Hi, @Chris_N_A3,
This is an excellent root cause analysis. Very interesting about SO_SNDBUF and SO_RCVBUF . According to the API docs for Socket.ReceiveBufferSize, the default is 8K.
/// <summary>Gets or sets a value that specifies the size of the receive buffer of the <see cref="T:System.Net.Sockets.Socket" />.</summary>
/// <exception cref="T:System.Net.Sockets.SocketException">An error occurred when attempting to access the socket.</exception>
/// <exception cref="T:System.ObjectDisposedException">The <see cref="T:System.Net.Sockets.Socket" /> has been closed.</exception>
/// <exception cref="T:System.ArgumentOutOfRangeException">The value specified for a set operation is less than 0.</exception>
/// <returns>An <see cref="T:System.Int32" /> that contains the size, in bytes, of the receive buffer. The default is 8192.</returns>
public int ReceiveBufferSize { get; set; }
According to MSDN docs for Socket.ReceiveBufferSize, the size is OS dependent:
An Int32 that contains the size, in bytes, of the receive buffer. The default value depends on the operating system.
The same is true for Socket.ReceiveBufferSize where API docs say 8K and MSDN says OS dependent.
In the .NET/C# Driver, we “increase” the TCP send and receive buffer sizes to 64KB in both MongoDefaults and the TcpStreamSettings constructor. Either the original documentation was incorrect regarding the 8K default or operating systems have adjusted their defaults upwards. MacOS Sonoma and Ubuntu 20.04 use 128KB default for both. (No rhyme or reason why I chose those particular OSes other than I happened to have them handy.)
You mentioned that adjusting the defaults in the driver still resulted in slow performance. Please confirm that you used code similar to the following:
var settings = MongoClientSettings.FromConnectionString(MONGODB_URI);
settings.ClusterConfigurator =
cc =>
cc.ConfigureTcp(tcp => tcp.With(receiveBufferSize: 128 * 1204, sendBufferSize: 128 * 1024));
var client = new MongoClient(settings);
Given that you are reading ~158K documents back, I suspect that your performance will be affected most by SO_RCVBUF. I notice that you’re projecting out the following fields: { _id:0, Eid:1, Data.Flags:1, Data.Lcn:1, Data.Rnd.Diff:1}. I can tell that Eid is an ObjectId, but what are the other data types (and sizes if they’re arrays or strings)? I’m trying to figure out how many 16MB batches the results would be split into. Even if it fit into a single 16MB batch, that batch would be transmitted as multiple TCP packets which would be affected by SO_RCVBUF.
We are going to try to reproduce the behaviour by retrieving larger payloads from an Atlas cluster. (My initial repro attempt was against a local cluster.) Knowing your approximate data sizes of the projected documents would be helpful in our investigation. It would also be useful to know the minimum/default/maximum values for SO_RCVBUF and SO_SNDBUF. Here are the values from my Ubuntu 20.04 VM:
$ cat /proc/sys/net/ipv4/tcp_rmem
4096 131072 6291456
$ cat /proc/sys/net/ipv4/tcp_wmem
4096 16384 4194304
Thanks in advance for your continued collaboration with investigating this issue.
Sincerely,
James