[.NET Mongo Driver] Connections dropping unexpectedly when under load

Background:
MongoDb provider: IBM Cloud
.NET mongo driver version - 2.12.1 (problem persisted with earlier version too)
Configuration:
Mongo Client is connected to a 2 server replica set (there are 3 more hidden ones, probably managed by ibm cloud)
Max connection pool size is 400
TcpKeep alive is enabled and the timeout is set to 120 seconds (from azure recommendations, but these problems exist in production and when running the application locally)
Mongo client is configured with these settings and registered as a singleton in our DI container

At my workplace we are facing a really peculiar issue. Whenever our application is under load we are getting dropped connections to mongodb.
From our application we can see these exceptions

MongoDB.Driver.MongoConnectionException:
       at MongoDB.Driver.Core.Connections.BinaryConnection+<OpenHelperAsync>d__51.MoveNext (MongoDB.Driver.Core, Version=2.12.1.0, Culture=neutral, PublicKeyToken=null)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at MongoDB.Driver.Core.Servers.Server+<GetChannelAsync>d__34.MoveNext (MongoDB.Driver.Core, Version=2.12.1.0, Culture=neutral, PublicKeyToken=null)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at MongoDB.Driver.Core.Operations.RetryableReadContext+<InitializeAsync>d__20.MoveNext (MongoDB.Driver.Core, Version=2.12.1.0, Culture=neutral, PublicKeyToken=null)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at MongoDB.Driver.Core.Operations.RetryableReadContext+<CreateAsync>d__1.MoveNext (MongoDB.Driver.Core, Version=2.12.1.0, Culture=neutral, PublicKeyToken=null)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at MongoDB.Driver.Core.Operations.FindOperation`1+<ExecuteAsync>d__120.MoveNext (MongoDB.Driver.Core, Version=2.12.1.0, Culture=neutral, PublicKeyToken=null)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at MongoDB.Driver.OperationExecutor+<ExecuteReadOperationAsync>d__3`1.MoveNext (MongoDB.Driver, Version=2.12.1.0, Culture=neutral, PublicKeyToken=null)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at MongoDB.Driver.MongoCollectionImpl`1+<ExecuteReadOperationAsync>d__98`1.MoveNext (MongoDB.Driver, Version=2.12.1.0, Culture=neutral, PublicKeyToken=null)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at MongoDB.Driver.MongoCollectionImpl`1+<UsingImplicitSessionAsync>d__106`1.MoveNext (MongoDB.Driver, Version=2.12.1.0, Culture=neutral, PublicKeyToken=null)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at MongoDB.Driver.IAsyncCursorSourceExtensions+<FirstOrDefaultAsync>d__5`1.MoveNext (MongoDB.Driver.Core, Version=2.12.1.0, Culture=neutral, PublicKeyToken=null)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at -
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at -
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at -
    Inner exception System.IO.IOException handled at MongoDB.Driver.Core.Connections.BinaryConnection+<OpenHelperAsync>d__51.MoveNext:
       at System.Net.Security.SslStream+<ReceiveBlobAsync>d__172`1.MoveNext (System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Net.Security.SslStream+<ForceAuthenticationAsync>d__171`1.MoveNext (System.Net.Security, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at MongoDB.Driver.Core.Connections.SslStreamFactory+<CreateStreamAsync>d__4.MoveNext (MongoDB.Driver.Core, Version=2.12.1.0, Culture=neutral, PublicKeyToken=null)
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
       at MongoDB.Driver.Core.Connections.BinaryConnection+<OpenHelperAsync>d__51.MoveNext (MongoDB.Driver.Core, Version=2.12.1.0, Culture=neutral, PublicKeyToken=null)

Most of the time not one, but a few connections get dropped at the same time.
If we enable verbose logging in mongodb - we find these log entries:

May 5 14:32:33 <db url> 2021-05-05T11:32:32.919047762Z stdout F 2021-05-05T11:32:32.917+0000 D2 NETWORK  [conn70191] Session from 172.30.205.189:50934 encountered a network error during SourceMessage: SocketException: stream truncated
May 5 14:32:33 <db url> 2021-05-05T11:32:32.919077528Z stdout F 2021-05-05T11:32:32.917+0000 I  NETWORK  [conn70191] end connection 172.30.205.189:50934 (410 connections now open)
May 5 14:32:33 <db url> 2021-05-05T11:32:32.919128227Z stdout F 2021-05-05T11:32:32.917+0000 D3 NETWORK  [conn70191] Cancelling outstanding I/O operations on connection to 172.30.205.189:50934
May 5 14:32:33 <db url> 2021-05-05T11:32:32.953175166Z stdout F 2021-05-05T11:32:32.952+0000 D3 NETWORK  [conn70281] Starting server-side compression negotiation
May 5 14:32:33 <db url> 2021-05-05T11:32:32.953386437Z stdout F 2021-05-05T11:32:32.953+0000 D3 NETWORK  [conn70281] Compression negotiation not requested by client
May 5 14:32:33 <db url> 2021-05-05T11:32:32.954950367Z stdout F 2021-05-05T11:32:32.954+0000 D2 NETWORK  [conn70380] Session from 172.30.63.237:52270 encountered a network error during SourceMessage: SocketException: stream truncated
May 5 14:32:33 <db url> 2021-05-05T11:32:32.95559245Z stdout F 2021-05-05T11:32:32.955+0000 I  NETWORK  [conn70380] end connection 172.30.63.237:52270 (409 connections now open)
May 5 14:32:33 <db url> 2021-05-05T11:32:32.955626373Z stdout F 2021-05-05T11:32:32.955+0000 D2 NETWORK  [conn70342] Session from 172.30.205.189:54648 encountered a network error during SourceMessage: SocketException: stream truncated
May 5 14:32:33 <db url> 2021-05-05T11:32:32.955796683Z stdout F 2021-05-05T11:32:32.955+0000 D2 NETWORK  [conn70387] Session from 172.30.63.237:52278 encountered a network error during SourceMessage: SocketException: stream truncated
May 5 14:32:33 <db url> 2021-05-05T11:32:32.955972883Z stdout F 2021-05-05T11:32:32.955+0000 D2 NETWORK  [conn70393] Session from 172.30.63.237:53290 encountered a network error during SourceMessage: SocketException: stream truncated
May 5 14:32:33 <db url> 2021-05-05T11:32:32.956105612Z stdout F 2021-05-05T11:32:32.955+0000 I  NETWORK  [conn70393] end connection 172.30.63.237:53290 (406 connections now open)
May 5 14:32:33 <db url> 2021-05-05T11:32:32.956157161Z stdout F 2021-05-05T11:32:32.955+0000 D3 NETWORK  [conn70393] Cancelling outstanding I/O operations on connection to 172.30.63.237:53290
May 5 14:32:33 <db url> 2021-05-05T11:32:32.956187301Z stdout F 2021-05-05T11:32:32.955+0000 D3 NETWORK  [conn70380] Cancelling outstanding I/O operations on connection to 172.30.63.237:52270
May 5 14:32:33 <db url> 2021-05-05T11:32:32.956322652Z stdout F 2021-05-05T11:32:32.956+0000 D2 NETWORK  [conn70370] Session from 172.30.205.189:54684 encountered a network error during SourceMessage: SocketException: stream truncated
May 5 14:32:33 <db url> 2021-05-05T11:32:32.95635403Z stdout F 2021-05-05T11:32:32.956+0000 I  NETWORK  [conn70370] end connection 172.30.205.189:54684 (405 connections now open)
May 5 14:32:33 <db url> 2021-05-05T11:32:32.956539989Z stdout F 2021-05-05T11:32:32.956+0000 D3 NETWORK  [conn70370] Cancelling outstanding I/O operations on connection to 172.30.205.189:54684
May 5 14:32:33 <db url> 2021-05-05T11:32:32.956881863Z stdout F 2021-05-05T11:32:32.955+0000 I  NETWORK  [conn70342] end connection 172.30.205.189:54648 (408 connections now open)
May 5 14:32:33 <db url> 2021-05-05T11:32:32.957024776Z stdout F 2021-05-05T11:32:32.956+0000 D3 NETWORK  [conn70342] Cancelling outstanding I/O operations on connection to 172.30.205.189:54648
May 5 14:32:33 <db url> 2021-05-05T11:32:32.95715975Z stdout F 2021-05-05T11:32:32.956+0000 D2 NETWORK  [conn70378] Session from 172.30.205.189:54694 encountered a network error during SourceMessage: SocketException: stream truncated
May 5 14:32:33 <db url> 2021-05-05T11:32:32.957189634Z stdout F 2021-05-05T11:32:32.957+0000 I  NETWORK  [conn70378] end connection 172.30.205.189:54694 (404 connections now open)
May 5 14:32:33 <db url> 2021-05-05T11:32:32.957218206Z stdout F 2021-05-05T11:32:32.957+0000 D3 NETWORK  [conn70378] Cancelling outstanding I/O operations on connection to 172.30.205.189:54694
May 5 14:32:33 <db url> 2021-05-05T11:32:32.958131873Z stdout F 2021-05-05T11:32:32.957+0000 D2 NETWORK  [conn70383] Session from 172.30.205.189:54700 encountered a network error during SourceMessage: SocketException: stream truncated

I have managed to reproduce the problem with wireshark running, and after inspecting the lifetime of a connection which causes these exceptions I can see these things:

It looks like we are opening a new connection to the databse, it idles for about 5 seconds, and then we receive a connection close (FIN) packet from the server. This happens to at least a few connections at the same time. It might seems that this is a problem on our provider side, but I have my doubts, considering way larger deployments run on these cloud services without any problems.

If anyone has any ideas or recommendations about this, it would be really appreciated, because we are a dead end in our investigation.