Hello,
in our MongoDB Cluster (3 mongos, 3 data-replicas, 3 config-replicas) sometime application catch errors like this
2020-09-17 16:25:47,2547 [48] () ERROR
An unhandled exception has occurred: An exception occurred while opening a connection to the server.
MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.
---> MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server.
---> System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
--- End of inner exception stack trace ---
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.GetResult(Int16 token)
at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
--- End of stack trace from previous location where exception was thrown ---
at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytesAsync(Stream stream, Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync()
--- End of inner exception stack trace ---
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync()
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBufferAsync(Int32 responseTo, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessageAsync(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
at MongoDB.Driver.Core.WireProtocol.CommandUsingQueryMessageWireProtocol`1.ExecuteAsync(IConnection connection, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.IsMasterHelper.GetResultAsync(IConnection connection, CommandWireProtocol`1 isMasterProtocol, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.ConnectionInitializer.InitializeConnectionAsync(IConnection connection, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at AspectCore.DynamicProxy.AspectActivator.InvokeTask[TResult](AspectActivatorContext activatorContext)
at Mango.Platform.ProductInventory.Controllers.v2.ProductController.GetProductsInternalAsync(ProductFilteredRequest request) in e:\\b\\Agent2\\13\\s\\Mango.Platform.ProductInventory\\Controllers\\v2\\ProductController.cs:line 342
at Mango.Platform.ProductInventory.Controllers.v2.ProductController.GetProductsPost(ProductFilteredRequest request) in e:\\b\\Agent2\\13\\s\\Mango.Platform.ProductInventory\\Controllers\\v2\\ProductController.cs:line 104
at Microsoft.AspNetCore.Mvc.Internal.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at System.Threading.Tasks.ValueTask`1.get_Result()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)
at Microsoft.AspNetCore.Builder.Extensions.MapWhenMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware.Invoke(HttpContext httpContext)
at Mango.Platform.Diagnostics.Middlewares.RequestResponseLoggingMiddleware.Invoke(HttpContext context)
at ServiceRegistry.AspCore.Middleware.MartiExecutionContextMiddleware.InvokeAsync(HttpContext context)
at ServiceRegistry.AspCore.Middleware.ServerStatisticsMiddleware.InvokeAsync(HttpContext context)
at Mango.Platform.Diagnostics.Middlewares.ExceptionHandlingMiddleware.Invoke(HttpContext context)
at this time in mongos-log (log level=3)
many timeouts like this
2020-09-17T16:25:47.094+0300 D NETWORK [TaskExecutorPool-0] Timer received error: CallbackCanceled: Callback was canceled
(complete logs attached).
When we connect directly to data-replicas - we have no prolem like this, problem is not reproduced.
So we need to investigate - what is the reason os this timeouts, when we working with mongos?
Application and mongos working on the same Windows-servers (Windows 2012 R2).
Data and config replicas working on others Linux-servers (4.1.12-124.32.1.el7uek.x86_64).
MongoDB version 4.0.12.
I’ve asked before in https://jira.mongodb.org/browse/SERVER-50942, but they recommended to ask in this community…
Please, help to find a reason of this timeouts…