Hello,
- My application is a spring boot microservices application which uses Mongo DB as its database.
- Our MongoDB cluster is a 5 server replica set.
- We are currently using Spring Boot Version 2.3.2 which comes bundled with spring-boot-actuator-2.3.2 and spring-data-mongodb-3.0.2. Our mongo DB Java drivers belong to version 4.0.5
- Following is the connection string we are using to connect to database -
`mongodb://${USERNAME}:${PASSWORD}@{PrimaryServerHostAndPort},{SecondaryServer1HostAndPort},{SecondaryServer2HostAndPort},{SecondaryServer3HostAndPort},{SecondaryServer4HostAndPort}/?replicaSet={replicaSetName}&authMechanism=PLAIN&authSource=$external;maxIdleTimeMS=60000;maxPoolSize=100;waitQueueMultiple=100'`
-
There is a consistent health check on our application instances which internally also check whether Mongo DB is up and running or not.
-
We are intermittently noticing the below error in our logs in our development environments.
2021-07-01 23:08:02.647 WARN 1 — [ XNIO-1 task-1] o.s.b.a.m.MongoHealthIndicator : MongoDB health check failed
org.springframework.data.mongodb.UncategorizedMongoDbException: Prematurely reached end of stream; nested exception is com.mongodb.MongoSocketReadException: Prematurely reached end of stream at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:133) ~[spring-data-mongodb-3.0.2.RELEASE.jar!/:3.0.2.RELEASE] at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2874) ~[spring-data-mongodb-3.0.2.RELEASE.jar!/:3.0.2.RELEASE] at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:541) ~[spring-data-mongodb-3.0.2.RELEASE.jar!/:3.0.2.RELEASE] at org.springframework.data.mongodb.core.MongoTemplate.executeCommand(MongoTemplate.java:457) ~[spring-data-mongodb-3.0.2.RELEASE.jar!/:3.0.2.RELEASE] at org.springframework.boot.actuate.mongo.MongoHealthIndicator.doHealthCheck(MongoHealthIndicator.java:46) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:85) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:44) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:99) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:110) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:96) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:74) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:61) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:71) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:60) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_291] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_291] at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282) ~[spring-core-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:77) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:305) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:388) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_291] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_291] at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) ~[spring-webmvc-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878) ~[spring-webmvc-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792) ~[spring-webmvc-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) ~[spring-webmvc-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) ~[spring-webmvc-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:645) ~[javax.servlet-api-4.0.1.jar!/:4.0.1] at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) ~[javax.servlet-api-4.0.1.jar!/:4.0.1] at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93) ~[spring-boot-actuator-2.3.2.RELEASE.jar!/:2.3.2.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) ~[spring-web-5.2.9.RELEASE.jar!/:5.2.9.RELEASE] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at wf.base.concurrent.ExecutionContextFilter.lambda$doFilter$0(ExecutionContextFilter.java:33) ~[wf-base-1.8.0.jar!/:1.8.0] at wf.base.concurrent.ExecutionContext.lambda$execute$3(ExecutionContext.java:199) ~[wf-base-1.8.0.jar!/:1.8.0] at wf.base.concurrent.ExecutionContext.execute(ExecutionContext.java:218) [wf-base-1.8.0.jar!/:1.8.0] at wf.base.concurrent.ExecutionContext.execute(ExecutionContext.java:198) [wf-base-1.8.0.jar!/:1.8.0] at wf.base.concurrent.ExecutionContextFilter.doFilter(ExecutionContextFilter.java:31) [wf-base-1.8.0.jar!/:1.8.0] at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) [undertow-core-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) [undertow-core-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) [undertow-servlet-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370) [undertow-core-2.1.3.Final.jar!/:2.1.3.Final] at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-2.1.3.Final.jar!/:2.1.3.Final] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [jboss-threads-3.1.0.Final.jar!/:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019) [jboss-threads-3.1.0.Final.jar!/:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558) [jboss-threads-3.1.0.Final.jar!/:3.1.0.Final] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449) [jboss-threads-3.1.0.Final.jar!/:3.1.0.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291] Caused by: com.mongodb.MongoSocketReadException: Prematurely reached end of stream at com.mongodb.internal.connection.SocketStream.read(SocketStream.java:112) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:579) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:444) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:298) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:258) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:500) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:71) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:224) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:202) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:118) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:110) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:343) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:334) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.operation.CommandOperationHelper.executeCommandWithConnection(CommandOperationHelper.java:220) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.operation.CommandOperationHelper$5.call(CommandOperationHelper.java:206) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:462) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:203) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.operation.CommandOperationHelper.executeCommand(CommandOperationHelper.java:198) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.internal.operation.CommandReadOperation.execute(CommandReadOperation.java:58) ~[mongodb-driver-core-4.0.5.jar!/:?] at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:190) ~[mongodb-driver-sync-4.0.5.jar!/:?] at com.mongodb.client.internal.MongoDatabaseImpl.executeCommand(MongoDatabaseImpl.java:194) ~[mongodb-driver-sync-4.0.5.jar!/:?] at com.mongodb.client.internal.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:163) ~[mongodb-driver-sync-4.0.5.jar!/:?] at com.mongodb.client.internal.MongoDatabaseImpl.runCommand(MongoDatabaseImpl.java:158) ~[mongodb-driver-sync-4.0.5.jar!/:?] at org.springframework.data.mongodb.core.MongoTemplate.lambda$executeCommand$2(MongoTemplate.java:457) ~[spring-data-mongodb-3.0.2.RELEASE.jar!/:3.0.2.RELEASE] at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:539) ~[spring-data-mongodb-3.0.2.RELEASE.jar!/:3.0.2.RELEASE] ... 93 more
Can you please advise how do we modify our connection string to resolve this health check issue ?
Regards,
Prateek