Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector

I am getting the following error

Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}.

Client view of cluster state is {type=REPLICA_SET, servers=[
  {address=mongo01:27017, type=UNKNOWN, state=CONNECTING*},
  {address=mongo02:27017, type=REPLICA_SET_SECONDARY, roundTripTime=2.5 ms, state=CONNECTED},
  {address=mongo03:27017, type=REPLICA_SET_SECONDARY, roundTripTime=1.1 ms, state=CONNECTED}
];

using spring data mongo

its a multitenant environment
change/create the mongoclient connection per request. Database/user/password change as tenant request.

Issue is intermittent and keeps happening randomly.

Hi @Kushal_Somkuwar,

Looking at the logging message - the driver is having trouble connecting to the primary node. The client view of the cluster state shows the state as seen from the Java driver (which spring data uses underneath).

Connecting appears to take longer than the 30s timeout to connect to the mongo01 node. So things to check are the server logs - has there been a change in the replicaset topology - eg new primary and secondaries? Also have there been any networking issues between the application and the mongodb nodes?

Ross

1 Like

Hi @Ross_Lawley ,

so application connection with mongodb works fine initially but if the data json is big than it become slow for application and start showing the above error.

we have checked the server logs there so no such of connection and I primary node is running fine there is no issue with cluster as well we suspect the same but why would it connect initially if there is a network issue.

Kushal

Hi @Kushal_Somkuwar,

If the driver initially connects then later cannot connect, then that points to some networking issue. It may be intermittent and not happen all the time. But from the drivers perspective it is timing out trying to select a server. The log message just shows you the current view of the servers after selection has failed, which is it is in the process of connecting to the mongo1 node.

Ross

Hi @Ross_Lawley

I also think the same but we are not getting any exception on mongodb server logs and application just get that error. Is there a way to identify what is exactly is the network issue ?

@Kushal_Somkuwar We are facing a similar issue and we are connecting to mongo from Google Cloud.
Did you find out what was the issue? If so, how did you resolve it?

@Ross_Lawley Kindly help. Below is the stack trace
“No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=xxxx, type=UNKNOWN, state=CONNECTING, exception={java.lang.OutOfMemoryError: Java heap space}}, ServerDescription{address=xxx, type=UNKNOWN, state=CONNECTING, exception={java.lang.OutOfMemoryError: Java heap space}}, ServerDescription{address=xxxx, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1401391508, setName=xxx’, canonicalAddress=xxx, hosts=[xxx], passives=[], arbiters=[], primary=xxxx’, tagSet=TagSet{[Tag{name=‘nodeType’, value=‘ELECTABLE’}, Tag{name=‘provider’, value=‘GCP’}, Tag{name=‘region’, value=‘EASTERN_US’}, Tag{name=‘workloadType’, value=‘OPERATIONAL’}]}, electionId=null, setVersion=6, lastWriteDate=Wed Apr 13 03:33:15 UTC 2022, lastUpdateTimeNanos=6819537012507}]}. Waiting for 30000 ms before timing out”

Hi @Sandhya_Kripalani,

I noticed this java.lang.OutOfMemoryError: Java heap space in the error message.

I’ve never seen that reported before in such a way (as part of the No server chosen exception) but its a sign that your JVM needs more resources.

Ross

1 Like

@Sandhya_Kripalani you will need to check mongoclient and the number of connections it is making with MongoDB it should be on the higher side. This issue comes when your application keep creating mongoclient object and not closing the existing mongoclient connections. Hope that will help

Hello @Ross_Lawley ,

I’m having the same issue here, but I use reactive mongodb in a spring boot application.

This is my logs messages:

org.springframework.data.mongodb.core.ReactiveMongoTemplate - Streaming aggregation: [{ "$match" : { "type" : { "$in" : ["INACTIVE_SITE", "DEVICE_NOT_BILLED", "NOT_REPLYING_POLLING", "MISSING_KEY_TECH_INFO", "MISSING_SITE", "ACTIVE_CIRCUITS_INACTIVE_RESOURCES", "INCONSISTENT_STATUS_VALUES", "TEST_SUPERVISION_RANGE"]}}}, { "$project" : { "extractionDate" : 1, "_id" : 0}}, { "$group" : { "_id" : null, "result" : { "$max" : "$extractionDate"}}}, { "$project" : { "_id" : 0}}] in collection kpi
org.mongodb.driver.cluster - No server chosen by com.mongodb.reactivestreams.client.internal.ClientSessionHelper$$Lambda$1183/0x000000080129dd60@292878db from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
luster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
com.obs.dqsc.interceptor.GlobalExceptionHandler - Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, {address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, {address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]
org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver - Resolved [org.springframework.dao.DataAccessResourceFailureException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, {address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, {address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]]
org.springframework.web.servlet.DispatcherServlet - Completed 500 INTERNAL_SERVER_ERROR

And this is my mongodb config:

    @Override
    public MongoClient reactiveMongoClient() {
        ConnectionString connectionString = new ConnectionString(
                "mongodb://" + username + ":" + password +
                        "@" + host1 + ":" + port
                        + "," + host2 + ":" + port +
                        "/?authSource=" + authSource +
                        "&replicaSet=" + replica
        );

        MongoClientSettings mongoClientSettings = MongoClientSettings.builder()
                .applyToSslSettings(builder ->
                        builder.enabled(true).invalidHostNameAllowed(true)
                )
                .applyToConnectionPoolSettings(
                        builder -> builder.minSize(10)
                                .maxSize(100)
                                .maxWaitTime(8,TimeUnit.MINUTES)
                )
                .applyToSocketSettings(builder -> builder.applySettings(SocketSettings.builder().connectTimeout(5,TimeUnit.MINUTES).readTimeout(5,TimeUnit.MINUTES).build()))
                .applyConnectionString(connectionString)
                .build();

        return MongoClients.create(mongoClientSettings);
    }

I tried to use different networks to avoid this problem, bit it always gives me the same error.
Could you help me please?

Hi Fellow developers,

I have recently started seeing this issues again as soon as I switched to ReactMongoTemplate.
In my case I am doing multiple find operations as part of flux streams and the connection does not go through after first few subscriptions. It also many times bring the whole cluster down.

Please tell me if someone managed to fix this issue.