Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector

I am getting the following error

Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}.

Client view of cluster state is {type=REPLICA_SET, servers=[
  {address=mongo01:27017, type=UNKNOWN, state=CONNECTING*},
  {address=mongo02:27017, type=REPLICA_SET_SECONDARY, roundTripTime=2.5 ms, state=CONNECTED},
  {address=mongo03:27017, type=REPLICA_SET_SECONDARY, roundTripTime=1.1 ms, state=CONNECTED}
];

using spring data mongo

its a multitenant environment
change/create the mongoclient connection per request. Database/user/password change as tenant request.

Issue is intermittent and keeps happening randomly.

Hi @Kushal_Somkuwar,

Looking at the logging message - the driver is having trouble connecting to the primary node. The client view of the cluster state shows the state as seen from the Java driver (which spring data uses underneath).

Connecting appears to take longer than the 30s timeout to connect to the mongo01 node. So things to check are the server logs - has there been a change in the replicaset topology - eg new primary and secondaries? Also have there been any networking issues between the application and the mongodb nodes?

Ross

1 Like

Hi @Ross_Lawley ,

so application connection with mongodb works fine initially but if the data json is big than it become slow for application and start showing the above error.

we have checked the server logs there so no such of connection and I primary node is running fine there is no issue with cluster as well we suspect the same but why would it connect initially if there is a network issue.

Kushal

1 Like

Hi @Kushal_Somkuwar,

If the driver initially connects then later cannot connect, then that points to some networking issue. It may be intermittent and not happen all the time. But from the drivers perspective it is timing out trying to select a server. The log message just shows you the current view of the servers after selection has failed, which is it is in the process of connecting to the mongo1 node.

Ross

1 Like

Hi @Ross_Lawley

I also think the same but we are not getting any exception on mongodb server logs and application just get that error. Is there a way to identify what is exactly is the network issue ?

@Kushal_Somkuwar We are facing a similar issue and we are connecting to mongo from Google Cloud.
Did you find out what was the issue? If so, how did you resolve it?

@Ross_Lawley Kindly help. Below is the stack trace
“No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=xxxx, type=UNKNOWN, state=CONNECTING, exception={java.lang.OutOfMemoryError: Java heap space}}, ServerDescription{address=xxx, type=UNKNOWN, state=CONNECTING, exception={java.lang.OutOfMemoryError: Java heap space}}, ServerDescription{address=xxxx, type=REPLICA_SET_SECONDARY, state=CONNECTED, ok=true, minWireVersion=0, maxWireVersion=8, maxDocumentSize=16777216, logicalSessionTimeoutMinutes=30, roundTripTimeNanos=1401391508, setName=xxx’, canonicalAddress=xxx, hosts=[xxx], passives=[], arbiters=[], primary=xxxx’, tagSet=TagSet{[Tag{name=‘nodeType’, value=‘ELECTABLE’}, Tag{name=‘provider’, value=‘GCP’}, Tag{name=‘region’, value=‘EASTERN_US’}, Tag{name=‘workloadType’, value=‘OPERATIONAL’}]}, electionId=null, setVersion=6, lastWriteDate=Wed Apr 13 03:33:15 UTC 2022, lastUpdateTimeNanos=6819537012507}]}. Waiting for 30000 ms before timing out”

Hi @Sandhya_Kripalani,

I noticed this java.lang.OutOfMemoryError: Java heap space in the error message.

I’ve never seen that reported before in such a way (as part of the No server chosen exception) but its a sign that your JVM needs more resources.

Ross

1 Like

@Sandhya_Kripalani you will need to check mongoclient and the number of connections it is making with MongoDB it should be on the higher side. This issue comes when your application keep creating mongoclient object and not closing the existing mongoclient connections. Hope that will help

Hello @Ross_Lawley ,

I’m having the same issue here, but I use reactive mongodb in a spring boot application.

This is my logs messages:

org.springframework.data.mongodb.core.ReactiveMongoTemplate - Streaming aggregation: [{ "$match" : { "type" : { "$in" : ["INACTIVE_SITE", "DEVICE_NOT_BILLED", "NOT_REPLYING_POLLING", "MISSING_KEY_TECH_INFO", "MISSING_SITE", "ACTIVE_CIRCUITS_INACTIVE_RESOURCES", "INCONSISTENT_STATUS_VALUES", "TEST_SUPERVISION_RANGE"]}}}, { "$project" : { "extractionDate" : 1, "_id" : 0}}, { "$group" : { "_id" : null, "result" : { "$max" : "$extractionDate"}}}, { "$project" : { "_id" : 0}}] in collection kpi
org.mongodb.driver.cluster - No server chosen by com.mongodb.reactivestreams.client.internal.ClientSessionHelper$$Lambda$1183/0x000000080129dd60@292878db from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
luster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=REPLICA_SET, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, ServerDescription{address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
com.obs.dqsc.interceptor.GlobalExceptionHandler - Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, {address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, {address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]
org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver - Resolved [org.springframework.dao.DataAccessResourceFailureException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, {address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.235.79.213:7915, type=UNKNOWN, state=CONNECTING}, {address=10.235.79.89:7915, type=UNKNOWN, state=CONNECTING}]]
org.springframework.web.servlet.DispatcherServlet - Completed 500 INTERNAL_SERVER_ERROR

And this is my mongodb config:

    @Override
    public MongoClient reactiveMongoClient() {
        ConnectionString connectionString = new ConnectionString(
                "mongodb://" + username + ":" + password +
                        "@" + host1 + ":" + port
                        + "," + host2 + ":" + port +
                        "/?authSource=" + authSource +
                        "&replicaSet=" + replica
        );

        MongoClientSettings mongoClientSettings = MongoClientSettings.builder()
                .applyToSslSettings(builder ->
                        builder.enabled(true).invalidHostNameAllowed(true)
                )
                .applyToConnectionPoolSettings(
                        builder -> builder.minSize(10)
                                .maxSize(100)
                                .maxWaitTime(8,TimeUnit.MINUTES)
                )
                .applyToSocketSettings(builder -> builder.applySettings(SocketSettings.builder().connectTimeout(5,TimeUnit.MINUTES).readTimeout(5,TimeUnit.MINUTES).build()))
                .applyConnectionString(connectionString)
                .build();

        return MongoClients.create(mongoClientSettings);
    }

I tried to use different networks to avoid this problem, bit it always gives me the same error.
Could you help me please?

Hi Fellow developers,

I have recently started seeing this issues again as soon as I switched to ReactMongoTemplate.
In my case I am doing multiple find operations as part of flux streams and the connection does not go through after first few subscriptions. It also many times bring the whole cluster down.

Please tell me if someone managed to fix this issue.

1 Like

Facing similar issue, any solution?

Did you find the root cause as similar issue i am facing.

Hi @sunny_tyagi welcome to the community!

By “similar issue”, I assume you see an error message that looks like this:

Is this correct?

If yes, then this is a typical timeout error that is shared across all official drivers, where it tries to connect to the primary member of a replica set, and gives up when it cannot find one after 30 seconds (default). See MongoClient settings serverSelectionTimeout.

Typically this is caused by network issues where the driver cannot reach the servers. It could be caused by network partition, security settings that prevent the client to reach the server (e.g. IP whitelisting issues), DNS issues, blocked port, among many others.

If your app used to be able to connect without issues and now it cannot, then perhaps there is something different now. Things to check may include whether you’re connecting from the same network as before, whether there are DNS changes to the server, whether security settings was changed in the server, or any other network reachability issues that wasn’t there before.

If you have checked everything and all seems to be in order, I would suggest you to consult your network administrator to troubleshoot your connectivity issues.

Just to be complete, if anyone else is encountering this error in the future, it’s best to create a new topic describing the exact error encountered, along with more specific information such as MongoDB version, driver version, topology description, and the complete error message, since everyone’s situation is different and the same error may be the result of two very different causes :slight_smile:

Best regards
Kevin

It was fixed my increasing the socket settings to the approximate time taken for cluster to come up again or reelect a leader. Also increase maxSize to like 10, spring could not handle 100 connections adequately. Now process is a bit slow but still more reliable.

could you please add the code snippet ? i am also facing the same issue

Hi, my configuration for the services that is using the mongodb service, using application.properties, to contact it with the uri mongodb://mongodbservice:27015/dbname. I found in the logs it is calling localhost:27017. Any clue?