Mongos Couldn't get a connection within the time limit

When client send request i see below error in mongos and client also gets timeout error

mongos error:

Dec 12 22:20:37 ip-10-255-161-52 docker[5830]: {“t”:{"$date":“2022-12-12T22:20:37.548+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“conn75710”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“rs4”,“host”:“ip-10-XXX-XXX-31.us-east-2.compute.internal:27017”,“error”:{“code”:202,“codeName”:“NetworkInterfaceExceededTimeLimit”,“errmsg”:“Couldn’t get a connection within the time limit of -31ms”},“action”:{“dropConnections”:false,“requestImmediateCheck”:false,“outcome”:{“host”:“ip-10-XXX-162-31.us-east-2.compute.internal:27017”,“success”:false,“errorMessage”:“NetworkInterfaceExceededTimeLimit: Couldn’t get a connection within the time limit of -31ms”}}}}
Dec 12 22:20:37 ip-10-XXX-XXX-52 docker[5830]: {“t”:{"$date":“2022-12-12T22:20:37.548+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“conn75722”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“rs4”,“host”:“ip-10-XXX-XXX-31.us-east-2.compute.internal:27017”,“error”:{“code”:202,“codeName”:“NetworkInterfaceExceededTimeLimit”,“errmsg”:“Couldn’t get a connection within the time limit of -31ms”},“action”:{“dropConnections”:false,“requestImmediateCheck”:false,“outcome”:{“host”:“ip-10-XXX-XXX-31.us-east-2.compute.internal:27017”,“success”:false,“errorMessage”:“NetworkInterfaceExceededTimeLimit: Couldn’t get a connection within the time limit of -31ms”}}}}

Client gets this error on thier logs:

Caused by: Job aborted due to stage failure: Query failed with error code 202 and error message ‘Couldn’t get a connection within the time limit of 663ms’ on server 10.XXX.XXX.78:27017
Caused by: MongoQueryException: Query failed with error code 202 and error message ‘Couldn’t get a connection within the time limit of 663ms’ on server 10.XXX.XXX.78:27017

Hi @giribabu_venugopal and welcome to the MongoDB community forum!!

To understand the issue further, could you help me with the following information:

  1. What is the topology for your deployment? Is this a replica set or a sharded cluster or a stand alone deployment?
  2. The output of sh.status() if this is a sharded cluster, the output of rs.status() from all the shards, and also from the config servers?
  3. From the log lines, this seems to be deployed on AWS, could you help with the steps or documentation followed to deploy database?
  4. The version details for MongoDB and docker image details.
  5. The message Host failed in replica set seems to imply connectivity issues between nodes. Have you done any connectivity tests to ensure the nodes can reach each other?
  6. Was this setup working in the past and are you recently seeing the error? Was something changed in between?
  7. Are you directly able to connect to the deployment through mongosh?

Best Regards
Aasawari

1 Like