Host Failed in replica set

Ops and Admin Sharding
1

I am not able to access MongoDB. While the mongodb is running, i am not able to access it. Below is the log:

“ctx”:“UpdateReplicaSetOnConfigServer”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“config”,“host”:“:port”,“success”:false,“errorMessage”:“NetworkInterfaceExceededTimeLimit: Couldn’t get a connection within the time limit”}}}}

“ctx”:“UserCacheInvalidator”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“config”,“host”:“20.55.103.161:27019"error”:{“code”:202,“codeName”:“NetworkInterfaceExceededTimeLimit”,“errmsg”:“Couldn’t get a connection within the time limit”},“action”:{“dropConnections”:false,“requestImmediateCheck”:false,“outcome”{“host”:“20.55.103.161:27019”,“success”:false,“errorMessage”:“NetworkInterfaceExceededTimeLimit: Couldn’t get a connection within the time limit”}}}}
“ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM Topology Change”,“attr”:{“replicaSet”:“config”,"newTopologyDescription { id: "2136769a-5c31-45f9-be7c-54f54a3c2bc8", topologyType: "ReplicaSetNoPrimary", servers: { 20.55.103.161:27019: { address: "20.55.103.161:27019", type: "Unknown", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: "config", compatible: true, maxSetVersion: 1, maxElectionId: ObjectI

It was working few days ago. Didn’t make any changes. But all of a sudden i am now not able to access the DB.
My mongos server and config server are in same virtual machine.
Any help or guidance will be highly appreciated. thank you.

2

Has anyone found the answer for this? This is most similar to the error I am seeing in our deployment. I am seeing a steady stream of both the “Host failed in replica set” and “RSM Topology Change” messages.

From our environment, I can open a mongo shell directly within a mongodb shard pod and execute a query. If I attempt to do the same query against the mongos service, however, I get “Authentication failed”.

3

Welcome to the MongoDB Community Forums @Lawson_Kendall!

Can you please start a new discussion topic with more details of your environment including:

  • Specific version of MongoDB server and O/S.
  • Are all components of your cluster (mongod, mongos) using identical versions? If not, please mention any variations.
  • Steps to reproduce the error you are encountering.
  • Was this deployment working previously or is this a fresh install?
  • Any other context on recent changes (for example, did issues start occurring after a server version upgrade?).

Since you mention “pod”, I’m guessing your deployment is using k8s.

It sounds like you can connect to a shard replica set directly, so two aspects you could verify are:

  • That the hostnames in your replica set and sharded cluster configuration are resolvable (and can be connected to) from all members of your cluster.

  • That you have configured Keyfile authentication for the members of your sharded cluster to be able to communicate with each other.

Regards,
Stennie