Hi,
I have one config and query router servers and 2 replicat sets - for each 1 primary and 1 secondary
Total 5 servers.
My MongoDB were running fine past couple of years. Suddenly, could not connect one day.
Found filesystem reached 100%. So cleared old log file and expanded space as well.
When restarting mongod in repicat set1 primary server, admin restarted it with root user instead of mongo user but ut did not restarted successfully. Found a mongod running process with root user, so I killed that process. Then log shows “Uncleaned shutdown detected”, So I cleared lock file and ran repair command.
After repair completed, when restarted mongod with mongo user, it shows permission issue for WiredTiger.turtle file files in journal folder. Those were owned by root, so I changed owner to mongo user.
Now when I restart mongod, its shows below error -
{"t":{"$date":"2023-11-17T11:38:57.268-06:00"},"s":"I", "c":"NETWORK", "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"ShardReplSet1","host":"172.22.145.156:27018","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to 172.22.145.156:27018 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":false,"outcome":{"host":"172.22.145.156:27018","success":false,"errorMessage":"HostUnreachable: Error connecting to 172.22.145.156:27018 :: caused by :: Connection refused"}}}}
{"t":{"$date":"2023-11-17T11:38:57.768-06:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Connecting","attr":{"hostAndPort":"172.22.145.156:27018"}}
IP 172.22.145.156 is of the same server where I am facing issue (replicaset1 primary1).
All other server’s replicaset1 secoundary1 and replicaset2 secoundary1 and replicaset2 secoundary2 mongod services are running fine. Also config and queryrouter mongod and mongos are also running fine.
Anyone please help me to bring my replicaset1primary1 server up