We didn’t detach server from replica set, looks like we had a reboot happen on the 2 servers (P and S).
After the servers cam online 2 of them were sitting in refused state where on startup we have mongo.service that starts the mongo with config we have. Some reasons on network we cant see 27017 port is not showing up. so that caused replica to sync.
Servers were hosted on AWS Linux.
This was on a running server, these were running 2.5 years minimum and no issues, this is sudden from Jan 28th.
Connection refused: This is because mongo service is running but replica/mongo can’t able to reach the network because on start-up 27017 port is not opened by OS. We tried to reboot and did so much for some reasons OS is not starting / opening the port.