we have 3 node mongodb cluster. Recently we had an issue where one of the mongodb node did not take connections causing intermittent issues to the application.
Reviewing the logs, we observed below errors.
socket errors for 9001 for multiple app instances.
2022-08-05T16:42:53.664+0000 [conn137308] SocketException handling request, closing client connection: 9001 socket exception [SEND_ERROR] server
connection to other two nodes failed.
2022-08-05T17:07:25.455+0000 [rsHealthPoll] couldn’t connect to prod-cirrusmongo-eu05: couldn’t connect to server prod-cirrusmongo-eu05-xxxxx27017 failed, connection attempt failed
2022-08-05T17:07:25.455+0000 [rsHealthPoll] warning: Failed to connect to 10.0.3.21:27017, reason: errno:106 Transport endpoint is already connected
syslog is showing below error.
[LIVE] root@prod-cirrusmongo-eu06 [/var/log]$ zcat syslog.5.gz | grep mongodb
Aug 5 16:42:52 prod-cirrusmongo-eu06 mongodb_exporter: E0805 16:42:52.923749 1760 connection.go:48] Cannot connect to server using url mongodb://localhost:27017: no reachable servers
The issue is finally resolved after restarting mongod service on the instance and taking connections now.
The node was primary earlier after restart it became secondary.
I would like to know the root cause of above errors to avoid the issue again.