Abnormalities in the entire cluster after a failover of one secondary member

Ops and Admin Sharding
,
1

Hi,

The settings of our Sharded Cluster are as follows.

  • 4 mongos
  • 1 CSRS (P-S-S Replica Set.)
  • 4 Shards (P-S-S Replica Set.)
  • writeConcern (w: 1)
  • readPreference (secondaryPrefered)
  • Operations are mainly focused on upsert.

After restarting a secondary member in one shard, the qps of the entire cluster drops to 0 for a short period of time.

Why does our entire cluster’s query throughput drop to zero? Is this normal behavior?

What settings should I look at to resolve this?

ddd
ddd1033×1013 208 KB

2

Did you see any interesting thing from mongodb logs or metrics?

3

@Kobe_W
Hi Kobe, Thank you for reply.
I am attaching logs from the time when query throughput drop to zero.

  • a Primary Member’s log (mongod.log)
{"t":{"$date":"2024-02-07T09:19:15.564+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":true}}}
{"t":{"$date":"2024-02-07T09:19:15.564+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:15.564+00:00"},"s":"I",  "c":"-",        "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","delayMillis":499}}
{"t":{"$date":"2024-02-07T09:19:15.564+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:15.564+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:15.987+00:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-11","msg":"Heartbeat failed after max retries","attr":{"target":"#.#.#.#:27017","maxHeartbeatRetries":2,"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"}}}
{"t":{"$date":"2024-02-07T09:19:16.000+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:63116","uuid":"06bf8abc-22ff-4d2a-9ca1-3a84d2ac894b","connectionId":7964,"connectionCount":83}}
{"t":{"$date":"2024-02-07T09:19:16.000+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn7964","msg":"Connection ended","attr":{"remote":"127.0.0.1:63116","uuid":"06bf8abc-22ff-4d2a-9ca1-3a84d2ac894b","connectionId":7964,"connectionCount":82}}
{"t":{"$date":"2024-02-07T09:19:16.050+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"ReplNetwork","msg":"Connecting","attr":{"hostAndPort":"#.#.#.#:27017"}}
{"t":{"$date":"2024-02-07T09:19:16.063+00:00"},"s":"I",  "c":"-",        "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"#.#.#.#:27017","error":"HostUnreachable: Error connecting to #.#.#.#:27017 :: caused by :: Connection refused","replicaSet":"shard_03","response":{}}}
{"t":{"$date":"2024-02-07T09:19:16.063+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":false,"outcome":{"host":"#.#.#.#:27017","success":false,"errorMessage":"HostUnreachable: Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"}}}}
{"t":{"$date":"2024-02-07T09:19:16.563+00:00"},"s":"I",  "c":"-",        "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"#.#.#.#:27017","error":"HostUnreachable: Error connecting to #.#.#.#:27017 :: caused by :: Connection refused","replicaSet":"shard_03","response":{}}}
{"t":{"$date":"2024-02-07T09:19:16.564+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":true}}}
{"t":{"$date":"2024-02-07T09:19:16.564+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:16.564+00:00"},"s":"I",  "c":"-",        "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","delayMillis":499}}
{"t":{"$date":"2024-02-07T09:19:16.564+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:16.564+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:16.697+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn6981","msg":"Connection ended","attr":{"remote":"#.#.#.#:37416","uuid":"e7770622-59ae-4058-9e44-4a54fa7e38ae","connectionId":6981,"connectionCount":81}}
{"t":{"$date":"2024-02-07T09:19:17.000+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:38126","uuid":"c595b2bc-5442-4e56-8af8-fbcc8d0c8d64","connectionId":7965,"connectionCount":82}}
{"t":{"$date":"2024-02-07T09:19:17.000+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn7965","msg":"Connection ended","attr":{"remote":"127.0.0.1:38126","uuid":"c595b2bc-5442-4e56-8af8-fbcc8d0c8d64","connectionId":7965,"connectionCount":81}}
{"t":{"$date":"2024-02-07T09:19:17.050+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"ReplNetwork","msg":"Connecting","attr":{"hostAndPort":"#.#.#.#:27017"}}
{"t":{"$date":"2024-02-07T09:19:17.063+00:00"},"s":"I",  "c":"-",        "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"#.#.#.#:27017","error":"HostUnreachable: Error connecting to #.#.#.#:27017 :: caused by :: Connection refused","replicaSet":"shard_03","response":{}}}
  • a Secondary Member’s log (mongod.log)
{"t":{"$date":"2024-02-07T09:19:14.441+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:25017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:25017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":true}}}
{"t":{"$date":"2024-02-07T09:19:14.441+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:25017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:14.441+00:00"},"s":"I",  "c":"-",        "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"shard_03","host":"#.#.#.#:25017","delayMillis":500}}
{"t":{"$date":"2024-02-07T09:19:14.441+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:25017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:14.441+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:25017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:14.878+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"TaskExecutorPool-0","msg":"Connecting","attr":{"hostAndPort":"#.#.#.#:25017"}}
{"t":{"$date":"2024-02-07T09:19:14.942+00:00"},"s":"I",  "c":"-",        "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"#.#.#.#:25017","error":"HostUnreachable: Error connecting to #.#.#.#:25017 :: caused by :: Connection refused","replicaSet":"shard_03","response":{}}}
{"t":{"$date":"2024-02-07T09:19:14.942+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:25017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:25017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":false,"outcome":{"host":"#.#.#.#:25017","success":false,"errorMessage":"HostUnreachable: Error connecting to #.#.#.#:25017 :: caused by :: Connection refused"}}}}
{"t":{"$date":"2024-02-07T09:19:15.000+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:44428","uuid":"d3b888b1-7537-4fb4-875c-1b17cfc52112","connectionId":7699,"connectionCount":79}}
{"t":{"$date":"2024-02-07T09:19:15.001+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn7699","msg":"Connection ended","attr":{"remote":"127.0.0.1:44428","uuid":"d3b888b1-7537-4fb4-875c-1b17cfc52112","connectionId":7699,"connectionCount":78}}
{"t":{"$date":"2024-02-07T09:19:15.140+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn7549","msg":"Connection ended","attr":{"remote":"#.#.#.#:50708","uuid":"817bfef9-bd81-4d59-9a25-1a82eb2a21c4","connectionId":7549,"connectionCount":77}}
{"t":{"$date":"2024-02-07T09:19:15.443+00:00"},"s":"I",  "c":"-",        "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"#.#.#.#:25017","error":"HostUnreachable: Error connecting to #.#.#.#:25017 :: caused by :: Connection refused","replicaSet":"shard_03","response":{}}}
{"t":{"$date":"2024-02-07T09:19:15.443+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:25017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:25017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":true}}}
{"t":{"$date":"2024-02-07T09:19:15.443+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:25017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:15.443+00:00"},"s":"I",  "c":"-",        "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"shard_03","host":"#.#.#.#:25017","delayMillis":500}}
{"t":{"$date":"2024-02-07T09:19:15.443+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:25017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:15.443+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:25017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:15.878+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"TaskExecutorPool-0","msg":"Connecting","attr":{"hostAndPort":"#.#.#.#:25017"}}
{"t":{"$date":"2024-02-07T09:19:15.944+00:00"},"s":"I",  "c":"-",        "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"#.#.#.#:25017","error":"HostUnreachable: Error connecting to #.#.#.#:25017 :: caused by :: Connection refused","replicaSet":"shard_03","response":{}}}
{"t":{"$date":"2024-02-07T09:19:15.944+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:25017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:25017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":false,"outcome":{"host":"#.#.#.#:25017","success":false,"errorMessage":"HostUnreachable: Error connecting to #.#.#.#:25017 :: caused by :: Connection refused"}}}}
{"t":{"$date":"2024-02-07T09:19:16.001+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:44438","uuid":"ac408a43-a2d5-41e8-a2f7-7860974adfe7","connectionId":7700,"connectionCount":78}}
{"t":{"$date":"2024-02-07T09:19:16.001+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn7700","msg":"Connection ended","attr":{"remote":"127.0.0.1:44438","uuid":"ac408a43-a2d5-41e8-a2f7-7860974adfe7","connectionId":7700,"connectionCount":77}}
{"t":{"$date":"2024-02-07T09:19:16.198+00:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-3","msg":"Heartbeat failed after max retries","attr":{"target":"#.#.#.#:25017","maxHeartbeatRetries":2,"error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:25017 :: caused by :: Connection refused"}}}
{"t":{"$date":"2024-02-07T09:19:16.445+00:00"},"s":"I",  "c":"-",        "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"#.#.#.#:25017","error":"HostUnreachable: Error connecting to #.#.#.#:25017 :: caused by :: Connection refused","replicaSet":"shard_03","response":{}}}
  • a mongos log (mongos.log)
{"t":{"$date":"2024-02-07T09:19:09.678+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":false,"outcome":{"host":"#.#.#.#:27017","success":false,"errorMessage":"HostUnreachable: Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"}}}}
{"t":{"$date":"2024-02-07T09:19:10.000+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:35038","uuid":"1b08030a-77ed-4f88-af40-66912b3725ad","connectionId":2672,"connectionCount":308}}
{"t":{"$date":"2024-02-07T09:19:10.000+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn2672","msg":"Connection ended","attr":{"remote":"127.0.0.1:35038","uuid":"1b08030a-77ed-4f88-af40-66912b3725ad","connectionId":2672,"connectionCount":307}}
{"t":{"$date":"2024-02-07T09:19:10.178+00:00"},"s":"I",  "c":"-",        "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"#.#.#.#:27017","error":"HostUnreachable: Error connecting to #.#.#.#:27017 :: caused by :: Connection refused","replicaSet":"shard_03","response":{}}}
{"t":{"$date":"2024-02-07T09:19:10.178+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":true}}}
{"t":{"$date":"2024-02-07T09:19:10.178+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:10.178+00:00"},"s":"I",  "c":"-",        "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","delayMillis":500}}
{"t":{"$date":"2024-02-07T09:19:10.178+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:10.178+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:10.676+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"TaskExecutorPool-0","msg":"Connecting","attr":{"hostAndPort":"#.#.#.#:27017"}}
{"t":{"$date":"2024-02-07T09:19:10.678+00:00"},"s":"I",  "c":"-",        "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"#.#.#.#:27017","error":"HostUnreachable: Error connecting to #.#.#.#:27017 :: caused by :: Connection refused","replicaSet":"shard_03","response":{}}}
{"t":{"$date":"2024-02-07T09:19:10.678+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":false,"outcome":{"host":"#.#.#.#:27017","success":false,"errorMessage":"HostUnreachable: Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"}}}}
{"t":{"$date":"2024-02-07T09:19:11.178+00:00"},"s":"I",  "c":"-",        "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"#.#.#.#:27017","error":"HostUnreachable: Error connecting to #.#.#.#:27017 :: caused by :: Connection refused","replicaSet":"shard_03","response":{}}}
{"t":{"$date":"2024-02-07T09:19:11.178+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":true}}}
{"t":{"$date":"2024-02-07T09:19:11.178+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:11.178+00:00"},"s":"I",  "c":"-",        "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","delayMillis":500}}
{"t":{"$date":"2024-02-07T09:19:11.178+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:11.178+00:00"},"s":"I",  "c":"-",        "id":4333227, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM monitoring host in expedited mode until we detect a primary","attr":{"host":"#.#.#.#:27017","replicaSet":"shard_03"}}
{"t":{"$date":"2024-02-07T09:19:11.676+00:00"},"s":"I",  "c":"CONNPOOL", "id":22576,   "ctx":"TaskExecutorPool-0","msg":"Connecting","attr":{"hostAndPort":"#.#.#.#:27017"}}
{"t":{"$date":"2024-02-07T09:19:11.678+00:00"},"s":"I",  "c":"-",        "id":4333222, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM received error response","attr":{"host":"#.#.#.#:27017","error":"HostUnreachable: Error connecting to #.#.#.#:27017 :: caused by :: Connection refused","replicaSet":"shard_03","response":{}}}
{"t":{"$date":"2024-02-07T09:19:11.678+00:00"},"s":"I",  "c":"NETWORK",  "id":4712102, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Host failed in replica set","attr":{"replicaSet":"shard_03","host":"#.#.#.#:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"},"action":{"dropConnections":true,"requestImmediateCheck":false,"outcome":{"host":"#.#.#.#:27017","success":false,"errorMessage":"HostUnreachable: Error connecting to #.#.#.#:27017 :: caused by :: Connection refused"}}}}

In my opinion, there doesn’t seem to be anything strange in the logs.