Mongodb with 4 nodes -2 nodes were down and no node became primary

Ops and Admin Replication
1

Mongodb has 4 nodes of replicaset with

Node 1 - Priority 1- Secondary
Node 2- Priority 1- Secondary
Node 3- Priority 1 -Primary
Node 4- Priority 0 - Secondary( used for Reading Data)

Node 3 Primary server disk was filled and running out of space and services got stopped and node 4 db services was also down due

Now here Node 1 or Node 2 Any one Node needs to became primary. But it failed to became primary

Node1 and Node2 were only Secondary , After that in the node 3 server Space was cleared and services was restarted, Node 1 became the primary and node 2 and node 3 were secondary,

My question why node 1 or node 2 didn;t became primary when the node 3 was services was down due to space constraints.

Mongodb version 5.0.20

OS
NAME=“CentOS Linux”
VERSION=“8”
ID=“centos”
ID_LIKE=“rhel fedora”
VERSION_ID=“8”
PLATFORM_ID=“platform:el8”
PRETTY_NAME=“CentOS Linux 8”
ANSI_COLOR=“0;31”
CPE_NAME=“cpe:/o:centos:centos:8”
HOME_URL=“https://centos.org/”
BUG_REPORT_URL=“https://bugs.centos.org/”
CENTOS_MANTISBT_PROJECT=“CentOS-8”
CENTOS_MANTISBT_PROJECT_VERSION=“8”

Please do suggest what can be the issue any one node should have became the primary during space issue on node 3 (Primary) services was down

2

If that is the configuration then yes Node 1 or 2 should have become primary.

Double check the replicaset configuration to be sure it is what you expect it to be. The logs will contain all the information about what occurred during the the period when node 3 and 4 were down and should indicate why a primary was not elected.

3

{“t”:{“$date”:“2023-10-04T03:57:36.344+05:30”},“s”:“I”, “c”:“REPL_HB”, “id”:23974, “ctx”:“ReplCoord-4164”,“msg”:“Heartbeat failed after max retries”,“attr”:{“target”:“his-air-db4:45431”,“maxHeartbeatRetries”:2,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”}}}
{“t”:{“$date”:“2023-10-04T03:57:36.972+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“action”:{“dropConnections”:true,“requestImmediateCheck”:true}}}
{“t”:{“$date”:“2023-10-04T03:57:36.972+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db4:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:36.972+05:30”},“s”:“I”, “c”:“-”, “id”:4333218, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Rescheduling the next replica set monitoring request”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“delayMillis”:0}}
{“t”:{“$date”:“2023-10-04T03:57:36.972+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“ip-10-20-110-22:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:36.972+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db3:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:36.972+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db2:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:36.973+05:30”},“s”:“I”, “c”:“-”, “id”:4333222, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM received error response”,“attr”:{“host”:“his-air-db4:45431”,“error”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”,“replicaSet”:“arcusapp”,“response”:“{}”}}
{“t”:{“$date”:“2023-10-04T03:57:36.973+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“action”:{“dropConnections”:true,“requestImmediateCheck”:false,“outcome”:{“host”:“his-air-db4:45431”,“success”:false,“errorMessage”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”}}}}
{“t”:{“$date”:“2023-10-04T03:57:37.205+05:30”},“s”:“I”, “c”:“CONNPOOL”, “id”:22576, “ctx”:“ReplNetwork”,“msg”:“Connecting”,“attr”:{“hostAndPort”:“his-air-db4:45431”}}
{“t”:{“$date”:“2023-10-04T03:57:37.474+05:30”},“s”:“I”, “c”:“-”, “id”:4333222, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM received error response”,“attr”:{“host”:“his-air-db4:45431”,“error”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”,“replicaSet”:“arcusapp”,“response”:“{}”}}
{“t”:{“$date”:“2023-10-04T03:57:37.474+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“action”:{“dropConnections”:true,“requestImmediateCheck”:true}}}
{“t”:{“$date”:“2023-10-04T03:57:37.474+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db4:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:37.474+05:30”},“s”:“I”, “c”:“-”, “id”:4333218, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Rescheduling the next replica set monitoring request”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“delayMillis”:499}}
{“t”:{“$date”:“2023-10-04T03:57:37.474+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“ip-10-20-110-22:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:37.474+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db3:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:37.474+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db2:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:37.974+05:30”},“s”:“I”, “c”:“-”, “id”:4333222, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM received error response”,“attr”:{“host”:“his-air-db4:45431”,“error”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”,“replicaSet”:“arcusapp”,“response”:“{}”}}
{“t”:{“$date”:“2023-10-04T03:57:37.974+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“action”:{“dropConnections”:true,“requestImmediateCheck”:false,“outcome”:{“host”:“his-air-db4:45431”,“success”:false,“errorMessage”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”}}}}
{“t”:{“$date”:“2023-10-04T03:57:38.205+05:30”},“s”:“I”, “c”:“CONNPOOL”, “id”:22576, “ctx”:“ReplNetwork”,“msg”:“Connecting”,“attr”:{“hostAndPort”:“his-air-db4:45431”}}
{“t”:{“$date”:“2023-10-04T03:57:38.346+05:30”},“s”:“I”, “c”:“REPL_HB”, “id”:23974, “ctx”:“ReplCoord-4145”,“msg”:“Heartbeat failed after max retries”,“attr”:{“target”:“his-air-db4:45431”,“maxHeartbeatRetries”:2,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”}}}
{“t”:{“$date”:“2023-10-04T03:57:38.474+05:30”},“s”:“I”, “c”:“-”, “id”:4333222, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM received error response”,“attr”:{“host”:“his-air-db4:45431”,“error”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”,“replicaSet”:“arcusapp”,“response”:“{}”}}
{“t”:{“$date”:“2023-10-04T03:57:38.474+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“action”:{“dropConnections”:true,“requestImmediateCheck”:true}}}
{“t”:{“$date”:“2023-10-04T03:57:38.475+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db4:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:38.475+05:30”},“s”:“I”, “c”:“-”, “id”:4333218, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Rescheduling the next replica set monitoring request”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“delayMillis”:499}}
{“t”:{“$date”:“2023-10-04T03:57:38.475+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“ip-10-20-110-22:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:38.475+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db3:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:38.475+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db2:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:38.975+05:30”},“s”:“I”, “c”:“-”, “id”:4333222, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM received error response”,“attr”:{“host”:“his-air-db4:45431”,“error”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”,“replicaSet”:“arcusapp”,“response”:“{}”}}
{“t”:{“$date”:“2023-10-04T03:57:38.975+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“action”:{“dropConnections”:true,“requestImmediateCheck”:false,“outcome”:{“host”:“his-air-db4:45431”,“success”:false,“errorMessage”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”}}}}
{“t”:{“$date”:“2023-10-04T03:57:39.205+05:30”},“s”:“I”, “c”:“CONNPOOL”, “id”:22576, “ctx”:“ReplNetwork”,“msg”:“Connecting”,“attr”:{“hostAndPort”:“his-air-db4:45431”}}
{“t”:{“$date”:“2023-10-04T03:57:39.475+05:30”},“s”:“I”, “c”:“-”, “id”:4333222, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM received error response”,“attr”:{“host”:“his-air-db4:45431”,“error”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”,“replicaSet”:“arcusapp”,“response”:“{}”}}
{“t”:{“$date”:“2023-10-04T03:57:39.476+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“action”:{“dropConnections”:true,“requestImmediateCheck”:true}}}
{“t”:{“$date”:“2023-10-04T03:57:39.476+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db4:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:39.476+05:30”},“s”:“I”, “c”:“-”, “id”:4333218, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Rescheduling the next replica set monitoring request”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“delayMillis”:499}}
{“t”:{“$date”:“2023-10-04T03:57:39.476+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“ip-10-20-110-22:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:39.476+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db3:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:39.476+05:30”},“s”:“I”, “c”:“-”, “id”:4333227, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM monitoring host in expedited mode until we detect a primary”,“attr”:{“host”:“his-air-db2:45431”,“replicaSet”:“arcusapp”}}
{“t”:{“$date”:“2023-10-04T03:57:39.975+05:30”},“s”:“I”, “c”:“-”, “id”:4333222, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM received error response”,“attr”:{“host”:“his-air-db4:45431”,“error”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”,“replicaSet”:“arcusapp”,“response”:“{}”}}
{“t”:{“$date”:“2023-10-04T03:57:39.976+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“action”:{“dropConnections”:true,“requestImmediateCheck”:false,“outcome”:{“host”:“his-air-db4:45431”,“success”:false,“errorMessage”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”}}}}
{“t”:{“$date”:“2023-10-04T03:57:40.205+05:30”},“s”:“I”, “c”:“CONNPOOL”, “id”:22576, “ctx”:“ReplNetwork”,“msg”:“Connecting”,“attr”:{“hostAndPort”:“his-air-db4:45431”}}

It was keep on looking for primary , but not elected as primary

4

{“t”:{“$date”:“2023-10-04T03:55:35.484+05:30”},“s”:“I”, “c”:“-”, “id”:4333222, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM received error response”,“attr”:{“host”:“his-air-db4:45431”,“error”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”,“replicaSet”:“arcusapp”,“response”:“{}”}}
{“t”:{“$date”:“2023-10-04T03:55:35.484+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“action”:{“dropConnections”:true,“requestImmediateCheck”:false,“outcome”:{“host”:“his-air-db4:45431”,“success”:false,“errorMessage”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”}}}}
{“t”:{“$date”:“2023-10-04T03:55:35.775+05:30”},“s”:“I”, “c”:“REPL_HB”, “id”:23974, “ctx”:“ReplCoord-4160”,“msg”:“Heartbeat failed after max retries”,“attr”:{“target”:“his-air-db4:45431”,“maxHeartbeatRetries”:2,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”}}}
{“t”:{“$date”:“2023-10-04T03:55:35.898+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4333213, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM Topology Change”,“attr”:{“replicaSet”:“arcusapp”,“newTopologyDescription”:“{ id: "4406317e-2a95-45db-a0cb-31cb34c560dd", topologyType: "ReplicaSetNoPrimary", servers: { his-air-db2:45431: { address: "his-air-db2:45431", topologyVersion: { processId: ObjectId(‘64fca3272b58dab17e865419’), counter: 51 }, roundTripTime: 472, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "his-air-db2:45431", setName: "arcusapp", setVersion: 286895, lastUpdateTime: new Date(1696371935898), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } }, his-air-db3:45431: { address: "his-air-db3:45431", topologyVersion: { processId: ObjectId(‘6505d5f2312accdf7dd80771’), counter: 48 }, roundTripTime: 668, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "his-air-db3:45431", setName: "arcusapp", setVersion: 286895, lastUpdateTime: new Date(1696371935292), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } }, his-air-db4:45431: { address: "his-air-db4:45431", type: "Unknown", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} }, ip-10-20-110-22:45431: { address: "ip-10-20-110-22:45431", topologyVersion: { processId: ObjectId(‘64e0c5ace2bc1369ed0b76d6’), counter: 24 }, roundTripTime: 106955, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "ip-10-20-110-22:45431", setName: "arcusapp", setVersion: 286895, lastUpdateTime: new Date(1696371932691), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } } }, logicalSessionTimeoutMinutes: 30, setName: "arcusapp", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId(‘7fffffff00000000000000b5’), setVersion: 286895 } }”,“previousTopologyDescription”:“{ id: "d7ad2052-5ba7-4084-9974-78550ebf6e48", topologyType: "ReplicaSetNoPrimary", servers: { his-air-db2:45431: { address: "his-air-db2:45431", topologyVersion: { processId: ObjectId(‘64fca3272b58dab17e865419’), counter: 51 }, roundTripTime: 472, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "his-air-db2:45431", setName: "arcusapp", setVersion: 286895, primary: "his-air-db3:45431", lastUpdateTime: new Date(1696371925888), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } }, his-air-db3:45431: { address: "his-air-db3:45431", topologyVersion: { processId: ObjectId(‘6505d5f2312accdf7dd80771’), counter: 48 }, roundTripTime: 668, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "his-air-db3:45431", setName: "arcusapp", setVersion: 286895, lastUpdateTime: new Date(1696371935292), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } }, his-air-db4:45431: { address: "his-air-db4:45431", type: "Unknown", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} }, ip-10-20-110-22:45431: { address: "ip-10-20-110-22:45431", topologyVersion: { processId: ObjectId(‘64e0c5ace2bc1369ed0b76d6’), counter: 24 }, roundTripTime: 106955, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "ip-10-20-110-22:45431", setName: "arcusapp", setVersion: 286895, lastUpdateTime: new Date(1696371932691), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } } }, logicalSessionTimeoutMinutes: 30, setName: "arcusapp", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId(‘7fffffff00000000000000b5’), setVersion: 286895 } }”}}
{“t”:{“$date”:“2023-10-04T03:55:35.985+05:30”},“s”:“I”, “c”:“-”, “id”:4333222, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM received error response”,“attr”:{“host”:“his-air-db4:45431”,“error”:“HostUnreachable: Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”,“replicaSet”:“arcusapp”,“response”:“{}”}}
{“t”:{“$date”:“2023-10-04T03:55:35.985+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4712102, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Host failed in replica set”,“attr”:{“replicaSet”:“arcusapp”,“host”:“his-air-db4:45431”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“action”:{“dropConnections”:true,“requestImmediateCheck”:true}}}
{“t”:{“$date”:“2023-10-04T03:55:36.088+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:4615652, “ctx”:“ReplCoord-4163”,“msg”:“Starting an election, since we’ve seen no PRIMARY in election timeout period”,“attr”:{“electionTimeoutPeriodMillis”:10000}}
{“t”:{“$date”:“2023-10-04T03:55:36.088+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:21438, “ctx”:“ReplCoord-4163”,“msg”:“Conducting a dry run election to see if we could be elected”,“attr”:{“currentTerm”:181}}
{“t”:{“$date”:“2023-10-04T03:55:36.088+05:30”},“s”:“I”, “c”:“REPL”, “id”:21752, “ctx”:“ReplCoord-4163”,“msg”:“Scheduling remote command request”,“attr”:{“context”:“vote request”,“request”:“RemoteCommand 116088808 – target:his-air-db4:45431 db:admin cmd:{ replSetRequestVotes: 1, setName: "arcusapp", dryRun: true, term: 181, candidateIndex: 0, configVersion: 286895, configTerm: 181, lastAppliedOpTime: { ts: Timestamp(1696371924, 2), t: 181 } }”}}
{“t”:{“$date”:“2023-10-04T03:55:36.088+05:30”},“s”:“I”, “c”:“REPL”, “id”:21752, “ctx”:“ReplCoord-4163”,“msg”:“Scheduling remote command request”,“attr”:{“context”:“vote request”,“request”:“RemoteCommand 116088809 – target:ip-10-20-110-22:45431 db:admin cmd:{ replSetRequestVotes: 1, setName: "arcusapp", dryRun: true, term: 181, candidateIndex: 0, configVersion: 286895, configTerm: 181, lastAppliedOpTime: { ts: Timestamp(1696371924, 2), t: 181 } }”}}
{“t”:{“$date”:“2023-10-04T03:55:36.088+05:30”},“s”:“I”, “c”:“REPL”, “id”:21752, “ctx”:“ReplCoord-4163”,“msg”:“Scheduling remote command request”,“attr”:{“context”:“vote request”,“request”:“RemoteCommand 116088810 – target:his-air-db3:45431 db:admin cmd:{ replSetRequestVotes: 1, setName: "arcusapp", dryRun: true, term: 181, candidateIndex: 0, configVersion: 286895, configTerm: 181, lastAppliedOpTime: { ts: Timestamp(1696371924, 2), t: 181 } }”}}
{“t”:{“$date”:“2023-10-04T03:55:36.088+05:30”},“s”:“I”, “c”:“CONNPOOL”, “id”:22576, “ctx”:“ReplNetwork”,“msg”:“Connecting”,“attr”:{“hostAndPort”:“ip-10-20-110-22:45431”}}
{“t”:{“$date”:“2023-10-04T03:55:36.089+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:51799, “ctx”:“ReplCoord-4159”,“msg”:“VoteRequester processResponse”,“attr”:{“term”:181,“dryRun”:true,“failReason”:“failed to receive response”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“from”:“his-air-db4:45431”}}
{“t”:{“$date”:“2023-10-04T03:55:36.089+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:51799, “ctx”:“ReplCoord-4162”,“msg”:“VoteRequester processResponse”,“attr”:{“term”:181,“dryRun”:true,“vote”:“yes”,“from”:“his-air-db3:45431”,“message”:{“term”:181,“voteGranted”:true,“reason”:“”,“ok”:1,“$clusterTime”:{“clusterTime”:{“$timestamp”:{“t”:1696371924,“i”:2}},“signature”:{“hash”:{“$binary”:{“base64”:“AAAAAAAAAAAAAAAAAAAAAAAAAAA=”,“subType”:“0”}},“keyId”:0}},“operationTime”:{“$timestamp”:{“t”:1696371924,“i”:2}}}}}
{“t”:{“$date”:“2023-10-04T03:55:36.135+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:51799, “ctx”:“ReplCoord-4160”,“msg”:“VoteRequester processResponse”,“attr”:{“term”:181,“dryRun”:true,“vote”:“yes”,“from”:“ip-10-20-110-22:45431”,“message”:{“term”:181,“voteGranted”:true,“reason”:“”,“ok”:1,“$clusterTime”:{“clusterTime”:{“$timestamp”:{“t”:1696371924,“i”:2}},“signature”:{“hash”:{“$binary”:{“base64”:“AAAAAAAAAAAAAAAAAAAAAAAAAAA=”,“subType”:“0”}},“keyId”:0}},“operationTime”:{“$timestamp”:{“t”:1696371924,“i”:2}}}}}
{“t”:{“$date”:“2023-10-04T03:55:36.135+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:21444, “ctx”:“ReplCoord-4160”,“msg”:“Dry election run succeeded, running for election”,“attr”:{“newTerm”:182}}
{“t”:{“$date”:“2023-10-04T03:55:36.135+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:6015300, “ctx”:“ReplCoord-4160”,“msg”:“Storing last vote document in local storage for my election”,“attr”:{“lastVote”:{“term”:182,“candidateIndex”:0}}}
{“t”:{“$date”:“2023-10-04T03:55:36.139+05:30”},“s”:“I”, “c”:“REPL”, “id”:21752, “ctx”:“ReplCoord-4160”,“msg”:“Scheduling remote command request”,“attr”:{“context”:“vote request”,“request”:“RemoteCommand 116088811 – target:his-air-db4:45431 db:admin cmd:{ replSetRequestVotes: 1, setName: "arcusapp", dryRun: false, term: 182, candidateIndex: 0, configVersion: 286895, configTerm: 181, lastAppliedOpTime: { ts: Timestamp(1696371924, 2), t: 181 } }”}}
{“t”:{“$date”:“2023-10-04T03:55:36.139+05:30”},“s”:“I”, “c”:“REPL”, “id”:21752, “ctx”:“ReplCoord-4160”,“msg”:“Scheduling remote command request”,“attr”:{“context”:“vote request”,“request”:“RemoteCommand 116088812 – target:ip-10-20-110-22:45431 db:admin cmd:{ replSetRequestVotes: 1, setName: "arcusapp", dryRun: false, term: 182, candidateIndex: 0, configVersion: 286895, configTerm: 181, lastAppliedOpTime: { ts: Timestamp(1696371924, 2), t: 181 } }”}}
{“t”:{“$date”:“2023-10-04T03:55:36.139+05:30”},“s”:“I”, “c”:“REPL”, “id”:21752, “ctx”:“ReplCoord-4160”,“msg”:“Scheduling remote command request”,“attr”:{“context”:“vote request”,“request”:“RemoteCommand 116088813 – target:his-air-db3:45431 db:admin cmd:{ replSetRequestVotes: 1, setName: "arcusapp", dryRun: false, term: 182, candidateIndex: 0, configVersion: 286895, configTerm: 181, lastAppliedOpTime: { ts: Timestamp(1696371924, 2), t: 181 } }”}}
{“t”:{“$date”:“2023-10-04T03:55:36.140+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:51799, “ctx”:“ReplCoord-4159”,“msg”:“VoteRequester processResponse”,“attr”:{“term”:182,“dryRun”:false,“failReason”:“failed to receive response”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“from”:“his-air-db4:45431”}}
{“t”:{“$date”:“2023-10-04T03:55:36.147+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:51799, “ctx”:“ReplCoord-4162”,“msg”:“VoteRequester processResponse”,“attr”:{“term”:182,“dryRun”:false,“vote”:“yes”,“from”:“his-air-db3:45431”,“message”:{“term”:182,“voteGranted”:true,“reason”:“”,“ok”:1,“$clusterTime”:{“clusterTime”:{“$timestamp”:{“t”:1696371924,“i”:2}},“signature”:{“hash”:{“$binary”:{“base64”:“AAAAAAAAAAAAAAAAAAAAAAAAAAA=”,“subType”:“0”}},“keyId”:0}},“operationTime”:{“$timestamp”:{“t”:1696371924,“i”:2}}}}}
{“t”:{“$date”:“2023-10-04T03:55:36.177+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:51799, “ctx”:“ReplCoord-4145”,“msg”:“VoteRequester processResponse”,“attr”:{“term”:182,“dryRun”:false,“vote”:“yes”,“from”:“ip-10-20-110-22:45431”,“message”:{“term”:182,“voteGranted”:true,“reason”:“”,“ok”:1,“$clusterTime”:{“clusterTime”:{“$timestamp”:{“t”:1696371924,“i”:2}},“signature”:{“hash”:{“$binary”:{“base64”:“AAAAAAAAAAAAAAAAAAAAAAAAAAA=”,“subType”:“0”}},“keyId”:0}},“operationTime”:{“$timestamp”:{“t”:1696371924,“i”:2}}}}}
{“t”:{“$date”:“2023-10-04T03:55:36.177+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:21450, “ctx”:“ReplCoord-4145”,“msg”:“Election succeeded, assuming primary role”,“attr”:{“term”:182}}
{“t”:{“$date”:“2023-10-04T03:55:36.177+05:30”},“s”:“I”, “c”:“REPL”, “id”:21358, “ctx”:“ReplCoord-4145”,“msg”:“Replica set state transition”,“attr”:{“newState”:“PRIMARY”,“oldState”:“SECONDARY”}}
{“t”:{“$date”:“2023-10-04T03:55:36.177+05:30”},“s”:“I”, “c”:“REPL”, “id”:21106, “ctx”:“ReplCoord-4145”,“msg”:“Resetting sync source to empty”,“attr”:{“previousSyncSource”:“:27017”}}
{“t”:{“$date”:“2023-10-04T03:55:36.177+05:30”},“s”:“I”, “c”:“REPL”, “id”:21359, “ctx”:“ReplCoord-4145”,“msg”:“Entering primary catch-up mode”}
{“t”:{“$date”:“2023-10-04T03:55:36.179+05:30”},“s”:“I”, “c”:“NETWORK”, “id”:4333213, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“RSM Topology Change”,“attr”:{“replicaSet”:“arcusapp”,“newTopologyDescription”:“{ id: "107740bc-dad1-4b26-84f8-484f9533e930", topologyType: "ReplicaSetNoPrimary", servers: { his-air-db2:45431: { address: "his-air-db2:45431", topologyVersion: { processId: ObjectId(‘64fca3272b58dab17e865419’), counter: 52 }, roundTripTime: 472, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "his-air-db2:45431", setName: "arcusapp", setVersion: 286895, electionId: ObjectId(‘7fffffff00000000000000b6’), primary: "his-air-db2:45431", lastUpdateTime: new Date(1696371936179), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } }, his-air-db3:45431: { address: "his-air-db3:45431", topologyVersion: { processId: ObjectId(‘6505d5f2312accdf7dd80771’), counter: 48 }, roundTripTime: 668, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "his-air-db3:45431", setName: "arcusapp", setVersion: 286895, lastUpdateTime: new Date(1696371935292), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } }, his-air-db4:45431: { address: "his-air-db4:45431", type: "Unknown", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} }, ip-10-20-110-22:45431: { address: "ip-10-20-110-22:45431", topologyVersion: { processId: ObjectId(‘64e0c5ace2bc1369ed0b76d6’), counter: 24 }, roundTripTime: 106955, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "ip-10-20-110-22:45431", setName: "arcusapp", setVersion: 286895, lastUpdateTime: new Date(1696371932691), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } } }, logicalSessionTimeoutMinutes: 30, setName: "arcusapp", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId(‘7fffffff00000000000000b5’), setVersion: 286895 } }”,“previousTopologyDescription”:“{ id: "4406317e-2a95-45db-a0cb-31cb34c560dd", topologyType: "ReplicaSetNoPrimary", servers: { his-air-db2:45431: { address: "his-air-db2:45431", topologyVersion: { processId: ObjectId(‘64fca3272b58dab17e865419’), counter: 51 }, roundTripTime: 472, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "his-air-db2:45431", setName: "arcusapp", setVersion: 286895, lastUpdateTime: new Date(1696371935898), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } }, his-air-db3:45431: { address: "his-air-db3:45431", topologyVersion: { processId: ObjectId(‘6505d5f2312accdf7dd80771’), counter: 48 }, roundTripTime: 668, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "his-air-db3:45431", setName: "arcusapp", setVersion: 286895, lastUpdateTime: new Date(1696371935292), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } }, his-air-db4:45431: { address: "his-air-db4:45431", type: "Unknown", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} }, ip-10-20-110-22:45431: { address: "ip-10-20-110-22:45431", topologyVersion: { processId: ObjectId(‘64e0c5ace2bc1369ed0b76d6’), counter: 24 }, roundTripTime: 106955, lastWriteDate: new Date(1696371924000), opTime: { ts: Timestamp(1696371924, 2), t: 181 }, type: "RSSecondary", minWireVersion: 9, maxWireVersion: 13, me: "ip-10-20-110-22:45431", setName: "arcusapp", setVersion: 286895, lastUpdateTime: new Date(1696371932691), logicalSessionTimeoutMinutes: 30, hosts: { 0: "his-air-db2:45431", 1: "his-air-db3:45431", 2: "his-air-db4:45431" }, arbiters: {}, passives: { 0: "ip-10-20-110-22:45431" } } }, logicalSessionTimeoutMinutes: 30, setName: "arcusapp", compatible: true, maxElectionIdSetVersion: { electionId: ObjectId(‘7fffffff00000000000000b5’), setVersion: 286895 } }”}}
{“t”:{“$date”:“2023-10-04T03:55:36.181+05:30”},“s”:“I”, “c”:“REPL_HB”, “id”:23974, “ctx”:“ReplCoord-4159”,“msg”:“Heartbeat failed after max retries”,“attr”:{“target”:“his-air-db4:45431”,“maxHeartbeatRetries”:2,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”}}}
{“t”:{“$date”:“2023-10-04T03:55:36.205+05:30”},“s”:“I”, “c”:“CONNPOOL”, “id”:22576, “ctx”:“ReplNetwork”,“msg”:“Connecting”,“attr”:{“hostAndPort”:“his-air-db4:45431”}}
{“t”:{“$date”:“2023-10-04T03:55:36.214+05:30”},“s”:“I”, “c”:“REPL”, “id”:21364, “ctx”:“ReplCoord-4163”,“msg”:“Caught up to the latest optime known via heartbeats after becoming primary”,“attr”:{“targetOpTime”:{“ts”:{“$timestamp”:{“t”:1696371924,“i”:2}},“t”:181},“myLastApplied”:{“ts”:{“$timestamp”:{“t”:1696371924,“i”:2}},“t”:181}}}
{“t”:{“$date”:“2023-10-04T03:55:36.214+05:30”},“s”:“I”, “c”:“REPL”, “id”:21363, “ctx”:“ReplCoord-4163”,“msg”:“Exited primary catch-up mode”}
{“t”:{“$date”:“2023-10-04T03:55:36.214+05:30”},“s”:“I”, “c”:“REPL”, “id”:21107, “ctx”:“ReplCoord-4163”,“msg”:“Stopping replication producer”}
{“t”:{“$date”:“2023-10-04T03:55:36.214+05:30”},“s”:“I”, “c”:“REPL”, “id”:21239, “ctx”:“ReplBatcher”,“msg”:“Oplog buffer has been drained”,“attr”:{“term”:182}}
{“t”:{“$date”:“2023-10-04T03:55:36.217+05:30”},“s”:“I”, “c”:“REPL”, “id”:21343, “ctx”:“RstlKillOpThread”,“msg”:“Starting to kill user operations”}
{“t”:{“$date”:“2023-10-04T03:55:36.217+05:30”},“s”:“I”, “c”:“REPL”, “id”:21344, “ctx”:“RstlKillOpThread”,“msg”:“Stopped killing user operations”}
{“t”:{“$date”:“2023-10-04T03:55:36.217+05:30”},“s”:“I”, “c”:“REPL”, “id”:21340, “ctx”:“RstlKillOpThread”,“msg”:“State transition ops metrics”,“attr”:{“metrics”:{“lastStateTransition”:“stepUp”,“userOpsKilled”:0,“userOpsRunning”:30}}}
{“t”:{“$date”:“2023-10-04T03:55:36.217+05:30”},“s”:“I”, “c”:“REPL”, “id”:4508103, “ctx”:“OplogApplier-0”,“msg”:“Increment the config term via reconfig”}

The logs not elected one as primary, it was keep on searching for the node to became primary, But not succeed

6

@chris Any update on this from the log it was trying to find but primary was not elected.

7

@chris from the logs also it was not stated that one of the secondary became the primary, There could be another reason?

8

if all 4 nodes have 1 as vote number, then no primary can be elected after two nodes are down.

9

We see that a primary was elected.

{“t”:{“$date”:“2023-10-04T03:55:36.177+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:21450, “ctx”:“ReplCoord-4145”,“msg”:“Election succeeded, assuming primary role”,“attr”:{“term”:182}}
{“t”:{“$date”:“2023-10-04T03:55:36.177+05:30”},“s”:“I”, “c”:“REPL”, “id”:21358, “ctx”:“ReplCoord-4145”,“msg”:“Replica set state transition”,“attr”:{“newState”:“PRIMARY”,“oldState”:“SECONDARY”}}
{“t”:{“$date”:“2023-10-04T03:55:36.177+05:30”},“s”:“I”, “c”:“REPL”, “id”:21106, “ctx”:“ReplCoord-4145”,“msg”:“Resetting sync source to empty”,“attr”:{“previousSyncSource”:“:27017”}}
{“t”:{“$date”:“2023-10-04T03:55:36.177+05:30”},“s”:“I”, “c”:“REPL”, “id”:21359, “ctx”:“ReplCoord-4145”,“msg”:“Entering primary catch-up mode”}

And yes, as @Kobe_W is saying, it looks like all members have 1 vote as this host is soliciting votes from 3 members, with his-air-db3:45431 and ip-10-20-110-22:45431 voting yes.

{“t”:{“$date”:“2023-10-04T03:55:36.140+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:51799, “ctx”:“ReplCoord-4159”,“msg”:“VoteRequester processResponse”,“attr”:{“term”:182,“dryRun”:false,“failReason”:“failed to receive response”,“error”:{“code”:6,“codeName”:“HostUnreachable”,“errmsg”:“Error connecting to his-air-db4:45431 (172.20.253.208:45431) :: caused by :: Connection refused”},“from”:“his-air-db4:45431”}}
{“t”:{“$date”:“2023-10-04T03:55:36.147+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:51799, “ctx”:“ReplCoord-4162”,“msg”:“VoteRequester processResponse”,“attr”:{“term”:182,“dryRun”:false,“vote”:“yes”,“from”:“his-air-db3:45431”,“message”:{“term”:182,“voteGranted”:true,“reason”:“”,“ok”:1,“$clusterTime”:{“clusterTime”:{“$timestamp”:{“t”:1696371924,“i”:2}},“signature”:{“hash”:{“$binary”:{“base64”:“AAAAAAAAAAAAAAAAAAAAAAAAAAA=”,“subType”:“0”}},“keyId”:0}},“operationTime”:{“$timestamp”:{“t”:1696371924,“i”:2}}}}}
{“t”:{“$date”:“2023-10-04T03:55:36.177+05:30”},“s”:“I”, “c”:“ELECTION”, “id”:51799, “ctx”:“ReplCoord-4145”,“msg”:“VoteRequester processResponse”,“attr”:{“term”:182,“dryRun”:false,“vote”:“yes”,“from”:“ip-10-20-110-22:45431”,“message”:{“term”:182,“voteGranted”:true,“reason”:“”,“ok”:1,“$clusterTime”:{“clusterTime”:{“$timestamp”:{“t”:1696371924,“i”:2}},“signature”:{“hash”:{“$binary”:{“base64”:“AAAAAAAAAAAAAAAAAAAAAAAAAAA=”,“subType”:“0”}},“keyId”:0}},“operationTime”:{“$timestamp”:{“t”:1696371924,“i”:2}}}}}

If you lost another node then the replicaset would be without quorum and would be degraded to secondaries only.

Updating your replicaset so the p:0 member also had votes:0 would have resulted in what you were expecting.

Use an Odd number of voting members:

11

Thanks Chis … after that it elecected as primary

If there is even number of nodes do we need to set prority like 1, 0.5 and 0.5 ,0 to avoid not making primary to secondary when primary db goes down

12

Hi @Samrat_Mehta

Its the votes you need to be concerned with, though a votes:0 member also has to be priority:0.

13

Thanks Chris, But my question was if the RS is Even do we need to set the priority on that like nodes to make the nodes from secondary to primary when proriity of one goes down

14

I’m not understanding your question.