Data Node did not became primary

Ops and Admin Replication
1

hi , just wanted to ask if there is an instance that an arbiter will not vote for a member to become primary ?

Amidst on the event regarding the BSOD today, our mongodb replica (7 Members , 6 Data Nodes 1 arbiter) some members of the replica has been down , below is the state it become

Node 1 : down - priority 1
Node 2 : UP - priority 0.5
Node 3 : down - priorty 0.5
Node 4 : UP - Priority 0
Node 5 : UP - Priority 0
Node 6 : UP - Priority 0
Node 7 : Arbiter

so apparently node 2 did not became primary even though it is the only remaining data node and there is an arbiter for the voting.

2

If the priority 0 hosts have votes:1 then yes, node 2 should become primary.

6 member + arbiter, 3 priority :0, 2 hosts down || click to expand... 
s0 [primary] test> rs.conf().members.map(x => {return {host: x.host, votes: x.votes, priority:x.priority}})
[
  { host: 'mongodb-0-a:27017', votes: 1, priority: 1 },
  { host: 'mongodb-0-b:27017', votes: 1, priority: 0.5 },
  { host: 'mongodb-0-c:27017', votes: 1, priority: 0.5 },
  { host: 'mongodb-0-d:27017', votes: 1, priority: 0 },
  { host: 'mongodb-0-e:27017', votes: 1, priority: 0 },
  { host: 'mongodb-0-f:27017', votes: 1, priority: 0 },
  { host: 'mongodb-a:27017', votes: 1, priority: 0 }
]
s0 [primary] test> rs.status().members.map(x => {return {name: x.name, health:x.health, stateStr:x.stateStr}})
[
  {
    name: 'mongodb-0-a:27017', health: 0, stateStr: '(not reachable/healthy)'
  },
  { name: 'mongodb-0-b:27017', health: 1, stateStr: 'PRIMARY' },
  {
    name: 'mongodb-0-c:27017', health: 0, stateStr: '(not reachable/healthy)'
  },
  { name: 'mongodb-0-d:27017', health: 1, stateStr: 'SECONDARY' },
  { name: 'mongodb-0-e:27017', health: 1, stateStr: 'SECONDARY' },
  { name: 'mongodb-0-f:27017', health: 1, stateStr: 'SECONDARY' },
  { name: 'mongodb-a:27017', health: 1, stateStr: 'ARBITER' }
]

There are scenarios where node 2 may not be in a position to assume primary though. If another remaining member (4, 5 or 6 ) have more recent writes than node 2 then node 2 will enter a catchup state. The node would report as Primary but would not process writes until caughtup.

See catchUpTimeoutMillis

Another possibility is that the priority: 0 members also have votes: 0 . This would lead to not enough votes to become primary as votingMembersCount: 4 requires a majorityVoteCount: 3 in this case add a vote or remove a vote/arbiter.

6 member + arbiter, 3 priority:0 votes:0, 2 hosts down || click to expand... 
s0 [direct: secondary] test> rs.conf().members.map(x => {return {host: x.host, votes: x.votes, priority:x.priority}})
[
  { host: 'mongodb-0-a:27017', votes: 1, priority: 1 },
  { host: 'mongodb-0-b:27017', votes: 1, priority: 0.5 },
  { host: 'mongodb-0-c:27017', votes: 1, priority: 0.5 },
  { host: 'mongodb-0-d:27017', votes: 0, priority: 0 },
  { host: 'mongodb-0-e:27017', votes: 0, priority: 0 },
  { host: 'mongodb-0-f:27017', votes: 0, priority: 0 },
  { host: 'mongodb-a:27017', votes: 1, priority: 0 }
]
s0 [direct: secondary] test> rs.status().members.map(x => {return {name: x.name, health:x.health, stateStr:x.stateStr}})
[
  {
    name: 'mongodb-0-a:27017', health: 0, stateStr: '(not reachable/healthy)'
  },
  { name: 'mongodb-0-b:27017', health: 1, stateStr: 'SECONDARY' },
  {
    name: 'mongodb-0-c:27017', health: 0, stateStr: '(not reachable/healthy)'
  },
  { name: 'mongodb-0-d:27017', health: 1, stateStr: 'SECONDARY' },
  { name: 'mongodb-0-e:27017', health: 1, stateStr: 'SECONDARY' },
  { name: 'mongodb-0-f:27017', health: 1, stateStr: 'SECONDARY' },
  { name: 'mongodb-a:27017', health: 1, stateStr: 'ARBITER' }
]
2 Likes
3

Hi Chris,

Thanks for the feedback , as you mentioned , i also check the number of votes needed and base from the logs , yes number of voters is insufficient. Is it possible that one of the nodes that has a 0 priority can be a voter ? thanks

4

Hi @Daniel_Inciong

Ye, if you looks at my first example: 6 member + arbiter, 3 priority :0, 2 hosts down || click to expand... all of the priority: 0 have votes: 1. Remember to keep the total number of votes to an odd number.

5

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.