Not able to resync my MongoDB database

Ops and Admin Replication
1

Hello

I have a configuration with 4 MongoDB databases version 3.4.6 on Redhat 6.10

• Node3 and Node4 are running one site
• Node1 and Node2 are running on another remote site
• Node1 is the PRIMARY node, Node2 is SECONDARY and I have to resync Node3 and Node4.
• I’m trying first to resync Node3 with the primary Node1
• I have launched the standard resync procedure to resync first Node3 as follows:

1- Connect to primary Node1 and remove Node3 from replica configuration with this command:
• rs.remove(“Node3”);

2 - Stop mongo service on node Node3 remove data folder and restart mongo service

3 - ReAdd the node to replica
• rs.add( { host: “Node3” } );

• During the resync Node3 was normally in STARTUP2 status.
• I could see that the Collections and the Indexes were created. It took a lot of time to recreate the Indexes by the way…
• Finally at the end the Node3 to be resynced switched to RECOVERING status without being able to reach SECONDARY status

Please find below ouput of rs.status() | egrep “name|stateStr” command

name" : “NODE3”,
“stateStr” : “RECOVERING”,
“name” : “NODE1”
“stateStr” : “PRIMARY”,
“name” : “NODE2”,
“stateStr” : “SECONDARY”,

Any idea why Node3 is staying in RECOVERING status ? and not be to be fully resynced properly? What can I check?

Many thanks for the suggestions.

2

Hi @MARCELIN_Richard,

It is probably due to resync failures when the resync attempts are exhausted the resync will fail in recovery mode.

There might be several issues:

  1. The new secondary cloud not catchup when the last phase started and fell off oplog.
  2. There was a failure in network or other issues with target source.
  3. The primary has run an operation like rename which is not compatible with automatic resync.
  4. Highly unlikely but the primary or source might be corrupted.

Check the logs of the syncing secondary to see the issue failure.

Thanks
Pavel