Mongo Cluster: Node unable to join the cluster after failure

Ops and Admin Replication
1

Created a cluster with 3 nodes (A - Primary, B, C). Everything worked fine. Now,

  1. Node C goes down
  2. Node C comes up and starts replicating with primary node A.
  3. While replicating is ongoing, node A (primary node) goes down.
  4. We receive an error in Pymongo saying: “primary stepped down while waiting for replication”
  5. Node A, comes up.
  6. But now, Node C is unable to join the cluster and the cluster status command shows only 2 nodes (A and B).

How can we recover the cluster and make node C functional?

3

Hi @Ronak_Shah and welcome to MongoDB community forums!!

Based on the above deployment configuration, I tried to insert sample data infinitely using the code below while I tried to perform the shutdown from the secondary and then primary as mentioned.

The sample code:

import random
from pymongo import MongoClient
replica_set_name = "rs02"
host1 = 'localhost:6000'
host2 = 'localhost:6001'
host3 = 'localhost:6002'
connection_string = f"mongodb://{host1},{host2},{host3}/?replicaSet={replica_set_name}"

try:
    client = MongoClient(connection_string)
    db = client['test']
    collection = db['sample']

    while True:
        random_number = random.randint(1, 100)
        collection.insert_one({'random_number': random_number})
        print(f"Inserted: {random_number}")

except Exception as e:
    print(f"An error occurred: {e}")

While the insertion was being performed, I tried the following steps:

  1. shutdown one of the secondary, the insertion was still continuing in the primary node.
  2. When the secondary came up, I tried to shutdown the primary node.
    Ideally, when the primary goes down, immediately the election takes places and one between the secondaries is selected as the primary node.
  3. The insertion still continued and the current status of the replica set had one primary and one secondary.
  4. When I restarted the primary, it resynced with the replica set and stayed as primary node.

Please note that the test has been conducted on MongoDB version 6.0.5 and the shutdown performed have been graceful shutdowns.

In order to triage your issue further could you help me with some information as:

  1. A sample pymongo code to replicate the issue.
  2. Are you facing the issue outside the python application?
  3. How does the node goes down?
  4. Do you see any error messages in the logs for the primary or secondary ?
  5. What is the MongoDB version you are using?

Regards
Aasawari