Re-add member in replica

Everyone, greetings …

I am a new user of mongoDB, and I set up a laboratory to study and train the replication configuration (1 Master and 2 slaves)

The initial configuration and putting everything to work, despite having gone through some errors, I managed to complete and leave the entire replication structure between the members.

When I tried to delete a member and re-add it didn’t work.
To delete the member, I used the command rs.remove (“mongodb02: 27017”) on the master node. And on the mongodb02 node I stopped the mongodb service and then deleted all files in the mongo’s data folder.

To include mongodb02 again in the replication group I did the following: on the master node, I ran the command rs.add (“mongodb02: 27017”). And on the slave node is I started the mongodb service.

On the slave node (mongodb02) the files are recreated, however the member does not return for replication.

In rs.status () in master node, I have the following return

grp-replic-001:PRIMARY> rs.status()
{
        "set" : "grp-replic-001",
        "date" : ISODate("2020-09-18T21:14:06.597Z"),
        "myState" : 1,
        "term" : NumberLong(6),
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "majorityVoteCount" : 2,
        "writeMajorityCount" : 2,
        "votingMembersCount" : 3,
        "writableVotingMembersCount" : 3,
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1600463636, 1),
                        "t" : NumberLong(6)
                },
                "lastCommittedWallTime" : ISODate("2020-09-18T21:13:56.959Z"),
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1600463636, 1),
                        "t" : NumberLong(6)
                },
                "readConcernMajorityWallTime" : ISODate("2020-09-18T21:13:56.959                                                                                      Z"),
                "appliedOpTime" : {
                        "ts" : Timestamp(1600463636, 1),
                        "t" : NumberLong(6)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1600463636, 1),
                        "t" : NumberLong(6)
                },
                "lastAppliedWallTime" : ISODate("2020-09-18T21:13:56.959Z"),
                "lastDurableWallTime" : ISODate("2020-09-18T21:13:56.959Z")
        },
        "lastStableRecoveryTimestamp" : Timestamp(1600463596, 1),
        "electionCandidateMetrics" : {
                "lastElectionReason" : "electionTimeout",
                "lastElectionDate" : ISODate("2020-09-18T20:34:20.457Z"),
                "electionTerm" : NumberLong(6),
                "lastCommittedOpTimeAtElection" : {
                        "ts" : Timestamp(0, 0),
                        "t" : NumberLong(-1)
                },
                "lastSeenOpTimeAtElection" : {
                        "ts" : Timestamp(1600451358, 1),
                        "t" : NumberLong(5)
                },
                "numVotesNeeded" : 2,
                "priorityAtElection" : 1,
                "electionTimeoutMillis" : NumberLong(10000),
                "numCatchUpOps" : NumberLong(0),
                "newTermStartDate" : ISODate("2020-09-18T20:34:24.959Z"),
                "wMajorityWriteAvailabilityDate" : ISODate("2020-09-18T20:34:26.                                                                                      581Z")
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "mongodb01:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 2874,
                        "optime" : {
                                "ts" : Timestamp(1600463636, 1),
                                "t" : NumberLong(6)
                        },
                        "optimeDate" : ISODate("2020-09-18T21:13:56Z"),
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "electionTime" : Timestamp(1600461260, 1),
                        "electionDate" : ISODate("2020-09-18T20:34:20Z"),
                        "configVersion" : 3,
                        "configTerm" : 6,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                },
                {
                        "_id" : 2,
                        "name" : "mongodb03:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 2395,
                        "optime" : {
                                "ts" : Timestamp(1600463636, 1),
                                "t" : NumberLong(6)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1600463636, 1),
                                "t" : NumberLong(6)
                        },
                        "optimeDate" : ISODate("2020-09-18T21:13:56Z"),
                        "optimeDurableDate" : ISODate("2020-09-18T21:13:56Z"),
                        "lastHeartbeat" : ISODate("2020-09-18T21:14:05.430Z"),
                        "lastHeartbeatRecv" : ISODate("2020-09-18T21:14:06.250Z"                                                                                      ),
                        "pingMs" : NumberLong(1),
                        "lastHeartbeatMessage" : "",
                        "syncSourceHost" : "mongodb01:27017",
                        "syncSourceId" : 0,
                        "infoMessage" : "",
                        "configVersion" : 3,
                        "configTerm" : 6
                },
                {
                        "_id" : 3,
                        "name" : "mongodb02:27017",
                        "health" : 0,
                        "state" : 8,
                        "stateStr" : "(not reachable/healthy)",
                        "uptime" : 0,
                        "optime" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(0, 0),
                                "t" : NumberLong(-1)
                        },
                        "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                        "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                        "lastHeartbeat" : ISODate("2020-09-18T21:14:06.037Z"),
                        "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                        "pingMs" : NumberLong(2),
                        "lastHeartbeatMessage" : "Our replica set configuration                                                                                       is invalid or does not include us",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : -2,
                        "configTerm" : -1
                }
        ],
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1600463636, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1600463636, 1)
}

But, rs.status() in node slave, I return

> rs.status()
{
        "operationTime" : Timestamp(0, 0),
        "ok" : 0,
        "errmsg" : "Our replica set config is invalid or we are not a member of it",
        "code" : 93,
        "codeName" : "InvalidReplicaSetConfig",
        "$clusterTime" : {
                "clusterTime" : Timestamp(1600463767, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }
}
>
>

Could someone help me find where I’m going wrong?

Hello @Mauricio_Tavares_Lei, welcome to the community.

When I tried to delete a member and re-add it didn’t work.
To delete the member, I used the command rs.remove (“mongodb02: 27017”) on the master node. And on the mongodb02 node I stopped the mongodb service and then deleted all files in the mongo’s data folder.

This means you just have to add a new member to the existing replica-set.

The following are the steps to add a new member to the existing replica-set. The links refer to documentation from the tutorial “Add Members to a Replica Set”. Make sure you follow these steps and see it works.

Requirements:

  • An active replica set.
  • A new MongoDB system capable of supporting your data set, accessible by the active replica set through the network.

Procedures:

  1. Prepare the Data Directory: This is about creating a data directory for the new member and manually copying data to it from an existing member.
  2. Add a Member to an Existing Replica Set:
    2.1 Start the new member’s mongod instance, pointing to the new data directory.
    2.2 Connect to the replica-set’s primary.
    2.3 Add the new member, using the rs.add() command.
    2.4 Check the status with rs.status() and see that the new member has the “SECONDARY” status.

Hi @Prasad_Saya,

I did the procedures you suggested, and the error persists …
I checked the slave node’s log file, and you’re giving this message …

{"t":{"$date":"2020-09-21T00:55:59.519-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-0","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}
{"t":{"$date":"2020-09-21T00:56:00.030-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-2","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}
{"t":{"$date":"2020-09-21T00:56:00.541-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-4","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}
{"t":{"$date":"2020-09-21T00:56:01.046-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-0","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}
{"t":{"$date":"2020-09-21T00:56:01.556-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-4","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}
{"t":{"$date":"2020-09-21T00:56:02.059-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-0","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}
{"t":{"$date":"2020-09-21T00:56:02.564-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-2","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}
{"t":{"$date":"2020-09-21T00:56:03.069-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-0","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}
{"t":{"$date":"2020-09-21T00:56:03.574-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-4","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}
{"t":{"$date":"2020-09-21T00:56:04.076-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-2","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}
{"t":{"$date":"2020-09-21T00:56:04.593-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-4","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}
{"t":{"$date":"2020-09-21T00:56:05.123-03:00"},"s":"I",  "c":"REPL_HB",  "id":23974,   "ctx":"ReplCoord-0","msg":"Heartbeat failed after max retries","attr":{"target":"mongodb03:27017","maxHeartbeatRetries":2,"error":{"code":93,"codeName":"InvalidReplicaSetConfig","errmsg":"Our replica set configuration is invalid or does not include us"}}}

Is it some kind of conflict between the two slave nodes?

are the node hosts as follows:
mongodb01 - 192.168.56.103 (node master)
mongodb02 - 192.168.56.104 (node slave - troubled node)
mongodb03 - 192.168.56.102 (node slave)

Hi @Mauricio_Tavares_Lei.

What was the status of the replica-set before adding the new member? Check the status was correct with the existing two members. Then add the new member using the above steps.

What is the MongoDB version you are using? Are you using any configuration files?

If possible, please post the exact commands of the steps you had tried.

when the 3 nodes were working, I had this return from the rs.status () command

cluster-001:SECONDARY> rs.status()
    {
            "set" : "econsular-cluster-001",
            "date" : ISODate("2020-09-21T04:11:21.789Z"),
            "myState" : 2,
            "term" : NumberLong(7),
            "syncSourceHost" : "mongodb01:27017",
            "syncSourceId" : 0,
            "heartbeatIntervalMillis" : NumberLong(2000),
            "majorityVoteCount" : 2,
            "writeMajorityCount" : 2,
            "votingMembersCount" : 3,
            "writableVotingMembersCount" : 3,
            "optimes" : {
                    "lastCommittedOpTime" : {
                            "ts" : Timestamp(1600478411, 1),
                            "t" : NumberLong(7)
                    },
                    "lastCommittedWallTime" : ISODate("2020-09-19T01:20:11.056Z"),
                    "readConcernMajorityOpTime" : {
                            "ts" : Timestamp(1600478411, 1),
                            "t" : NumberLong(7)
                    },
                    "readConcernMajorityWallTime" : ISODate("2020-09-19T01:20:11.056Z"),
                    "appliedOpTime" : {
                            "ts" : Timestamp(1600478411, 1),
                            "t" : NumberLong(7)
                    },
                    "durableOpTime" : {
                            "ts" : Timestamp(1600478411, 1),
                            "t" : NumberLong(7)
                    },
                    "lastAppliedWallTime" : ISODate("2020-09-19T01:20:11.056Z"),
                    "lastDurableWallTime" : ISODate("2020-09-19T01:20:11.056Z")
            },
            "lastStableRecoveryTimestamp" : Timestamp(1600478350, 1),
            "electionParticipantMetrics" : {
                    "votedForCandidate" : true,
                    "electionTerm" : NumberLong(7),
                    "lastVoteDate" : ISODate("2020-09-21T03:51:25.877Z"),
                    "electionCandidateMemberId" : 0,
                    "voteReason" : "",
                    "lastAppliedOpTimeAtElection" : {
                            "ts" : Timestamp(1600477027, 1),
                            "t" : NumberLong(6)
                    },
                    "maxAppliedOpTimeInSet" : {
                            "ts" : Timestamp(1600477027, 1),
                            "t" : NumberLong(6)
                    },
                    "priorityAtElection" : 1,
                    "newTermStartDate" : ISODate("2020-09-19T00:58:39.312Z"),
                    "newTermAppliedDate" : ISODate("2020-09-21T03:51:25.911Z")
            },
            "members" : [
                    {
                            "_id" : 0,
                            "name" : "mongodb01:27017",
                            "health" : 1,
                            "state" : 1,
                            "stateStr" : "PRIMARY",
                            "uptime" : 1197,
                            "optime" : {
                                    "ts" : Timestamp(1600478411, 1),
                                    "t" : NumberLong(7)
                            },
                            "optimeDurable" : {
                                    "ts" : Timestamp(1600478411, 1),
                                    "t" : NumberLong(7)
                            },
                            "optimeDate" : ISODate("2020-09-19T01:20:11Z"),
                            "optimeDurableDate" : ISODate("2020-09-19T01:20:11Z"),
                            "lastHeartbeat" : ISODate("2020-09-21T04:11:21.451Z"),
                            "lastHeartbeatRecv" : ISODate("2020-09-21T04:11:20.655Z"),
                            "pingMs" : NumberLong(1),
                            "lastHeartbeatMessage" : "",
                            "syncSourceHost" : "",
                            "syncSourceId" : -1,
                            "infoMessage" : "",
                            "electionTime" : Timestamp(1600477119, 1),
                            "electionDate" : ISODate("2020-09-19T00:58:39Z"),
                            "configVersion" : 5,
                            "configTerm" : 7
                    },
                    {
                            "_id" : 2,
                            "name" : "mongodb03:27017",
                            "health" : 1,
                            "state" : 2,
                            "stateStr" : "SECONDARY",
                            "uptime" : 1200,
                            "optime" : {
                                    "ts" : Timestamp(1600478411, 1),
                                    "t" : NumberLong(7)
                            },
                            "optimeDate" : ISODate("2020-09-19T01:20:11Z"),
                            "syncSourceHost" : "mongodb01:27017",
                            "syncSourceId" : 0,
                            "infoMessage" : "",
                            "configVersion" : 5,
                            "configTerm" : 7,
                            "self" : true,
                            "lastHeartbeatMessage" : ""
                    },

Version 4.4
Yes, I am using the replication settings via the mongo.conf file

In the replication session I inserted the following parameters

replication:
  oplogSizeMB: 128
  replSetName: econsular-cluster-001
  enableMajorityReadConcern: true
  1. Node mongodb01
    1.1. mongo
    1.2. rs.remove(“mongodb02:27017”)

  2. Node mongodb02
    1.1. systemctl stop mongod
    1.2. rm -rf /var/lib/mongodb/

  3. Node mongodb03
    2.1. systemctl stop mongod

  4. Node mongodb02
    3.1. rsync -arv root@192.168.56.102:/var/lib/mongo/ .
    3.2. systemctl start mongod

  5. Node mongodb01
    4.1. rs.add(“mongodb02:27017”)

Hi @Mauricio_Tavares_Lei, I tried the same actions: From a 3 member replica-set removed a member and added it again. See the steps I followed on a Windows and MongoDB v4.2.8.

1. Create a replica-set with three members:

conf1.cfg:

storage:
  dbPath: G:\mongo\mongo-stuff\misc\db\node1
net:
  bindIp: localhost
  port: 27011
systemLog:
  destination: file
  path: G:\mongo\mongo-stuff\misc\db\node1\mongod.log
  logAppend: true
replication:
  replSetName: myrset

Similarly, I created conf2.cfg and conf3.cfg on ports 27012 and 27013 with data directories on node2 and node3 folders (instead of node1).

Started the three mongods:

mongod -f conf1.cfg
mongod -f conf2.cfg
mongod -f conf3.cfg

Connected to a member:

mongo --port 27011

Initiated the replica-set and added the other two members.

rs.initiate()
rs.add("localhost:27012")
rs.add("localhost:27013")

rs.status()

All okay, replica-set created and running.
Add some data, etc.

2. Remove a member:

I connected to the secondary member “localhost:27012”, and stopped the instance.

use admin
db.shutdownServer()

[ NOTE: Corrected the typo rs.shutdownServer() to db.shutdownServer() ]

From the primary, removed the member “localhost:27012” from the replica-set:

rs.remove("localhost:27012")

Deleted the data directory of member (“localhost:27012”): G:\mongo\mongo-stuff\misc\db\node2

3. Adding the member again (“localhost:27012” to the replica-set)

Prepared the Data Directory:

  • Created the directory G:\mongo\mongo-stuff\misc\db\node2
  • Copied the data (all the files except the log) from the node3 to node2. There were no writes happening when I made a copy of the data.

NOTE from documentation: If your storage system does not support snapshots, you can copy the files directly using cp, rsync, or a similar tool. Since copying multiple files is not an atomic operation, you must stop all writes to the mongod before copying the files. Otherwise, you will copy the files in an invalid state.

Started the instance of the tobe-added member.

mongod -f conf2.cfg

Connect to replica-set’s primary. Add the new member “localhost:27012”.
Check the status before and after adding.

rs.status()
rs.add("localhost:27012")
rs.status()

All okay.

1 Like

My friend, your script worked perfectly …
Thank you very much!

I will use this new environment that I set up based on your script and will continue testing.

There is only one thing that I was very doubtful about.
In the step that you tell me to copy the files from another node but with the exception of the log, what was the log you were referring to?
If it is opLog, help me find it, because I don’t know where it is recorded.

Hi @Mauricio_Tavares_Lei , it is just the server log. But, you could be having this log at different location.

In this case it would be the log that is configured there in the configuration file ???

systemLog:
  destination: file
  path: G:\mongo\mongo-stuff\misc\db\node1\mongod.log
  logAppend: true

Correct. That is the log file I was referring to.

Interesting!!!

Well, now that my laboratory is working, I will proceed with the studies, because now I need to understand how to mount the backup and restore including the oplog. And that I already have some doubts …

@Prasad_Saya , thank you very much for the help and attention you gave to my problem !!!

I hope one day I can help you too.

1 Like

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.