Lab: Deploy a Replica Set -- ERROR while adding secondary nodes to the replica set

Unable to add secondary nodes to the replica set after primary node is initiated.
Below are the complete steps

user@M103# ls -lrt
total 3
-rw-rw-r--    1 1000     1000           271 Nov 24 01:03 mongod_2.conf
-rw-rw-r--    1 1000     1000           271 Nov 24 01:04 mongod_3.conf
-rw-rw-r--    1 1000     1000           271 Nov 24 01:04 mongod_1.conf
user@M103# openssl rand -base64 741 > /var/mongodb/pki/m103-keyfile
user@M103# chmod 400 /var/mongodb/pki/m103-keyfile
user@M103# mongod -config mongod_1.conf
about to fork child process, waiting until server is ready for connections.
forked process: 333
child process started successfully, parent exiting
user@M103# mongod -config mongod_2.conf
about to fork child process, waiting until server is ready for connections.
forked process: 366
child process started successfully, parent exiting
user@M103# mongod -config mongod_3.conf
about to fork child process, waiting until server is ready for connections.
forked process: 394
child process started successfully, parent exiting
user@M103# mongo --port 27001
MongoDB shell version v4.0.5
connecting to: mongodb://127.0.0.1:27001/?gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("c235e601-30a7-4b48-a6cf-76d16921cadb") }
MongoDB server version: 4.0.5
> rs.initiate()
{
        "info2" : "no configuration specified. Using a default configuration for the set",
        "me" : "localhost:27001",
        "ok" : 1
}

m103-repl:OTHER> use admin
switched to db admin

m103-repl:PRIMARY> db.createUser({
...   user: "m103-admin",
...   pwd: "m103-pass",
...   roles: [
...     {role: "root", db: "admin"}
...   ]
... })
Successfully added user: {
        "user" : "m103-admin",
        "roles" : [
                {
                        "role" : "root",
                        "db" : "admin"
                }
        ]
}
user@M103# mongo --host "m103-repl/localhost:27001" -u "m103-admin" -p "m103-pass" --authenticationDatabase "admin"
MongoDB shell version v4.0.5
connecting to: mongodb://localhost:27001/?authSource=admin&gssapiServiceName=mongodb&replicaSet=m103-repl
2021-11-24T01:16:26.598+0000 I NETWORK  [js] Starting new replica set monitor for m103-repl/localhost:27001
2021-11-24T01:16:26.618+0000 I NETWORK  [js] Successfully connected to localhost:27001 (1 connections now open to localhost:27001 with a 5 second timeout)
Implicit session: session { "id" : UUID("4e3cda3f-8493-4ac3-96dc-2e3647a3afa5") }
MongoDB server version: 4.0.5
Server has startup warnings: 
2021-11-24T01:13:43.071+0000 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2021-11-24T01:13:43.071+0000 I CONTROL  [initandlisten] 
---
m103-repl:PRIMARY> rs.status()
{
        "set" : "m103-repl",
        "date" : ISODate("2021-11-24T01:17:09.148Z"),
        "myState" : 1,
        "term" : NumberLong(1),
        "syncingTo" : "",
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1637716628, 1),
                        "t" : NumberLong(1)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1637716628, 1),
                        "t" : NumberLong(1)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1637716628, 1),
                        "t" : NumberLong(1)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1637716628, 1),
                        "t" : NumberLong(1)
                }
        },
        "lastStableCheckpointTimestamp" : Timestamp(1637716618, 1),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "localhost:27001",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 208,
                        "optime" : {
                                "ts" : Timestamp(1637716628, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2021-11-24T01:17:08Z"),
                        "syncingTo" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "electionTime" : Timestamp(1637716497, 2),
                        "electionDate" : ISODate("2021-11-24T01:14:57Z"),
                        "configVersion" : 1,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1637716628, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1637716628, 1),
                "signature" : {
                        "hash" : BinData(0,"Q8Fw/cpAPOmx+jMJxJ2yvmhAXp8="),
                        "keyId" : NumberLong("7033938799029649410")
                }
        }
}
m103-repl:PRIMARY> rs.add("localhost:27002")
{
        "operationTime" : Timestamp(1637716718, 1),
        "ok" : 0,
        "errmsg" : "Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: localhost:27001; the following nodes did not respond affirmatively: localhost:27002 failed with not running with --replSet",
        "code" : 74,
        "codeName" : "NodeNotFound",
        "$clusterTime" : {
                "clusterTime" : Timestamp(1637716718, 1),
                "signature" : {
                        "hash" : BinData(0,"PWT771ozy6PXe5LN4BSzk5N4RzE="),
                        "keyId" : NumberLong("7033938799029649410")
                }
        }
}
m103-repl:PRIMARY> rs.add("localhost:27003")
{
        "operationTime" : Timestamp(1637716778, 1),
        "ok" : 0,
        "errmsg" : "Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: localhost:27001; the following nodes did not respond affirmatively: localhost:27003 failed with not running with --replSet",
        "code" : 74,
        "codeName" : "NodeNotFound",
        "$clusterTime" : {
                "clusterTime" : Timestamp(1637716778, 1),
                "signature" : {
                        "hash" : BinData(0,"lD0JdlCTO5e14pWeI6M3USMvLdU="),
                        "keyId" : NumberLong("7033938799029649410")
                }
        }
}

I have also verified the mongod processes for all 3 nodes are up and running.

user@M103# ps -ef|grep mongod
333 root 0:08 mongod -config mongod_1.conf
366 root 0:04 mongod -config mongod_2.conf
394 root 0:04 mongod -config mongod_3.conf
502 root 0:00 grep mongod
user@M103#

1 Like

Check if your config files for other two nodes are ok or not
Are they using same replicaset name?

One of the best, if not the best, post ever that gives almost everything we need to help.

The content of configuration files is also important to have. It looks like, as mentioned by Ramachandra_Tummala, that the replication options are not what they should be.

The output of netstat -tlnp is also nice to have.