Could not add second node to shard cluster

MongoDB University M103: Basic Cluster Administration
#1

Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: 192.168.103.100:26001; the following nodes did not respond affirmatively: 192.168.103.100:26002 failed with Received heartbeat from member with the same member ID as ourself: 0

Config Files 1
sharding:
clusterRole: configsvr
replication:
replSetName: m103-csrs
security:
keyFile: /var/mongodb/pki/m103-keyfile
net:
bindIp: localhost,192.168.103.100
port: 26001
systemLog:
destination: file
path: /var/mongodb/db/csrs1/mongod.log
logAppend: true
processManagement:
fork: true
storage:
dbPath: /var/mongodb/db/csrs1

Config File 2:
sharding:
clusterRole: configsvr
replication:
replSetName: m103-csrs
security:
keyFile: /var/mongodb/pki/m103-keyfile
net:
bindIp: localhost,192.168.103.100
port: 26002
systemLog:cmd
destination: file
path: /var/mongodb/db/csrs2/mongod.log
logAppend: true
processManagement:
fork: true
storage:
dbPath: /var/mongodb/db/csrs2

Config File 3:
sharding:
clusterRole: configsvr
replication:
replSetName: m103-csrs
security:
keyFile: /var/mongodb/pki/m103-keyfile
net:
bindIp: localhost,192.168.103.100
port: 26003
systemLog:
destination: file
path: /var/mongodb/db/csrs3/mongod.log
logAppend: true
processManagement:
fork: true
storage:
dbPath: /var/mongodb/db/csrs3

Error occuring while running the command
rs.add(“192.168.103.100:26002”)

#2

Are your mongods up and running?

M103 “errmsg” : “Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded:

#3

Hi,

Yes they are running.

#4

Are you able to connect to each node and replicaset as a whole?
What does rs.status() & rs.conf() show
Please check mongod.log of each node
Follow the clear instructions from 007_jb in the above link about user creation,connect to local host etc

1 Like
#5

This what i get when connecting to my replicaSet

vagrant@m103:~$ mongod --replSet m103-csrs
2020-03-23T08:48:11.426+0000 I CONTROL [initandlisten] MongoDB starting : pid=4739 port=27017 dbpath=/data/db 64-bit host=m103
2020-03-23T08:48:11.426+0000 I CONTROL [initandlisten] db version v3.6.17
2020-03-23T08:48:11.426+0000 I CONTROL [initandlisten] git version: 3d6953c361213c5bfab23e51ab274ce592edafe6
2020-03-23T08:48:11.426+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
2020-03-23T08:48:11.426+0000 I CONTROL [initandlisten] allocator: tcmalloc
2020-03-23T08:48:11.426+0000 I CONTROL [initandlisten] modules: enterprise
2020-03-23T08:48:11.426+0000 I CONTROL [initandlisten] build environment:
2020-03-23T08:48:11.426+0000 I CONTROL [initandlisten] distmod: ubuntu1404
2020-03-23T08:48:11.426+0000 I CONTROL [initandlisten] distarch: x86_64
2020-03-23T08:48:11.426+0000 I CONTROL [initandlisten] target_arch: x86_64
2020-03-23T08:48:11.426+0000 I CONTROL [initandlisten] options: { replication: { replSet: “m103-csrs” } }
2020-03-23T08:48:11.433+0000 I - [initandlisten] Detected data files in /data/db created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.
2020-03-23T08:48:11.434+0000 I STORAGE [initandlisten]
2020-03-23T08:48:11.434+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2020-03-23T08:48:11.434+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2020-03-23T08:48:11.434+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=488M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release=“3.0”,require_max=“3.0”),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2020-03-23T08:48:12.347+0000 I STORAGE [initandlisten] WiredTiger message [1584953292:347798][4739:0x7fd4372c6ac0], txn-recover: Main recovery loop: starting at 25/6016
2020-03-23T08:48:12.439+0000 I STORAGE [initandlisten] WiredTiger message [1584953292:439654][4739:0x7fd4372c6ac0], txn-recover: Recovering log 25 through 26
2020-03-23T08:48:12.498+0000 I STORAGE [initandlisten] WiredTiger message [1584953292:498004][4739:0x7fd4372c6ac0], txn-recover: Recovering log 26 through 26
2020-03-23T08:48:12.542+0000 I STORAGE [initandlisten] WiredTiger message [1584953292:542209][4739:0x7fd4372c6ac0], txn-recover: Set global recovery timestamp: 0
2020-03-23T08:48:12.558+0000 I CONTROL [initandlisten]
2020-03-23T08:48:12.559+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2020-03-23T08:48:12.559+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2020-03-23T08:48:12.559+0000 I CONTROL [initandlisten]
2020-03-23T08:48:12.560+0000 I CONTROL [initandlisten] ** WARNING: This server is bound to localhost.
2020-03-23T08:48:12.560+0000 I CONTROL [initandlisten] ** Remote systems will be unable to connect to this server.
2020-03-23T08:48:12.560+0000 I CONTROL [initandlisten] ** Start the server with --bind_ip to specify which IP
2020-03-23T08:48:12.561+0000 I CONTROL [initandlisten] ** addresses it should serve responses from, or with --bind_ip_all to
2020-03-23T08:48:12.561+0000 I CONTROL [initandlisten] ** bind to all interfaces. If this behavior is desired, start the
2020-03-23T08:48:12.561+0000 I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disable this warning.
2020-03-23T08:48:12.561+0000 I CONTROL [initandlisten]
2020-03-23T08:48:12.576+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory ‘/data/db/diagnostic.data’
2020-03-23T08:48:12.582+0000 I REPL [initandlisten] Did not find local initialized voted for document at startup.
2020-03-23T08:48:12.586+0000 I REPL [initandlisten] Rollback ID is 1
2020-03-23T08:48:12.586+0000 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset
2020-03-23T08:48:12.586+0000 I NETWORK [initandlisten] listening via socket bound to 127.0.0.1
2020-03-23T08:48:12.587+0000 I CONTROL [LogicalSessionCacheRefresh] Sessions collection is not set up; waiting until next sessions refresh interval: Replication has not yet been configured
2020-03-23T08:48:12.587+0000 I NETWORK [initandlisten] listening via socket bound to /tmp/mongodb-27017.sock
2020-03-23T08:48:12.587+0000 I NETWORK [initandlisten] waiting for connections on port 27017

vagrant@m103:~$ mongo --port 26001
MongoDB shell version v3.6.17
connecting to: mongodb://127.0.0.1:26001/?gssapiServiceName=mongodb
Implicit session: session { “id” : UUID(“d67bf999-cd71-4629-9289-401c6696d76a”) }
MongoDB server version: 3.6.17
MongoDB Enterprise m103-csrs:PRIMARY> use admin
switched to db admin
MongoDB Enterprise m103-csrs:PRIMARY> rs.status()
{
“operationTime” : Timestamp(1584953523, 2),
“ok” : 0,
“errmsg” : “there are no users authenticated”,
“code” : 13,
“codeName” : “Unauthorized”,
“$gleStats” : {
“lastOpTime” : Timestamp(0, 0),
“electionId” : ObjectId(“7fffffff000000000000000b”)
},
“$clusterTime” : {
“clusterTime” : Timestamp(1584953523, 2),
“signature” : {
“hash” : BinData(0,“ZgKuOlPoV2Ra1PQiUTGSyjHunTw=”),
“keyId” : NumberLong(“6805838811130495004”)
}
}
}

MongoDB Enterprise m103-csrs:PRIMARY> rs.config()
2020-03-23T08:52:36.175+0000 E QUERY [thread1] Error: Could not retrieve replica set config: {
“operationTime” : Timestamp(1584953553, 2),
“ok” : 0,
“errmsg” : “there are no users authenticated”,
“code” : 13,
“codeName” : “Unauthorized”,
“$gleStats” : {
“lastOpTime” : Timestamp(0, 0),
“electionId” : ObjectId(“7fffffff000000000000000b”)
},
“$clusterTime” : {
“clusterTime” : Timestamp(1584953553, 2),
“signature” : {
“hash” : BinData(0,“oNs5eukHRiXAI2G5qmflszZpnYc=”),
“keyId” : NumberLong(“6805838811130495004”)
}
}
} :
rs.conf@src/mongo/shell/utils.js:1412:11
@(shell):1:1

Looks like my replication is not up. How do i check the mongod.log files

#6

After rs.initiate() did you create the user and authenticated?
On config server also you need to create the user

#7

Yes i did that

#8

Can you connect to each node of replica with that user and replicaset as a whole?

#9

Hi @Harshal_40444,

In addition to @Ramachandra_37567,

What’s the output of these commands :

ps -ef | grep mongod

mongo --host “m103-csrs/192.168.103.100:26001” -u m103-admin -p m103-pass

~ Shubham

#10

One more clarification
You have shown us the config files but why was below command run?
How did you bring up mongods by config file or command line?

vagrant@m103:~$ mongod --replSet m103-csrs

You don’t connect to replicaset with mongod
Please use mongo

By running mongod you have spun a mongod on default port 27017

Please check if all required mongods are running or not and kill all those not needed

#11

@Harshal_40444, from @Ramachandra_37567’s first post, he suggested that you go through all the points in this specific post. You clearly haven’t done that so I’d suggest that you do before posting.

Lastly, you’re now in Chapter 3. In Chapters 1 and 2, it covered what a mongod and mongo is, how to connect to a replica set, how to authenticate, create first user, check running nodes etc. From all your posts I feel that there’s a gap in knowledge and it will be beneficial for you to revise Chapters 1 and 2 again to get up to speed.

#12
#13