Hi,
I have a replica set on which replication was going fine. Suddenly I am seeing below value in replication status output and looks like there is an issue with replication. What might be causing it and any resolution for this please?
optimes" : { “lastCommittedOpTime” : { “ts” : { “$timestamp” : { “t” : 0, “i” : 0 } }, “t” : -1 },
Thanks,
Akshaya Srinivasan
Welcome @Akshaya_Srinivasan!
To help investigate this problem can you please share:
lastCommittedOpTime set to 0rs.status() output from the current primary with the hostnames redactedThanks,
Stennie
Hi Stennie,
lastCommittedOpTime set to 0 --> I do not have authentication enabled on this server. Server is started and access as Windows administrator user.rs.status() output from the current primary with the hostnames redacted myShard_0:PRIMARY> rs.status()
{
"set" : "myShard_0",
"date" : ISODate("2020-03-16T06:53:24.078Z"),
"myState" : 1,
"term" : NumberLong(57),
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(0, 0),
"t" : NumberLong(-1)
},
"appliedOpTime" : {
"ts" : Timestamp(1584341601, 1),
"t" : NumberLong(57)
},
"durableOpTime" : {
"ts" : Timestamp(1584341601, 1),
"t" : NumberLong(57)
}
},
"members" : [
{
"_id" : 0,
"name" : "xxxx:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 256214,
"optime" : {
"ts" : Timestamp(1584084717, 1),
"t" : NumberLong(2)
},
"optimeDurable" : {
"ts" : Timestamp(1584084717, 1),
"t" : NumberLong(2)
},
"optimeDate" : ISODate("2020-03-13T07:31:57Z"),
"optimeDurableDate" : ISODate("2020-03-13T07:31:57Z"),
"lastHeartbeat" : ISODate("2020-03-16T06:53:22.760Z"),
"lastHeartbeatRecv" : ISODate("2020-03-16T06:53:22.777Z"
),
"pingMs" : NumberLong(0),
"syncingTo" : "xxxx:27018",
"configVersion" : 104289
},
{
"_id" : 1,
"name" : "xxxx:27018",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 256217,
"optime" : {
"ts" : Timestamp(1584341601, 1),
"t" : NumberLong(57)
},
"optimeDate" : ISODate("2020-03-16T06:53:21Z"),
"electionTime" : Timestamp(1584086043, 1),
"electionDate" : ISODate("2020-03-13T07:54:03Z"),
"configVersion" : 104289,
"self" : true
}
],
"ok" : 1
}
I would strongly recommend upgrading to the latest (and final) version of MongoDB 3.4 (3.4.24). There have been a significant number of bug fixes and stability improvements in the 3.4 release series and upgrading will confirm you are not encountering a known issue since 3.4.5 was released back in June, 2017. Minor releases do not introduce any backward-breaking changes.
i would also note that MongoDB 3.4 reached end of life in January, 2020 so I would also plan to upgrade to 3.6 for continued stability & security improvements.
I was referring to the roles of replica set members (primary, secondary, arbiter). However, that isn’t relevant since you have provided the rs.status() info.
A lastCommittedOpTime of -1 is expected if you are running a replica set with the older replication protocol 0 (pv0).
You can confirm your protocol version using rs.conf().protocolVersion.
Regards,
Stennie
Thanks Stennie.
Protocol version:
myShard_0:PRIMARY> rs.conf().protocolVersion
NumberLong(1)
myShard_0:PRIMARY>
Hrm, in this case the lastCommittedOpTime should normally have a t value.
Can you also share the output of rs.conf() from your primary with the hostnames redacted?
In your original post you mentioned replication was going fine previously. Did anything specific change in your deployment (configuration, upgrades, election, …) to prompt noticing this change in the replication status output?
Thanks,
Stennie
Thanks Stennie. I just did role reversal. Primary to secondary and vice versa.
myShard_0:PRIMARY> rs.conf()
{
"_id" : "myShard_0",
"version" : 104289,
"protocolVersion" : NumberLong(1),
"members" : [
{
"_id" : 0,
"host" : "xxx:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 1,
"host" : "cccwin:27018",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 0.5,
"tags" : {
},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"catchUpTimeoutMillis" : 2000,
"getLastErrorModes" : {
},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("5e6b3245ccc189cd553cf737")
}
}
myShard_0:PRIMARY>How can I resolve this @Stennie_X. Please help.
Thanks,
Akshaya Srinivasan
Hi,
I am seeing this issue again. i am using MongoDB version 3.6.20 , OS is RHEL 6.7 . Please can some one help? Also protocolVersion is not listed in the rs.conf() output.
Secondary node:
[root@akscent1 Log_Files]# /root/mongodb-linux-x86_64-rhel62-3.6.20/bin/mongo --port 27097 -u xxxx -p’xxxx’ --authenticationDatabase admin --eval “rs.status().optimes.lastCommittedOpTime”
MongoDB shell version v3.6.20
{ “ts” : Timestamp(0, 0), “t” : NumberLong(-1) }
Secondary node:
[root@akscent1 Log_Files]# /root/mongodb-linux-x86_64-rhel62-3.6.20/bin/mongo --port 27098 -u xxxx -p’xxxx’ --authenticationDatabase admin --eval “rs.status().optimes.lastCommittedOpTime”
MongoDB shell version v3.6.20
{ “ts” : Timestamp(0, 0), “t” : NumberLong(-1) }
Primary node:
[root@akscent1 Log_Files]# /root/mongodb-linux-x86_64-rhel62-3.6.20/bin/mongo --port 27099 -u xxxx -p’xxxxx’ --authenticationDatabase admin --eval “rs.status().optimes.lastCommittedOpTime”
MongoDB shell version v3.6.20
{ “ts” : Timestamp(1607574936, 1), “t” : NumberLong(-1) }
Thanks,
Akshaya Srinivasan
What’s the whole output of rs.status() and rs.conf()? Also please show the output of rs.printReplicationInfo().
Best regards,
Kevin