We have upgraded the mongodb data base from 3.2 to 4.0 as replica set and after that we are unable to start the database on secondary node

Ops and Admin
1

We have upgraded the Mongo db from 3.2->3.4->3.6->4.0.27 after that we are facing issue when we are trying to start the database using “systemctl” it is giveing us some error and service is not starting but when we are using “mongod --replSet “rs0” --bind_ip 0.0.0.0” command, we can see the data base is running and it’s also successfully attached in replica set, also we are able to access the mongo console.

There are three node in the replica set and mentioned issue is occurring in the secondary node.

Pasting the error below:

2021-11-16T19:35:06.723+0000 F REPL [repl writer worker 0] writer worker caught exception: BadValue: Invalid value for version, found 3.2, expected '4.0' or '3.6'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.2" }. See . on: { op: "u", ns: "admin.system.version", o: { _id: "featureCompatibilityVersion", version: "3.2" }, o2: { _id: "featureCompatibilityVersion" }, ts: Timestamp(1636726345, 1), t: 1, h: -4488578241200778356, v: 2 }

2021-11-16T19:35:06.723+0000 F REPL [initandlisten] Failed to apply batch of operations. Number of operations in batch: 1. First operation: { op: "u", ns: "admin.system.version", o: { _id: "featureCompatibilityVersion", version: "3.2" }, o2: { _id: "featureCompatibilityVersion" }, ts: Timestamp(1636726345, 1), t: 1, h: -4488578241200778356, v: 2 }. Last operation: { op: "u", ns: "admin.system.version", o: { _id: "featureCompatibilityVersion", version: "3.2" }, o2: { _id: "featureCompatibilityVersion" }, ts: Timestamp(1636726345, 1), t: 1, h: -4488578241200778356, v: 2 }. Oplog application failed in writer thread 8: BadValue: Invalid value for version, found 3.2, expected '4.0' or '3.6'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.2" }. See http://dochub.mongodb.org/core/4.0-feature-compatibility.

2021-11-16T19:35:06.723+0000 F REPL [initandlisten] Caught exception during replication recovery: BadValue: Invalid value for version, found 3.2, expected '4.0' or '3.6'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.2" }. 

2021-11-16T19:35:06.723+0000 F - [initandlisten] terminate() called. An exception is active; attempting to gather more information

2021-11-16T19:35:06.732+0000 F - [initandlisten] DBException::toString(): BadValue: Invalid value for version, found 3.2, expected '4.0' or '3.6'. Contents of featureCompatibilityVersion document in admin.system.version: { _id: "featureCompatibilityVersion", version: "3.2" }. 
Actual exception type: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)2, mongo::AssertionException>
0x561302f9c541 0x561302f9bf25 0x5613030913a6 0x5613030913f1 0x5613017ed50c 0x56130182ab05 0x56130182b4f7 0x56130154552c 0x5613015477bb 0x5613014d3669 0x7f1563c99840 0x561301541d59*

Thanks,
Adway Deshpande

2

Hi @Adway_Deshpande welcome to the community

I think the error message is quite descriptive regarding the issue:

BadValue: Invalid value for version, found 3.2, expected ‘4.0’ or ‘3.6’. Contents of featureCompatibilityVersion document in admin.system.version

What I’m guessing happened is that when the upgrade was being done, the binaries are upgraded but the feature compatibility version step of the upgrade was not done for some reason. Please review the page in Upgrade a Replica Set to 4.0 and ensure that all steps are done for a successful upgrade.

Just in case, I would recommend you to also back up your data should there be issues with the upgrade.

Best regards
Kevin

3

Hi Kevin,

As mention by you we have checked the compatibility versions of the primary and secondary and it is showing that the version is ‘4.0’.

rs0:SECONDARY> db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )
{
        "featureCompatibilityVersion" : {
                "version" : "4.0"
        },
        "ok" : 1,
        "operationTime" : Timestamp(1637229199, 1),
        "$clusterTime" : {
                "clusterTime" : Timestamp(1637229199, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        }

Also when we are trying to start the Mongodb we can see the version i.e. 4.0.27.

When we try to start the mongdb with ‘systemctl’ command the services are not getting started but when we use ‘mongod --replSet “rs0” --bind_ip 0.0.0.0’ it is starting but after sometime i.e. 8 or 9 hours it is again going down automatically.

For the other node we are able to start the mongo service with ‘systemctl’ command.

Thanks,
Adway.

4

Hi All ,

we have renamed dbPath folder and again created new dbpath folder . after that we are able to start mongod service, but if we need old data of dbpath then we might restore data to older directory. so could you please suggest how we can start the services with older dbpath directory content.

Thanks ​