Since the last reboot of my server I can longer access mongo at all. When I attempt to run the mongodb service I see the following error logged in /var/log/mongodb/mongod.log:
{
"t": {"$date":"2022-08-16T12:45:18.536-04:00"},
"s": "F",
"c": "CONTROL",
"id": 20573,
"ctx": "initandlisten",
"msg": "Wrong mongod version",
"attr": {
"error": "UPGRADE PROBLEM: Found an invalid featureCompatibilityVersion document (ERROR: Location4926900: Invalid featureCompatibilityVersion document in admin.system.version: { _id: \"featureCompatibilityVersion\", version: \"4.4\" }. See https://docs.mongodb.com/master/release-notes/5.0-compatibility/#feature-compatibility. :: caused by :: Invalid feature compatibility version value, expected '5.0' or '5.3' or '6.0. See https://docs.mongodb.com/master/release-notes/5.0-compatibility/#feature-compatibility.). If the current featureCompatibilityVersion is below 5.0, see the documentation on upgrading at https://docs.mongodb.com/master/release-notes/5.0/#upgrade-procedures."
}
}
The instructions for upgrading at the provided link require me to run an adminCommand from the mongo shell, which I cannot do because I cannot access it at all.
The mongo shell is gone:
$ mongo
-bash: mongo: command not found
And the mongosh shell doesn’t work:
$ mongosh
Current Mongosh Log ID: [redacted]
Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&appName=mongosh+1.5.4
MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017
My lock file is empty, but I attempted to repair mongod anyway to no avail.
$ ls -l /var/lib/mongodb/mongod.lock
-rw------- 1 mongodb daemon 0 Aug 16 12:45 /var/lib/mongodb/mongod.lock
I don’t really care about recovering my databases, just getting mongo back into a working state. However, I don’t even know how I can wipe my databases without being able to start the mongo service at all. How can I get mongo running again?
I have been able to replicate this error by starting up MongoDB 4.4.15. Once it was running I stopped it. I then started up MongoDB 6.0.0 and pointed it at the same path that the 4.4.15 version wrote its database files to and I got the FCV error and the process stopped. This seems to be what you are seeing.
If you want to save your data, you can fix this, by downloading the compressed archive of 5.0.x and extract it. From this extracted folder you can run ./bin/mongod --dbpath <current database path>. Connect to this instance with mongosh and run db.adminCommand( { setFeatureCompatibilityVersion: "5.0" } ). This will change the FCV for you. You can then exit mongosh and then shutdown the mongod instance and finally start your version 6.0.0 server. You will want to change the FCV here as well to be 6.0.
If you don’t care about any of your data, you can just delete the entire database directory that the current data is stored in.
Thanks! This worked like a charm. Since I’m on Arch Linux I just needed to install mongodb50-bin from the AUR to downgrade, ran the adminCommand, then installed mongodb-bin again to upgrade my binary back to 6.0.0.
Is it good practice to update the FCV after each major release of mongo? Is there a reason it doesn’t happen automatically?
Good to know in case I run into similar issues in the future.
I think the reason for that is that if you need to roll back due to a failed upgrade having the FCV automatically change could cause even more problems. It’s better to leave the FCV at the upgraded from version for a bit to make sure things work as planned and then only change that to the upgraded to version.