Failed to start WiredTiger after system upgrade

I recently was having so many problems with my computer (gdm issues unrelated to mongo), that I wound up upgrading my OS from Ubuntu 20.04 to Pop_OS 20.10. In the process, I needed to reinstall mongo-org. Fortunately, my old database was on a secondary drive and hence did not get refreshed.

However, when I try to start mongod, it is failing to start. I’m getting the following message in the log file.

{"t":{"$date":"2020-12-15T15:27:43.546-05:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"127.0.0.1","port":27017},"processManagement":{"timeZoneInfo":"/usr/share/zoneinfo"},"storage":{"dbPath":"/home1/mongo/db/","journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"path":"/home1/mongo/log/mongod.log"}}}}
{"t":{"$date":"2020-12-15T15:27:43.590-05:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/home1/mongo/db/","storageEngine":"wiredTiger"}}
{"t":{"$date":"2020-12-15T15:27:43.590-05:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
{"t":{"$date":"2020-12-15T15:27:43.590-05:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=15515M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2020-12-15T15:27:43.974-05:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2020-12-15T15:27:43.974-05:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"95: Operation not supported"}}
{"t":{"$date":"2020-12-15T15:27:43.975-05:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":1123}}
{"t":{"$date":"2020-12-15T15:27:43.975-05:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

I’m guessing that I upgraded mongo in the process of upgrading the rest of the system, and that somehow the old database is not adjusting itself for compatability?

Is there some kind of database repair I can run to fix this problem? (Fortunately, I can restore any mission critical data from backup if need be).

Hi Russell_Almond,

Share the below detail-
1.) Version of MongoDB server before upgrade.
2.) Version of MongoDB server after upgrade.

Thanks
Braj Mohan

No, you’re going to have to run a mongod that can handle that compatibility level. Best use the last version that you ran. Then upgrade following the procedures in the release note.

I’m still not sure of the original version, but given that I’m at version 3.6 on my laptop, it looks like a multistep process.

Fortunately, in my case, the critical data is backed up as JSON files, so it is probably faster to kill the old database and then rebuild from JSON.

Is there an easy way to tell what version of Mongo the database was built under? That might help for a more informative error message.

@Russell_Almond see this post. It show which version it successfully ran under last. Use the most recent metrics file.

1 Like

Just because this pops up first when searching for the Error

Failed to start WiredTiger after system upgrade

In our case we had mongo-4.2.8 running and wanted to upgrade to mongo-4.4.4 which should work just fine, but the featureCompatibilityVersion was still set to “4.0” instead of “4.2” so the upgrade did not succeed.

You can solve this easily with:

  • downgrade to 4.2.8 (in our case) again
  • Follow the migration steps to set the featureCompatibilityVersion to 4.2
  • Shut down your database (or replica set) cleanly as stated in the migration guide
  • Upgrade the binary to 4.4.4 (in our case)
  • Everything should start correctly. Now you set the featureCompatibilityVersion to 4.4 for future upgrades.
3 Likes

thx, this is really helping :+1:

1 Like