Hi,
I’m having issues upgrading a Debian 9.11 (Stretch) server with MongoDB 3.4.23 to 3.6.17. MongoDB 3.4.23 is from the Jessie repository and 3.6.17 is in the Stretch repository
The cluster is a ‘simple’ 3 node cluster, all with mongod running happily and having been upgraded continuously from the 2.2 days
The upgrade of a secondary appears to go well, until mongod starts, at which point it explodes with an error about DBException::toString(): Location40415: BSON field ‘MinValidDocument.h’ is an unknown field. This happens on any secondary I try to upgrade, and is also reverted reasonably well by just downgrading back to 2.4.23
I’ve replicated this in a Vagrant test cluster, with a copy of our database, but am at a loss as to how to proceed to fix this. Any input would be appreciated.
Cheers
Kyle
2020-02-05T12:04:12.148+0000 I CONTROL [signalProcessingThread] now exiting
2020-02-05T12:04:12.148+0000 I CONTROL [signalProcessingThread] shutting down with code:0
2020-02-05T12:04:58.379+0000 I CONTROL [main] ***** SERVER RESTARTED *****
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] MongoDB starting : pid=19342 port=27017 dbpath=/var/lib/mongodb 64-bit host=mongodb-2
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] db version v3.6.17
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] git version: 3d6953c361213c5bfab23e51ab274ce592edafe6
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.0l 10 Sep 2019
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] allocator: tcmalloc
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] modules: none
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] build environment:
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] distmod: debian92
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] distarch: x86_64
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] target_arch: x86_64
2020-02-05T12:04:58.393+0000 I CONTROL [initandlisten] options: { config: “/etc/mongod.conf”, net: { bindIp: “192.168.56.102”, port: 27017 }, replication: { replSetName: “linndb” }, storage: { dbPath: “/var/lib/mongodb”, journal: { enabled: true } }, systemLog: { destination: “file”, logAppend: true, path: “/var/log/mongodb/mongod.log” } }
2020-02-05T12:04:58.394+0000 I - [initandlisten] Detected data files in /var/lib/mongodb created by the ‘mmapv1’ storage engine, so setting the active storage engine to ‘mmapv1’.
2020-02-05T12:04:58.403+0000 I JOURNAL [initandlisten] journal dir=/var/lib/mongodb/journal
2020-02-05T12:04:58.403+0000 I JOURNAL [initandlisten] recover : no journal files present, no recovery needed
2020-02-05T12:04:58.404+0000 I JOURNAL [durability] Durability thread started
2020-02-05T12:04:58.405+0000 I JOURNAL [journal writer] Journal writer thread started
2020-02-05T12:04:58.405+0000 I CONTROL [initandlisten]
2020-02-05T12:04:58.405+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2020-02-05T12:04:58.405+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2020-02-05T12:04:58.405+0000 I CONTROL [initandlisten]
2020-02-05T12:04:58.913+0000 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory ‘/var/lib/mongodb/diagnostic.data’
2020-02-05T12:04:58.984+0000 I REPL [initandlisten] Did not find local initialized voted for document at startup.
2020-02-05T12:04:58.985+0000 I REPL [initandlisten] Did not find local Rollback ID document at startup. Creating one.
2020-02-05T12:04:58.985+0000 I STORAGE [initandlisten] createCollection: local.system.rollback.id with no UUID.
2020-02-05T12:04:58.986+0000 I REPL [initandlisten] Initialized the rollback ID to 1
2020-02-05T12:04:59.032+0000 F REPL [initandlisten] Caught exception during replication recovery: Location40415: BSON field ‘MinValidDocument.h’ is an unknown field.
2020-02-05T12:04:59.032+0000 F - [initandlisten] terminate() called. An exception is active; attempting to gather more information
2020-02-05T12:04:59.055+0000 F - [initandlisten] DBException::toString(): Location40415: BSON field ‘MinValidDocument.h’ is an unknown field.
Actual exception type: mongo::error_details::throwExceptionForStatus(mongo::Status const&)::NonspecificAssertionException0x55bb915f6511 0x55bb915f5ef5 0x55bb916ea486 0x55bb916ea4d1 0x55bb90183cba 0x55bb904afadf 0x55bb904b027f 0x55bb8fd81d94 0x55bb8fd83db2 0x55bb8fd07ec9 0x7fae51a382e1 0x55bb8fd6c0da
----- BEGIN BACKTRACE -----
{“backtrace”:[{“b”:“55BB8F36B000”,“o”:“228B511”,“s”:“_ZN5mongo15printStackTraceERSo”},{“b”:“55BB8F36B000”,“o”:“228AEF5”},{“b”:“55BB8F36B000”,“o”:“237F486”,“s”:“_ZN10__cxxabiv111__terminateEPFvvE”},{“b”:“55BB8F36B000”,“o”:“237F4D1”},{“b”:“55BB8F36B000”,“o”:“E18CBA”,“s”:“_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextE”},{“b”:“55BB8F36B000”,“o”:“1144ADF”,“s”:“_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE”},{“b”:“55BB8F36B000”,“o”:“114527F”,“s”:“_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE”},{“b”:“55BB8F36B000”,“o”:“A16D94”},{“b”:“55BB8F36B000”,“o”:“A18DB2”,“s”:“ZN5mongo11mongoDbMainEiPPcS1”},{“b”:“55BB8F36B000”,“o”:“99CEC9”,“s”:“main”},{“b”:“7FAE51A18000”,“o”:“202E1”,“s”:“__libc_start_main”},{“b”:“55BB8F36B000”,“o”:“A010DA”,“s”:“_start”}],“processInfo”:{ “mongodbVersion” : “3.6.17”, “gitVersion” : “3d6953c361213c5bfab23e51ab274ce592edafe6”, “compiledModules” : , “uname” : { “sysname” : “Linux”, “release” : “4.9.0-11-amd64”, “version” : “#1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11)”, “machine” : “x86_64” }, “somap” : [ { “b” : “55BB8F36B000”, “elfType” : 3, “buildId” : “E93AC4BD48D98B49945C22FA72B32037154B506B” }, { “b” : “7FFCC75E2000”, “path” : “linux-vdso.so.1”, “elfType” : 3, “buildId” : “E4710F184535B7465A293C8D84196F236A12D2CF” }, { “b” : “7FAE53001000”, “path” : “/lib/x86_64-linux-gnu/libresolv.so.2”, “elfType” : 3, “buildId” : “EAD5FD817712E63C1212D1EE7D7EE1B9C29F93A7” }, { “b” : “7FAE52B67000”, “path” : “/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1”, “elfType” : 3, “buildId” : “BC8265C9A7EB28F1A9677B7437E4561F383C44D9” }, { “b” : “7FAE528FB000”, “path” : “/usr/lib/x86_64-linux-gnu/libssl.so.1.1”, “elfType” : 3, “buildId” : “4EAE4DD6DA20F6CECBEA6688C8ED8E9987CD800F” }, { “b” : “7FAE526F7000”, “path” : “/lib/x86_64-linux-gnu/libdl.so.2”, “elfType” : 3, “buildId” : “DB2CAEEEC37482A98AB1416D0A9AFE2944930DE9” }, { “b” : “7FAE524EF000”, “path” : “/lib/x86_64-linux-gnu/librt.so.1”, “elfType” : 3, “buildId” : “86B35D63FACD97D22973E99EE9863F7714C4F53A” }, { “b” : “7FAE521EB000”, “path” : “/lib/x86_64-linux-gnu/libm.so.6”, “elfType” : 3, “buildId” : “4E49714C557CE0472C798F39365CA10F9C0E1933” }, { “b” : “7FAE51FD4000”, “path” : “/lib/x86_64-linux-gnu/libgcc_s.so.1”, “elfType” : 3, “buildId” : “51AD5FD294CD6C813BED40717347A53434B80B7A” }, { “b” : “7FAE51DB7000”, “path” : “/lib/x86_64-linux-gnu/libpthread.so.0”, “elfType” : 3, “buildId” : “16D609487BCC4ACBAC29A4EAA2DDA0D2F56211EC” }, { “b” : “7FAE51A18000”, “path” : “/lib/x86_64-linux-gnu/libc.so.6”, “elfType” : 3, “buildId” : “775143E680FF0CD4CD51CCE1CE8CA216E635A1D6” }, { “b” : “7FAE53218000”, “path” : “/lib64/ld-linux-x86-64.so.2”, “elfType” : 3, “buildId” : “606DF9C355103E82140D513BC7A25A635591C153” } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55bb915f6511]
mongod(+0x228AEF5) [0x55bb915f5ef5]
mongod(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x55bb916ea486]
mongod(+0x237F4D1) [0x55bb916ea4d1]
mongod(_ZN5mongo4repl23ReplicationRecoveryImpl16recoverFromOplogEPNS_16OperationContextE+0xBDA) [0x55bb90183cba]
mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl21_startLoadLocalConfigEPNS_16OperationContextE+0x52F) [0x55bb904afadf]
mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl7startupEPNS_16OperationContextE+0x1DF) [0x55bb904b027f]
mongod(+0xA16D94) [0x55bb8fd81d94]
mongod(ZN5mongo11mongoDbMainEiPPcS1+0x872) [0x55bb8fd83db2]
mongod(main+0x9) [0x55bb8fd07ec9]
libc.so.6(__libc_start_main+0xF1) [0x7fae51a382e1]
mongod(_start+0x2A) [0x55bb8fd6c0da]
----- END BACKTRACE -----