Recover from "commit timestamp is less than the oldest timestamp"

Hi,

I’m am testing how mongod (4.2.8) is handling system date modification with replicaSet configuration. There are different problems:

  • the 2038 date cf Mongodb replicaset cluster maximum logicalTime => 2038 year max. We have bypassed this problem by clipping the system date to 2037
  • the maxAcceptableLogicalClockDriftSecs ste to 1year per default, if the user is changing its system date while running mongoDB instance in a cluster (of only one node). We have bypassed this problem by setting setParameter: maxAcceptableLogicalClockDriftSecs: 2207520000 (70 years). Do you see any drawbacks of doing such configuration ? Moreover we are delete the system.local database at boot since we are having a replicationSet of only 1 node (in order to use change stream functionality).
  • the “commit timestamp is less than the oldest timestamp” error that makes mongod crash when applying the replicationSet configuration.

It’s the last problem that is bothering me. I am not sure how to reproduce it but it should occurs when doing this workflow

  • user set the system date to 2070 => mongod crash

  • the user reboots => we are clamping the system date to 2037

  • the user change the system date to 2021

  • the user reboot => after the reboot we are starting the mongod as replicationSet and are applying the below configuration => which make mongod crash

    client = pymongo.MongoClient(‘mongodb://127.0.0.1:27017’)
    config = {’_id’: ‘rs0’, ‘members’: [{’_id’: 0, ‘host’: ‘127.0.0.1:27017’}]}
    client.admin.command(‘replSetInitiate’, config)

logs

    2020-01-01T01:15:40.582+0000 I  NETWORK  [listener] connection accepted from 127.0.0.1:45184 #1 (1 connection now open)
2020-01-01T01:15:40.583+0000 I  NETWORK  [conn1] received client metadata from 127.0.0.1:45184 conn1: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.15.0-54-lowlatency" }, platform: "CPython 3.7.3.final.0" }
2020-01-01T01:15:40.584+0000 I  NETWORK  [listener] connection accepted from 127.0.0.1:45186 #2 (2 connections now open)
2020-01-01T01:15:40.584+0000 I  NETWORK  [conn2] received client metadata from 127.0.0.1:45186 conn2: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "4.15.0-54-lowlatency" }, platform: "CPython 3.7.3.final.0" }
2020-01-01T01:15:40.585+0000 I  REPL     [conn2] replSetInitiate admin command received from client
2020-01-01T01:15:40.585+0000 I  REPL     [conn2] replSetInitiate config object with 1 members parses ok
2020-01-01T01:15:40.585+0000 I  STORAGE  [conn2] createCollection: local.system.replset with generated UUID: e389f0cc-6d30-4178-a9e7-7e96ce9e9061 and options: {}
2020-01-01T01:15:40.589+0000 I  INDEX    [conn2] index build: done building index _id_ on ns local.system.replset
2020-01-01T01:15:40.589+0000 E  STORAGE  [conn2] WiredTiger error (22) [1577841340:589883][48749:0x7fd095d37700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1577841340, 1) is less than the oldest timestamp (2114386099, 16): Invalid argument Raw: [1577841340:589883][4
8749:0x7fd095d37700], WT_SESSION.timestamp_transaction: __wt_txn_set_commit_timestamp, 683: commit timestamp (1577841340, 1) is less than the oldest timestamp (2114386099, 16): Invalid argument
2020-01-01T01:15:40.589+0000 F  -        [conn2] Fatal assertion 28560 BadValue: timestamp_transaction 22: Invalid argument at src/mongo/db/repl/local_oplog_info.cpp 133
2020-01-01T01:15:40.589+0000 F  -        [conn2] 

***aborting after fassert() failure


2020-01-01T01:15:40.615+0000 F  -        [conn2] Got signal: 6 (Aborted).
=----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"563F6A833000","o":"28BE5C1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"563F6A833000","o":"28BDBDC"},{"b":"563F6A833000","o":"28BDC66"},{"b":"7FD0A8FA1000","o":"12730"},{"b":"7FD0A8DE0000","o":"377BB","s":"gsignal"},{"b":"7FD0A8DE0000","o":"22535","s":"abort"},{"b":"563F6A833000","o":"D148CB","s":"_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj"},{"b":"563F6A833000","o":"B7FFA6"},{"b":"563F6A833000","o":"1679B73","s":"_ZN5mongo4repl5logOpEPNS_16OperationContextEPKcRKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEERKNS_7BSONObjEPSD_bNS_6Date_tERKNS_20OperationSessionInfoENS9_IiEERKNS0_9OplogLinkERKNS0_6OpTimeE"},{"b":"563F6A833000","o":"1461ADB"},{"b":"563F6A833000","o":"146329D","s":"_ZN5mongo14OpObserverImpl19onInternalOpMessageEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEERKNS_7BSONObjENS7_ISA_EE"},{"b":"563F6A833000","o":"DA214F","s":"_ZN5mongo18OpObserverRegistry19onInternalOpMessageEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEERKNS_7BSONObjENS7_ISA_EE"},{"b":"563F6A833000","o":"103ED16","s":"_ZN5mongo10OpObserver11onOpMessageEPNS_16OperationContextERKNS_7BSONObjE"},{"b":"563F6A833000","o":"103987A"},{"b":"563F6A833000","o":"1039B3B","s":"_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl24initializeReplSetStorageEPNS_16OperationContextERKNS_7BSONObjE"},{"b":"563F6A833000","o":"1077280","s":"_ZN5mongo4repl26ReplicationCoordinatorImpl22processReplSetInitiateEPNS_16OperationContextERKNS_7BSONObjEPNS_14BSONObjBuilderE"},{"b":"563F6A833000","o":"1032568","s":"_ZN5mongo4repl18CmdReplSetInitiate3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE"},{"b":"563F6A833000","o":"22AD78F","s":"_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE"},{"b":"563F6A833000","o":"118BECE"},{"b":"563F6A833000","o":"118E614"},{"b":"563F6A833000","o":"118F3AA","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE"},{"b":"563F6A833000","o":"117CF2C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"563F6A833000","o":"1188CBC","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"563F6A833000","o":"1185F9F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"563F6A833000","o":"1187C2C"},{"b":"563F6A833000","o":"1FE3CA2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"563F6A833000","o":"1182E6D","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"563F6A833000","o":"11846E3","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"563F6A833000","o":"1185296","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"563F6A833000","o":"1185EFB","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"563F6A833000","o":"1187C2C"},{"b":"563F6A833000","o":"1FE410B"},{"b":"563F6A833000","o":"26399D5"},{"b":"563F6A833000","o":"2639A34"},{"b":"7FD0A8FA1000","o":"7FA3"},{"b":"7FD0A8DE0000","o":"F94CF","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.8", "gitVersion" : "43d25964249164d76d5e04dd6cf38f6111e21f5f", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-54-lowlatency", "version" : "#58-Ubuntu SMP PREEMPT Mon Jun 24 11:45:10 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "563F6A833000", "elfType" : 3, "buildId" : "20706737C609D70734BA93E4512D1197C0298271" }, { "b" : "7FFD15B47000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "8E01B5042FF894DB5D8616C392BEDDE0B89FAA59" }, { "b" : "7FD0A9505000", "path" : "/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "3F557C34A9F02DD9040B5D4F6EEC46521B0ED3F9" }, { "b" : "7FD0A94EB000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "026C3BA167F64F631EB8781FCA2269FBC2EE7CA5" }, { "b" : "7FD0A9202000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "749142D5EE728A76E7CDC61FD79D2311A77405A2" }, { "b" : "7FD0A9170000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "7DF8C588A7CA055565FE8BD49A604746FB2EB1AE" }, { "b" : "7FD0A916B000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D3583C742DD47AAA860C5AE0C0C5BDBCD2D54F61" }, { "b" : "7FD0A9161000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "5DCF98AD684962BE494AF28A1051793FD39E4EBC" }, { "b" : "7FD0A8FDC000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "885DDA4B4A5CEA600E7B5B98C1AD86996C8D2299" }, { "b" : "7FD0A8FC2000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "DE6B14E57AEA9BBEAF1E81EB6772E2222101AA6E" }, { "b" : "7FD0A8FA1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "E91114987A0147BD050ADDBD591EB8994B29F4B3" }, { "b" : "7FD0A8DE0000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "18B9A9A8C523E5CFE5B5D946D605D09242F09798" }, { "b" : "7FD0A959E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F25DFD7B95BE4BA386FD71080ACCAE8C0732B711" }, { "b" : "7FD0A8DB8000", "path" : "/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "11070FEAA71B4F7C2E5714A61B66028FA86EAE5E" }, { "b" : "7FD0A8D99000", "path" : "/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "93835C08B4818817E355044CEF05F7F5BA573386" }, { "b" : "7FD0A8B7A000", "path" : "/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "F8F137851A6C9F76F2AFB296C77499E3DB004E4B" }, { "b" : "7FD0A8B4C000", "path" : "/lib/x86_64-linux-gnu/libssh2.so.1", "elfType" : 3, "buildId" : "4AEBD6D1D4181EACBCA6F6E30CB293A73FF25FD4" }, { "b" : "7FD0A8B39000", "path" : "/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "E7463248F4FD5ADA5D53F36A7F11BA66C9A7DA3C" }, { "b" : "7FD0A8AEC000", "path" : "/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "FC25787CDEC113023F7F3EF1A85C44C2B96E7DE5" }, { "b" : "7FD0A8A0C000", "path" : "/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "A878F36044AB31951F1A275F2134757C5854B87C" }, { "b" : "7FD0A89D8000", "path" : "/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "BE410728CA057034FC08978FC156B9F0553D7D71" }, { "b" : "7FD0A89D0000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "DFFD546CDF7248805473C118886139F88BF01415" }, { "b" : "7FD0A897C000", "path" : "/lib/x86_64-linux-gFD0A9170000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "7DF8C588A7CA055565FE8BD49A604746FB2EB1AE" }, { "b" : "7FD0A916B000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D3583C742DD47AAA860C5AE0C0C5BDBCD2D54F61" }, { "b" : "7FD0A9161000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "5DCF98AD684962BE494AF28A1051793FD39E4EBC" }, { "b" : "7FD0A8FDC000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "885DDA4B4A5CEA600E7B5B98C1AD86996C8D2299" }, { "b" : "7FD0A8FC2000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "DE6B14E57AEA9BBEAF1E81EB6772E2222101AA6E" }, { "b" : "7FD0A8FA1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "E91114987A0147BD050ADDBD591EB8994B29F4B3" }, { "b" : "7FD0A8DE0000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "18B9A9A8C523E5CFE5B5D946D605D09242F09798" }, { "b" : "7FD0A959E000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "F25DFD7B95BE4BA386FD71080ACCAE8C0732B711" }, { "b" : "7FD0A8DB8000", "path" : "/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "11070FEAA71B4F7C2E5714A61B66028FA86EAE5E" }, { "b" : "7FD0A8D99000", "path" : "/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "93835C08B4818817E355044CEF05F7F5BA573386" }, { "b" : "7FD0A8B7A000", "path" : "/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "F8F137851A6C9F76F2AFB296C77499E3DB004E4B" }, { "b" : "7FD0A8B4C000", "path" : "/lib/x86_64-linux-gnu/libssh2.so.1", "elfType" : 3, "buildId" : "4AEBD6D1D4181EACBCA6F6E30CB293A73FF25FD4" }, { "b" : "7FD0A8B39000", "path" : "/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "E7463248F4FD5ADA5D53F36A7F11BA66C9A7DA3C" }, { "b" : "7FD0A8AEC000", "path" : "/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "FC25787CDEC113023F7F3EF1A85C44C2B96E7DE5" }, { "b" : "7FD0A8A0C000", "path" : "/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "A878F36044AB31951F1A275F2134757C5854B87C" }, { "b" : "7FD0A89D8000", "path" : "/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "BE410728CA057034FC08978FC156B9F0553D7D71" }, { "b" : "7FD0A89D0000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "DFFD546CDF7248805473C118886139F88BF01415" }, { "b" : "7FD0A897C000", "path" : "/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "28276FC4A83169D06A484C4463705F082A038F66" }, { "b" : "7FD0A896B000", "path" : "/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "F8E79FDAC986891B754C708238ECE03F83EF370F" }, { "b" : "7FD0A874D000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "3AF7C4BCEB19B6C83F76E2822B9A23041D85F6D1" }, { "b" : "7FD0A85C9000", "path" : "/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "2B976CABA5F5BF345388917673C45EE626A576D0" }, { "b" : "7FD0A841A000", "path" : "/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "BF7D77672D078E8B8E4CE26A48E80E3770237D37" }, { "b" : "7FD0A83E1000", "path" : "/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "B548A14003EE05ADA36686A3B48D1913BACD540D" }, { "b" : "7FD0A83A9000", "path" : "/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "696C145020FC52F49A604B409E80C0F604514CBE" }, { "b" : "7FD0A8326000", "path" : "/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "CF7737ED0FEB1A97D13F3EF9BBAD9AE2E0EEEF48" }, { "b" : "7FD0A8208000", "path" : "/lib/x86_64-linux-gnu/libgcrypt.so.20", "elfType" : 3, "buildId" : "74F7633B27B762D83B0C72F001917B3424313C5B" }, { "b" : "7FD0A81F9000", "path" : "/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "C54EF3584F013660152971AE32E58D4078BB80BE" }, { "b" : "7FD0A81F0000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "B33B7F30AEA5D2BC14A939FA750862D09A4AC80E" }, { "b" : "7FD0A81D3000", "path" : "/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "99BF5A225908FD4124228D4F3E19C67D7138144F" }, { "b" : "7FD0A80A4000", "path" : "/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "7B23874F495FC6224CFE5D00B0268F309A659301" }, { "b" : "7FD0A7E91000", "path" : "/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "9D60C41CEC3F57BC859B75C1E834187E04DF7C99" }, { "b" : "7FD0A7E6E000", "path" : "/lib/x86_64-linux-gnu/libgpg-error.so.0", "elfType" : 3, "buildId" : "0B8984CF2F0DD4F4901E9100CDB9410D7EBE7930" }, { "b" : "7FD0A7E62000", "path" : "/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "9ED5213748F3F5D008D615DFF0368A6E38E1DE55" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x563f6d0f15c1]
 mongod(+0x28BDBDC) [0x563f6d0f0bdc]
 mongod(+0x28BDC66) [0x563f6d0f0c66]
 libpthread.so.0(+0x12730) [0x7fd0a8fb3730]
 libc.so.6(gsignal+0x10B) [0x7fd0a8e177bb]
 libc.so.6(abort+0x121) [0x7fd0a8e02535]
 mongod(_ZN5mongo42fassertFailedWithStatusNoTraceWithLocationEiRKNS_6StatusEPKcj+0x0) [0x563f6b5478cb]
 mongod(+0xB7FFA6) [0x563f6b3b2fa6]
 mongod(_ZN5mongo4repl5logOpEPNS_16OperationContextEPKcRKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEERKNS_7BSONObjEPSD_bNS_6Date_tERKNS_20OperationSessionInfoENS9_IiEERKNS0_9OplogLinkERKNS0_6OpTimeE+0x4E3) [0x563f6beacb73]
 mongod(+0x1461ADB) [0x563f6bc94adb]
 mongod(_ZN5mongo14OpObserverImpl19onInternalOpMessageEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEERKNS_7BSONObjENS7_ISA_EE+0x11D) [0x563f6bc9629d]
 mongod(_ZN5mongo18OpObserverRegistry19onInternalOpMessageEPNS_16OperationContextERKNS_15NamespaceStringEN5boost8optionalINS_4UUIDEEERKNS_7BSONObjENS7_ISA_EE+0xDF) [0x563f6b5d514f]
 mongod(_ZN5mongo10OpObserver11onOpMessageEPNS_16OperationContextERKNS_7BSONObjE+0x56) [0x563f6b871d16]
 mongod(+0x103987A) [0x563f6b86c87a]
 mongod(_ZN5mongo4repl39ReplicationCoordinatorExternalStateImpl24initializeReplSetStorageEPNS_16OperationContextERKNS_7BSONObjE+0xAB) [0x563f6b86cb3b]
 mongod(_ZN5mongo4repl26ReplicationCoordinatorImpl22processReplSetInitiateEPNS_16OperationContextERKNS_7BSONObjEPNS_14BSONObjBuilderE+0x850) [0x563f6b8aa280]
 mongod(_ZN5mongo4repl18CmdReplSetInitiate3runEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjERNS_14BSONObjBuilderE+0xF58) [0x563f6b865568]
 mongod(_ZN5mongo12BasicCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE+0xAF) [0x563f6cae078f]
 mongod(+0x118BECE) [0x563f6b9beece]
 mongod(+0x118E614) [0x563f6b9c1614]
 mongod(_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE+0x41A) [0x563f6b9c23aa]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3C) [0x563f6b9aff2c]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xEC) [0x563f6b9bbcbc]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x17F) [0x563f6b9b8f9f]
 mongod(+0x1187C2C) [0x563f6b9bac2c]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x182) [0x563f6c816ca2]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x10D) [0x563f6b9b5e6d]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x753) [0x563f6b9b76e3]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x316) [0x563f6b9b8296]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xDB) [0x563f6b9b8efb]
 mongod(+0x1187C2C) [0x563f6b9bac2c]
 mongod(+0x1FE410B) [0x563f6c81710b]
 mongod(+0x26399D5) [0x563f6ce6c9d5]
 mongod(+0x2639A34) [0x563f6ce6ca34]
 libpthread.so.0(+0x7FA3) [0x7fd0a8fa8fa3]
 libc.so.6(clone+0x3F) [0x7fd0a8ed94cf]
-----  END BACKTRACE  -----

Questions:

  • What does the “commit timestamp is less than the oldest timestamp” means ? What is the oldest_timestamp used in __wt_txn_set_commit_timestamp, ? How can I retrieve it ? How to recover the application from such state ?
  • Do you see any drawbacks of using setParameter: maxAcceptableLogicalClockDriftSecs: 2207520000 (70 years) ?