Mongodb replicaset corruption when electric is gone

Hi,

I’m using mongodb 5.0.6 version with 3 members (PSA) replica set. When electric is gone suddenly, mongodb generally corrupted. After electric is come, mongodb primary and secondary members is not working.

Centos 7.9
Mongodb 5.0.6 with replica set (PSA)
File system ext4

Could you please help me ?

So after power came back what is the status of your mongods
How are they configured?Do they start auto on reboot or you start them manually?
Is this prod or development?
Do you have backups configured
How you came to conclusion it is corrupted
What exactly you mean by not working?
What does mongod.logs show?

2 Likes

Hi Ramachandra
It’s configured with Primary - Secondary - Arbiter. Mongodb is starting with service. So starting auto
It will be prod in 1 month. So it’s critical for us.
Yes we have a backup strategy.
I saw some “database corruption” error in the logs file.
When i run “systemctl start mongod” command, I’m taking below error and mongodb doesn’t open.

mongod.log files is below for 2 data bearing nodes.
I only changed our hostname to “hostname”

Node1:

{“t”:{“$date”:“2022-09-26T08:36:01.858+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:20698, “ctx”:“-”,“msg”:“***** SERVER RESTARTED **“}
{“t”:{”$date":“2022-09-26T08:36:01.858+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:23285, “ctx”:“-”,“msg”:“Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’”}
{“t”:{“$date”:“2022-09-26T08:36:01.863+03:00”},“s”:“I”, “c”:“NETWORK”, “id”:4915701, “ctx”:“main”,“msg”:“Initialized wire specification”,“attr”:{“spec”:{“incomingExternalClient”:{“minWireVersion”:0,“maxWireVersion”:13},“incomingInternalClient”:{“minWireVersion”:0,“maxWireVersion”:13},“outgoing”:{“minWireVersion”:0,“maxWireVersion”:13},“isInternalClient”:true}}}
{“t”:{“$date”:“2022-09-26T08:36:01.863+03:00”},“s”:“W”, “c”:“ASIO”, “id”:22601, “ctx”:“main”,“msg”:“No TransportLayer configured during NetworkInterface startup”}
{“t”:{“$date”:“2022-09-26T08:36:01.863+03:00”},“s”:“I”, “c”:“NETWORK”, “id”:4648601, “ctx”:“main”,“msg”:“Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize.”}
{“t”:{“$date”:“2022-09-26T08:36:01.938+03:00”},“s”:“W”, “c”:“ASIO”, “id”:22601, “ctx”:“main”,“msg”:“No TransportLayer configured during NetworkInterface startup”}
{“t”:{“$date”:“2022-09-26T08:36:01.938+03:00”},“s”:“W”, “c”:“ASIO”, “id”:22601, “ctx”:“main”,“msg”:“No TransportLayer configured during NetworkInterface startup”}
{“t”:{“$date”:“2022-09-26T08:36:01.938+03:00”},“s”:“I”, “c”:“REPL”, “id”:5123008, “ctx”:“main”,“msg”:“Successfully registered PrimaryOnlyService”,“attr”:{“service”:“TenantMigrationDonorService”,“ns”:“config.tenantMigrationDonors”}}
{“t”:{“$date”:“2022-09-26T08:36:01.939+03:00”},“s”:“I”, “c”:“REPL”, “id”:5123008, “ctx”:“main”,“msg”:“Successfully registered PrimaryOnlyService”,“attr”:{“service”:“TenantMigrationRecipientService”,“ns”:“config.tenantMigrationRecipients”}}
{“t”:{“$date”:“2022-09-26T08:36:01.939+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:5945603, “ctx”:“main”,“msg”:“Multi threading initialized”}
{“t”:{“$date”:“2022-09-26T08:36:01.939+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:4615611, “ctx”:“initandlisten”,“msg”:“MongoDB starting”,“attr”:{“pid”:94507,“port”:27017,“dbPath”:“/opt/mongo”,“architecture”:“64-bit”,“host”:“hostname”}}
{“t”:{“$date”:“2022-09-26T08:36:01.939+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:23403, “ctx”:“initandlisten”,“msg”:“Build Info”,“attr”:{“buildInfo”:{“version”:“5.0.6”,“gitVersion”:“212a8dbb47f07427dae194a9c75baec1d81d9259”,“openSSLVersion”:“OpenSSL 1.0.1e-fips 11 Feb 2013”,“modules”:[],“allocator”:“tcmalloc”,“environment”:{“distmod”:“rhel70”,“distarch”:“x86_64”,“target_arch”:“x86_64”}}}}
{“t”:{“$date”:“2022-09-26T08:36:01.939+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:51765, “ctx”:“initandlisten”,“msg”:“Operating System”,“attr”:{“os”:{“name”:“CentOS Linux release 7.9.2009 (Core)”,“version”:“Kernel 3.10.0-1160.el7.x86_64”}}}
{“t”:{“$date”:“2022-09-26T08:36:01.939+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:21951, “ctx”:“initandlisten”,“msg”:“Options set by command line”,“attr”:{“options”:{“config”:“/etc/mongod.conf”,“net”:{“bindIp”:“127.0.0.1,hostname”,“port”:27017},“processManagement”:{“fork”:true,“pidFilePath”:“/var/run/mongodb/mongod.pid”,“timeZoneInfo”:“/usr/share/zoneinfo”},“replication”:{“replSetName”:“eirs”},“security”:{“authorization”:“enabled”,“keyFile”:“/opt/mongo/mongokeyfile”},“storage”:{“dbPath”:“/opt/mongo”,“journal”:{“enabled”:true}},“systemLog”:{“destination”:“file”,“logAppend”:true,“path”:“/var/log/mongodb/mongod.log”}}}}
{“t”:{“$date”:“2022-09-26T08:36:01.940+03:00”},“s”:“W”, “c”:“STORAGE”, “id”:22271, “ctx”:“initandlisten”,“msg”:“Detected unclean shutdown - Lock file is not empty”,“attr”:{“lockFile”:“/opt/mongo/mongod.lock”}}
{“t”:{“$date”:“2022-09-26T08:36:01.941+03:00”},“s”:“I”, “c”:“STORAGE”, “id”:22270, “ctx”:“initandlisten”,“msg”:“Storage engine to use detected by data files”,“attr”:{“dbpath”:“/opt/mongo”,“storageEngine”:“wiredTiger”}}
{“t”:{“$date”:“2022-09-26T08:36:01.941+03:00”},“s”:“W”, “c”:“STORAGE”, “id”:22302, “ctx”:“initandlisten”,“msg”:“Recovering data from the last clean checkpoint.”}
{“t”:{“$date”:“2022-09-26T08:36:01.941+03:00”},“s”:“I”, “c”:“STORAGE”, “id”:22297, “ctx”:“initandlisten”,“msg”:“Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem",“tags”:["startupWarnings”]}
{“t”:{“$date”:“2022-09-26T08:36:01.941+03:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,cache_size=15527M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],”}}
{“t”:{“$date”:“2022-09-26T08:36:02.439+03:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:{“error”:-31809,“message”:“[1664170562:439217][94507:0x7f741c10abc0], connection: __wt_turtle_read, 391: WiredTiger.turtle: fatal turtle file read error: WT_TRY_SALVAGE: database corruption detected”}}
{“t”:{“$date”:“2022-09-26T08:36:02.439+03:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:{“error”:-31804,“message”:“[1664170562:439270][94507:0x7f741c10abc0], connection: __wt_turtle_read, 391: the process must exit and restart: WT_PANIC: WiredTiger library panic”}}
{“t”:{“$date”:“2022-09-26T08:36:02.439+03:00”},“s”:“F”, “c”:“-”, “id”:23089, “ctx”:“initandlisten”,“msg”:“Fatal assertion”,“attr”:{“msgid”:50853,“file”:“src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp”,“line”:538}}
{“t”:{“$date”:“2022-09-26T08:36:02.439+03:00”},“s”:“F”, “c”:“-”, “id”:23090, “ctx”:“initandlisten”,“msg”:"\n\n
aborting after fassert() failure\n\n”}
{“t”:{“$date”:“2022-09-26T08:36:02.439+03:00”},“s”:“F”, “c”:“CONTROL”, “id”:4757800, “ctx”:“initandlisten”,“msg”:“Writing fatal message”,“attr”:{“message”:“Got signal: 6 (Aborted).\n”}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31380, “ctx”:“initandlisten”,“msg”:“BACKTRACE”,“attr”:{“bt”:{“backtrace”:[{“a”:“55B2A3A23FA5”,“b”:“55B29FBA3000”,“o”:“3E80FA5”,“s”:“_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357”,“s+”:“215”},{“a”:“55B2A3A26A39”,“b”:“55B29FBA3000”,“o”:“3E83A39”,“s”:“_ZN5mongo15printStackTraceEv”,“s+”:“29”},{“a”:“55B2A3A1F076”,“b”:“55B29FBA3000”,“o”:“3E7C076”,“s”:“abruptQuit”,“s+”:“66”},{“a”:“7F741A65E630”,“b”:“7F741A64F000”,“o”:“F630”,“s”:“_L_unlock_13”,“s+”:“34”},{“a”:“7F741A2B7387”,“b”:“7F741A281000”,“o”:“36387”,“s”:“gsignal”,“s+”:“37”},{“a”:“7F741A2B8A78”,“b”:“7F741A281000”,“o”:“37A78”,“s”:“abort”,“s+”:“148”},{“a”:“55B2A0F5FBAB”,“b”:“55B29FBA3000”,“o”:“13BCBAB”,“s”:“_ZN5mongo25fassertFailedWithLocationEiPKcj”,“s+”:“F6”},{“a”:“55B2A0A584AC”,“b”:“55B29FBA3000”,“o”:“EB54AC”,“s”:“_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1216”,“s+”:“16”},{“a”:“55B2A1260B63”,“b”:“55B29FBA3000”,“o”:“16BDB63”,“s”:“__eventv”,“s+”:“403”},{“a”:“55B2A0A6AD89”,“b”:“55B29FBA3000”,“o”:“EC7D89”,“s”:“__wt_panic_func”,“s+”:“114”},{“a”:“55B2A0A6487D”,“b”:“55B29FBA3000”,“o”:“EC187D”,“s”:“__wt_turtle_read.cold.7”,“s+”:“4C”},{“a”:“55B2A1228B24”,“b”:“55B29FBA3000”,“o”:“1685B24”,“s”:“__wt_turtle_validate_version”,“s+”:“234”},{“a”:“55B2A11DC49D”,“b”:“55B29FBA3000”,“o”:“163949D”,“s”:“wiredtiger_open”,“s+”:“2B9D”},{“a”:“55B2A11875A9”,“b”:“55B29FBA3000”,“o”:“15E45A9”,“s”:“ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8”,“s+”:“B9”},{“a”:“55B2A1192AA8”,“b”:“55B29FBA3000”,“o”:“15EFAA8”,“s”:“_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb”,“s+”:“1138”},{“a”:“55B2A11694C1”,“b”:“55B29FBA3000”,“o”:“15C64C1”,“s”:“_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createEPNS_16OperationContextERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE”,“s+”:“171”},{“a”:“55B2A1F3AE59”,“b”:“55B29FBA3000”,“o”:“2397E59”,“s”:“_ZN5mongo23initializeStorageEngineEPNS_16OperationContextENS_22StorageEngineInitFlagsE”,“s+”:“419”},{“a”:“55B2A10D2CCD”,“b”:“55B29FBA3000”,“o”:“152FCCD”,“s”:“_ZN5mongo12_GLOBAL__N_114_initAndListenEPNS_14ServiceContextEi.isra.1896”,“s+”:“47D”},{“a”:“55B2A10D564F”,“b”:“55B29FBA3000”,“o”:“153264F”,“s”:“_ZN5mongo11mongod_mainEiPPc”,“s+”:“CDF”},{“a”:“55B2A0F72F0E”,“b”:“55B29FBA3000”,“o”:“13CFF0E”,“s”:“main”,“s+”:“E”},{“a”:“7F741A2A3555”,“b”:“7F741A281000”,“o”:“22555”,“s”:“__libc_start_main”,“s+”:“F5”},{“a”:“55B2A10CFB3E”,“b”:“55B29FBA3000”,“o”:“152CB3E”,“s”:“_start”,“s+”:“29”}],“processInfo”:{“mongodbVersion”:“5.0.6”,“gitVersion”:“212a8dbb47f07427dae194a9c75baec1d81d9259”,“compiledModules”:,“uname”:{“sysname”:“Linux”,“release”:“3.10.0-1160.el7.x86_64”,“version”:“#1 SMP Mon Oct 19 16:18:59 UTC 2020”,“machine”:“x86_64”},“somap”:[{“b”:“55B29FBA3000”,“elfType”:3,“buildId”:“6B144064C4AA51D5B9894904F22879DB438E9C3B”},{“b”:“7F741A64F000”,“path”:“/lib64/libpthread.so.0”,“elfType”:3,“buildId”:“2B482B3BAE79DEF4E5BC9791BC6BBDAE0E93E359”},{“b”:“7F741A281000”,“path”:“/lib64/libc.so.6”,“elfType”:3,“buildId”:“F9FAFDE281E0E0E2AF45911AD0FA115B64C2CEA8”}]}}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A3A23FA5”,“b”:“55B29FBA3000”,“o”:“3E80FA5”,“s”:“_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357”,“s+”:“215”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A3A26A39”,“b”:“55B29FBA3000”,“o”:“3E83A39”,“s”:“_ZN5mongo15printStackTraceEv”,“s+”:“29”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A3A1F076”,“b”:“55B29FBA3000”,“o”:“3E7C076”,“s”:“abruptQuit”,“s+”:“66”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“7F741A65E630”,“b”:“7F741A64F000”,“o”:“F630”,“s”:“_L_unlock_13”,“s+”:“34”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“7F741A2B7387”,“b”:“7F741A281000”,“o”:“36387”,“s”:“gsignal”,“s+”:“37”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“7F741A2B8A78”,“b”:“7F741A281000”,“o”:“37A78”,“s”:“abort”,“s+”:“148”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A0F5FBAB”,“b”:“55B29FBA3000”,“o”:“13BCBAB”,“s”:“_ZN5mongo25fassertFailedWithLocationEiPKcj”,“s+”:“F6”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A0A584AC”,“b”:“55B29FBA3000”,“o”:“EB54AC”,“s”:“_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1216”,“s+”:“16”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A1260B63”,“b”:“55B29FBA3000”,“o”:“16BDB63”,“s”:“__eventv”,“s+”:“403”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A0A6AD89”,“b”:“55B29FBA3000”,“o”:“EC7D89”,“s”:“__wt_panic_func”,“s+”:“114”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A0A6487D”,“b”:“55B29FBA3000”,“o”:“EC187D”,“s”:“__wt_turtle_read.cold.7”,“s+”:“4C”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A1228B24”,“b”:“55B29FBA3000”,“o”:“1685B24”,“s”:“__wt_turtle_validate_version”,“s+”:“234”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A11DC49D”,“b”:“55B29FBA3000”,“o”:“163949D”,“s”:“wiredtiger_open”,“s+”:“2B9D”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A11875A9”,“b”:“55B29FBA3000”,“o”:“15E45A9”,“s”:“ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8”,“s+”:“B9”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A1192AA8”,“b”:“55B29FBA3000”,“o”:“15EFAA8”,“s”:“_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb”,“s+”:“1138”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A11694C1”,“b”:“55B29FBA3000”,“o”:“15C64C1”,“s”:“_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createEPNS_16OperationContextERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE”,“s+”:“171”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A1F3AE59”,“b”:“55B29FBA3000”,“o”:“2397E59”,“s”:“_ZN5mongo23initializeStorageEngineEPNS_16OperationContextENS_22StorageEngineInitFlagsE”,“s+”:“419”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A10D2CCD”,“b”:“55B29FBA3000”,“o”:“152FCCD”,“s”:“_ZN5mongo12_GLOBAL__N_114_initAndListenEPNS_14ServiceContextEi.isra.1896”,“s+”:“47D”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A10D564F”,“b”:“55B29FBA3000”,“o”:“153264F”,“s”:“_ZN5mongo11mongod_mainEiPPc”,“s+”:“CDF”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A0F72F0E”,“b”:“55B29FBA3000”,“o”:“13CFF0E”,“s”:“main”,“s+”:“E”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“7F741A2A3555”,“b”:“7F741A281000”,“o”:“22555”,“s”:“__libc_start_main”,“s+”:“F5”}}}
{“t”:{“$date”:“2022-09-26T08:36:02.570+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55B2A10CFB3E”,“b”:“55B29FBA3000”,“o”:“152CB3E”,“s”:“_start”,“s+”:“29”}}}

node2:

{“t”:{“$date”:“2022-09-26T08:37:05.088+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:20698, “ctx”:“-”,“msg”:“***** SERVER RESTARTED **“}
{“t”:{”$date":“2022-09-26T08:37:05.089+03:00”},“s”:“I”, “c”:“NETWORK”, “id”:4915701, “ctx”:“main”,“msg”:“Initialized wire specification”,“attr”:{“spec”:{“incomingExternalClient”:{“minWireVersion”:0,“maxWireVersion”:13},“incomingInternalClient”:{“minWireVersion”:0,“maxWireVersion”:13},“outgoing”:{“minWireVersion”:0,“maxWireVersion”:13},“isInternalClient”:true}}}
{“t”:{“$date”:“2022-09-26T08:37:05.090+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:23285, “ctx”:“main”,“msg”:“Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols ‘none’”}
{“t”:{“$date”:“2022-09-26T08:37:05.093+03:00”},“s”:“W”, “c”:“ASIO”, “id”:22601, “ctx”:“main”,“msg”:“No TransportLayer configured during NetworkInterface startup”}
{“t”:{“$date”:“2022-09-26T08:37:05.093+03:00”},“s”:“I”, “c”:“NETWORK”, “id”:4648601, “ctx”:“main”,“msg”:“Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize.”}
{“t”:{“$date”:“2022-09-26T08:37:05.168+03:00”},“s”:“W”, “c”:“ASIO”, “id”:22601, “ctx”:“main”,“msg”:“No TransportLayer configured during NetworkInterface startup”}
{“t”:{“$date”:“2022-09-26T08:37:05.169+03:00”},“s”:“W”, “c”:“ASIO”, “id”:22601, “ctx”:“main”,“msg”:“No TransportLayer configured during NetworkInterface startup”}
{“t”:{“$date”:“2022-09-26T08:37:05.169+03:00”},“s”:“I”, “c”:“REPL”, “id”:5123008, “ctx”:“main”,“msg”:“Successfully registered PrimaryOnlyService”,“attr”:{“service”:“TenantMigrationDonorService”,“ns”:“config.tenantMigrationDonors”}}
{“t”:{“$date”:“2022-09-26T08:37:05.169+03:00”},“s”:“I”, “c”:“REPL”, “id”:5123008, “ctx”:“main”,“msg”:“Successfully registered PrimaryOnlyService”,“attr”:{“service”:“TenantMigrationRecipientService”,“ns”:“config.tenantMigrationRecipients”}}
{“t”:{“$date”:“2022-09-26T08:37:05.169+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:5945603, “ctx”:“main”,“msg”:“Multi threading initialized”}
{“t”:{“$date”:“2022-09-26T08:37:05.169+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:4615611, “ctx”:“initandlisten”,“msg”:“MongoDB starting”,“attr”:{“pid”:122834,“port”:27017,“dbPath”:“/opt/mongo”,“architecture”:“64-bit”,“host”:“hostname”}}
{“t”:{“$date”:“2022-09-26T08:37:05.169+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:23403, “ctx”:“initandlisten”,“msg”:“Build Info”,“attr”:{“buildInfo”:{“version”:“5.0.6”,“gitVersion”:“212a8dbb47f07427dae194a9c75baec1d81d9259”,“openSSLVersion”:“OpenSSL 1.0.1e-fips 11 Feb 2013”,“modules”:[],“allocator”:“tcmalloc”,“environment”:{“distmod”:“rhel70”,“distarch”:“x86_64”,“target_arch”:“x86_64”}}}}
{“t”:{“$date”:“2022-09-26T08:37:05.169+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:51765, “ctx”:“initandlisten”,“msg”:“Operating System”,“attr”:{“os”:{“name”:“CentOS Linux release 7.9.2009 (Core)”,“version”:“Kernel 3.10.0-1160.el7.x86_64”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.169+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:21951, “ctx”:“initandlisten”,“msg”:“Options set by command line”,“attr”:{“options”:{“config”:“/etc/mongod.conf”,“net”:{“bindIp”:“127.0.0.1,hostname”,“port”:27017},“processManagement”:{“fork”:true,“pidFilePath”:“/var/run/mongodb/mongod.pid”,“timeZoneInfo”:“/usr/share/zoneinfo”},“replication”:{“replSetName”:“eirs”},“security”:{“authorization”:“enabled”,“keyFile”:“/opt/mongo/mongokeyfile”},“storage”:{“dbPath”:“/opt/mongo”,“journal”:{“enabled”:true}},“systemLog”:{“destination”:“file”,“logAppend”:true,“path”:“/var/log/mongodb/mongod.log”}}}}
{“t”:{“$date”:“2022-09-26T08:37:05.170+03:00”},“s”:“W”, “c”:“STORAGE”, “id”:22271, “ctx”:“initandlisten”,“msg”:“Detected unclean shutdown - Lock file is not empty”,“attr”:{“lockFile”:“/opt/mongo/mongod.lock”}}
{“t”:{“$date”:“2022-09-26T08:37:05.170+03:00”},“s”:“I”, “c”:“STORAGE”, “id”:22270, “ctx”:“initandlisten”,“msg”:“Storage engine to use detected by data files”,“attr”:{“dbpath”:“/opt/mongo”,“storageEngine”:“wiredTiger”}}
{“t”:{“$date”:“2022-09-26T08:37:05.170+03:00”},“s”:“W”, “c”:“STORAGE”, “id”:22302, “ctx”:“initandlisten”,“msg”:“Recovering data from the last clean checkpoint.”}
{“t”:{“$date”:“2022-09-26T08:37:05.170+03:00”},“s”:“I”, “c”:“STORAGE”, “id”:22297, “ctx”:“initandlisten”,“msg”:“Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem",“tags”:["startupWarnings”]}
{“t”:{“$date”:“2022-09-26T08:37:05.170+03:00”},“s”:“I”, “c”:“STORAGE”, “id”:22315, “ctx”:“initandlisten”,“msg”:“Opening WiredTiger”,“attr”:{“config”:“create,cache_size=15527M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],”}}
{“t”:{“$date”:“2022-09-26T08:37:05.674+03:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:{“error”:-31809,“message”:“[1664170625:674753][122834:0x7f77fd5a8bc0], connection: __wt_turtle_read, 391: WiredTiger.turtle: fatal turtle file read error: WT_TRY_SALVAGE: database corruption detected”}}
{“t”:{“$date”:“2022-09-26T08:37:05.674+03:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:{“error”:-31804,“message”:“[1664170625:674804][122834:0x7f77fd5a8bc0], connection: __wt_turtle_read, 391: the process must exit and restart: WT_PANIC: WiredTiger library panic”}}
{“t”:{“$date”:“2022-09-26T08:37:05.674+03:00”},“s”:“F”, “c”:“-”, “id”:23089, “ctx”:“initandlisten”,“msg”:“Fatal assertion”,“attr”:{“msgid”:50853,“file”:“src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp”,“line”:538}}
{“t”:{“$date”:“2022-09-26T08:37:05.674+03:00”},“s”:“F”, “c”:“-”, “id”:23090, “ctx”:“initandlisten”,“msg”:"\n\n
aborting after fassert() failure\n\n”}
{“t”:{“$date”:“2022-09-26T08:37:05.674+03:00”},“s”:“F”, “c”:“CONTROL”, “id”:4757800, “ctx”:“initandlisten”,“msg”:“Writing fatal message”,“attr”:{“message”:“Got signal: 6 (Aborted).\n”}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31380, “ctx”:“initandlisten”,“msg”:“BACKTRACE”,“attr”:{“bt”:{“backtrace”:[{“a”:“55FBFEAF2FA5”,“b”:“55FBFAC72000”,“o”:“3E80FA5”,“s”:“_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357”,“s+”:“215”},{“a”:“55FBFEAF5A39”,“b”:“55FBFAC72000”,“o”:“3E83A39”,“s”:“_ZN5mongo15printStackTraceEv”,“s+”:“29”},{“a”:“55FBFEAEE076”,“b”:“55FBFAC72000”,“o”:“3E7C076”,“s”:“abruptQuit”,“s+”:“66”},{“a”:“7F77FBAFC630”,“b”:“7F77FBAED000”,“o”:“F630”,“s”:“_L_unlock_13”,“s+”:“34”},{“a”:“7F77FB755387”,“b”:“7F77FB71F000”,“o”:“36387”,“s”:“gsignal”,“s+”:“37”},{“a”:“7F77FB756A78”,“b”:“7F77FB71F000”,“o”:“37A78”,“s”:“abort”,“s+”:“148”},{“a”:“55FBFC02EBAB”,“b”:“55FBFAC72000”,“o”:“13BCBAB”,“s”:“_ZN5mongo25fassertFailedWithLocationEiPKcj”,“s+”:“F6”},{“a”:“55FBFBB274AC”,“b”:“55FBFAC72000”,“o”:“EB54AC”,“s”:“_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1216”,“s+”:“16”},{“a”:“55FBFC32FB63”,“b”:“55FBFAC72000”,“o”:“16BDB63”,“s”:“__eventv”,“s+”:“403”},{“a”:“55FBFBB39D89”,“b”:“55FBFAC72000”,“o”:“EC7D89”,“s”:“__wt_panic_func”,“s+”:“114”},{“a”:“55FBFBB3387D”,“b”:“55FBFAC72000”,“o”:“EC187D”,“s”:“__wt_turtle_read.cold.7”,“s+”:“4C”},{“a”:“55FBFC2F7B24”,“b”:“55FBFAC72000”,“o”:“1685B24”,“s”:“__wt_turtle_validate_version”,“s+”:“234”},{“a”:“55FBFC2AB49D”,“b”:“55FBFAC72000”,“o”:“163949D”,“s”:“wiredtiger_open”,“s+”:“2B9D”},{“a”:“55FBFC2565A9”,“b”:“55FBFAC72000”,“o”:“15E45A9”,“s”:“ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8”,“s+”:“B9”},{“a”:“55FBFC261AA8”,“b”:“55FBFAC72000”,“o”:“15EFAA8”,“s”:“_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb”,“s+”:“1138”},{“a”:“55FBFC2384C1”,“b”:“55FBFAC72000”,“o”:“15C64C1”,“s”:“_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createEPNS_16OperationContextERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE”,“s+”:“171”},{“a”:“55FBFD009E59”,“b”:“55FBFAC72000”,“o”:“2397E59”,“s”:“_ZN5mongo23initializeStorageEngineEPNS_16OperationContextENS_22StorageEngineInitFlagsE”,“s+”:“419”},{“a”:“55FBFC1A1CCD”,“b”:“55FBFAC72000”,“o”:“152FCCD”,“s”:“_ZN5mongo12_GLOBAL__N_114_initAndListenEPNS_14ServiceContextEi.isra.1896”,“s+”:“47D”},{“a”:“55FBFC1A464F”,“b”:“55FBFAC72000”,“o”:“153264F”,“s”:“_ZN5mongo11mongod_mainEiPPc”,“s+”:“CDF”},{“a”:“55FBFC041F0E”,“b”:“55FBFAC72000”,“o”:“13CFF0E”,“s”:“main”,“s+”:“E”},{“a”:“7F77FB741555”,“b”:“7F77FB71F000”,“o”:“22555”,“s”:“__libc_start_main”,“s+”:“F5”},{“a”:“55FBFC19EB3E”,“b”:“55FBFAC72000”,“o”:“152CB3E”,“s”:“_start”,“s+”:“29”}],“processInfo”:{“mongodbVersion”:“5.0.6”,“gitVersion”:“212a8dbb47f07427dae194a9c75baec1d81d9259”,“compiledModules”:,“uname”:{“sysname”:“Linux”,“release”:“3.10.0-1160.el7.x86_64”,“version”:“#1 SMP Mon Oct 19 16:18:59 UTC 2020”,“machine”:“x86_64”},“somap”:[{“b”:“55FBFAC72000”,“elfType”:3,“buildId”:“6B144064C4AA51D5B9894904F22879DB438E9C3B”},{“b”:“7F77FBAED000”,“path”:“/lib64/libpthread.so.0”,“elfType”:3,“buildId”:“2B482B3BAE79DEF4E5BC9791BC6BBDAE0E93E359”},{“b”:“7F77FB71F000”,“path”:“/lib64/libc.so.6”,“elfType”:3,“buildId”:“F9FAFDE281E0E0E2AF45911AD0FA115B64C2CEA8”}]}}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFEAF2FA5”,“b”:“55FBFAC72000”,“o”:“3E80FA5”,“s”:“_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.357”,“s+”:“215”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFEAF5A39”,“b”:“55FBFAC72000”,“o”:“3E83A39”,“s”:“_ZN5mongo15printStackTraceEv”,“s+”:“29”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFEAEE076”,“b”:“55FBFAC72000”,“o”:“3E7C076”,“s”:“abruptQuit”,“s+”:“66”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“7F77FBAFC630”,“b”:“7F77FBAED000”,“o”:“F630”,“s”:“_L_unlock_13”,“s+”:“34”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“7F77FB755387”,“b”:“7F77FB71F000”,“o”:“36387”,“s”:“gsignal”,“s+”:“37”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“7F77FB756A78”,“b”:“7F77FB71F000”,“o”:“37A78”,“s”:“abort”,“s+”:“148”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFC02EBAB”,“b”:“55FBFAC72000”,“o”:“13BCBAB”,“s”:“_ZN5mongo25fassertFailedWithLocationEiPKcj”,“s+”:“F6”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFBB274AC”,“b”:“55FBFAC72000”,“o”:“EB54AC”,“s”:“_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold.1216”,“s+”:“16”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFC32FB63”,“b”:“55FBFAC72000”,“o”:“16BDB63”,“s”:“__eventv”,“s+”:“403”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFBB39D89”,“b”:“55FBFAC72000”,“o”:“EC7D89”,“s”:“__wt_panic_func”,“s+”:“114”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFBB3387D”,“b”:“55FBFAC72000”,“o”:“EC187D”,“s”:“__wt_turtle_read.cold.7”,“s+”:“4C”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFC2F7B24”,“b”:“55FBFAC72000”,“o”:“1685B24”,“s”:“__wt_turtle_validate_version”,“s+”:“234”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFC2AB49D”,“b”:“55FBFAC72000”,“o”:“163949D”,“s”:“wiredtiger_open”,“s+”:“2B9D”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFC2565A9”,“b”:“55FBFAC72000”,“o”:“15E45A9”,“s”:“ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8”,“s+”:“B9”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFC261AA8”,“b”:“55FBFAC72000”,“o”:“15EFAA8”,“s”:“_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbbbb”,“s+”:“1138”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFC2384C1”,“b”:“55FBFAC72000”,“o”:“15C64C1”,“s”:“_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createEPNS_16OperationContextERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE”,“s+”:“171”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFD009E59”,“b”:“55FBFAC72000”,“o”:“2397E59”,“s”:“_ZN5mongo23initializeStorageEngineEPNS_16OperationContextENS_22StorageEngineInitFlagsE”,“s+”:“419”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFC1A1CCD”,“b”:“55FBFAC72000”,“o”:“152FCCD”,“s”:“_ZN5mongo12_GLOBAL__N_114_initAndListenEPNS_14ServiceContextEi.isra.1896”,“s+”:“47D”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFC1A464F”,“b”:“55FBFAC72000”,“o”:“153264F”,“s”:“_ZN5mongo11mongod_mainEiPPc”,“s+”:“CDF”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFC041F0E”,“b”:“55FBFAC72000”,“o”:“13CFF0E”,“s”:“main”,“s+”:“E”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“7F77FB741555”,“b”:“7F77FB71F000”,“o”:“22555”,“s”:“__libc_start_main”,“s+”:“F5”}}}
{“t”:{“$date”:“2022-09-26T08:37:05.806+03:00”},“s”:“I”, “c”:“CONTROL”, “id”:31445, “ctx”:“initandlisten”,“msg”:“Frame”,“attr”:{“frame”:{“a”:“55FBFC19EB3E”,“b”:“55FBFAC72000”,“o”:“152CB3E”,“s”:“_start”,“s+”:“29”}}}

For a corrupted replica restore is the only option
Please take advice from Stennie and other experts before you do anything
Also check this link

Hi @Kadir_USTUN

I agree with @Ramachandra_Tummala 's assessment that restoring from backup is probably the best way forward.

However I’m curious about one thing. Here’s the error message from node 1:

{“t”:{"$date":“2022-09-26T08:36:02.439+03:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:{“error”:-31809,“message”:"[1664170562:439217][94507:0x7f741c10abc0], connection: __wt_turtle_read, 391: WiredTiger.turtle: fatal turtle file read error: WT_TRY_SALVAGE: database corruption detected"}}
{“t”:{"$date":“2022-09-26T08:36:02.439+03:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:{“error”:-31804,“message”:"[1664170562:439270][94507:0x7f741c10abc0], connection: __wt_turtle_read, 391: the process must exit and restart: WT_PANIC: WiredTiger library panic"}}

and here’s the error message from node 2:

{“t”:{"$date":“2022-09-26T08:37:05.674+03:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:{“error”:-31809,“message”:"[1664170625:674753][122834:0x7f77fd5a8bc0], connection: __wt_turtle_read, 391: WiredTiger.turtle: fatal turtle file read error: WT_TRY_SALVAGE: database corruption detected"}}
{“t”:{"$date":“2022-09-26T08:37:05.674+03:00”},“s”:“E”, “c”:“STORAGE”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error”,“attr”:{“error”:-31804,“message”:"[1664170625:674804][122834:0x7f77fd5a8bc0], connection: __wt_turtle_read, 391: the process must exit and restart: WT_PANIC: WiredTiger library panic"}}

It strikes me as odd that both of them seem to have an identical error:

WiredTiger.turtle: fatal turtle file read error: WT_TRY_SALVAGE: database corruption detected

I understand that this is a PSA setup, but I noticed that the two secondaries have the exact same error. Note that the file WiredTiger.turtle is a vital file, so WiredTiger is very, very careful in handling this file in particular.

To have the same error of this magnitude at the same time on two different nodes is so highly unlikely that there may be other reason behind this. How are you deploying the mongod processes? Are they sharing disk, CPU, or something? What’s the spec of the deployment hardware/architecture?

Best regards
Kevin

3 Likes

Hi Kevin,

We have the same thoughts. 2 mongod process working with 2 seperate rugged pc. And 2 rugged pc has own local disc. I suspect discs. Discs, CPU or everything are seperate. The corruption on the 1st server somehow breaks the 2nd server as well.

This is how we are going now.
Corruption happens to often :frowning:
I tried --repair option before open a ticket here, but i didn’t found any solution.

Out disc is ext4 and mongodb is strongly recommended to use xfs file system. You think could this be the problem?

I’m working a lot of project with mongo. We are using virtual machine for mongo and the disc coming from storage. I didn’t have any problems with those projects.

Yes agree but what is the problem? I need to find problem. Maybe hardware problem. If I can prove it’s a hardware problem, I can request a hardware replacement.

Do you have any idea ?
Thanks.

Yes this is a strange issue. MongoDB replication works logically instead of physically, and each node manage their own storage, so any physical level corruption won’t be replicated to the other nodes. For this to happen to two separate nodes at the same time on the same error is highly unlikely, and I would perhaps consider a hardware issue is at play here.

As far as I am aware, Ext4 has issues with performance in the early days, but not corruption. However at this point I think there’s no harm in trying XFS since you’re restoring from backup anyway :slight_smile:

In the meantime, you might want to double check that your deployment follows the recommendations in the production notes and operations checklist.

Best regards
Kevin

1 Like

Hi Kevin,

Actually i read this articles :slight_smile:
Btw I was using mongodb 4.4.6 version and i had same problem. But in the mongodb releases notes for 4.4.6 , theew was an error like below

WARNING

MongoDB version 4.4.6 is not recommended for production use due to critical issue WT-7995, fixed in later versions. Use the latest available patch release version.

So i upgraded my mongodb version to 5.0.6 and i thought i resolve the problem.

https://jira.mongodb.org/browse/WT-7995

Thanks.

@kevinadi do you have any idea?

Hi @Kadir_USTUN

Yes MongoDB 4.4.2 - 4.4.8 was not recommended for production usage anymore due to the issues you mentioned. However I don’t think this is the cause for a WiredTiger.turtle error you’re seeing. The turtle file is a metadata for the main WiredTiger metadata file (see WiredTiger: Metadata for a complete explanation of what the turtle file contains).

In short, it’s very peculiar that two nodes have the exact same complaint about turtle file corruption. Typically these errors are generated if the hardware is having issues, but there’s not much more knowledge we can get from the logs themselves.

Best regards
Kevin

2 Likes

Hi,

I tried this situation on another workstation and some vm’s. (Hp Z4). Lıke you said we didn’t see any error when electric is gone. After restart the machines, mongodb worked automaticly and without error.

But when we try to electric our rugged pc, 2 mongo instance couldn’t open.
So we decided that the problem is with our rugged pc. We will try to change our hard disk.

Thank you.

2 Likes