MongoDB secondary nodes down with WiredTiger error

Our secondary MongoDB nodes went down with wiredtiger error. we try to restart the servers and it was not started. I don’t know what is it and how to fix it. This project is very urgent, so I hope someone can help me fix this problem. please find errors below.

2022-08-02T11:37:48.690+0300 E STORAGE  [repl writer worker 11] WiredTiger error (0) [1659429468:685187][2567:0x7f20df43c700], file:index-25--485386741746989461.wt, WT_CURSOR.insert: read checksum error for 12288B block at offset 18314407936: block header checksum of 604962355 doesn't match expected checksum of 3306781470
2022-08-02T11:37:48.699+0300 E STORAGE  [repl writer worker 11] WiredTiger error (0) [1659429468:690748][2567:0x7f20df43c700], file:index-25--485386741746989461.wt, WT_CURSOR.insert: index-25--485386741746989461.wt: encountered an illegal file format or internal value
2022-08-02T11:37:48.699+0300 E STORAGE  [repl writer worker 11] WiredTiger error (-31804) [1659429468:699213][2567:0x7f20df43c700], file:index-25--485386741746989461.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic
2022-08-02T11:37:48.699+0300 I -        [repl writer worker 11] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2022-08-02T11:37:48.699+0300 I -        [repl writer worker 11]

***aborting after fassert() failure


2022-08-02T11:37:48.702+0300 I -        [repl writer worker 3] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.702+0300 I -        [repl writer worker 4] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.702+0300 I -        [repl writer worker 3]

***aborting after fassert() failure


2022-08-02T11:37:48.702+0300 I -        [repl writer worker 4]

***aborting after fassert() failure


2022-08-02T11:37:48.702+0300 I -        [repl writer worker 5] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.703+0300 I -        [repl writer worker 5]

***aborting after fassert() failure


2022-08-02T11:37:48.703+0300 I -        [repl writer worker 2] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.703+0300 I -        [repl writer worker 9] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.704+0300 I -        [repl writer worker 2]

***aborting after fassert() failure


2022-08-02T11:37:48.704+0300 I -        [repl writer worker 7] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.704+0300 I -        [repl writer worker 1] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.704+0300 I -        [repl writer worker 9]

***aborting after fassert() failure


2022-08-02T11:37:48.704+0300 I -        [repl writer worker 7]

***aborting after fassert() failure


2022-08-02T11:37:48.704+0300 I -        [repl writer worker 12] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.704+0300 I -        [repl writer worker 1]

***aborting after fassert() failure


2022-08-02T11:37:48.704+0300 I -        [repl writer worker 14] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.704+0300 I -        [repl writer worker 8] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.704+0300 I -        [repl writer worker 12]

***aborting after fassert() failure


2022-08-02T11:37:48.704+0300 I -        [repl writer worker 8]

***aborting after fassert() failure


2022-08-02T11:37:48.704+0300 I -        [repl writer worker 14]

***aborting after fassert() failure


2022-08-02T11:37:48.704+0300 I -        [repl writer worker 6] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.704+0300 I -        [repl writer worker 6]

***aborting after fassert() failure


2022-08-02T11:37:48.704+0300 E STORAGE  [thread2] WiredTiger error (-31804) [1659429468:704562][2567:0x7f20fac73700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic
2022-08-02T11:37:48.704+0300 I -        [thread2] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2022-08-02T11:37:48.704+0300 I -        [thread2]

***aborting after fassert() failure


2022-08-02T11:37:48.707+0300 I -        [repl writer worker 15] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.707+0300 I -        [repl writer worker 13] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.707+0300 I -        [repl writer worker 10] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:48.707+0300 I -        [repl writer worker 15]

***aborting after fassert() failure


2022-08-02T11:37:48.707+0300 I -        [repl writer worker 13]

***aborting after fassert() failure


2022-08-02T11:37:48.707+0300 I -        [repl writer worker 10]

***aborting after fassert() failure


2022-08-02T11:37:48.708+0300 E STORAGE  [thread3] WiredTiger error (-31804) [1659429468:708427][2567:0x7f20fa472700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic
2022-08-02T11:37:48.708+0300 I -        [thread3] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2022-08-02T11:37:48.708+0300 I -        [thread3]

***aborting after fassert() failure


2022-08-02T11:37:48.708+0300 E STORAGE  [thread4] WiredTiger error (-31804) [1659429468:708474][2567:0x7f20f9470700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic
2022-08-02T11:37:48.708+0300 I -        [thread4] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2022-08-02T11:37:48.708+0300 I -        [thread4]

***aborting after fassert() failure


2022-08-02T11:37:49.001+0300 I -        [ftdc] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:49.001+0300 I -        [ftdc]

***aborting after fassert() failure


2022-08-02T11:37:49.348+0300 E STORAGE  [thread5] WiredTiger error (-31804) [1659429469:348068][2567:0x7f20fb474700], log-server: log server error: WT_PANIC: WiredTiger library panic
2022-08-02T11:37:49.348+0300 I -        [thread5] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2022-08-02T11:37:49.348+0300 I -        [thread5]

***aborting after fassert() failure


2022-08-02T11:37:52.761+0300 I -        [repl writer worker 0] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 64
2022-08-02T11:37:52.761+0300 I -        [repl writer worker 0]

***aborting after fassert() failure


2022-08-02T11:38:04.048+0300 F -        [repl writer worker 11] Got signal: 6 (Aborted).

 0x7f21019bf6b1 0x7f21019be8c9 0x7f21019bedad 0x7f20ff09e100 0x7f20fed035f7 0x7f20fed04ce8 0x7f2100c63267 0x7f21016cfcb6 0x7f2100c6d576 0x7f2100c6d792 0x7f2100c6d9f4 0x7f21022c9e35 0x7f21022e3bbb 0x7f21022ead93 0x7f210230bcb0 0x7f21022d9b32 0x7f2102328d54 0x7f21016a3a69 0x7f21016a5914 0x7f21010b2a03 0x7f2100e5896a 0x7f2100e58c0d 0x7f2100e58ce3 0x7f2100e30a51 0x7f2100e30cb6 0x7f21013ccd51 0x7f21014ad9a4 0x7f21014a3e92 0x7f21014a6b5e 0x7f21014a75bd 0x7f21014a76cd 0x7f21014ad06e 0x7f21014ad663 0x7f21014a2c7f 0x7f21014a3d6a 0x7f2101938f1c 0x7f21019399cc 0x7f210193a3b6 0x7f2102436d40 0x7f20ff096dc5 0x7f20fedc41cd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"7F2100447000","o":"15786B1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7F2100447000","o":"15778C9"},{"b":"7F2100447000","o":"1577DAD"},{"b":"7F20FF08F000","o":"F100"},{"b":"7F20FECCE000","o":"355F7","s":"gsignal"},{"b":"7F20FECCE000","o":"36CE8","s":"abort"},{"b":"7F2100447000","o":"81C267","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"7F2100447000","o":"1288CB6"},{"b":"7F2100447000","o":"826576","s":"__wt_eventv"},{"b":"7F2100447000","o":"826792","s":"__wt_err"},{"b":"7F2100447000","o":"8269F4","s":"__wt_panic"},{"b":"7F2100447000","o":"1E82E35","s":"__wt_bm_read"},{"b":"7F2100447000","o":"1E9CBBB","s":"__wt_bt_read"},{"b":"7F2100447000","o":"1EA3D93","s":"__wt_page_in_func"},{"b":"7F2100447000","o":"1EC4CB0","s":"__wt_row_search"},{"b":"7F2100447000","o":"1E92B32","s":"__wt_btcur_insert"},{"b":"7F2100447000","o":"1EE1D54"},{"b":"7F2100447000","o":"125CA69","s":"_ZN5mongo23WiredTigerIndexStandard7_insertEP11__wt_cursorRKNS_7BSONObjERKNS_8RecordIdEb"},{"b":"7F2100447000","o":"125E914","s":"_ZN5mongo15WiredTigerIndex6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb"},{"b":"7F2100447000","o":"C6BA03","s":"_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl"},{"b":"7F2100447000","o":"A1196A","s":"_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EEPl"},{"b":"7F2100447000","o":"A11C0D","s":"_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EEPl"},{"b":"7F2100447000","o":"A11CE3","s":"_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EEPl"},{"b":"7F2100447000","o":"9E9A51","s":"_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bPNS_7OpDebugE"},{"b":"7F2100447000","o":"9E9CB6","s":"_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_PNS_7OpDebugEbb"},{"b":"7F2100447000","o":"F85D51","s":"_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbSt8functionIFvvEE"},{"b":"7F2100447000","o":"10669A4","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbSt8functionIFvvEEEPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_S8_ObOSB_"},{"b":"7F2100447000","o":"105CE92"},{"b":"7F2100447000","o":"105FB5E","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS7_IFvvEEEES7_IFS8_S3_S6_bEESC_"},{"b":"7F2100447000","o":"10605BD","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb"},{"b":"7F2100447000","o":"10606CD"},{"b":"7F2100447000","o":"106606E","s":"_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjEbEE"},{"b":"7F2100447000","o":"1066663","s":"_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE"},{"b":"7F2100447000","o":"105BC7F"},{"b":"7F2100447000","o":"105CD6A"},{"b":"7F2100447000","o":"14F1F1C","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"7F2100447000","o":"14F29CC","s":"_ZN5mongo10ThreadPool13_consumeTasksEv"},{"b":"7F2100447000","o":"14F33B6","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE"},{"b":"7F2100447000","o":"1FEFD40"},{"b":"7F20FF08F000","o":"7DC5"},{"b":"7F20FECCE000","o":"F61CD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.10", "gitVersion" : "078f28920cb24de0dd479b5ea6c66c644f6326e9", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-327.el7.x86_64", "version" : "#1 SMP Thu Oct 29 17:29:29 EDT 2015", "machine" : "x86_64" }, "somap" : [ { "b" : "7F2100447000", "elfType" : 3, "buildId" : "94C7FAB092E567C9338D13DB9B68751363D15EFD" }, { "b" : "7FFF283D9000", "elfType" : 3, "buildId" : "17A121B1F7BBB010F54735FFDE3347B27B33884D" }, { "b" : "7F20FFFB6000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "8DB4545998776514159031B754BB67F7F396F83A" }, { "b" : "7F20FFBCF000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "038F79F7C3F6E60C29184B8E70D0B1E62525D64D" }, { "b" : "7F20FF9C7000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "A1D9E0B471D827008C36FA72BAB34BE08FE54B33" }, { "b" : "7F20FF7C3000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "5958E57738366BCC217783F3CD4C836437F7C45F" }, { "b" : "7F20FF4C1000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "02C4E38A8145201D9C574499CF75132551835CEB" }, { "b" : "7F20FF2AB000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "97D5E2F5739B715C3A0EC9F95F7336E232346CA8" }, { "b" : "7F20FF08F000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "FA15B7D2CA650B34E6A0C9AD999BA6625AEC4068" }, { "b" : "7F20FECCE000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "B0A1DFA62C6AF7AA62487E3C260DC4B9C24D8BF8" }, { "b" : "7F2100223000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "CEB78DAE1EE5B4C544047DC26F88A8E4586A34D2" }, { "b" : "7F20FEA82000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "8AB5682155DE13D0916B984306B4E044E216B2EB" }, { "b" : "7F20FE79D000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "BE8968836D439581B2816CE3827642FCF4B8BF4A" }, { "b" : "7F20FE599000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "B25574847B066A26CD593C8101DF6779898FF2C2" }, { "b" : "7F20FE367000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "F5784ED7E64118BAFE898DBF178DC9E37CBDA4AA" }, { "b" : "7F20FE151000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "FC37913FB197B822BCDBF3697D061E248698CEC1" }, { "b" : "7F20FDF42000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4BBED12CFDC9647C8771A4B897E0D5A4F217ED7C" }, { "b" : "7F20FDD3E000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8CA73C16CFEB9A8B5660015B9223B09F87041CAD" }, { "b" : "7F20FDB24000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "D08CF135D143704DA93E5F025AE6AE6943838F03" }, { "b" : "7F20FD8FF000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "5062031216B995004A297D555D834C0109F7598C" }, { "b" : "7F20FD69E000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "8E3819A80BF876382A6F0CB2A08F82F1742EE8DB" }, { "b" : "7F20FD479000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "61D7D46225E85F144221E1424B87FBF3CB2B9D3F" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x7f21019bf6b1]
 mongod(+0x15778C9) [0x7f21019be8c9]
 mongod(+0x1577DAD) [0x7f21019bedad]
 libpthread.so.0(+0xF100) [0x7f20ff09e100]
 libc.so.6(gsignal+0x37) [0x7f20fed035f7]
 libc.so.6(abort+0x148) [0x7f20fed04ce8]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x7f2100c63267]
 mongod(+0x1288CB6) [0x7f21016cfcb6]
 mongod(__wt_eventv+0x3D7) [0x7f2100c6d576]
 mongod(__wt_err+0x9D) [0x7f2100c6d792]
 mongod(__wt_panic+0x2E) [0x7f2100c6d9f4]
 mongod(__wt_bm_read+0x135) [0x7f21022c9e35]
 mongod(__wt_bt_read+0x1FB) [0x7f21022e3bbb]
 mongod(__wt_page_in_func+0x1303) [0x7f21022ead93]
 mongod(__wt_row_search+0x660) [0x7f210230bcb0]
 mongod(__wt_btcur_insert+0xB62) [0x7f21022d9b32]
 mongod(+0x1EE1D54) [0x7f2102328d54]
 mongod(_ZN5mongo23WiredTigerIndexStandard7_insertEP11__wt_cursorRKNS_7BSONObjERKNS_8RecordIdEb+0x149) [0x7f21016a3a69]
 mongod(_ZN5mongo15WiredTigerIndex6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb+0xF4) [0x7f21016a5914]
 mongod(_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl+0x1F3) [0x7f21010b2a03]
 mongod(_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EEPl+0xDA) [0x7f2100e5896a]
 mongod(_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EEPl+0x1FD) [0x7f2100e58c0d]
 mongod(_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EEPl+0x73) [0x7f2100e58ce3]
 mongod(_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bPNS_7OpDebugE+0x4F1) [0x7f2100e30a51]
 mongod(_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_PNS_7OpDebugEbb+0x176) [0x7f2100e30cb6]
 mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEbSt8functionIFvvEE+0x17F1) [0x7f21013ccd51]
 mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbSt8functionIFvvEEEPSC_E9_M_invokeERKSt9_Any_dataOS3_OS5_S8_ObOSB_+0x74) [0x7f21014ad9a4]
 mongod(+0x105CE92) [0x7f21014a3e92]
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS7_IFvvEEEES7_IFS8_S3_S6_bEESC_+0x54E) [0x7f21014a6b5e]
 mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xFD) [0x7f21014a75bd]
 mongod(+0x10606CD) [0x7f21014a76cd]
 mongod(_ZN5mongo4repl22multiSyncApply_noAbortEPNS_16OperationContextEPSt6vectorIPKNS0_10OplogEntryESaIS6_EESt8functionIFNS_6StatusES2_RKNS_7BSONObjEbEE+0xFEE) [0x7f21014ad06e]
 mongod(_ZN5mongo4repl14multiSyncApplyEPSt6vectorIPKNS0_10OplogEntryESaIS4_EEPNS0_8SyncTailE+0x73) [0x7f21014ad663]
 mongod(+0x105BC7F) [0x7f21014a2c7f]
 mongod(+0x105CD6A) [0x7f21014a3d6a]
 mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x14C) [0x7f2101938f1c]
 mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xBC) [0x7f21019399cc]
 mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x96) [0x7f210193a3b6]
 mongod(+0x1FEFD40) [0x7f2102436d40]
 libpthread.so.0(+0x7DC5) [0x7f20ff096dc5]
 libc.so.6(clone+0x6D) [0x7f20fedc41cd]

Hi @veerendra_pulapa,

Please provide some more details on your environment:

  • Specific version of MongoDB server
  • Replica set configuration
  • Health/status of replica set members

Fatal assertions indicate data issues that cannot be automatically recovered. In these situations the recommended approach is normally to Resync from a healthy member of your replica set.

Regards,
Stennie

Hi @Stennie,

  • Specific version of MongoDB server

MongoDB Version 3.4

  • Replica set configuration

cat /etc/mongod.conf | grep -v ^#
systemLog:
destination: file
logAppend: true
path: /var/log/mongodb/mongod.log
storage:
dbPath: /data/mongo
journal:
enabled: true
processManagement:
fork: true # fork and run in background
pidFilePath: /var/run/mongodb/mongod.pid # location of pidfile
net:
port: 27017
bindIp: 127.0.0.1
security:
authorization: “enabled”
keyFile: /etc/mongodb_repl_prd.key
replication:
replSetName: “prd”

  • Health/status of replica set members

for secondary servers it is showing unhealthy.

Hi Team,

what could be the main reason, does secondary mongodb nodes went down?