Mongod Service Crash, Ubuntu Linux

Russell_Morris · September 1, 2025, 9:31pm

Hi,

Having a problem getting the service (systemctl) to start on Linux … it’s even failing if I try to start it manually, with the command,

sudo /usr/bin/mongod --config /etc/mongod.conf

Checking /var/crash/_usr_bin_mongod.0.crash, I see some info like below, seems to be a pretty low level crash. Any thoughts? FYI, I’m running version 8.0.13.

ProblemType: Crash
ApportVersion: 2.28.1-0ubuntu3.8
Architecture: amd64
CasperMD5CheckResult: unknown
CrashCounter: 1
...
Disassembly:
 => 0x7c18dda9eb2c <__GI___pthread_kill+284>:   mov    %eax,%r14d
    0x7c18dda9eb2f <__GI___pthread_kill+287>:   neg    %r14d
    0x7c18dda9eb32 <__GI___pthread_kill+290>:   cmp    $0xfffff000,%eax
    0x7c18dda9eb37 <__GI___pthread_kill+295>:   mov    $0x0,%eax
    0x7c18dda9eb3c <__GI___pthread_kill+300>:   cmovbe %eax,%r14d
    0x7c18dda9eb40 <__GI___pthread_kill+304>:   jmp    0x7c18dda9eac0 <__GI___pthread_kill+176>
    0x7c18dda9eb45 <__GI___pthread_kill+309>:   nopl   (%rax)
    0x7c18dda9eb48 <__GI___pthread_kill+312>:   mov    %r13,%rdi
    0x7c18dda9eb4b <__GI___pthread_kill+315>:   call   0x7c18dda98ee0 <__GI___lll_lock_wait_private>
    0x7c18dda9eb50 <__GI___pthread_kill+320>:   jmp    0x7c18dda9ea8e <__GI___pthread_kill+126>
    0x7c18dda9eb55 <__GI___pthread_kill+325>:   nopl   (%rax)
    0x7c18dda9eb58 <__GI___pthread_kill+328>:   mov    %r13,%rdi
    0x7c18dda9eb5b <__GI___pthread_kill+331>:   call   0x7c18dda98fa0 <__GI___lll_lock_wake_private>
    0x7c18dda9eb60 <__GI___pthread_kill+336>:   jmp    0x7c18dda9eaa9 <__GI___pthread_kill+153>
    0x7c18dda9eb65 <__GI___pthread_kill+341>:   call   0x7c18ddb37e90 <__stack_chk_fail>
    0x7c18dda9eb6a:     nopw   0x0(%rax,%rax,1)
DistroRelease: Ubuntu 24.04
ExecutablePath: /usr/bin/mongod
ExecutableTimestamp: 1387431706

Any thoughts or debugging suggestions much appreciated. Thanks!

Mattew_wade · September 2, 2025, 11:20am

It looks like the crash might be related to memory or threading issues. I’d check the MongoDB logs (/var/log/mongodb/mongod.log), ensure your system has enough resources, and make sure both Ubuntu and MongoDB are fully updated.

Russell_Morris · September 2, 2025, 12:55pm

Thanks! Both seem to be fully updated, but of course I could be wrong ;). I do see in the log file you noted, a msg “WiredTiger error message”. I admit, not sure what that means - any thoughts?

Thanks again.

dogs_Cute · September 2, 2025, 1:45pm

Can you share the contents of the log?

Russell_Morris · September 2, 2025, 2:44pm

Sure, attached - hope this helps. Thanks!

PS: Tried to paste inline, but too long => attached text file.

PSS: That failed also … “new user” can’t upload. So “limited” log below (stripped out a bunch of the chunks).

{"t":{"$date":"2025-09-02T09:31:11.472-05:00"},"s":"I",  "c":"CONTROL",  "id":20698,   "ctx":"main","msg":"***** SERVER RESTARTED *****"}
{"t":{"$date":"2025-09-02T09:31:11.472-05:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2025-09-02T09:31:11.474-05:00"},"s":"I",  "c":"CONTROL",  "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
{"t":{"$date":"2025-09-02T09:31:11.474-05:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set at least one of the related parameters","attr":{"relatedParameters":["tcpFastOpenServer","tcpFastOpenClient","tcpFastOpenQueueSize"]}}
{"t":{"$date":"2025-09-02T09:31:11.480-05:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":25},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":25},"outgoing":{"minWireVersion":6,"maxWireVersion":25},"isInternalClient":true}}}
{"t":{"$date":"2025-09-02T09:31:11.480-05:00"},"s":"I",  "c":"TENANT_M", "id":7091600, "ctx":"main","msg":"Starting TenantMigrationAccessBlockerRegistry"}
{"t":{"$date":"2025-09-02T09:31:11.480-05:00"},"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":3259868,"port":27017,"dbPath":"/var/lib/mongodb","architecture":"64-bit","host":"linuxServer.rmorris.home"}}
{"t":{"$date":"2025-09-02T09:31:11.480-05:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"8.0.13","gitVersion":"8dc5cd2a30c4524132e2d44bb314544dc477e611","openSSLVersion":"OpenSSL 3.0.13 30 Jan 2024","modules":[],"allocator":"tcmalloc-google","environment":{"distmod":"ubuntu2404","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2025-09-02T09:31:11.480-05:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"24.04"}}}
{"t":{"$date":"2025-09-02T09:31:11.480-05:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"127.0.0.1","port":27017},"processManagement":{"timeZoneInfo":"/usr/share/zoneinfo"},"storage":{"dbPath":"/var/lib/mongodb"},"systemLog":{"destination":"file","logAppend":true,"path":"/var/log/mongodb/mongod.log"}}}}
{"t":{"$date":"2025-09-02T09:31:11.481-05:00"},"s":"W",  "c":"STORAGE",  "id":22271,   "ctx":"initandlisten","msg":"Detected unclean shutdown - Lock file is not empty","attr":{"lockFile":"/var/lib/mongodb/mongod.lock"}}
{"t":{"$date":"2025-09-02T09:31:11.481-05:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/var/lib/mongodb","storageEngine":"wiredTiger"}}
{"t":{"$date":"2025-09-02T09:31:11.481-05:00"},"s":"W",  "c":"STORAGE",  "id":22302,   "ctx":"initandlisten","msg":"Recovering data from the last clean checkpoint."}
{"t":{"$date":"2025-09-02T09:31:11.481-05:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
{"t":{"$date":"2025-09-02T09:31:11.481-05:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=15495M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],prefetch=(available=true,default=false),"}}
{"t":{"$date":"2025-09-02T09:31:11.593-05:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":0,"message":{"ts_sec":1756823471,"ts_usec":593039,"thread":"3259868:0x7d89cce74240","session_dhandle_name":"file:WiredTiger.wt","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__wt_block_read_off:279:WiredTiger.wt: potential hardware corruption, read checksum error for 4096B block at offset 102400: block header checksum of 0x6e726574 doesn't match expected checksum of 0x406110a5"}}}
{"t":{"$date":"2025-09-02T09:31:11.593-05:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":0,"message":{"ts_sec":1756823471,"ts_usec":593123,"thread":"3259868:0x7d89cce74240","session_dhandle_name":"file:WiredTiger.wt","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__bm_corrupt_dump:86:{0: 102400, 4096, 0x406110a5}: (chunk 1 of 4): 69 7a ... 35 36 "}}}
{"t":{"$date":"2025-09-02T09:31:11.593-05:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":0,"message":{"ts_sec":1756823471,"ts_usec":593215,"thread":"3259868:0x7d89cce74240","session_dhandle_name":"file:WiredTiger.wt","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__bm_corrupt_dump:86:{0: 102400, 4096, 0x406110a5}: (chunk 2 of 4): 32 39 ... 73 61 "}}}
{"t":{"$date":"2025-09-02T09:31:11.593-05:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":0,"message":{"ts_sec":1756823471,"ts_usec":593293,"thread":"3259868:0x7d89cce74240","session_dhandle_name":"file:WiredTiger.wt","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__bm_corrupt_dump:86:{0: 102400, 4096, 0x406110a5}: (chunk 3 of 4): 67 65 ... 67 65 "}}}
{"t":{"$date":"2025-09-02T09:31:11.593-05:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":0,"message":{"ts_sec":1756823471,"ts_usec":593369,"thread":"3259868:0x7d89cce74240","session_dhandle_name":"file:WiredTiger.wt","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__bm_corrupt_dump:86:{0: 102400, 4096, 0x406110a5}: (chunk 4 of 4): 5f 6d ... 00 00 "}}}
{"t":{"$date":"2025-09-02T09:31:11.593-05:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":-31802,"message":{"ts_sec":1756823471,"ts_usec":593414,"thread":"3259868:0x7d89cce74240","session_dhandle_name":"file:WiredTiger.wt","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__wt_block_read_off:291:WiredTiger.wt: fatal read error","error_str":"WT_ERROR: non-specific WiredTiger error","error_code":-31802}}}
{"t":{"$date":"2025-09-02T09:31:11.593-05:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":-31804,"message":{"ts_sec":1756823471,"ts_usec":593428,"thread":"3259868:0x7d89cce74240","session_dhandle_name":"file:WiredTiger.wt","session_name":"connection","category":"WT_VERB_DEFAULT","category_id":12,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__wt_block_read_off:291:the process must exit and restart","error_str":"WT_PANIC: WiredTiger library panic","error_code":-31804}}}
{"t":{"$date":"2025-09-02T09:31:11.593-05:00"},"s":"F",  "c":"ASSERT",   "id":23089,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":791}}
{"t":{"$date":"2025-09-02T09:31:11.593-05:00"},"s":"F",  "c":"ASSERT",   "id":23090,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
{"t":{"$date":"2025-09-02T09:31:11.593-05:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"initandlisten","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"63E9254AAD08","b":"63E91C532000","o":"8F78D08","s":"_ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"38"},{"a":"63E92548815D","b":"63E91C532000","o":"8F5615D","s":"_ZN5mongo12_GLOBAL__N_115printErrorBlockEv","C":"mongo::(anonymous namespace)::printErrorBlock()","s+":"1BD"},{"a":"63E92548824D","b":"63E91C532000","o":"8F5624D","s":"abruptQuit","s+":"5D"},{"a":"7D89CD845330","b":"7D89CD800000","o":"45330"},{"a":"7D89CD89EB2C","b":"7D89CD800000","o":"9EB2C","s":"__GI___pthread_kill","s+":"11C"},{"a":"7D89CD84527E","b":"7D89CD800000","o":"4527E","s":"__GI_raise","s+":"1E"},{"a":"7D89CD8288FF","b":"7D89CD800000","o":"288FF","s":"__GI_abort","s+":"DF"},{"a":"63E925478270","b":"63E91C532000","o":"8F46270","s":"_ZN5mongo12_GLOBAL__N_19callAbortEv","C":"mongo::(anonymous namespace)::callAbort()","s+":"3C"},{"a":"63E92547A324","b":"63E91C532000","o":"8F48324","s":"_ZN5mongo14fassert_detail6failedENS_14SourceLocationENS0_5MsgIdE","C":"mongo::fassert_detail::failed(mongo::SourceLocation, mongo::fassert_detail::MsgId)","s+":"165"},{"a":"63E921B00DF3","b":"63E91C532000","o":"55CEDF3","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold","C":"mongo::(anonymous namespace)::mdb_handle_error_with_startup_suppression(__wt_event_handler*, __wt_session*, int, char const*) [clone .cold]","s+":"16"},{"a":"63E921CE2C67","b":"63E91C532000","o":"57B0C67","s":"__eventv","s+":"10D7"},{"a":"63E921CE33EC","b":"63E91C532000","o":"57B13EC","s":"__wt_panic_func","s+":"154"},{"a":"63E921B1F245","b":"63E91C532000","o":"55ED245","s":"__wt_block_read_off.cold","s+":"2D"},{"a":"63E921B1C09B","b":"63E91C532000","o":"55EA09B","s":"__wti_block_extlist_read","s+":"9B"},{"a":"63E921B1C5EC","b":"63E91C532000","o":"55EA5EC","s":"__wti_block_extlist_read_avail","s+":"2C"},{"a":"63E921B14458","b":"63E91C532000","o":"55E2458","s":"__wt_block_checkpoint_load","s+":"278"},{"a":"63E921B2C1A6","b":"63E91C532000","o":"55FA1A6","s":"__bm_checkpoint_load","s+":"36"},{"a":"63E921B50AB1","b":"63E91C532000","o":"561EAB1","s":"__wt_btree_open","s+":"E71"},{"a":"63E921BB7775","b":"63E91C532000","o":"5685775","s":"__wt_conn_dhandle_open","s+":"435"},{"a":"63E921CD1FF0","b":"63E91C532000","o":"579FFF0","s":"__wt_session_get_dhandle","s+":"D0"},{"a":"63E921CD23D2","b":"63E91C532000","o":"57A03D2","s":"__wt_session_get_dhandle","s+":"4B2"},{"a":"63E921CD2D83","b":"63E91C532000","o":"57A0D83","s":"__wt_session_get_btree_ckpt","s+":"873"},{"a":"63E921BE2A74","b":"63E91C532000","o":"56B0A74","s":"__wt_curfile_open","s+":"114"},{"a":"63E921CCB300","b":"63E91C532000","o":"5799300","s":"__session_open_cursor_int","s+":"5C0"},{"a":"63E921CCAC78","b":"63E91C532000","o":"5798C78","s":"__wt_open_cursor","s+":"58"},{"a":"63E921C6D39C","b":"63E91C532000","o":"573B39C","s":"__wt_metadata_cursor_open","s+":"6C"},{"a":"63E921C6D46B","b":"63E91C532000","o":"573B46B","s":"__wt_metadata_cursor","s+":"4B"},{"a":"63E921BAC918","b":"63E91C532000","o":"567A918","s":"wiredtiger_open","s+":"2718"},{"a":"63E921AB8951","b":"63E91C532000","o":"5586951","s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_","C":"mongo::WiredTigerKVEngine::_openWiredTiger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)","s+":"A1"},{"a":"63E921AC4623","b":"63E91C532000","o":"5592623","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbb","C":"mongo::WiredTigerKVEngine::WiredTigerKVEngine(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::ClockSource*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, bool, bool)","s+":"BE3"},{"a":"63E921A85CB2","b":"63E91C532000","o":"5553CB2","s":"_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createEPNS_16OperationContextERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE","C":"mongo::(anonymous namespace)::WiredTigerFactory::create(mongo::OperationContext*, mongo::StorageGlobalParams const&, mongo::StorageEngineLockFile const*) const","s+":"152"},{"a":"63E921D29B41","b":"63E91C532000","o":"57F7B41","s":"_ZN5mongo23initializeStorageEngineEPNS_16OperationContextENS_22StorageEngineInitFlagsEPNS_14BSONObjBuilderE","C":"mongo::initializeStorageEngine(mongo::OperationContext*, mongo::StorageEngineInitFlags, mongo::BSONObjBuilder*)","s+":"441"},{"a":"63E920C26591","b":"63E91C532000","o":"46F4591","s":"_ZN5mongo11mongod_mainEiPPc","C":"mongo::mongod_main(int, char**)","s+":"B91"},{"a":"63E920C09A89","b":"63E91C532000","o":"46D7A89","s":"main","s+":"9"},{"a":"7D89CD82A1CA","b":"7D89CD800000","o":"2A1CA","s":"__libc_start_call_main","s+":"7A"},{"a":"7D89CD82A28B","b":"7D89CD800000","o":"2A28B","s":"__libc_start_main_alias_2","s+":"8B"},{"a":"63E920C09965","b":"63E91C532000","o":"46D7965","s":"_start","s+":"25"}],"processInfo":{"mongodbVersion":"8.0.13","gitVersion":"8dc5cd2a30c4524132e2d44bb314544dc477e611","compiledModules":[],"uname":{"sysname":"Linux","release":"6.8.0-71-generic","version":"#71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025","machine":"x86_64"},"somap":[{"b":"63E91C532000","elfType":3,"buildId":"58D5E8D4FB929389"},{"b":"7D89CD800000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"282C2C16E7B6600B0B22EA0C99010D2795752B5F"}]}}},"tags":[]}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E9254AAD08","b":"63E91C532000","o":"8F78D08","s":"_ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"38"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E92548815D","b":"63E91C532000","o":"8F5615D","s":"_ZN5mongo12_GLOBAL__N_115printErrorBlockEv","C":"mongo::(anonymous namespace)::printErrorBlock()","s+":"1BD"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E92548824D","b":"63E91C532000","o":"8F5624D","s":"abruptQuit","s+":"5D"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"7D89CD845330","b":"7D89CD800000","o":"45330"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"7D89CD89EB2C","b":"7D89CD800000","o":"9EB2C","s":"__GI___pthread_kill","s+":"11C"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"7D89CD84527E","b":"7D89CD800000","o":"4527E","s":"__GI_raise","s+":"1E"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"7D89CD8288FF","b":"7D89CD800000","o":"288FF","s":"__GI_abort","s+":"DF"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E925478270","b":"63E91C532000","o":"8F46270","s":"_ZN5mongo12_GLOBAL__N_19callAbortEv","C":"mongo::(anonymous namespace)::callAbort()","s+":"3C"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E92547A324","b":"63E91C532000","o":"8F48324","s":"_ZN5mongo14fassert_detail6failedENS_14SourceLocationENS0_5MsgIdE","C":"mongo::fassert_detail::failed(mongo::SourceLocation, mongo::fassert_detail::MsgId)","s+":"165"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921B00DF3","b":"63E91C532000","o":"55CEDF3","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc.cold","C":"mongo::(anonymous namespace)::mdb_handle_error_with_startup_suppression(__wt_event_handler*, __wt_session*, int, char const*) [clone .cold]","s+":"16"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921CE2C67","b":"63E91C532000","o":"57B0C67","s":"__eventv","s+":"10D7"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921CE33EC","b":"63E91C532000","o":"57B13EC","s":"__wt_panic_func","s+":"154"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921B1F245","b":"63E91C532000","o":"55ED245","s":"__wt_block_read_off.cold","s+":"2D"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921B1C09B","b":"63E91C532000","o":"55EA09B","s":"__wti_block_extlist_read","s+":"9B"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921B1C5EC","b":"63E91C532000","o":"55EA5EC","s":"__wti_block_extlist_read_avail","s+":"2C"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921B14458","b":"63E91C532000","o":"55E2458","s":"__wt_block_checkpoint_load","s+":"278"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921B2C1A6","b":"63E91C532000","o":"55FA1A6","s":"__bm_checkpoint_load","s+":"36"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921B50AB1","b":"63E91C532000","o":"561EAB1","s":"__wt_btree_open","s+":"E71"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921BB7775","b":"63E91C532000","o":"5685775","s":"__wt_conn_dhandle_open","s+":"435"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921CD1FF0","b":"63E91C532000","o":"579FFF0","s":"__wt_session_get_dhandle","s+":"D0"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921CD23D2","b":"63E91C532000","o":"57A03D2","s":"__wt_session_get_dhandle","s+":"4B2"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921CD2D83","b":"63E91C532000","o":"57A0D83","s":"__wt_session_get_btree_ckpt","s+":"873"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921BE2A74","b":"63E91C532000","o":"56B0A74","s":"__wt_curfile_open","s+":"114"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921CCB300","b":"63E91C532000","o":"5799300","s":"__session_open_cursor_int","s+":"5C0"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921CCAC78","b":"63E91C532000","o":"5798C78","s":"__wt_open_cursor","s+":"58"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921C6D39C","b":"63E91C532000","o":"573B39C","s":"__wt_metadata_cursor_open","s+":"6C"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921C6D46B","b":"63E91C532000","o":"573B46B","s":"__wt_metadata_cursor","s+":"4B"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921BAC918","b":"63E91C532000","o":"567A918","s":"wiredtiger_open","s+":"2718"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921AB8951","b":"63E91C532000","o":"5586951","s":"_ZN5mongo18WiredTigerKVEngine15_openWiredTigerERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_","C":"mongo::WiredTigerKVEngine::_openWiredTiger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)","s+":"A1"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921AC4623","b":"63E91C532000","o":"5592623","s":"_ZN5mongo18WiredTigerKVEngineC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mmbb","C":"mongo::WiredTigerKVEngine::WiredTigerKVEngine(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::ClockSource*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, unsigned long, bool, bool)","s+":"BE3"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921A85CB2","b":"63E91C532000","o":"5553CB2","s":"_ZNK5mongo12_GLOBAL__N_117WiredTigerFactory6createEPNS_16OperationContextERKNS_19StorageGlobalParamsEPKNS_21StorageEngineLockFileE","C":"mongo::(anonymous namespace)::WiredTigerFactory::create(mongo::OperationContext*, mongo::StorageGlobalParams const&, mongo::StorageEngineLockFile const*) const","s+":"152"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E921D29B41","b":"63E91C532000","o":"57F7B41","s":"_ZN5mongo23initializeStorageEngineEPNS_16OperationContextENS_22StorageEngineInitFlagsEPNS_14BSONObjBuilderE","C":"mongo::initializeStorageEngine(mongo::OperationContext*, mongo::StorageEngineInitFlags, mongo::BSONObjBuilder*)","s+":"441"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E920C26591","b":"63E91C532000","o":"46F4591","s":"_ZN5mongo11mongod_mainEiPPc","C":"mongo::mongod_main(int, char**)","s+":"B91"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E920C09A89","b":"63E91C532000","o":"46D7A89","s":"main","s+":"9"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"7D89CD82A1CA","b":"7D89CD800000","o":"2A1CA","s":"__libc_start_call_main","s+":"7A"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"7D89CD82A28B","b":"7D89CD800000","o":"2A28B","s":"__libc_start_main_alias_2","s+":"8B"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"initandlisten","msg":"Frame","attr":{"frame":{"a":"63E920C09965","b":"63E91C532000","o":"46D7965","s":"_start","s+":"25"}}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"\n"}}
{"t":{"$date":"2025-09-02T09:31:11.679-05:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"\n"}}

Jack_Woehr · September 2, 2025, 3:00pm

Make sure your computer meets the minimum hardware requirements

chris · September 2, 2025, 4:25pm

Looks like it is time to restore from backup, checksum error:

{“t”:{“$date”:“2025-09-02T09:31:11.593-05:00”},“s”:“E”, “c”:“WT”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error message”,“attr”:{“error”:0,“message”:{“ts_sec”:1756823471,“ts_usec”:593039,“thread”:“3259868:0x7d89cce74240”,“session_dhandle_name”:“file:WiredTiger.wt”,“session_name”:“connection”,“category”:“WT_VERB_DEFAULT”,“category_id”:12,“verbose_level”:“ERROR”,“verbose_level_id”:-3,“msg”:“__wt_block_read_off:279:WiredTiger.wt: potential hardware corruption, read checksum error for 4096B block at offset 102400: block header checksum of 0x6e726574 doesn’t match expected checksum of 0x406110a5”}}}

Though try a repair first.

Russell_Morris · September 2, 2025, 4:54pm

That seemed to help, thanks! No, no crash from the command line … but ;).

I was running from the command line as sudo (so root), tried to restart the service, and it fails => thinking this is a permissions issue, because I ran command line as root, and now I see in the log (when trying to start the service),

{"t":{"$date":"2025-09-02T11:52:14.510-05:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":13,"message":"[1756831934:510430][3381989:0x7e358d57b240], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 924: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied"}}
{"t":{"$date":"2025-09-02T11:52:14.510-05:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":13,"message":"[1756831934:510714][3381989:0x7e358d57b240], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 924: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied"}}
{"t":{"$date":"2025-09-02T11:52:14.510-05:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error message","attr":{"error":13,"message":"[1756831934:510930][3381989:0x7e358d57b240], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 924: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied"}}
{"t":{"$date":"2025-09-02T11:52:14.510-05:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2025-09-02T11:52:14.510-05:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"13: Permission denied"}}
{"t":{"$date":"2025-09-02T11:52:14.510-05:00"},"s":"F",  "c":"ASSERT",   "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":754}}
{"t":{"$date":"2025-09-02T11:52:14.510-05:00"},"s":"F",  "c":"ASSERT",   "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

Any thoughts how to correct this? Thanks again!

Russell_Morris · September 2, 2025, 6:56pm

Got it working - thanks for the help! After repair I had to fix permissions - not only on log and lock files, but also in /var/lib/mongodb/journal.

Thanks again!

system · September 7, 2025, 6:57pm

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.