Why does mongo crash?

its a clean install of latest ver as per instructions for installation on ubuntu server 22.04.

I have a /data directory which is an (unlocked) LUKS partition.

i have updated mongod.conf to store data in subdirs of /data :

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /data/var/lib/mongodb
#  engine:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

Mongod when started. Logs read:

{"t":{"$date":"2023-08-09T12:28:18.565+00:00"},"s":"I",  "c":"CONTROL",  "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2023-08-09T12:28:18.565+00:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2023-08-09T12:28:18.565+00:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"initandlisten","msg":"Now exiting"}
{"t":{"$date":"2023-08-09T12:28:18.565+00:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":100}}

{"t":{"$date":"2023-08-09T12:36:18.215+00:00"},"s":"I",  "c":"CONTROL",  "id":20698,   "ctx":"-","msg":"***** SERVER RESTARTED *****"}
{"t":{"$date":"2023-08-09T12:36:18.215+00:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"main","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingIn>
{"t":{"$date":"2023-08-09T12:36:18.216+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2023-08-09T12:36:18.217+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueu>
{"t":{"$date":"2023-08-09T12:36:18.225+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","namespace":"config.tenantMigrati>
{"t":{"$date":"2023-08-09T12:36:18.225+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","namespace":"config.tenantMig>
{"t":{"$date":"2023-08-09T12:36:18.225+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"ShardSplitDonorService","namespace":"config.tenantSplitDonors">
{"t":{"$date":"2023-08-09T12:36:18.225+00:00"},"s":"I",  "c":"CONTROL",  "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
{"t":{"$date":"2023-08-09T12:36:18.225+00:00"},"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":73698,"port":27017,"dbPath":"/data/var/lib/mongodb","architecture":"64-bit","host":">
{"t":{"$date":"2023-08-09T12:36:18.225+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"6.0.8","gitVersion":"3d84c0dd4e5d99be0d69003652313e7eaf4cdd74","openSSLV>
{"t":{"$date":"2023-08-09T12:36:18.225+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"22.04"}}}
{"t":{"$date":"2023-08-09T12:36:18.225+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"127.0.0.1","port":27017>
{"t":{"$date":"2023-08-09T12:36:18.226+00:00"},"s":"I",  "c":"STORAGE",  "id":22297,   "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/c>
{"t":{"$date":"2023-08-09T12:36:18.226+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=15484M,session_max=33000,eviction=(threads_min=4,threads_max>
{"t":{"$date":"2023-08-09T12:36:19.079+00:00"},"s":"I",  "c":"STORAGE",  "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":853}}
{"t":{"$date":"2023-08-09T12:36:19.079+00:00"},"s":"I",  "c":"RECOVERY", "id":23987,   "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2023-08-09T12:36:19.480+00:00"},"s":"W",  "c":"CONTROL",  "id":22120,   "ctx":"initandlisten","msg":"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted","ta>
{"t":{"$date":"2023-08-09T12:36:19.480+00:00"},"s":"W",  "c":"CONTROL",  "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":102400,"maxConns":51200},"tags":["sta>
{"t":{"$date":"2023-08-09T12:36:19.500+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"admin.system.version","uuidDisposition":"provided","uuid":{"uuid":{"$uuid":"4>
{"t":{"$date":"2023-08-09T12:36:19.690+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"458a0e57-e541-4be3-bc87-a9be>
{"t":{"$date":"2023-08-09T12:36:19.690+00:00"},"s":"I",  "c":"REPL",     "id":20459,   "ctx":"initandlisten","msg":"Setting featureCompatibilityVersion","attr":{"newVersion":"6.0"}}
{"t":{"$date":"2023-08-09T12:36:19.690+00:00"},"s":"I",  "c":"REPL",     "id":5853300, "ctx":"initandlisten","msg":"current featureCompatibilityVersion value","attr":{"featureCompatibilityVersion":"6.0","context":"setFCV"}}
{"t":{"$date":"2023-08-09T12:36:19.690+00:00"},"s":"I",  "c":"NETWORK",  "id":4915702, "ctx":"initandlisten","msg":"Updated wire specification","attr":{"oldSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"in>
{"t":{"$date":"2023-08-09T12:36:19.690+00:00"},"s":"I",  "c":"NETWORK",  "id":4915702, "ctx":"initandlisten","msg":"Updated wire specification","attr":{"oldSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"in>
{"t":{"$date":"2023-08-09T12:36:19.690+00:00"},"s":"I",  "c":"REPL",     "id":5853300, "ctx":"initandlisten","msg":"current featureCompatibilityVersion value","attr":{"featureCompatibilityVersion":"6.0","context":"startup"}}
{"t":{"$date":"2023-08-09T12:36:19.690+00:00"},"s":"I",  "c":"STORAGE",  "id":5071100, "ctx":"initandlisten","msg":"Clearing temp directory"}
{"t":{"$date":"2023-08-09T12:36:19.690+00:00"},"s":"I",  "c":"CONTROL",  "id":20536,   "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
{"t":{"$date":"2023-08-09T12:36:19.691+00:00"},"s":"I",  "c":"FTDC",     "id":20625,   "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/data/var/lib/mongodb/diagnostic.data"}}
{"t":{"$date":"2023-08-09T12:36:19.693+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"initandlisten","msg":"createCollection","attr":{"namespace":"local.startup_log","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"c9a>
{"t":{"$date":"2023-08-09T12:36:19.899+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"initandlisten","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"c9a0e516-d2c8-4526-a0c5-7a87>
{"t":{"$date":"2023-08-09T12:36:19.899+00:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigReplicationDisabled","oldState":"ConfigPreStart"}}
{"t":{"$date":"2023-08-09T12:36:19.900+00:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
{"t":{"$date":"2023-08-09T12:36:19.901+00:00"},"s":"I",  "c":"CONTROL",  "id":20712,   "ctx":"LogicalSessionCacheReap","msg":"Sessions collection is not set up; waiting until next sessions reap interval","attr":{"error":"NamespaceNo>
{"t":{"$date":"2023-08-09T12:36:19.902+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"LogicalSessionCacheRefresh","msg":"createCollection","attr":{"namespace":"config.system.sessions","uuidDisposition":"generated","uuid":{"u>
{"t":{"$date":"2023-08-09T12:36:19.902+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
{"t":{"$date":"2023-08-09T12:36:19.902+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"127.0.0.1"}}
{"t":{"$date":"2023-08-09T12:36:19.902+00:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
{"t":{"$date":"2023-08-09T12:36:20.200+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"LogicalSessionCacheRefresh","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"420a6230-492e-4>
{"t":{"$date":"2023-08-09T12:36:20.200+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"LogicalSessionCacheRefresh","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"420a6230-492e-4>
{"t":{"$date":"2023-08-09T12:36:20.200+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"LogicalSessionCacheRefresh","msg":"Slow query","attr":{"type":"command","ns":"config.system.sessions","command":{"createIndexes":"system.s>
{"t":{"$date":"2023-08-09T12:43:08.394+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"thread26","msg":"WiredTiger error message","attr":{"error":13,"message":{"ts_sec":1691584988,"ts_usec":394320,"thread":"73698:0x7fcde9c086>
{"t":{"$date":"2023-08-09T12:43:08.394+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"thread26","msg":"WiredTiger error message","attr":{"error":13,"message":{"ts_sec":1691584988,"ts_usec":394483,"thread":"73698:0x7fcde9c086>
{"t":{"$date":"2023-08-09T12:43:08.394+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"thread26","msg":"WiredTiger error message","attr":{"error":13,"message":{"ts_sec":1691584988,"ts_usec":394538,"thread":"73698:0x7fcde9c086>
{"t":{"$date":"2023-08-09T12:43:08.394+00:00"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"thread26","msg":"WiredTiger error message","attr":{"error":-31804,"message":{"ts_sec":1691584988,"ts_usec":394599,"thread":"73698:0x7fcde9>
{"t":{"$date":"2023-08-09T12:43:08.394+00:00"},"s":"F",  "c":"ASSERT",   "id":23089,   "ctx":"thread26","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":712}}
{"t":{"$date":"2023-08-09T12:43:08.394+00:00"},"s":"F",  "c":"ASSERT",   "id":23090,   "ctx":"thread26","msg":"\n\n***aborting after fassert() failure\n\n"}
{"t":{"$date":"2023-08-09T12:43:08.394+00:00"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"thread26","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31380,   "ctx":"thread26","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"5611E42FDD14","b":"5611DF499000","o":"4E64D14","s":"_ZN5mongo18stack_trace_de>
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"5611E42FDD14","b":"5611DF499000","o":"4E64D14","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__>
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"5611E4300259","b":"5611DF499000","o":"4E67259","s":"_ZN5mongo15printStackTraceEv","C":"mongo>
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"5611E42F9F36","b":"5611DF499000","o":"4E60F36","s":"abruptQuit","s+":"66"}}}
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"7FCDEE662520","b":"7FCDEE620000","o":"42520","s":"__sigaction","s+":"50"}}}
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"7FCDEE6B6A7C","b":"7FCDEE620000","o":"96A7C","s":"pthread_kill","s+":"12C"}}}
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"7FCDEE662476","b":"7FCDEE620000","o":"42476","s":"raise","s+":"16"}}}
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"7FCDEE6487F3","b":"7FCDEE620000","o":"287F3","s":"abort","s+":"D3"}}}
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"5611E13BB518","b":"5611DF499000","o":"1F22518","s":"_ZN5mongo25fassertFailedWithLocationEiPK>
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"5611E0F640BA","b":"5611DF499000","o":"1ACB0BA","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_erro>
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"5611E1F444C1","b":"5611DF499000","o":"2AAB4C1","s":"__eventv","s+":"E61"}}}
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"5611E0F783F9","b":"5611DF499000","o":"1ADF3F9","s":"__wt_panic_func","s+":"13A"}}}
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"5611E1EC06D5","b":"5611DF499000","o":"2A276D5","s":"__log_server","s+":"485"}}}
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"7FCDEE6B4B43","b":"7FCDEE620000","o":"94B43","s":"pthread_condattr_setpshared","s+":"513"}}}
{"t":{"$date":"2023-08-09T12:43:08.470+00:00"},"s":"I",  "c":"CONTROL",  "id":31445,   "ctx":"thread26","msg":"Frame","attr":{"frame":{"a":"7FCDEE746A00","b":"7FCDEE620000","o":"126A00","s":"__xmknodat","s+":"23

Since I feel out of my depth, here, I resorted to using GPT 3.5, it tells me:

It looks like the MongoDB server encountered a fatal error and has crashed. The log you've provided shows a sequence of events leading up to the crash. Here's a summary of what's happening:

1. The server starts and goes through its initialization process.
2. Various components and services are registered, and the server initializes its storage engine (WiredTiger).
3. The server starts listening on certain addresses and ports for incoming connections.
4. Several index builds and collections are created.
5. Various configuration options and settings are applied.

However, at some point, the server encounters a critical error:

1. There are several error messages from the WiredTiger storage engine, indicating various types of errors.
2. The error messages are related to file system operations, thread operations, and other internal operations of the storage engine.

Finally, the server crashes:

1. An "assertion" failure is triggered, which is a mechanism in programming to catch logical inconsistencies.
2. The assertion failure leads to the server generating a "fatal message" and ultimately aborting.

The exact cause of this crash could be due to a variety of factors, such as hardware issues, misconfiguration, corruption of data files, or software bugs. To troubleshoot and resolve this issue, you might need to analyze the error messages in more detail, inspect the MongoDB configuration settings, and investigate the health of the underlying hardware and file system. If you have access to MongoDB support or a community forum, reaching out for assistance can also be helpful.

Please can someone shed light on this. I’m stoked to go live, but can’t proceed without mongo! :slight_smile:

Thanks all,
Sam

That was a great demonstration of how useless ChatGPT can be :smile:
Everything it told you is right there in the error dump.

SIG 6 is abort() called by the C++ library.

It has some problem with your file system or its understanding of the layout, I am inferring.

If you set the configuration back the way it was before you changed it, does it work?

This is like the old joke:
“Doctor, it hurts when I do this.”
“Don’t do that!”

1 Like

I absolutely must use my /data dir.

I will revert the db path and get back to you in a couple of hours.

Many thanks for your support

Okay, yes: The db server stays up when using the default path.

(Now I have an error regarding my multer-gridfs-storage connection from my express api, which doesn’t exist in my local version… I set the env vars when launching in pm2, so not sure what th- e issue is, but I’ll investigate tomorrow. Now back to topic!)

Please tell me how I can keep the server up, whilst storing all my user data in subdirs of /data

Thanks for your support :slight_smile:
Sam

I am afraid I don’t know what the problem is. All I can suggest is that you carefully read all information on MongoDB configuration and try to determine what causes the problem.

If your file system is unusual, some kind of mount, that might be the problem. I suspect that the engine wants direct file system access.

But I’m not really sure. Perhaps a MongoDB employee will have a thought on this.

Oh dear. I’ll do some more work on this tomorrow, then. It is an encrypted volume, that is part of a vg, mounted at /data

My host os is ubuntu 22.04

I really hope I can get this completed

Hopefully someone knows…

Thanks,
Sam

@Tarun_Gaur do you have any suggestions for this user ? ^^^^