Mongod service doesn't stop and restart after fatal exception

[Nathaniel_brewer]

OS: Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1099-aws x86_64)
MongoDB: v4.4.5

MongoDB experiences fatal exceptions such this one logged in mongod.log

{"t":{"$date":"2023-03-30T00:03:11.041+00:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"ftdc","msg":"Writing fatal message","attr":{"message":"terminate() called. An exception is active; attempting to gather more information"}}

After that, the mongod.service log shows systemd indicating that it is stopping mongod (output from journalctl -u mongod.service)

Mar 30 00:03:11 ip-xxx-xxx-xxx-xxx mongod[27818]: {"t":{"$date":"2023-03-30T00:03:11.933+00:00"},"s":"F",  "c":"CONTROL",Mar 30 14:30:05 ip-xxx-xxx-xxx-xxx systemd[1]: Stopping MongoDB Database Server...

But checking the status of the service with systemctl status mongod.service shows that it is still active (output from sudo journalctl -u mongod.service) but there are not further entries in mongod.log.

● mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2023-03-27 11:14:27 UTC; 72h ago
     Docs: https://docs.mongodb.org/manual
 Main PID: 29624 (mongod)
   CGroup: /system.slice/mongod.service
           └─29624 /usr/bin/mongod --config /etc/mongod.conf

My mongod.conf is pretty cookie cutter, as far as I can tell (will include it below).

Is there anything I can do get mongod to shutdown and restart properly? Or anything I can do to help mongod recovery from the exception?

This is my first post in this community so, if I haven’t included something that would be helpful, please just let me know and I will add it as quickly as I can.

Thank you!

# mongod.conf
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

net:
  port: 27017
  bindIp: 0.0.0.0

processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:
#operationProfiling:
#replication:
#sharding:
## Enterprise-Only Options:
#auditLog:
#snmp:

[Ramachandra_Tummala]

Try with latest mongodb version
Check this thread

MongoDB random crash from time to time, version 4.4.15, Ubuntu 20.04.5 LTS

Hello everyone, We are facing some issues with mongoDB and still have no clue what is going on, so I am posting here for some suggestion/help. The issue is that mongod service is getting stopped/down a few times a day or once a week - completely random. This is what we can see in the mongo logs: {"t":{"$date":"2023-03-25T05:04:26.904+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"WTCheckpointThread","msg":"WiredTiger message","attr":{"message":"[1679720666:903992][803:0x7f0b1dc21700], …

[Nathaniel_brewer]

Ah yes. I do always see those same FTDC log lines. I will start working on updating mongo to v6 in a test env, but would like to try disabling FTDC in production and see if that helps. Do you think that’s a good thing to try?

[Benjamin_Beganovic]

Yes, you could try that. I guess you will get rid of these log lines (because FTDC is creating them), but the Mongo will be still crashing. You should check ulimit (if you are on Linux), antivirus (if any) activity for Mongo directories, and sure, disk space and RAM memory utilization. Keep in mind that disk space and memory utilization could be temporarily high, which means you should have more space than it is consuming at the load.

[Nathaniel_brewer]

Great! I will carry on updating mongodb. Hopefully that will do the trick
Thank you for helping.