Debug Out of Memory Crash

We recently moved to a manual install of Rocketchat which uses MongoDB for it’s database and I am completely new to MongoDB.

After running for a weeks without issue today it appears the kernel OOM killer went to work on MongoDB and killed the main server process. Normally when things like this happen, my assumption is that:

  1. there may be a query design flaw, missing index or some other underlying issue causing certain processes to stall or consume more memory than normal.
  2. there may also be some server or db configuration that needs tuning to limit memory consumption

It would also be nice if the Mongo server restarted itself after such events but that’s another story.

As an absolute beginner, can someone please help guide me in getting to the root of what happened here? I see lots of "msg":"Slow query" and "msg":"serverStatus was very slow" messages in the logs among other things. Please let me know what further info I can provide here that would be of use debugging this issue.

Hi @billy_noah,
I think you need to create index to permorf a faster query:

I suggest you to read the production note to have the better environment for mongo (and avoid oom killer) :

And finally i suggest you to install mongodb compass:

Best regards

Thank you for the resources. I was just wondering if the “slow query” messages could simply be related to server memory issues stemming from some other database problem - or if we can confidently assume that they are the root cause of running out of memory? Coming over from MySQL, I’ve seen a single long running process do things like that and affect many other queries and overall performance.

At any rate, is this a place where I can post a few log snippets to get more specific advice?

Hi @billy_noah

Sorry you’re having issues with this. OOMkiller is a common issue that typically points to the lack of resources in the server itself for the workload it’s supposed to do. You are correct that it may be caused by query issues. However if you’re only running Rocketchat, there’s probably not much to be done on that front. That is, other than provisioning a larger server, or setting up swap (this may delay but not eliminate the main issue).

Are you running Rocketchat and mongod in the same server? If yes, it may benefit both apps if you separate them into their own servers. A single mongod process was designed to wholly take over the server to give you the best possible performance out of the hardware you have. Consequently in a busy server they frequently have resource contention issues with other apps, or with other mongod running on the same machine.

Yes this is typical message you see when a server is being overwhelmed by work. MongoDB calls serverStatus every second for telemetry purposes and the call should be very quick to complete (it was designed to place negligible burden on the server). If these calls start to get bogged down, it means the server is really busy.

Yes if you have some log snippets that you think may be helpful, please do post them here.

Best regards
Kevin