A couple of days ago, one of our servers hosting our mongoDB containers experienced an incredibly high load (more than 200x higher than the usual average, we’re still investigating where that came from). However, as all our apps relying on the databases seemed to behave normally and the general metrics of the server were still fine (it was responsive to requests, total cpu was < 20%, total memory < 50%, total disk usage < 30%, no active swapping), we decided to wait a while until we restarted the whole server to bring it back into a stable state (we had to, as we didn’t have ssh or console access at that point anymore due to the high load, but that is a different issue).
Again, during the incident all our apps behaved normally (nodejs + mongoose) and claimed that data had been written to the database and could also be retrieved normally.
The interesting (as in “wtf”) thing happened once we restarted the whole server once there was no user-interaction anymore, gained back control, restarted the containers and everything seemed to be back to normal. The only problem: All the data, that should’ve been written (according to the apps), was gone.
So, my question:
Is it possible, that mongoDB stores data in memory only, even without writing to the journal for a long time (1h+) in case there are problems like an extremely high load?
For me it simply doesn’t make any sense that the apps were behaving normally, users could interact with it as they usually do, were able to “save” data to the database and retrieve it later on (no client-side caching involved), but the data was not persisted to the database at all. We were also able to confirm that there had been no overwriting of data on our side.
We’re using an Ubuntu 18_04 Server that hosts vanilla mongo 4.0.7 instances (journaling enabled, no replica sets). Client is nodeJs + mongoose.