Mongo DB crashing with an error: Not enough storage is available to process this command

Hi,
Mongo DB is crashing which is causing the application to fail. When extracted mongo Db logs found below error :-1:

2022-09-11T00:36:21.977+0000 E STORAGE [WTJournalFlusher] WiredTiger (-28992) [1662856581:400819][7072:1995194080], WT_SESSION.log_flush: journal/WiredTigerLog.0000158373 handle-sync: FlushFileBuffers error: Not enough storage is available to process this command.

2022-09-11T00:36:21.989+0000 I - [WTJournalFlusher] Invariant failure: s->log_flush(s, “sync=on”) resulted in status UnknownError: -28992: Not enough storage is available to process this command.
This is the second occurrence and After restarting application servers mongo started working as expected.

what is the possible reason for this issue? and do we have any work around to avoid future occurences?

Attaching logs for your reference.

Thanks in advance!

2022-09-11T00:36:21.849+0000 I COMMAND  [conn3987] insert meolutdb.rawDetection ninserted:40 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { w: 1 } } } 113ms
2022-09-11T00:36:21.977+0000 E STORAGE  [WTJournalFlusher] WiredTiger (-28992) [1662856581:400819][7072:1995194080], WT_SESSION.log_flush: journal/WiredTigerLog.0000158373 handle-sync: FlushFileBuffers error: Not enough storage is available to process this command.

2022-09-11T00:36:21.989+0000 I -        [WTJournalFlusher] Invariant failure: s->log_flush(s, "sync=on") resulted in status UnknownError: -28992: Not enough storage is available to process this command.

 at src\mongo\db\storage\wiredtiger\wiredtiger_session_cache.cpp 203
2022-09-11T00:36:25.065+0000 I COMMAND  [conn4275] update meolutdb.eMSStatus query: { _id: "421d0553-e28b-4feb-a48e-7bfb6ed129fc" } update: { _id: "421d0553-e28b-4feb-a48e-7bfb6ed129fc", _class: "com.emsgt.lut.il.dal.api.EMSStatus", subsystem_id: 3, service_id: 10005, service_description: "Motor Temperature", timestamp: 1662856584631000000, status: "OK", errors: " 34.00 celsius", subsystem_type: "Antenna" } keysExamined:1 docsExamined:1 nMatched:1 nModified:1 keyUpdates:6 writeConflicts:0 numYields:1 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 2 } } } 113ms
2022-09-11T00:36:26.505+0000 I COMMAND  [conn3965] insert meolutdb.rawDetection ninserted:40 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { w: 1 } } } 119ms
2022-09-11T00:36:26.632+0000 I COMMAND  [conn4001] insert meolutdb.rawDetection ninserted:40 keyUpdates:0 writeConflicts:0 numYields:0 locks:{ Global: { acquireCount: { r: 1, w: 1 } }, Database: { acquireCount: { w: 1 } }, Collection: { acquireCount: { w: 1 } } } 156ms
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(174)                                   mongo::printStackTrace+0x43
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\log.cpp(136)                                                  mongo::logContext+0xa8
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\assert_util.cpp(164)                                          mongo::invariantOKFailed+0x14c
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_session_cache.cpp(203)            mongo::WiredTigerSessionCache::waitUntilDurable+0x2bd
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp(97)                 mongo::WiredTigerKVEngine::WiredTigerJournalFlusher::run+0x1a8
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    ...\src\mongo\util\background.cpp(152)                                           mongo::BackgroundJob::jobBody+0x1b1
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    c:\program files (x86)\microsoft visual studio 12.0\vc\include\thr\xthread(188)  std::_LaunchPad<std::_Bind<0,void,std::_Bind<1,void,std::_Pmf_wrap<void (__cdecl mongo::dur::JournalWriter::*)(void) __ptr64,void,mongo::dur::JournalWriter>,mongo::dur::JournalWriter * __ptr64 const> > >::_Go+0x1c
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    f:\dd\vctools\crt\crtw32\stdcpp\thr\threadcall.cpp(28)                           _Call_func+0x14
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    f:\dd\vctools\crt\crtw32\startup\threadex.c(376)                                 _callthreadstartex+0x17
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] mongod.exe    f:\dd\vctools\crt\crtw32\startup\threadex.c(354)                                 _threadstartex+0x102
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] kernel32.dll                                                                                   BaseThreadInitThunk+0xd
2022-09-11T00:36:50.126+0000 I CONTROL  [WTJournalFlusher] 
2022-09-11T00:36:50.126+0000 I -        [WTJournalFlusher] 

***aborting after invariant() failure

Hello @Mamatha_K and welcome to the MongoDB community forums. :wave:

This log entry seems to be stating that your disk is full and that the process cannot write to the disk. Have you verified that amount of free space you have on the drive you’re writing to?

3 Likes

@Doug_Duncan : Yes we have 800gb in D drive, 40-50gb is C drive. And mongo DB is around 720gb.

It looks like you should have enough free space on your drives, but I’m not sure if the numbers given are the amount free or the amount total. You also don’t state where MongoDB data is located.

If you have 800G free on the D drive and your MongoDB data is also on the D drive, then you definitely have more than enough free space for most things. You might run into problems if you were to back up to that drive or do some other admin-y type things that would require you to basically copy the data to that drive.

If you have 800G total on the D drive and your MongoDB data is also on the D drive, then you might still have enough space for running the database. I would be concerned that I was sitting at 90% usage on the drive however, especially if your database is growing in size. Once you run out of space (as your error indicates) the system will shutdown to mitigate any potential corruption.

If you have 40 - 50G free on the C drive and your MongoDB data is also on the C drive, then again you might have enough space for running the database. I would caution again that this is not a lot of space compared to the size of the database.

If you have 40 - 50G total on the C drive then your MongoDB data will not be there because it’s just not possible. :wink:

You can take a look at the <dbdata path>/journal/WiredTigerLog.* files to see how big in size they are on average as that might give more insight into what’s going on.