Hi,
We are using MongoDB 4.2.0 running in Windows Server 2019 environment. We have a replica set with 2 data nodes and an arbiter. After restarting the service on one of the data nodes, it tried to recover, but complained about not finding a specific file (WiredTigerLog.0000000006):
2021-11-19T09:38:24.226+0100 I STORAGE [initandlisten] WiredTiger message [1637311104:225994][5108:140713466155616], txn-recover: Recovering log 6 through 273
2021-11-19T09:38:24.228+0100 E STORAGE [initandlisten] WiredTiger error (2) [1637311104:228003][5108:140713466155616], txn-recover: __win_open_file, 539: C:\ProgramData\MongoDB\db\journal\WiredTigerLog.0000000006: handle-open: CreateFileW: The system cannot find the file specified.
: No such file or directory Raw: [1637311104:228003][5108:140713466155616], txn-recover: __win_open_file, 539: C:\ProgramData\MongoDB\db\journal\WiredTigerLog.0000000006: handle-open: CreateFileW: The system cannot find the file specified.
: No such file or directory
From now on, the number of these journal files started to grow rapidly, filling the disk completely in less than 10 minutes.
2021-11-19T09:47:33.226+0100 E STORAGE [initandlisten] WiredTiger error (28) [1637311653:226007][5484:140713466155616], connection: __win_file_set_end, 394: C:\ProgramData\MongoDB\db\journal\WiredTigerTmplog.0000000001: handle-set-end: SetEndOfFile: There is not enough space on the disk.
: No space left on device Raw: [1637311653:226007][5484:140713466155616], connection: __win_file_set_end, 394: C:\ProgramData\MongoDB\db\journal\WiredTigerTmplog.0000000001: handle-set-end: SetEndOfFile: There is not enough space on the disk.
: No space left on device
What is strange, the last recovery log lines were encountered 10 days earlier:
2021-11-09T09:59:12.657+0100 I STORAGE [serviceShutdown] WiredTiger message [1636448352:656659][9476:140720848654944], txn-recover: Recovering log 271 through 272
2021-11-09T09:59:12.809+0100 I STORAGE [serviceShutdown] WiredTiger message [1636448352:809657][9476:140720848654944], txn-recover: Recovering log 272 through 272
No problem was detected in the meantime, unit 2021-11-19.
Maybe it is just by coincidence, but it looks strange that it tries to search for file with index 6 up to 273, which is one more as 272 10 days before.
Did someone have a similar problem or experience?
Thank you in advance
Best regards
Lubo