We are using MongoDB ReplicaSet with three nodes (One Primary and two secondaries). We are performing load testing. We got a weird Disk Space issue on one of the secondaries even though there is plenty of disk space available.
2022-05-01T19:57:03.058+0000 W FTDC [ftdc] Uncaught exception in 'FileStreamFailed: F ailed to write to interim file buffer for full-time diagnostic data capture: /var/data/mon godb/diagnostic.data/metrics.interim.temp' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
2022-05-01T19:57:04.607+0000 E STORAGE [WTCheckpointThread] WiredTiger error (28) [16514 35024:607232][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_ write, 539: /var/data/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No space left on device Raw: [1651435024:607232][80:0x7f0b7fe7e70 0], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_write, 539: /var/data/mongodb/ WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No sp ace left on device
2022-05-01T19:57:04.607+0000 E STORAGE [WTCheckpointThread] WiredTiger error (28) [16514 35024:607232][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_ write, 539: /var/data/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No space left on device Raw: [1651435024:607232][80:0x7f0b7fe7e70 0], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_write, 539: /var/data/mongodb/ WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No space left on device
10624 2022-05-01T19:57:04.608+0000 E STORAGE [WTCheckpointThread] WiredTiger error (28) [1651435024:608377][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_ write, 539: /var/data/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No space left on device Raw: [1651435024:608377][80:0x7f0b7fe7e70 0], file:WiredTiger.wt, WT_SESSION.checkpoint: __posix_file_write, 539: /var/data/mongodb/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1262 bytes at offset 0: No space left on device
10625 2022-05-01T19:57:04.608+0000 E STORAGE [WTCheckpointThread] WiredTiger error (28) [16514 35024:608480][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_turtle_u pdate, 391: WiredTiger.turtle: fatal turtle file update error: No space left on device Raw : [1651435024:608480][80:0x7f0b7fe7e700], file:WiredTiger.wt, WT_SESSION.checkpoint: __wt_ turtle_update, 391: WiredTiger.turtle: fatal turtle file update error: No space left on device
2022-05-01T19:57:04.608+0000 F - [WTCheckpointThread] Fatal Assertion 50853 at src /mongo/db/storage/wiredtiger/wiredtiger_util.cpp 486
10628 2022-05-01T19:57:04.608+0000 F - [WTCheckpointThread] \n\n***aborting after fassert() failure\n\n