Disk out of space, Can't start MongoDB

Hello,
I have MongoDB running on Ubuntu 22.04 and have my drive formatted as XFS. Some time in the past few days my 512GB drive was suddenly full because of the /var/log/mongodb log file. Now MongoDB won’t start.
I’ve read about enabling log rotation, but I cannot start MongoDB to enable this.
I have a lot of data in MongoDB that I have not backed up in a few weeks so really want to be careful with any steps I take.

What options are possible to get it working?

I suppose you can just remove the log file. It’s for logging only, and doesn’t have any real user data.

At startup mongodb will create the log file if not exist.

That’s what I was wondering. But I had read that removing the log file could result in data loss or problems with the database. Do you think that is a real concern?

from where? log files are for troubleshooting only and any operation on it should never cause user data loss.

If you’re feeling paranoid you can:

truncate -s0 /var/log/mongodb/mongod.log this will keep everything about the file the same except its contents.

If I ever come accross asituation like this I gzip the offending file in place and pkill -USR1 -x mongod as the mongod is not running in your installation just start mongod as @Kobe_W has suggested.

Thanks for the suggestions so far.

I gzipped the file to another directory to clear the space off my XFS /var mount. It compressed down to just 20mb.

I then ran the truncate command and then systemctl started mongod. Just connected with Compass and confirmed all is just fine.

Thanks a lot!
Now I can set up log rotation to prevent this happening again.

1 Like