Huge WiredTigerHS.wt file and very slow server startup/shudown

I am having a problem with growth of the WiredTigerHS.wt file in the database directory and its supposed effect on MongoDB server startup and shutdown duration.

I run a single node server. Collections files are stored on HDD array (with SSD cache though), indexes and journals are moved to SSD.
Total collections size on disk is about 50 TB.
The overall performance is OK, it’s just the slow server startup and shutdown times what’s really messes up my MongoDB experience.

The longer Mongo instance works uninterruptedly, the slower gets the shutdown or the recovery process after unclean shutdown. Once it took more than 3 days to startup MongoDB after a power failure, and clean shutdown didn’t happen because mongo was shutting down for more than 30 minutes and then my UPS just gave up.
When I stop Mongo instance normally, it could also take a day or two to finish shutdown process.

Monitoring the long recovery pocess, I found that the only thing than MongoDB process does most of the time is just reading the WiredTigerHS.wt file. This file could grow up to multiple TBs is size (once I saw a 3.1TB), and it seems that the larger this file is, the longer it takes Mongo to startup or shutdown properly. After some time, the size of the file reduces, but I didn’t find when exactly does this happen. I certainly did not observe it shrunk after a clean shutdown or right after a startup, must be somewhere during normal DB operation.

I don’t really mind the size of the WiredTigerHS.wt, but the start/stop times is what really matters to me.

I would like to know why is this file gets so large and if there any way to improve MongoDB startup and shutdown times, because with multiple days downtime after a server shutdown or upgrade, the whole system just gets unusable.

Right now I use MongoDB 7.0.5, but I definitely observed the same behavior on 5.x and 6.x versions.

I read this topic, but it doesn’t contain a solution for WiredTigherHS.wt growth.

1 Like

Hi,

Same here. Anyone with same problem? After upgrade from 4.4 to 5.0.23, WiredTigherHS.wt is very huge and is growing every day. The used size in /mongodb/data is different on PRIMARY and SECONDARY member because of this file.

ps: minSnapshotHistoryWindowInSeconds parameter is set to 300 seconds.