Huge WiredTigerHS.wt file and very slow server startup/shudown

Ops and Admin
1

I am having a problem with growth of the WiredTigerHS.wt file in the database directory and its supposed effect on MongoDB server startup and shutdown duration.

I run a single node server. Collections files are stored on HDD array (with SSD cache though), indexes and journals are moved to SSD.
Total collections size on disk is about 50 TB.
The overall performance is OK, it’s just the slow server startup and shutdown times what’s really messes up my MongoDB experience.

The longer Mongo instance works uninterruptedly, the slower gets the shutdown or the recovery process after unclean shutdown. Once it took more than 3 days to startup MongoDB after a power failure, and clean shutdown didn’t happen because mongo was shutting down for more than 30 minutes and then my UPS just gave up.
When I stop Mongo instance normally, it could also take a day or two to finish shutdown process.

Monitoring the long recovery pocess, I found that the only thing than MongoDB process does most of the time is just reading the WiredTigerHS.wt file. This file could grow up to multiple TBs is size (once I saw a 3.1TB), and it seems that the larger this file is, the longer it takes Mongo to startup or shutdown properly. After some time, the size of the file reduces, but I didn’t find when exactly does this happen. I certainly did not observe it shrunk after a clean shutdown or right after a startup, must be somewhere during normal DB operation.

I don’t really mind the size of the WiredTigerHS.wt, but the start/stop times is what really matters to me.

I would like to know why is this file gets so large and if there any way to improve MongoDB startup and shutdown times, because with multiple days downtime after a server shutdown or upgrade, the whole system just gets unusable.

Right now I use MongoDB 7.0.5, but I definitely observed the same behavior on 5.x and 6.x versions.

I read this topic, but it doesn’t contain a solution for WiredTigherHS.wt growth.

1 Like
2

Hi,

Same here. Anyone with same problem? After upgrade from 4.4 to 5.0.23, WiredTigherHS.wt is very huge and is growing every day. The used size in /mongodb/data is different on PRIMARY and SECONDARY member because of this file.

ps: minSnapshotHistoryWindowInSeconds parameter is set to 300 seconds.

3

Recently I updated to MongoDB 7.0.12, which has more verbose recovery logging, and found that after unclean shutdown, followed by startup and clean restart, both global recovery timestamp and global oldest timestamp are reset to zero:

t={"$date":"2024-06-29T14:40:16.985+00:00"} s=I c=WTRECOV id=22430 ctx=initandlisten msg=WiredTiger message attr={"message":{"ts_sec":1719672016,"ts_usec":985427,"thread":"1:0x7f9524a84c80","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"Set global recovery timestamp: (0, 0)"}}

t={"$date":"2024-06-29T14:40:16.985+00:00"} s=I c=WTRECOV id=22430 ctx=initandlisten msg=WiredTiger message attr={"message":{"ts_sec":1719672016,"ts_usec":985436,"thread":"1:0x7f9524a84c80","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"Set global oldest timestamp: (0, 0)"}}

what leads to a complete re-reading of WiredTigerHS.wt, which takes forever (for some reason, it’s size is about 8TB on a 66TB single node DB).

I’m no MongoDB internals expert, but shouldn’t those timestamps be at some reasonable point in time, like within a few minutes before the unclean shutdown?