Growing WiredTigerHS.wt

Hello:

I am using mongodb 4.4.x in several replicasets, & ! ran into frequent problem with growing file size for WiredTigerHS.wt. The file grows non-stop, eventually eating up all disk space on the server, killing the mongod process.

I want to see if anyone has any idea what might be gone wrong. Below is the mongod config we use.

Thanks in advance.
Eric

storage:
  dbPath: "/var/lib/mongo/corp_group1"
  directoryPerDB: true
  journal:
    enabled: true
  engine: "wiredTiger"
  wiredTiger:
    engineConfig:
      directoryForIndexes: false
    collectionConfig:
      blockCompressor: "snappy"
systemLog:
  quiet: true
  destination: file
  path: "/var/log/mongo/corp_group1/mongod.log"
  logAppend: true
  timeStampFormat: "iso8601-utc"
processManagement:
  fork: true
setParameter:
  cursorTimeoutMillis: 600000
  honorSystemUmask: true
net:
  bindIp: 0.0.0.0
  port: 27018
  unixDomainSocket:
    pathPrefix: "/var/lib/mongo"
sharding:
  clusterRole: "shardsvr"
replication:
  replSetName: "corp_group1"
  enableMajorityReadConcern: false

Hi @Eric_Wong

The file WiredTigerHS.wt contains WiredTiger’s history store. From the relevant page in WiredTiger documentation:

The history store in WiredTiger tracks historical versions of records required to service older readers. By having these records in storage separate from the current version, they can be used to service long running transactions and be evicted as necessary, without interfering with activity that uses the most recent committed versions.

Basically it’s used to keep historical state of your data, due to some process that’s requiring MongoDB to hold onto those old data versions. Typically this was caused by the application opening a long running transaction that never commit/rollback.

I am using mongodb 4.4.x

If you’re not using 4.4.17, I would suggest you to upgrade to it since it’s the latest version in the 4.4 series, just be sure that you’re not hitting an old issue that was resolved. Also note that versions 4.4.2-4.4.8 are not recommended to be used anymore due to some serious issues. Please see the 4.4 Release Notes for more details.

If upgrading doesn’t solve this, then you may need to take a look in your application code for any workload that opens a transaction, a no-timeout cursor, or anything similar.

Best regards
Kevin

1 Like