Growing WiredTigerHS.wt

Ops and Admin
1

Hello:

I am using mongodb 4.4.x in several replicasets, & ! ran into frequent problem with growing file size for WiredTigerHS.wt. The file grows non-stop, eventually eating up all disk space on the server, killing the mongod process.

I want to see if anyone has any idea what might be gone wrong. Below is the mongod config we use.

Thanks in advance.
Eric

storage:
  dbPath: "/var/lib/mongo/corp_group1"
  directoryPerDB: true
  journal:
    enabled: true
  engine: "wiredTiger"
  wiredTiger:
    engineConfig:
      directoryForIndexes: false
    collectionConfig:
      blockCompressor: "snappy"
systemLog:
  quiet: true
  destination: file
  path: "/var/log/mongo/corp_group1/mongod.log"
  logAppend: true
  timeStampFormat: "iso8601-utc"
processManagement:
  fork: true
setParameter:
  cursorTimeoutMillis: 600000
  honorSystemUmask: true
net:
  bindIp: 0.0.0.0
  port: 27018
  unixDomainSocket:
    pathPrefix: "/var/lib/mongo"
sharding:
  clusterRole: "shardsvr"
replication:
  replSetName: "corp_group1"
  enableMajorityReadConcern: false
2

Hi @Eric_Wong

The file WiredTigerHS.wt contains WiredTiger’s history store. From the relevant page in WiredTiger documentation:

The history store in WiredTiger tracks historical versions of records required to service older readers. By having these records in storage separate from the current version, they can be used to service long running transactions and be evicted as necessary, without interfering with activity that uses the most recent committed versions.

Basically it’s used to keep historical state of your data, due to some process that’s requiring MongoDB to hold onto those old data versions. Typically this was caused by the application opening a long running transaction that never commit/rollback.

I am using mongodb 4.4.x

If you’re not using 4.4.17, I would suggest you to upgrade to it since it’s the latest version in the 4.4 series, just be sure that you’re not hitting an old issue that was resolved. Also note that versions 4.4.2-4.4.8 are not recommended to be used anymore due to some serious issues. Please see the 4.4 Release Notes for more details.

If upgrading doesn’t solve this, then you may need to take a look in your application code for any workload that opens a transaction, a no-timeout cursor, or anything similar.

Best regards
Kevin

1 Like
3

but why it grows in standalone mongodb without replSet?
stat WiredTigerHS.wt
File: WiredTigerHS.wt
Size: 362814038016 Blocks: 708621176 IO Block: 4096 regular file

and how to disable history and compact this file?

mongdb 6.0.5
overall collections size ~14Tb / 800M docs

4

you can still have transaction on it. So doesn’t matter.

5

Hi @Tema_Gordiyenko welcome to the community!

Could you post the ouput of db.adminCommand( { getCmdLineOpts: 1 } ), and also the output of rs.status()?

You cannot disable the history store as far as I know.

Best regards
Kevin

6

**MongoServerError**: not running with --replSet
i can’t set neither read preference nor write concern - it is “you can’t - not replica set”

db.adminCommand( { getCmdLineOpts: 1 } )
{
  argv: [ '/usr/bin/mongod', '--quiet', '--config', '/etc/mongod.conf' ],
  parsed: {
    config: '/etc/mongod.conf',
    net: { bindIp: '::1,127.0.0.1', ipv6: true, port: 27017 },
    operationProfiling: { slowOpSampleRate: 1, slowOpThresholdMs: 10000 },
    processManagement: {
      fork: false,
      pidFilePath: '/var/run/mongodb/mongod.pid',
      timeZoneInfo: '/usr/share/zoneinfo'
    },
    setParameter: {
      ShardingTaskExecutorPoolMinSize: '150',
      allowDiskUseByDefault: '1',
      connPoolMaxConnsPerHost: '1000',
      connPoolMaxInUseConnsPerHost: '300',
      globalConnPoolIdleTimeoutMinutes: '300'
    },
    storage: {
      dbPath: '/db/mongo/d',
      directoryPerDB: true,
      engine: 'wiredTiger',
      journal: { commitIntervalMs: 300, enabled: true },
      wiredTiger: {
        collectionConfig: { blockCompressor: 'zstd' },
        engineConfig: {
          cacheSizeGB: 128,
          configString: 'eviction=(threads_min=20,threads_max=20),checkpoint=(wait=60),eviction_dirty_trigger=15,eviction_dirty_target=3,eviction_trigger=95,eviction_target=90',
          directoryForIndexes: true,
          journalCompressor: 'none',
          zstdCompressionLevel: 10
        }
      }
    },
    systemLog: {
      destination: 'file',
      logAppend: true,
      path: '/db/mongo/log/mongod.log',
      quiet: true
    }
  },
  ok: 1
}

I do not want replSet but any transactional stuff is documented only for replicas.

7

413290467328 May 29 12:23 WiredTigerHS.wt
how to stop it from growing ???

8

ok. it can. but app not use this functionality.
so why?

9

Hi @Tema_Gordiyenko

Sorry you’re seeing this. This is pretty much unexpected, as typically the WiredTigerHS.wt file growth is usually related to replication. This is quite strange as you don’t use replication.

At the moment, there are some options that may be possible to do:

  1. Is this still happening when you don’t have any client connected to the database at all? Let’s say you restart the node without any client connected. Do you still see it grow? Or do you see it grow once a certain client connect to the database? This may be important to determine what’s causing this phenomenon.
  2. Try to upgrade to MongoDB 6.0.6 (latest in the 6.0 series currently) and see if this is still an issue. This is to ensure that you’re not seeing an old fixed issue.
  3. Make a copy of the dbpath without the WiredTigerHS.wt file, and try to run mongod using that dbpath. You might need to run mongod --repair to make it work.
  4. If this is still happening, perhaps opening a SERVER ticket is the next thing, as this will require a deeper knowledge into your situation and deployment. You might need to attach some files and give further information about your data & deployment to assist troubleshooting in the ticket.

Best regards
Kevin

10

it is growing not monotonically. Somthing causing it to grow - maybe TTL drops or gridfs?

418498617344 May 30 05:55 WiredTigerHS.wt

  1. can’t stop - it is production DB
  2. will try this weekend
  3. will try this weekend
    3.1 - will split main db and gridfs to separate mongod
11 
422 291 902 464 May 31 02:42 WiredTigerHS.wt
427 954 937 856 May 31 06:01 WiredTigerHS.wt
428 556 591 104 May 31 07:00 WiredTigerHS.wt
12

it was very stupid solution run --repair on 16Tb data
mongo use 1 thread to read and verify collections and indexes it is processing about 2Tb per 24h
on 32CPU/256GB/io2 64kiops …
terminated and restored from snapshot.
also created clone and restored from backup
mongo 6.0.6

2658304 Jun 4 14:48 WiredTigerHS.wt
10194944 Jun 4 14:51 WiredTigerHS.wt
7569408 Jun 5 07:13 WiredTigerHS.wt
39284736 Jun 5 14:39 WiredTigerHS.wt
8404992 Jun 8 15:00 WiredTigerHS.wt

increasing/decreasing …