Mongod high disk utilisation by ApplyBa.Journal process after upgrading OS from Ubuntu 16.04 LTS to 22.04 LTS

Hi,

I have an installation with three shards (PSA) of MongoDB 4.2. I have to reinstall OS on the servers from Ubuntu 16.04 to 22.04. After I completed the first server from the first shard I had to rsync all data from another member of this shard. Partition on which database is stored has XFS filesystem. The same as previously. After that mongod process started and everything seemed to be ok. However, I noticed very high disk utilisation on the reinstalled server. All other servers have about 5-15% of disk utilisation and this one have about 76%. iotop shows some mongod process called ApplyBa.Journal which creates almost all the load. I wasn’t able to find any information about this process.
I checked the other server from the same shard (and other shards) and there is no such process so it seems that for some reason it runs only on this “fresh” server.

I’m looking for some information about this process. What it does exactly (it is quite obvious that it’s related to journal) and if/how it is possible to do anything with it to decrease disk utilisation.

Hi @luso welcome to the community!

Note that as per the production notes, Ubuntu 22 is not officially supported yet.

Having said that, are you seeing the high IO utilization consistently, or does it taper off to normal levels after a while? The mongod logs may be able to provide further clues as to what is happening. Do you see this only in one Ubuntu 22 installation, or do you have other Ubuntu 22 installations that do not exhibit this behaviour?

Best regards
Kevin

Hi kevinadi,

High IO utilization is consistent since the process started with copied data. Before that it was going through initial sync and IO was high but not that high.

I have one more installation that run mongoDB 4.4.18 on the same OS version without any issues and process ApplyBa.Journal doesn’t appear on that installation.
I thougt it might be related to sharding but there is another mongod instance on the same server which is a member of config server replicaset and I noticed that it also runs the same process.

I checked logs of both mongod and mongod-cfgsrv instances and they do not indicate any problems.

Here is a screenshot from iotop.

there is another mongod instance on the same server which is a member of config server replicaset

If you’re running multiple mongod processes within the same machine, note that this is strongly not recommended since: 1) it can result in resource contention and, 2) can have difiicult to diagnose issues.

I have one more installation that run mongoDB 4.4.18 on the same OS version without any issues and process ApplyBa.Journal doesn’t appear on that installation.

If I understand correctly, this is only happening on MongoDB 4.2 series on Ubuntu 22, and only in this particular hardware. Is this correct?

The MongoDB 4.4.18 installation that doesn’t exhibit this issue, is this on the same hardware, or on different hardware? It is a known possibility that a bad hardware could cause this unexplained disk utilization.

I would try to troubleshoot this in this manner:

  1. Ensure that the deployment follows the production nodes recommendations. This is an important first step to ensure that it’s not caused by suboptimal configuration
  2. Determine whether the config server process or the shard mongod process is causing this
  3. Try using the latest in the 4.2 series (currently 4.2.23) to see if it solves it
  4. Upgrade to MongoDB 4.4.18 since it does not exhibit this issue in at least one instance
  5. Try to migrate the config server node and the shard node to two separate hardware instances to see if this is caused by hardware issues

Best of luck, hopefully this helps!

Best regards
Kevin