High RAM, frequent replication lag and high secondary lock usage in 4.2 WT compare to 4.0 MMAP

Hi Team,

Currently, I am using mongo 4.0 MMapV1 in the environment. The setup is in PSSSAAA(7 members) architecture. As Mongo 4.0 will be EoLed in April 2022, so upgrading the setup to mongo 4.2.

Below are the difference we observed for same scenerio in 4.0 MMAP and 4.2 WT

Environment:
VM - 64 Gb RAM

  • 4.0 mmap:
    no of instances - 4 (PS members)
    storage - tmpfs (13 gb for 1 instance)

  • 4.2 WT:
    no of instances - 4 (PS members)
    storage - tmpfs (13 gb for 1 instance)
    wiredtiger cache value for PS members: 5 Gb for 1 member

Difference found in below statistics:

  • available RAM(free+ cache) : it is decreased in 4.2 WT storage engine by 4-5 GB compare to 4.0 mmap for above environment

  • Locking:
    we are using below statistics for checking primary and secondary locking
    (timeAcquiringMicros/acquireWaitCount)
    Observations:
    for secondary members, lock frequency is increased for WT and lock time is also increased (for mmap max lock time is 6 ms, for WT it is 14 ms).
    for primary memebers, there is approx zero lock observed in WT but mmap we got some locks

  • Replication Lag:
    In WT replication lag frequency is increased , time value (1ms) is same in both mmap and WT.

Queries:

  • Any tuning can be done to avoid this additional resource utilization in WT

  • Secondary lock % & replication lag, may cause problem in long hours. so we wanted to know, whether this is expected in WT and do we see any problem because of this.

Kindy provide your inputs.

Thanks,
Kapil

Hi @Kapil_Gupta

MMAPv1 and WiredTiger works very differently and would require a different tuning. WiredTiger has a lot more features vs. MMAPv1, like MVCC, compression, transactions, better concurrency control, etc. while MMAPv1 supports none of these features. Optimization methods that works with MMAPv1 will not work the same way in WiredTiger, especially since WiredTiger was designed to handle much higher concurrency than MMAPv1 can ever do.

To directly answer your questions:

No I don’t think so. I would say that extra features would imply different resource utilization metrics. Tuning WiredTiger to use less resources will not increase its performance.

This would depend on many things. If your deployment was specifically tuned for MMAPv1, it would need a rethink on how it can cater to WiredTiger better.

Having said that, I’d like to know more about your deployment:

Is there a reason why you need so many arbiters? I would not recommend using that many arbiters unless 1) You absolutely have no other options, and 2) You know exactly how the arbiters will affect your writes and availability. Specifically, using arbiters will hinder the use of majority write concern and read concern.

Since you have a 64GB VM, why do you set the WiredTiger cache at 5GB? That sounds extremely low to me, unless you’re running multiple mongod inside one VM. It is strongly not recommended to run multiple mongods since you’ll have resource contention issues.

To start with, I would recommend:

  • Not run multiple mongods in a node, and set the WiredTiger cache to its default value (~50% of available RAM)
  • Do not use tmpfs. Rather, use the recommended filesystem for WiredTiger (either XFS or Ext4 for Linux). Also, please follow the settings recommended in the production notes
  • See if you can run without any arbiters, to gain better write concern & read concern guarantees

Best regards
Kevin

2 Likes