Troubleshoot Replication Lag

Network Latency

Replication lag can build up when cluster nodes cannot reliably communicate with each other.

Check the network routes between the members of your replica set to ensure that there is no packet loss or network routing issues.

Use tools including ping to test latency between set members and traceroute to expose the routing of packets between network endpoints.

Alternatively, run replSetGetStatus and examine the pingMs field. This returns the current network latency in milliseconds between the primary and secondary nodes.

Secondary Resource Exhaustion

Secondary nodes can experience resource contention when they cannot efficiently handle incoming read operations from the primary. This can cause memory issues such as cache contention. When the cache reaches critical thresholds, the server uses application threads to evict pages, which reduces the number of threads available to manage replication.

To see whether the server redirects application threads to eviction tasks, run the following command in your mongosh shell:

db.serverStatus().wiredTiger.cache['pages evicted by application threads']

• If Result = 0: No application threads are paused for eviction.

• If Result > 0 (and increasing): The database is under cache pressure. Incoming queries must evict cached data before they execute, which can increase replication lag.

Atlas users can also monitor the WiredTiger Cache Activity and Page Faults metrics to investigate cache-related issues.

For potential strategies to fix this issue, see Scale your resources .

Disk-Related Issues

If a secondary node cannot flush dirty data to disk quickly enough, it falls behind the primary. This phenomenon occurs when the volume of writes coming from the primary exceeds the write speed of the secondary's disk.

Disk-related issues are prevalent on multi-tenant systems, including virtualized instances, and can be transient if the system accesses disk devices over an IP network.

To assess disk status, use system-level tools, such as iostat or vmstat.

Atlas users can access Atlas metrics such as Disk IOPS and Disk Space Used to investigate disk issues.

Some common causes of disk issues include:

• Under provisioning: The secondary has slower disks, or lower IOPS, than the primary.

• Virtualization Overhead: In shared environments, other virtual machines may saturate the physical disk controller.

Some possible solutions to disk issues include:

• Increasing provisioned IOPS

• Upgrading to NVMe Storage

• Upgrading to a higher Atlas cluster tier. For more information, see Atlas Cluster Sizing and Tier Selection.

Long Running Operations

In some cases, long-running operations on the primary can block replication on secondaries. For best results, configure write concern to require confirmation of replication to secondaries. This prevents write operations from returning if replication cannot keep up with the write load.

You can also use the database profiler to identify slow queries or long-running operations that correlate with the observed lag.

Excessive Write Load

Bulk write operations can exceed replica sets' ability to replicate in a timely manner, causing replication lag.

The following sub-sections offer possible solutions to this issue:

db.collName.deleteMany({createdDate: {$gte: new Date("2018-12-01"), $lt: new Date("2019-01-01")}});
db.collName.deleteMany({createdDate: {$gte: new Date("2018-11-01"), $lt: new Date("2018-12-01")}});
db.collName.deleteMany({createdDate: {$gte: new Date("2018-10-01"), $lt: new Date("2018-11-01")}});

db.collection.createIndex({ status: 1 });

db.collection.deleteMany({ status: "inactive" });

oplog Window Size

If your oplog window is too small for the amount of data you are syncing, you might experience replication lag. A larger oplog can give a replica set a greater tolerance for lag.

To check the size of the oplog and the date ranges of its operations for a given replica set member, connect to the member in mongosh and run the rs.printReplicationInfo() method. The oplog should be long enough to hold all transactions for the longest downtime you expect on a secondary. [1] At a minimum, an oplog should be able to hold minimum 24 hours of operations; however, many users prefer to have 72 hours or even a week's work of operations.

To change oplog size, see the Change the Oplog Size of Self-Managed Replica Set Members tutorial.