oplog Sizing

This version of the documentation is archived and no longer maintained. View the current documentation for up-to-date guidance on mongosync usage and instructions on how to upgrade your version of mongosync.

The mongosync program uses change streams to synchronize data between source and destination clusters. mongosync does not access the oplog directly, but when a change stream returns events from the past, the events must be within the oplog time range.

mongosync applies operations in the oplog on the source cluster to the data on the destination cluster. When operations that mongosync has not applied roll off the oplog on the source cluster, the sync fails and mongosync exits.

Note

mongosync does not replicate applyOps operations made on the source cluster during sync to the destination cluster.

During the initial sync, mongosync may apply operations at a slower rate due to copying documents concurrently. After the initial sync, mongosync applies changes faster and is more likely to maintain a position in the oplog that is close to the real-time writes occurring on the source cluster.

If you anticipate syncing a large data set, or if you plan to pause synchronization for an extended period of time, you might exceed the oplog window. Use the oplogSizeMB setting to increase the size of the oplog on the source cluster.

Monitor oplog Size Needed for Initial Sync

Determine oplog Window

To get the difference in seconds between the first and last entry in the oplog run db.getReplicationInfo(). If you are replicating a sharded cluster, run the command on each shard.

db.getReplicationInfo().timeDiff

The value returned is the minimum oplog window of the cluster. If there are multiple shards, the smallest number is the minimum oplog window.

Determine mongosync Replication Lag

To get the lagTimeSeconds value, run the /progress command. The lag time is the time in seconds between the last event applied by mongosync and time of the current latest event on the source cluster.

It is a measure of how far behind the source cluster mongosync is.

Validate oplog Size

If the lag time approaches the minimum oplog window, make one of the following changes:

Increase the oplog window. Use replSetResizeOplog to set minRetentionHours greater than the current oplog window.
Scale up the mongosync instance. Add CPU or memory to scale up the mongosync node so that it has a higher copy rate.

Note

The oplog window and rate of change for replication lag may vary during synchronization. Repeat these steps during a migration to monitor the progress.

Back

Regular Expressions

Limitations