Docs Home → MongoDB Cluster-to-Cluster Sync
oplog Sizing
On this page
The mongosync program uses change streams to synchronize data between source and destination
clusters. mongosync does not access the oplog directly,
but when a change stream returns events from the past, the events must
be within the oplog time range.
The oplog in the source cluster must be large enough to track events that happen during the time it takes to complete the initial sync to the destination cluster.
If you anticipate syncing a large data set, or if you plan to pause
synchronization for an extended period of time, increase the size of the
replica set oplog in the source cluster.
Estimate oplog Size Needed for Initial Sync
Determine the oplog Time Span
To get the difference in seconds between the first and last entry
in the oplog run db.getReplicationInfo(). If you
are replicating a sharded cluster, run the command on each shard.
db.getReplicationInfo().timeDiff
The value returned is the minimum oplog window of the
cluster. If there are multiple shards, the smallest number is the
minimum oplog window.
Estimate Copy Rate During Synching
To gather performance data while synching, start the sync process and monitor how fast data is transferred between clusters.
To start syncing, run the /start command.
To get the copy_rate:
run the /progress command to get
estimatedCopiedBytes_time01wait a second or two
run the /progress command to get
estimatedCopiedBytes_time02
The copy_rate is:
copy_rate = ( estimatedCopiedBytes_time02 - estimatedCopiedBytes_01) / time_between_requests
Estimate Copy Time
Estimate the time needed to copy the entire collection. The estimated copy time is:
estimatedCopyTime = estimatedTotalBytes / copy_rate
Validate oplog Size
If the estimated time is larger than the minimum oplog window you must cancel synchronization. Before restarting, make one of the following changes:
Increase the oplog window. Use
replSetResizeOplogto setminRetentionHoursgreater than the estimated copy time.Scale up the
mongosyncinstance. Add cpu or memory to scale up themongosyncnode so that it has a higher copy rate.
Note
The copy rate may vary during synchronization. To monitor progress, repeat the steps to estimate the copy rate and verify that the copy rate stays about the same.