Docs Menu

Docs HomeMongoDB Cluster-to-Cluster Sync

oplog Sizing

On this page

  • Estimate oplog Size Needed for Initial Sync
  • Determine the oplog Time Span
  • Estimate Copy Rate During Synching
  • Estimate Copy Time
  • Validate oplog Size

The mongosync program uses change streams to synchronize data between source and destination clusters. mongosync does not access the oplog directly, but when a change stream returns events from the past, the events must be within the oplog time range.

The oplog in the source cluster must be large enough to track events that happen during the time it takes to complete the initial sync to the destination cluster.

If you anticipate syncing a large data set, or if you plan to pause synchronization for an extended period of time, increase the size of the replica set oplog in the source cluster.

1

To get the difference in seconds between the first and last entry in the oplog run db.getReplicationInfo(). If you are replicating a sharded cluster, run the command on each shard.

db.getReplicationInfo().timeDiff

The value returned is the minimum oplog window of the cluster. If there are multiple shards, the smallest number is the minimum oplog window.

2

To gather performance data while synching, start the sync process and monitor how fast data is transferred between clusters.

To start syncing, run the /start command.

To get the copy_rate:

  • run the /progress command to get estimatedCopiedBytes_time01

  • wait a second or two

  • run the /progress command to get estimatedCopiedBytes_time02

The copy_rate is:

copy_rate = ( estimatedCopiedBytes_time02 - estimatedCopiedBytes_01) / time_between_requests
3

Estimate the time needed to copy the entire collection. The estimated copy time is:

estimatedCopyTime = estimatedTotalBytes / copy_rate
4

If the estimated time is larger than the minimum oplog window you must cancel synchronization. Before restarting, make one of the following changes:

  • Increase the oplog window. Use replSetResizeOplog to set minRetentionHours greater than the estimated copy time.

  • Scale up the mongosync instance. Add cpu or memory to scale up the mongosync node so that it has a higher copy rate.

Note

The copy rate may vary during synchronization. To monitor progress, repeat the steps to estimate the copy rate and verify that the copy rate stays about the same.

←  Filtered SyncUse mongosync for Disaster Recovery →
Share Feedback
© 2023 MongoDB, Inc.

About

  • Careers
  • Investor Relations
  • Legal Notices
  • Privacy Notices
  • Security Information
  • Trust Center
© 2023 MongoDB, Inc.