Cluster-to-Cluster Sync Quickstart

Overview

MongoDB Cluster-to-Cluster Sync connects MongoDB clusters and provides a way to synchronize data between them. The tool that makes the connection is mongosync. For more details, please see the rest of the Cluster-to-Cluster Sync documentation.

mongosync syncs data between two clusters.

• Each cluster can be a replica set or a sharded cluster. If either is a sharded cluster, consult the sharded cluster limitations for mongosync.

• Sharded clusters are not required to have the same number of shards.

• The destination cluster must be the same version or up to two versions ahead of the source cluster. The patch version is not important, so long as they meet the minimum patch version requirements.

• The source cluster can remain active until commit because mongosync syncs writes on the source cluster for the duration of the migration until commit is called.

Follow the instructions below to set up Cluster-to-Cluster Sync, connect your clusters, and synchronize your data.

Setup

Connect the Clusters

mongodb://[username:password@]host1[:port1][,...hostN[:portN]][/[defaultauthdb][?options]]

cluster0:
mongodb://clusterAdmin:superSecret@clusterOne01.fancyCorp.com:20020,clusterOne02.fancyCorp.com:20020,clusterOne03.fancyCorp.com:20020
cluster1:
mongodb://clusterAdmin:superSecret@clusterTwo01.fancyCorp.com:20020,clusterTwo02.fancyCorp.com:20020,clusterTwo03.fancyCorp.com:20020

./bin/mongosync \
      --logPath /var/log/mongosync \
      --cluster0 "mongodb://clusterAdmin:superSecret@clusterOne01.fancyCorp.com:20020,clusterOne02.fancyCorp.com:20020,clusterOne03.fancyCorp.com:20020" \
      --cluster1 "mongodb://clusterAdmin:superSecret@clusterTwo01.fancyCorp.com:20020,clusterTwo02.fancyCorp.com:20020,clusterTwo03.fancyCorp.com:20020"

Synchronize Data Between Clusters

The start endpoint initiates data synchronization. To start syncing, use curl or a similar program to issue the start request:

curl localhost:27182/api/v1/start -XPOST \
--data '
   {
      "source": "cluster0",
      "destination": "cluster1"
   } '

If the start request is successful, mongosync returns { "success": true } and starts to synchronize existing data on the source cluster with the destination cluster. At this point, mongosync enters the RUNNING state and applies subsequent source cluster writes to the destination cluster.

To check the status of the sync, call the progress endpoint:

curl localhost:27182/api/v1/progress -XGET

If the progress response includes the field canCommit: true, the clusters are in sync and the destination cluster continuously replicates data from the source cluster.

The command interface for mongosync is an HTTP server that publishes an HTTP API. To control mongosync, use the API endpoints. The API documentation provides details on using the following endpoints:

Finalize Cutover Process

You can finalize a migration and transfer your application workload from the source to the destination cluster using the mongosync cutover process.

For more information, see Finalize Cutover Process.

One-Time Sync

After initializing data synchronization, call the progress endpoint to see the status of the synchronization process:

curl localhost:27182/api/v1/progress -XGET

For a one time sync, verify that the progress response includes the following field values:

• state: "RUNNING"

• canCommit: true

• lagTimeSeconds is near 0 (Recommended, but not required)

Then, call the commit endpoint to commit the synchronization operation to the destination cluster and stop continuous replication:

curl localhost:27182/api/v1/commit -XPOST --data '{ }'

If the commit request is successful, mongosync returns { "success": true } and enters the COMMITTING state. After the sync is complete, mongosync enters the COMMITTED state and the clusters are no longer in continuous sync.

Data Verification

Before transferring your application load from the source cluster to the destination, check your data to ensure that the sync was successful.

For more information, see Verify Data Transfer.

Synchronization Notes

• The default port for the HTTP API is 27182. Use the --port option with mongosync to configure another port.

• mongosync can swap the source and destination clusters to enable reverse synchronization.

  For more information, see the reverse endpoint.

• The user specified in the mongosync connection string must have the required permissions on the source and destination clusters. The permissions vary depending on your environment and if you want to modify write-blocking settings or use reverse sync.

  To determine the correct the user permissions for your use case, see User Permissions.

• You may need to increase the file descriptor ulimits on the host that is running mongosync. This applies to any UNIX-like system, but macOS in particular has low defaults. See UNIX ulimit settings.

• To estimate the size of oplog needed for initial synchronization, see oplog Sizing.