Docs Menu

Docs HomeMongoDB Cluster-to-Cluster Sync

start

On this page

  • Description
  • Requirements
  • Request
  • Response
  • Example: Start a Sync Job
  • Example: Start a Reversible Sync Job
  • Example: Start a Filtered Sync Job
  • Example: Start Sync from Replica Set to Sharded Cluster
  • Behavior

Starts the synchronization between a source and destination cluster.

To use the start endpoint, mongosync must be in the IDLE state.

To set enableUserWriteBlocking, the mongosync user must have a role that includes the setUserWriteBlockMode and bypassWriteBlockingMode ActionTypes.

Note

When using enableUserWriteBlocking, writes are only blocked for users that do not have the bypassWriteBlockingMode ActionType. Users who have this ActionType are able to perform writes.

To set a custom role for the mongosync user:

  1. To create a custom role, use the createRole command:

    db.adminCommand( {
    createRole: "reverseSync",
    privileges: [ {
    resource: { db: "", collection: "" },
    actions: [ "setUserWriteBlockMode", "bypassWriteBlockingMode" ]
    } ],
    roles: []
    } )
  2. To grant the custom role to the mongosync user, use the grantRolesToUser command:

    db.adminCommand( {
    grantRolesToUser: "mongosync-user",
    roles: [ { role: "reverseSync", db: "admin" } ]
    } )

Ensure that you use this configured mongosync user in the connection strings for the cluster0 or cluster1 settings when you start mongosync.

Note

When you configure multiple mongosync instances to sync between sharded clusters, you must send identical API endpoint commands to each mongosync instance.

For more information, see Start Multiple Mongosyncs.

POST /api/v1/start
Parameter
Type
Necessity
Description
source
string
Required
Name of the source cluster.
destination
string
Required
Name of the destination cluster.
buildIndexes
string
Optional

Configures index builds during sync.

Supported Options:

  • beforeDataCopy (the default) causes mongosync to build indexes on the destination cluster. These include both existing indexes and any indexes created during migration on the source cluster.

  • never causes mongosync to skip building unnecessary indexes during sync. This can improve migration performance, especially with index heavy workloads.

    Warning

    Do not manually build indexes while mongosync is performing a migration. Wait until the migration is fully committed.

    For more information on the indexes it does build, see Required Indexes.

/start returns an error when buildIndexes is set to never and reversible is set to true.

If you call /start without specifying the buildIndexes option, mongosync builds indexes on the destination cluster.

New in version 1.3.0.

enableUserWriteBlocking
boolean
Optional

If set to true, blocks writes on the destination cluster while the synchronization is in progress. After the synchronization is committed to the destination cluster, the original source cluster blocks writes and the destination cluster accepts writes.

To reverse sync, the enableUserWriteBlocking field must be set to true.

Default value is false.

includeNamespaces
array
Optional

Filters the databases or collections to include in sync.

If you configure a filter on a source cluster that has multiple databases, mongosync only syncs the databases specified in the filter definition. mongosync will not sync the other existing databases.

If you want to modify the filter to add a newly created database, you have to restart the filtered sync from the beginning.

For more details, see Filtered Sync.

For current limitations, see Filtered Sync.

New in version 1.1.

excludeNamespaces
array
Optional

Filters the databases or collections to exclude from sync.

If you configure a filter on a source cluster that has multiple databases, mongosync only syncs the databases specified in the filter definition. mongosync will not sync the other existing databases.

If you want to modify the filter to add a newly created database, you have to restart the filtered sync from the beginning.

For more details, see Filtered Sync.

For current limitations, see Filtered Sync.

New in version 1.6.

reversible
boolean
Optional

If set to true, enables the sync operation to be reversed.

To reverse sync, the enableUserWriteBlocking field must be set to true.

This option is not supported for the following configurations:

  • Sync from a replica set to a sharded cluster

  • Sync sharded clusters that have different numbers of shards

  • Reversible sync when buildIndexes is set to never.

For more information, see the reverse endpoint.

Default value is false.

sharding
document
Optional

Configures sync between a replica set and sharded cluster. Sync from a replica set to a sharded cluster requires this option.

For more information, see Sharding Parameters.

New in version 1.1.

New in version 1.1.

To sync from a replica set to a sharded cluster, set the sharding option to shard collections on the destination cluster.

The sharding option has the following parameters:

Parameter
Type
Description
createSupportingIndexes
boolean
Optional. Sets whether sync creates a supporting index for the shard key, if none exists. Defaults to false.
shardingEntries
array of documents

Required. Sets the namespace and key of collections to shard during sync.

Collections not included in this array sync to unsharded collections on the destination cluster. If set with an empty array, no collections are sharded.

shardingEntries
.collection
string
Sets the collection to shard.
shardingEntries
.database
string
Sets the database of the collection to shard.
shardingEntries
.shardCollection
document
Sets the shard key to generate on the destination cluster.
shardingEntries
.shardCollection
.key
array

Sets the fields to use for the shard key.

For more information, see Shard Keys.

mongosync throws an error if the sharding option is not set when syncing from a replica set to a sharded cluster. mongosync also throws an error if the sharding option is set with any other configuration.

Field
Type
Description
success
boolean
When the request is successful, this value is true.
error
string
If an error occurred, indicates the name of the error. This field is omitted from the response when success is true.
errorDescription
string
Detailed description of the error that occurred. This field is omitted from the response when success is true.

The following example starts a sync job between cluster0 and cluster1. The source cluster is cluster0 and the destination cluster is cluster1.

curl localhost:27182/api/v1/start -XPOST \
--data '
{
"source": "cluster0",
"destination": "cluster1"
} '
{"success":true}

The following example starts a sync job between cluster0 and cluster1. The source cluster is cluster0 and the destination cluster is cluster1.

The reversible and enableUserWriteBlocking fields allow the sync to be reversed. To reverse the sync direction, see: reverse.

curl localhost:27182/api/v1/start -XPOST \
--data '
{
"source": "cluster0",
"destination": "cluster1",
"reversible": true,
"enableUserWriteBlocking": true
} '
{"success":true}

The following example starts a sync job between cluster0 and cluster1. The source cluster is cluster0 and the destination cluster is cluster1.

cluster0 contains the sales, marketing, and engineering databases.

The sales database contains the EMEA, APAC, and AMER collections.

The includeNamespaces array in this example defines a filter on two of the databases, sales and marketing.

The sales database also filters on the EMEA and APAC collections.

"includeNamespaces" : [
{
"database" : "sales",
"collections": [ "EMEA", "APAC" ]
},
{
"database" : "marketing"
}
]

After you call the /start API with this filter in place, mongosync:

  • Syncs all of the collections in the marketing database

  • Filters out the engineering database

  • Syncs the EMEA and APAC collections from the sales database

  • Filters out the AMER collection

The includeNamespaces option creates a filter. To filter the sync, see: Filtered Sync

curl -X POST "http://localhost:27182/api/v1/start" --data '
{
"source": "cluster0",
"destination": "cluster1",
"includeNamespaces": [
{
"database": "sales",
"collectionsRegex": {
"pattern": "^accounts_.+$",
"options": "i"
}
}, {
"database": "marketing"
}
]
} '
{"success":true}
curl localhost:27182/api/v1/start -XPOST \
--data '
{
"source": "cluster0",
"destination": "cluster1",
"sharding": {
"createSupportingIndexes": true,
"shardingEntries": [
{
"database": "accounts",
"collection": "us_east",
"shardCollection": {
"key": [
{ "location": 1 },
{ "region": 1 },
]
}
}
]
}
} '
{"success":true}

If the start request is successful, mongosync enters the RUNNING state.

New in version 1.1.

When mongosync syncs to a sharded cluster, it pre-splits chunks for sharded collections on the destination cluster. This is supported in the following configurations:

  • Sync from a replica set to a sharded cluster.

  • Sync between sharded clusters that differ in the number of shards.

New in version 1.1.

Sync from a replica set to a sharded cluster requires the sharding option. This option configures how mongosync shards collections.

The sharding.shardEntries array specifies the collections to shard. Collections that are not listed in this array replicate as unsharded.

New in version 1.1.

mongosync syncs indexes from the source cluster to the destination cluster. But, when syncing from a replica set to a sharded cluster, mongosync may require an additional index to support the shard key, which may not exist on the source cluster.

mongosync can create supporting indexes for sharded collections during sync. This is done by setting the sharding.createSupportingIndexes option.

When sharding.createSupportingIndexes is false (the default):

  • Each shard key you provide for the sharding.shardEntries option must have an existing index on the source cluster.

  • One of the indexes used for the shard key must have simple collation if the collection uses any other collation.

  • To use a unique index in the shard key, you must specify its uniqueness when you create the index on the source cluster.

  • Unique indexes on the source cluster that are incompatible with the requested shard key on the destination cluster, such as a unique index on the source that does not contain the requested shard key as a prefix on the destination, can cause mongosync to fail.

When sharding.createSupportingIndexes is true:

  • If the supporting indexes exist on the source cluster, mongosync syncs the indexes to the destination cluster and uses them as shard keys.

  • If the supporting indexes don't exist, mongosync creates them on the destination cluster.

The sharding.createSupportingIndexes option affects all sharded collections.

New in version 1.1.

Collections listed in the sharding.shardEntries array when synced from a replica set to a sharded cluster become sharded collections on the destination cluster.

Renaming a collection (such as with the renameCollection command) on the source cluster after calling start but before mongosync begins to copy the collection can block the collection from sharding on the destination.

Note

Renaming collections to use a different database while syncing from a replica set to a sharded cluster is not supported.

To check whether it is safe to rename collections, call the progress endpoint and check the value of the collectionCopy.estimatedCopiedBytes field in the return document.

  • A value of 0 indicates that mongosync has not started to copy the collection.

    Renaming a collection at this point may result in an unsharded collection on the destination cluster, as the transition to copying can happen before the rename takes effect on the source.

  • A value greater than 0 indicates that mongosync has started the copy. Renaming the collection from this point on does not block its sharding on the destination cluster, even in the event of a crash.

When you call /start with the buildIndexes option set to never, mongosync skips building unnecessary indexes.

Indexes that are always built include:

  • mongosync builds an index on the _id field of every collection it copies.

  • mongosync builds dummy indexes that support the shard key for each sharded collection, which are removed after commit. When buildIndexes is set to never, mongosync retains this index after commit.

←  mongosync API Endpointsprogress →