> For the complete MongoDB documentation index, see www.mongodb.com/docs/llms.txt

# Filtered Sync

**New in version 1.1**

Mongosync migrates data between two MongoDB clusters. You can use filtered sync to specify which databases and collections the [mongosync](https://www.mongodb.com/docs/mongosync/current/reference/mongosync-binary.md#std-label-c2c-mongosync) utility transfers between the source and destination clusters.

Starting in 1.1, `mongosync` supports inclusion filters to specify which databases and collections to include in sync.  Starting in 1.6, `mongosync` also supports exclusion filters and regular expressions.

- With inclusion filters, `mongosync` syncs matching databases and collections.

- With exclusion filters, `mongosync` syncs all databases and collections, except for those that match the filters.

- With both inclusion and exclusion filters, `mongosync` only syncs databases and collections that match the inclusion filters then excludes any that also match the exclusion filters.

- With no filters, `mongosync` syncs all databases and collections.

## Filter Syntax

The [`start`](https://www.mongodb.com/docs/mongosync/current/reference/api/start.md#std-label-c2c-api-start) API endpoint accepts two fields that configure filtered sync: `includeNamespaces` and `excludeNamespaces`. Each field takes an array of filters that specify the databases and collections to include or exclude from sync.

**Note:**

If the [`start`](https://www.mongodb.com/docs/mongosync/current/reference/api/start.md#std-label-c2c-api-start) call uses both `includeNamespaces` and `excludeNamespaces` parameters, `mongosync` first matches databases and collections from the inclusion filters, then excludes those that also match an exclusion filter.

Filters have the following syntax:

```json
"includeNamespaces": [
    {
       "database": "<database-name>",
       "collections": [
          "<collection-name>"
       ]
       "databaseRegex": {
          "pattern": "<regex-pattern>",
          "options": "<options>"
       },
       "collectionsRegex": {
          "pattern": "<regex-pattern>",
          "options": "<options>"
       }
    }
 ],
 "excludeNamespaces": [
    {
       "database": "<database-name>",
       "collections": [
          "<collection-name>"
       ]
       "databaseRegex": {
          "pattern": "<regex-pattern>",
          "options": "<options>"
       },
       "collectionsRegex": {
          "pattern": "<regex-pattern>",
          "options": "<options>"
       }
    }
 ]

```

To learn more about the `pattern` and `options` regular expression syntax, see [Regular Expressions in Filters.](https://www.mongodb.com/docs/mongosync/current/reference/collection-level-filtering/filter-regex.md#std-label-c2c-filter-regex)

Filters must include either the `database` field or the `databaseRegex` field.

If you need the filter to match specific collections, you can use either the `collections` array to specify collections individually or define a regular expression using the `collectionsRegex` field.

## Configure a Filter

**Important:**

Once you start `mongosync` with a filter in place, the filter cannot be modified. If you do need to create a new filter, see: [Restart `mongosync`.](https://www.mongodb.com/docs/mongosync/current/reference/restart-mongosync.md#std-label-mongosync-restart-migration)

1. Identify Databases and Collections.

   Identify the databases and collections that you want to sync to the destination cluster.

   - When you add a set of databases to the filter, you also exclude any other databases in the cluster.

   - When you specify a collection in your filter, you also exclude any other collections that are in the same database.

2. Create a Filter.

   The [`start`](https://www.mongodb.com/docs/mongosync/current/reference/api/start.md#std-label-c2c-api-start) API accepts two parameters that configure optional filters:

   - The `includeNamespaces` parameter takes an array of filters, which are used to determines which databases and collections `mongosync` should include in the sync.

   - The `excludeNamespaces` parameter takes an array of filters, which are used to determine which databases and collections `mongosync` should exclude from the sync.

   If you don't specify a filter, `mongosync` performs a full cluster sync.

   Create inclusion and/or exclusion filters to identify the databases and collections you want to sync.

   For example, this inclusion filter would configure `mongosync` to only sync collections whose names begin with `accounts_` from the `sales` database, except for the `accounts_old` collection:

   ```json
   "includeNamespaces": [
      {
         "database": "sales",
         "collectionsRegex": {
            "pattern": "^accounts_.+?$",
            "options": "ms"
         }
   ],
   "excludeNamespaces": [
       {
           "database": "sales",
           "collections": [
               "accounts_old"
           ]
       }
   ]

   ```

   For more information on filters, see [Filter Syntax.](https://www.mongodb.com/docs/mongosync/current/reference/collection-level-filtering.md#std-label-c2c-filter-syntax)

3. Use the Filter.

   To use the filter, attach the filter json when you make the [/start](https://www.mongodb.com/docs/mongosync/current/reference/api/start.md#std-label-c2c-api-start) API call to begin syncing.

   ```bash
   curl -X POST "http://localhost:27182/api/v1/start" --data '
      {
         "source": "cluster0",
         "destination": "cluster1",
         "includeNamespaces": [
            {
                "database": "sales",
                "collectionsRegex": {
                   "pattern": "^accounts_.+$",
                   "options": "i"
                }
            }, {
               "database": "marketing"
            }
         ]
      } '

   ```

For an example configuration, see: [Start `mongosync` with a Filter.](https://www.mongodb.com/docs/mongosync/current/reference/collection-level-filtering.md#std-label-c2c-example-start-with-filter)

## Replace an Existing Filter

You cannot update an existing filter. You must stop the ongoing sync process, prepare the destination cluster, and restart `mongosync` with a new filter. To restart `mongosync`, see [Restart `mongosync`.](https://www.mongodb.com/docs/mongosync/current/reference/restart-mongosync.md#std-label-mongosync-restart-migration)

## Adding and Renaming Collections

You can, with some restrictions, add or [`rename a collection`](https://www.mongodb.com/docs/manual/reference/method/db.collection.renameCollection.md#mongodb-method-db.collection.renameCollection) during a filtered sync.

**Warning:**

If your renaming operation violates the renaming restrictions, `mongosync` stops syncing and reports an error.

To clean up and restart after an error, follow the steps to [replace an existing filter.](https://www.mongodb.com/docs/mongosync/current/reference/collection-level-filtering.md#std-label-c2c-change-filter)

### Adding and Renaming Within a Single Database

You can add new collections or rename an existing collection if the entire database is part of the filter.

You can also rename a collection if the old name and the new name are both specified in the filter.

See the [renaming examples.](https://www.mongodb.com/docs/mongosync/current/reference/collection-level-filtering.md#std-label-c2c-example-rename-while-syncing)

### Renaming Across Different Databases

You can only rename a collection across databases if the entire target database is part of a filter. If the filter specifies individual collections in the target database, renaming across databases does not work.

See the [renaming examples.](https://www.mongodb.com/docs/mongosync/current/reference/collection-level-filtering.md#std-label-c2c-example-rename-while-syncing)

## Filtering with mapReduce and $out

To use the [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) aggregation stage or the [`mapReduce`](https://www.mongodb.com/docs/manual/reference/command/mapReduce.md#mongodb-dbcommand-dbcmd.mapReduce) command (when set to create or replace a collection) with filtering, you must filter the whole database and not just the specified collection.

For example, consider this aggregation pipeline:

```javascript
use library
db.books.aggregate( [
   { $group : { _id : "$author", titles: { $push: "$title" } } },
   { $out : "authors" }
] )

```

The `$out` stage creates the `authors` collection in the `library` database. If you want to sync the `authors` collection, you must specify the entire `library` database in your filter. The filter will not work if you only specify the `authors` collection.

This filter works:

```json
"includeNamespaces": [
  {
    "database": "library"
  }
]

```

This filter does not work:

```json
"includeNamespaces": [
  {
    "database": "library",
    "collections": [ "authors", "books" ]  // DOES NOT WORK WITH $OUT
  }
]

```

## Limitations

- Filtering is not supported with [reversible sync.](https://www.mongodb.com/docs/mongosync/current/reference/api/reverse.md#std-label-c2c-api-reverse)

- The destination cluster must not contain user data prior to starting unless you set the [preExistingDestinationData](https://www.mongodb.com/docs/mongosync/current/reference/api/start.md#std-label-c2c-pre-existing-dest-data) parameter to `true` when you call `/start`.

- The destination cluster must not contain the `__mdb_internal_mongosync` system database prior to starting.

- You cannot modify a filter that is in use. To create a new filter, see: [Replace an Existing Filter.](https://www.mongodb.com/docs/mongosync/current/reference/collection-level-filtering.md#std-label-c2c-change-filter)

- You can only rename collections in certain situations. For more details see: [Adding and Renaming Collections.](https://www.mongodb.com/docs/mongosync/current/reference/collection-level-filtering.md#std-label-c2c-filter-renaming-collections)

- If a filter includes a [view](https://www.mongodb.com/docs/manual/core/views.md#std-label-views-landing-page) but not the base collection, only the view metadata syncs to the destination cluster. To include the view documents, you must also sync the base collection.

- You cannot specify system collections or system databases in a filter.

- To use the [`$out`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/out.md#mongodb-pipeline-pipe.-out) aggregation stage or the [`mapReduce`](https://www.mongodb.com/docs/manual/reference/command/mapReduce.md#mongodb-dbcommand-dbcmd.mapReduce) command (when set to create or replace a collection) with filtering, you must configure the filter to use the entire database. You cannot limit the filter to collections within the database.

  For more information, see [Filtering with mapReduce and $out.](https://www.mongodb.com/docs/mongosync/current/reference/collection-level-filtering.md#std-label-c2c-filter-with-out)

## Examples

### Start `mongosync` with a Filter

The following example starts a sync job between the source cluster `cluster0` and the destination cluster `cluster1`.

`cluster0` contains the `sales`, `marketing`, and `engineering` databases.

The `sales` database contains the `EMEA`, `APAC`, and `AMER` collections.

The `includeNamespaces` array in this example defines a filter on two of the databases, `sales` and `marketing`.

The `sales` database also filters on the `EMEA` and `APAC` collections.

```json
"includeNamespaces" : [
      {
          "database" : "sales",
          "collections": [ "EMEA", "APAC" ]
      },
      {
          "database" : "marketing"
      }
   ]

```

After you call the `/start` API with this filter in place, `mongosync`:

- Syncs all of the collections in the `marketing` database

- Filters out the `engineering` database

- Syncs the `EMEA` and `APAC` collections from the `sales` database

- Filters out the `AMER` collection

### Adding and Renaming Collections While Syncing

The following example starts a sync job between the source cluster `cluster0` and the destination cluster `cluster1`.

`cluster0` contains the `students`, `staff`, and `prospects` databases.

- The `students` database contains the `undergrad`, `graduate`, and `adjuncts` collections.

- The `staff` database contains the `employees` and `contractors` collections.

The `includeNamespaces` array in this example defines a filter on two of the databases:

```json
{
   "source": "cluster0",
   "destination": "cluster1",
   "includeNamespaces":
      [
         { "database" : "students", "collections": ["undergrad", "graduate", "adjuncts"] },
         { "database" : "staff" }
      ]
}

```

With this filter in place, `mongosync` syncs:

- The entire `staff` database

- The `undergrad`, `graduate`, and `adjuncts` collections in the `students` database

`mongosync` does not sync any information from the `prospects` database.

#### Adding a Collection

`mongosync` syncs the entire `staff` database. If you add new collections to the `staff` database, `mongosync` syncs them too.

`mongosync` does not sync new collections that are added to the `students` database unless the collection is a part of the filter.

For example, `mongosync` does not sync the new collection if you add the `postdocs` collection to the `students` database. If you add the `adjuncts` collection, `mongosync` syncs it since `adjuncts` is part of the filter.

#### Renaming a Collection

You can rename any collection in the `staff` database.

```javascript
// This code works 
use admin
db.runCommand( { renameCollection: "staff.employees", to: "staff.salaried" } )

```

You can only rename a collection within the `students` database if the new and old names are both in the filter. If either of the names is not in the filter, `monogsync` reports an error and exits.

```javascript
// This code works 
use admin
db.runCommand( { renameCollection: "students.graduate", to: "students.adjuncts" } )

```

If a collection is specified in the filter, you can drop it, but you cannot rename it to remove it from the filter.

```javascript
// This code produces an error and mongosync stops syncing 
use admin
db.runCommand( { renameCollection: "students.graduate", to: "students.notAFilteredCollection" } )

```

When the whole target database is included in the filter, you can rename collections to add them to the filter:

- Source collection is specified in the filter

  ```javascript
  use admin
  db.runCommand( { renameCollection: "students.adjuncts", to: "staff.adjuncts" } )

  ```

- Source collection is not specified in the filter

  ```javascript
  use admin
  db.runCommand( { renameCollection: "prospects.current", to: "staff.newHires" } )

  ```

You can also rename collections in the source database when the whole target database is in the filter:

```javascript
use admin
db.runCommand( { renameCollection: "staff.employees", to: "staff.onPayroll" } )

```

**Important:**

If you anticipate renaming collections, consider adding the entire database to the filter rather than specifying individual collections.