stores

array

required

Array of objects where each object represents a data store to associate with the federated database instance. The federated database instance store captures documents in the Atlas cluster. Atlas Data Federation can only access data stores defined in the stores object.

stores.[n].name

string

required

Name of the federated database instance store. The databases.[n].collections.[n].dataSources.[n].storeName field references this value as part of mapping configuration.

stores.[n].provider

string

required

Defines where the data is stored. Value must be atlas for a collection in an Atlas cluster.

stores.[n].clusterName

string

required

Name of the Atlas cluster on which the store is based. The cluster must exist in the same project as your federated database instance. The source field on the data partition is the name of the Atlas cluster.

stores.[n].projectId

string

required

Unique identifier of the project that contains the Atlas cluster on which the store is based.

stores.[n].readPreference

boolean

optional

Cluster read preference, which describes how to route read requests to the cluster.

For example:

The following readPreference setting specifies secondary mode and ANALYTICS nodeType.

{
   ...
   "stores": [
     {
       "provider": "atlas",
       "clusterName": <CLUSTER_NAME>,
       "name": <STORE_NAME>,
       "projectId": <PROJECT_ID>,
       "readPreference": {
         "mode": "secondary",
         "tagSets": [
           [
             {
               "name": "nodeType",
               "value": "ANALYTICS"
             }
           ],
           ...
         ]
       }
     }
   ]
 }

stores.[n].readPreference.mode

string

optional

Read preference mode that specifies which replica set member to route the read requests to. Value can be one of the following:

• primary - to route all read requests to the replica set primary

• primaryPreferred - to route all read requests the replica set primary and to secondary members only if primary is unavailable

• secondary - to route all read requests to the secondary members of the replica set

• secondaryPreferred - to route all read requests to the secondary members of the replica set and the primary on sharded clusters only if secondary members are unavailable

• nearest - to route all read requests to random eligible replica set member, irrespective of whether that member is a primary or secondary

If omitted, defaults to local.

stores.[n].readPreference.tagSets

array

optional

Arrays of tag sets or tag specification documents that contain name and value pairs for the replica set member. If specified, Atlas Data Federation routes read requests to replica set member or members that are associated with the specified tags. To learn more, Read Preference Tag Sets.

IMPORTANT: Atlas Data Federation doesn't support tagSets for sharded clusters.

stores.[n].readPreference.maxStalenessSeconds

string

optional

Maximum replication lag, or "staleness", for reads from secondaries. To learn more about maxStalenessSeconds, see Read Preference maxStalenessSeconds.

stores.[n].readConcern

string

optional

Consistency and isolation properties of the data read from an Atlas cluster. To learn more, see Read Concern. The value for the level of consistency and availability can be one of the following:

• local - to return data from an instance with no guarantee that the data has been written to a majority of the replica set members. To learn more, see Read Concern "local".

• available - to return data from an instance with no guarantee that the data has been written to a majority of the replica set members. This can return orphaned documents when reading from a sharded collection. To learn more, see Read Concern "available".

• majority - to return data that has been acknowledged by a majority of the nodes on the Atlas cluster. To learn more, see Read Concern "majority".

• linearizable - to return data that reflects all successful majority-acknowledged writes that completed prior to the start of the read operation. To learn more, see Read Concern "linearizable".

• snapshot - to return majority-committed data as it appears across shards from a specific single point in time in the recent past. To learn more, see Read Concern "snapshot".

databases

array

required

Array of objects where each object represents a database, its collections, and, optionally, any views on the collections. Each database can have multiple collections and views objects.

databases.[n].name

string

required

Name of the database to which Atlas Data Federation maps the data contained in the data store. You can generate databases dynamically by specifying * for the database name. Dynamically generated databases:

• Can exist alongside explicitly defined databases. However, Atlas Data Federation won't dynamically generate databases with names that conflict with explicitly defined databases in the storage configuration.

• Can only be from a single Atlas cluster. Atlas Data Federation won't dynamically generate databases from multiple Atlas clusters or other data stores.

databases.[n].collections

array

required

Array of objects where each object represents a collection and data sources that map to a stores federated database instance store. For dynamically generated databases, you can define only one wildcard (*) collection object in the storage configuration.

databases.[n].collections.[n].name

string

required

Name of the collection to which Atlas Data Federation maps the data contained in each databases.[n].collections.[n].dataSources.[n].storeName. Each object in the array represents the mapping between the collection and an object in the stores array.

You can generate collection names dynamically by specifying * for the collection name and omitting the collection field. To dynamically generate wildcard (*) collections for dynamically generated wilcard (*) databases, specify the databases.[n].collections.[n].dataSources.[n].storeName option and omit the databases.[n].collections.[n].dataSources.[n].database option. Note that for dynamically generated databases, you can define only one wildcard (*) collection object in the storage configuration.

For wildcard (*) collections, you can also define regex patterns using databases.[n].collections.[n].dataSources.[n].collectionRegex field to filter the collections only.

databases.[n].collections.[n].dataSources

array

required

Array of objects where each object represents a stores federated database instance store to map with the collection.

databases.[n].collections.[n].dataSources.[n].storeName

string

required

Name of a federated database instance store to map to the <collection>. Must match the name of an object in the stores array.

databases.[n].collections.[n].dataSources.[n].database

string

required

Name of the database on the Atlas cluster that contains the collection. You must omit this setting to:

• Create a wildcard (*) collection for a wilcard (*) database.

• Glob multiple databases.

databases.[n].collections.[n].dataSources.[n].databaseRegex

string

optional

Regex pattern to use for globbing databases to combine multiple collections. If you specify this option, the federated database instance instance contains a single database with collections from multiple databases. For globbing databases, you must do the following:

• Omit the database field.

• Specify a valid name for the databases.[n].collections.[n].dataSources.[n].collection field.

Suppose you have 2 databases named foo and bar that each have a collection named Sales. You can combine the Sales collection from foo and bar using the databaseRegex option in your storage configuration:

 {
   "databases": [
     {
       "name": "Transactions",
       "collections": [
         {
           "name": "AllSales",
           "dataSources": [
             {
               "storeName": "atlasStore",
               "databaseRegex": ".*",
               "collection": "Sales"
             }
           ]
         }
       ]
     }
   ]
 }
For the preceding :datalakeconf-atlas:`databases` object, Atlas Data Federation
generates the following in your federated database instance:
- A virtual database named ``Transactions``.
- A virtual collection named ``AllSales`` that contains data from
  the collection named ``Sales`` in all the databases whose name
  match the regex pattern specified in the ``databaseRegex``
  option.

If you specify this option, you must specify the name of the collection. You can't specify this option for wildcard collections.

databases.[n].collections.[n].dataSources.[n].collection

string

required

Name of the collection in the Atlas cluster on which the federated database instance store is based. You must omit this setting for:

• Creating a wildcard (*) collection.

• Creating wildcard collection names that match regex patterns.

• Combining multiple collections in a database using regex patterns.

databases.[n].collections.[n].dataSources.[n].collectionRegex

string

Optional for wildcard collections Required for combining collections in a database

Regex pattern to use for creating the wildcard (*) collection or for combining mulitple collections in a database.

To use regex patterns for wildcard (*) collection names, you must do the following:

• Specify wildcard (*) as the value for databases.[n].collections.[n].name.

• Omit databases.[n].collections.[n].dataSources.[n].collection.

If you specify this field for generating wildcard collections, the federated database instance instance only contains collections with names that match the specified regular expression. The collections in the federated database instance storage configuration use their original names in the Atlas cluster.

To use regex patterns for combining multiple collections in a database, you must do the following:

• Specify a name that isn't the wildcard (*) as the value for databases.[n].collections.[n].name.

• Omit databases.[n].collections.[n].dataSources.[n].collection.

If you specify this field for combining multiple collections, the collection in the federated database instance contains data from all the Atlas collections with names that match the specified regular expression. The collection in the federated database instance storage configuration uses the name that you specify as value for databases.[n].collections.[n].name.

To learn more about the regex syntax, see Go programming language.

databases.[n].collections.[n].dataSources.[n].provenanceFieldName

string

required

Name for the field that includes the provenance of the documents in the results. If you specify this setting in the storage configuration, Atlas Data Federation returns the following fields for each document in the result:

You can't configure this setting using the Visual Editor in the Atlas UI.

databases.[n].views

array

required

Array of objects where each object represents an aggregation pipeline on a collection. To learn more about views, see Views.

databases.[n].views.[n].name

string

required

Name of the view.

databases.[n].views.[n].source

string

required

Name of the source collection for the view. If you want to create a view with a $sql stage, you must omit this field as the SQL statement will specify the source collection.

databases.[n].views.[n].pipeline

string

required

Aggregation pipeline stage(s) to apply to the source collection. You can also create views using the $sql stage.