> For the complete MongoDB documentation index, see www.mongodb.com/docs/llms.txt # Configuration File Formats for Data Stores Atlas Data Federation supports various data store providers as federated database instance stores. You must define mappings in your federated database instance to data stores to run queries against your data. **Important:** Information in your storage configuration is visible at MongoDB and stored as operational data to improve the performance of Atlas Data Federation. Avoid using personally identifiable information (PII) in your configurations. Select your provider from the dropdown to configure your data store: ## AWS S3 Bucket ### Overview Atlas Data Federation supports AWS (Amazon Web Services) S3 (Simple Storage Service) buckets as federated database instance stores. You must define mappings in your federated database instance to your AWS (Amazon Web Services) S3 (Simple Storage Service) bucket to run queries against your data. ### Configuration File Format To define a federated database instance store for an AWS (Amazon Web Services) S3 (Simple Storage Service) bucket, you can specify the configuration parameters in JSON (Javascript Object Notation). The configuration contains the AWS (Amazon Web Services) data store and maps it to virtual collections that you can query. The JSON (Javascript Object Notation) configuration for data in AWS (Amazon Web Services) S3 (Simple Storage Service) bucket uses the following fields: ```json { "stores" : [ { "name" : "", "provider": "", "region" : "", "bucket" : "", "additionalStorageClasses" : [""], "prefix" : "", "includeTags": , "delimiter": "", "public": } ], "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "path" : "", "defaultFormat" : "", "provenanceFieldName": "", "omitAttributes": true | false } ] } ], "maxWildcardCollections" : , "views" : [ { "name" : "", "source" : "", "pipeline" : "" } ] } ] } ``` The JSON (Javascript Object Notation) configuration for AWS (Amazon Web Services) S3 (Simple Storage Service) data store contains two top-level objects: [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-stores-reference) and [`databases`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-aws-databases-reference) #### `stores` The `stores` object defines each data store associated with the federated database instance. This store captures files in an AWS (Amazon Web Services) S3 (Simple Storage Service) bucket. Data Federation can only access data stores defined within the `stores` object. The `stores` object contains the following fields: ```json "stores" : [ { "name" : "", "provider" : "", "region" : "", "bucket" : "", "additionalStorageClasses" : [""], "prefix" : "", "delimiter" : "", "includeTags": , "public": } ] ``` The following table describes the fields in the `stores` object: | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | required | Array of objects where each object represents a data store to associate with the federated database instance. The store captures files in an AWS (Amazon Web Services) S3 (Simple Storage Service) bucket. Atlas Data Federation can only access data stores defined in the `stores` object. | | | string | required | Name of the federated database instance store. The [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.storeName) field references this value as part of mapping configuration. | | | string | required | Defines where the data is stored. Value must be `s3` for an AWS (Amazon Web Services) S3 (Simple Storage Service) bucket. | | | string | required | Name of the AWS (Amazon Web Services) region in which the AWS (Amazon Web Services) S3 (Simple Storage Service) bucket is hosted. For a list of valid region names, see [Amazon Web Services (AWS).](https://www.mongodb.com/docs/atlas/reference/amazon-aws/#amazon-aws) | | | string | required | Name of the AWS (Amazon Web Services) S3 bucket. Must exactly match the name of an AWS (Amazon Web Services) S3 (Simple Storage Service) bucket which Atlas Data Federation can access with the configured AWS (Amazon Web Services) IAM credentials. | | | array | optional | Array of AWS (Amazon Web Services) S3 (Simple Storage Service) [storage classes](https://aws.amazon.com/s3/storage-classes/). Atlas Data Federation will include the files in these storage classes in the query results. Valid values are: `INTELLIGENT_TIERING` to include files in the [Intelligent Tiering](https://aws.amazon.com/s3/storage-classes/#Unknown_or_changing_access) storage class; `STANDARD_IA` to include files in the [Standard-Infrequent Access](https://aws.amazon.com/s3/storage-classes/#Infrequent_access) storage class IMPORTANT: Files in the [Standard](https://aws.amazon.com/s3/storage-classes/#General_purpose) storage class are supported by default. | | | string | optional | Adds a prefix to search paths for files in the AWS (Amazon Web Services) S3 (Simple Storage Service) bucket. Atlas Data Federation prepends the value of `prefix` to the [`databases.[n].collections.[n].dataSources.[n].path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) to create the full path for files to ingest. If omitted, Atlas Data Federation searches all files from the root of the AWS (Amazon Web Services) S3 (Simple Storage Service) bucket. | | | string | optional | Sets a delimiter that separates path segments in the federated database instance store. Data Federation uses the delimiter to efficiently traverse AWS (Amazon Web Services) S3 (Simple Storage Service) buckets with a hierarchical directory structure. You can specify any character supported by the AWS (Amazon Web Services) S3 (Simple Storage Service) [object keys](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html) as the delimiter. For example, you can specify an underscore (`_`) or a plus sign (`+`) or multiple characters such as double underscores (`__`) as the delimiter. If omitted, defaults to `"/"`. | | | boolean | optional | Determines whether to use AWS (Amazon Web Services) S3 (Simple Storage Service) tags on the files in the given path as additional partition attributes. Valid values are `true` and `false`. If omitted, defaults to `false`. If set to `true`, Atlas Data Federation does the following: Adds the AWS (Amazon Web Services) S3 (Simple Storage Service) tags as additional partition attributes.; Adds new top level BSON elements that associate each tag to each document for the tagged files. WARNING: If set to `true`, Atlas Data Federation processes the files for additional partition attributes by making extra calls to AWS (Amazon Web Services) S3 (Simple Storage Service) to get the tags. This behavior might impact performance. | | | boolean | optional | Specifies whether the bucket is public. If set to `true`, Atlas Data Federation doesn't use the configured AWS (Amazon Web Services) IAM (Identity and Access Management) role to access the AWS (Amazon Web Services) S3 (Simple Storage Service) bucket. If set to `false`, the configured AWS (Amazon Web Services) IAM (Identity and Access Management) must include permissions to access the AWS (Amazon Web Services) S3 (Simple Storage Service) bucket, even if that bucket is public. If omitted, defaults to `false`. | #### `databases` The `databases` object defines the mapping between each federated database instance store defined in `stores` and MongoDB collections in the databases. The `database` object contains the following fields: ```json "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "defaultFormat" : "", "path" : "", "provenanceFieldName": "", "omitAttributes": } ] } ], "maxWildcardCollections" : , "views" : [ { "name" : "", "source" : "", "pipeline" : "" } ] } ] ``` The following table describes the fields in the `database` object: | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | required | Array of objects where each object represents a database, its collections, and, optionally, any [views](https://www.mongodb.com/docs/manual/core/views.md#std-label-views-landing-page) on the collections. Each database can have multiple `collections` and `views` objects. | | | string | required | Name of the database to which Atlas Data Federation maps the data contained in the data store. | | | array | required | Array of objects where each object represents a collection and data sources that map to a [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores) federated database instance store. | | | string | required | Name of the collection to which Atlas Data Federation maps the data contained in each [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.storeName). Each object in the array represents the mapping between the collection and an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores) array. You can generate collection names dynamically from file paths by specifying `*` for the collection name and the `collectionName()` function in the [`path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) field. See [Generate Dynamic Collection Names from File Path](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-path-generate-collection) for examples. | | | array | required | Array of objects where each object represents a [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores) federated database instance store to map with the collection. | | | string | required | Name of a federated database instance store to map to the ``. Must match the [`name`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores.-n-.name) of an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores) array. | | | string | required | Controls how Atlas Data Federation searches for and parses files in the [`storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.storeName) before mapping them to the ``. federated database instance prepends the [`stores.[n].prefix`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores.-n-.prefix) to the `path` to build the full path to search within. Specify `/` to capture all files and folders from the `prefix` path. For example, consider an S3 bucket `metrics` with the following structure: ` metrics |--hardware |--software |--computed` A `path` of `/` directs Atlas Data Federation to search all files and folders in the `metrics` bucket. A `path` of `/hardware` directs Atlas Data Federation to search only that path for files to ingest. If the [`prefix`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores.-n-.prefix) is `software`, Atlas Data Federation searches for files only in the path `/software/computed`. Appending the `*` wildcard character to the path directs Atlas Data Federation to include all files and folders from that point in the path. For example, `/software/computed*` would match files like `/software/computed-detailed`, `/software/computedArchive`, and `/software/computed/errors`. [`path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) supports additional syntax for parsing filenames, including: Generating document fields from filenames.; Using regular expressions to control field generation.; Setting boundaries for bucketing filenames by timestamp. See [Define Path Syntax for Data Stores](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-path-syntax) for more information. When specifying the [`path`:](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) Specify the data type for the partition attribute.; Ensure that the partition attribute type matches the data type to parse.; Use the delimiter specified in [`delimiter`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores.-n-.delimiter) When specifying attributes of the same type, do any of the following: Add a constant separator between the attributes.; Use regular expressions to describe the search pattern. To learn more, see [Unsupported Parsing Functions.](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-unsupported-parsing-funcs) Default format that Data Federation assumes if it encounters a file without an extension while searching the `databases.[n].collections.[n].dataSources.[n].storeName`. The following values are valid for the `defaultFormat` field: `.json`, `.json.gz`, `.bson`, `.bson.gz`, `.avro,` `.avro.gz`, `.orc`, `.tsv`, `.tsv.gz`, `.csv`, `.csv.gz`, `.parquet` IMPORTANT: If your file format is `CSV` or `TSV`, you must include a header row in your data. See [CSV and TSV](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/data-formats/csv-tsv-data-files.md#std-label-adf-csv-tsv-data) for more information. If omitted, Data Federation attempts to detect the file type by processing a few bytes of the file. For more information, see [Supported Data Formats.](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/supported-data-formats.md#std-label-adf-data-formats) | | | string | optional | Name for the field that includes the provenance of the documents in the results. If you specify this setting in the storage configuration, Atlas Data Federation returns the following fields for each document in the result: Field NameDescription`provider`Provider ([`stores.[n].provider`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores.-n-.provider)) in the federated database instance storage configuration`region`AWS (Amazon Web Services) region ([`stores.[n].region`)](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores.-n-.region)`bucket`Name of the AWS (Amazon Web Services) S3 (Simple Storage Service) bucket ([`stores.[n].bucket`)](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores.-n-.bucket)`key`Path ([`databases.[n].collections.[n].dataSources.[n].path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path)) to the document`lastModified`Date and time the document was last modified. You can't configure this setting using the Visual Editor in the Atlas UI. | | | boolean | optional | Flag that specifies whether to omit the attributes (key and value pairs) that Atlas Data Federation adds to documents in the collection. You can specify one of the following values: `false` - to add the attributes; `true` - to omit the attributes If omitted, defaults to `false` and Atlas Data Federation adds the attributes. For example: Consider a file named `/employees/949-555-0195.json` for which you configure the [`path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) `/employees/{phone string}`. Atlas Data Federation adds the attribute `phone: 949-555-0195` to documents in this file if `omitAttributes` is `false`, regardless of whether the key-value pair already exists in the document. If you set `omitAttributes` to `true`, Atlas Data Federation doesn't add the attribute to the document in the virtual collection. Atlas Data Federation adds partition attributes to documents even when a field with the same name already exists in the document, which creates duplicate field names. MongoDB doesn't support [duplicate field names](https://www.mongodb.com/docs/manual/reference/limits/#mongodb-does-not-support-duplicate-field-names). To avoid duplicate field names, set `omitAttributes` to `true` or use partition attribute names that don't appear in your documents. | | | integer | optional | *Optional.* Maximum number of wildcard `*` collections in the database. Each wildcard collection can have only one data source. Value can be between `1` and `1000`, inclusive. If omitted, defaults to `100`. | | | array | optional | Array of objects where each object represents an [aggregation pipeline](https://www.mongodb.com/docs/manual/core/aggregation-pipeline.md#std-label-aggregation-pipeline) on a collection. To learn more about views, see [Views.](https://www.mongodb.com/docs/manual/core/views.md#std-label-views-landing-page) | | | string | required | Name of the view. | | | string | required | Name of the source collection for the view. If you want to create a view with a [$sql](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/sql.md#std-label-adf-sql-stage) stage, you must omit this field as the SQL statement specifies the source collection. | | | array | optional | Array of [Aggregation pipeline stage(s)](https://www.mongodb.com/docs/manual/reference/mql/aggregation-stages.md#std-label-aggregation-pipeline-operator-reference) to apply to the [`source`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.views.-n-.source) collection. You can also create views using the [$sql](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/sql.md#std-label-adf-sql-stage) stage. | ### Example Configuration for S3 Data Store **Example:** Consider a S3 bucket `datacenter-alpha` containing data collected from a datacenter: ```none |--metrics |--hardware ``` The `/metrics/hardware` path stores JSON files with metrics derived from the datacenter hardware, where each filename is the UNIX timestamp in milliseconds of the 24 hour period covered by that file: ```none /hardware/1564671291998.json ``` The following configuration: - Defines a federated database instance store on the `datacenter-alpha` S3 bucket in the `us-east-1` AWS region. The federated database instance store is specifically restricted to only datafiles in the `metrics` folder path. - Maps files from the `hardware` folder to a MongoDB database `datacenter-alpha-metrics` and collection `hardware`. The configuration mapping includes parsing logic for capturing the timestamp implied in the filename. ```json { "stores" : [ { "name" : "datacenter-alpha", "provider" : "s3", "region" : "us-east-1", "bucket" : "datacenter-alpha", "additionalStorageClasses" : [ "STANDARD_IA" ], "prefix" : "/metrics", "delimiter" : "/" } ], "databases" : [ { "name" : "datacenter-alpha-metrics", "collections" : [ { "name" : "hardware", "dataSources" : [ { "storeName" : "datacenter-alpha", "path" : "/hardware/{date date}" } ] } ] } ] } ``` Atlas Data Federation parses the S3 bucket `datacenter-alpha` and processes all files under `/metrics/hardware/`. The `collections` uses the [path parsing syntax](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-path-syntax) to map the filename to the `date` field, which is an ISO-8601 date, in each document. If a matching `date` field does not exist in a document, it will be added. Users connected to the federated database instance can use the MongoDB Query Language and supported aggregations to analyze data in the AWS (Amazon Web Services) S3 (Simple Storage Service) bucket through the `datacenter-alpha-metrics.hardware` collection. **See also:** - [Querying Data in S3](https://www.mongodb.com/docs/atlas/data-federation/query/query-federated-database.md#std-label-query-s3) - [Query a Federated Database Instance](https://www.mongodb.com/docs/atlas/data-federation/query/query-federated-database.md#std-label-query-fdi) ## Azure Blob Storage ### Overview Atlas Data Federation supports Azure (Microsoft Azure) Blob Storage containers as federated database instance stores. You must define mappings in your federated database instance to your Azure Blob Storage containers to run queries against your data. **Note:** While we refer to blobs as files and delimiter-separated prefixes as directories in this page, these blob storage services are not actually file systems and don't have the same behaviors in all cases as files on a hard drive. ### Configuration File Format To define a federated database instance store for an Azure Blob Storage container, you can specify the configuration parameters in JSON format. The configuration contains the Azure Blob Storage data store and maps it to virtual collections that you can query. The JSON configuration for data in Azure Blob Storage containers uses the following fields: ```json { "stores" : [ { "name" : "", "provider": "", "region" : "", "serviceURL" : "", "containerName" : "", "delimiter" : "", "prefix": "", "public": } ], "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "path" : "", "defaultFormat" : "", "provenanceFieldName": "", "omitAttributes": } ] } ], "maxWildcardCollections" : , "views" : [ { "name" : "", "source" : "", "pipeline" : "" } ] } ] } ``` The JSON configuration for an Azure Blob Storage contains two top-level objects: [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-azure-stores-reference) and [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-azure-databases-reference) #### `stores` The `stores` object defines each data store associated with the federated database instance. The federated database instance store captures files in a Azure Blob Storage container. Data Federation can only access data stores defined in the `stores` object. The `stores` object contains the following fields: ```json "stores" : [ { "name" : "", "provider" : "", "region" : "", "serviceURL" : "", "containerName" : "", "delimiter": "", "public": } ] ``` The following table describes the fields in the stores object: | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | required | Array of objects where each object represents a data store to associate with the federated database instance. The federated database instance store captures: Files in an Azure Blob Storage container; Documents in an Atlas cluster; Files stored at public URL (Uniform Resource Locator)s. Atlas Data Federation can only access data stores defined in the `stores` object. | | | string | required | Name of the federated database instance store. The [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.databases.-n-.collections.-n-.dataSources.-n-.storeName) field references this value as part of mapping configuration. | | | string | required | Defines where the data is stored. Value must be `azure` for an Azure Blob Storage container. | | | string | required | Name of the Azure (Microsoft Azure) region in which the data is stored. | | | string | required | URL of the Azure Blob Storage account that contains your blob containers. The `serviceURL` must be in the following format: ` https://.blob.core.windows.net/` where `storage-account-name` is the name of your Azure Blob Storage account. | | | string | required | Name of the Azure Blob Storage container that contains the files. | | | string | optional | Prefix Atlas Data Federation applies when searching for files in the Azure Blob Storage. For example, consider an an Azure Blob Storage container `metrics` with the following structure: ` metrics |--hardware |--software |--computed` The federated database instance store prepends the value of `prefix` to the [`databases.[n].collections.[n].dataSources.[n].path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.databases.-n-.collections.-n-.dataSources.-n-.path) to create the full path for files to ingest. Setting the `prefix` to `/software` restricts any [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.databases) objects using the federated database instance store to only subpaths `/software`. If omitted, Atlas Data Federation searches all files from the root of the Azure Blob Storage container. | | | string | optional | The delimiter that separates [`databases.[n].collections.[n].dataSources.[n].path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.databases.-n-.collections.-n-.dataSources.-n-.path) segments in the federated database instance store. Data Federation uses the delimiter to efficiently traverse Azure Blob Storage containers with a hierarchical directory structure. If omitted, defaults to `"/"`. | | | boolean | optional | Specifies whether the Azure Blob Storage container is public. If set to `true`, Atlas Data Federation doesn't use the configured Azure (Microsoft Azure) Service Principal to access your Azure Blob Storage. If set to `false`, the configured Service Principal must include permissions to access the blob container, even if that blob container is public. If omitted, defaults to `false`. | #### `databases` The `databases` object defines the mapping between each federated database instance store defined in `stores` and MongoDB collections in the databases. The `databases` object contains the following fields: ```json "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "defaultFormat" : "", "path" : "", "provenanceFieldName": "", "omitAttributes": } ] } ], "maxWildcardCollections" : , "views" : [ { "name" : "", "source" : "", "pipeline" : "" } ] } ] ``` The following table describes the fields in the databases object: | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | required | Array of objects where each object represents a database, its collections, and, optionally, any [views](https://www.mongodb.com/docs/manual/core/views.md#std-label-views-landing-page) on the collections. Each database can have multiple `collections` and `views` objects. | | | string | required | Name of the database to which Atlas Data Federation maps the data contained in the data store. | | | array | required | Array of objects where each object represents a collection and data sources that map to a [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.stores) federated database instance store. | | | string | required | Name of the collection to which Atlas Data Federation maps the data contained in each [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.databases.-n-.collections.-n-.dataSources.-n-.storeName). Each object in the array represents the mapping between the collection and an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.stores) array. You can generate collection names dynamically from file paths by specifying `*` for the collection name and the `collectionName()` function in the [`path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.databases.-n-.collections.-n-.dataSources.-n-.path) field. | | | array | required | Array of objects where each object represents a [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.stores) federated database instance store to map with the collection. | | | string | required | Name of a federated database instance store to map to the ``. Must match the [`name`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.stores.-n-.name) of an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.stores) array. | | | string | required | Controls how Atlas Data Federation searches for and parses files in the [`storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.databases.-n-.collections.-n-.dataSources.-n-.storeName) before mapping them to the ``. Atlas Data Federation prepends the [`stores.[n].prefix`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.stores.-n-.prefix) to the `path` to build the full path to search within. Specify `/` to capture all files and directories from the `prefix` path. For example, consider an Azure Blob Storage container `metrics` with the following structure: ` metrics |--hardware |--software |--computed` A `path` of `/` directs Atlas Data Federation to search all files and directories in the `metrics` bucket. A `path` of `/hardware` directs Atlas Data Federation to search only that path for files to ingest. If the [`prefix`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.stores.-n-.prefix) is `software`, Atlas Data Federation searches for files only in the path `/software/computed`. Appending the `*` wildcard character to the path directs Atlas Data Federation to include all files and directories from that point in the path. For example, `/software/computed*` would match files like `/software/computed-detailed`, `/software/computedArchive`, and `/software/computed/errors`. [`path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.databases.-n-.collections.-n-.dataSources.-n-.path) supports additional syntax for parsing filenames, including: Generating document fields from filenames.; Using regular expressions to control field generation.; Setting boundaries for bucketing filenames by timestamp. See [Define Path Syntax for Data Stores](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-path-syntax) for more information. When specifying the [`path`:](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) Specify the data type for the partition attribute.; Ensure that the partition attribute type matches the data type to parse.; Use the delimiter specified in [`delimiter`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores.-n-.delimiter) When specifying attributes of the same type, do any of the following: Add a constant separator between the attributes.; Use regular expressions to describe the search pattern. To learn more, see [Unsupported Parsing Functions.](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-unsupported-parsing-funcs) | | | string | optional | Default format that Data Federation assumes if it encounters a file without an extension while searching the [`databases.[n].collections.[n].dataSources.[n].storeName`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.databases.-n-.collections.-n-.dataSources.-n-.storeName) The following values are valid for the defaultFormat field: `.json` `.json.gz` `.bson` `.bson.gz` `.avro` `.avro.gz` `.orc` `.tsv` `.tsv.gz` `.csv` `.csv.gz` `.parquet` For more information, see [Supported Data Formats](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/supported-data-formats.md#std-label-adf-data-formats) | | | string | required | Name for the field that includes the provenance of the documents in the results. If you specify this setting in the storage configuration, Atlas Data Federation returns the following fields for each document in the result: Field NameDescription`provider`Provider ([`stores.[n].provider`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.stores.-n-.provider)) in the federated database instance storage configuration.`region`Azure (Microsoft Azure) region ([`stores.[n].region`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.stores.-n-.region)).`serviceURL`URL (Uniform Resource Locator) of the Azure Blob Storage account that contains your blob containers ([`stores.[n].serviceURL`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.stores.-n-.serviceURL)).`containerName`Name of the Azure Blob Storage container ([`stores.[n].containerName`)](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.stores.-n-.containerName)`key`Path ([`databases.[n].collections.[n].dataSources.[n].path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.databases.-n-.collections.-n-.dataSources.-n-.path)) to the file.IMPORTANT: You can't configure this setting using the Visual Editor in the Atlas UI. | | | boolean | required | Flag that specifies whether to omit the attributes (key and value pairs) that Atlas Data Federation adds to the collection. You can specify one of the following values: `false` - to add the attributes; `true` - to omit the attributes If omitted, defaults to `false` and Atlas Data Federation adds the attributes. **For example:** Consider a file named `/employees/949-555-0195.json` for which you configure the [`path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) `/employees/{phone string}`. Atlas Data Federation adds the attribute `phone: 949-555-0195` to the document if you set `omitAttributes` to `false`, regardless of whether the key-value pair already exists in the document. If you set `omitAttributes` to `true`, Atlas Data Federation doesn't add the attribute to the document in the virtual collection. Atlas Data Federation adds partition attributes to documents even when a field with the same name already exists in the document, which creates duplicate field names. MongoDB doesn't support [duplicate field names](https://www.mongodb.com/docs/manual/reference/limits/#mongodb-does-not-support-duplicate-field-names). To avoid duplicate field names, set `omitAttributes` to `true` or use partition attribute names that don't appear in your documents. | | | integer | optional | Maximum number of wildcard * collections in the database. Each wildcard collection can have only one data source. Value can be between 1 and 1000, inclusive. If omitted, defaults to 100. | | | array | required | Array of objects where each object represents an [aggregation pipeline](https://www.mongodb.com/docs/manual/core/aggregation-pipeline.md#std-label-aggregation-pipeline) on a collection. To learn more about views, see [Views.](https://www.mongodb.com/docs/manual/core/views.md#std-label-views-landing-page) | | | string | required | Label that identifies the view. | | | string | required | Name of the source collection for the view. If you want to create a view with a [$sql](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/sql.md#std-label-adf-sql-stage) stage, you must omit this field as the SQL statement specifies the source collection. | | | array | optional | [Aggregation pipeline stage(s)](https://www.mongodb.com/docs/manual/reference/mql/aggregation-stages.md#std-label-aggregation-pipeline-operator-reference) to apply to the [`source`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-azure-datalakeconf-azure.databases.-n-.views.-n-.source) collection. You can also create views using the [$sql](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/sql.md#std-label-adf-sql-stage) stage. | ### Example Configuration for Azure Blob Storage Data Store **Example:** Consider Azure Blob Storage container `datacenter-alpha` containing data collected from a datacenter: ```none |--metrics |--hardware ``` The `/metrics/hardware` path stores JSON files with metrics derived from the datacenter hardware, where each filename is the UNIX timestamp in milliseconds of the 24 hour period covered by that file: ```none /hardware/1564671291998.json ``` The following configuration: - Defines a federated database instance store on the `datacenter-alpha` Azure Blob Storage container in the `eastus2` Azure (Microsoft Azure) region. The federated database instance store is specifically restricted to include only data files in the `metrics` directory path. - Maps files from the `hardware` directory to a MongoDB database `datacenter-alpha-metrics` and collection `hardware`. The configuration mapping includes parsing logic for capturing the timestamp implied in the filename. ```json { "stores" : [ { "name" : "datacenter", "provider" : "azure", "region" : "eastus2", "containerName" : "datacenter-alpha", "serviceURL" : "https://mystorageaccount.blob.core.windows.net/" } ], "databases" : [ { "name" : "datacenter-alpha-metrics", "collections" : [ { "name" : "hardware", "dataSources" : [ { "storeName" : "datacenter", "path" : "/hardware/{date date}" } ] } ] } ] } ``` Atlas Data Federation parses the Azure Blob Storage container `datacenter-alpha` and processes all files under `/metrics/hardware/`. The `collections` uses the [path parsing syntax](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-path-syntax) to map the filename to the `date` field, which is an ISO-8601 date, in each document. If a matching `date` field does not exist in a document, Atlas Data Federation adds it. Users connected to the federated database instance can use the MongoDB Query Language and supported aggregations to analyze data in the Azure Blob Storage container through the `datacenter-alpha-metrics.hardware` collection. ## Google Cloud Storage Bucket ### Overview Atlas Data Federation supports Google Cloud Storage buckets as federated database instance stores. You must define mappings in your federated database instance to your Cloud Storage bucket to run queries against your data. **Note:** We refer to objects as files and delimiter-separated prefixes as directories in this page. However, these object storage services aren't actually file systems and don't have the same behaviors in all cases as files on a hard drive. ### Configuration File Format To define a federated database instance store with Google Cloud, you can specify the configuration parameters in JSON format. The configuration contains the Google Cloud data store and maps it to virtual collections that you can query. The JSON configuration for data in Google Cloud uses the following fields: ```json { "stores" : [ { "name" : "", "provider" : "", "region" : "", "bucket" : "", "prefix": "", "delimiter": "" } ], "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "path" : "", "defaultFormat" : "", "provenanceFieldName": "", "omitAttributes": } ] } ], "maxWildcardCollections" : , "views" : [ { "name" : "", "source" : "", "pipeline" : "" } ] } ] } ``` The JSON configuration for Google Cloud contains two top-level objects: [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-gcp-stores-reference) and [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-gcp-databases-reference) #### `stores` The `stores` object defines each data store associated with the federated database instance. The federated database instance store captures files in Google Cloud. Data Federation can only access data stores defined in the `stores` object. The `stores` object contains the following fields: ```json "stores" : [ { "name" : "", "provider" : "", "region" : "", "bucket" : "", "prefix": "", "delimiter": "" } ] ``` The following table describes the fields in the stores object: | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | Required | Array of objects where each object represents a data store to associate with the federated database instance. The federated database instance store captures: Files in a Google Cloud Storage bucket; Documents in an Atlas cluster; Files stored at public URL (Uniform Resource Locator)s Atlas Data Federation can only access data stores defined in the `stores` object. | | | string | Required | Name of the federated database instance store. The [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.databases.-n-.collections.-n-.dataSources.-n-.storeName) field references this value as part of mapping configuration. | | | string | Required | Name of the cloud provider where the data is stored. The value must be `gcs` for a Google Cloud Storage bucket. | | | string | Required | Name of the Google Cloud region in which the Google Cloud Storage bucket is hosted. For a list of valid region names, see [Google Cloud.](https://www.mongodb.com/docs/atlas/reference/google-gcp.md#std-label-google-gcp) | | | string | Required | Name of the Google Cloud Storage bucket. Must exactly match the name of a Google Cloud Storage bucket that Atlas Data Federation must access. | | | string | Optional | Prefix Atlas Data Federation applies when searching for files in the Google Cloud Storage bucket. For example, consider a Google Cloud Storage bucket `metrics` with the following structure: `metrics |--hardware |--software |--computed` The federated database instance store prepends the value of `prefix` to [`databases.[n].collections.[n].dataSources.[n].path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.databases.-n-.collections.-n-.dataSources.-n-.path) to create the full path for files to ingest. Setting `prefix` to `/software` restricts any [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.databases) objects using the federated database instance to only subpaths of `/software`. Defaults to the root of the Google Cloud Storage bucket, retrieving all files. | | | string | Optional | Delimiter that separates [`databases.[n].collections.[n].dataSources.[n].path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.databases.-n-.collections.-n-.dataSources.-n-.path) segments in the federated database instance store. Atlas Data Federation uses the delimiter to efficiently traverse Google Cloud Storage buckets with a simulated hierarchical directory structure. | #### `databases` The `databases` object defines the mapping between each federated database instance store defined in `stores` and MongoDB collections in the databases. The `databases` object contains the following fields: ```json "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "path" : "", "defaultFormat" : "", "provenanceFieldName": "", "omitAttributes": } ] } ], "maxWildcardCollections" : , "views" : [ { "name" : "", "source" : "", "pipeline" : "" } ] } ] ``` The following table describes the fields in the databases object: | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | Required | Array of objects where each object represents a database, its collections, and, optionally, any [views](https://www.mongodb.com/docs/manual/core/views.md#std-label-views-landing-page) on the collections. Each database can have multiple `collections` and `views` objects. | | | string | Required | Name of the database to which Atlas Data Federation maps the data contained in the data store. | | | array | Required | Array of objects where each object represents a collection and data sources that map to a [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.stores) federated database instance store. | | | string | Required | Name of the collection to which Atlas Data Federation maps the data contained in each [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.databases.-n-.collections.-n-.dataSources.-n-.storeName). Each object in the array represents the mapping between the collection and an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.stores) array. You can generate collection names dynamically from file paths by specifying `*` for the collection name and the `collectionName()` function in the path field. See [Generate Dynamic Collection Names from File Path](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-advanced-path-generate-collection) for examples. | | | array | Required | Array of objects where each object represents a [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.stores) federated database instance store to map with the collection. | | | string | Required | Name of a federated database instance store to map to the ``. Must match the `name` of an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.stores) array. | | | string | Required | Controls how Atlas Data Federation searches for and parses files in the [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.databases.-n-.collections.-n-.dataSources.-n-.storeName) before mapping them to the ``. federated database instance prepends the [`stores.[n].prefix`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.stores.-n-.prefix) to the `path` to build the full path to search within. Specify `/` to capture all files and folders from the `prefix` path. For example, consider a Google Cloud Storage bucket named `metrics` with the following structure: `metrics |--hardware |--software |--computed` A `path` of `/` directs Atlas Data Federation to search all files and folders in the `metrics` bucket. A `path` of `/hardware` directs Atlas Data Federation to search only that path for files to ingest. If the [`stores.[n].prefix`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.stores.-n-.prefix) is `software`, Atlas Data Federation searches for files only in the path `/software/computed`. Appending the `*` wildcard character to the path directs Atlas Data Federation to include all files and folders from that point in the path. For example, `/software/computed*` would match files like `/software/computed-detailed`, `/software/computedArchive`, and `/software/computed/errors`. [`databases.[n].collections.[n].dataSources.[n].path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.databases.-n-.collections.-n-.dataSources.-n-.path) supports additional syntax for parsing filenames, including: Generating document fields from filenames.; Using regular expressions to control field generation.; Setting boundaries for bucketing filenames by timestamp. See [Define Path Syntax for Data Stores](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-path-syntax) for more information. When specifying the [`path`:](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.databases.-n-.collections.-n-.dataSources.-n-.path) Specify the data type for the partition attribute.; Ensure that the partition attribute type matches the data type to parse.; Use the delimiter specified in [`delimiter`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-aws-datalakeconf-aws.stores.-n-.delimiter) When specifying attributes of the same type, do any of the following: Add a constant separator between the attributes.; Use regular expressions to describe the search pattern. To learn more, see [Unsupported Parsing Functions.](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-unsupported-parsing-funcs) | | | string | Optional | Default format that Data Federation assumes if it encounters a file without an extension while searching the [`databases.[n].collections.[n].dataSources.[n].storeName`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.databases.-n-.collections.-n-.dataSources.-n-.storeName) The following values are valid for the `defaultFormat` field: `.json`, `.json.gz`, `.bson`, `.bson.gz`, `.avro,` `.avro.gz`, `.orc`, `.tsv`, `.tsv.gz`, `.csv`, `.csv.gz`, `.parquet` If your file format is `CSV` or `TSV`, you must include a header row in your data. See [CSV and TSV](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/data-formats/csv-tsv-data-files.md#std-label-adf-csv-tsv-data) for more information. If omitted, Data Federation attempts to detect the file type by processing a few bytes of the file. **See also:** [Supported Data Formats](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/supported-data-formats.md#std-label-adf-data-formats) | | | string | Optional | Name for the field that includes the provenance of the documents in the results. If you specify this setting in the storage configuration, Atlas Data Federation returns the following fields for each document in the result: Field NameDescription`provider`Provider ([`stores.[n].provider`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.stores.-n-.provider)) in the federated database instance storage configuration`region`Google Cloud region ([`stores.[n].region`)](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.stores.-n-.region)`bucket`Name of the Google Cloud Storage bucket ([`stores.[n].bucket`)](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.stores.-n-.bucket)`key`Path ([`databases.[n].collections.[n].dataSources.[n].path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.databases.-n-.collections.-n-.dataSources.-n-.path)) to the document`lastModified`Date and time the document was last modified. You can't configure this setting using the Visual Editor in the Atlas UI. | | | boolean | Optional | Flag that specifies whether to omit the attributes (key and value pairs) that Atlas Data Federation adds to documents in the collection. You can specify one of the following values: `false` - to add the attributes; `true` - to omit the attributes If omitted, defaults to `false` and Atlas Data Federation adds the attributes. For example, consider a file named `/employees/949-555-0195.json` for which you configure the [`databases.[n].collections.[n].dataSources.[n].path`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-gcp-datalakeconf-gcp.databases.-n-.collections.-n-.dataSources.-n-.path) `/employees/{phone string}`. Atlas Data Federation adds the attribute `phone: 949-555-0195` to documents in this file if `omitAttributes` is `false`, regardless of whether the key-value pair already exists in the document. If you set `omitAttributes` to `true`, Atlas Data Federation doesn't add the attribute to the document in the virtual collection. Atlas Data Federation adds partition attributes to documents even when a field with the same name already exists in the document, which creates duplicate field names. MongoDB doesn't support [duplicate field names](https://www.mongodb.com/docs/manual/reference/limits/#mongodb-does-not-support-duplicate-field-names). To avoid duplicate field names, set `omitAttributes` to `true` or use partition attribute names that don't appear in your documents. | ### Example Configuration for Google Cloud Storage Bucket Consider a Google Cloud Storage bucket `datacenter-alpha` containing data collected from a datacenter: ```none |--metrics |--hardware ``` The `/metrics/hardware` path stores JSON files with metrics derived from the datacenter hardware, where each filename is the UNIX timestamp in milliseconds of the 24 hour period covered by that file: ```none /hardware/1564671291998.json ``` The following configuration: - Defines a federated database instance store on the `datacenter-alpha` Google Cloud Storage bucket in the `us-central1` Google Cloud region. The federated database instance store is specifically restricted to include only data files in the `metrics` directory path. A delimiter of `/` is defined to simulate a file system hierarchy for ease of navigation and retrieval. - Maps files from the `hardware` directory to a MongoDB database `datacenter-alpha-metrics` and collection `hardware`. The configuration mapping includes parsing logic for capturing the timestamp implied in the filename. ```json { "stores" : [ { "name" : "datacenter-alpha", "provider" : "gcs", "region" : "us-central1", "bucket" : "datacenter-alpha", "prefix": "metrics", "delimiter": "/" } ], "databases" : [ { "name" : "datacenter-alpha-metrics", "collections" : [ { "name" : "hardware", "dataSources" : [ { "storeName" : "datacenter-alpha", "path" : "/hardware/{date date}" } ] } ] } ] } ``` Atlas Data Federation parses the Google Cloud Storage bucket `datacenter-alpha` and processes all files under `/metrics/hardware/`. The `collections` object uses the [path parsing syntax](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples.md#std-label-adf-path-syntax) to map the filename to the `date` field, which is an ISO-8601 date, in each document. If a matching `date` field does not exist in a document, Atlas Data Federation adds it. Users connected to the federated database instance can use the MongoDB Query Language and supported aggregations to analyze data in the Google Cloud Storage bucket through the `datacenter-alpha-metrics.hardware` collection. ## HTTP URL ### Overview Atlas Data Federation supports publicly accessible URL (Uniform Resource Locator)s as federated database instance stores. You must define mappings in your federated database instance to your HTTP (HyperText Transport Protocol) data stores to run queries against your data. **Important:** Information in your storage configuration is visible internally at MongoDB and stored as operational data to monitor and improve the performance of Atlas Data Federation. So, we recommend that you do not use PII (Personally Identifiable Information) in your configurations. ### Configuration Format The federated database instance configuration has the following format: ```json { "stores" : [ { "name" : "", "provider": "", "defaultFormat" : "", "allowInsecure": , "urls": [""] } ], "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "allowInsecure" : , "urls" : [""], "defaultFormat" : "", "provenanceFieldName": "" } ] } ], "views" : [ { "name" : "", "source" : "", "pipeline" : "" } ] } ] } ``` [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-http-stores-reference) The `stores` object defines each data store associated with the federated database instance. The federated database instance store captures files stored at publicly accessible URL (Uniform Resource Locator)s. Data Federation can only access data stores defined in the `stores` object. [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-http-databases-reference) The `databases` object defines the mapping between each federated database instance store defined in `stores` and MongoDB collections in the databases. #### `stores` ```json "stores" : [ { "name" : "", "provider" : "", "allowInsecure": , "urls" : [""], "defaultFormat" : "" } ] ``` | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | required | Array of objects where each object represents a data store to associate with the federated database instance. The federated database instance store captures files stored at publicly accessible URL (Uniform Resource Locator)s. Atlas Data Federation can only access data stores defined in the `stores` object. | | | string | required | Name of the federated database instance store. The [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.databases.-n-.collections.-n-.dataSources.-n-.storeName) field references this value as part of mapping configuration. | | | string | required | Defines where the data is stored. Value can be one of the following:`s3` for an AWS (Amazon Web Services) S3 (Simple Storage Service) bucket.; `atlas` for a collection in an Atlas cluster.; `http` for data in files hosted at publicly accessible URL (Uniform Resource Locator)s. | | | boolean | required | *Optional.* Validates the scheme in the specified URL (Uniform Resource Locator)s. Value can be one of the following: `true` to allow insecure HTTP (HyperText Transport Protocol) scheme; `false` to only allow secure HTTPS (Secure HyperText Transport Protocol) scheme (default) If true, Atlas Data Federation: Does not verify the server's certificate chain and hostname.; Accepts any certificate with any hostname presented by the server. WARNING: If you set this to `true`, your data might become vulnerable to a man-in-the-middle attack, which can compromise the confidentiality and integrity of your data. Set this to `true` only for testing and getting started with Atlas Data Federation. If omitted, defaults to `false`. If set to `false`, Data Federation returns an error similar to the following if a specified URL contains insecure HTTP (HyperText Transport Protocol) scheme: `The insecure HTTP scheme is not supported by default - please add a "allowInsecure: true" flag to query from such URLs.` | | | array | optional | Comma-separated list of publicly accessible HTTP (HyperText Transport Protocol) URL (Uniform Resource Locator)s where data is stored. You can't specify URL (Uniform Resource Locator)s that require authentication. | | | string | optional | Default format that Data Federation assumes if it encounters a file without an extension while searching the `databases.[n].collections.[n].dataSources.[n].storeName`. The following values are valid for the `defaultFormat` field: `.json`, `.json.gz`, `.bson`, `.bson.gz`, `.avro,` `.avro.gz`, `.orc`, `.tsv`, `.tsv.gz`, `.csv`, `.csv.gz`, `.parquet` IMPORTANT: If your file format is `CSV` or `TSV`, you must include a header row in your data. See [CSV and TSV](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/data-formats/csv-tsv-data-files.md#std-label-adf-csv-tsv-data) for more information. If omitted, Data Federation attempts to detect the file type by processing a few bytes of the file. The specified format only applies to the URL (Uniform Resource Locator)s specified in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.stores) object. | #### `databases` ```json "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "allowInsecure" : , "urls" : [""], "defaultFormat" : "", "provenanceFieldName": "" } ] } ] } ] ``` | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | required | Array of objects where each object represents a database, its collections, and, optionally, any [views](https://www.mongodb.com/docs/manual/core/views.md#std-label-views-landing-page) on the collections. Each database can have multiple `collections` and `views` objects. | | | string | required | Name of the database to which Atlas Data Federation maps the data contained in the data store. | | | array | required | Array of objects where each object represents a collection and data sources that map to a [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.stores) federated database instance store. For dynamically generated databases, you can define only one wildcard (`*`) collection object in the storage configuration. | | | string | required | Name of the collection to which Atlas Data Federation maps the data contained in each [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.databases.-n-.collections.-n-.dataSources.-n-.storeName). Each object in the array represents the mapping between the collection and an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.stores) array. IMPORTANT: Wildcard `*` collections are not available for HTTP (HyperText Transport Protocol) stores. | | | array | string | Array of objects where each object represents a [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.stores) federated database instance store to map with the collection. | | | string | required | Name of a federated database instance store to map to the ``. Must match the [`name`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.stores.-n-.name) of an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.stores) array. | | | boolean | required | *Optional.* Validates the scheme in the specified URL (Uniform Resource Locator)s. Value can be one of the following: `true` to allow insecure HTTP (HyperText Transport Protocol) scheme; `false` to only allow secure HTTPS (Secure HyperText Transport Protocol) scheme (default) If true, Atlas Data Federation: Does not verify the server's certificate chain and hostname.; Accepts any certificate with any hostname presented by the server. WARNING: If you set this to `true`, your data might become vulnerable to a man-in-the-middle attack, which can compromise the confidentiality and integrity of your data. Set this to `true` only for testing and getting started with Atlas Data Federation. If omitted, defaults to `false`. If set to `false`, Data Federation returns an error similar to the following if a specified URL contains insecure HTTP (HyperText Transport Protocol) scheme: `The insecure HTTP scheme is not supported by default - please add a "allowInsecure: true" flag to query from such URLs.` | | | array | optional | Comma-separated list of publicly accessible URL (Uniform Resource Locator)s where the data is stored. Federated Database Instance creates a partition for each URL (Uniform Resource Locator). You can specify URL (Uniform Resource Locator)s that are not in the [`urls`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.stores.-n-.urls); however, the collection will contain a union of data from URL (Uniform Resource Locator)s in both the [`urls`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.stores.-n-.urls) and [`urls`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.databases.-n-.collections.-n-.dataSources.-n-.urls) If omitted, Data Federation uses the [`urls`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.stores.-n-.urls) in the specified [`storeName`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.databases.-n-.collections.-n-.dataSources.-n-.storeName) | | | string | optional | Default format that Data Federation assumes if it encounters a file without an extension while searching the `databases.[n].collections.[n].dataSources.[n].storeName`. The following values are valid for the `defaultFormat` field: `.json`, `.json.gz`, `.bson`, `.bson.gz`, `.avro,` `.avro.gz`, `.orc`, `.tsv`, `.tsv.gz`, `.csv`, `.csv.gz`, `.parquet` IMPORTANT: If your file format is `CSV` or `TSV`, you must include a header row in your data. See [CSV and TSV](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/data-formats/csv-tsv-data-files.md#std-label-adf-csv-tsv-data) for more information. If omitted, Data Federation attempts to detect the file type by processing a few bytes of the file. The specified format only applies to the URL (Uniform Resource Locator)s specified in the [`databases.[n].collections.[n].dataSources`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.databases.-n-.collections.-n-.dataSources) object. | | | string | required | Name for the field that includes the provenance of the documents in the results. If you specify this setting in the storage configuration, Atlas Data Federation returns the following fields for each document in the result: Field NameDescription`provider`Provider ([`stores.[n].provider`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.stores.-n-.provider)) in the federated database instance storage configuration`uri`HTTP (HyperText Transport Protocol) or HTTPS (Secure HyperText Transport Protocol) URL (Uniform Resource Locator) ([`databases.[n].collections.[n].dataSources.[n].urls`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.databases.-n-.collections.-n-.dataSources.-n-.urls)) of the document You can't configure this setting using the Visual Editor in the Atlas UI. | | | array | required | Array of objects where each object represents an [aggregation pipeline](https://www.mongodb.com/docs/manual/core/aggregation-pipeline.md#std-label-aggregation-pipeline) on a collection. To learn more about views, see [Views.](https://www.mongodb.com/docs/manual/core/views.md#std-label-views-landing-page) | | | string | required | Name of the view. | | | string | required | Name of the source collection for the view. If you want to create a view with a [$sql](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/sql.md#std-label-adf-sql-stage) stage, you must omit this field as the SQL statement will specify the source collection. | | | array | required | [Aggregation pipeline stage(s)](https://www.mongodb.com/docs/manual/reference/mql/aggregation-stages.md#std-label-aggregation-pipeline-operator-reference) to apply to the [`source`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-http-datalakeconf-http.databases.-n-.views.-n-.source) collection. You can also create views using the [$sql](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/sql.md#std-label-adf-sql-stage) stage. | ### Example Configuration for HTTP Data Store Consider URL (Uniform Resource Locator)s `https://www.datacenter-hardware.com/data.json`, `https://www.datacenter-software.com/data.json`, and `https://www.datacenter-metrics.com/data.json` containing data collected from a datacenter. The following configuration: - Specifies the publicly accessible URLs that contain data in files as a federated database instance store. - Creates a partition for each URL (Uniform Resource Locator). ```json { "stores" : [ { "name" : "httpStore", "provider" : "http", "allowInsecure" : false, "urls" : [ "https://www.datacenter-hardware.com/data.json", "https://www.datacenter-software.com/data.json" ], "defaultFormat" : ".json" } ], "databases" : [ { "name" : "dataCenter", "collections" : [ { "name" : "inventory", "dataSources" : [ { "storeName" : "httpStore", "allowInsecure" : false, "urls" : [ "https://www.datacenter-metrics.com/data" ], "defaultFormat" : ".json" } ] } ] } ] } ``` **See also:** [Querying Data at a HTTP (HyperText Transport Protocol) or HTTPS (Secure HyperText Transport Protocol) URL (Uniform Resource Locator)](https://www.mongodb.com/docs/atlas/data-federation/query/query-federated-database.md#std-label-query-http) **See also:** - [Query a Federated Database Instance](https://www.mongodb.com/docs/atlas/data-federation/query/query-federated-database.md#std-label-query-fdi) ## Atlas Cluster ### Overview Atlas Data Federation supports Atlas clusters as federated database instance stores. You must define mappings in your federated database instance to your Atlas cluster to run queries against your data. **Important:** Information in your storage configuration is visible internally at MongoDB and stored as operational data to monitor and improve the performance of Atlas Data Federation. So, we recommend that you do not use PII (Personally Identifiable Information) in your configurations. ### Configuration Format To define a federated database instance store for an Atlas cluster, you can specify the following JSON (Javascript Object Notation) configuration parameters in your [Federated Database Instance Configuration File](https://www.mongodb.com/docs/atlas/data-federation/config/config-data-stores.md#std-label-adf-setstorageconfig). The configuration contains the Atlas cluster and maps it to virtual collections that you can query. The following JSON (Javascript Object Notation) configuration shows the format of the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-atlas-stores-reference) and [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-atlas-databases-reference) configuration fields, which you must set in your Federated Database Instance configuration file to define an Atlas cluster as a federated database instance store: ```json { "stores" : [ { "name" : "", "provider": "", "clusterName": "", "projectId": "", "readPreference": { "mode": "", "tagSets": [ [{"name": "", "value": ""},...], ... ], "maxStalenessSeconds": } } ], "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "database" : "", "databaseRegex": "", "collection" : "", "collectionRegex" : "", "provenanceFieldName": "" } ] } ], "views" : [ { "name" : "", "source" : "", "pipeline" : "" } ] } ] } ``` The JSON configuration for an Atlas cluster contains two top-level objects: [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-atlas-stores-reference) and [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-atlas-databases-reference) #### `stores` The `stores` object defines each data store associated with the federated database instance. This store captures documents in the Atlas cluster. Federated Database Instances can only access data stores defined in the `stores` object. ```json "stores" : [ { "name" : "", "provider" : "", "clusterName" : "", "projectId": "" "readPreference": { "mode": "", "tagSets": [ [{"name": "", "value": ""},...], ... ], "maxStalenessSeconds": }, "readConcern": { "level": "" } } ] ``` The following table describes the fields in the `stores` object: | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | required | Array of objects where each object represents a data store to associate with the federated database instance. The federated database instance store captures documents in the Atlas cluster. Atlas Data Federation can only access data stores defined in the `stores` object. | | | string | required | Name of the federated database instance store. The [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.dataSources.-n-.storeName) field references this value as part of mapping configuration. | | | string | required | Defines where the data is stored. Value must be `atlas` for a collection in an Atlas cluster. | | | string | required | Name of the Atlas cluster on which the store is based. The cluster must exist in the same project as your federated database instance. The `source` field on the data partition is the name of the Atlas cluster. | | | string | required | Unique identifier of the project that contains the Atlas cluster on which the store is based. | | | boolean | optional | Cluster [read preference](https://www.mongodb.com/docs/manual/core/read-preference.md#std-label-replica-set-read-preference), which describes how to route read requests to the cluster. For example: The following `readPreference` setting specifies `secondary` [mode](https://www.mongodb.com/docs/manual/core/read-preference.md#std-label-read-pref-modes-summary) and `ANALYTICS` [nodeType.](https://www.mongodb.com/docs/atlas/reference/replica-set-tags.md#std-label-repl-set-node-types) `{ ... "stores": [ { "provider": "atlas", "clusterName": , "name": , "projectId": , "readPreference": { "mode": "secondary", "tagSets": [ [ { "name": "nodeType", "value": "ANALYTICS" } ], ... ] } } ] }` | | | string | optional | [Read preference mode](https://www.mongodb.com/docs/manual/core/read-preference.md#std-label-read-pref-modes-summary) that specifies which replica set member to route the read requests to. Value can be one of the following: `primary` - to route all read requests to the replica set [primary](https://www.mongodb.com/docs/manual/reference/glossary/#term-primary); `primaryPreferred` - to route all read requests the replica set [primary](https://www.mongodb.com/docs/manual/reference/glossary/#term-primary) and to [secondary](https://www.mongodb.com/docs/manual/reference/glossary/#term-secondary) members only if `primary` is unavailable; `secondary` - to route all read requests to the [secondary](https://www.mongodb.com/docs/manual/reference/glossary/#term-secondary) members of the replica set; `secondaryPreferred` - to route all read requests to the [secondary](https://www.mongodb.com/docs/manual/reference/glossary/#term-secondary) members of the replica set and the [primary](https://www.mongodb.com/docs/manual/reference/glossary/#term-primary) on sharded clusters only if `secondary` members are unavailable; `nearest` - to route all read requests to random eligible replica set member, irrespective of whether that member is a [primary](https://www.mongodb.com/docs/manual/reference/glossary/#term-primary) or [secondary](https://www.mongodb.com/docs/manual/reference/glossary/#term-secondary) If omitted, defaults to `local`. | | | array | optional | Arrays of [tag sets](https://www.mongodb.com/docs/manual/core/read-preference-tags.md#std-label-replica-set-read-preference-tag-sets) or tag specification documents that contain name and value pairs for the replica set member. If specified, Atlas Data Federation routes read requests to replica set member or members that are associated with the specified tags. To learn more, [Read Preference Tag Sets.](https://www.mongodb.com/docs/manual/core/read-preference-tags.md#std-label-replica-set-read-preference-tag-sets) IMPORTANT: Atlas Data Federation doesn't support `tagSets` for sharded clusters. | | | string | optional | Maximum replication lag, or "staleness", for reads from secondaries. To learn more about `maxStalenessSeconds`, see [Read Preference maxStalenessSeconds.](https://www.mongodb.com/docs/manual/core/read-preference-staleness.md#std-label-replica-set-read-preference-max-staleness) | | | string | optional | Consistency and isolation properties of the data read from an Atlas cluster. To learn more, see [Read Concern](https://www.mongodb.com/docs/manual/reference/read-concern.md#std-label-read-concern). The value for the level of consistency and availability can be one of the following: `local` - to return data from an instance with no guarantee that the data has been written to a majority of the replica set members. To learn more, see [Read Concern "local".](https://www.mongodb.com/docs/manual/reference/read-concern-local/); `available` - to return data from an instance with no guarantee that the data has been written to a majority of the replica set members. This can return [orphaned documents](https://www.mongodb.com/docs/manual/reference/glossary.md#std-term-orphaned-document) when reading from a sharded collection. To learn more, see [Read Concern "available".](https://www.mongodb.com/docs/manual/reference/read-concern-available/); `majority` - to return data that has been acknowledged by a majority of the nodes on the Atlas cluster. To learn more, see [Read Concern "majority".](https://www.mongodb.com/docs/manual/reference/read-concern-majority/); `linearizable` - to return data that reflects all successful majority-acknowledged writes that completed prior to the start of the read operation. To learn more, see [Read Concern "linearizable".](https://www.mongodb.com/docs/manual/reference/read-concern-linearizable/); `snapshot` - to return majority-committed data as it appears across shards from a specific single point in time in the recent past. To learn more, see [Read Concern "snapshot".](https://www.mongodb.com/docs/manual/reference/read-concern-snapshot/) | #### `databases` The `databases` object defines the mapping between each federated database instance store defined in `stores` and MongoDB collections in the databases. ```json "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "database" : "", "databaseRegex": "", "collection" : "", "collectionRegex" : "", "provenanceFieldName": "" } ] } ] } ] ``` The following table describes the fields in the `databases` object: | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | required | Array of objects where each object represents a database, its collections, and, optionally, any [views](https://www.mongodb.com/docs/manual/core/views.md#std-label-views-landing-page) on the collections. Each database can have multiple `collections` and `views` objects. | | | string | required | Name of the database to which Atlas Data Federation maps the data contained in the data store. You can generate databases dynamically by specifying `*` for the database name. Dynamically generated databases: Can exist alongside explicitly defined databases. However, Atlas Data Federation won't dynamically generate databases with names that conflict with explicitly defined databases in the storage configuration.; Can only be from a single Atlas cluster. Atlas Data Federation won't dynamically generate databases from multiple Atlas clusters or other data stores. | | | array | required | Array of objects where each object represents a collection and data sources that map to a [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.stores) federated database instance store. For dynamically generated databases, you can define only one wildcard (`*`) collection object in the storage configuration. | | | string | required | Name of the collection to which Atlas Data Federation maps the data contained in each [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.dataSources.-n-.storeName). Each object in the array represents the mapping between the collection and an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.stores) array. You can generate collection names dynamically by specifying `*` for the collection name and omitting the [`collection`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.dataSources.-n-.collection) field. To dynamically generate wildcard (`*`) collections for dynamically generated wilcard (`*`) databases, specify the [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.dataSources.-n-.storeName) option and omit the [`databases.[n].collections.[n].dataSources.[n].database`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.dataSources.-n-.database) option. Note that for dynamically generated databases, you can define only one wildcard (`*`) collection object in the storage configuration. For wildcard (`*`) collections, you can also define regex patterns using [`databases.[n].collections.[n].dataSources.[n].collectionRegex`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.dataSources.-n-.collectionRegex) field to filter the collections only. | | | array | required | Array of objects where each object represents a [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.stores) federated database instance store to map with the collection. | | | string | required | Name of a federated database instance store to map to the ``. Must match the [`name`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.stores.-n-.name) of an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.stores) array. | | | string | required | Name of the database on the Atlas cluster that contains the collection. You must omit this setting to: Create a wildcard (`*`) collection for a wilcard (`*`) database.; Glob multiple databases. | | | string | optional | Regex pattern to use for globbing databases to combine multiple collections. If you specify this option, the federated database instance instance contains a single database with collections from multiple databases. For globbing databases, you must do the following: Omit the [`database`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.dataSources.-n-.database) field.; Specify a valid name for the [`databases.[n].collections.[n].dataSources.[n].collection`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.dataSources.-n-.collection) field. Suppose you have 2 databases named `foo` and `bar` that each have a collection named `Sales`. You can combine the `Sales` collection from `foo` and `bar` using the `databaseRegex` option in your storage configuration: ` { "databases": [ { "name": "Transactions", "collections": [ { "name": "AllSales", "dataSources": [ { "storeName": "atlasStore", "databaseRegex": ".*", "collection": "Sales" } ] } ] } ] } For the preceding :datalakeconf-atlas:`databases` object, Atlas Data Federation generates the following in your federated database instance: - A virtual database named ``Transactions``. - A virtual collection named ``AllSales`` that contains data from the collection named ``Sales`` in all the databases whose name match the regex pattern specified in the ``databaseRegex`` option.` If you specify this option, you must specify the name of the collection. You can't specify this option for wildcard collections. | | | string | required | Name of the collection in the Atlas cluster on which the federated database instance store is based. You must omit this setting for: Creating a wildcard (`*`) collection.; Creating wildcard collection names that match regex patterns.; Combining multiple collections in a database using regex patterns. | | | string | Optional for wildcard collections Required for combining collections in a database | Regex pattern to use for creating the wildcard (`*`) collection or for combining mulitple collections in a database. To use regex patterns for wildcard (`*`) collection names, you must do the following: Specify wildcard (`*`) as the value for [`databases.[n].collections.[n].name`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.name); Omit [`databases.[n].collections.[n].dataSources.[n].collection`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.dataSources.-n-.collection) If you specify this field for generating wildcard collections, the federated database instance instance only contains collections with names that match the specified regular expression. The collections in the federated database instance storage configuration use their original names in the Atlas cluster. To use regex patterns for combining multiple collections in a database, you must do the following: Specify a name that isn't the wildcard (`*`) as the value for [`databases.[n].collections.[n].name`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.name); Omit [`databases.[n].collections.[n].dataSources.[n].collection`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.dataSources.-n-.collection) If you specify this field for combining multiple collections, the collection in the federated database instance contains data from all the Atlas collections with names that match the specified regular expression. The collection in the federated database instance storage configuration uses the name that you specify as value for [`databases.[n].collections.[n].name`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.name) To learn more about the regex syntax, see [Go programming language.](https://golang.org/pkg/regexp/) | | | string | required | Name for the field that includes the provenance of the documents in the results. If you specify this setting in the storage configuration, Atlas Data Federation returns the following fields for each document in the result: Field NameDescription`provider`Provider ([`stores.[n].provider`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.stores.-n-.provider)) in the federated database instance storage configuration`clusterName`Name of the Atlas cluster ([`stores.[n].clusterName`)](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.stores.-n-.clusterName)`databaseName`Name of the database ([`databases.[n].collections.[n].dataSources.[n].database`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.dataSources.-n-.database)) in the Atlas cluster`collectionName`Name of the collection ([`databases.[n].collections.[n].name`)](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.collections.-n-.name) You can't configure this setting using the Visual Editor in the Atlas UI. | | | array | required | Array of objects where each object represents an [aggregation pipeline](https://www.mongodb.com/docs/manual/core/aggregation-pipeline.md#std-label-aggregation-pipeline) on a collection. To learn more about views, see [Views.](https://www.mongodb.com/docs/manual/core/views.md#std-label-views-landing-page) | | | string | required | Name of the view. | | | string | required | Name of the source collection for the view. If you want to create a view with a [$sql](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/sql.md#std-label-adf-sql-stage) stage, you must omit this field as the SQL statement will specify the source collection. | | | string | required | [Aggregation pipeline stage(s)](https://www.mongodb.com/docs/manual/reference/mql/aggregation-stages.md#std-label-aggregation-pipeline-operator-reference) to apply to the [`source`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-atlas-datalakeconf-atlas.databases.-n-.views.-n-.source) collection. You can also create views using the [$sql](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/sql.md#std-label-adf-sql-stage) stage. | ### Example Configuration for Atlas Data Store Consider a `M10` or higher Atlas cluster named `myDataCenter` containing data in the `metrics.hardware` collection. The `metrics.hardware` collection contains JSON documents with metrics derived from the hardware in a datacenter. The following configuration: - Specifies the Atlas cluster named `myDataCenter` in the specified project as a federated database instance store. - Maps documents from the `metrics.hardware` collection in the Atlas cluster to the `dataCenter.inventory` collection in the storage configuration. ```json { "stores" : [ { "name" : "atlasClusterStore", "provider" : "atlas", "clusterName" : "myDataCenter", "projectId" : "5e2211c17a3e5a48f5497de3" } ], "databases" : [ { "name" : "dataCenter", "collections" : [ { "name" : "inventory", "dataSources" : [ { "storeName" : "atlasClusterStore", "database" : "metrics", "collection" : "hardware" } ] } ] } ] } Atlas Data Federation maps all the documents in the ``metrics.hardware`` collection to the ``dataCenter.inventory`` collection in the storage configuration. Users connected to the federated database instance can use the MongoDB Query Language and supported aggregations to analyze data in the |service| cluster through the ``dataCenter.inventory`` collection. When you run queries, the query first goes to Atlas Data Federation. Therefore, if you run aggregation queries that are supported by your |service| cluster but not by Atlas Data Federation, the queries will fail. To learn more about supported and unsupported commands in Data Federation, see :ref:`adf-mql-support`. ``` **See also:** - [Querying Data in Your Atlas Cluster](https://www.mongodb.com/docs/atlas/data-federation/query/query-federated-database.md#std-label-query-atlas) - [Configure Atlas Data Federation](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-configuration-file-atlas) - [Query a Federated Database Instance](https://www.mongodb.com/docs/atlas/data-federation/query/query-federated-database.md#std-label-query-fdi) ## Online Archives **Note:** You can only use *new* Online Archives as a data source in Atlas Data Federation. Read the MongoDB blog post [New Online Archive with Performance Improvements and Enhanced Metrics](https://www.mongodb.com/blog/post/new-online-archive-performance-improvements-enhanced-metrics) for more information. ### Overview Atlas Data Federation supports Atlas [online archives](https://www.mongodb.com/docs/atlas/online-archive/overview.md#std-label-online-archive-overview) as federated database instance stores. You must define mappings in your federated database instance storage configuration to your online archive to run queries against your data. **Important:** Information in your storage configuration is visible internally at MongoDB and stored as operational data to monitor and improve the performance of Atlas Data Federation. So, we recommend that you do not use PII (Personally Identifiable Information) in your configurations. ### Configuration Format To define a federated database instance store for an Atlas Online Archive, you can specify the following JSON (Javascript Object Notation) configuration parameters in your [Federated Database Instance Configuration File](https://www.mongodb.com/docs/atlas/data-federation/config/config-data-stores.md#std-label-adf-setstorageconfig). The configuration contains the Atlas online archive and maps it to virtual collections that you can query. The following JSON (Javascript Object Notation) configuration shows the format of the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-oa-stores-reference) and [`databases`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-oa-databases-reference) configuration fields, which you must set in your Federated Database Instance configuration file to define an Atlas Online Archive as a federated database instance store: ```json { "stores" : [ { "name" : "", "provider": "", "region": "" } ], "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "datasetName" : "", "datasetPrefix": "", "trimLevel": , "provenanceFieldName": "", "maxDatasets": } ] } ], "views" : [ { "name" : "", "source" : "", "pipeline" : "" } ] } ] } ``` The JSON configuration for Atlas Online Archive data stores contains two top-level objects: [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-oa-stores-reference) and [`databases`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#std-label-adf-oa-databases-reference) #### `stores` The stores object defines each data store associated with the federated database instance. This store captures documents in an Atlas Online Archive. Data Federation can only access data stores defined within the stores object. The `stores` object contains the following fields: ```json "stores" : [ { "name" : "", "provider": "", "region": "" } ] ``` The following table describes the fields in the `stores` object: | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | required | Array of objects where each object represents a data store to associate with the federated database instance. The federated database instance store references to an Atlas online archives. Atlas Data Federation can only access data stores defined in the `stores` object. | | | string | required | Name of the federated database instance store. The [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.databases.-n-.collections.-n-.dataSources.-n-.storeName) field references this value as part of mapping configuration. | | | string | required | Cloud provider where the snapshot data is stored. Value must be `dls:` for a snapshot. Atlas Data Federation supports the following subtypes: `aws`, for which the value must be `dls:aws` | | | string | required | Region name of your online archive. Each store is associated with a single region, where the archived data is stored. If you have multiple online archives in different regions, you must add a store for each region to map data in that region to virtual databases and collection in federated database instance. To learn more about the supported regions for AWS (Amazon Web Services), see [Atlas Data Federation Regions.](https://www.mongodb.com/docs/atlas/data-federation/adf-overview/regions.md#std-label-atlas-data-federation-regions) | #### `databases` The databases object defines the mapping between each federated database instance store defined in stores and MongoDB collections in the databases. The database object contains the following fields: ```json "databases" : [ { "name" : "", "collections" : [ { "name" : "", "dataSources" : [ { "storeName" : "", "datasetName" : "", "datasetPrefix": "", "trimLevel": , "provenanceFieldName": "", "maxDatasets": } ] } ], "views" : [ { "name" : "", "source" : "", "pipeline" : "" } ] } ] ``` The following table describes the fields in the database object: | Field | Type | Necessity | Description | | --- | --- | --- | --- | | | array | required | Array of objects that define the mapping between each federated database instance store defined in `stores` and online archives. Each object represents a database, its collections, and, optionally, any [views](https://www.mongodb.com/docs/manual/core/views.md#std-label-views-landing-page) on the collections. Each database can have multiple `collections` and `views` objects. | | | string | required | Name of the database to which Atlas Data Federation maps the data contained in the data store. | | | array | required | Array of objects where each object represents a collection and data sources that map to a [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.stores) federated database instance store. | | | string | required | Name of the collection to which Atlas Data Federation maps the data contained in each [`databases.[n].collections.[n].dataSources.[n].storeName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.databases.-n-.collections.-n-.dataSources.-n-.storeName). Each object in the array represents the mapping between the collection and an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.stores) array. You can generate collection names dynamically by specifying `*` for the collection name. To dynamically generate collection names, you must also specify the following: [`databases.[n].collections.[n].dataSources.[n].datasetPrefix`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.databases.-n-.collections.-n-.dataSources.-n-.datasetPrefix); [`databases.[n].collections.[n].dataSources.[n].trimLevel`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.databases.-n-.collections.-n-.dataSources.-n-.trimLevel) For wildcard collections, Atlas Data Federation maps a dataset name to a collection name first by splitting the [namespace](https://www.mongodb.com/docs/manual/reference/limits.md#std-label-faq-dev-namespace) into a list of fields on the `$` delimiter, then by trimming a number of fields from the left of the list, and finally by combining the remaining fields using `_`. | | | array | required | Array of objects where each object represents a federated database instance store in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.stores) array to map with the collection. You can specify multiple `dataSources` for a wildcard collection only if all the `dataSources` for the collection map to the online archive [`stores`.](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.stores) | | | string | required | Name of a federated database instance store to map to the ``. Must match the [`name`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.stores.-n-.name) of an object in the [`stores`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.stores) array. | | | string | required | Name of the online archive dataset to map with the collection. The `datasetName` is in the following format: `$$$$$$` For example, consider the following online archive name. `v1$atlas$archive$clusterName$dbName$collections$archiveId` Here: `v1` is the version; `atlas` is the type of data source; `archive` is the subtype; `clusterName` is the name of the Atlas cluster; `dbName` is the name of the database on the Atlas cluster; `collection` is the the collections in the database; `archiveId` is the unique identifier of the archived data IMPORTANT: For a non-wildcard collection, you can't specify this option if you specify the [`databases.[n].collections.[n].dataSources.[n].datasetPrefix`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.databases.-n-.collections.-n-.dataSources.-n-.datasetPrefix) option because Atlas Data Federation automatically generates collections for the latest dataset using the name specified through [`databases.[n].collections.[n].dataSources.[n].datasetPrefix`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.databases.-n-.collections.-n-.dataSources.-n-.datasetPrefix) option. | | | string | Required for wildcard collections. | Dataset name prefix to match against the online archive dataset name. When you set this for wildcard collections, Atlas Data Federation maps collections to only to the online archive dataset names whose prefix match the value specified here. If you specify this setting for a non-wildcard collection, Atlas Data Federation maps only the latest dataset (for the most recently captured snapshot) to the collection. You can't specify the [`databases.[n].collections.[n].dataSources.[n].datasetName`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.databases.-n-.collections.-n-.dataSources.-n-.datasetName) also if you specify this option for a non-wildcard collection. For example: `{ ..., "name": "myFederatedDbCollection", "dataSources": [ { "storeName": "aws-dl-store", "datasetPrefix": "v1$atlas$archive$MyCluster$MyDB$MyArchiveId" } ] }` | | | integer | required | Unsigned integer that specifies how many fields of the dataset name to trim from the left of the dataset name before mapping the remaining fields to a wildcard collection name. Value must be greater than `0` and less than `7`. You can set this setting for wildcard collections only. You can't configure this setting using the Visual Editor in the Atlas UI. Therefore, this setting defaults to trim level `5` for configurations using the Visual Editor. | | | string | required | Name for the field that includes the provenance of the documents in the results. If you specify this setting in the storage configuration, Atlas Data Federation returns the following fields for each document in the result: Field NameDescription`provider`Provider ([`stores.[n].provider`](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.stores.-n-.provider)) in the federated database instance storage configuration`clusterName`Name of the Atlas cluster`databaseName`Name of the database on the Atlas cluster`collectionName`Name of the collection`snapshotID`Unique 24-hexadecimal character string that identifies the snapshot`dataSetName`Name of the online archive dataset ([`databases.[n].collections.[n].dataSources.[n].datasetName`)](https://www.mongodb.com/docs/atlas/data-federation/config/adf-config-file-formats.md#mongodb-datalakeconf-adl-datalakeconf-adl.databases.-n-.collections.-n-.dataSources.-n-.datasetName) You can't configure this setting using the Visual Editor in the Atlas UI. | | | integer | required | Unsigned integer that specifies the maximum number of datasets from which to dynamically generate collections for the data source. You must provide a value greater than `0`. You can set this setting for wildcard collections only. Atlas Data Federation returns datasets in reverse lexicographical order. IMPORTANT: You can't configure this setting using the Visual Editor in the Atlas UI. Therefore, Atlas Data Federation configuration doesn't include a limit on the number of datasets for configurations using the Visual Editor. | ### Example Configuration for Atlas Online Archive Data Store Consider an Atlas online archive backed by AWS (Amazon Web Services) with the following ID: ```text v1$atlas$archive$V3metrics$hardware$69437250-b9da-4ae9-a1cd-8811462c38a4$64679f7c09f07374b4dcc914 ``` The online archive contains the archived data from the `metrics.hardware` in an Atlas cluster. The following configuration: - Specifies the Atlas online archive in the `us-east-1` AWS (Amazon Web Services) region as a federated database instance store. - Maps documents from the Atlas online archive to the `dataCenter.inventory` collection in the storage configuration. ```json { "stores" : [ { "name" : "atlasOnlineArchiveStore", "provider" : "dls:aws", "region" : "us-east-1" } ], "databases" : [ { "name" : "dataCenter", "collections" : [ { "name" : "inventory", "dataSources" : [ { "storeName" : "atlasOnlineArchiveStore", "datasetName" : "v1$atlas$archive$V3metrics$hardware$69437250-b9da-4ae9-a1cd-8811462c38a4$64679f7c09f07374b4dcc914", "provider" : "dls:aws" } ] } ] } ] } ``` **Note:** Since AWS (Amazon Web Services) backs the online archive, the `provider` is set to `dls:aws` in the example. If Azure (Microsoft Azure) backed the online archive, the `provider` would be `dls:azure`. Atlas Data Federation maps all the documents in the online archive to the `dataCenter.inventory` collection in the storage configuration. Users connected to the federated database instance can use the MongoDB Query Language and supported aggregations to analyze data in the Atlas cluster through the `dataCenter.inventory` collection.