/ /

Define Data Stores

Docs Home

Deploy a Federated Database Instance Data Store

A federated database instance data store connects Atlas to a cloud storage source so you can run federated queries across your data. You can deploy a data store to AWS S3, Azure Blob Storage, Google Cloud Storage, HTTP or HTTPS endpoints, or an Atlas cluster.

To deploy a federated database instance Data Store, select your data store from the dropdown menu:

Deploy a Federated Database Instance in AWS S3

This page describes how to deploy a federated database instance for accessing data in AWS S3.

Required Access

To deploy a federated database instance, you must have Project Owner access to the project.

Important

Users with Organization Owner access must add themselves as a Project Owner to the project before deploying their federated database instance.

Prerequisites

Before deploying your federated database instance in your AWS S3 bucket, you have to complete the following steps:

Create a MongoDB Atlas account, if you do not have one already.
Install the AWS CLI.
Configure the AWS CLI to access your AWS account. Alternatively, you must have access to the AWS Management Console with permission to create IAM roles.

Optional. Set Up Unified AWS Access.

Procedure

To create a new Data Federation database using the Atlas CLI, run the following command:

atlas dataFederation create <name> [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas dataFederation create.

Deploy a Federated Database Instance in AWS S3

This page describes how to deploy a federated database instance for accessing data in AWS S3.

Required Access

To deploy a federated database instance, you must have Project Owner access to the project.

Important

Users with Organization Owner access must add themselves as a Project Owner to the project before deploying their federated database instance.

Prerequisites

Before deploying your federated database instance in your AWS S3 bucket, you have to complete the following steps:

Create a MongoDB Atlas account, if you do not have one already.
Install the AWS CLI.
Configure the AWS CLI to access your AWS account. Alternatively, you must have access to the AWS Management Console with permission to create IAM roles.

Optional. Set Up Unified AWS Access.

Procedure

Log into MongoDB Atlas

Select Data Federation option from the Services on the left

Create your federated database instance

Click the Create New Federated Database dropdown
Select Manual setup from the menu

Select your federated database instance cloud provider

Select AWS as your cloud provider
Type in your federated database instance name
Click Continue

IMPORTANT: Your federated database instance name defaults to FederatedDatabaseInstance[n]. Once your federated database instance is created, you cannot rename it.

Select your preferred configuration interface method on the top-right corner

For a visual interface, click Visual.
For editing directly in the JSON file, click JSON Editor.

Specify your AWS S3 data store and configure federated databases and virtual collections that map to your data store.

Select the dataset for your federated database instance from the Data Sources section.
Click Add Data Sources to select your data store.
Specify your data store.
Choose Amazon S3 to configure a federated database instance for data in AWS S3 buckets.
Corresponds to stores.[n].provider JSON configuration setting.
Select an AWS IAM role for Atlas.
You can select an existing AWS IAM role that Atlas is authorized for from the role selection dropdown list or choose Authorize an AWS IAM Role to authorize a new role.
If you selected an existing role that Atlas is authorized for, proceed to the next step to list your AWS S3 buckets.
If you are authorizing Atlas for an existing role or are creating a new role, complete the following steps before proceeding to the next step:
1. Select Authorize an AWS IAM Role to authorize a new role or select an existing role from the dropdown and click Next.
2. Use the AWS ARN and unique External ID in the Add Atlas to the trust relationships of your AWS IAM role section to add Atlas to the trust relationships of an existing or new AWS IAM role.
  In the Atlas UI, click and expand one of the following:
  - The Create New Role with the AWS CLI shows how to use the ARN and the unique External ID to add Atlas to the trust relationships of a new AWS IAM role. Follow the steps in the Atlas UI for creating a new role. To learn more, see Create New Role with the AWS CLI.
    
    When authorizing a new role, if you quit the Configure a New Data Lake workflow:
    
    Before validating the role, Atlas will not create the federated database instance. You can go to the Atlas Integrations page to authorize a new role. You can resume the workflow when you have the AWS IAM role ARN.
    
    After validating the role, Atlas will not create the federated database instance. However, the role is available in the role selection dropdown and can be used to create a federated database instance. You do not need to authorize the role again.
  - The Add Trust Relationships to an Existing Role shows how to use the ARN and the unique External ID to add Atlas to the trust relationships of an existing AWS IAM role. Follow the steps in the Atlas UI for adding Atlas to the trust relationship to an existing role. To learn more, see Add Trust Relationships to an Existing Role .
  IMPORTANT: If you modify your custom AWS role ARN in the future, verify that the access policy for the role includes the appropriate access to the S3 resources for the federated database instance.
  For further information, see
  
  Set Up Unified AWS Access
  
  Create a Cloud Provider Access Role
3. Click Next.
Enter the S3 bucket information.
1. Enter the name of your S3 bucket.
  Corresponds to stores.[n].bucket JSON configuration setting.
2. Specify whether the bucket is Read-only or both Read and write.
  Atlas can only query Read-only buckets; if you wish to query and save query results to your S3 bucket, choose Read and write. To save query results to your S3 bucket, the role policy that grants Atlas access to your AWS resources must include the s3:PutObject and s3:DeleteObject permissions in addition to the s3:ListBucket, s3:GetObject, s3:GetObjectVersion, and s3:GetBucketLocation permissions, which grant read access. See step 4 below to learn more about assigning access policy to your AWS IAM role.
3. Select the region of the S3 bucket.
  Corresponds to stores.[n].region JSON configuration setting.
  IMPORTANT: You can't create a federated database instance if Atlas Data Federation is unable to retrieve the region of the specified S3 bucket.
4. Optional. Specify a prefix that Data Federation should use when searching the files in the S3 bucket. If omitted, Data Federation does a recursive search for all files from the root of the S3 bucket.
  Corresponds to stores.[n].prefix JSON configuration setting.
5. Click Next.

Assign an access policy to your AWS IAM role.

Follow the steps in the Atlas user interface to assign an access policy to your AWS IAM role.

Your role policy for read-only or read and write access should look similar to the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        <role arn>
      ]
    }
  ]
}

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetBucketLocation",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        <role arn>
      ]
    }
  ]
}

Click Next.

Define the path structure for your files in the S3 bucket and click Next.
For example:
s3://<bucket-name>/<path>/<to>/<files>/<filename>.<file-extension>
To add additional paths to data on your S3 bucket, click Add Data Source and enter the path. To learn more about paths, see Define Path Syntax for Data Stores.
Corresponds to databases.[n].collections.[n].dataSources.[n].path JSON configuration setting.
Create the virtual databases, collections, and views and map the databases, collections, and views to your data store.
1. (Optional) Click the for the:
  - Database to edit the database name. Defaults to VirtualDatabase[n].
    
    Corresponds to databases.[n].name JSON configuration setting.
  - Collection to edit the collection name. Defaults to VirtualCollection[n].
    
    Corresponds to databases.[n].collections.[n].name JSON configuration setting.
  - View to edit the view name.
  You can click:
  - Add Database to add databases and collections.
  - associated with the database to add collections to the database.
  - associated with the collection to add views on the collection. To create a view, you must specify:
    
    The name of the view.
    
    The pipeline to apply to the view.
    
    The view definition pipeline cannot include the $out or the $merge stage. If the view definition includes nested pipeline stages such as $lookup or $facet, this restriction applies to those nested pipelines as well.
    
    To learn more about views, see:
    
    Views
    
    db.createView
  - associated with the database, collection, or view to remove it.
2. Select AWS S3 from the dropdown in the Data Sources section.
3. Drag and drop the data store to map with the collection.
  Corresponds to databases.[n].collections.[n].dataSources JSON configuration setting.

Your configuration for AWS S3 data store should look similar to the following:

{
  "stores" : [
    {
      "name" : "<string>",
      "provider": "<string>",
      "region" : "<string>",
      "bucket" : "<string>",
      "additionalStorageClasses" : ["<string>"],
      "prefix" : "<string>",
      "includeTags": <boolean>,
      "delimiter": "<string>",
      "public": <boolean>
    }
  ],
  "databases" : [
    {
      "name" : "<string>",
      "collections" : [
        {
          "name" : "<string>",
          "dataSources" : [
            {
              "storeName" : "<string>",
              "path" : "<string>",
              "defaultFormat" : "<string>",
              "provenanceFieldName": "<string>",
              "omitAttributes": true | false
            }
          ]
        }
      ],
      "maxWildcardCollections" : <integer>,
      "views" : [ 
        {
          "name" : "<string>", 
          "source" : "<string>", 
          "pipeline" : "<string>" 
        }
      ] 
    }
  ]
}

For more information on the configuration settings, see Configure Data Stores for a Federated Database Instance.

Define your AWS S3 data store.

Edit the JSON configuration settings shown in the UI for stores. Your stores cofiguration setting should resemble the following:

   "stores" : [
     {
       "name" : "<string>",
       "provider" : "<string>",
       "region" : "<string>",
       "bucket" : "<string>",
       "additionalStorageClasses" : ["<string>"],
       "prefix" : "<string>",
       "delimiter" : "<string>",
       "includeTags": <boolean>,
       "public": <boolean>
     }
   ]

To learn more about these configuration settings, see stores.

Define your federated database instance virtual databases, collections, and views.

Edit the JSON configuration settings shown in the UI for databases. Your databases cofiguration setting should resemble the following:

"databases" : [
  {
    "name" : "<string>",
    "collections" : [
      {
        "name" : "<string>",
        "dataSources" : [
          {
            "storeName" : "<string>",
            "defaultFormat" : "<string>",
            "path" : "<string>",
            "provenanceFieldName": "<string>",
            "omitAttributes": <boolean>
          }
        ]
      }
    ], 
    "maxWildcardCollections" : <integer>,
    "views" : [
      {
        "name" : "<string>",
        "source" : "<string>",
        "pipeline" : "<string>"
      }
    ]
  }
]

To learn more about these configuration settings, see databases.

Optional: Repeat steps 1 to 6 to define additional AWS S3 data stores.

Click the Save button to create the federated database instance

Deploy a Federated Database Instance in Microsoft Azure

This page describes how to deploy a federated database instance for accessing data in your Azure Blob Storage containers.

Prerequisites

Before you begin, complete the following prerequisites:

Create a MongoDB Atlas account, if you don't have one already.
Install Azure PowerShell or Azure CLI. To learn more about these tools, see Choose the right Azure command-line tool.
Configure Azure PowerShell or Azure CLI.
Optional. Set Up Azure Service Principal Access.

Procedure

To create a new Data Federation database using the Atlas CLI, run the following command:

atlas dataFederation create <name> [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas dataFederation create.

Deploy a Federated Database Instance in Microsoft Azure

This page describes how to deploy a federated database instance for accessing data in your Azure Blob Storage containers.

Prerequisites

Before you begin, complete the following prerequisites:

Create a MongoDB Atlas account, if you don't have one already.
Install Azure PowerShell or Azure CLI. To learn more about these tools, see Choose the right Azure command-line tool.
Configure Azure PowerShell or Azure CLI.
Optional. Set Up Azure Service Principal Access.

Procedure

Log in to MongoDB Atlas.

Select the Data Federation option on the left-hand navigation.

Create a federated database instance.

Click the Create New Federated Database dropdown.
Select Manual Setup.

Select the cloud provider where Atlas Data Federation will process your queries against your federated database instance.

You can select AWS, Azure, or Google Cloud. Once your federated database instance is created, you can't change the cloud provider where Atlas Data Federation processes your queries.

If you are configuring a federated database instance for data in an Azure Blob Storage container, you can't choose a cloud provider that is different from the cloud provider that is hosting your data. That is, you must choose Azure.

Type a name for your federated database instance in the Federated Database Instance Name field and click Continue.

Defaults to FederatedDatabaseInstance[n]. Once your federated database instance is created, you can't change its name.

Select the configuration method.

For a guided experience, click Visual Editor.
To edit the raw JSON, click JSON Editor.

Specify your Azure Blob Storage data store and configure federated database instance and virtual collections that map to your data store.

Select the dataset for your federated database instance from the Data Sources section.
Click Add Data Sources to select your data store.
Specify your data store.
Choose Azure to configure a federated database instance for data in Azure Blob Storage containers.
Corresponds to stores.[n].provider JSON configuration setting.
Select an Azure Service Principal for Atlas.
You can select an existing Azure Service Principal that Atlas is authorized for from the Service Principal dropdown or choose Authorize an Azure Service Principal to authorize a new service principal.
If you selected an existing service principal that Atlas is authorized for, proceed to the next step.
If you are authorizing Atlas for an existing service principal or are creating a new service principal, complete the following steps before proceeding to the next step:
1. Select Authorize an Azure Service Principal to authorize a new or existing service principal and click Continue.
  Use the Azure Service Principal AppId in the Add Atlas to your Azure Service Principal section to grant Atlas access through an existing or new Azure Service Principal.
2. Follow the steps in the PowerShell or AzureCLI tab in the Atlas UI to create a new service prinicpal or modify an existing service principal.
3. Enter the tenant ID and Service Principal ID in the respective fields after completing the steps in the Atlas UI.
4. Click Validate and Finish to proceed to the next step.
Configure access to your Azure Blob Storage.
In the Configure Azure Blob Storage page, you must configure Azure Storage Account credential delegation and Storage Container access. To do these:
1. Enter your Storage Account Resource ID in the Storage Account Credential Delegation field.
  To learn more, see Get the resource ID for a storage account.
2. Copy and run the command from the Storage Account Credential Delegation step in your Azure PowerShell to set up the credentials delegation.
3. Specify whether the storage container allows Read only or Read and write operations.
  Atlas can only query Read-only containers; if you wish to query and save query results to your Azure Blob Storage container, choose Read and write.
  Atlas Data Federation doesn't support writes to your Azure Blob Storage container using $out.
4. Enter your Storage Container name.
5. Copy and run the command shown from the Storage Container Access step in your Azure PowerShell to set up blob container access.
6. Click Continue.
Define the path structure for your files in the Azure Blob Storage container and click Next.
1. Enter the storage path to your Azure Blob Storage container.
  For example:
  https://<storage-account>.blob.core.windows.net/<container>/<file-name>
  To add additional paths to data on your Azure Blob Storage container, click Add Data Source and enter the path. To learn more about paths, see Define Path Syntax for Data Stores.
  Corresponds to databases.[n].collections.[n].dataSources.[n].path JSON configuration setting.
2. Optional. Specify the partition fields that Data Federation should use when searching the files in the Azure Blob Storage container and the field value type.
  If omitted, Data Federation does a recursive search for all files from the root of the Azure Blob Storage container. If you don't select a specific field value type, Data Federation adds any value in that path in all queries.
  Corresponds to stores.[n].prefix and databases.[n].collections.[n].dataSources.[n].path JSON configuration settings.
Create the virtual databases, collections, and views and map the databases, collections, and views to your data store.
1. (Optional) Click the for the:
  - Database to edit the database name. Defaults to VirtualDatabase[n].
    
    Corresponds to databases.[n].name JSON configuration setting.
  - Collection to edit the collection name. Defaults to VirtualCollection[n].
    
    Corresponds to databases.[n].collections.[n].name JSON configuration setting.
  - View to edit the view name.
  You can click:
  - Add Database to add databases and collections.
  - associated with the database to add collections to the database.
  - associated with the collection to add views on the collection. To create a view, you must specify:
    
    The name of the view.
    
    The pipeline to apply to the view.
    
    The view definition pipeline cannot include the $out or the $merge stage. If the view definition includes nested pipeline stages such as $lookup or $facet, this restriction applies to those nested pipelines as well.
    
    To learn more about views, see:
    
    Views
    
    db.createView
  - associated with the database, collection, or view to remove it.
2. Select Azure Blob Storage from the dropdown in the Data Sources section.
3. Drag and drop the data store to map with the collection.
  Corresponds to databases.[n].collections.[n].dataSources JSON configuration setting.

Your configuration for Azure Blob Storage data store should look similar to the following:

{
    "stores" : [
      {
        "name" : "<string>",
        "provider": "<string>",
        "region" : "<string>",
        "serviceURL" : "<string>",
        "containerName" : "<string>",
        "delimiter" : "<string>",
        "prefix": "<string>",
        "public": <boolean>
      }
    ],
    "databases" : [
      {
        "name" : "<string>",
        "collections" : [
          {
            "name" : "<string>",
            "dataSources" : [
              {
                "storeName" : "<string>",
                "path" : "<string>",
                "defaultFormat" : "<string>",
                "provenanceFieldName": "<string>",
                "omitAttributes": <boolean>
              }
            ]
          }
        ],
        "maxWildcardCollections" : <integer>,
        "views" : [ 
          {
            "name" : "<string>", 
            "source" : "<string>", 
            "pipeline" : "<string>" 
          }
        ] 
      }
    ]
  }

For more information on the configuration settings, see Configure Data Stores for a Federated Database Instance.

Define your Azure Blob Storage data store.

Edit the JSON configuration settings shown in the UI for stores. Your stores cofiguration setting should resemble the following:

"stores" : [
    {
      "name" : "<string>",
      "provider" : "<string>",
      "region" : "<string>",
      "serviceURL" : "<string>",
      "containerName" : "<string>",
      "delimiter": "<string",
      "prefix" : "<string>",
      "public": <boolean>
    }
  ]

To learn more about these configuration settings, see stores.

Define your federated database instance virtual databases, collections, and views.

Edit the JSON configuration settings shown in the UI for databases. Your databases cofiguration setting should resemble the following:

"databases" : [
  {
    "name" : "<string>",
    "collections" : [
      {
        "name" : "<string>",
        "dataSources" : [
          {
            "storeName" : "<string>",
            "defaultFormat" : "<string>",
            "path" : "<string>",
            "provenanceFieldName": "<string>",
            "omitAttributes": <boolean>
          }
        ]
      }
    ], 
    "maxWildcardCollections" : <integer>,
    "views" : [
      {
        "name" : "<string>",
        "source" : "<string>",
        "pipeline" : "<string>"
      }
    ]
  }
]

To learn more about these configuration settings, see databases.

Optional: Repeat steps in the Visual Editor or JSON Editor tab above to define additional Azure Blob Storage data stores.

To add other data stores for federated queries, see:

Create Atlas Data Store From the UI
Create HTTP or HTTPS Data Store From the UI

Note

You can't create an AWS data store for running federated queries across different cloud providers.

Click Save to create the federated database instance.

Deploy a Federated Database Instance in Google Cloud

This page describes how to deploy a federated database instance for accessing data in your Google Cloud Storage buckets.

Required Access

To deploy a federated database instance, you must have Project Owner access to the project.

Important

Users with Organization Owner access must add themselves as a Project Owner to the project before deploying their federated database instance.

Prerequisites

Before you begin, you will need to:

Create a MongoDB Atlas account, if you do not have one already.
Install the gcloud CLI.
Configure the gcloud CLI to access your Google Cloud account. Alternatively, you must have access to the Google Cloud console with permission to create IAM roles.
Optional. Set Up GCP Service Account Access.

Procedure

To create a new Data Federation database using the Atlas CLI, run the following command:

atlas dataFederation create <name> [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas dataFederation create.

Deploy a Federated Database Instance in Google Cloud

This page describes how to deploy a federated database instance for accessing data in your Google Cloud Storage buckets.

Required Access

To deploy a federated database instance, you must have Project Owner access to the project.

Important

Users with Organization Owner access must add themselves as a Project Owner to the project before deploying their federated database instance.

Prerequisites

Before you begin, you will need to:

Create a MongoDB Atlas account, if you do not have one already.
Install the gcloud CLI.
Configure the gcloud CLI to access your Google Cloud account. Alternatively, you must have access to the Google Cloud console with permission to create IAM roles.
Optional. Set Up GCP Service Account Access.

Procedure

Log in to MongoDB Atlas.

Select the Data Federation option on the left-hand navigation.

Create a federated database instance.

Click the Create New Federated Database dropdown.
Select Manual Setup.

Select the cloud provider where Atlas Data Federation will process your queries against your federated database instance.

You can select AWS, Azure, or Google Cloud. Once your federated database instance is created, you can't change the cloud provider where Atlas Data Federation processes your queries.

You must configure your federated database instance on the same cloud provider as the data stores to which your federated database instance maps.

Type a name for your federated database instance in the Federated Database Instance Name field and click Continue.

Defaults to FederatedDatabaseInstance[n]. Once your federated database instance is created, you can't change its name.

Select the configuration method.

For a guided experience, click Visual Editor.
To edit the raw JSON, click JSON Editor.

Specify your Google Cloud Storage bucket and configure federated databases and virtual collections that map to your data store.

Select the dataset for your federated database instance from the Data Sources section.
Click Add Data Sources to select your data store.
Specify your data store.
Choose Google Cloud Storage to configure a federated database instance for data in Google Cloud Storage buckets.
Corresponds to stores.[n].provider JSON configuration setting.
Select a Google Cloud Service Account for Atlas.
You can select an existing Google Cloud Service Account that Atlas is authorized for from the role selection dropdown list or choose Create a Google Service Account.
If you selected an existing account that Atlas is authorized for, click Next and proceed to the next step to list your Google Cloud Storage buckets.
If you are creating a new service account, select Create a Google Service Account and click Next.
Tip
- Set Up GCP Service Account Access.
- Create a Cloud Provider Access Role
In the Configure Google Cloud Storage modal, follow the provided instructions to configure the Google Cloud CLI, then click Next.
Configure Google Cloud Storage.
1. Enter the name of your Google Cloud Storage bucket.
  Corresponds to the stores.[n].bucket JSON configuration setting.
2. Specify whether the bucket is Read-only or both Read and write.
  Atlas can only query Read-only buckets; if you wish to query and save query results to your Google Cloud Storage bucket, choose Read and write.
3. Select the region of the Google Cloud Storage bucket.
  Corresponds to the stores.[n].region JSON configuration setting.
  Note
  You can't create a federated database instance if Atlas Data Federation is unable to retrieve the region of the specified Google Cloud Storage bucket.
4. Grant access to your Google Cloud project.
  1. In the Google Cloud console for the project that hosts your Google Cloud Storage bucket, navigate to IAM and Admin, then navigate to IAM.
  2. Click Grant Access. In the modal that appears, in the New principals field, enter the Google Cloud Service Account associated with your federated database instance.
  3. To grant read-only access to the bucket, apply the storage.viewer role. To grant read-write access to the bucket, additionally apply the storage.editor role.
5. Optional. Specify a prefix that Data Federation should use when searching the files in the Google Cloud Storage bucket. If omitted, Data Federation does a recursive search for all files from the root of the Google Cloud Storage bucket.
  Corresponds to the stores.[n].prefix JSON configuration setting.
6. Click Validate and finish.
Define the path structure for your files in the Google Cloud Storage bucket and click Next.
For example:
https://storage.googleapis.com/<path>/<to>/<files>/<filename>.<file-extension>
To add additional paths to data on your Google Cloud Storage bucket, click Add Data Source and enter the path. To learn more about paths, see Define Path Syntax for Data Stores.
Corresponds to the databases.[n].collections.[n].dataSources.[n].path JSON configuration setting.
Create the virtual databases, collections, and views and map the databases, collections, and views to your data store.
1. (Optional) Click the for the:
  - Database to edit the database name. Defaults to VirtualDatabase[n].
    
    Corresponds to databases.[n].name JSON configuration setting.
  - Collection to edit the collection name. Defaults to VirtualCollection[n].
    
    Corresponds to databases.[n].collections.[n].name JSON configuration setting.
  - View to edit the view name.
  You can click:
  - Add Database to add databases and collections.
  - associated with the database to add collections to the database.
  - associated with the collection to add views on the collection. To create a view, you must specify:
    
    The name of the view.
    
    The pipeline to apply to the view.
    
    The view definition pipeline cannot include the $out or the $merge stage. If the view definition includes nested pipeline stages such as $lookup or $facet, this restriction applies to those nested pipelines as well.
    
    To learn more about views, see:
    
    Views
    
    db.createView
  - associated with the database, collection, or view to remove it.
2. Select Google Cloud Storage from the dropdown in the Data Sources section.
3. Drag and drop the data store to map with the collection.
  Corresponds to databases.[n].collections.[n].dataSources JSON configuration setting.

Your configuration for Google Cloud Storage data store should look similar to the following:

{
  "stores" : [
    {
      "name" : "<string>",
      "provider" : "<string>",
      "region" : "<string>",
      "bucket" : "<string>",
      "prefix": "<string>",
      "delimiter": "<string>"
    }
  ],
  "databases" : [
    {
      "name" : "<string>",
      "collections" : [
	{
	  "name" : "<string>",
	  "dataSources" : [
	    {
	      "storeName" : "<string>",
	      "path" : "<string>",
	      "defaultFormat" : "<string>",
	      "provenanceFieldName": "<string>",
	      "omitAttributes": <boolean>
	    }
	  ]
	}
      ],
      "maxWildcardCollections" : <integer>,
      "views" : [ 
	{
	  "name" : "<string>", 
	  "source" : "<string>", 
	  "pipeline" : "<string>" 
	}
      ] 
    }
  ]
}

For more information on the configuration settings, see Configure Data Stores for a Federated Database Instance.

Define your AWS S3 data store.

Edit the JSON configuration settings shown in the UI for stores. Your stores configuration setting should resemble the following:

   "stores" : [
     {
       "name" : "<string>",
       "provider" : "<string>",
       "region" : "<string>",
       "bucket" : "<string>",
       "additionalStorageClasses" : ["<string>"],
       "prefix" : "<string>",
       "delimiter" : "<string>",
       "includeTags": <boolean>,
       "public": <boolean>
     }
   ]

To learn more about these configuration settings, see stores.

Define your federated database instance virtual databases, collections, and views.

Edit the JSON configuration settings shown in the UI for databases. Your databases configuration setting should resemble the following:

"databases" : [
  {
    "name" : "<string>",
    "collections" : [
      {
        "name" : "<string>",
        "dataSources" : [
          {
            "storeName" : "<string>",
            "defaultFormat" : "<string>",
            "path" : "<string>",
            "provenanceFieldName": "<string>",
            "omitAttributes": <boolean>
          }
        ]
      }
    ], 
    "maxWildcardCollections" : <integer>,
    "views" : [
      {
        "name" : "<string>",
        "source" : "<string>",
        "pipeline" : "<string>"
      }
    ]
  }
]

To learn more about these configuration settings, see databases.

Optional: Repeat steps in the Visual Editor or JSON Editor tab above to define additional Azure Azure Blob Storage data stores.

To add other data stores for federated queries, see:

Create Atlas Data Store From the UI
Create HTTP or HTTPS Data Store From the UI

Note

You can't connect an Azure Blob Storage data store for running federated queries across cloud providers.

Click Save to create the federated database instance.

Deploy a Federated Database Instance in an HTTP Data Store

This page describes how to deploy a federated database instance for accessing data in an HTTP data store.

Required Access

To deploy a federated database instance, you must have Project Owner access to the project.

Important

Users with Organization Owner access must add themselves as a Project Owner to the project before deploying their federated database instance.

Prerequisites

Before you begin, you will need to:

Create a MongoDB Atlas account, if you do not have one already.
Format your data store using one of the supported data formats.

Note
If your file format is CSV or TSV, you must include a header row in your data. See CSV and TSV for more information.
Make your data store accessible over the public Internet.

Important

If your HTTP data store is not accessible over HTTPS, you must use the JSON Editor to configure your data store. In your JSON configuration, you must set the stores.[n].allowInsecure setting to true.
Atlas Data Federation does not support HTTP data store URLs that require authentication.

Procedure

To create a new Data Federation database using the Atlas CLI, run the following command:

atlas dataFederation create <name> [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas dataFederation create.

Deploy a Federated Database Instance in an HTTP Data Store

This page describes how to deploy a federated database instance for accessing data in an HTTP data store.

Required Access

To deploy a federated database instance, you must have Project Owner access to the project.

Important

Users with Organization Owner access must add themselves as a Project Owner to the project before deploying their federated database instance.

Prerequisites

Before you begin, you will need to:

Create a MongoDB Atlas account, if you do not have one already.
Format your data store using one of the supported data formats.

Note
If your file format is CSV or TSV, you must include a header row in your data. See CSV and TSV for more information.
Make your data store accessible over the public Internet.

Important

If your HTTP data store is not accessible over HTTPS, you must use the JSON Editor to configure your data store. In your JSON configuration, you must set the stores.[n].allowInsecure setting to true.
Atlas Data Federation does not support HTTP data store URLs that require authentication.

Procedure

Log in to MongoDB Atlas.

Select the Data Federation option on the left-hand navigation.

Create a federated database instance.

Click the Create New Federated Database dropdown.
Select Manual Setup.

Select the cloud provider where Atlas Data Federation will process your queries against your federated database instance.

You can select AWS, Azure, or Google Cloud. Once your federated database instance is created, you can't change the cloud provider where Atlas Data Federation processes your queries.

Type a name for your federated database instance in the Federated Database Instance Name field and click Continue.

Defaults to FederatedDatabaseInstance[n]. Once your federated database instance is created, you can't change its name.

Select the configuration method.

For a guided experience, click Visual Editor.
To edit the raw JSON, click JSON Editor.

Specify your data store.

Select the dataset for your federated database instance from the Data Sources section.
Click Add Data Sources to select your data store.
Specify your data store.
Choose HTTP(S) to configure a federated database instance for data in publicly accessible HTTP and HTTPS URLs.
Corresponds to stores.[n].provider JSON configuration setting.
Enter a name for your HTTP data store into the HTTP(S) Store Name field.
Note
The data store's name must be unique within your federated database instance.
Corresponds to stores.[n].name JSON configuration setting.
Enter the publicly accessible URL of the file where data is stored.
Atlas Data Federation supports JSON, BSON, CSV, TSV, Avro (gzipped or uncompressed), Parquet, and ORC file types.
Tip
Click Use Sample URL to add a sample HTTP data store.
For each additional HTTP data store that you want to add, click Add Another URL, then enter the HTTP data store URLs.
Corresponds to stores.[n].urls JSON configuration setting.
Click Next to configure virtual databases and collections.
Create the virtual databases, collections, and views and map the databases, collections, and views to your data store.
1. (Optional) Click the for the:
  - Database to edit the database name. Defaults to VirtualDatabase[n].
    
    Corresponds to databases.[n].name JSON configuration setting.
  - Collection to edit the collection name. Defaults to VirtualCollection[n].
    
    Corresponds to databases.[n].collections.[n].name JSON configuration setting.
  - View to edit the view name.
  You can click:
  - Add Database to add databases and collections.
  - associated with the database to add collections to the database.
  - associated with the collection to add views on the collection. To create a view, you must specify:
    
    The name of the view.
    
    The pipeline to apply to the view.
    
    The view definition pipeline cannot include the $out or the $merge stage. If the view definition includes nested pipeline stages such as $lookup or $facet, this restriction applies to those nested pipelines as well.
    
    To learn more about views, see:
    
    Views
    
    db.createView
  - associated with the database, collection, or view to remove it.
2. Select HTTP from the dropdown in the Data Sources section.
3. Drag and drop the data store to map with the collection.
  Corresponds to databases.[n].collections.[n].dataSources JSON configuration setting.

Your configuration for an HTTP data store should look similar to the following:

1 {
2   "stores" : [
3     {
4       "name" : "<string>",
5       "provider": "<string>",
6       "defaultFormat" : "<string>",
7       "allowInsecure": <boolean>,
8        "urls": ["<string>"]
9     }
10   ],
11   "databases" : [
12     {
13       "name" : "<string>",
14       "collections" : [
15         {
16           "name" : "<string>",
17           "dataSources" : [
18             {
19               "storeName" : "<string>",
20               "allowInsecure" : <boolean>,
21               "urls" : ["<string>"],
22               "defaultFormat" : "<string>",
23               "provenanceFieldName": "<string>"
24             }
25           ]
26         }
27       ],
28       "views" : [ 
29         {
30           "name" : "<string>", 
31           "source" : "<string>", 
32           "pipeline" : "<string>" 
33         }
34       ] 
35     }
36   ]
37 }

For more information on the configuration settings, see HTTP URL.

Define your HTTP data store.
Edit the JSON configuration settings shown in the UI for stores. Your stores cofiguration setting should resemble the following:
"stores" : [ { "name" : "<string>", "provider" : "<string>", "allowInsecure": <boolean>, "urls" : ["<string>"], "defaultFormat" : "<string>" } ]
To learn more about these configuration settings, see stores.

Define your federated database instance virtual databases, collections, and views.

Edit the JSON configuration settings shown in the UI for databases. Your databases cofiguration setting should resemble the following:

"databases" : [
  {
    "name" : "<string>",
    "collections" : [
      {
        "name" : "<string>",
        "dataSources" : [
          {
            "storeName" : "<string>",
            "allowInsecure" : <boolean>,
            "urls" : ["<string>"],
            "defaultFormat" : "<string>",
            "provenanceFieldName": "<string>"
          }
        ]
      }
    ]
  }
]

To learn more about these configuration settings, see databases.

Optional: Repeat steps in the Visual Editor or JSON Editor tab above to define additional HTTP URLs.

To add other data stores for federated queries, see:

Create AWS Data Store From the UI or Create Azure Data Store From the UI
Create Atlas Data Store From the UI

Click Save to create the federated database instance.

Deploy a Federated Database Instance with Atlas Cluster Data

This page describes how to deploy a federated database instance for accessing data in an Atlas cluster.

Required Access

To deploy a federated database instance, you must have Project Owner access to the project.

Important

Users with Organization Owner access must add themselves as a Project Owner to the project before deploying their federated database instance.

Prerequisites

Before you begin, you will need to:

Create a MongoDB Atlas account, if you do not have one already.
Create an Atlas Cluster, if you do not have one already. Atlas Data Federation supports Atlas clusters deployed to AWS, Azure, or Google Cloud.

Note
To use your Atlas cluster as a data store, you must deploy it to the same project as your federated database instance.
Add data to at least one collection on your Atlas cluster if you have not already.

Procedure

To create a new Data Federation database using the Atlas CLI, run the following command:

atlas dataFederation create <name> [options]

To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas dataFederation create.

Deploy a Federated Database Instance with Atlas Cluster Data

This page describes how to deploy a federated database instance for accessing data in an Atlas cluster.

Required Access

To deploy a federated database instance, you must have Project Owner access to the project.

Important

Users with Organization Owner access must add themselves as a Project Owner to the project before deploying their federated database instance.

Prerequisites

Before you begin, you will need to:

Create a MongoDB Atlas account, if you do not have one already.
Create an Atlas Cluster, if you do not have one already. Atlas Data Federation supports Atlas clusters deployed to AWS, Azure, or Google Cloud.

Note
To use your Atlas cluster as a data store, you must deploy it to the same project as your federated database instance.
Add data to at least one collection on your Atlas cluster if you have not already.

Procedure

Log in to MongoDB Atlas.

Select the Data Federation option on the left-hand navigation.

Create a federated database instance.

Click the Create New Federated Database dropdown.
Select Manual Setup.

Select the cloud provider where Atlas Data Federation will process your queries against your federated database instance.

You can select AWS, Azure, or Google Cloud. Once your federated database instance is created, you can't change the cloud provider where Atlas Data Federation processes your queries.

If you want to query data in an Atlas cluster, we recommend that you select that same cloud provider as the cloud provider for your cluster. To query data in object storage such as AWS S3 or Azure Blob Storage, you must select the same cloud for your federated database instance as the cloud for your object storage.

Type a name for your federated database instance in the Federated Database Instance Name field and click Continue.

Defaults to FederatedDatabaseInstance[n]. Once your federated database instance is created, you can't change its name.

Select the configuration method.

For a guided experience, click Visual Editor.
To edit the raw JSON, click JSON Editor.

Specify your data store.

Select the dataset for your federated database instance from the Data Sources section.
Click Add Data Sources to select your data store.

Specify your data source.

Choose Atlas Cluster to configure a federated database instance for data on your Atlas cluster.
Corresponds to stores.[n].provider JSON configuration setting.
Select the Atlas cluster that you want to use as a data store in the Provide Namespaces in this project section.
Corresponds to stores.[n].clusterName JSON configuration setting.
Expand the databases and select the collections that you want to add to your federated database instance.
To filter the databases and collections, enter text into the Search database or collection field. The dialog box displays only databases and collections with names that match your search criteria.
Corresponds to the databases.[n].collections.[n].dataSources.[n].database and databases.[n].collections.[n].dataSources.[n].collection JSON configuration settings.

Optional. Specify the Cluster Read Preference settings by expanding the section.

Corresponds to stores.[n].readPreference.

Field Name	Description
Read Preference Mode	Specifies the replica set member to which you want to route the read requests. You can choose one of the following from the dropdown: `primary` - to route all read requests to the replica set primary `primaryPreferred` - to route all read requests the replica set primary and to secondary members only if `primary` is unavailable `secondary` - to route all read requests to the secondary members of the replica set `secondaryPreferred` - to route all read requests to the secondary members of the replica set and the primary on sharded clusters only if `secondary` members are unavailable `nearest` - to route all read requests to random eligible replica set member, irrespective of whether that member is a primary or secondary IMPORTANT If you add an Atlas cluster as a store, the default value is `secondary`. If you don't set anything in your federated database instance storage configuration, the default value is `nearest`. To learn more, see Read preference mode. Corresponds to `stores.[n].readPreference.mode`.
TagSets	Specifies the list of tags or tag specification documents that contain name and value pairs for the replica set member to which you want to route read requests. To learn more, see Read Preference Tag Sets. Corresponds to `stores.[n].readPreference.tagSets`.
Maxstaleness Seconds	Specifies the maximum replication lag, or "staleness", for reads from secondaries. To learn more, see Read Preference maxStalenessSeconds. Corresponds to `stores.[n].readPreference.maxStalenessSeconds`.

Click Next.

Create the virtual databases, collections, and views and map the databases, collections, and views to your data store.
1. (Optional) Click the for the:
  - Database to edit the database name. Defaults to VirtualDatabase[n].
    
    Corresponds to databases.[n].name JSON configuration setting.
  - Collection to edit the collection name. Defaults to VirtualCollection[n].
    
    Corresponds to databases.[n].collections.[n].name JSON configuration setting.
  - View to edit the view name.
  You can click:
  - Add Database to add databases and collections.
  - associated with the database to add collections to the database.
  - associated with the collection to add views on the collection. To create a view, you must specify:
    
    The name of the view.
    
    The pipeline to apply to the view.
    
    The view definition pipeline cannot include the $out or the $merge stage. If the view definition includes nested pipeline stages such as $lookup or $facet, this restriction applies to those nested pipelines as well.
    
    To learn more about views, see:
    
    Views
    
    db.createView
  - associated with the database, collection, or view to remove it.
2. Select Atlas Cluster from the dropdown in the Data Sources section.
3. Drag and drop the data store to map with the collection.
  Corresponds to databases.[n].collections.[n].dataSources JSON configuration setting.

Your configuration for an Atlas cluster data store should look similar to the following:

1 {
2   "stores" : [
3     {
4       "name" : "<string>",
5       "provider": "<string>",
6       "clusterName": "<string>", 
7       "projectId": "<string>",
8       "readPreference": {
9         "mode": "<string>",
10         "tagSets": [
11           [{"name": "<string>", "value": "<string>"},...],
12           ...
13         ],
14         "maxStalenessSeconds": <int>
15       }
16     }
17   ],
18   "databases" : [
19     {
20       "name" : "<string>",
21       "collections" : [
22         {
23           "name" : "<string>",
24           "dataSources" : [
25             {
26               "storeName" : "<string>",
27               "database" : "<string>",
28               "databaseRegex": "<string>",
29               "collection" : "<string>",
30               "collectionRegex" : "<string>",
31               "provenanceFieldName": "<string>"
32             }
33           ]
34         }
35       ],
36       "views" : [ 
37         {
38           "name" : "<string>", 
39           "source" : "<string>", 
40           "pipeline" : "<string>" 
41         }
42       ] 
43     }
44   ]
45 }

To learn more about these configuration settings, see Configure Data Stores for a Federated Database Instance.

Define your Atlas data store.

Edit the JSON configuration settings shown in the UI for stores. Your stores cofiguration setting should resemble the following:

"stores" : [
  {
    "name" : "<string>",
    "provider" : "<string>",
    "clusterName" : "<string>",
    "projectId": "<string>"
    "readPreference": {
      "mode": "<string>",
      "tagSets": [
        [{"name": "<string>", "value": "<string>"},...],
        ...
      ],
      "maxStalenessSeconds": <int>
    },
    "readConcern": {
      "level": "<string>"
    }
  }
]

To learn more about these configuration settings, see stores.

Define your federated database instance virtual databases, collections, and views.

Edit the JSON configuration settings shown in the UI for databases. Your databases cofiguration setting should resemble the following:

"databases" : [
  {
    "name" : "<string>",
    "collections" : [
      {
        "name" : "<string>",
        "dataSources" : [
          {
            "storeName" : "<string>",
            "database" : "<string>",
            "databaseRegex": "<string>",
            "collection" : "<string>",
            "collectionRegex" : "<string>",
            "provenanceFieldName": "<string>" 
          }
        ]
      }
    ]
  }
]

To learn more about these configuration settings, see databases.

Optional: Repeat steps in the Visual Editor or JSON Editor tab above to define additional Atlas clusters.

To add other data stores for federated queries, see:

Create AWS Data Store From the UI or Create Azure Data Store From the UI
Create HTTP or HTTPS Data Store From the UI

Click Save to create the federated database instance.

Back

Define Data Stores

Define a Configuration File

1	{
2	"stores" : [
3	{
4	"name" : "<string>",
5	"provider": "<string>",
6	"defaultFormat" : "<string>",
7	"allowInsecure": <boolean>,
8	"urls": ["<string>"]
9	}
10	],
11	"databases" : [
12	{
13	"name" : "<string>",
14	"collections" : [
15	{
16	"name" : "<string>",
17	"dataSources" : [
18	{
19	"storeName" : "<string>",
20	"allowInsecure" : <boolean>,
21	"urls" : ["<string>"],
22	"defaultFormat" : "<string>",
23	"provenanceFieldName": "<string>"
24	}
25	]
26	}
27	],
28	"views" : [
29	{
30	"name" : "<string>",
31	"source" : "<string>",
32	"pipeline" : "<string>"
33	}
34	]
35	}
36	]
37	}