Manage Atlas Data Federation Query Limits
Overview
You can limit the amount of data that Atlas Data Federation processes for your federated database instances to control costs. To limit the amount of processed data, you can configure query limits per federated database instance and for all federated database instances in your project. The query limits that you configure only apply to data processing costs and don't apply to other Atlas Data Federation costs such as data retrieval and transfer.
You can configure one limit per query, per day, per week, and per month per project and per federated database instance. Atlas Data Federation considers each limit of each type independent of other configured limits. If you configure the same type of limit for the project and for a federated database instance in the project, Atlas Data Federation enforces the more restrictive limit of the two. Limit values must be between 1GiB (1024^3 bytes) and 1EiB.
By default, Atlas Data Federation sets a 100TB limit on the amount of processed data per month for all new federated database instances.
Note
Atlas Data Federation enforces the project limit per cloud object storage. For example, suppose you have 2 federated database instances, one on AWS and another on Azure. Atlas Data Federation allows scans of up to 100TB per federated database instance for a total of 200TB for the project.
What Happens When Atlas Data Federation Reaches the Data Limit?
When the amount of processed data reaches the configured limit amount, Atlas Data Federation stops processing data for the query that has reached the limit if the limit type is per query. For limits of other types, Atlas Data Federation doesn't execute any new queries until the limit resets based on the configured limit type. That is, Atlas Data Federation doesn't execute any new queries until:
The next calendar day beginning at
00:00UTC for limit type of per day.The next calendar week beginning on Monday at
00:00UTC for limit type of per week.The next calendar month beginning on the first day (1st) of the month at
00:00UTC for limit type of per month.
If you configure the same type of limit for both the project and federated database instances in the project, but with different limit amounts for the project and the federated database instances in the project, the following apply:
If Atlas Data Federation reaches the project limit amount before reaching the limit amount for any federated database instance in the project, Atlas Data Federation allows all running queries against all federated database instances in the project to complete.
If Atlas Data Federation doesn't reach the limit amount for the project, but reaches the limit amount set for any federated database instance in the project, Atlas Data Federation allows all running queries against that federated database instance to complete, but doesn't execute any new queries against that federated database instance. Atlas Data Federation continues to execute queries against other federated database instances in the project until it reaches the limit amount set for the project. When Atlas Data Federation reaches the project limit amount, Atlas Data Federation allows all running queries against all federated database instances in the project to complete, but doesn't execute any new queries against any federated database instances in the project.
Atlas Data Federation displays a warning in the Atlas UI when Atlas Data Federation reaches the data processing limit. Additionally, you can configure Atlas Data Federation to terminate queries that exceed the limit ASAP when Atlas Data Federation reaches the limit instead of allowing running queries to complete.
What Happens When You Enable Query Termination?
If you configure Atlas Data Federation to terminate queries that exceed the limit, Atlas Data Federation attempts to terminate the queries when Atlas Data Federation reaches the applicable limit. While Atlas Data Federation attempts to terminate the query, Atlas Data Federation might exceed the limit marginally. Atlas Data Federation doesn't return any results and returns only an error. However, you will see data processing charges for the amount of data that Atlas Data Federation processed before it reached the limit.
If you configure the same type of limit for both the project and federated database instances in the project, but with different limit amounts and query termination settings, Atlas Data Federation will terminate the query only when the amount of data that Atlas Data Federation processes reaches the limit amount that you associated with query termination.
Example
Suppose a 100 GB per week project limit with termination enabled and a 60 GB per week federated database instance limit with termination disabled.
When one or more queries against the federated database instance reach the 60 GB limit, Atlas Data Federation terminates all running queries against the federated database instance and doesn't execute any new queries. Atlas Data Federation continues to execute all running and new queries against other federated database instances in the project until Atlas Data Federation reaches the project limit of 100 GB.
When one or more queries reach the 100 GB limit for the project, Atlas Data Federation doesn't execute any new queries and terminates all queries against all federated database instances in the project.
Required Access
To add, edit, or delete
query limits, you must have Project Owner access to the
project. Users with Organization Owner access must add
themselves as a Project Owner to the project before adding, editing,
or deleting query limits.
To view limits on queries, you must have
Project Read Only access to the project or higher.
Procedures
Add Query Limits
You can configure limits on the amount of data processed for your queries from the Atlas CLI, Atlas UI, and API.
To create a new Data Federation query limit using the Atlas CLI, run the following command:
atlas dataFederation queryLimits create <name> [options]
To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas dataFederation queryLimits create.
Click Manage Query Limits to configure limits per federated database instance or for all federated database instances in the project.
You can configure limits for the project and per federated database instance. Project level query limits apply to all federated database instances in the project, which prevents new queries against any federated database instance when the amount of processed data reaches the limit. Query limits on a federated database instance apply to only that federated database instance, and Atlas Data Federation won't execute any new queries against that federated database instance when the amount of processed data reaches the limit. You can also, optionally, enable termination of queries when Atlas Data Federation reaches the configured limit.
Click Add Query Limit to configure the limit.
You can configure the following fields in the Add Query Limit window:
Field Name | Description |
|---|---|
Limit For | Specify whether the limit is for a project or federated database instance. Click the dropdown and select the project or federated database instance to apply the limit to. After you add the limit, you can't modify this setting. |
Limit Type | Specify the limit duration. Click the dropdown to choose one of the following:
After you add the limit, you can't modify this setting. |
Limit Amount | Specify the amount of data in |
Terminate Queries | Toggle to enable query termination. You can't enable query termination for Per query limit type because, by default, Atlas Data Federation terminates the query when it reaches the limit for this limit type. |
Click Add Query Limit for the changes to take effect.
For limits of type Per query, the changes take effect immediately and Atlas Data Federation enforces the limit for all new queries only. For limits of other types, the following apply:
Per day - The changes take effect at the start of Monday in the current week in UTC time.
Per week - The changes take effect at the start of Monday in the current week in UTC time.
Per month - The changes take effect on the first day, the beginning of the the monthly billing period, for the project in UTC time.
To configure query limits from the API, send a POST request
to the limits endpoint.
To configure a limit that applies to all federated database instances in the project, you must specify the unique ID of the project and the name of the limit to set. To learn more about the syntax and options for setting limits for all federated database instances in the project, see API.
To configure a limit for one federated database instance in a project, specify the unique ID of the project and the name of the federated database instance in the project. To learn more about the syntax and options for setting limits for one federated database instance in a project, see API.
View Query Limits
You can view the project and federated database instance limits on queries from the Atlas CLI, Atlas UI, and API. You can also view the amount of data that Atlas Data Federation processed per instance per day.
To list Data Federation query limits for your project using the Atlas CLI, run the following command:
atlas dataFederation queryLimits list [options]
To return the details for the Data Federation query limit you specify using the Atlas CLI, run the following command:
atlas dataFederation queryLimits describe <name> [options]
To learn more about the syntax and parameters for the previous commands, see the Atlas CLI documentation for atlas dataFederation queryLimits list and atlas dataFederation queryLimits describe.
Click Manage Query Limits to view the limits per federated database instance and for all federated database instances in the project.
The Data Federation Query Limits page displays the following:
Column Name | Column Description |
|---|---|
Limit for | Specifies the name of the project if the limit is for the project or the name of the federated database instance if the limit is for a federated database instance. |
Data Processed / Limit | Indicates the total amount of data processed by the queries. You can hover your mouse over the processed data to view the following:
|
Limit Type | Indicates the type of limit. Value can be:
|
Terminate Queries | Indicates whether the flag to terminate running queries when Atlas Data Federation reaches the limit is enabled. |
Actions | Displays the actions you can take on the limit. You can do the following: |
To retrieve the query limits from the API, send a GET request
to the limits endpoint.
To retrieve one limit that applies to all federated database instances in the project, you must specify the unique ID of the project and the name of the limit to retrieve. To learn more about the syntax and options for retrieving one project limit, see API. To retrieve all limits for a project, specify the unique ID of the project. To learn more about the syntax and options for retrieving all federated database instance limits, see API.
To retrieve one limit for one federated database instance in a project, specify the unique ID of the project, the name of federated database instance, and the name of the limit to retrieve. To learn more about the syntax and options for retrieving one limit for one federated database instance in a project, see API. To retrieve all limits for a federated database instance in a project, specify the unique ID of the project and the name of the federated database instance. To learn more about the syntax and options for retrieving all federated database instance limits, see API.
Edit Query Limits
You can edit the project and per federated database instance limits from the Atlas UI and API.
Make changes as needed to the following.
You can modify any of the following settings:
Field Name | Description |
|---|---|
Limit Amount | Specify the amount of data in
|
Terminate Queries | Toggle to enable query termination. You can't enable query termination for Per query limit type because for this limit type, Atlas Data Federation will attempt to terminate each query as soon as it reaches the limit by default. If you enable query termination for any other type, Atlas Data Federation terminates the running queries that have reached the limit. If you disable termination, Atlas Data Federation won't restart any terminated queries and won't terminate any new or running query when Atlas Data Federation reaches the limit amount. Instead, your running queries will be able to finish and scan additional data. To learn more, see What Happens When You Enable Query Termination?. |
To modify the query limits from the API, send a PATCH request
to the limits endpoint.
To update one limit that applies to all federated database instances in the project, you must specify the unique ID of the project and the name of the limit to update. To learn more about the syntax and options for updating one project limit, see API.
To update one limit for one federated database instance in a project, specify the unique ID of the project, the name of the federated database instance, and the name of the limit to update. To learn more about the syntax and options for updating one limit for one federated database instance in a project, see API.
Delete Query Limits
You can delete a project or per federated database instance limit from the Atlas CLI, Atlas UI, and API.
To remove the specified Data Federation query limit from your project using the Atlas CLI, run the following command:
atlas dataFederation queryLimits delete <name> [options]
To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas dataFederation queryLimits delete.
To delete the query limits from the API, send a DELETE request
to the limits endpoint.
To delete one limit that applies to all federated database instances in the project, you must specify the unique ID of the project and the name of the limit to delete. To learn more about the syntax and options for deleting one project limit, see API.
To delete one limit for one federated database instance in a project, specify the unique ID of the project and the name of the limit to delete. To learn more about the syntax and options for deleting one limit for one federated database instance in a project, see API.