Manage Atlas Data Federation Query Limits
On this page
Overview
You can limit the amount of data that Atlas Data Federation processes for your federated database instances to control costs. To limit the amount of processed data, you can configure query limits per federated database instance and for all federated database instances in your project. The query limits that you configure only apply to data processing costs and don't apply to other Atlas Data Federation costs such as data retrieval and transfer.
You can configure one limit per query, per day, per week, and per month per project and per federated database instance. Atlas Data Federation considers each limit of each type independent of other configured limits. If you configure the same type of limit for the project and for a federated database instance in the project, Atlas Data Federation enforces the more restrictive limit of the two. Limit values must be between 1GiB (1024^3 bytes) and 1PiB.
By default, Atlas Data Federation sets a 100TB limit on the amount of processed data per month for all new federated database instances.
Note
Atlas Data Federation enforces the project limit per cloud object storage. For example, suppose you have 2 federated database instances, one on AWS and another on Azure. Atlas Data Federation allows scans of up to 100TB per federated database instance for a total of 200TB for the project.
What Happens When Atlas Data Federation Reaches the Data Limit?
When the amount of processed data reaches the configured limit amount, Atlas Data Federation stops processing data for the query that has reached the limit if the limit type is per query. For limits of other types, Atlas Data Federation doesn't execute any new queries until the limit resets based on the configured limit type. That is, Atlas Data Federation doesn't execute any new queries until:
The next calendar day beginning at
00:00UTC for limit type of per day.The next calendar week beginning on Monday at
00:00UTC for limit type of per week.The next calendar month beginning on the first day (1st) of the month at
00:00UTC for limit type of per month.
If you configure the same type of limit for both the project and federated database instances in the project, but with different limit amounts for the project and the federated database instances in the project, the following apply:
If Atlas Data Federation reaches the project limit amount before reaching the limit amount for any federated database instance in the project, Atlas Data Federation allows all running queries against all federated database instances in the project to complete.
If Atlas Data Federation doesn't reach the limit amount for the project, but reaches the limit amount set for any federated database instance in the project, Atlas Data Federation allows all running queries against that federated database instance to complete, but doesn't execute any new queries against that federated database instance. Atlas Data Federation continues to execute queries against other federated database instances in the project until it reaches the limit amount set for the project. When Atlas Data Federation reaches the project limit amount, Atlas Data Federation allows all running queries against all federated database instances in the project to complete, but doesn't execute any new queries against any federated database instances in the project.
Atlas Data Federation displays a warning in the Atlas UI when Atlas Data Federation reaches the data processing limit. Additionally, you can configure Atlas Data Federation to terminate queries that exceed the limit ASAP when Atlas Data Federation reaches the limit instead of allowing running queries to complete.
What Happens When You Enable Query Termination?
If you configure Atlas Data Federation to terminate queries that exceed the limit, Atlas Data Federation attempts to terminate the queries when Atlas Data Federation reaches the applicable limit. While Atlas Data Federation attempts to terminate the query, Atlas Data Federation might exceed the limit marginally. Atlas Data Federation doesn't return any results and returns only an error. However, you will see data processing charges for the amount of data that Atlas Data Federation processed before it reached the limit.
If you configure the same type of limit for both the project and federated database instances in the project, but with different limit amounts and query termination settings, Atlas Data Federation will terminate the query only when the amount of data that Atlas Data Federation processes reaches the limit amount that you associated with query termination.
Example
Suppose a 100 GB per week project limit with termination enabled and a 60 GB per week federated database instance limit with termination disabled.
When one or more queries against the federated database instance reach the 60 GB limit, Atlas Data Federation terminates all running queries against the federated database instance and doesn't execute any new queries. Atlas Data Federation continues to execute all running and new queries against other federated database instances in the project until Atlas Data Federation reaches the project limit of 100 GB.
When one or more queries reach the 100 GB limit for the project, Atlas Data Federation doesn't execute any new queries and terminates all queries against all federated database instances in the project.
Required Access
To add, edit, or delete
query limits, you must have Project Owner access to the
project. Users with Organization Owner access must add
themselves as a Project Owner to the project before adding, editing,
or deleting query limits.
To view limits on queries, you must have
Project Read Only access to the project or higher.
Procedures
Add Query Limits
You can configure limits on the amount of data processed for your queries from the Atlas CLI, Atlas UI, and API.
View Query Limits
You can view the project and federated database instance limits on queries from the Atlas CLI, Atlas UI, and API. You can also view the amount of data that Atlas Data Federation processed per instance per day.
Edit Query Limits
You can edit the project and per federated database instance limits from the Atlas UI and API.
Delete Query Limits
You can delete a project or per federated database instance limit from the Atlas CLI, Atlas UI, and API.