> For the complete MongoDB documentation index, see www.mongodb.com/docs/llms.txt # Configure Online Archive for Standard Collections **Important:** The feature described on this page is not currently supported on Atlas Infinite clusters during public preview. For the features supported in public preview, see [Public Preview Availability.](https://www.mongodb.com/docs/atlas/infinite/atlas-infinite-landing/#public-preview-availability) ## Overview You can configure data in a standard collection to be archived by specifying an archiving rule. The archiving rule for a standard collection can be one of the following: - A date field paired with a retention period in days, which determines when Atlas moves data from the cluster to the archive (Date Match). - A custom JSON (Javascript Object Notation) query that selects the documents to archive (Custom Criteria). To configure your Atlas cluster for online archive: 1. Create an archiving rule by providing the collection namespace and the criteria for selecting data to archive. 2. (Optional) Specify commonly queried fields to partition archived data. When you configure an Online Archive on your cluster, Atlas creates 2 federated database instances on your cluster for your archive only and for your cluster and archive. For limitations that apply to all online archives, see [Limitations.](https://www.mongodb.com/docs/atlas/online-archive/configure-online-archive.md#std-label-config-online-archive-limitations) ## Prerequisites Before configuring an Online Archive, ensure that your Atlas cluster is healthy. If one node is down, the archive remains in `PENDING` status until the cluster is fully operational. ## Required Access To create an Online Archive, you must have [`Project Data Access Admin`](https://www.mongodb.com/docs/atlas/reference/user-roles.md#mongodb-authrole-Project-Data-Access-Admin) access or higher to the project. To watch for an archive to be available, you must have [`Project Read Only`](https://www.mongodb.com/docs/atlas/reference/user-roles.md#mongodb-authrole-Project-Read-Only) access or higher to the project. ## Configure Through the Atlas CLI To create an online archive for a cluster using the Atlas CLI, run the following command: ```sh atlas clusters onlineArchives create [options] ``` To watch for a specific online archive to become available using the Atlas CLI, run the following command: ```sh atlas clusters onlineArchives watch [options] ``` To learn more about the syntax and parameters for the previous commands, see the Atlas CLI documentation for [atlas clusters onlineArchives create](https://www.mongodb.com/docs/atlas/cli/current/command/atlas-clusters-onlineArchives-create/) and [atlas clusters onlineArchives watch.](https://www.mongodb.com/docs/atlas/cli/current/command/atlas-clusters-onlineArchives-watch/) **See also: Related Links** - [Install the Atlas CLI](https://www.mongodb.com/docs/atlas/cli/current/install-atlas-cli/) - [Connect to the Atlas CLI](https://www.mongodb.com/docs/atlas/cli/current/connect-atlas-cli/) ## Configure Through the Atlas Administration API To configure an online archive from the API, send a `POST` request to the [onlineArchives](https://www.mongodb.com/docs/api/doc/atlas-admin-api-v2/group/endpoint-online-archive) endpoint. **Note:** If you use the `DATE` criteria, you must specify the `date` field as part of the partition keys. If the cluster already has an `Active` online archive with the same archiving rule for the same database and collection, the operation fails. However, if the existing online archive is in `Paused` or `Deleted` state, the new online archive is created and its status is set to `Active`. To learn more about the syntax and options, see [API.](https://www.mongodb.com/docs/api/doc/atlas-admin-api-v2/operation/operation-creategroupclusteronlinearchive) ## Configure Through the User Interface To configure an online archive for a standard collection, in your Atlas UI: 1. In Atlas, go to the Clusters page for your project. If it's not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar. If it's not already displayed, select your desired project from the Projects menu in the navigation bar. In the sidebar, click Clusters under the Database heading. The [Clusters](https://cloud.mongodb.com/go?l=https%3A%2F%2Fcloud.mongodb.com%2Fv2%2F%3Cproject%3E%23%2Fclusters) page displays. 2. Go to the Online Archive page for your cluster. In the sidebar, click Online Archive. The [Online Archive](https://cloud.mongodb.com/go?l=https%3A%2F%2Fcloud.mongodb.com%2Fv2%2F%3Cproject%3E%23%2Fclusters%2FonlineArchive%2F%3Ccluster%3E) page displays. 3. Start configuring online archive for your collection. To configure an online archive for your collection, click: - Configure Online Archive button the first time. - Add Archive button subsequently. 4. Review the Online Archive Overview and click Next to proceed. 5. Create an Archiving Rule by providing the following information. Specify the collection namespace, which includes the database name, the dot (`.`) separator, and the collection name (that is, `.`), in the Namespace field. You can't modify the [namespace](https://www.mongodb.com/docs/manual/reference/limits/#faq-dev-namespace) once the online archive is created. Create an index on the Date field of your collection to ensure optimal performance. Select the cloud provider region where you want to store your archived data. IMPORTANT: We recommend that you select the same region as your cluster if possible because you might incur higher data transfer cost if you choose a different region. Atlas displays the cloud provider regions based on the cloud provider where your cluster is deployed. For multi-cloud clusters, Atlas displays the cloud provider regions of the highest priority provider. Atlas displays a next to the region that closely or exactly matches the region where your cluster is deployed. ### AWS For Atlas clusters deployed on AWS (Amazon Web Services), you can select one of the following regions: | Data Federation Region | AWS (Amazon Web Services) Region | Atlas Region | | --- | --- | --- | | Northern Virginia, USA | `us-east-1` | `US_EAST_1` | | Oregon, USA | `us-west-2` | `US_WEST_2` | | Sao Paulo, Brazil | `sa-east-1` | `SA_EAST_1` | | Ireland | `eu-west-1` | `EU_WEST_1` | | London, England, UK | `eu-west-2` | `EU_WEST_2` | | Frankfurt, Germany | `eu-central-1` | `EU_CENTRAL_1` | | Tokyo, Japan | `ap-northeast-1` | `AP_NORTHEAST_1` | | Seoul, South Korea | `ap-northeast-2` | `AP_NORTHEAST_2` | | Mumbai, India | `ap-south-1` | `AP_SOUTH_1` | | Singapore | `ap-southeast-1` | `AP_SOUTHEAST_1` | | Sydney, Australia | `ap-southeast-2` | `AP_SOUTHEAST_2` | | Montreal, QC, Canada | `ca-central-1` | `CA_CENTRAL_1` | IMPORTANT: Once Atlas creates the online archive, you can't modify the storage region. Specify the criteria for selecting documents to archive under the Date Match or Custom Criteria tab. ### Date Match To select documents from the collection using a combination of a date field and number of days: - Specify an already indexed date field from the documents in the collection. To specify a nested field, use the [dot notation.](https://www.mongodb.com/docs/manual/core/document/#dot-notation) - Specify the number of days to keep the data in the Atlas cluster. - Choose the date format of the specified date field. Date can be expressed as an ISO-8601-formatted date, a timestamp in UNIX epoch format, or an ObjectId. The UNIX epoch timestamp can be expressed in nanoseconds, milliseconds, or seconds. If you choose any of the following formats, the value of specified date field must be the [BSON type](https://www.mongodb.com/docs/manual/reference/bson-types/) `long`: - `EPOCH_SECONDS` - `EPOCH_MILLIS` - `EPOCH_NANOSECONDS` IMPORTANT: You can't modify the date field once the online archive is created. **Important:** Atlas runs an index sufficiency query during the first archive run to determine the efficiency of the archival process. If the number of documents scanned to the number of documents returned is 10 or more, the query result triggers an `Index Sufficiency Warning`. This warning indicates that you have insufficient indexes for an efficient archival process. For date-based archives, you must index the date field. For custom criteria that use an expression, Atlas might first convert a value before it evaluates it against the query. The index sufficiency query stops running after sufficient indexes are detected. If an index on the collection gets dropped later and slows down the archiving process, Atlas does not issue an `Index Sufficiency Warning`. 6. Specify how many days you want to store data in the online archive and a time window when you want Atlas to run the archiving job. (Optional) Specify a Data Retention Period. By default, Atlas doesn't delete archived data. However, if you specify the Data Retention Period, you can specify between `7` to `9125` days (25 years) to keep archived data. Atlas deletes archived data after the number of days you specify here. This data expiration rule takes effect `24` hours after you set the Data Retention Period. WARNING: Once Atlas deletes the data, you can't recover the data. (Optional) Specify a Schedule Archiving Window. By default, Atlas periodically [runs a query to archive data](https://www.mongodb.com/docs/atlas/online-archive/overview.md#std-label-adf-oa-performance-expectations). However, you can toggle the Schedule Archiving Window to explicitly schedule the time window during which you want Atlas to archive data. You can specify the following: - Frequency. You can choose to run the job every day, on a specific day of the week, or on a specific date every month. If you wish to schedule the data archiving job on the 29th, 30th, or 31st of every month, Atlas doesn't run the archiving job for months without these dates (for example, February). - Time window, in hours. Select the period of time during which you want Atlas to run the data archiving job. You must specify a minimum of two hours. If a running job doesn't complete during the specified time window, Atlas continues to run the job until it completes. 7. Click Next to specify the most commonly queried fields. 8. Specify the two most frequently queried fields in your collection to create partitions in your online archive. IMPORTANT: Archive must have at least one partition field. ### Date Match Enter up to two most commonly queried fields from the collection in the Second most commonly queried field and Third most commonly queried field fields respectively. To specify nested fields, use the [dot notation](https://www.mongodb.com/docs/manual/core/document/#dot-notation). Do not include quotes (`""`) around nested fields that you specify using [dot notation.](https://www.mongodb.com/docs/manual/core/document/#dot-notation) **Warning:** You can't specify field names that contain periods (`.`) for partitioning. The specified fields are used to partition your archived data. Partitions are similar to folders. The date field is in the first position of the partition by default for the Date Match criteria. You can move another field to the first position of the partition if you frequently query by that field. The order of fields listed in the path is important, as data in the specified path is partitioned first by the value of the first field, then by the value of the second field, and so on. Atlas supports queries on the specified fields using the partitions. For example, suppose you are configuring the online archive for the `movies` collection in the `sample_mflix` database. If your archived field is the `released` date field, which you moved to the third position, your first queried field is `title`, and your second queried field is `plot`, your partition will look similar to the following: ```sh /title/plot/released ``` Atlas creates partitions first for the `title` field, followed by the `plot` field, and then the `released` field. Atlas uses the partitions for queries on the following fields: - the `title` field, - the `title` field and the `plot` field, - the `title` field and the `plot` field and the `released` field. Atlas can also use the partitions to support a query on the `title` and `released` fields. However, in this case, Atlas would not be as efficient in supporting the query as it would be if the query were on the `title` and `plot` fields only. To query the Online Archive effectively, include all defined partition fields in the order in which they're defined. Querying all partition fields in order targets specific partitions and minimizes the amount of data scanned. Since a query on `title` and `released` omits `plot`, Atlas uses the `title` partition more efficiently than the `released` partition to support this query. Atlas can't use the partitioning strategy to efficiently support queries on fields not specified here. Also, Atlas can't use the partitions to support queries that include the following fields without the `title` field: - the `plot` field, - the `released` field, or - the `plot` and `released` fields. * Choose fields that contain only characters supported on AWS (Amazon Web Services). To learn more about the characters to avoid, see [Creating object key names](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html). Atlas skips and doesn't archive documents that contain unsupported characters. * Choose fields that do not contain polymorphic data. Atlas determines the data type of a partition field by sampling 10 documents from the collection. Atlas will not archive a document if the specified field value in a document does not match values in other documents in the same collection. * Choose fields that you query frequently and order them from the most frequently queried in the first position to the least queried field in the last position. For example, if you frequently query on the date field, then leave the date field in the first position. But if you frequently query on another field, then that field should be in the first position. IMPORTANT: For Online Archives created before June 2023, MongoDB doesn't recommend `string` type fields with high cardinality as a query field for Online Archives. For fields of type `string` with high cardinality, Atlas creates a large number of partitions. This doesn't apply to Online Archives created after June 2023. To learn more, read the [MongoDB blog post.](https://www.mongodb.com/blog/post/new-online-archive-performance-improvements-enhanced-metrics?tck=docs) Atlas supports the following partition attribute types: - `date` - `double` - `int` - `long` - `objectId` - `string` - `boolean` To learn more about the supported partition attribute types, see [Partition Attribute Types.](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/supported-partition-attributes.md#std-label-adf-path-attribute-types) IMPORTANT: You can use the [explain](https://www.mongodb.com/docs/manual/reference/command/explain/) command to return information about the data partitions used to satisfy a query. To learn more, see [`explain`.](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/diagnostic-commands.md#std-label-adf-diagnostic-cmd-explain) While partitions improve query performance, queries that don't contain these fields require a full collection scan of all archived documents, which will take longer and increase your costs. To learn more about how partitions improve your query performance in Atlas Data Federation, see [Data Structure in S3.](https://www.mongodb.com/docs/atlas/data-federation/admin/optimize-query-performance.md#std-label-optimize-query-perf) 9. Click Next to review and confirm the online archive settings. You can review the following archiving rule settings: - The name of the database and collection - The name of the cloud provider and the cloud provider region - The name of the date field (for Date Match only) - The number of days to keep data on the Atlas cluster (for Date Match only) - The number of days after which to delete archived data - The frequency and time window for archiving data - The custom query to use to identify data to archive (for Custom Criteria only) - The partition fields Click Back to edit these settings if needed. 10. Copy and run the displayed query in your [`mongosh`](https://www.mongodb.com/docs/mongodb-shell.md#mongodb-binary-bin.mongosh) shell to see the documents that match the criteria in the rule you defined. You can run [explain](https://www.mongodb.com/docs/manual/reference/method/cursor.explain/) on the query to check whether it uses an index. Proceed to the next step to create the index if the fields are not indexed. If the fields are already indexed, skip to the final step. 11. (Optional) Copy and run the displayed query in your [`mongosh`](https://www.mongodb.com/docs/mongodb-shell.md#mongodb-binary-bin.mongosh) to create the required index. This ensures that your data is indexed for optimal performance. 12. Verify and confirm your archiving rule. Click Begin Archiving in the Confirm an online archive tab. Click Confirm in the Begin Archiving window. WARNING: Once your document is queued for archiving, you can no longer edit the document. See [Restore Specific Documents from Online Archive](https://www.mongodb.com/docs/atlas/online-archive/restore-archived-data-with-merge.md#std-label-restore-archived-data) to move archived data back into the live Atlas cluster.