Docs Menu

Manage Online Archives

On this page

Important
Serverless Instances are in Preview

Serverless instances are in preview and do not support this feature at this time. To learn more, see Serverless Instance Limitations.

You can run queries against your archived data.

To run queries, you must first connect to your Online Archive. Your cluster connection string allows you to only query data in your Atlas cluster. To query your Online Archive, you must use one of the following:

  • Connect to Online Archive and Cluster - this read-only connection string allows you to read data directly from the live cluster, thereby impacting available resources for IOPS, and from your Online Archive.
  • Connect to Online Archive - this read-only connection string allows you to read data from the Online Archive only and doesn't affect resources of the cluster.

In general, your queries against archived data will be much slower than your queries against data on the Atlas cluster. When you query your cluster and archived data through the federated connection string:

  • Blocking queries, such as sorts that consume and process all input documents to the sort operation before returning results, have performance characteristics associated with the slowest storage, the archive, being queried. The sort operations require all data from the sources being queried before returning the results.
  • Streaming queries, such as finds, have performance characteristics associated with the highest performing storage, the Atlas cluster, being queried. Atlas returns the results as soon as they are available, which means returning results from the archive takes longer than returning results from the Atlas cluster.

For your federated and archive-only queries, you incur costs for the following items.

During data scan, Atlas processes data from both the cluster and the archive. Atlas runs as much of the query on the cluster as it can to minimize the amount of data it needs to scan. For example, for a match query that specifies a specific value, Atlas only retrieves documents with the specified value from the cluster. Atlas then combines the retrieved documents with the archived data and returns.

For blocking queries that need to access all data stored in the underlying cluster, Atlas retrieves all data. For example, for a sort (with no match), Atlas retrieves all data from the cluster and archive to be sorted.

MongoDB charges a fee for each partition that you query in the archive. If your query requires querying specific partitions, MongoDB downloads the partitions and each downloaded partition corresponds to a single access.

To find partitions based on the query and query fields, Atlas runs operations on the archive. Each such operation that Atlas runs finds up to 1000 partitions. Atlas runs the minimum number of required operations to find the partitions required to satisfy the query. For example, if your query requires 100 partitions that are covered in your query fields, Atlas runs only one operation to satisfy the query.

Data that is transferred to the federated infrastructure incurs data transfer costs.

Important
Serverless Instances are in Preview

Serverless instances are in preview and do not support this feature at this time. To learn more, see Serverless Instance Limitations.

You can view the list of online archives for a cluster through the Atlas UI and API.

To view the list of Online Archives:

1
  1. If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.
  2. If it is not already displayed, select your desired project from the Projects menu in the navigation bar.
  3. If the Database Deployments page is not already displayed, click Databases in the sidebar.
2
  1. Click the name of the cluster.
  2. Click the Online Archive tab to view the list of online archives, if any, for the cluster.

The page displays the online archives for the cluster. For each online archive, you can see the following information:

Column Name
Description
Archive Field
The date field based on which documents are archived.
Archive Last Updated
The date when the archive was last modified.
Archival Age Limit
The number of days used to qualify documents for archiving.
Deletion Age Limit
The number of days after which to delete the data in the archive.
Partition Fields
The other commonly used query fields used for partitioning data on the cloud object storage.
Status

The status of the Online Archive. Value can be one of the following:

Pending
Indicates documents are queued for archive, but archiving has not yet started.
Archiving
Indicates archiving has started. In this state, the documents that meet the criteria for archiving are being archived.
Idle
Indicates online archive is waiting for the next archival job to start.
Pausing
Indicates that you have requested to pause archiving. In this state, Atlas is finishing the running archiving operation and therefore, Atlas has not yet put archiving on hold. The online archive transitions to the Paused state when the running archiving operation finishes.
Paused
Indicates archiving has been temporarily stopped. In this state, previously archived documents continue to be available on the cloud object storage for querying, but the specified archiving operation on the active cluster is put on hold and additional documents are not archived. You can resume archiving for paused archives at any time.
Orphaned
Indicates collection associated with an active or paused online archive was deleted. Atlas will not automatically delete the archived data. You must manually delete the online archives associated with the deleted collection.
Deleted
Indicates online archive was deleted. When you delete an online archive, associated archived documents are removed from the cloud object storage.
Actions
Operations that you can perform on the Online Archive.

To retrieve an online archive through the API, send a GET request to the onlineArchives endpoint with the unique ID of the online archive to retrieve. To learn more about the API syntax and options, see Get One Online Archive.

To retrieve all the online archives configured for a cluster using the API, send a GET request to the onlineArchives endpoint for the cluster. To learn more about the syntax and options, see Get all Online Archives for Cluster.

Important
Serverless Instances are in Preview

Serverless instances are in preview and do not support this feature at this time. To learn more, see Serverless Instance Limitations.

You can modify the number of days to keep data on the Atlas cluster (the Age Limit) or the custom JSON query used to select documents for archiving from the Atlas UI and API. You can't change the archiving criteria from Date Match to Custom Filter, or vice versa.

To edit an online archive, in your Atlas UI:

1
  1. If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.
  2. If it is not already displayed, select your desired project from the Projects menu in the navigation bar.
  3. If the Database Deployments page is not already displayed, click Databases in the sidebar.
2
  1. Click the name of the cluster.
  2. Click the Online Archive tab to view the list of online archives, if any, for the cluster.
3

You can:

  • Pause Archiving (only if state is Active)
  • Edit Archive
  • Delete Archive
  • Resume Archiving (only if state is Paused)
4

You can change the archiving criteria and the number of days after which to delete archived data.

  • To edit Date Match criteria, modify the number of days Atlas stores data on the active Atlas cluster in the Archival Age Limit section.
  • To edit Custom Criteria, enter a valid JSON filter to select the documents for archiving.

    Note

    Atlas uses the specified query with the db.collection.find(query) command. Custom queries do not support JavaScript expressions. Also, you can't pass an empty document {} to return all documents.

  • To modify the number of days after which Atlas deletes archived data, enter or modify the number of days in the Deletion Age Limit section. You can specify between 7 and 9125 days, or leave the field empty to disable the data expiration rule. It takes 24 hours for this change to take effect.

    Note

    Atlas Online Archive data expiration is available as a Preview feature. The feature and the corresponding documentation may change at any time during the Preview stage.

5
Note

It takes 24 hours for changes to Deletion Age Limit to take effect.

You can't modify the partition fields or structure from the Atlas UI or API. However, you can manually migrate the data from the cloud object storage using mongodump, delete the online archive, use mongorestore to restore the data on the Atlas cluster, and then create a new online archive for the collection with the desired partition fields and structure.

To edit an online archive through the API, send a PATCH request to the onlineArchives endpoint with the unique ID of the online archive to update. To learn more about the API syntax and options, see Update an Online Archive.

Important
Serverless Instances are in Preview

Serverless instances are in preview and do not support this feature at this time. To learn more, see Serverless Instance Limitations.

You can delete an online archive through the Atlas UI and API. When you remove an online archive, you also delete data on the cloud object storage for the online archive. After you delete this data, you can't restore it.

If you drop a database or collection configured for online archive, the data from the collection, if archived, continues to be available on the cloud object storage. You incur costs for storage on the cloud object storage. Alternatively, if you delete the cluster, Atlas deletes all the online archives configured for the cluster. This also deletes any archived data from the cloud object storage.

If you delete all the online archives, you also delete the Data Lake and you create a new Data Lake when you create an online archive again.

After you delete an online archive, its state moves to Deleted. You can create another online archive for the same database, collection, and fields as the deleted online archive if there is no other online archive for the same database, collection, and fields in the Active state.

To delete an online archive, in your Atlas UI:

1
  1. If it is not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar.
  2. If it is not already displayed, select your desired project from the Projects menu in the navigation bar.
  3. If the Database Deployments page is not already displayed, click Databases in the sidebar.
2
  1. Click the name of the cluster.
  2. Click the Online Archive tab to view the list of online archives, if any, for the cluster.
3

You can:

  • Pause Archiving (only if state is Active)
  • Edit Archive
  • Delete Archive
  • Resume Archiving (only if state is Paused)
4
5

To delete an online archive through the API, send a DELETE request to the onlineArchives endpoint with the unique ID of the online archive to delete. To learn more about the API syntax and options, see Delete an Online Archive.

←  Connect to Your Online ArchiveManage Private Endpoints for Online Archives →
Give Feedback
© 2022 MongoDB, Inc.

About

  • Careers
  • Investor Relations
  • Legal Notices
  • Privacy Notices
  • Security Information
  • Trust Center
© 2022 MongoDB, Inc.