Rachelle Palmer

## Contributed By
This article was contributed by Prem Krishna, a Senior Product Manager for Analytics at MongoDB.

## Introduction
With Atlas Online Archive, you can tier off cold data or infrequently accessed data from your MongoDB cluster to a MongoDB-managed cloud object storage - Amazon S3 or Microsoft Azure Blob Storage. This can lower the cost via archival cloud storage for old data, while active data that is more often accessed and queried remains in the primary database.  

> FYI: If using Online Archive and also using MongoDB's Atlas Data Federation, users can also see a unified view of production data, and *archived data*  side by side through a read-only, federated database instance.

In this blog, we are going to be discussing how to improve the performance of your online archive by choosing the correct partitioning fields.

## Why is partitioning so critical when configuring Online Archive?
Once you have started archiving data, you cannot edit any partition fields as the structure of how the data will be stored in the object storage becomes fixed after the archival job begins. Therefore, you'll want to think critically about your partitioning strategy beforehand.

Also, archival query performance is determined by how the data is structured in object storage, so it is important to not only choose the correct partitions but also choose the correct order of partitions. 

## Do this...
**Choose the most frequently queried fields.** You can choose up to 2 partition fields for a custom query-based archive or up to three fields on a date-based online archive.  Ensure that the most frequently queried fields for the archive are chosen. Note that we are talking about how you are going to query the archive and not the custom query criteria provided at the time of archiving!

**Check the order of partitioned fields.** While selecting the partitions is important, it is equally critical to choose the correct *order* of partitions.  The most frequently queried field should be the first chosen partition field, followed by the second and third. That's simple enough.

## Not this
**Don't add irrelevant fields as partitions.**  If you are not querying a specific field from the archive, then that field should not be added as a partition field. Remember that you can add a maximum of 2 or 3 partition fields, so it is important to choose these fields carefully based on how you query your archive.

**Don't ignore the “Move down” option.** The “Move down” option is applicable to an archive with a data-based rule. For example, if you want to query on Field_A the most, then Field_B, and then on exampleDate, ensure you are selecting the “Move Down” option next to the “Archive date field” on top.
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Online_Archive_Configuration_3abb277a04.png)

**Don't choose high cardinality partition(s).** Choosing a high cardinality field such as `_id` will create a large number of partitions in the object storage. Then querying the archive for any aggregate based queries will cause increased latency. The same is applicable if multiple partitions are selected such that the collective fields when grouped together can be termed as high cardinality.  For example, if you are selecting Field_A, Field_B and Field_C as your partitions and if a combination of these fields are creating unique values, then it will result in high cardinality partitions. 
> Please note that this is **not applicable** for new [Online Archives](https://www.mongodb.com/blog/post/new-online-archive-performance-improvements-enhanced-metrics). 

## Additional guidance
In addition to the partitioning guidelines, there are a couple of additional considerations that are relevant for the optimal configuration of your data archival strategy.

**Add data expiration rules and scheduled windows**
These fields are optional but are relevant for your use cases and can improve your archival speeds and for how long your data needs to be present in the archive. 

**Index required fields**
Before archiving the data, ensure that your data is indexed for optimal performance. You can run an explain plan on the archival query to verify whether the archival rule will use an index. 

## Conclusion
It is important to follow these do’s and don’ts before hitting “Begin Archiving” to archive your data so that the partitions are correctly configured thereby optimizing the performance of your online archives.

For more information on configuration or Online Archive, please see the documentation for [setting up an Online Archive](https://www.mongodb.com/docs/atlas/online-archive/configure-online-archive/) and our [blog post](https://www.mongodb.com/developer/products/atlas/atlas-data-lake-online-archive/) on how to create an Online Archive. 

Dig deeper into this topic with [this tutorial.](https://www.mongodb.com/developer/products/atlas/how-to-use-custom-archival-rules-and-partitioning-on-mongodb-atlas-online-archive/)


  

Get all the do's and don'ts around optimization of your data archival strategy.

Optimizing your Online Archive for Query Performance

Joe Karlsson

>As of June 2022, the functionality previously known as Atlas Data Lake is now named Atlas Data Federation. Atlas Data Federation’s functionality is unchanged and you can learn more about it [here](http://mongodb.com/atlas/data-federation). Atlas Data Lake will remain in the Atlas Platform, with newly introduced functionality that you can learn about [here](http://mongodb.com/atlas/data-lake).

Okay, so you've set up a simple MongoDB Atlas Online Archive, and now you might be wondering, "What's next?" In this post, we will cover some more advanced Online Archive use cases, including setting up custom archival rules and how to improve query performance through partitioning.

## Prerequisites

-   The Online Archive feature is available on [M10](https://docs.atlas.mongodb.com/cluster-tier/) and greater Atlas clusters that run MongoDB 3.6 or later. So for this demo, you will need to create a [M10 cluster](https://docs.atlas.mongodb.com/cluster-tier/) in MongoDB Atlas. [Click here for information on setting up a new MongoDB Atlas cluster](https://docs.atlas.mongodb.com/getting-started/) or check out [How to Manage Data at Scale With MongoDB Atlas Online Archive](https://developer.mongodb.com/how-to/manage-data-at-scale-with-online-archive/).

-   Ensure that each database has been seeded by [loading sample data into our Atlas cluster](https://docs.atlas.mongodb.com/sample-data/). I will be using the `sample_analytics.customers` dataset for this demo.

## Creating a Custom Archival Rule

Creating an Online Archive rule based on the date makes sense for a lot of archiving situations, such as automatically archiving documents that are over X years old, or that were last updated Y months ago. But what if you want to have more control over what gets archived? Some examples of data that might be eligible to be archived are:

-   Data that has been flagged for archival by an administrator.
-   Discontinued products on your eCommerce site.
-   User data from users that have closed their accounts on your platform (unless they are European citizens).
-   Employee data from employees that no longer work at your company.

There are lots of reasons why you might want to set up custom rules for archiving your cold data. Let's dig into how you can achieve this using custom archive rules with MongoDB Atlas Online Archive. For this demo, we will be setting up an automatic archive of all users in the `sample_analytics.customers` collection that have the 'active' field set to `false`.

In order to configure our Online Archive, first navigate to the Cluster page for your project, click on the name of the cluster you want to configure Online Archive for, and click on the **Online Archive** tab.

Next, click the Configure Online Archive button the first time and the **Add Archive** button subsequently to start configuring Online Archive for your collection. Then, you will need to create an Archiving Rule by specifying the collection namespace, which will be `sample_analytics.customers`.

You will also need to specify your custom criteria for archiving documents. You can specify the documents you would like to filter for archival with a MongoDB query, in JSON, the same way as you would write filters in MongoDB Atlas.

> Note: You can use any valid MongoDB Query Language (MQL) query, however, you cannot use the empty document argument ({}) to return all documents.

To retrieve the documents staged for archival, we will use the following find command. This will retrieve all documents that have the \`active\` field set to \`false\` or do not have an \`active\` key at all.

```
{ $or: [
    { active: false }, 
    { active: null }
] }
```
Continue setting up your archive, and then you should be done!

> Note: It's always a good idea to run your custom queries in the [mongo shell](https://docs.mongodb.com/mongodb-shell/install/) first to ensure that you are archiving the correct documents.

> Note: Once you initiate an archive and a MongoDB document is queued for archiving, you can no longer edit the document.

## Improving Query Performance Through Partitioning

One of the reasons we archive data is to access and query it in the future, if for some reason we still need to use it. In fact, you might be accessing this data quite frequently! That's why it's useful to be able to partition your archived data and speed up query times. With Atlas Online Archive, you can specify the two most frequently queried fields in your collection to create partitions in your online archive.

Fields with a moderate to high cardinality (or the number of elements in a set or grouping) are good choices to be used as a partition. Queries that don't contain these fields will require a full collection scan of all archived documents, which will take longer and increase your costs. However, it's a bit of a bit of a balancing act. 

For example, fields with low cardinality wont partition the data well and therefore wont improve performance greatly. However, this may be OK for range queries or collection scans,  but will result in fast archival performance.

Fields with mid to high cardinality will partition the data better leading to better general query performance, but maybe slightly slower archival performance.

Fields with extremely high cardinality like `_id` will lead to poor query performance for everything but "point queries" that query on _id, and will lead to terrible archival performance due to writing many partitions.

> Note: Online Archive is powered by MongoDB Atlas Data Lake. To learn more about how partitions improve your query performance in Data Lake, see [Data Structure in cloud object storage - Amazon S3 or Microsoft Azure Blob Storage](https://docs.mongodb.com/datalake/admin/optimize-query-performance#data-structure-in-s3).

The specified fields are used to partition your archived data for optimal query performance. Partitions are similar to folders. You can move whichever field to the first position of the partition if you frequently query by that field.

The order of fields listed in the path is important in the same way as it is in [Compound Indexes](https://docs.mongodb.com/manual/core/index-compound/). Data in the specified path is partitioned first by the value of the first field, and then by the value of the next field, and so on. Atlas supports queries on the specified fields using the partitions.

You can specify the two most frequently queried fields in your collection and order them from the most frequently queried in the first position to the least queried field in the second position. For example, suppose you are configuring the online archive for your `customers` collection in the `sample_analytics` database. If your archived field is set to the custom archival rule in our example above, your first queried field is `username`, and your second queried field is `email`, your partition will look like the following:

```
/username/email
```

Atlas creates partitions first for the `username` field, followed by the `email`. Atlas uses the partitions for queries on the following fields:

-   the `username` field
-   the ` username` field and the `email` field

> Note: The value of a partition field can be up to a maximum of 700 characters. Documents with values exceeding 700 characters are not archived.

![Figure showing how Atlas Online Archive partitions data and how it uses it to query documents quickly.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Online_Archive_Partition_3466b335d6.png)

For more information on how to partition data in your Online Archive, please [refer to the documentation](https://docs.atlas.mongodb.com/online-archive/configure-online-archive/).

## Summary

In this post, we covered some advanced use cases for Online Archive to help you take advantage of this MongoDB Atlas feature. We initialized a demo project to show you how to set up custom archival rules with Atlas Online Archive, as well as improve query performance through partitioning your archived data.

If you have questions, please head to our [developer community website](https://community.mongodb.com/) where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.


So you've set up a simple MongoDB Atlas Online Archive, and now you might be wondering, "What's next?" In this post, we will cover some more advanced Online Archive use cases, including setting up custom archival rules and how to improve query performance through partitioning.

How to Use Custom Archival Rules and Partitioning on MongoDB Atlas Online Archive

Store infrequently accessed data elsewhere to lower storage costs

Online Archive

All Online Archive Content

Optimizing your Online Archive for Query Performance

How to Use Custom Archival Rules and Partitioning on MongoDB Atlas Online Archive

Related Topics