Explore Developer Center's New Chatbot! MongoDB AI Chatbot can be accessed at the top of your navigation to answer all your MongoDB questions.

Join us at AWS re:Invent 2024! Learn how to use MongoDB for AI use cases.
MongoDB Developer
Atlas
plus
Sign in to follow topics
MongoDB Developer Centerchevron-right
Developer Topicschevron-right
Productschevron-right
Atlaschevron-right

How to Use Custom Archival Rules and Partitioning on MongoDB Atlas Online Archive

Joe Karlsson5 min read • Published Jun 08, 2021 • Updated May 31, 2023
AtlasOnline Archive
Facebook Icontwitter iconlinkedin icon
Rate this tutorial
star-empty
star-empty
star-empty
star-empty
star-empty
As of June 2022, the functionality previously known as Atlas Data Lake is now named Atlas Data Federation. Atlas Data Federation’s functionality is unchanged and you can learn more about it here. Atlas Data Lake will remain in the Atlas Platform, with newly introduced functionality that you can learn about here.
Okay, so you've set up a simple MongoDB Atlas Online Archive, and now you might be wondering, "What's next?" In this post, we will cover some more advanced Online Archive use cases, including setting up custom archival rules and how to improve query performance through partitioning.

Prerequisites

Creating a Custom Archival Rule

Creating an Online Archive rule based on the date makes sense for a lot of archiving situations, such as automatically archiving documents that are over X years old, or that were last updated Y months ago. But what if you want to have more control over what gets archived? Some examples of data that might be eligible to be archived are:
  • Data that has been flagged for archival by an administrator.
  • Discontinued products on your eCommerce site.
  • User data from users that have closed their accounts on your platform (unless they are European citizens).
  • Employee data from employees that no longer work at your company.
There are lots of reasons why you might want to set up custom rules for archiving your cold data. Let's dig into how you can achieve this using custom archive rules with MongoDB Atlas Online Archive. For this demo, we will be setting up an automatic archive of all users in the sample_analytics.customers collection that have the 'active' field set to false.
In order to configure our Online Archive, first navigate to the Cluster page for your project, click on the name of the cluster you want to configure Online Archive for, and click on the Online Archive tab.
Next, click the Configure Online Archive button the first time and the Add Archive button subsequently to start configuring Online Archive for your collection. Then, you will need to create an Archiving Rule by specifying the collection namespace, which will be sample_analytics.customers.
You will also need to specify your custom criteria for archiving documents. You can specify the documents you would like to filter for archival with a MongoDB query, in JSON, the same way as you would write filters in MongoDB Atlas.
Note: You can use any valid MongoDB Query Language (MQL) query, however, you cannot use the empty document argument ({}) to return all documents.
To retrieve the documents staged for archival, we will use the following find command. This will retrieve all documents that have the `active` field set to `false` or do not have an `active` key at all.
1{ $or: [
2 { active: false },
3 { active: null }
4] }
Continue setting up your archive, and then you should be done!
Note: It's always a good idea to run your custom queries in the mongo shell first to ensure that you are archiving the correct documents.
Note: Once you initiate an archive and a MongoDB document is queued for archiving, you can no longer edit the document.

Improving Query Performance Through Partitioning

One of the reasons we archive data is to access and query it in the future, if for some reason we still need to use it. In fact, you might be accessing this data quite frequently! That's why it's useful to be able to partition your archived data and speed up query times. With Atlas Online Archive, you can specify the two most frequently queried fields in your collection to create partitions in your online archive.
Fields with a moderate to high cardinality (or the number of elements in a set or grouping) are good choices to be used as a partition. Queries that don't contain these fields will require a full collection scan of all archived documents, which will take longer and increase your costs. However, it's a bit of a bit of a balancing act.
For example, fields with low cardinality wont partition the data well and therefore wont improve performance greatly. However, this may be OK for range queries or collection scans, but will result in fast archival performance.
Fields with mid to high cardinality will partition the data better leading to better general query performance, but maybe slightly slower archival performance.
Fields with extremely high cardinality like _id will lead to poor query performance for everything but "point queries" that query on _id, and will lead to terrible archival performance due to writing many partitions.
Note: Online Archive is powered by MongoDB Atlas Data Lake. To learn more about how partitions improve your query performance in Data Lake, see Data Structure in cloud object storage - Amazon S3 or Microsoft Azure Blob Storage.
The specified fields are used to partition your archived data for optimal query performance. Partitions are similar to folders. You can move whichever field to the first position of the partition if you frequently query by that field.
The order of fields listed in the path is important in the same way as it is in Compound Indexes. Data in the specified path is partitioned first by the value of the first field, and then by the value of the next field, and so on. Atlas supports queries on the specified fields using the partitions.
You can specify the two most frequently queried fields in your collection and order them from the most frequently queried in the first position to the least queried field in the second position. For example, suppose you are configuring the online archive for your customers collection in the sample_analytics database. If your archived field is set to the custom archival rule in our example above, your first queried field is username, and your second queried field is email, your partition will look like the following:
1/username/email
Atlas creates partitions first for the username field, followed by the email. Atlas uses the partitions for queries on the following fields:
  • the username field
  • the username field and the email field
Note: The value of a partition field can be up to a maximum of 700 characters. Documents with values exceeding 700 characters are not archived.
Figure showing how Atlas Online Archive partitions data and how it uses it to query documents quickly.
For more information on how to partition data in your Online Archive, please refer to the documentation.

Summary

In this post, we covered some advanced use cases for Online Archive to help you take advantage of this MongoDB Atlas feature. We initialized a demo project to show you how to set up custom archival rules with Atlas Online Archive, as well as improve query performance through partitioning your archived data.
If you have questions, please head to our developer community website where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.

Facebook Icontwitter iconlinkedin icon
Rate this tutorial
star-empty
star-empty
star-empty
star-empty
star-empty
Related
Quickstart

Getting Your Free MongoDB Atlas Cluster


Jan 26, 2023 | 1 min read
Tutorial

Securely Connect MongoDB to Cloud-Offered Kubernetes Clusters


Sep 09, 2024 | 4 min read
Tutorial

Leveraging MongoDB Atlas Vector Search With LangChain


Sep 18, 2024 | 6 min read
Tutorial

Simplify Semantic Search With LangChain and MongoDB


Oct 28, 2024 | 4 min read
Table of Contents