Utilizing Collection Globbing and Provenance in Data Federation
Rate this tutorial
A common pattern for users of MongoDB running multi-tenant services is to model your data by splitting your different customers into different databases. This is an excellent strategy for keeping your various customers’ data separate from one another, as well as helpful for scaling in the future. But one downside to this strategy is that you can end up struggling to get a holistic view of your data across all of your customers. There are many ways to mitigate this challenge, and one of the primary ones is to copy and transform your data into another storage solution. However, this can lead to some unfortunate compromises. For example, you are now paying more to store your data twice. You now need to manage the copy and transformation process, which can become onerous as you add more customers. And lastly, and perhaps most importantly, you are now looking at a delayed state of your data.
To solve these exact challenges, we’re thrilled to announce two features that will completely transform how you use your cluster data and the ease with which you can remodel it. The first feature is called Provenance. This functionality allows you to tell Data Federation to inject fields into your documents during query time that indicate where they are coming from. For example, you can add the source collection on the Atlas cluster when federating across clusters or you can add the path from your AWS S3 bucket where the data is being read. The great thing is that you can now also query on these fields to only get data from the source of your choice!
The other feature we’re adding is a bit nuanced, and we are calling it “globbing.” For those of you familiar with Atlas Data Federation, you probably know about our “wildcard collections.” This functionality allows you to generate collection names based on the collections that exist in your underlying Atlas clusters or based on sections of paths to your files in S3. This is a handy feature to avoid having to explicitly define everything in your storage configuration. “Globbing” is somewhat similar, except that instead of dynamically generating new collections for each collection in your cluster, it will dynamically merge collections to give you a “global” view of your data automatically. To help illustrate this, I’m going to walk you through an example.
Imagine you are running a successful travel agency on top of MongoDB. For various reasons, you have chosen to store your customers data in different databases based on their location. (Maybe you are going to shard based on this and will have different databases in different regions for compliance purposes.)
This has worked well, but now you’d like to query your data based on this information and get a holistic view of your data across geographies in real time (without impacting your operational workloads). So let’s discuss how to solve this challenge!
In order to follow along with this tutorial yourself, you will need the following:
- Experience with Atlas Data Federation.
- An Atlas cluster with the sample data in it.
Here is how the data is modeled in my cluster (data in your cluster can be spread out among collections however your application requires):
- Cluster: MongoTravelServices
- Database: ireland
- Collection: user_feedback (8658 Documents)
- Collection: passengers
- Collection: flights
- Database: israel
- Collection: user_feedback (8658 Documents)
- Collection: passengers
- Collection: flights
- Database: usa
- Collection: user_feedback (8660 Documents)
- Collection: passengers
- Collection: flights
The goal here is to consolidate this data into one database, and then have each of the collections for user feedback, passengers, and flights represent the data stored in the collections from each database on the cluster. Lastly, we also want to be able to query on the “database” name as if it were part of our documents.
- The first thing you’ll need to do is navigate to the “Data Federation” tab on the left-hand side of your Atlas dashboard and then click “set up manually” in the "create new federated database" dropdown in the top right corner of the UI.
- Then, for this example, we’re going to manually edit the storage configuration as these capabilities are not yet available in the UI editor.
Now when you connect, you will see:
And a simple count results in the count of all three collections globbed together:
25976 is the sum of 8660 feedback documents from the USA, 8658 from Israel, and 8658 from Ireland.
And lastly, I can query on the provenance metadata using the field “provenancedata.databaseName”:
So, what have we done and what have we learned?
- We saw how quickly and easily you can create a Federated Database in MongoDB Atlas.
- We learned how you can easily combine and reshape data from your underlying Atlas clusters inside of Atlas Data Federation with Collection Globbing. Now, you can easily query one user_feedback collection and have it query data in the user_feedback collections in each database.
- We saw how to add provenance data to our documents and query it.
- Collection globbing is a new feature that applies to Atlas cluster sources and allows dynamic manipulation of source collections similar to “wildcard collections.”
- Provenance allows you to include additional metadata with your documents. You can indicate that data federation should include additional attributes such as source cluster, database, collection, the source path in S3, and more.
- Currently, this is only supported in the Data Federation JSON editor or via setting the Storage Configuration in the shell, not the visual storage configuration editor.
- This is particularly powerful for multi-tenant implementations done in MongoDB.
To learn more about Atlas Data Federation and whether it would be the right solution for you, check out our documentation and tutorials or get started today.