Joe Karlsson

Senior Developer Advocate

Modern always-on applications rely on automatic failover capabilities and real-time data access. [MongoDB Atlas already supports automatic backups](https://docs.mongodb.com/manual/core/backups/) out of the box, but you might still want to copy your data into another location to run advanced analytics on your data or isolate your operational workload. For this reason, it can be incredibly useful to set up automatic continuous replication of your data for your workload.

In this post, we are going to set up a way to continuously copy data from a MongoDB database into an [AWS S3](https://aws.amazon.com/s3/) bucket in the [Parquet](https://parquet.apache.org/) data format by using [MongoDB Atlas Database Triggers](https://docs.atlas.mongodb.com/triggers/). We will first set up a Federated Database Instance using [MongoDB Atlas Data Federation](https://www.mongodb.com/docs/atlas/data-federation/overview/) to consolidate a MongoDB database and our AWS S3 bucket. Next, we will set up a Trigger to automatically add a new document to a collection every minute, and another Trigger to automatically copy our data to our S3 bucket. Then, we will run a test to ensure that our data is being continuously copied into S3 from MongoDB. Finally, we’ll cover some items you’ll want to consider when building out something like this for your application.

Note: The values we use for certain parameters in this blog are for demonstration and testing purposes. If you plan on utilizing this functionality, we recommend you look at the “Production Considerations” section and adjust based on your needs.

## What is Parquet?

For those of you not familiar with [Parquet](https://parquet.apache.org/), it's an amazing file format that does a lot of the heavy lifting to ensure blazing fast query performance on data stored in files. This is a popular file format in the Data Warehouse and Data Lake space as well as for a variety of machine learning tasks.

One thing we frequently see users struggle with is getting NoSQL data into Parquet as it is a [columnar format](https://en.wikipedia.org/wiki/Column-oriented_DBMS).  Historically, you would have to write some custom code to get the data out of the database, transform it into an appropriate structure, and then probably utilize a third-party library to write it to Parquet.  Fortunately, with [MongoDB Atlas Data Federation](https://www.mongodb.com/docs/atlas/data-federation/overview/)'s $out to S3, you can now convert MongoDB Data into Parquet with little effort.

## Prerequisites

In order to follow along with this tutorial yourself, you will need to
do the following:

1.  Create a [MongoDB Atlas](https://www.mongodb.com/cloud/atlas?tck=docs_atlas) account, if you do not have one already.
2.  Create [an AWS account with privileges to create IAM Roles and S3 Buckets](https://docs.mongodb.com/datalake/deployment/deploy-s3/) (to give Data Federation access to write data to your S3 bucket). Already have an AWS account? Atlas supports paying for usage via the AWS Marketplace (AWS MP) without any upfront commitment — simply <a href="https://aws.amazon.com/marketplace/pp/prodview-pp445qepfdy34?trk=38088f03-4928-4b00-9a39-86781df6ab9b&sc_channel=el" target="_blank" rel="noreferrer">sign up for MongoDB Atlas via AWS Marketplace.<a>
3.  [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html).  4.  [Configure the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration).
5.  *Optional*: [Set up unified AWS access](https://docs.atlas.mongodb.com/security/set-up-unified-aws-access/).

## Create a Federated Database Instance and Connect to S3

We need to set up a Federated Database Instance to copy our MongoDB data and utilize [MongoDB Atlas Data Federation](https://www.mongodb.com/docs/atlas/data-federation/overview/)'s $out to S3 to convert our MongoDB Data into Parquet and land it in an S3 bucket.

The first thing you'll need to do is navigate to "Data Federation" on the left-hand side of your Atlas Dashboard and then click “set up manually” in the "create new federated database" dropdown in the top right corner of the UI.

![Creating new federated database](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc94edfdcc342265b/649c786b84a4c7eb46d8db62/data-federation.png)

Then, you need to go ahead and connect your S3 bucket to your Federated Database Instance. This is where we will write the Parquet files. The setup wizard should guide you through this pretty quickly, but you will need access to your credentials for AWS.

>Note: For more information, be sure to refer to the documentation on [deploying a Federated Database Instance for a S3 data store](https://www.mongodb.com/docs/atlas/data-federation/deployment/deploy-s3/). (Be sure to give Atlas Data Federation "Read and Write" access to the bucket, so it can write the Parquet files there).

![Screenshot of the add data source modal](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2022_10_08_at_1_34_12_PM_024ac20c81.png)

Select an AWS IAM role for Atlas.

-   If you created a role that Atlas is already authorized to read and write to your S3 bucket, select this user.
-   If you are authorizing Atlas for an existing role or are creating a new role, be sure to [refer to the documentation](https://docs.mongodb.com/datalake/deployment/deploy-s3/#select-an-iam-role-for) for how to do this.

Enter the S3 bucket information.

-   Enter the name of your S3 bucket. I named my bucket `mongodb-data-lake-demo`.
-   Choose Read and write, to be able to write documents to your S3 bucket.

Assign an access policy to your AWS IAM role.

-   Follow the steps in the Atlas user interface to assign an access policy to your AWS IAM role.
-   Your role policy for read-only or read and write access should look similar to the following:

``` json
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Effect": "Allow",
            "Action": [
               "s3:ListBucket",
               "s3:GetObject",
               "s3:GetObjectVersion",
               "s3:GetBucketLocation"
            ],
            "Resource": [
               <role arn>
            ]
      }
   ]
}
```

-   Define the path structure for your files in the S3 bucket and click Next.
-   Once you've connected your S3 bucket, we're going to create a simple data source to query the data in S3, so we can verify we've written the data to S3 at the end of this tutorial.

## Connect Your MongoDB Database to Your Federated Database Instance

Now, we're going to connect our Atlas Cluster, so we can write data from it into the Parquet files on S3. This involves picking the cluster from a list of clusters in your Atlas project and then selecting the databases and collections you'd like to create Data Sources from and dragging them into your Federated Database Instance.

![Screenshot of the Add Data Source modal with collections selected](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2022_10_08_at_12_45_36_PM_8755ec364c.png)

![Screenshot of the configuration screen](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2022_10_08_at_12_50_25_PM_3518d76180.png)


## Create a MongoDB Atlas Trigger to Create a New Document Every Minute

Now that we have all of our data sources set up in our brand new Federated Database Instance, we can now set up a [MongoDB Database Trigger](https://docs.atlas.mongodb.com/triggers/) to automatically generate new documents every minute for our continuous replication demo. **Triggers** allow you to execute server-side logic in response to database events or according to a schedule. Atlas provides two kinds of Triggers: **Database** and **Scheduled** triggers. We will use a **Scheduled** trigger to ensure that these documents are automatically archived in our S3 bucket.

1.  Click the Atlas tab in the top navigation of your screen if you have not already navigated to Atlas.
2.  Click Triggers in the left-hand navigation.
3.  On the Overview tab of the Triggers page, click Add Trigger to open the trigger configuration page.
4.  Enter these [configuration values](https://docs.atlas.mongodb.com/triggers/trigger-configuration/#std-label-database-trigger-configuration) for our trigger:

![Screenshot from MongoDB Atlas Trigger configuration page showing the options for our new Trigger. Trigger type: Scheduled. Name: Create_Event_Every_Min_Trigger. Enable: On. Schedule Type: Basic. Link Data Sources: Cluster 1 and atlas_data_lake. Select an Event Type: Function.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/how-to/automated-continuous-data-copying-from-mongodb-to-s3/afd684b2c8ef9c090fdfd721833ed1592ea88db0.png)

And our Trigger function looks like this:

``` javascript
exports = function () {

   const mongodb = context.services.get("NAME_OF_YOUR_ATLAS_SERVICE");
   const db = mongodb.db("NAME_OF_YOUR DATABASE")
   const events = db.collection("NAME_OF_YOUR_COLLECTION");

   const event = events.insertOne(
      {
            time: new Date(),
            aNumber: Math.random() * 100,
            type: "event"
      }
   );

   return JSON.stringify(event);

};
```

Lastly, click Run and check that your database is getting new documents inserted into it every 60 seconds.

![Screenshot from the MongoDB Atlas webpage showing the data that was generated by the Create_Event_Every_Min_Trigger with a red box around the new data.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/how-to/automated-continuous-data-copying-from-mongodb-to-s3/02d89f1e430a91cad359369fcba9930129c2dca8.png)

## Create a MongoDB Atlas Trigger to Copy New MongoDB Data into S3 Every Minute

Alright, now is the fun part. We are going to create a new MongoDB Trigger that copies our MongoDB data every 60 seconds utilizing MongoDB Atlas Data Federation's $out to S3 aggregation pipeline. Create a new Trigger and use these configuration settings.

![Screenshot from MongoDB Atlas Trigger configuration page showing the options for our new Trigger](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2022_10_08_at_1_56_39_PM_19367c8d81.png)

Your Trigger function will look something like this. But there's a lot going on, so let's break it down.

* First, we are going to connect to our new Federated Database Instance. This is different from the previous Trigger that connected to our Atlas database. Be sure to put your virtual database name in for `context.services.get`. You must [connect](https://docs.mongodb.com/datalake/tutorial/connect/#std-label-data-lake-connect) to your Federated Database Instance to use [$out](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/pipeline/out/) to S3.
* Next, we are going to create an aggregation pipeline function to first query our MongoDB data that's more than 60 seconds old.
* Then, we will utilize the $out aggregate operator to replicate the data from our previous aggregation stage into S3.
* In the format, we're going to specify *parquet* and determine a maxFileSize and maxRowGroupSize.
    * *maxFileSize* is going to determine the maximum size each
        partition will be.
    *maxRowGroupSize* is going to determine how records are grouped inside of the parquet file in "[row groups](https://parquet.apache.org/docs/concepts/)" which will impact performance querying your Parquet files similarly to file size.
* Lastly, we’re going to set our S3 path to match the value of the data.

``` javascript
exports = function () {

   const service = context.services.get("NAME_OF_YOUR_FEDERATED_DATA_SERVICE");
   const db = service.db("NAME_OF_YOUR_VIRTUAL_DATABASE")
   const events = db.collection("NAME_OF_YOUR_VIRTUAL_COLLECTION");

   const pipeline = [
      {
            $match: {
               "time": {
                  $gt: new Date(Date.now() - 60 * 60 * 1000),
                  $lt: new Date(Date.now())
               }
            }
      }, {
            "$out": {
               "s3": {
                  "bucket": "mongodb-federated-data-demo",
                  "region": "us-east-1",
                  "filename": "events",
                  "format": {
                        "name": "parquet",
                        "maxFileSize": "10GB",
                        "maxRowGroupSize": "100MB"
                  }
               }
            }
      }
   ];

   return events.aggregate(pipeline);
};
```

If all is good, you should see your new Parquet document in your S3 bucket. I've enabled the AWS GUI to show you the versions so that you can see how it is being updated every 60 seconds automatically.

![Screenshot from AWS S3 management console showing the new events.parquet document that was generated by our $out trigger function.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/how-to/automated-continuous-data-copying-from-mongodb-to-s3/2694de087922972709633334785ab9f5cce3f333.png)

## Production Considerations

Some of the configurations chosen above were done so to make it easy to set up and test, but if you’re going to use this in production, you’ll want to adjust them.

Firstly, this blog was setup with a “deltas” approach. This means that we are only copying the new documents from our collection into our Parquet files. Another approach would be to do a full snapshot, i.e., copying the entire collection into Parquet each time. The approach you’re taking should depend on how much data is in your collection and what’s required by the downstream consumer.

Secondly, regardless of how much data you’re copying, ideally you want Parquet files to be larger, and for them to be partitioned based on how you’re going to query. [Apache recommends](https://parquet.apache.org/docs/file-format/configurations/) row group sizes of 512MB to 1GB. You can go smaller depending on your requirements, but as you can see, you want larger files. The other consideration is if you plan to query this data in the parquet format, you should partition it so that it aligns with your query pattern. If you’re going to query on a date field, for instance, you might want each file to have a single day's worth of data.

Lastly, depending on your needs, it may be appropriate to look into an alternative scheduling device to triggers, like [Temporal](https://docs.temporal.io/) or [Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/index.html).

## Wrap Up

In this post, we walked through how to set up an automated continuous replication from a MongoDB database into an [AWS S3](https://aws.amazon.com/s3/) bucket in the [Parquet](https://parquet.apache.org/) data format by using [MongoDB Atlas Data Federation](https://www.mongodb.com/docs/atlas/data-federation/overview/) and [MongoDB Atlas Database Triggers](https://docs.atlas.mongodb.com/triggers/). First, we set up a new Federated Database Instance to consolidate a MongoDB database and our AWS S3 bucket. Then, we set up a Trigger to automatically add a new document to a collection every minute, and another Trigger to automatically back up these new automatically generated documents into our S3 bucket.

We also discussed how Parquet is a great format for your MongoDB data when you need to use columnar-oriented tools like Tableau for visualizations or Machine Learning frameworks that use Data Frames. Parquet can be quickly and easily converted into Pandas Data Frames in Python.

>If you have questions, please head to our [developer community website](https://community.mongodb.com/) where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.

Additional Resources:

-   [Data Federation: Getting Started Documentation](https://docs.mongodb.com/datalake/tutorial/getting-started/)
-   [$out S3 Data Lake Documentation](https://docs.mongodb.com/datalake/reference/pipeline/out/#std-label-adl-out-stage)

Learn how to set up a continuous copy from MongoDB into an AWS S3 bucket in Parquet.

How to Automate Continuous Data Copying from MongoDB to S3

Benjamin Flast

Product Manager, Data Lake

For those of you not familiar with [Parquet](https://parquet.apache.org/), it’s an amazing file format that does a lot of the heavy lifting to ensure blazing fast query performance on data stored in files. This is a popular file format in the Data Warehouse and Data Lake space as well as for a variety of machine learning tasks.

One thing we frequently see users struggle with is getting NoSQL data into Parquet as it is a columnar format. Historically, you would have to write some custom code to get the data out of the database, transform it into an appropriate structure, and then probably utilize a third-party library to write it to Parquet. Fortunately, with MongoDB Atlas Data Federation's $out to cloud object storage - Amazon S3 or Microsoft Azure Blob Storage, you can now convert MongoDB Data into Parquet with little effort.

In this blog post, I’m going to walk you through the steps necessary to write data from your Atlas Cluster directly to cloud object storage in the Parquet format and then finish up by reviewing some things to keep in mind when using Parquet with NoSQL data. I’m going to use a sample data set that contains taxi ride data from New York City.

## Prerequisites

In order to follow along with this tutorial yourself, you will need the following:
An Atlas cluster with some data in it. (It can be the sample data.)
An AWS account with privileges to create IAM Roles and cloud object storage buckets (to give us access to write data to your cloud object storage bucket).

## Create a Federated Database Instance and Connect to cloud object storage

The first thing you'll need to do is navigate to the "Data Federation" tab on the left hand side of your Atlas Dashboard and then click “set up manually” in the "create new federated database" dropdown in the top right corner of the UI.

![Data Federation](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc94edfdcc342265b/649c786b84a4c7eb46d8db62/data-federation.png)

Then, you need to connect your cloud object storage bucket to your Federated Database Instance. This is where we will write the Parquet files. The setup wizard should guide you through this pretty quickly but you will need access to your credentials for AWS. (Be sure to give Atlas Data Federation “Read and Write” access to the bucket so it can write the Parquet files there.)

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_12_19_125037_440fb93c33.jpg)

Once you’ve connected your cloud object storage bucket, we’re going to create a simple data source to query the data in cloud object storage so we can verify we’ve written the data to cloud object storage at the end of this tutorial. Our new setup tool makes it easier than ever to configure your Federated Database Instance to take advantage of the partitioning of data in cloud object storage. Partitioning allows us to only select the relevant data to process in order to satisfy your query. (I’ve put a sample file in there for this test that will fit how we’re going to partition the data by \_cab\_type).

``` bash
mongoimport --uri mongodb+srv://<USERNAME>:<PASSWORD>@<MONGODB_URI>/<DATABASE> --collection <COLLECTION> --type json --file <FILENAME>
```

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_12_20_164216_22544c3908.jpg)

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_12_20_163429_6d7a53327c.jpg)

## Connect Your Federated Database Instance to an Atlas Cluster

Now we’re going to connect our Atlas cluster, so we can write data from it into the Parquet files. This involves picking the cluster from a list of clusters in your Atlas project and then selecting the databases and collections you’d like to create Data Sources from and dragging them into your Federated Database Instance.

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_12_20_164727_7f084fa42e.jpg)

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_12_20_164926_350b9a7a74.jpg)

## $out to cloud object storage in Parquet

Now we’re going to connect to our Federated Database Instance using the mongo shell and execute the following command. This is going to do quite a few things, so I’m going to explain the important ones.
- First, you can use the ‘filename’ field of the $out stage to have your Federated Database Instance partition files by “_cab_type”, so all the green cabs will go in one set of files and all the yellow cabs will go in another.
- Then in the format, we’re going to specify parquet and determine a maxFileSize and maxRowGroupSize.
   -- maxFileSize is going to determine the maximum size each partition will be.
   -- maxRowGroupSize is going to determine how records are grouped inside of the Parquet file in “row groups” which will impact performance querying your Parquet files, similarly to file size.
- Lastly, we’re using a special Atlas Data Federation aggregation “background: true” which simply tells the Federated Database Instance to keep executing the query even if the client disconnects. (This is handy for long running queries or environments where your network connection is not stable.)

``` js
db.getSiblingDB("clusterData").getCollection("trips").aggregate([
    {
        "$out" : {
            "s3" : {
                "bucket" : "ben.flast",
                "region" : "us-east-1",
                "filename" : {
                    "$concat" : [
                        "taxi-trips/",
                        "$_cab_type",
                        "/"
                    ]
                },
                "format" : {
                    "name" : "parquet",
                    "maxFileSize" : "10GB",
                    "maxRowGroupSize" : "100MB"
                }
            }
        }
    }
], {
    background: true
})
```

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/aws_s3_interface_f21ce56fa9.png)

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/aws_s3_yellow_cab_folder_55e21a7e89.png)

## Blazing Fast Queries on Parquet Files

Now, to give you some idea of the potential performance improvements for Object Store Data you can see, I’ve written three sets of data, each with 10 million documents: one in Parquet, one in uncompressed JSON, and another in compressed JSON. And I ran a count command on each of them with the following results.

*db.trips.count()*
10,000,000

| Type | Data Size (GB) | Count Command Latency (Seconds) |
| ---- | -------------- | ------------------------------- |
| JSON (Uncompressed) | \~16.1 | 297.182 |
| JSON (Compressed) | \~1.1 | 78.070 |
| Parquet | \~1.02 | 1.596 |

## In Review

So, what have we done and what have we learned?
- We saw how quickly and easily you can create a Federated Database Instance in MongoDB Atlas.
- We connected an Atlas cluster to our Federated Database Instance.
- We used our Federated Database Instance to write Atlas cluster data to cloud object storage in Parquet format.
- We demonstrated how fast and space-efficient Parquet is when compared to JSON.

## A Couple of Things to Remember About Atlas Data Federation

- Parquet is a super fast columnar format that can be read and written with Atlas Data Federation.
- Atlas Data Federation takes advantage of various pieces of metadata contained in Parquet files, not just the maxRowGroupSize. For instance, if your first stage in an aggregation pipeline was $project: {fieldA: 1, filedB: 1}, we would only read the two columns from the Parquet file which results in faster performance and lower costs as we are scanning less data.
- Atlas Data Federation writes Parquet files flexibly so if you have polymorphic data, we will create union columns so you can have ‘Column A - String’ and ‘Column A - Int’. Atlas Data Federation will read union columns back in as one field but other tools may not handle union types. So if you’re going to be using these Parquet files with other tools, you should transform your data before the $out stage to ensure no union columns.
- Atlas Data Federation will also write files with different schemas if it encounters data with varying schemas throughout the aggregation. It can handle different schemas across files in one collection, but other tools may require a consistent schema across files. So if you’re going to be using these Parquet files with other tools, you should do a $project with $convert’s before the $out stage to ensure a consistent schema across generated files.
- Parquet is a great format for your MongoDB data when you need to use columnar oriented tools like Tableau for visualizations or machine learning frameworks that use data frames. Parquet can be quickly and easily converted into Pandas data frames in Python.

Learn how to transform MongoDB data to Parquet with Atlas Data Federation.

Parquet

All Parquet Content

How to Automate Continuous Data Copying from MongoDB to S3

How to Get MongoDB Data into Parquet in 10 Seconds or Less

Related Topics