Ado Kukic

Robert Belson

The network edge has been one of the most explosive cloud computing opportunities in recent years. As mobile contactless experiences become the norm and as businesses move ever-faster to digital platforms and services, edge computing is positioned as a faster, cheaper, and more reliable alternative for data processing and compute at scale.

While mobile devices continue to increase their hardware capabilities with built-in GPUs, custom chipsets, and more storage, even the most cutting-edge devices will suffer the same fundamental problem: each device serves as a single point of failure and, thus, cannot effectively serve as a persistent data storage layer. Said differently, wouldn’t it be nice to have the high-availability of the cloud but with the topological distance to your end users of the smartphone?

Mobile edge computing promises to precisely address this problem—bringing low latency compute to the edge of networks with the high-availability and scale of cloud computing. Through Verizon 5G Edge with AWS Wavelength, we saw the opportunity to explore how to take existing compute-intensive workflows and overlay a data persistence layer with MongoDB, utilizing the MongoDB Atlas management platform, to enable ultra-immersive experiences with personalized experience—reliant on existing database structures in the parent region with the seamlessness to extend to the network edge. 

![About Verizon 5G Edge and MongoDB](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/mongodb_verizon_5g_edge_66e1e2d536.png)

In this article, learn how Verizon and MongoDB teamed up to deliver on this vision, a quick Getting Started guide to build your first MongoDB application at the edge, and advanced architectures for those proficient with MongoDB.

Let’s get started!

## About Verizon 5G Edge and MongoDB

Through Verizon 5G Edge, AWS developers can now deploy parts of their application that require low latency at the edge of 4G and 5G networks using the same AWS APIs, tools, and functionality they use today, while seamlessly connecting back to the rest of their application and the full range of cloud services running in an AWS Region. By embedding AWS compute and storage services at the edge of the network, use cases such as ML inference, real-time video streaming, remote video production, and game streaming can be rapidly accelerated.

However, for many of these use cases, a persistent storage layer is required that extends beyond the native storage capabilities of AWS Wavelength—namely Elastic Block Storage (EBS) volumes. However, using MongoDB Enterprise, developers can leverage the underlying compute (i.e.,. EC2 instances) at the edge to deploy MongoDB clusters either a) as standalone clusters or b) highly available replica sets that can synchronize data seamlessly.

MongoDB is a general purpose, document-based, distributed database built for modern application developers. With MongoDB Atlas, developers can get up and running even faster with fully managed MongoDB databases deployed across all major cloud providers.

While [MongoDB Atlas](https://mongodb.com/atlas) today does not support deployments within Wavelength Zones, [MongoDB Cloud Manager](https://www.mongodb.com/cloud/cloud-manager) can automate, monitor, and back up your MongoDB infrastructure. Cloud Manager Automation enables you to configure and maintain MongoDB nodes and clusters, whereby MongoDB Agents running on each MongoDB host can maintain your MongoDB deployments. In this example, we’ll start with a fairly simple architecture highlighting the relationship between Wavelength Zones (the edge) and the Parent Region (core cloud):

Just like any other architecture, we’ll begin with a VPC consisting of two subnets. Instead of one public subnet and one private subnet, we’ll have one public subnet and one carrier subnet —a new way to describe subnets exposed within Wavelength Zones to the mobile network only.

![AWS Diagram](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/diagram_49b0c08be0.png)

* **Public Subnet**: Within the us-west-2 Oregon region, we launched a subnet in us-west-2a availability zone consisting of a single EC2 instance with a public IP address. From a routing perspective, we attached an Internet Gateway to the VPC to provide outbound connectivity and attached the Internet Gateway as the default route (0.0.0.0/0) to the subnet’s associated route table.
* **Carrier Subnet**: Also within the us-west-2 Oregon region, our second subnet is in the San Francisco Wavelength Zone (us-west-2-wl1-sfo-wlz-1) —an edge data center within the Verizon carrier network but part of the us-west-2 region. In this subnet, we also deploy a single EC2 instance, this time with a carrier IP address—a carrier network-facing IP address exposed to Verizon mobile devices. From a routing perspective, we attached a Carrier Gateway to the VPC to provide outbound connectivity and attached the Carrier Gateway as the default route (0.0.0.0/0) to the subnet’s associated route table.

Next, let’s configure the EC2 instance in the parent region. Once you get the IP address (54.68.26.68) of the launched EC2 instance, SSH into the instance itself and begin to download the MongoDB agent.

```bash
ssh -i "mongovz.pem" ec2-user@ec2-54-68-26-68.us-west-2.compute.amazonaws.com
```

Once you are in, download and install the packages required for the MongoDB MMS Automation Agent. Run the following command:

```bash
sudo yum install cyrus-sasl cyrus-sasl-gssapi \
     cyrus-sasl-plain krb5-libs libcurl \
     lm_sensors-libs net-snmp net-snmp-agent-libs \
     openldap openssl tcp_wrappers-libs xz-libs
```

Once within the instance, download the MongoDB MMS Automation Agent, and install the agent using the RPM package manager.

```bash
curl -OL https://cloud.mongodb.com/download/agent/automation/mongodb-mms-automation-agent-manager-10.30.1.6889-1.x86_64.rhel7.rpm

sudo rpm -U mongodb-mms-automation-agent-manager-10.30.1.6889-1.x86_64.rhel7.rpm
```

Next, navigate to the **/etc/mongodb-mms/** and edit the **automation-agent.config** file to include your MongoDB Cloud Manager API Key. To create a key, head over to MongoDB Atlas at [https://mongodb.com/atlas](https://mongodb.com/atlas) and either login to an existing account, or sign up for a new free account.

Once you are logged in, create a new organization, and for the cloud service, be sure to select Cloud Manager.

![Cloud Manager Setup](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/cloud_manager_1_7118a3c187.png)

With your organization created, next we’ll create a new Project. When creating a new project, you may be asked to select a cloud service, and you’ll choose Cloud Manager again.

![Selecting Cloud Manager on Atlas](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/cloud_manager_2_bdc0385fbc.png)

Next, you’ll name your project. You can select any name you like, we’ll go with Verizon for our project name. After you give your project a name, you will be given a prompt to invite others to the project. You can skip this step for now as you can always add additional users in the future.

![Creating a Cloud Manager Project](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/cloud_manager_3_0afe0790e2.png)

Finally, you are ready to deploy MongoDB to your environment using Cloud Manager. With Cloud Manager, you can deploy both standalone instances as well as Replica Sets of MongoDB. Since we want high availability, we’ll deploy a replica set.

![Creating a Replica Set on Cloud Manager](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/cloud_manager_4_0b43dada9f.png)

Clicking on the **New Replica Set** button will bring us to the user interface to configure our replica set. At this point, we’ll probably get a message saying that no servers were detected, and that’s fine since we haven’t started our MongoDB Agents yet. 

![Replica Set Configuration](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/cloud_manager_5_d24084d445.png)

Click on the “see instructions” link to get more details on how to install the MongoDB Agent. On the modal that pops up, it will have familiar instructions that we’re already following, but it will also have two pieces of information that we’ll need. The **mmsApiKey** and **mmsGroupId** will be displayed here and you’ll likely have to click the Generate Key button to generate a new mmsAPIKey which will be automatically populated. Make note of these **mmsGroupId** and **mmsApiKey** values as we’ll need when configuring our MongoDB Agents next.

![Replica Set Instructions](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/cloud_manager_6_6e86d6ddbd.png)

Head back to your terminal for the EC2 instance and navigate to the **/etc/mongodb-mms/** and edit the **automation-agent.config** file to include your MongoDB Cloud Manager API Key. 

![Automation Agent Configuration](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/automation_agent_config_cf4383b9e7.png)

In this example, we edited the **mmsApiKey** and **mmsGroupId** variables. From there, we’ll create the data directory and start our MongoDB agent!

```bash
sudo mkdir -p /data
sudo chown mongod:mongod /data
sudo systemctl start mongodb-mms-automation-agent.service
```

Once you’ve completed these configuration steps, go ahead and do the same for your Wavelength Zone instance. Note that you will not be able to SSH directly to the instance’s Carrier IP (155.146.16.178/). Instead, you must use the parent region instance as a bastion host to “jump” onto the edge instance itself. To do so, find the private IP address of the edge instance (10.0.0.54) and, from the parent region instance, SSH into the second instance using the same key pair you used.

```bash
ssh -i "mongovz.pem" ec2-user@10.0.0.54
```

After completing configuration of the second instance, which follows the same instructions from above, it’s time for the fun part —launching the ReplicaSet on the Cloud Manager Console! The one thing to note for the replica set, since we’ll have three nodes, on the edge instance we’ll create a /data and /data2 directories to allow for two separate directories to host the individual nodes data. Head back over to [https://mongodb.com/atlas](https://mongodb.com/atlas) and the Cloud Manager to complete setup.

Refresh the Create New Replica Set page and now since the MongoDB Agents are running you should see a lot of information pre-populated for you. Make sure that it matches what you’d expect and when you’re satisfied hit the Create Replica Set button.

![Finalize Creating a Replica Set on Cloud Manager](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/cloud_manager_7_03e29fb2aa.png)

Click on the “Create Replica Set” button to finalize the process.

Within a few minutes the replica set cluster will be deployed to the servers and your MongoDB cluster will be up and running. 

![Cloud Manager Replica Set Deployed](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/cloud_manager_8_debf956fbc.png)

With the replica set deployed, you should now be able to connect to your MongoDB cluster hosted on either the standard Us-West or Wavelength zone. To do this, you’ll need the public address for the cluster and the port as well as Authentication enabled in Cloud Manager. To enable Authentication, simply click on the Enabled/Disabled button underneath the Auth section of your replica set and you’ll be given a number of options to connect to the client. We’ll select Username/password.

![Cloud Manager User Account Setup](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/cloud_manager_9_dcdd123ceb.png)

Click Next, and the subsequent modal will have your username and password to connect to the cluster with.

![Cloud Manager User Account](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/cloud_manager_10_38fbb3c3bf.png)

You are all set. Next, let’s see how the MongoDB performs at the edge. We’ll test this by reading data from both our standard US-West node as well as the Wavelength zone and compare our results.

## Racing MongoDB at the Edge

After laying out the architecture, we wanted to see the power of 5G Edge in action. To that end, we designed a very simple “race.” Over 1,000 trials we would read data from our MongoDB database, and timestamp each operation both from the client to the edge and to the parent region. 

```python
from pymongo import MongoClient
import time
client = MongoClient('155.146.144.134', 27017)
mydb = client["mydatabase"]
mycol = mydb["customers"]
mydict = { "name": "John", "address": "Highway 37" }

# Load dataset
for i in range(1000):
    x = mycol.insert(mydict)

# Measure reads from Parent Region
edge_latency=[]
for i in range(1000):
    t1=time.time()
    y = mycol.find_one({"name":"John"})
    t2=time.time()
    edge_latency.append(t2-t1)

print(sum(edge_latency)/len(edge_latency))

client = MongoClient('52.42.129.138', 27017)
mydb = client["mydatabase"]
mycol = mydb["customers"]
mydict = { "name": "John", "address": "Highway 37" }

# Measure reads from Wavelength Region
edge_latency=[]
for i in range(1000):
    t1=time.time()
    y = mycol.find_one({"name":"John"})
    t2=time.time()
    edge_latency.append(t2-t1)

print(sum(edge_latency)/len(edge_latency))
```

After running this experiment, we found that our MongoDB node at the edge performed **over 40% faster** than the parent region! But why was that the case? 

Given that the Wavelength Zone nodes were deployed within the mobile network, packets never had to leave the Verizon network and incur the latency penalty of traversing through the public internet—prone to incremental jitter, loss, and latency. In our example, our 5G Ultra Wideband connected device in San Francisco had two options: connect to a local endpoint within the San Francisco mobile network or travel 500+ miles to a data center in Oregon. Thus, we validated the significant performance savings of MongoDB on Verizon 5G Edge relative to the next best alternative: deploying the same architecture in the core cloud.

## Getting started on 5G Edge with MongoDB

While Verizon 5G Edge alone enables developers to build ultra-immersive applications, how can immersive applications become personalized and localized?

Enter MongoDB. 

From real-time transaction processing, telemetry capture for your IoT application, or personalization using profile data for localized venue experiences, bringing MongoDB ReplicaSets to the edge allows you to maintain the low latency characteristics of your application without sacrificing access to user profile data, product catalogues, IoT telemetry, and more.

There’s no better time to start your edge enablement journey with Verizon 5G Edge and MongoDB. To learn more about Verizon 5G Edge, you can visit our [developer resources page](https://www.verizon.com/business/solutions/5g/edge-computing/developer-resources/). If you have any questions about this blog post, find us in the [MongoDB community](https://community.mongodb.com).

In our next post, we will demonstrate how to build your first image classifier on 5G Edge using MongoDB to identify VIPs at your next sporting event, developer conference, or large-scale event.

From real-time transaction processing, telemetry capture for your IoT application, or personalization using profile data for localized venue experiences, bringing MongoDB to the edge allows you to maintain the low latency characteristics of your application without sacrificing access to data.

Real-time Data Architectures with MongoDB Cloud Manager and Verizon 5G Edge

Joe Karlsson

>As of June 2022, the functionality previously known as Atlas Data Lake is now named Atlas Data Federation. Atlas Data Federation’s functionality is unchanged and you can learn more about it [here](http://mongodb.com/atlas/data-federation). Atlas Data Lake will remain in the Atlas Platform, with newly introduced functionality that you can learn about [here](http://mongodb.com/atlas/data-lake).

Modern always-on applications rely on automatic failover capabilities and real-time data access. [MongoDB Atlas already supports automatic backups](https://docs.mongodb.com/manual/core/backups/) out of the box, but you might still want to copy your data into another location to run advanced analytics on your data or isolate your operational workload. For this reason, it can be incredibly useful to set up automatic continuous replication of your data for your workload.

In this post, we are going to set up a way to continuously copy data from a MongoDB database into an [AWS S3](https://aws.amazon.com/s3/) bucket in the [Parquet](https://parquet.apache.org/) data format by using [MongoDB Atlas Database Triggers](https://docs.atlas.mongodb.com/triggers/). We will first set up a [MongoDB Atlas Data Lake](https://docs.mongodb.com/datalake/) to consolidate a MongoDB database and our AWS S3 bucket. Next, we will set up a Trigger to automatically add a new document to a collection every minute, and another Trigger to automatically copy our data to our S3 bucket. Then, we will run a test to ensure that our data is being continuously copied into S3 from MongoDB. Finally, we’ll cover some items you’ll want to consider when building out something like this for your application.

Note: The values we use for certain parameters in this blog are for demonstration and testing purposes. If you plan on utilizing this functionality, we recommend you look at the “Production Considerations” section and adjust based on your needs.

## What is Parquet?

For those of you not familiar with [Parquet](https://parquet.apache.org/), it's an amazing file format that does a lot of the heavy lifting to ensure blazing fast query performance on data stored in files. This is a popular file format in the Data Warehouse and Data Lake space as well as for a variety of machine learning tasks.

One thing we frequently see users struggle with is getting NoSQL data into Parquet as it is a [columnar format](https://en.wikipedia.org/wiki/Column-oriented_DBMS).  Historically, you would have to write some custom code to get the data out of the database, transform it into an appropriate structure, and then probably utilize a third-party library to write it to Parquet.  Fortunately, with [MongoDB Atlas Data Lake](https://docs.mongodb.com/datalake/)'s $out to S3, you can now convert MongoDB Data into Parquet with little effort.

## Prerequisites

In order to follow along with this tutorial yourself, you will need to
do the following:

1.  Create a [MongoDB Atlas](https://www.mongodb.com/cloud/atlas?tck=docs_atlas) account, if you do not have one already.
2.  Create [an AWS account with privileges to create IAM Roles and S3 Buckets](https://docs.mongodb.com/datalake/deployment/deploy-s3/) (to give Data Lake access to write data to your S3 bucket).
3.  [Install the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html).  4.  [Configure the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration).
5.  *Optional*: [Set up unified AWS access](https://docs.atlas.mongodb.com/security/set-up-unified-aws-access/).

## Create a MongoDB Atlas Data Lake and Connect to S3

We need to set up a MongoDB Atlas Data Lake to copy our MongoDB data and utilize [MongoDB Atlas Data Lake](https://docs.mongodb.com/datalake/)'s $out to S3 to convert our MongoDB Data into Parquet and land it in an S3 bucket.

The first thing you'll need to do is navigate to "Data Lake" on the left-hand side of your Atlas Dashboard and then click "Create Data Lake" or "Configure a New Data Lake."

![Screenshot from MongoDB Atlas showing the Create a Data Lake page with red arrows pointing to the Create Data Lake and Configure a New Data Lake buttons.](https://www.mongodb.com/developer/images/how-to/automated-continuous-data-copying-from-mongodb-to-s3/317875632652054d11686821e75c4a29513f622b.png)

Then, you need to go ahead and connect your S3 bucket to your Atlas Data Lake. This is where we will write the Parquet files. The setup wizard should guide you through this pretty quickly, but you will need access to your credentials for AWS.

>Note: For more information, be sure to refer to the documentation on [deploying a Data Lake for a S3 data store](https://docs.mongodb.com/datalake/deployment/deploy-s3/). (Be sure to give Atlas Data Lake "Read and Write" access to the bucket, so it can write the Parquet files there).

![Screenshot from MongoDB Atlas Data Lake configuration modal with the AWS S3 data store option selected.](https://www.mongodb.com/developer/images/how-to/automated-continuous-data-copying-from-mongodb-to-s3/db52bafc35e995edaf8b6e37d7ba8b89c50d68ef.png)

Select an AWS IAM role for Atlas.

-   If you created a role that Atlas is already authorized to read and write to your S3 bucket, select this user.
-   If you are authorizing Atlas for an existing role or are creating a new role, be sure to [refer to the documentation](https://docs.mongodb.com/datalake/deployment/deploy-s3/#select-an-iam-role-for) for how to do this.

Enter the S3 bucket information.

-   Enter the name of your S3 bucket. I named my bucket `mongodb-data-lake-demo`.
-   Choose Read and write, to be able to write documents to your S3 bucket.

Assign an access policy to your AWS IAM role.

-   Follow the steps in the Atlas user interface to assign an access policy to your AWS IAM role.
-   Your role policy for read-only or read and write access should look similar to the following:

``` json
{
   "Version": "2012-10-17",
   "Statement": [
      {
            "Effect": "Allow",
            "Action": [
               "s3:ListBucket",
               "s3:GetObject",
               "s3:GetObjectVersion",
               "s3:GetBucketLocation"
            ],
            "Resource": [
               <role arn>
            ]
      }
   ]
}
```

-   Define the path structure for your files in the S3 bucket and click Next.
-   Once you've connected your S3 bucket, we're going to create a simple data source to query the data in S3, so we can verify we've written the data to S3 at the end of this tutorial.

## Connect Your MongoDB Database to Your Data Lake

Now, we're going to connect our Atlas Cluster, so we can write data from it into the Parquet files on S3. This involves picking the cluster from a list of clusters in your Atlas project and then selecting the databases and collections you'd like to create Data Sources from and dragging them into your Data Lake.

![Screenshot from MongoDB Atlas showing the MongoDB Atlas Cluster option selected and the cluster with our event data in it also selected.](https://www.mongodb.com/developer/images/how-to/automated-continuous-data-copying-from-mongodb-to-s3/0018f0560604fd378ff2dba5e3137f192ddf280d.png)

![Screenshot from MongoDB Atlas Data Lake configuration page showing how the Atlas Cluster and S3 data source have been moved into our new Data Lake.](https://www.mongodb.com/developer/images/how-to/automated-continuous-data-copying-from-mongodb-to-s3/f428363886c9d2fa47518fcd552a2318f06929bb.png)

## Create a MongoDB Atlas Trigger to Create a New Document Every Minute

Now that we have all of our data sources set up in our brand new Data Lake, we can now set up a [MongoDB Database Trigger](https://docs.atlas.mongodb.com/triggers/) to automatically generate new documents every minute for our continuous replication demo. **Triggers** allow you to execute server-side logic in response to database events or according to a schedule. Atlas provides two kinds of Triggers: **Database** and **Scheduled** triggers. We will use a **Scheduled** trigger to ensure that these documents are automatically archived in our S3 bucket.

1.  Click the Atlas tab in the top navigation of your screen if you have not already navigated to Atlas.
2.  Click Triggers in the left-hand navigation.
3.  On the Overview tab of the Triggers page, click Add Trigger to open the trigger configuration page.
4.  Enter these [configuration values](https://docs.atlas.mongodb.com/triggers/trigger-configuration/#std-label-database-trigger-configuration) for our trigger:

![Screenshot from MongoDB Atlas Trigger configuration page showing the options for our new Trigger. Trigger type: Scheduled. Name: Create_Event_Every_Min_Trigger. Enable: On. Schedule Type: Basic. Link Data Sources: Cluster 1 and atlas_data_lake. Select an Event Type: Function.](https://www.mongodb.com/developer/images/how-to/automated-continuous-data-copying-from-mongodb-to-s3/afd684b2c8ef9c090fdfd721833ed1592ea88db0.png)

And our Trigger function looks like this:

``` javascript
exports = function () {

   const mongodb = context.services.get("NAME_OF_YOUR_ATLAS_SERVICE");
   const db = mongodb.db("NAME_OF_YOUR DATABASE")
   const events = db.collection("NAME_OF_YOUR_COLLECTION");

   const event = events.insertOne(
      {
            time: new Date(),
            aNumber: Math.random() * 100,
            type: "event"
      }
   );

   return JSON.stringify(event);

};
```

Lastly, click Run and check that your database is getting new documents inserted into it every 60 seconds.

![Screenshot from the MongoDB Atlas webpage showing the data that was generated by the Create_Event_Every_Min_Trigger with a red box around the new data.](https://www.mongodb.com/developer/images/how-to/automated-continuous-data-copying-from-mongodb-to-s3/02d89f1e430a91cad359369fcba9930129c2dca8.png)

## Create a MongoDB Atlas Trigger to Copy New MongoDB Data into S3 Every Minute

Alright, now is the fun part. We are going to create a new MongoDB Trigger that copies our MongoDB data every 60 seconds utilizing MongoDB Data Lake's $out to S3 aggregation pipeline. Create a new Trigger and use these configuration settings.

![Screenshot from MongoDB Atlas Trigger configuration page showing the options for our new Trigger. Trigger type: Scheduled. Name: archive_on_s3_every_min. Enable: On. Schedule Type: Basic. Link Data Sources: Cluster 1 and atlas_data_lake. Select an Event Type: Function.](https://www.mongodb.com/developer/images/how-to/automated-continuous-data-copying-from-mongodb-to-s3/bdb6fec69d5bd9ea2e1afc0d6378a01e8b4e346d.png)

Your Trigger function will look something like this. But there's a lot going on, so let's break it down.

* First, we are going to connect to our new Data Lake. This is different from the previous Trigger that connected to our Atlas database. Be sure to put your Data Lake name in for `context.services.get`. You must [connect](https://docs.mongodb.com/datalake/tutorial/connect/#std-label-data-lake-connect) to your Data Lake to use [$out](https://docs.mongodb.com/manual/reference/operator/aggregation/out/) to S3.
* Next, we are going to create an aggregation pipeline function to first query our MongoDB data that's more than 60 seconds old.
* Then, we will utilize the $out aggregate operator to replicate the data from our previous aggregation stage into S3.
* In the format, we're going to specify *parquet* and determine a maxFileSize and maxRowGroupSize.
    * *maxFileSize* is going to determine the maximum size each
        partition will be.
    *maxRowGroupSize* is going to determine how records are grouped inside of the parquet file in "[row groups](https://parquet.apache.org/documentation/latest/)" which will impact performance querying your Parquet files similarly to file size.
* Lastly, we’re going to set our S3 path to match the value of the data.

``` javascript
exports = function () {

   const datalake = context.services.get("NAME_OF_YOUR_DATA_LAKE_SERVICE");
   const db = datalake.db("NAME_OF_YOUR_DATA_LAKE_DATABASE")
   const events = db.collection("NAME_OF_YOUR_DATA_LAKE_COLLECTION");

   const pipeline = [
      {
            $match: {
               "time": {
                  $gt: new Date(Date.now() - 60 * 60 * 1000),
                  $lt: new Date(Date.now())
               }
            }
      }, {
            "$out": {
               "s3": {
                  "bucket": "mongodb-data-lake-demo",
                  "region": "us-east-1",
                  "filename": "events",
                  "format": {
                        "name": "parquet",
                        "maxFileSize": "10GB",
                        "maxRowGroupSize": "100MB"
                  }
               }
            }
      }
   ];

   return events.aggregate(pipeline);
};
```

If all is good, you should see your new Parquet document in your S3 bucket. I've enabled the AWS GUI to show you the versions so that you can see how it is being updated every 60 seconds automatically.

![Screenshot from AWS S3 management console showing the new events.parquet document that was generated by our $out trigger function.](https://www.mongodb.com/developer/images/how-to/automated-continuous-data-copying-from-mongodb-to-s3/2694de087922972709633334785ab9f5cce3f333.png)

## Production Considerations

Some of the configurations chosen above were done so to make it easy to set up and test, but if you’re going to use this in production, you’ll want to adjust them.

Firstly, this blog was setup with a “deltas” approach. This means that we are only copying the new documents from our collection into our Parquet files. Another approach would be to do a full snapshot, i.e., copying the entire collection into Parquet each time. The approach you’re taking should depend on how much data is in your collection and what’s required by the downstream consumer.

Secondly, regardless of how much data you’re copying, ideally you want Parquet files to be larger, and for them to be partitioned based on how you’re going to query. [Apache recommends](https://parquet.apache.org/documentation/latest/) row group sizes of 512MB to 1GB. You can go smaller depending on your requirements, but as you can see, you want larger files. The other consideration is if you plan to query this data in the parquet format, you should partition it so that it aligns with your query pattern. If you’re going to query on a date field, for instance, you might want each file to have a single day's worth of data.

Lastly, depending on your needs, it may be appropriate to look into an alternative scheduling device to triggers, like [Temporal](https://docs.temporal.io/) or [Apache Airflow](https://airflow.apache.org/docs/apache-airflow/stable/index.html).

## Wrap Up

In this post, we walked through how to set up an automated continuous replication from a MongoDB database into an [AWS S3](https://aws.amazon.com/s3/) bucket in the [Parquet](https://parquet.apache.org/) data format by using [MongoDB Atlas Data Lake](https://docs.mongodb.com/datalake/) and [MongoDB Atlas Database Triggers](https://docs.atlas.mongodb.com/triggers/). First, we set up a new MongoDB Atlas Data Lake to consolidate a MongoDB database and our AWS S3 bucket. Then, we set up a Trigger to automatically add a new document to a collection every minute, and another Trigger to automatically back up these new automatically generated documents into our S3 bucket.

We also discussed how Parquet is a great format for your MongoDB data when you need to use columnar-oriented tools like Tableau for visualizations or Machine Learning frameworks that use Data Frames. Parquet can be quickly and easily converted into Pandas Data Frames in Python.

>If you have questions, please head to our [developer community website](https://community.mongodb.com/) where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.

Additional Resources:

-   [Data Lake: Getting Started Documentation](https://docs.mongodb.com/datalake/tutorial/getting-started/)
-   [$out S3 Data Lake Documentation](https://docs.mongodb.com/datalake/reference/pipeline/out/#std-label-adl-out-stage)


Learn how to set up a continuous copy from MongoDB into an AWS S3 bucket in Parquet.

How to Automate Continuous Data Copying from MongoDB to S3

Aaron Bassett

Using MongoDB Realm WebHooks with Amazon Kinesis Data Firehose

Maxime Beugnet

>As of June 2022, the functionality previously known as Atlas Data Lake is now named Atlas Data Federation. Atlas Data Federation’s functionality is unchanged and you can learn more about it [here](http://mongodb.com/atlas/data-federation). Atlas Data Lake will remain in the Atlas Platform, with newly introduced functionality that you can learn about [here](http://mongodb.com/atlas/data-lake).

Last year at MongoDB World 2019, Eliot announced that [MongoDB Atlas Data Lake](https://www.mongodb.com/atlas/data-lake) was a new tool available in beta in the [MongoDB Cloud Platform](https://www.mongodb.com/cloud).

During this last year, MongoDB has been working closely with many customers to test this new tool and gathered much feedback to make it even better.

Today, after a year of refinement and improvement, MongoDB is proud to announce that **MongoDB Atlas Data Lake** is now generally available and can be used with confidence in your production environment.

In this tutorial, I will show you a new feature of MongoDB Atlas Data Lake called [Federated Query](https://docs.mongodb.com/datalake/reference/config-files/data-lake-configuration) that allows you to access your archived documents in S3 **AND** your documents in your MongoDB Atlas cluster with a **SINGLE** MQL query.

!["MongoDB Atlas Data Lake Federated Queries"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake/federated-queries.png))

This feature is really amazing because it allows you to have easy access to your archived data in S3 along with your "hot" data in your Atlas cluster. This feature could help you prevent your Atlas clusters from growing in size indefinitely and reduce your costs drastically. It also makes it easier to gain new insights by easily querying data residing in S3 and exposing it to your real-time app.

Finally, I will show you how to use the new version of the [$out aggregation pipeline stage](https://docs.mongodb.com/datalake/reference/pipeline/out) to write documents from a MongoDB Atlas cluster into an AWS S3 bucket.

## Prerequisistes

In order to follow along this tutorial, you need to:

-   create at least an [M10 cluster](https://docs.atlas.mongodb.com/cluster-tier/) in MongoDB Atlas,
-   create a user in the **Database Access** menu,
-   add your IP address in the Network Access List in the **Network Access** menu,
-   [setup your MongoDB Atlas Data Lake environment in MongoDB Atlas](/how-to/atlas-data-lake-setup),
-   have python 3 with `pymongo` and `dnspython` libs installed, -   you will also need to have a [Mongo Shell](https://docs.mongodb.com/manual/mongo/) installed or maybe the new [MongoDB Shell](https://www.mongodb.com/blog/post/introducing-the-new-shell).

If you did these actions correctly, you should have an M10 (or bigger) cluster running in your MongoDB Atlas project.

!["MongoDB Atlas M10 cluster"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake/M10-cluster.png)

And your Data Lake page should look like this:

!["MongoDB Atlas Data Lake setup"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake/data-lake-setup.png)

You should also have an AWS S3 bucket linked to your Atlas Data Lake setup. Mine is called `cold-data-mongodb` in this tutorial.

>
>
>A MongoDB Atlas [M10 or bigger cluster](https://docs.atlas.mongodb.com/cluster-tier/) is required here because MongoDB Atlas Data Lake uses X.509 certificates which are not supported on MongoDB Atlas shared tier at this time.
>
>

## We need some data

To illustrate how `$out` and federated queries work, I will use an overly simple dataset to keep things as easy as possible to understand.

Connect to your M10 cluster using the `CONNECT` button using the MongoDB Shell and let's insert these 4 documents in the `test` database:

``` javascript
db.orders.insertMany(
    [
        {
            "_id": 1,
            "created": new Date("2020-05-30"),
            "items": [1, 3],
            "price": 20
        },
        {
            "_id": 2,
            "created": new Date("2020-05-31"),
            "items": [2, 3],
            "price": 25
        },
        {
            "_id": 3,
            "created": new Date("2020-06-01"),
            "items": [1, 3],
            "price": 20
        },
        {
            "_id": 4,
            "created": new Date("2020-06-02"),
            "items": [1, 2],
            "price": 15
        },
    ]
)
```

This is the result you should see in your terminal:

!["4 documents inserted in MongoDB Atlas M10 cluster"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake/insert-docs.png)

## Archive Data to S3 with $out

Now that we have a "massive" collection of orders, we can consider archiving the oldest orders to an S3 bucket. Let's imagine that once a month is over, I can archive all the orders from the previous month. I will create one JSON file in S3 for all the orders created during the previous month.

Let's transfer these orders to S3 using the [aggregation pipeline stage $out](https://docs.mongodb.com/datalake/reference/pipeline/out).

But first, we need to configure Atlas Data Lake correctly.

### Data Lake Configuration

The first thing we need to do is to make sure we can write to our S3 bucket and read the archived orders as well as the current orders in my M10 cluster.

This new feature in MongoDB Atlas Data Lake is called [Federated Queries](https://docs.mongodb.com/datalake/reference/config-files/data-lake-configuration).

Now head to your Data Lake configuration:

!["MongoDB Atlas Data Lake configuration"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake/data-lake-configuration.png)

And let's use the following configuration:

``` json
{
  "databases": [
    {
      "name": "test",
      "collections": [
        {
          "name": "orders",
          "dataSources": [
            {
              "path": "/{min(created) isodate}-{max(created) isodate}*",
              "storeName": "cold-data-mongodb"
            },
            {
              "collection": "orders",
              "database": "test",
              "storeName": "BigCluster"
            }
          ]
        }
      ],
      "views": []
    }
  ],
  "stores": [
    {
      "provider": "s3",
      "bucket": "cold-data-mongodb",
      "delimiter": "/",
      "includeTags": false,
      "name": "cold-data-mongodb",
      "region": "eu-west-1"
    },
    {
      "provider": "atlas",
      "clusterName": "BigCluster",
      "name": "BigCluster",
      "projectId": "5e78e83fc61ce37535921257"
    }
  ]
}
```

In this configuration, you can see that we have configured:

-   an S3 `store`: this is my S3 bucket that I named "cold-data-mongodb",
-   an Atlas `store`: this is my M10 cluster,
-   a database `test` with a collection `orders` that contains the data from this S3 store **AND** the data from my collection `test.orders` from my M10 cluster.

>
>
>Feel free to replace `cold-data-mongodb` with your own bucket name. The very same one that you used during the Atlas Data Lake setup.
>
>

You can find your MongoDB Atlas project ID in your project settings:

!["MongoDB Atlas Project ID"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake/atlas-project-id.png)

In the `path`, I also told Atlas Data Lake that the JSON filename contains the min and max `created` dates of the orders it contains. This is useful for performance purposes: Atlas Data Lake won't have to scan all the files if I'm looking for an order on a given date. You can read more about [data partitioning](https://docs.mongodb.com/datalake/reference/examples/path-syntax-examples#create-partitions-from-file-path) in the Data Lake documentation.

### $out to S3

Let's now collect the URI we are going to use to connect to Atlas Data Lake.

Click on the connect button:

!["MongoDB Atlas Data Lake connect button"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake/data-lake-connect-button.png)

Click on "Connect your application" and collect your URI:

!["MongoDB Atlas Data Lake URI"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake/data-lake-uri.png)

Now let's use Python to execute our aggregation pipeline and archive the 2 orders from May 2020 in our S3 bucket.

``` python
from datetime import datetime

from pymongo import MongoClient

client = MongoClient('<YOUR_ATLAS_DATA_LAKE_URI>')
db = client.get_database("test")
coll = db.get_collection("orders")

start_date = datetime(2020, 5, 1, 0, 0, 0)  # May 1st
end_date = datetime(2020, 6, 1, 0, 0, 0)  # June 1st

pipeline = [
    {
        '$match': {
            'created': {
                '$gte': start_date,
                '$lt': end_date
            }
        }
    },
    {
        '$out': {
            's3': {
                'bucket': 'cold-data-mongodb',
                'region': 'eu-west-1',
                'filename': start_date.isoformat('T', 'milliseconds') + 'Z-' + end_date.isoformat('T', 'milliseconds') + 'Z',
                'format': {'name': 'json', 'maxFileSize': "200MiB"}
            }
        }
    }
]

coll.aggregate(pipeline)
print('Archive created!')
```

To execute this code, make sure you have Python 3 and the dependencies:

``` none
$ python3 --version
Python 3.8.3
$ pip3 install pymongo dnspython
Requirement already satisfied: pymongo in /home/polux/.local/lib/python3.8/site-packages (3.10.1)
Requirement already satisfied: dnspython in /home/polux/.local/lib/python3.8/site-packages (1.16.0)
$ python3 archive.py
Archive created!
```

And now we can confirm that our archive was created correctly in our S3 bucket:

!["file in the S3 bucket"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake/s3-bucket.png)

### Finish the Work

Now that our orders are safe in S3, I can delete these 2 orders from my Atlas cluster. Let's use Python again but this time, we need to use the URI from our Atlas cluster. The Atlas Data Lake URI doesn't allow this kind of operation.

``` python
from datetime import datetime

from pymongo import MongoClient

client = MongoClient('<YOUR-ATLAS-URI>')
db = client.get_database("test")
coll = db.get_collection("orders")

start_date = datetime(2020, 5, 1, 0, 0, 0)  # May 1st
end_date = datetime(2020, 6, 1, 0, 0, 0)  # June 1st
query = {
    'created': {
        '$gte': start_date,
        '$lt': end_date
    }
}

result = coll.delete_many(query)
print('Deleted', result.deleted_count, 'orders.')
```

Let's run this code:

``` none
$ python3 remove.py
Deleted 2 orders.
```

Now let's double check what we have in S3. Here is the content of the S3 file I downloaded:

``` json
{"_id":{"$numberDouble":"1.0"},"created":{"$date":{"$numberLong":"1590796800000"}},"items":[{"$numberDouble":"1.0"},{"$numberDouble":"3.0"}],"price":{"$numberDouble":"20.0"}}
{"_id":{"$numberDouble":"2.0"},"created":{"$date":{"$numberLong":"1590883200000"}},"items":[{"$numberDouble":"2.0"},{"$numberDouble":"3.0"}],"price":{"$numberDouble":"25.0"}}
```

And here is what's left in my MongoDB Atlas cluster.

!["Documents left in MongoDB Atlas cluster"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake/docs-atlas.png)

### Federated Queries

As mentioned above already, federated queries in MongoDB Atlas Data Lake allow me to retain easy access to 100% of my data. I actually already used this feature when I ran the aggregation pipeline with the `$out` stage.

Let's verify this one last time with Python:

``` python
from pymongo import MongoClient

client = MongoClient('<YOUR_ATLAS_DATA_LAKE_URI>')
db = client.get_database("test")
coll = db.get_collection("orders")

print("All the docs from S3 + Atlas:")
docs = coll.find()
for d in docs:
    print(d)

pipeline = [
    {
        '$group': {
            '_id': None,
            'total_price': {
                '$sum': '$price'
            }
        }
    }, {
        '$project': {
            '_id': 0
        }
    }
]

print('\nI can also run an aggregation.')
print(coll.aggregate(pipeline).next())
```

Here is the output:

``` none
All the docs from S3 + Atlas:
{'_id': 1.0, 'created': datetime.datetime(2020, 5, 30, 0, 0), 'items': [1.0, 3.0], 'price': 20.0}
{'_id': 2.0, 'created': datetime.datetime(2020, 5, 31, 0, 0), 'items': [2.0, 3.0], 'price': 25.0}
{'_id': 3.0, 'created': datetime.datetime(2020, 6, 1, 0, 0), 'items': [1.0, 3.0], 'price': 20.0}
{'_id': 4.0, 'created': datetime.datetime(2020, 6, 2, 0, 0), 'items': [1.0, 2.0], 'price': 15.0}

I can also run an aggregation:
{'total_price': 80.0}
```

## Wrap Up

MongoDB Atlas Data Lake is now production-ready and generally available starting today.

If you have a lot of infrequently access data in your Atlas cluster but you still need to be able to query it and access it easily once you've archived it to S3, Atlas Data Lake and the new Federated Query feature will help you save tons of money. If you're looking for an automated way to archive your data from Atlas Clusters to fully-managed S3 storage, then check out our new [Atlas Online Archive feature](https://docs.atlas.mongodb.com/online-archive/manage-online-archive/)!

Storage on S3 is a lot cheaper than scaling up your MongoDB Atlas cluster because your cluster is full of cold data and needs more RAM & storage size to operate correctly.

All the python code is available in this [Github Repository](https://github.com/mongodb-developer/data-lake-tutorial).

Please let me know on Twitter if you liked my blog post: [@MBeugnet](https://twitter.com/MBeugnet).

If you have questions, please head to our [developer community website](https://community.mongodb.com/) where the MongoDB engineers and the MongoDB community will give you a hand.


Learn how to use MongoDB Atlas Data Lake federated queries and $out to S3.

MongoDB Atlas Data Lake Tutorial: Federated Queries and $out to AWS S3

Ken W. Alger

>As of June 2022, the functionality previously known as Atlas Data Lake is now named Atlas Data Federation. Atlas Data Federation’s functionality is unchanged and you can learn more about it [here](http://mongodb.com/atlas/data-federation). Atlas Data Lake will remain in the Atlas Platform, with newly introduced functionality that you can learn about [here](http://mongodb.com/atlas/data-lake).

At MongoDB World 2019, we announced the Beta version of MongoDB Atlas Data Lake which allows for the querying of data stored in Amazon S3 buckets. We're able to query uncompressed or compressed data stored in JSON, BSON, CSV, TSV, Avro, and Parquet formats right in S3. This is a great feature of MongoDB Atlas to enable the exploration of data stored on S3 without having to import it into a database.

Before diving into the Atlas Data Lake, however, there are some setup steps that need to be done. To start, you'll need an Amazon Web Services (AWS) account, access to the AWS Management Console, and [Command Line Interface (CLI) tools](http://bit.ly/cli-aws). Let's take a look.

## Setting Up the AWS CLI

The first thing we'll need to create is an Identity and Access Management (IAM) user. From the AWS Management Console, I'll search for IAM.

!["AWS Management Console Search"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake-setup/AWS_Management_Console.png)

On the left navigation, I'll select `Users`, and `Add User`. I'll make a new user, I'll just call it `DataLakeUser`, and assign Programmatic access.

!["Add User Step"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake-setup/AddUser.png)

Then I'll click on `Next: Permissions` and assign, or I can create, a group that has Administrator Access privileges.

!["Set Permissions"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake-setup/setPermissions.png)

I'll click on `Next: Tags` and since I'm not assigning any tags at this point, I'll move onto `Next: Review`. And everything looks good, so I'll create the user.

!["Access and Secret Key Information"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake-setup/keyInformation.png)

I'll need the `Access key ID` and `Secret access key` information to configure my command-line interface. I'll download the .csv file and store it locally. Note that once you leave this screen you won't have access to this information again and will have to go through the credential creation process to get a new set of keys. That's why downloading that information at this point is important.

Next, in a terminal window I'll run:

``` bash
aws configure
```

Here, I'll be prompted for four pieces of information to configure my AWS [Command Line Interface (CLI) tools](http://bit.ly/cli-aws), the Access Key Id and Secret Access Key we just saved in the last step, my [AWS region](https://docs.aws.amazon.com/sns/latest/dg/sns-supported-regions-countries.html) that I want to use, and the output format.

``` bash
AWS Access Key ID [****************UHHP]: AKI********HS
AWS Secret Access Key [****************EcNP]: uJ*******Lpk
Default region name [us-west-2]: 
Default output format [json]:
```

Great, the AWS CLI is now configured for my AWS account. The next step is organizing the S3 bucket directories.

## AWS S3 Bucket Structure

While I could go through and create the bucket structure using the [cli commands](http://bit.ly/cli-commands), I find it to be faster and easier with the S3 console. I'll navigate to the S3 service and locate the bucket where my data is stored. For this example, I have several datasets inside an `atlas-data-lake-demo` bucket.

>
>
>Download the sample data and bucket structure [here](https://developer-advocacy-public.s3-eu-west-1.amazonaws.com/blog/sampledata.zip).
>
>

Inside that bucket, I have several folders, each with a different data type:

!["S3 Buckets"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake-setup/S3_Buckets.png)

Let's take a look inside the CSV folder:

!["CSV Folder Contents"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake-setup/CSV_Folder_Contents.png)

Inside each of these folders, I have the actual CSV data files.  Similarly, in the other data file format folders (avro, json, parquet, and tsv), I have datasets of those respective data types. These folder names, `comics` and `seattleLibrary`, in this CSV example, will become collections inside Altas Data Lake. This structure is what's outlined in our Data Lake setup walkthrough. It *can* be configured differently, but the structure here works pretty well.

With the CLI configured and the S3 directory structure in place, we're ready to set up the MongoDB Atlas Data Lake.

## Connecting MongoDB Atlas to S3

Now it's time to get our feet wet with setting up the connection between Atlas Data Lake and our S3 bucket. Inside the MongoDB Atlas console, on the left side, I'll click on `Data Lake` which takes me to the MongoDB Atlas Data Lake dashboard.

!["Try MongoDB Atlas Data Lake"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake-setup/DL1.png)

From here I'll select `Try MongoDB Atlas Data Lake` which will start the step-by-step setup of the connection, but I'll point out a few things along the way as well. After the opening splash screen, we move onto naming our Data Lake. Once you pick a name here, you can't change it. For now, I'll call this LakeDemo.

!["Name Your Data Lake"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake-setup/DL_Naming.png)

Next up is the S3 bucket or buckets, this Data Lake will attach to. The S3 bucket I'm using is called `atlas-data-lake-demo`... not very creative I know... but, I'll put that into this text box. If you are connecting to more than a single S3 bucket, you can add a comma-separated list in the text box as well.

!["S3 Buckets to Query"](https://www.mongodb.com/developer/images/how-to/atlas-data-lake-setup/S3_Buckets_To_Query.png)

### Role & Policy Task

The next step in the configuration process is the *Role & Policy* task.  This is where the AWS CLI configuration we did previously pays off.  Something to note here: As the warning box states, the information for the External ID and ARN aren't shown inside the MongoDB Atlas interface in this format again, so we'll want to save this information. I like to copy this information and put it into a `config.txt` file that I can find later. The information is also available in the JSON file we'll create in a moment, but it's good to save it separately.

The next step is to enter a new IAM role for MongoDB Atlas to use. I'll go with the default suggestion here of `mdb-dl-role`.

There is a generated trust policy that we'll copy the text from and save into a file named `dl-role-trust.json`. This allows MongoDB Atlas to be trusted in AWS S3.

Using that JSON file, we can create an AWS IAM role from the command line... see, I said our previous work would pay off. We copy the command, go to the terminal in the same directory as the `dl-role-trust.json` file we just saved, and run the command.

``` bash
aws iam create-role \
  --role-name mdb-dl-role \
  --assume-role-policy-document file://dl-role-trust.json
```

We should get returned a JSON document that confirms our role.

>
>
>There's some information we'll need out of this response in a later step, so don't clear your screen!
>
>

You can also find the information from the AWS Console, but since it was just provided, we can use it from here.

One more bit of JSON to copy and paste into a file. This time the file is called `dl-s3-role-policy.json`, and it defines the IAM role that we just created. Once that's saved, we can apply that policy to the role with the command here in the Roles & Policy step 5.

``` bash
aws iam put-role-policy \
  --role-name mdb-dl-role \
  --policy-name mdb-dl-role-policy \
  --policy-document file://dl-s3-role-policy.json
```

We run this command in the terminal, again in the same folder that the `dl-s3-role-policy.json` file resides.

The next step in the process is to validate the connection. Remember that piece of information I said we'd need? Well, now's the time. We want the get the Amazon Resource Name (ARN) string from the response we got when we created the IAM role.

If we scroll up a bit in the terminal window we can find it. Note that we want the ARN under the `Role` information, **not** the ARN under the `Principal` information. We paste that into the text box and we don't need to touch the bucket name box. We next click on `Validate & Launch` which makes sure that all of our configurations are in place and correct.

Congratulations! You've successfully created an Atlas Data Lake.

## Swimming in the Data Lake

Let's connect to the MongoDB Atlas Data Lake using the Mongo Shell. From the Data Lake console, I'll click on Step 2 and `connect`. Then Connect with the Mongo Shell. I already have the Mongo Shell installed, so I'll choose that option. If you don't have it already, you can download it for your OS platform.

I'll copy the connection string and paste it into the command line.

``` bash
mongo "<<CONNECTION STRING>>" --ssl --authenticationDatabase 
      admin --username <username> --password <password>
```

Note that I need to replace the connection string, username, and password portions of this with the appropriate information for database access. This user **must** have Atlas Admin privileges to be able to configure the database. You can configure the users and their permissions from the **Database Access** tab on the left.

After I run that command from the terminal, I should be inside the Mongo Shell. The Atlas Data Lake console will take me to the next step: database configuration. By clicking on Step 3 here, I'm taken to the documentation for a `storageSetConfig` command that I need to run to enable Atlas Data Lake to pull in the data from S3.

I need to update the information in the sample code to match my S3 bucket region and bucket name. Recall from an earlier step when I set up the directory structure in S3 I mentioned that those directories would come into Atlas Data Lake as collections. This command is where that is defined. We can define the databases that are generated in MongoDB Atlas as well as the collections, and what that structure looks like.  Additionally, to improve performance you can structure your data on S3 in a way that aligns with common queries similar to indexing strategies and make the accompanying optimizations to your Storage Config

For example, if our data is stored in S3 in yearly folders and inside each of those folders is data of different file types, e.g. JSON, CSV, TSV, etc. MongoDB Atlas Data Lake will intelligently read the S3 partitions and make the partition names park of the documents. This allows for the efficient querying of data by only opening the needed partitions. Meaning that data in a 2018 folder will be pulled into the query logic and searches for 2018 data will only open 2018 folders.

For now, I'll stick with a single database called `sample`, but I'll bring in different datasets from the different file types. I just need to copy these lines here:

``` bash
{                      
   /* Works with the definition to create a
   collection the data in each directory */
   "store": "s3store", // Links to the S3 store above
   "definition": "/json/{collectionName()}"
}
```

Paste them in and update json to the different file types.

``` bash
use admin;
db.runCommand( { "storageSetConfig": {
    "stores": [{
      "s3": {
        "name": "s3store", // Creates an S3 store

        "region":"us-west-2",  // Update with the bucket region code

        "bucket": "atlas-data-lake-demo" // Update with your bucket name

      }
    }],
    "databases": {
      "sample": {  // Creates a database named sample
        "*": [
    {                      
        /* Works with the definition to create a
        collection the data in each directory */
        "store": "s3store",  // Links to the S3 store above
        "definition": "/avro/{collectionName()}"
      },
    {
        /* Works with the definition to create a
        collection the data in each directory */
        "store": "s3store",  // Links to the S3 store above
        "definition": "/csv/{collectionName()}"
      },
      {
        /* Works with the definition to create a
        collection the data in each directory */
        "store": "s3store",  // Links to the S3 store above
        "definition": "/json/{collectionName()}"
      },
      {
        /* Works with the definition to create a
        collection the data in each directory */
        "store": "s3store",  // Links to the S3 store above
        "definition": "/parquet/{collectionName()}"
      },
      {
        /* Works with the definition to create a
        collection the data in each directory */
        "store": "s3store",  // Links to the S3 store above
        "definition": "/tsv/{collectionName()}"
      }]
    }
  }
}})
```

Then I run the command from inside the Mongo Shell. I should then get back a `{ "ok" : 1 }` response indicating that all is well.

From here, depending on the size of the datasets, it can take a bit for Atlas Data Lake to generate everything needed. You'll know it's ready to go when you can run a

``` bash
show dbs
```

command and you are given a list of databases. In the case of this example, I'll get back a single database named sample.

Then with a

``` bash
use sample
```

command and a

``` bash
show collections
```

I should get a list of the collections.

The Data Lake console will have three green boxes with checkmarks at this point, indicating that the database has been configured. The last step is to run a query.

Back in the Mongo Shell, I can query any of those collections using MQL, the MongoDB Query Language. I'll run a simple findOne() on the comics collection. Behind the scenes here, this operation should have fairly reasonable performance as this will only open the associated S3 partition that holds this data.

``` bash
db.comics.findOne()

{
  "align" : "Good",
  "alive" : "Living Characters",
  "appearances" : "4043",
  "eye" : "Hazel Eyes",
  "first_appear" : "Aug-1962",
  "gender" : "Male",
  "gsm" : "NA",
  "hair" : "Brown Hair",
  "id" : "Secret",
  "name" : "Spider-Man (Peter Parker)",
  "publisher" : "marvel"
}
```

After running that query our Data Lake setup dashboard should be cleared and an actual dashboard will show up.

## Wrap Up

Excellent work! We've come a long way in a few relatively short steps.  You now have the power to explore your data stored in S3 using MQL.  Don't stop with just the Mongo Shell though. You can also [explore data using MongoDB Compass](https://alger.me/compass-data-lake) and programmatically using MongoDB Drivers for your favorite language.

### See MongoDB Atlas Data Lake In Action


:youtube[]{vid=2AGRICOvg2Q}


Enjoy this great new feature of [MongoDB Atlas](http://bit.ly/mongodb-meetatlas). Let us know how you're putting it to use, we're always excited to hear how folks are using our products and features.

Until next time, Happy Coding!


Learn how to setup your MongoDB Atlas Data Lake connection to explore data stored in AWS S3 buckets.

MongoDB Atlas Data Lake Setup

Jason Mimick



<div class="introduction">

It's pretty amazing that we can now deploy and control massive systems
in the cloud from our laptops and phones. And it's so easy to take for
granted when it all works, but not so awesome when everything is broken
after coming back on Monday morning after a long weekend! On top of
that, the tooling that's available is constantly changing and updating
and soon you are drowning in dependabot PRs.

The reality of setting up and configuring all the tools necessary to
deploy an app is time-consuming, error-prone, and can result in security
risks if you're not careful. These are just a few of the reasons we've
all witnessed the incredible growth of DevOps tooling as we continue the
evolution to and in the cloud.

[AWS CloudFormation](https://aws.amazon.com/cloudformation/) is an
infrastructure-as-code (IaC) service that helps you model and set up
your Amazon Web Services resources so that you can spend less time
managing those resources and more time focusing on your applications
that run in AWS. CloudFormation, or CFN, let's users create and manage
AWS resources directly from templates which provide dependable out of
the box blueprint deployments for any kind of cloud app.

To better serve customers using modern cloud-native workflows, MongoDB
Atlas supports native CFN templates with a new set of Resource Types.
This new integration allows you to manage complete MongoDB Atlas
deployments through the AWS CloudFormation console and CLI so your apps
can securely consume data services with full AWS cloud ecosystem
support.

</div>

<div class="content">

## Launch a MongoDB Atlas Stack on AWS CloudFormation

We created a helper project that walks you through an end-to-end example
of setting up and launching a MongoDB Atlas stack in AWS CloudFormation.

The
[get-started-aws-cfn](https://github.com/mongodb-developer/get-started-aws-cfn)
project builds out a complete MongoDB Atlas deployment, which includes a
MongoDB Atlas project, cluster, AWS IAM role-type database user, and IP
access list entry.

>
>
>You can also use the [AWS Quick Start for MongoDB
>Atlas](https://aws.amazon.com/quickstart/architecture/mongodb-atlas/)
>that uses the same resources for CloudFormation and includes network
>peering to a new or existing VPC.
>
>

<img src="https://www.mongodb.com/developer/images/aws-cloudformation/get-started-aws-cfn-demo.gif" class="align-center" alt="Deploy MongoDB Atlas using AWS CloudFormation" />

You're most likely already set up to run the
[get-started-aws-cfn](https://github.com/mongodb-developer/get-started-aws-cfn)
since the project uses common tools like the AWS CLI and Docker, but
just in case, head over to the
[prerequisites](https://github.com/mongodb-developer/get-started-aws-cfn#pre-requisites)
section to check your development machine. (If you haven't already,
you'll want to [create a MongoDB Atlas
account](https://www.mongodb.com/cloud/atlas/register).)

The project has two main parts: "get-setup' will deploy and configure
the MongoDB Atlas CloudFormation resources into the AWS region of your
choice, while "get-started' will launch your complete Atlas deployment.

## Step 1) Get Set Up

Clone the
[get-started-aws-cfn](https://github.com/mongodb-developer/get-started-aws-cfn)
repo and [get
setup](https://github.com/mongodb-developer/get-started-aws-cfn#get-setupsh):

``` bash
git clone https://github.com/mongodb-developer/get-started-aws-cfn
cd get-started-aws-cfn
./get-setup.sh
```

## Step 2) Get Started

Run the get-started script:

``` bash
./get-started.sh
```

<img src="https://www.mongodb.com/developer/images/aws-cloudformation/get-started-aws-cfn-pic-1.png" class="align-center" alt="Launch a MongoDB Atlas deployment with the AWS CLI using CloudFormation with the Get Started project" />

Once the stack is launched, you will start to see resources getting
created in the AWS CloudFormation console. The Atlas cluster takes a few
minutes to spin up completely, and you can track the progress in the
console or through the AWS CLI.

<img src="https://www.mongodb.com/developer/images/aws-cloudformation/get-started-aws-cfn-pic-2.png" class="align-center" alt="View and manage MongoDB Atlas resources from the AWS CloudFormation console." />

## Step 3) Get Connected

Once your MongoDB Atlas cluster is deployed successfully, you can find
its connection string under the Outputs tab as the value for the
"ClusterSrvAddress' key.

<img src="https://www.mongodb.com/developer/images/aws-cloudformation/get-started-aws-cfn-pic-3.png" class="align-center" alt="Use the ClusterSrvAddress output in AWS CloudFormation to connect to your MongoDB Atlas cluster." />

The Get-Started project also has a helper script to combine the AWS and
MongoDB shells to securely connect via an AWS IAM role session. Check
out [connecting to your
cluster](https://github.com/mongodb-developer/get-started-aws-cfn#connecting-to-your-cluster)
for more information.

What next? You can connect to your MongoDB Atlas cluster from [the mongo
shell](https://docs.atlas.mongodb.com/mongo-shell-connection/), [MongoDB
Compass](https://docs.atlas.mongodb.com/compass-connection/), or any of
our supported
[drivers](https://docs.atlas.mongodb.com/driver-connection/). We have
guides for those using Atlas with popular languages: Here's one for [how
to connect to Atlas with
Node.js](https://developer.mongodb.com/quickstart/node-connect-mongodb-3-3-2/)
and another for [getting started with
Java](https://developer.mongodb.com/quickstart/java-setup-crud-operations/).

</div>

<div class="summary">

## Conclusion

Use the MongoDB Atlas CloudFormation Resources to power everything from
the most basic "hello-world' apps to the most advanced devops pipelines.
Jump start your new projects today with the [MongoDB Atlas AWS
CloudFormation Get-Started
project](https://github.com/mongodb-developer/get-started-aws-cfn)!

>
>
>If you have questions, please head to our [developer community
>website](https://community.mongodb.com/) where the MongoDB engineers and
>the MongoDB community will help you build your next big idea with
>MongoDB.
>
>

</div>


Learn how to get started with MongoDB Atlas and AWS CloudFormation.

Get Started with MongoDB Atlas and AWS CloudFormation

Nic Raboy



<div class="introduction">

The popularity of [Amazon Alexa](https://developer.amazon.com/en-US/alexa) and virtual assistants in general is no question, huge. Having a web application and mobile application isn't enough for most organizations anymore, and now you need to start supporting voice operated applications.

So what does it take to create something for Alexa? How different is it from creating a web application?

In this tutorial we're going to see how to create an Amazon Alexa Skill, also referred to as an Alexa application, that interacts with a MongoDB cluster using the Go programming language (Golang) and AWS Lambda.

![](https://www.mongodb.com/developer/images/how-to/demo-s4cgc58hq1.gif
"")

## The Requirements

A few requirements must be met prior to starting this tutorial:

-   Golang must be installed and configured
-   A MongoDB Atlas cluster

If you don't have a [MongoDB Atlas](https://www.mongodb.com/cloud) cluster, you can configure one for free. For this example an M0 cluster is more than sufficient.

Make sure the Atlas cluster has the proper IP addresses on the Network Access List for AWS services. If AWS Lambda cannot reach your cluster then requests made by Alexa will fail.

Having an Amazon Echo or other Amazon Alexa enabled device is not necessary to be successful with this tutorial. Amazon offers a really great simulator that can be used directly in the web browser.

## Designing an Alexa Skill with an Invocation Term and Sample Utterances

When it comes to building an Alexa Skill, it doesn't matter if you start with the code or the design. For this tutorial we're going to start with the design, directly in the [Amazon Developer Portal](https://developer.amazon.com) for Alexa.

Sign into the portal and choose to create a new custom Skill. After creating the Skill, you'll be brought to a dashboard with several checklist items:

![](https://www.mongodb.com/developer/images/how-to/amazon-alexa-portal-j4mv42a9lt.png
"")

In the checklist, you should take note of the following:

-   Invocation Name
-   Intents, Samples, and Slots
-   Endpoint

There are other items, one being optional and the other being checked naturally as the others complete.

The first step is to define the invocation name. This is the name that users will use when they speak to their virtual assistant. It should not be confused with the Skill name because the two do not need to match. The Skill name is what would appear in the online marketplace.

For our invocation name, let's use **recipe manager**, something that is easy to remember and easy to pronounce. With the invocation name in place, we can anticipate using our Skill like the following:

``` none
Alexa, ask Recipe Manager to INTENT
```

The user would not literally speak **INTENT** in the command. The intent
is the command that will be defined through sample utterances, also
known as sample phrases or data. You can, and probably should, have
multiple intents for your Skill.

Let's start by creating an intent titled **GetIngredientsForRecipeIntent** with the following sample utterances:

``` none
what ingredients do i need for {recipe}
what do i need to cook {recipe}
to cook {recipe} what ingredients do i need
```

There are a few things to note about the above phrases:

-   The `{recipe}` tag is a slot variable which is going to be user defined when spoken.
-   Every possible spoken phrase to execute the command should be listed.

Alexa operates from machine learning, so the more sample data the better. When defining the `{recipe}` variable, it should be assigned a type of `AMAZON.Food`.

When all said and done, you could execute the intent by doing something like:

``` none
Alexa, ask Recipe Manager what do I need to cook Chocolate Chip Cookies
```

Having one intent in your Alexa Skill is no fun, so let's create another intent with its own set of sample phrases. Choose to create a new intent titled `GetRecipeFromIngredientsIntent` with the following sample utterances:

``` none
what can i cook with {ingredientone} and {ingredienttwo}
what are some recipes with {ingredientone} and {ingredienttwo}
if i have {ingredientone} and {ingredienttwo} what can i cook
```

This time around we're using two slot variables instead of one. Like previously mentioned, it is probably a good idea to add significantly more sample utterances to get the best results. Alexa needs to be able to process the data to send to your Lambda function.

At this point in time, the configuration in the Alexa Developer Portal is about complete. The exception being the endpoint which doesn't exist yet.

## Building a Lambda Function with Golang and MongoDB

Alexa, for the most part should be able to direct requests, so now we need to create our backend to receive and process them. This is where Lambda, Go, and MongoDB come into play.

Assuming Golang has been properly installed and configured, create a new project within your **$GOPATH** and within that project, create a **main.go** file. As boilerplate to get the ball rolling, this file should contain the following:

``` go
package main

func main() { }
```

With the boilerplate code added, now we can install the MongoDB Go driver. To do this, you could in theory do a `go get`, but the preferred approach as of now is to use the [dep](https://github.com/golang/dep) package management tool for Golang. To do this, after having installed the tool, execute the following:

``` bash
dep init
dep ensure -add "go.mongodb.org/mongo-driver/mongo"
```

We're using `dep` so that way the version of the driver that we're using in our project is version locked.

In addition to the MongoDB Go driver, we're also going to need to get the [AWS Lambda SDK for Go](https://github.com/aws/aws-lambda-go) as well as an [unofficial SDK for Alexa](https://github.com/arienmalec/alexa-go), since no official SDK exists. To do this, we can execute:

``` bash
dep ensure -add "github.com/arienmalec/alexa-go"
dep ensure -add "github.com/aws/aws-lambda-go/lambda"
```

With the dependencies available to us, we can modify the project's **main.go** file. Open the file and add the following code:

``` go
package main

import (
    "context"
    "os"

    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

// Stores a handle to the collection being used by the Lambda function
type Connection struct {
    collection *mongo.Collection
}

func main() {
    ctx := context.Background()
    client, err := mongo.Connect(ctx, options.Client().ApplyURI(os.Getenv("ATLAS_URI")))
    if err != nil {
        panic(err)
    }

    defer client.Disconnect(ctx)

    connection := Connection{
        collection: client.Database("alexa").Collection("recipes"),
    }
}
```

In the `main` function we are creating a client using the connection string of our cluster. In this case, I'm using an environment variable on my computer that points to my MongoDB Atlas cluster. Feel free to configure that connection string however you feel the most confident.

Upon connecting, we are getting a handle of a `recipes` collection for an `alexa` database and storing it in a `Connection` data structure. Because we won't be writing any data in this example, both the `alexa` database and the `recipes` collection should exist prior to running this application.

You can check out more information about connecting to MongoDB with the Go programming language in a [previous tutorial](https://www.mongodb.com/blog/post/quick-start-golang--mongodb--starting-and-setup) I wrote.

So why are we storing the collection handle in a `Connection` data structure?

AWS Lambda behaves a little differently when it comes to web applications. Instead of running the `main` function and then remaining alive for as long as your server remains alive, Lambda functions tend to suspend or shutdown when they are not used. For this reason, we cannot rely on our connection being available and we also don't want to establish too many connections to our database in the scenario where our function hasn't shut down. To handle this, we can pass the connection from our `main` function to our logic function.

Let's make a change to see this in action:

``` go
package main

import (
    "context"
    "os"

    "github.com/arienmalec/alexa-go"
    "github.com/aws/aws-lambda-go/lambda"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

// Stores a handle to the collection being used by the Lambda function
type Connection struct {
    collection *mongo.Collection
}

func (connection Connection) IntentDispatcher(ctx context.Context, request alexa.Request) (alexa.Response, error) {
    // Alexa logic here...
}

func main() {
    ctx := context.Background()
    client, err := mongo.Connect(ctx, options.Client().ApplyURI(os.Getenv("ATLAS_URI")))
    if err != nil {
        panic(err)
    }

    defer client.Disconnect(ctx)

    connection := Connection{
        collection: client.Database("alexa").Collection("recipes"),
    }

    lambda.Start(connection.IntentDispatcher)
}
```

Notice in the above code that we've added a `lambda.Start` call in our `main` function that points to an `IntentDispatcher` function. We're designing this function to use the connection information established in the `main` function, which based on our Lambda knowledge, may not run every time the function is executed.

So we've got the foundation to our Alexa Skill in place. Now we need to design the logic for each of our intents that were previously defined in the Alexa Developer Portal.

Since this is going to be a recipe related Skill, let's model our MongoDB documents like the following:

``` json
{
    "_id": ObjectID("234232358943"),
    "name": "chocolate chip cookies",
    "ingredients": [
        "flour",
        "egg",
        "sugar",
        "chocolate"
    ]
}
```

There is no doubt that our documents could be more extravagant, but for this example it will work out fine. Within the MongoDB Atlas cluster, create the **alexa** database if it doesn't already exist and add a document modeled like the above in a **recipes** collection.

In the `main.go` file of the project, add the following data structure:

``` go
// A data structure representation of the collection schema
type Recipe struct {
    ID          primitive.ObjectID `bson:"_id"`
    Name        string             `bson:"name"`
    Ingredients []string           `bson:"ingredients"`
}
```

With the MongoDB Go driver, we can annotate Go data structures with BSON
so that way we can easily map between the two. It essentially makes our
lives a lot easier when working with MongoDB and Go.

Let's circle back to the `IntentDispatcher` function:

``` go
func (connection Connection) IntentDispatcher(ctx context.Context, request alexa.Request) (alexa.Response, error) {
    var response alexa.Response
    switch request.Body.Intent.Name {
    case "GetIngredientsForRecipeIntent":
    case "GetRecipeFromIngredientsIntent":
    default:
        response = alexa.NewSimpleResponse("Unknown Request", "The intent was unrecognized")
    }
    return response, nil
}
```

Remember the two intents from the Alexa Developer Portal? We need to assign logic to them.

Essentially, we're going to do some database logic and then use the `NewSimpleResponse` function to create a response the the results.

Let's start with the `GetIngredientsForRecipeIntent` logic:

``` go
case "GetIngredientsForRecipeIntent":
    var recipe Recipe
    recipeName := request.Body.Intent.Slots["recipe"].Value
    if recipeName == "" {
        return alexa.Response{}, errors.New("Recipe name is not present in the request")
    }
    if err := connection.collection.FindOne(ctx, bson.M{"name": recipeName}).Decode(&recipe); err != nil {
        return alexa.Response{}, err
    }
    response = alexa.NewSimpleResponse("Ingredients", strings.Join(recipe.Ingredients, ", "))
```

In the above snippet, we are getting the slot variable that was passed and are issuing a `FindOne` query against the collection. The filter for the query says that the `name` field of the document must match the recipe that was passed in as a slot variable.

If there was a match, we are serializing the array of ingredients into a string and are returning it back to Alexa. In theory, Alexa should then read back the comma separated list of ingredients.

Now let's take a look at the `GetRecipeFromIngredientsIntent` intent logic:

``` go
case "GetRecipeFromIngredientsIntent":
    var recipes []Recipe
    ingredient1 := request.Body.Intent.Slots["ingredientone"].Value
    ingredient2 := request.Body.Intent.Slots["ingredienttwo"].Value
    cursor, err := connection.collection.Find(ctx, bson.M{
        "ingredients": bson.D{
            {"$all", bson.A{ingredient1, ingredient2}},
        },
    })
    if err != nil {
        return alexa.Response{}, err
    }
    if err = cursor.All(ctx, &recipes); err != nil {
        return alexa.Response{}, err
    }
    var recipeList []string
    for _, recipe := range recipes {
        recipeList = append(recipeList, recipe.Name)
    }
    response = alexa.NewSimpleResponse("Recipes", strings.Join(recipeList, ", "))
```

In the above snippet, we are taking both slot variables that represent
ingredients and are using them in a `Find` query on the collection. This
time around we are using the `$all` operator because we want to filter
for all recipes that contain both ingredients anywhere in the array.

With the results of the `Find`, we can create create an array of the
recipe names and serialize it to a string to be returned as part of the
Alexa response.

If you'd like more information on the `Find` and `FindOne` commands for
Go and MongoDB, check out my [how to read documents
tutorial](https://www.mongodb.com/blog/post/quick-start-golang--mongodb--how-to-read-documents)
on the subject.

While it might seem simple, the code for the Alexa Skill is actually
complete. We've coded scenarios for each of the two intents that we've
set up in the Alexa Developer Portal. We could improve upon what we've
done or create more intents, but it is out of the scope of what we want
to accomplish.

Now that we have our application, we need to build it for Lambda.

Execute the following commands:

``` bash
GOOS=linux go build
zip handler.zip ./project-name
```

So what's happening in the above commands? First we are building a Linux compatible binary. We're doing this because if you're developing on Mac or Windows, you're going to end up with a binary that is incompatible. By defining the operating system, we're telling Go what to build for.

For more information on cross-compiling with Go, check out my [Cross Compiling Golang Applications For Use On A Raspberry Pi](https://www.thepolyglotdeveloper.com/2017/04/cross-compiling-golang-applications-raspberry-pi/) post.

Next, we are creating an archive of our binary. It is important to replace the `project-name` with that of your actual binary name. It is important to remember the name of the file as it is used in the Lambda dashboard.

![](https://www.mongodb.com/developer/images/how-to/amazon-lambda-portal-635pa2aya3.jpg
"")

When you choose to create a new Lambda function within AWS, make sure Go is the development technology. Choose to upload the ZIP file and add the name of the binary as the handler.

Now it comes down to linking Alexa with Lambda.

Take note of the **ARN** value of your Lambda function. This will be added in the Alexa Portal. Also, make sure you add the Alexa Skills Kit as a trigger to the function. It is as simple as selecting it from the list.

Navigate back to the [Alexa Developer Portal](https://developer.amazon.com/alexa/console/ask) and choose the **Endpoint** checklist item. Add the ARN value to the default region and choose to build the Skill using the **Build Model** button.

When the Skill is done building, you can test it using the simulator that Amazon offers as part of the Alexa Developer Portal. This simulator can be accessed using the **Test** tab within the portal.

If you've used the same sample utterances that I have, you can try entering something like this:

``` none
ask recipe manager what can i cook with flour and sugar
ask recipe manager what chocolate chip cookies requires
```

Of course the assumption is that you also have collection entries for chocolate chip cookies and the various ingredients that I used above. Feel free to modify the variable terms with those of your own data.

## Conclusion

You just saw how to build an Alexa Skill with [MongoDB](https://www.mongodb.com), Golang, and AWS Lambda. Knowing how to develop applications for voice assistants like Alexa is great because they are becoming increasingly popular, and the good news is that they aren't any more difficult than writing standard applications.

As previously mentioned, MongoDB Atlas makes pairing MongoDB with Lambda and Alexa very convenient. You can use the free tier or upgrade to something better.

If you'd like to expand your Alexa with Go knowledge and get more practice, check out a previous tutorial I wrote titled [Build an Alexa Skill with Golang and AWS Lambda](https://www.thepolyglotdeveloper.com/2018/10/build-alexa-skill-golang-aws-lambda/).

Learn how to develop Amazon Alexa Skills that interact with MongoDB using the Go programming language and AWS Lambda.

Developing Alexa Skills with MongoDB and Golang

Aydrian Howard

>Please note: This article discusses Stitch. Stitch is now MongoDB Realm. All the same features and functionality, now with a new name. Learn more [here](https://www.mongodb.com/realm). We will be updating this article in due course.

Computers can now look at a video or image and know what's going on and, sometimes, who's in it. Amazon Web Service Rekognition gives your applications the eyes it needs to label visual content. In the following, you can see how to use Rekognition along with MongoDB Stitch to supplement new content with information as it is inserted into the database.

You can easily detect labels or faces in images or videos in your MongoDB application using the built-in AWS service. Just [add the AWS service](https://docs.mongodb.com/realm/services/aws/) and use the Stitch client to execute the AWS SES request right from your React.js application or create a Stitch function and Trigger. In a recent Stitchcraft live coding session on [my Twitch channel](https://twitch.tv/itsaydrian), I wanted to tag an image using [label detection](https://docs.aws.amazon.com/rekognition/latest/dg/labels-detect-labels-image.html). I set up a trigger that executed a function after an image was uploaded to my S3 bucket and its metadata was inserted into a collection.

``` javascript
exports = function(changeEvent) {
  const aws = context.services.get('AWS');
  const mongodb = context.services.get("mongodb-atlas");
  const insertedPic = changeEvent.fullDocument;

  const args = {
    Image: {
      S3Object: {
        Bucket: insertedPic.s3.bucket,
        Name: insertedPic.s3.key
      }
    },
    MaxLabels: 10,
    MinConfidence: 75.0
  };

  return aws.rekognition()
    .DetectLabels(args)
    .then(result => {
      return mongodb
        .db('data')
        .collection('picstream')
        .updateOne({_id: insertedPic._id}, {$set: {tags: result.Labels}});
    });
};
```

With just a couple of service calls, I was able to take an image, stored in S3, analyse it with Rekognition, and add the tags to its document. Want to see how it all came together? [Watch the recording on YouTube](https://youtu.be/VK2ffPLnF0o) with the Github repo in the description. [Follow me on Twitch](https://twitch.tv/itsaydrian) to join me and ask questions live.

Use MongoDB with AWS Rekognition to tag and analyse images.

Using AWS Rekognition to Analyse and Tag Uploaded Images

John Page

## Introduction

This article will walk through creating an API using the [Amazon API Gateway](https://aws.amazon.com/api-gateway/)  in front of the  MongoDB Atlas Data API.  When integrating with the Amazon API Gateway, it is possible but undesirable to use a driver, as drivers are designed to be long-lived and maintain connection pooling. Using serverless  functions with a driver can result in either a performance hit – if the driver is instantiated on each call and must authenticate – or excessive connection numbers if the underlying mechanism persists between calls, as you have no control over when code  containers  are reused or created.

The[MongoDB Atlas Data API is](https://docs.atlas.mongodb.com/api/data-api/) an HTTPS-based API that allows us to read and write data in Atlas where a MongoDB driver library is either not available or not desirable. For example, when creating serverless microservices with MongoDB.

AWS (Amazon Web Services) describe their API Gateway as:

>  "A fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. APIs act as the "front door" for applications to access data, business logic, or functionality from your backend services. Using API Gateway, you can create RESTful APIs and WebSocket APIs that enable real-time two-way communication applications. API Gateway supports containerized and serverless workloads, as well as web applications.
> API Gateway handles all the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls, including traffic management, CORS support, authorization and access control, throttling, monitoring, and API version management. API Gateway has no minimum fees or startup costs. You pay for the API calls you receive and the amount of data transferred out and, with the API Gateway tiered pricing model, you can reduce your cost as your API usage scales."

## Prerequisites.

A core requirement for this walkthrough is to have an Amazon Web Services account, the API Gateway is available as part of the AWS free tier, allowing up to 1 million API calls per month, at no charge, in your first 12 months with AWS.

We will also need an Atlas Cluster for which we have enabled the Data API – and our endpoint URL and API Key. You can learn how to get these in this [Article](https://www.mongodb.com/developer/quickstart/atlas_data_api_introduction) or this [Video](https://www.youtube.com/watch?v=46I0wZiTFi4) if you do not have them already.


A common use of Atlas with the Amazon API Gateway might be to provide a managed API to a restricted subset of data in our cluster, which is a common need for a microservice architecture. To demonstrate this, we first need to have some data available in MongoDB Atlas. This can be added by selecting the three dots next to our cluster name and choosing "Load Sample Dataset", or following instructions [here](https://docs.mongodb.com/charts/saas/tutorial/order-data/prerequisites-setup/). 
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_11_29_at_09_50_02_6532382c89.png)


## Creating an API with the Amazon API Gateway and the Atlas Data API
## 
The instructions here are an extended variation from Amazon's own "[Getting Started with the API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started.html)" tutorial. I do not presume to teach you how best to use Amazon's API Gateway as Amazon itself has  many fine resources for this, what we will do here is use it to get a basic Public API enabled that uses the Data API.

> The Data API itself is currently in an early preview with a flat security model  allowing all users who have an API key to query or update any database or collection. Future versions will have more granular security. We would not want to simply expose the current data API as a 'Public' API but we can use it on the back-end to create more restricted and specific access to our data. 
> 
We are going to create an API which allows users to GET the ten films for any given year which received the most awards - a notional "Best Films of the Year". We will restrict this API to performing only that operation and supply the year as part of the URL

We will first create the API, then analyze the code we used for it.

## Create a AWS Lambda Function to retrieve data with the Data API


1. Sign in to the Lambda console at[https://console.aws.amazon.com/lambda](https://console.aws.amazon.com/lambda).
2. Choose **Create function**.
3. For **Function name**, enter top-movies-for-year.
4. Choose **Create function**.
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_24_30_c38ba7b6cb.png)


When you see the Javascript editor that looks like this

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_25_59_2ba3959057.png)
Replace the code with the following, changing the API-KEY and APP-ID to the values for your Atlas cluster. Save and click **Deploy**  (In a production application you might look to store these in AWS Secrets manager , I have simplified by putting them in the code here).

```
const https = require('https');
 
const atlasEndpoint = "/app/APP-ID/endpoint/data/beta/action/find";
const atlasAPIKey = "API-KEY";
 
 
exports.handler = async(event) => {
 
   if (!event.queryStringParameters || !event.queryStringParameters.year) {
      return { statusCode: 400, body: 'Year not specified' };
   }
 
   //Year is a number but the argument is a string so we need to convert as MongoDB is typed
 
 
   let year = parseInt(event.queryStringParameters.year, 10);
   console.log(`Year = ${year}`)
   if (Number.isNaN(year)) { return { statusCode: 400, body: 'Year incorrectly specified' }; }
 
 
   const payload = JSON.stringify({
      dataSource: "Cluster0",
      database: "sample_mflix",
      collection: "movies",
      filter: { year },
      projection: { _id: 0, title: 1, awards: "$awards.wins" },
      sort: { "awards.wins": -1 },
      limit: 10
   });
 
 
   const options = {
      hostname: 'data.mongodb-api.com',
      port: 443,
      path: atlasEndpoint,
      method: 'POST',
      headers: {
         'Content-Type': 'application/json',
         'Content-Length': payload.length,
         'api-key': atlasAPIKey
      }
   };
 
   let results = '';
 
   const response = await new Promise((resolve, reject) => {
      const req = https.request(options, res => {
         res.on('data', d => {
            results += d;
         });
         res.on('end', () => {
            console.log(`end() status code = ${res.statusCode}`);
            if (res.statusCode == 200) {
               let resultsObj = JSON.parse(results)
               resolve({ statusCode: 200, body: JSON.stringify(resultsObj.documents, null, 4) });
            }
            else {
               reject({ statusCode: 500, body: 'Your request could not be completed, Sorry' }); //Backend Problem like 404 or wrong API key
            }
         });
      });
      //Do not give the user clues about backend issues for security reasons
      req.on('error', error => {
         reject({ statusCode: 500, body: 'Your request could not be completed, Sorry' }); //Issue like host unavailable
      });
 
      req.write(payload);
      req.end();
   });
   return response;
 
};

```

Alternatively, if you are familiar with working with packages and Lambda, you could upload an HTTP package like Axios to Lambda as a zipfile, allowing you to  use the following simplified code.

```

const axios = require('axios');

const atlasEndpoint = "https://data.mongodb-api.com/app/APP-ID/endpoint/data/beta/action/find";
const atlasAPIKey = "API-KEY";


exports.handler = async(event) => {

   if (!event.queryStringParameters || !event.queryStringParameters.year) {
      return { statusCode: 400, body: 'Year not specified' };
   }

   //Year is a number but the argument is a string so we need to convert as MongoDB is typed


   let year = parseInt(event.queryStringParameters.year, 10);
   console.log(`Year = ${year}`)
   if (Number.isNaN(year)) { return { statusCode: 400, body: 'Year incorrectly specified' }; }


   const payload = {
      dataSource: "Cluster0",
      database: "sample_mflix",
      collection: "movies",
      filter: { year },
      projection: { _id: 0, title: 1, awards: "$awards.wins" },
      sort: { "awards.wins": -1 },
      limit: 10
   };

   try {
      const response = await axios.post(atlasEndpoint, payload, { headers: { 'api-key': atlasAPIKey } });
      return response.data.documents;
   }
   catch (e) {
      return { statusCode: 500, body: 'Unable to service request' }
   }
};
```





## Create an HTTP endpoint for our custom API function
## 
We now need to route an HTTP endpoint to our Lambda function using the HTTP API. 

The HTTP API provides an HTTP endpoint for your Lambda function. API Gateway routes requests to your Lambda function, and then returns the function's response to clients.

1. Go to the API Gateway console at[https://console.aws.amazon.com/apigateway.](https://console.aws.amazon.com/apigateway)
2. Do one of the following:
  To create your first API, for HTTP API, choose **Build**.
   If you've created an API before, choose **Create API**, and then choose **Build** for HTTP API.
3. For Integrations, choose **Add integration**.
4. Choose **Lambda**.
5. For **Lambda function**, enter top-movies-for-year.
6. For **API name**, enter movie-api.
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_38_55_cfcd82c333.png)
8. Choose **Next**.

8. Review the route that API Gateway creates for you, and then choose **Next**.
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_42_20_23bd84911f.png)
9. Review the stage that API Gateway creates for you, and then choose **Next**.
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_43_43_ebab329b15.png)
10. Choose **Create**.

Now you've created an HTTP API with a Lambda integration and the Atlas Data API that's ready to receive requests from clients.

## Test your API

You should now be looking at API Gateway details that look like this, if not you can get to it by going to[https://console.aws.amazon.com/apigateway](https://console.aws.amazon.com/apigateway)and clicking on **movie-api**

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_46_17_06301a223e.png)


Take a note of the **Invoke URL**, this is the base URL for your API

Now, in a new browser tab,  browse to  `<Invoke URL>/top-movies-for-year?year=2001` . Changing `<Invoke URL> `to the Invoke URL shown in AWS. You should see the results of your API call - JSON listing the top 10 "Best" films of 2001.

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/unnamed_33c3be7d0f.png)
## Reviewing our Function.
## 
We start by importing the Standard node.js https library - the Data API needs no special libraries to call it. We also define our API Key and the path to our find endpoint, You get both of these from the Data API tab in Atlas.


```
const https = require('https');
 
const atlasEndpoint = "/app/data-amzuu/endpoint/data/beta/action/find";
const atlasAPIKey = "YOUR-API-KEY";
```




Now we check that the API call included a parameter for year and that it's a number - we need to convert it to a number as in MongoDB, "2001" and 2001  are different values, and searching for one will not find the other. The collection uses a number for the movie release year.


 
```
exports.handler = async (event) => {
 
   if (!event.queryStringParameters || !event.queryStringParameters.year) {
       return { statusCode: 400, body: 'Year not specified' };
   }
   //Year is a number but the argument is a string so we need to convert as MongoDB is typed
   let year = parseInt(event.queryStringParameters.year, 10);
   console.log(`Year = ${year}`)
   if (Number.isNaN(year)) { return { statusCode: 400, body: 'Year incorrectly specified' }; }
 
 
   const payload = JSON.stringify({
       dataSource: "Cluster0", database: "sample_mflix", collection: "movies",
       filter: { year }, projection: { _id: 0, title: 1, awards: "$awards.wins" }, sort: { "awards.wins": -1 }, limit: 10
   });

```



THen we construct our payload - the parameters for the Atlas API Call, we are querying for year = year, projecting just the title and the number of awards, sorting by the numbers of awards descending and limiting to 10.


 
```
  const payload = JSON.stringify({
       dataSource: "Cluster0", database: "sample_mflix", collection: "movies",
       filter: { year }, projection: { _id: 0, title: 1, awards: "$awards.wins" },  
       sort: { "awards.wins": -1 }, limit: 10
   });

```



We then construct the options for the HTTPS POST request to the Data API - here we pass the Data API API-KEY as a header.


```
  const options = {
       hostname: 'data.mongodb-api.com',
       port: 443,
       path: atlasEndpoint,
       method: 'POST',
       headers: {
           'Content-Type': 'application/json',
           'Content-Length': payload.length,
           'api-key': atlasAPIKey
       }
   };


```


Finally we use some fairly standard code to call the API and handle errors. We can get Request errors - such as being unable to contact the server - or Response errors where we get any Response code other than 200 OK - In both cases we return a  500 Internal error from our simplified API to not leak any details of the internals to a potential hacker.


 
```
   let results = '';
 
   const response = await new Promise((resolve, reject) => {
       const req = https.request(options, res => {
           res.on('data', d => {
               results += d;
           });
           res.on('end', () => {
               console.log(`end() status code = ${res.statusCode}`);
               if (res.statusCode == 200) {
                   let resultsObj = JSON.parse(results)
                   resolve({ statusCode: 200, body: JSON.stringify(resultsObj.documents, null, 4) });
               } else {
                   reject({ statusCode: 500, body: 'Your request could not be completed, Sorry' }); //Backend Problem like 404 or wrong API key
               }
           });
       });
       //Do not give the user clues about backend issues for security reasons
       req.on('error', error => {
           reject({ statusCode: 500, body: 'Your request could not be completed, Sorry' }); //Issue like host unavailable
       });
 
       req.write(payload);
       req.end();
   });
   return response;
 
};
```




Our Axios verison is just the same functionality as above but simplified by the use of a library.
## Conclusion

As we can see, calling the Atlas Data API from AWS Lambda function is incredibly simple, especially if making use of a library like Axios. The Data API is also stateless, so there are no concerns about connection setup times or maintaining long lived connections as there would be using a  Driver. 


In this article we look at how the Atlas Data API is a great choice for accessing MongoDB Atlas from AWS Lambda Functions by creating a custom API with the AWS API Gateway. 

Creating an API with the AWS API Gateway and the Atlas Data API

Pierre Petersson

Johannes Brännström

## Why CI/CD?

To increase the speed and quality of development, you may use continuous delivery strategies to manage and deploy your application code changes. However, continuous delivery for databases is often a manual process.

Adopting continuous integration and continuous delivery (CI/CD) for managing the lifecycle of a database has the following benefits:

* An automated multi-environment setup enables you to move faster and focus on what really matters.
* The confidence level of the changes applied increases.
* The process is easier to reproduce.
* All changes to database configuration will be traceable.

### Why CI/CD for MongoDB Atlas?

[MongoDB Atlas](https://www.mongodb.com/cloud/atlas) is a multi-cloud application data platform, providing an integrated suite of cloud database and data services to accelerate and simplify how you build with data. MongoDB Atlas also provides a comprehensive [API](https://docs.atlas.mongodb.com/api/atlas-admin-api/), making CI/CD for the actual data platform itself possible.

In this blog, we’ll demonstrate how to set up CI/CD for MongoDB Atlas, in a typical production setting. The intended audience is developers, solutions architects, and database administrators with knowledge of MongoDB Atlas, AWS, and Terraform.

## Our CI/CD Solution Requirements

* Ensure that each environment (dev, test, prod) is isolated to minimize blast radius in case of a human error or from a security perspective. MongoDB Atlas Projects and API Keys will be utilized to enable environment isolation.
* All services used in this solution will use managed services. This to minimize the time needed to spend on managing infrastructure.
* Minimize commercial agreements required. Use as much as possible from AWS and the Atlas ecosystem so that there is no need to purchase external tooling, such as HashiCorp Vault. 
* Minimize time spent on installing local dev tooling, such as git and Terraform. The solution will provide a docker image, with all tooling required to run provisioning of Terraform templates. The same image will be used to also run the pipeline in AWS CodeBuild. 

## Implementation

Enough talk—let’s get to the action. As developers, we love working examples as a way to understand how things work. So, here’s how we did it.

### Prerequisites

First off, we need to have at least an Atlas account to provision Atlas and then somewhere to run our automation. You can [get an Atlas account for free](https://www.mongodb.com/cloud/atlas/register) at mongodb.com. If you want to take this demo for a spin, take the time and create your Atlas account now. Next, you’ll need to [create an organization-level API key](https://docs.atlas.mongodb.com/tutorial/manage-programmatic-access/). If you or your org already have an Atlas account you’d like to use, you’ll need the organization owner to create the organization-level API key.

Second, you’ll need an AWS account. For more information on how to create an AWS account, see [How do I create an AWS account?](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/) For this demo, we’ll be using some for-pay services like S3, but you get 12 months free. 

You will also need to have Docker installed as we are using a docker container to run all provisioning. For more information on how to install Docker, see [Get Started with Docker](https://www.docker.com/get-started). We are using Docker as it will make it easier for you to get started, as all the tooling is packaged in the container—such as AWS cli, mongosh, and Terraform.

### What You Will Build

* [MongoDB Atlas Projects](https://docs.atlas.mongodb.com/tutorial/manage-projects/) for dev, test, prod environments, to minimize blast radius in case of a human error and from a security perspective.
* [MongoDB Atlas Cluster](https://docs.atlas.mongodb.com/tutorial/create-new-cluster/) in each Atlas project (dev, test, prod). MongoDB Atlas is a fully managed data platform for modern applications. Storing data the way it is accessed as documents makes developers more productive. It provides a document-based database that is cost-efficient and resizable while automating time-consuming administration tasks such as hardware provisioning, database setup, patching, and backups. It allows you to focus on your applications by providing the foundation of high performance, high availability, security, and compatibility they need.
* [CodePipeline](https://aws.amazon.com/codepipeline/) orchestrates the CI/CD database migration stages.
* [IAM](https://aws.amazon.com/iam/) roles and policies allow cross-account access to applicable AWS resources.
* [CodeCommit](https://aws.amazon.com/codecommit/) creates a repo to store the SQL statements used when running the database migration.
* [Amazon S3](https://aws.amazon.com/s3/) creates a bucket to store pipeline artifacts.
* [CodeBuild](https://aws.amazon.com/codebuild) creates a project in target accounts using Flyway to apply database changes.
* VPC [security groups](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) ensure the secure flow of traffic between a CodeBuild project deployed within a VPC and MongoDB Atlas. AWS Private Link will also be provisioned.
* [AWS Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) stores secrets securely and centrally, such as the Atlas API keys and database username and password.
* [Amazon SNS](https://aws.amazon.com/sns/) notifies you by email when a developer pushes changes to the CodeCommit repo.

### Step 1: Bootstrap AWS Resources

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/aws_codepipeline_161a68794c.png)

Next, we’ll fire off the script to bootstrap our AWS environment and Atlas account as shown in Diagram 1 using Terraform.

You will need to use [programmatic access keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html) for your AWS account and the Atlas organisation-level API key that you have created as described in the prerequisites.This is also the only time you’ll need to handle the keys manually. 

```
# Set your environment variables

# You'll find this in your Atlas console as described in prerequisites
export ATLAS_ORG_ID=60388113131271beaed5

# The public part of the Atlas Org key you created previously 
export ATLAS_ORG_PUBLIC_KEY=l3drHtms

# The private part of the Atlas Org key you created previously 
export ATLAS_ORG_PRIVATE_KEY=ab02313b-e4f1-23ad-89c9-4b6cbfa1ed4d


# Pick a username, the script will create this database user in Atlas
export DB_USER_NAME=demouser

# Pick a project base name, the script will appended -dev, -test, -prod depending on environment
export ATLAS_PROJECT_NAME=blogcicd6


# The AWS region you want to deploy into
export AWS_DEFAULT_REGION=eu-west-1


# The AWS public programmatic access key
export AWS_ACCESS_KEY_ID=AKIAZDDBLALOZWA3WWQ


# The AWS private programmatic access key
export AWS_SECRET_ACCESS_KEY=nmarrRZAIsAAsCwx5DtNrzIgThBA1t5fEfw4uJA


```

Once all the parameters are defined, you are ready to run the script that will create your CI/CD pipeline.

```
# Clone solution code repository
$ git clone https://github.com/mongodb-developer/atlas-cicd-aws
$ cd atlas-cicd

# Start docker container, which contains all the tooling e.g terraform, mongosh, and other, 
$ docker container run -it --rm -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION -e ATLAS_ORG_ID -e ATLAS_ORG_PUBLIC_KEY -e ATLAS_ORG_PRIVATE_KEY -e DB_USER_NAME -e ATLAS_PROJECT_NAME -v ${PWD}/terraform:/terraform piepet/cicd-mongodb:46


$ cd terraform   

# Bootstrap AWS account and Atlas Account
$ ./deploy_baseline.sh $AWS_DEFAULT_REGION $ATLAS_ORG_ID $ATLAS_ORG_PUBLIC_KEY $ATLAS_ORG_PRIVATE_KEY $DB_USER_NAME $ATLAS_PROJECT_NAME base apply




```

When deploy:baseline.sh is invoked, provisioning of AWS resources starts, using Terraform templates. The resources created are shown in Diagram 1.


From here on, you'll be able to operate your Atlas infrastructure without using your local docker instance. If you want to blaze through this guide, including cleaning it all up, you might as well keep the container running, though. The final step of tearing down the AWS infrastructure requires an external point like your local docker instance.

Until you’ve committed anything, the pipeline will have a failed Source stage. This is because it tries to check out a branch that does not exist in the code repository. After you’ve committed the Terraform code you want to execute, you’ll see that the Source stage will restart and proceed as expected. You can find the pipeline in the AWS console at this url: [https://eu-west-1.console.aws.amazon.com/codesuite/codepipeline/pipelines?region=eu-west-1](https://eu-west-1.console.aws.amazon.com/codesuite/codepipeline/pipelines?region=eu-west-1)

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/failing_pipeline_in_codepipeline_bc5629211b.png)

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/codepipeline_5ec9666ae0.png)

### Step 2: Deploy Atlas Cluster

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/deploying_an_atlas_cluster_f951a572c1.png)

Next is to deploy the Atlas cluster (projects, users, API keys, etc). This is done by pushing a configuration into the new AWS CodeCommit repo. 

If you’re like me and want to see how provisioning of the Atlas cluster works before setting up IAM properly, you can push the original github repo to AWS CodeCommit directly inside the docker container (inside the Terraform folder) using a bit of a hack. By pushing to the CodeCommit repo, AWS CodePipeline will be triggered and provisioning of the Atlas cluster will start. 

```
cd /terraform
# Push default settings to AWS Codecommit
./git_push_terraform.sh


```

To set up access to the CodeCommit repo properly, for use that survives stopping the docker container, you’ll need a proper git CodeCommit user. Follow the steps in the [AWS documentation](https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-gc.html) to create and configure your CodeCommit git user in AWS IAM. Then clone the AWS CodeCommit repository that was created in the bootstrapping, outside your docker container, perhaps in another tab in your shell, using your IAM credentials. If you did not use the “hack” to initialize it, it’ll be empty, so copy the Terraform folder that is provided in this solution, to the root of the cloned CodeCommit repository, then commit and push to kick off the pipeline. Now you can use this repo to control your setup! You should now see in the AWS CodePipeline console that the pipeline has been triggered. The pipeline will create Atlas clusters in each of the Atlas Projects and configure AWS PrivateLink. 

Let’s dive into the stages [defined in this Terraform pipeline file](https://github.com/mongodb-developer/atlas-cicd-aws/blob/main/terraform/modules/aws/code-pipeline/main.tf).

**Deploy-Base**
This is basically re-applying what we did in the bootstrapping. This stage ensures we can improve on the AWS pipeline infrastructure itself over time.

This stage creates the projects in Atlas, including Atlas project API keys, Atlas project users, and database users. 

**Deploy-Dev**

This stage creates the corresponding Private Link and MongoDB cluster.

**Deploy-Test**

This stage creates the corresponding Private Link and MongoDB cluster.

**Deploy-Prod**

This stage creates the corresponding Private Link and MongoDB cluster.

**Gate**

Approving means we think it all looks good. Perhaps counter intuitively but great for demos, it proceeds to teardown. This might be one of the first behaviours you’ll change. :)

**Teardown**

This decommissions the dev, test, and prod resources we created above. To decommission the base resources, including the pipeline itself, we recommend you run that externally—for example, from the Docker container on your laptop. We’ll cover that later.

As you advance towards the Gate stage, you’ll see the Atlas clusters build out. Below is an example where the Test stage is creating a cluster. Approving the Gate will undeploy the resources created in the dev, test, and prod stages, but keep projects and users.

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/project_names_and_clusters_041271acd0.png)

### Step 3: Make a Change!

Assuming you took the time to set up IAM properly, you can now work with the infrastructure as code directly from your laptop outside the container. If you just deployed using the hack inside the container, you can continue interacting using the repo created inside the Docker container, but at some point, the container will stop and that repo will be gone. So, beware.

Navigate to the root of the clone of the CodeCommit repo. For example, if you used the script in the container, you’d run, also in the container:

```
cd /${ATLAS_PROJECT_NAME}-base-repo/

```

Then you can edit, for example, the MongoDB version by changing 4.4 to 5.0 in `terraform/environment/dev/variables.tf`.

```
variable "cluster_mongodbversion" {
 description = "The Major MongoDB Version"
 default = "5.0"
}
```

Then push (git add, commit, push) and you’ll see a new run initiated in CodePipeline.

### Step 4: Clean Up Base Infrastructure

Now, that was interesting. Time for cleaning up! To decommission the full environment, you should first approve the Gate stage to execute the teardown job. When that’s been done, only the base infrastructure remains. Start the container again as in Step 1 if it’s not running, and then execute deploy_baseline.sh, replacing the word ***apply*** with ***destroy***: 

```
# inside the /terraform folder of the container

# Clean up AWS and Atlas Account
./deploy_baseline.sh $AWS_DEFAULT_REGION $ATLAS_ORG_ID $ATLAS_ORG_PUBLIC_KEY $ATLAS_ORG_PRIVATE_KEY $DB_USER_NAME $ATLAS_PROJECT_NAME base destroy

```

## Lessons Learned

In this solution, we have separated the creation of AWS resources and the Atlas cluster, as the changes to the Atlas cluster will be more frequent than the changes to the AWS resources. 

When implementing infrastructure as code for a MongoDB Atlas Cluster, you have to consider not just the cluster creation but also a strategy for how to separate dev, qa, and prod environments and how to store secrets. This to minimize blast radius. 

We also noticed how useful resource tagging is to make Terraform scripts portable. By setting tags on AWS resources, the script does not need to know the names of the resources but can look them up by tag instead.

## Conclusion

By using CI/CD automation for Atlas clusters, you can speed up deployments and increase the agility of your software teams. 

MongoDB Atlas offers a powerful API that, in combination with AWS CI/CD services and Terraform, can support continuous delivery of MongoDB Atlas clusters, and version-control the database lifecycle. You can apply the same pattern with other CI/CD tools that aren’t specific to AWS. 

In this blog, we’ve offered an exhaustive, reproducible, and reusable deployment process for MongoDB Atlas, including traceability. A devops team can use our demonstration as inspiration for how to quickly deploy MongoDB Atlas, automatically embedding organisation best practices. 

In this blog, we’ll demonstrate how to set up CI/CD for MongoDB Atlas, in a typical production setting.

Building a Multi-Environment Continuous Delivery Pipeline for MongoDB Atlas

>As of June 2022, the functionality previously known as Atlas Data Lake is now named Atlas Data Federation. Atlas Data Federation’s functionality is unchanged and you can learn more about it [here](http://mongodb.com/atlas/data-federation). Atlas Data Lake will remain in the Atlas Platform, with newly introduced functionality that you can learn about [here](http://mongodb.com/atlas/data-lake).

Have you ever needed to make queries across databases, clusters, data
centers, or even mix it with data stored in a AWS S3 blob? You probably
haven't had to do all of these at once, but I'm guessing you've needed
to do at least one of these at some point in your career. I'll also bet
that you didn't know that this is possible (and easy) to do with MongoDB
federated queries on a [MongoDB Atlas Data
Lake](https://docs.mongodb.com/datalake/)! These allow you to configure
multiple remote MongoDB deployments, and enable federated queries across
all the configured deployments.

![MongoDB Atlas Data Lake design graphic showing a lake with JSON brackets for waves and a sample data overlay.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/atlas_data_lake_d52555700e.png)

**MongoDB Federated Query** allows you to perform queries across many
MongoDB systems, including Clusters, Databases, and even AWS S3 buckets.
Here's how MongoDB federated query works in practice.

![Diagram showing how MongoDB Atlas Data Lake uses a compute plane to distribute and perform queries across multiple MongoDB Databases.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/data_lake_diagram_421db09adb.png)

Note: In this post, we will be demoing how to query from two separate
databases. However, if you want to query data from two separate
[collections](https://docs.mongodb.com/manual/reference/glossary/#term-collection)
that are in the same database, I would personally recommend that you use
the [$lookup (aggregation
pipeline)](https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/index.html)
query. $lookup performs a left outer join to an unsharded collection in
the same database to filter documents from the "joined" collection for
processing. In this scenario, using a data lake is not necessary.

*tl;dr: In this post, I will guide you through the process of creating
and connecting to a Data Lake in MongoDB Atlas, configuring paths to
collections in two separate MongoDB databases stored in separate
datacenters, and querying data from both databases using only a single
query.*

## Prerequisites

In order to follow along this tutorial, you need to:

-   Create at least two [M10
    clusters](https://docs.atlas.mongodb.com/cluster-tier/) in MongoDB
    Atlas. For this demo, I have created two databases deployed to
    separate Cloud Providers (AWS and GCP). [Click here for information
    on setting up a new MongoDB Atlas
    cluster](https://docs.atlas.mongodb.com/getting-started/).

![Screenshot of MongoDB Atlas Cluster overview page. There are red boxes highlighting that the two clusters used in this example are being hosted in two different cloud providers, AWS and GCP.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/multiple_clusters_7832e1ce14.png)

-   Ensure that each database has been seeded by [loading sample data
    into our Atlas
    cluster](https://docs.atlas.mongodb.com/sample-data/).
-   Have a [Mongo Shell](https://docs.mongodb.com/manual/mongo/)
    installed.

## Deploy a Data Lake

First, make sure you are logged into [MongoDB
Atlas](https://account.mongodb.com/account/login). Next, select the Data
Lake option on the left-hand navigation.

![Screenshot from the MongoDB Atlas cluster overview page with a red box highlighting the Data Lake navigation button on the right side of the screen.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/atlas_dashboard_data_lake_ba99cb1a5e.png)

Create a Data Lake.

-   For your first Data Lake, click Create a Data Lake.
-   For your subsequent Data Lakes, click Configure a New Data Lake.

![Screenshot from the MongoDB Atlas Data Lake overview page with a red box highlighting the Create Data Lake button.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_data_lake_cd93e36eb7.png)

Click **Connect Data** on the Data Lake Configuration page, and select
**MongoDB Atlas Cluster**. Select your first cluster, input
`sample_airbnb` as the databases and `listingsAndReviews` as the
collection. For this tutorial, we will be analyzing Airbnb rental data
and some sample weather data to see if we can draw any insights into
renting behaviors on Airbnb and the weather.

![Screenshot from the MongoDB Atlas Data Lake creation modal showing how I filled in the form for this demo.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/data_lake_creation_initial_1e4fe48c35.png)

Repeat the steps above to connect the data for your other cluster and
data source.

![Screenshot from the MongoDB Atlas Data Lake creation modal showing how I filled in the form for this demo.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/data_lake_creation_b2ead5c2bd.png)

Next, drag these new data stores into your data lake and click **save**.
It should look like this.

![Screenshot from the MongoDB Atlas Data Lake overview page with a red box highlighting the data sources we created in the previous step and how I positioned them in the data lake.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/highlight_data_sources_a7d1f99f2e.png)

## Connect to Your Data Lake

The next thing we are going to need to do after setting up our data lake
is to connect to it so we can start running queries on all of our data.
First, click connect in the second box on the data lake overview page.

![Screenshot from the MongoDB Atlas Data Lake overview page with a red box highlighting the "Connect to your Data Lake" button.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/connect_data_lake_9ac7b7c19c.png)

Click Add Your Current IP Address. Enter your IP address and an optional
description, then click **Add IP Address**. In the **Create a MongoDB
User** step of the dialog, enter a Username and a Password for your
database user. (Note: You'll use this username and password combination
to access data on your cluster.)

## Run Queries Against Your Data Lake

You can run your queries any way you feel comfortable. You can use
MongoDB Compass, the MongoDB Shell, connect to an application, or
anything you see fit. For this demo, I'm going to be running my queries
using [MongoDB Visual Studio Code
plugin](https://marketplace.visualstudio.com/items?itemName=mongodb.mongodb-vscode)
and leveraging its
[Playgrounds](https://docs.mongodb.com/mongodb-vscode/playgrounds/)
feature. For more information on using this plugin, check out this
[post](https://developer.mongodb.com/how-to/mongodb-visual-studio-code-plugin/)
on our Developer Hub.

Make sure you are using the connection string for your data lake and not
for your individual MongoDB databases. To get the connection string for
your new data lake, click the connect button on the MongoDB Atlas Data
Lake overview page. Then click on Connect using **MongoDB Compass**.
Copy this connection string to your clipboard. Note: You will need to
add the password of the user that you authorized to access your data
lake here.

![Screenshot from the MongoDB Atlas Data Lake connection modal with a red box highlighting your connection string that you will use to connect to your data lake.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/highlight_connection_string_16ba260a30.png)

You're going to paste this connection string into the MongoDB Visual
Studio Code plugin when you add a new connection.

![Screenshot from the MongoDB Visual Studio Code plugin showing where to paste your MongoDB Atlas Data Lake Connection string.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/15_vscode_enter_connection_string_b785102dae.png)

Note: If you need assistance with getting started with the MongoDB
Visual Studio Code Plugin, be sure to check out my post, [How To Use The
MongoDB Visual Studio Code
Plugin](https://developer.mongodb.com/how-to/mongodb-visual-studio-code-plugin/),
and the [official
documentation](https://docs.mongodb.com/mongodb-vscode/).

You can run operations using the MongoDB Query Language (MQL) which
includes most, but not all, standard server commands. To learn which MQL
operations are supported, see the [MQL
Support](https://docs.mongodb.com/datalake/supported-unsupported/mql-support/#std-label-data-lake-mql-support)
documentation.

The following queries use the paths that you added to your Data Lake
during
[deployment](https://docs.mongodb.com/datalake/tutorial/deploy/#std-label-gst-deploy-adl).

For this query, I wanted to construct a unique aggregation that could
only be used if both sample datasets were combined using federated query
and MongoDB Atlas Data Lake. For this example, I am running a query to
determine the number of theaters and restaurants in each zip code, by
analyzing the `sample_restaurants.restaurants` and the
`sample_mflix.theaters` datasets. If you haven't added these data
sources to your data lake, be sure to do that before moving forward with
this query.

I want to make it clear that these data sources are still being stored
in different MongoDB databases in completely different datacenters, but
by leveraging MongoDB Atlas Data Lake, we can query all of our databases
at once as if all of our data is in a single collection! The following
query is only possible using federated search! How cool is that?

``` javascript
// MongoDB Playground

// Select the database to use. Database0 is the default name for a MongoDB Atlas Data Lake database. If you renamed your database, be sure to put in your data lake database name here.
use('Database0');

// We are connecting to `Collection0` since this is the default collection that MongoDB Atlas Data Lake calls your collection. If you renamed it, be sure to put in your data lake collection name here.
db.Collection0.aggregate([

  // In the first stage of our aggregation pipeline, we extract and normalize the dataset to only extract zip code data from our dataset.
  {
    '$project': {
      'restaurant_zipcode': '$address.zipcode',
      'theater_zipcode': '$location.address.zipcode',
      'zipcode': {
        '$ifNull': [
          '$address.zipcode', '$location.address.zipcode'
        ]
      }
    }
  },

  // In the second stage of our aggregation, we group the data based on the zip code it resides in. We also push each unique restaurant and theater into an array, so we can get a count of the number of each in the next stage.
  // We are calculating the `total` number of theaters and restaurants by using the aggregator function on $group. This sums all the documents that share a common zip code.
  {
    '$group': {
      '_id': '$zipcode',
      'total': {
        '$sum': 1
      },
      'theaters': {
        '$push': '$theater_zipcode'
      },
      'restaurants': {
        '$push': '$restaurant_zipcode'
      }
    }
  },

  // In the third stage, we get the size or length of the `theaters` and `restaurants` array from the previous stage. This gives us our totals for each category.
  {
    '$project': {
      'zipcode': '$_id',
      'total': '$total',
      'total_theaters': {
        '$size': '$theaters'
      },
      'total_restaurants': {
        '$size': '$restaurants'
      }
    }
  },

  // In our final stage, we sort our data in descending order so that the zip codes with the most number of restaurants and theaters are listed at the top.
  {
    '$sort': {
      'total': -1
    }
  }
])
```

This outputs the zip codes with the most theaters and restaurants.

``` json
[
  {
    "_id": "10003",
    "zipcode": "10003",
    "total": 688,
    "total_theaters": 2,
    "total_restaurants": 686
  },
  {
    "_id": "10019",
    "zipcode": "10019",
    "total": 676,
    "total_theaters": 1,
    "total_restaurants": 675
  },
  {
    "_id": "10036",
    "zipcode": "10036",
    "total": 611,
    "total_theaters": 0,
    "total_restaurants": 611
  },
  {
    "_id": "10012",
    "zipcode": "10012",
    "total": 408,
    "total_theaters": 1,
    "total_restaurants": 407
  },
  {
    "_id": "11354",
    "zipcode": "11354",
    "total": 379,
    "total_theaters": 1,
    "total_restaurants": 378
  },
  {
    "_id": "10017",
    "zipcode": "10017",
    "total": 378,
    "total_theaters": 1,
    "total_restaurants": 377
  }
 ]
```

## Wrap-Up

Congratulations! You just set up an Atlas Data Lake that contains
databases being run in different cloud providers. Then, you queried both
databases using the MongoDB Aggregation pipeline by leveraging AtlasData
Lake and federated queries. This allows us to more easily run queries on
data that is stored in multiple MongoDB database deployments across
clusters, data centers, and even in different formats, including S3 blob
storage.

![Screenshot from the MongoDB Atlas Data Lake overview page showing the information for our new Data Lake.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/data_lake_overview_521ed9ba4f.png)

Screenshot from the MongoDB Atlas Data Lake overview page showing the
information for our new Data Lake.

>
>
>If you have questions, please head to our [developer community
>website](https://community.mongodb.com/) where the MongoDB engineers and
>the MongoDB community will help you build your next big idea with
>MongoDB.
>
>

## Additional Resources

-   [Getting Started with MongoDB Atlas Data Lake
    \[Docs\]](https://docs.mongodb.com/datalake/tutorial/getting-started/)
-   [AWS re\:Invent 2020 Auto-archiving and federated
    queries](https://www.youtube.com/watch?v=1VWh55uRfnY)
-   [Tutorial: Federated Queries and $out to AWS
    S3](https://developer.mongodb.com/how-to/atlas-data-lake-federated-queries-out-aws-s3/)


Learn how to query from multiple MongoDB databases using MongoDB Atlas Data Lake.

How to Query from Multiple MongoDB Databases Using MongoDB Atlas Data Lake

Frank Wang

[Serverless computing](https://www.mongodb.com/serverless) is now becoming the standard for how applications are developed in the cloud. Serverless at its core lets developers create small packages of code that are executed by a cloud provider to respond to events. These events can range from HTTP requests to cron jobs, or even file upload notifications. These packages of code are called functions or Lambda functions, named after [AWS Lambda](https://aws.amazon.com/lambda/), the AWS service that powers them. This model allows serverless services to scale with ease and be incredibly cost effective, as you only pay for the exact number of milliseconds it takes to execute them.

However, working with Lambda functions locally can be tricky. You’ll need to either emulate the events locally or redeploy them to the cloud every time you make a change. On the other hand, due to the event-based execution of these functions, you’ll need to use services that support a similar model as well. For instance, a traditional database expects you to hold on to a connection and reuse it to make queries. This doesn’t work well in the serverless model, since Lambda functions are effectively stateless. Every invocation of a Lambda function creates a new environment. The state from all previous invocations is lost unless committed to persistent storage.

Over the years, there has been a steady stream of improvements from the community to address these challenges. The developer experience is now at a point where it’s incredibly easy to build full-stack applications with serverless. In this post, we’ll look at the new way for building serverless apps. This includes using:

* [Serverless Stack $SST$](https://serverless-stack.com), a framework for building serverless apps with a great local developer experience.
* A [serverless database](https://www.mongodb.com/cloud/atlas/serverless) in MongoDB Atlas, the most advanced cloud database service on the market.

> “With MongoDB Atlas and SST, it’s now easier than ever to build full-stack serverless applications.” — Andrew Davidson, Vice President, Product Management, MongoDB 

Let’s look at how using these tools together can make it easy to build serverless applications.

## Developing serverless apps locally

Lambda functions are packages of code that are executed in response to cloud events. This makes it a little tricky to work with them locally, since you want them to respond to events that happen in the cloud. You can work around this by emulating these events locally or by redeploying your functions to test them. Both these approaches don’t work well in practice.

### Live Lambda development with SST

[SST](https://serverless-stack.com) is a framework for building serverless applications that allows developers to work on their Lambda functions locally. It features something called [Live Lambda Development](https://docs.serverless-stack.com/live-lambda-development) that proxies the requests from the cloud to your local machine, executes them locally, and sends the results back. This allows you to work on your functions locally without having to redeploy them or emulate the events.

Live Lambda development thus allows you to work on your serverless applications, just as you would with a traditional server-based application.

## Serverless databases

Traditional databases operate under the assumption that there is a consistent connection between the application and the database. It also assumes that you’ll be responsible for scaling the database capacity, just as you would with your server-based application.

However, in a serverless context, you are simply writing the code and the cloud provider is responsible for scaling and operating the application. You expect your database to behave similarly as well. You also expect it to handle new connections automatically.

### On-demand serverless databases with MongoDB Atlas

To address this issue, MongoDB Atlas launched [serverless instances](https://www.mongodb.com/cloud/atlas/serverless), currently available in preview. This allows developers to use MongoDB’s world class developer experience without any setup, maintenance, or tuning. You simply pick the region and you’ll receive an on-demand database endpoint for your application.

You can then make queries to your database, just as you normally would, and everything else is taken care of by Atlas. Serverless instances automatically scale up or down depending on your usage, so you never have to worry about provisioning more than you need and you only pay for what you use.

## Get started

In this post, we saw a quick overview of the current serverless landscape, and how serverless computing abstracts and automates away many of the lower level infrastructure decisions. So, you can focus on building features that matter to your business!

For help building your first serverless application with SST and MongoDB Atlas, check out our tutorial: [How to use MongoDB Atlas in your serverless app](https://serverless-stack.com/examples/how-to-use-mongodb-atlas-in-your-serverless-app.html).

>"MongoDB Atlas’ serverless instances and SST allow developers to leverage MongoDB’s unparalleled developer experience to build full-stack serverless apps." — Jay V, CEO, SST 

Also make sure to check out the [quick start](https://www.mongodb.com/developer/how-to/atlas-serverless-quick-start/) and join the [community forums](https://www.mongodb.com/community/forums/) for more insights on building serverless apps with MongoDB Atlas.


The developer experience is at a point where it’s easy to build full-stack applications with serverless. In this post, we’ll look at the new way of building serverless apps.

How to Build Serverless Applications with SST and MongoDB Atlas

## Introduction

This article will walk through creating an API using the [Amazon API Gateway](https://aws.amazon.com/api-gateway/)  in front of the  MongoDB Atlas Data API.  When integrating with the Amazon API Gateway, it is possible but undesirable to use a driver, as drivers are designed to be long-lived and maintain connection pooling. Using serverless  functions with a driver can result in either a performance hit – if the driver is instantiated on each call and must authenticate – or excessive connection numbers if the underlying mechanism persists between calls, as you have no control over when code  containers  are reused or created.

The[MongoDB Atlas Data API is](https://docs.atlas.mongodb.com/api/data-api/) an HTTPS-based API that allows us to read and write data in Atlas where a MongoDB driver library is either not available or not desirable. For example, when creating serverless microservices with MongoDB.

AWS (Amazon Web Services) describe their API Gateway as:

>  "A fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. APIs act as the "front door" for applications to access data, business logic, or functionality from your backend services. Using API Gateway, you can create RESTful APIs and WebSocket APIs that enable real-time two-way communication applications. API Gateway supports containerized and serverless workloads, as well as web applications.
> API Gateway handles all the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls, including traffic management, CORS support, authorization and access control, throttling, monitoring, and API version management. API Gateway has no minimum fees or startup costs. You pay for the API calls you receive and the amount of data transferred out and, with the API Gateway tiered pricing model, you can reduce your cost as your API usage scales."

## Prerequisites.

A core requirement for this walkthrough is to have an Amazon Web Services account, the API Gateway is available as part of the AWS free tier, allowing up to 1 million API calls per month, at no charge, in your first 12 months with AWS.

We will also need an Atlas Cluster for which we have enabled the Data API – and our endpoint URL and API Key. You can learn how to get these in this [Article](https://www.mongodb.com/developer/quickstart/atlas_data_api_introduction) or this [Video](https://www.youtube.com/watch?v=46I0wZiTFi4) if you do not have them already.

A common use of Atlas with the Amazon API Gateway might be to provide a managed API to a restricted subset of data in our cluster, which is a common need for a microservice architecture. To demonstrate this, we first need to have some data available in MongoDB Atlas. This can be added by selecting the three dots next to our cluster name and choosing "Load Sample Dataset", or following instructions [here](https://docs.mongodb.com/charts/saas/tutorial/order-data/prerequisites-setup/). 
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_11_29_at_09_50_02_6532382c89.png)

## Creating an API with the Amazon API Gateway and the Atlas Data API

The instructions here are an extended variation from Amazon's own "[Getting Started with the API Gateway](https://docs.aws.amazon.com/apigateway/latest/developerguide/getting-started.html)" tutorial. I do not presume to teach you how best to use Amazon's API Gateway as Amazon itself has  many fine resources for this, what we will do here is use it to get a basic Public API enabled that uses the Data API.

> The Data API itself is currently in an early preview with a flat security model  allowing all users who have an API key to query or update any database or collection. Future versions will have more granular security. We would not want to simply expose the current data API as a 'Public' API but we can use it on the back-end to create more restricted and specific access to our data. 
> 
We are going to create an API which allows users to GET the ten films for any given year which received the most awards - a notional "Best Films of the Year". We will restrict this API to performing only that operation and supply the year as part of the URL

We will first create the API, then analyze the code we used for it.

## Create a AWS Lambda Function to retrieve data with the Data API


1. Sign in to the Lambda console at [https://console.aws.amazon.com/lambda](https://console.aws.amazon.com/lambda).
2. Choose **Create function**.
3. For **Function name**, enter top-movies-for-year.
4. Choose **Create function**.
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_24_30_c38ba7b6cb.png)

When you see the Javascript editor that looks like this

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_25_59_2ba3959057.png)
Replace the code with the following, changing the API-KEY and APP-ID to the values for your Atlas cluster. Save and click **Deploy**  (In a production application you might look to store these in AWS Secrets manager , I have simplified by putting them in the code here).

```
const https = require('https');
 
const atlasEndpoint = "/app/APP-ID/endpoint/data/beta/action/find";
const atlasAPIKey = "API-KEY";
 
 
exports.handler = async(event) => {
 
   if (!event.queryStringParameters || !event.queryStringParameters.year) {
      return { statusCode: 400, body: 'Year not specified' };
   }
 
   //Year is a number but the argument is a string so we need to convert as MongoDB is typed
 
 
   let year = parseInt(event.queryStringParameters.year, 10);
   console.log(`Year = ${year}`)
   if (Number.isNaN(year)) { return { statusCode: 400, body: 'Year incorrectly specified' }; }
 
 
   const payload = JSON.stringify({
      dataSource: "Cluster0",
      database: "sample_mflix",
      collection: "movies",
      filter: { year },
      projection: { _id: 0, title: 1, awards: "$awards.wins" },
      sort: { "awards.wins": -1 },
      limit: 10
   });
 
 
   const options = {
      hostname: 'data.mongodb-api.com',
      port: 443,
      path: atlasEndpoint,
      method: 'POST',
      headers: {
         'Content-Type': 'application/json',
         'Content-Length': payload.length,
         'api-key': atlasAPIKey
      }
   };
 
   let results = '';
 
   const response = await new Promise((resolve, reject) => {
      const req = https.request(options, res => {
         res.on('data', d => {
            results += d;
         });
         res.on('end', () => {
            console.log(`end() status code = ${res.statusCode}`);
            if (res.statusCode == 200) {
               let resultsObj = JSON.parse(results)
               resolve({ statusCode: 200, body: JSON.stringify(resultsObj.documents, null, 4) });
            }
            else {
               reject({ statusCode: 500, body: 'Your request could not be completed, Sorry' }); //Backend Problem like 404 or wrong API key
            }
         });
      });
      //Do not give the user clues about backend issues for security reasons
      req.on('error', error => {
         reject({ statusCode: 500, body: 'Your request could not be completed, Sorry' }); //Issue like host unavailable
      });
 
      req.write(payload);
      req.end();
   });
   return response;
 
};

```

Alternatively, if you are familiar with working with packages and Lambda, you could upload an HTTP package like Axios to Lambda as a zipfile, allowing you to  use the following simplified code.

```

const axios = require('axios');

const atlasEndpoint = "https://data.mongodb-api.com/app/APP-ID/endpoint/data/beta/action/find";
const atlasAPIKey = "API-KEY";


exports.handler = async(event) => {

   if (!event.queryStringParameters || !event.queryStringParameters.year) {
      return { statusCode: 400, body: 'Year not specified' };
   }

   //Year is a number but the argument is a string so we need to convert as MongoDB is typed


   let year = parseInt(event.queryStringParameters.year, 10);
   console.log(`Year = ${year}`)
   if (Number.isNaN(year)) { return { statusCode: 400, body: 'Year incorrectly specified' }; }


   const payload = {
      dataSource: "Cluster0",
      database: "sample_mflix",
      collection: "movies",
      filter: { year },
      projection: { _id: 0, title: 1, awards: "$awards.wins" },
      sort: { "awards.wins": -1 },
      limit: 10
   };

   try {
      const response = await axios.post(atlasEndpoint, payload, { headers: { 'api-key': atlasAPIKey } });
      return response.data.documents;
   }
   catch (e) {
      return { statusCode: 500, body: 'Unable to service request' }
   }
};
```

## Create an HTTP endpoint for our custom API function

We now need to route an HTTP endpoint to our Lambda function using the HTTP API. 

The HTTP API provides an HTTP endpoint for your Lambda function. API Gateway routes requests to your Lambda function, and then returns the function's response to clients.

1. Go to the API Gateway console at [https://console.aws.amazon.com/apigateway.](https://console.aws.amazon.com/apigateway)
2. Do one of the following:
  To create your first API, for HTTP API, choose **Build**.
   If you've created an API before, choose **Create API**, and then choose **Build** for HTTP API.
3. For Integrations, choose **Add integration**.
4. Choose **Lambda**.
5. For **Lambda function**, enter top-movies-for-year.
6. For **API name**, enter movie-api.
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_38_55_cfcd82c333.png)
8. Choose **Next**.

8. Review the route that API Gateway creates for you, and then choose **Next**.
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_42_20_23bd84911f.png)
9. Review the stage that API Gateway creates for you, and then choose **Next**.
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_43_43_ebab329b15.png)
10. Choose **Create**.

Now you've created an HTTP API with a Lambda integration and the Atlas Data API that's ready to receive requests from clients.

## Test your API

You should now be looking at API Gateway details that look like this, if not you can get to it by going to[https://console.aws.amazon.com/apigateway](https://console.aws.amazon.com/apigateway)and clicking on **movie-api**

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2021_12_07_at_15_46_17_06301a223e.png)

Take a note of the **Invoke URL**, this is the base URL for your API

Now, in a new browser tab,  browse to  `<Invoke URL>/top-movies-for-year?year=2001` . Changing `<Invoke URL> `to the Invoke URL shown in AWS. You should see the results of your API call - JSON listing the top 10 "Best" films of 2001.

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/unnamed_33c3be7d0f.png)
## Reviewing our Function.

We start by importing the Standard node.js https library - the Data API needs no special libraries to call it. We also define our API Key and the path to our find endpoint, You get both of these from the Data API tab in Atlas.

```
const https = require('https');
 
const atlasEndpoint = "/app/data-amzuu/endpoint/data/beta/action/find";
const atlasAPIKey = "YOUR-API-KEY";
```

Now we check that the API call included a parameter for year and that it's a number - we need to convert it to a number as in MongoDB, "2001" and 2001  are different values, and searching for one will not find the other. The collection uses a number for the movie release year.

```
exports.handler = async (event) => {
 
   if (!event.queryStringParameters || !event.queryStringParameters.year) {
       return { statusCode: 400, body: 'Year not specified' };
   }
   //Year is a number but the argument is a string so we need to convert as MongoDB is typed
   let year = parseInt(event.queryStringParameters.year, 10);
   console.log(`Year = ${year}`)
   if (Number.isNaN(year)) { return { statusCode: 400, body: 'Year incorrectly specified' }; }
 
 
   const payload = JSON.stringify({
       dataSource: "Cluster0", database: "sample_mflix", collection: "movies",
       filter: { year }, projection: { _id: 0, title: 1, awards: "$awards.wins" }, sort: { "awards.wins": -1 }, limit: 10
   });

```

Then we construct our payload - the parameters for the Atlas API Call, we are querying for year = year, projecting just the title and the number of awards, sorting by the numbers of awards descending and limiting to 10.
 
```
  const payload = JSON.stringify({
       dataSource: "Cluster0", database: "sample_mflix", collection: "movies",
       filter: { year }, projection: { _id: 0, title: 1, awards: "$awards.wins" },  
       sort: { "awards.wins": -1 }, limit: 10
   });

```

We then construct the options for the HTTPS POST request to the Data API - here we pass the Data API API-KEY as a header.

```
  const options = {
       hostname: 'data.mongodb-api.com',
       port: 443,
       path: atlasEndpoint,
       method: 'POST',
       headers: {
           'Content-Type': 'application/json',
           'Content-Length': payload.length,
           'api-key': atlasAPIKey
       }
   };


```

Finally we use some fairly standard code to call the API and handle errors. We can get Request errors - such as being unable to contact the server - or Response errors where we get any Response code other than 200 OK - In both cases we return a  500 Internal error from our simplified API to not leak any details of the internals to a potential hacker.
 
```
   let results = '';
 
   const response = await new Promise((resolve, reject) => {
       const req = https.request(options, res => {
           res.on('data', d => {
               results += d;
           });
           res.on('end', () => {
               console.log(`end() status code = ${res.statusCode}`);
               if (res.statusCode == 200) {
                   let resultsObj = JSON.parse(results)
                   resolve({ statusCode: 200, body: JSON.stringify(resultsObj.documents, null, 4) });
               } else {
                   reject({ statusCode: 500, body: 'Your request could not be completed, Sorry' }); //Backend Problem like 404 or wrong API key
               }
           });
       });
       //Do not give the user clues about backend issues for security reasons
       req.on('error', error => {
           reject({ statusCode: 500, body: 'Your request could not be completed, Sorry' }); //Issue like host unavailable
       });
 
       req.write(payload);
       req.end();
   });
   return response;
 
};
```

Our Axios verison is just the same functionality as above but simplified by the use of a library.
## Conclusion

As we can see, calling the Atlas Data API from AWS Lambda function is incredibly simple, especially if making use of a library like Axios. The Data API is also stateless, so there are no concerns about connection setup times or maintaining long lived connections as there would be using a  Driver. 

AWS

Featured

How to Build Serverless Applications with SST and MongoDB Atlas

Building a Multi-Environment Continuous Delivery Pipeline for MongoDB Atlas

How to Query from Multiple MongoDB Databases Using MongoDB Atlas Data Lake

All AWS Content