Babu Srinivasan

Senior Partner Solutions Architect

In this tutorial, you'll find a tangible showcase of how AWS Glue, Amazon Kinesis, and MongoDB Atlas seamlessly integrate, creating a streamlined data streaming solution alongside extract, transform, and load (ETL) capabilities. This repository also harnesses the power of AWS CDK to automate deployment across diverse environments, enhancing the efficiency of the entire process.

To follow along with this tutorial, you should have intermediate proficiency with AWS and MongoDB services.

## Architecture diagram

![architecture diagram][1]

In the architecture described above, various streams of data, such as Order and Customer, are retrieved via the Amazon Kinesis stream. Subsequently, AWS Glue Studio is utilized to enrich the data. The enriched data is backed up in an S3 bucket, while the consolidated stream is stored in MongoDB Atlas and made accessible as data APIs for downstream systems.

## Implementation steps

### Prerequisites

 - [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) installed and configured
 - [NVM/NPM](https://nvm.sh/) installed and configured
 - [AWS CDK](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) installed and configured
 - [MongoDB Atlas account](https://www.mongodb.com/docs/atlas/tutorial/create-atlas-account/), with the [Organization](https://www.mongodb.com/docs/atlas/government/tutorial/create-organization/#create-an-organization) set up
 - [Python packages](https://www.python.org/downloads/)
     - Python3 - `yum install -y python3`
     - Python Pip - `yum install -y python-pip`
     - Virtualenv - `pip3 install virtualenv`

>This repo is developed taking us-east-1 as the default region. Please update the scripts to your specific region (if required). This repo will create a MongoDB Atlas project and a free-tier database cluster automatically. No need to create a database cluster manually. This repo is created for a demo purpose and IP access is not restricted (0.0.0.0/0). Ensure you strengthen the security by updating the relevant IP address (if required).

### Setting up the environment

#### Get the application code

`git clone https://github.com/mongodb-partners/Stream_Data_into_MongoDB_AWS_Glue
cd kinesis-glue-aws-cdk`

#### Prepare the dev environment to run AWS CDK

a. Set up the AWS Environment variable AWS Access Key ID, AWS Secret Access Key, and optionally, the AWS Session Token.

```
export AWS_ACCESS_KEY_ID = <"your AWS access key">
   export AWS_SECRET_ACCESS_KEY =<"your AWS secret access key">
   export AWS_SESSION_TOKEN = <"your AWS session token">
```
b. We will use CDK to make our deployments easier. 

You should have npm pre-installed.
If you don’t have CDK installed:
`npm install -g aws-cdk`

Make sure you’re in the root directory.
`python3 -m venv .venv`
`source .venv/bin/activate`
`pip3 install -r requirements.txt`

> For development setup, use requirements-dev.txt.

c. Bootstrap the application with the AWS account.

`cdk bootstrap`

d. Set the ORG_ID as an environment variable in the .env file. All other parameters are set to default in global_args.py in the kinesis-glue-aws-cdk folder. MONGODB_USER and MONGODB_PASSWORD parameters are set directly in mongodb_atlas_stack.py and glue_job_stack.py

The below screenshot shows the location to get the Organization ID from MongoDB Atlas.

![Organization ID in MongoDB Atlas][2]

Please note that using "0.0.0.0/0" as an IP_ADDRESS, we are allowing access to the database from anywhere. This might be suitable for development or testing purposes but is highly discouraged for production environments because it exposes the database to potential attacks from unauthorized sources.

e. List the CDKs:

`cdk ls`

You should see an output of the available stacks:

    `aws-etl-mongo-atlas-stack`
    `aws-etl-kinesis-stream-stack`
    `aws-etl-bucket-stack`
   ` aws-etl-glue-job-stack`

### Deploying the application

Let’s walk through each of the stacks:

**Stack for MongoDB Atlas: aws-etl-mongo-atlas-stack**
This stack will create a MongoDB Atlas project and a free-tier database cluster with user and network permission (open).

#### Prerequisites

a. Create an AWS role with its trust relationship as a CloudFormation service.

Use the [template](https://github.com/mongodb/mongodbatlas-cloudformation-resources/blob/master/examples/execution-role.yaml) to create a new [CloudFormation stack](https://console.aws.amazon.com/cloudformation/home#/stacks/create) to create the execution role.

![Creating a new CloudFormation stsack][3]

b.The following Public Extension in the CloudFormation Registry should be activated with the Role created in the earlier step. After logging into the AWS console, use this link to register extensions on CloudFormation.

```
          MongoDB::Atlas::Project,
          MongoDB::Atlas::DatabaseUser,
          MongoDB::Atlas::Cluster,
          MongoDB::Atlas::ProjectIpAccessList
```
Pass the ARN of the role from the earlier step as input to activate the MongoDB resource in Public Extension.

MongoDB Resource Activation in Public Extension:

![Activation of Registry Public Extensions][4]

The above screenshot shows the activation of the Registry Public Extensions for the MongoDB::Atlas::Cluster.

Alternatively, you can activate the above public extension through AWS CLI.

Command to list the MongoDB Public Extensions. Note down the arns for the above four public extensions.

```
aws cloudformation list-types \
  --visibility PUBLIC \
  --filters "Category=THIRD_PARTY,TypeNamePrefix=Mongodb"
```

Command to activate the Public Extension. Use this command to activate all four public extensions mentioned in the previous steps.

```
aws cloudformation activate-type --region us-east-1 --public-type-arn "<arn for the public extension noted down in the previous step>" --execution-role-arn "<arn of the role created in step a>"
```

c. Log into the MongoDB console and note down the organization ID. Ensure the Organization ID is updated in the global_args.py

![Cluster organization settings][5]

The above screenshot shows the MongoDB Cluster Organization settings.

d. Create an API Key in an organization with Organization Owner access. Note down the API credentials.

![Edit API key][6]

The above screenshot shows the access managers for the API Key created in the MongoDB Atlas cluster.

Restrict the access to the Organization API with the API Access list. We provided an open access 0.0.0.0/1 for demo purposes only. We strongly discourage the use of this in any production environment or equivalent.

e. A profile should be created in the AWS Secrets Manager containing the MongoDB Atlas Programmatic API Key.

Use the [template](https://github.com/mongodb/mongodbatlas-cloudformation-resources/blob/master/examples/profile-secret.yaml) to create a new CloudFormation [stack](https://console.aws.amazon.com/cloudformation/home#/stacks/create) for the default profile that all resources will attempt to use unless a different override is specified.

#### Profile secret stack

![AWS CloudFormation stack][7]

The above screenshot shows the parameters for the AWS CloudFormation stack.

Initiate the deployment with the following command:

```bash
  cdk deploy aws-etl-mongo-atlas-stack
  ```
After successfully deploying the stack, validate the Outputs section of the stack and MongoDB Atlas cluster. You will find the stdUrl and stdSrvUrl for the connection string.

#### Stack:

![Output of CloudFormation stack][8]

The above screenshot shows the output of the CloudFormation stack.

#### MongoDB Atlas cluster:

![Creation of a MongoDB Atlas cluster][9]

The above screenshot shows the successful creation of the MongoDB Atlas cluster.

### Stack for creating the Kinesis stream: aws-etl-kinesis-stream-stack

This stack will create two Kinesis data streams. Each producer runs an ingesting stream of events for different customers with their orders.

Initiate the deployment with the following command:

  ```bash
  cdk deploy aws-etl-kinesis-stream-stack
  ```
After successfully deploying the stack, Check the Outputs section of the stack. You will find the CustomerOrderKinesisDataStream Kinesis function.

#### Stack:

![Output of the CloudFormation stack][10]

The above screenshot shows the output of the CloudFormation stack for Kinesis streams.

#### Amazon Kinesis data stream:

![Kinesis stream][11]

The above screenshot shows the Kinesis stream created.

### Stack for creating the S3 bucket: aws-etl-bucket-stack

This stack will create an S3 bucket that will be used by AWS Glue jobs to persist the incoming customer and order details.

  ```bash
  cdk deploy aws-etl-bucket-stack
  ```
After successfully deploying the stack, check the Outputs section of the stack. You will find the 
S3SourceBucket resource.

#### Stack:

![Output of the CloudFormation stack][12]

The above screenshot shows the Output of the CloudFormation stack.

AWS S3 Bucket:

![S3 buckets created][13]

The above screenshot shows the S3 buckets created.

### Stack for creating the AWS Glue job and parameters: aws-etl-glue-job-stack

This stack will create two AWS Glue jobs: one job for the customer and another for the order. The code is in the location glue_job_stack/glue_job_scripts/customer_kinesis_streams_s3.py and glue_job_stack/glue_job_scripts/order_kinesis_streams_s3.py.

  ```bash
  cdk deploy aws-etl-glue-job-stack
  ```
#### Stack:

![Output of CloudFormation stack][14]

The above screenshot shows the output of the CloudFormation stack.

#### AWS Glue job:

![AWS Glue Studio][15]

The above screenshot shows the AWS Glue Studio job to set up the job.

#### Note

The MongoDB URL of the newly created cluster and other parameters will be passed to the AWS Glue job programmatically. Update these parameters to your values (if required).

"Spark UI logs path" and "Temporary path" details will be maintained in the same bucket location with folder name /sparkHistoryLogs and /temporary.

#### Spark UI logs path:

  `s3://<S3_BUCKET_NAME>/sparkHistoryLogs` 

#### Temporary path:

  `s3://<S3_BUCKET_NAME>/temporary/`

#### Screenshot of the AWS Glue parameters

![AWS Glue parameters][16]

The above screenshot shows the AWS Glue parameters.

Once you are ready with all stacks, start the producers for the customer and order. The code is in this location, to ingest data into a kinesis data stream and also start the Glue job for both.

  `producer/customer.py` and `producer/order.py`

#### Sample record in MongoDB Atlas:

```
{
  "_id": "1",
  "country_id": "1",
  "customer_name": "NICK",
  "email_id": "nick@gmail.com",
  "orders": [
    {
      "order_id": "8",
      "product_name": "Artisanal Cheese Selection",
      "quantity": "8",
      "price": "29.17"
    }
  ]
}
```

## Clean up

Use `cdk destroy` to clean up all the AWS CDK resources. 

## Troubleshooting

Refer to [GitHub](https://github.com/mongodb/mongodbatlas-cloudformation-resources/tree/master#troubleshooting) to resolve some common issues encountered when using AWS CloudFormation/CDK with MongoDB Atlas Resources.

## Useful commands

`cdk ls` lists all stacks in the app.
`cdk synth` emits the synthesized CloudFormation template.
`cdk deploy` deploys this stack to your default AWS account/region.
`cdk diff` compares the deployed stack with the current state.
`cdk docs` opens CDK documentation.


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt95d66e4812fd56ed/661e9e36e9e603e1aa392ef0/architecture-diagram.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt401e52a6c6ca2f6f/661ea008f5bcd1bf540c99bd/organization-settings.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltffa00dde10848a90/661ea1fe190a257fcfbc5b4e/cloudformation-stack.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltcc9e58d51f80206d/661ea245ad926e2701a4985b/registry-public-extensions.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc5a089f4695e6897/661ea2c60d5626cbb29ccdfb/cluster-organization-settings.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1a0bed34c4fbaa4f/661ea2ed243a4fa958838c90/edit-api-key.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt697616247b2b7ae9/661ea35cdf48e744da7ea2bd/aws-cloud-formation-stack.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltcba40f2c7e837849/661ea396f19ed856a2255c19/output-cloudformation-stack.png
  [9]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1beb900395ae5533/661ea3d6a7375b6a462d7ca2/creation-mongodb-atlas-cluster.png
  [10]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt298896d4b3f0ecbb/661ea427e9e6030914392f35/output-cloudformation.png
  [11]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt880cc957dbb069b9/661ea458a7375b89a52d7cb8/kinesis-stream.png
  [12]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4fd8a062813bee88/661ea4c0a3e622865f4be23e/output.png
  [13]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt355c545ee15e9222/661ea50645b6a80f09390845/s3-buckets-created.png
  [14]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb4c1cdd506b90172/661ea54ba7375b40172d7cc5/output-2.png
  [15]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta91c3d8a8bd9d857/661ea57a243a4fe9d4838caa/aws-glue-studio.png
  [16]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1993ccc1ca70cd57/661ea5bc061bb15fd5421300/aws-glue-parameters.png

In this tutorial, find a tangible showcase of how AWS Glue, Amazon Kinesis, and MongoDB Atlas seamlessly integrate.

Stream Data Into MongoDB Atlas Using AWS Glue

Abilash Ravikumar

Product Manager at Datadog

[Datadog](https://www.datadoghq.com/) is a well-known monitoring and security platform for cloud applications. Datadog’s software-as-a-service (SaaS) platform integrates and automates infrastructure monitoring, application performance monitoring, and [log management](https://docs.datadoghq.com/logs/log_collection/) to provide unified, real-time observability of a customer’s entire technology stack.

MongoDB Atlas on Amazon Web Services (AWS) already supports [easy integration with Datadog](https://www.mongodb.com/docs/atlas/tutorial/datadog-integration/) for alerts and events right within the Atlas UI (select the three vertical dots → Integration → Datadog). With the Log Forwarding feature, it's now possible to send Atlas Application Services logs to Datadog. This blog outlines the configuration steps necessary as well as strategies for customizing the view to suit the need.

![Datadog Configuration from Atlas ](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/datadog_configuration_atlas_00347e84a6.png)

[**Atlas Application Services**](https://www.mongodb.com/docs/atlas/app-services/) (formerly MongoDB Realm) is a set of enhanced services that complement the Atlas database to simplify the development of backend applications. App Services-based apps can react to changes in your MongoDB Atlas data, connect that data to other systems, and scale to meet demand without the need to manage the associated server infrastructure.

App Services provides user authentication and management, schema validation and data access rules, event-driven serverless functions, secure client-side queries with HTTPS Endpoints, and best of all, synchronization of data across devices with the Realm Mobile SDK.

With App Services and Datadog, you can simplify the end-to-end development and monitoring of your application. **Atlas App Services** specifically enables the forwarding of logs to Datadog via a serverless function that can also give more fine-grained control over how these logs appear in Datadog, via customizing the associated tags.

## Atlas setup

We assume that you already have an Atlas account. If not, you can sign up for a [free account](https://www.mongodb.com/realm/register) on MongoDB or the [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-pp445qepfdy34?trk=38088f03-4928-4b00-9a39-86781df6ab9b&sc_channel=el). Once you have an Atlas account, if you haven't had a chance to try App Services with Atlas, you can follow one of our [tutorials](https://www.mongodb.com/docs/atlas/app-services/tutorials/) to get a running application working quickly.

To initiate custom log forwarding, follow the instructions for App Services to configure [log forwarding](https://www.mongodb.com/docs/atlas/app-services/logs/forward-logs/). Specifically, choose the “To Function” option:

![MongoDB Atlas Log configuration](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/mongodb_atlas_log_configuration_1f15e0de05.png)

Within Atlas App Services, we can create a custom function that provides the mapping and ingesting logs into Datadog. Please note the intake endpoint URL from Datadog first, which is [documented by Datadog](https://docs.datadoghq.com/logs/log_collection/?tab=host).

Here’s a sample function that provides that basic capability:

```
exports = async function(logs) {
  // `logs` is an array of 1-100 log objects
  // Use an API or library to send the logs to another service.
  await context.http.post({
    url: "https://http-intake.logs.datadoghq.com/api/v2/logs",
    headers: {
      "DD-API-KEY": ["XXXXXX"],
      "DD-APPLICATION-KEY": ["XXXXX"], 
      "Content-Type": ["application/json"]
    },
    body: logs.map(x => {return {
      "ddsource": "mongodb.atlas.app.services",
      "ddtags": "env:test,user:igor",
      "hostname": "RealmApp04",
      "service": "MyRealmService04",
      "message" : JSON.stringify(x)
    }}),
    encodeBodyAsJSON: true
  });
}
```

One of the capabilities of the snippet above is that it allows you to modify the function to supply your Datadog [API and application keys](https://docs.datadoghq.com/account_management/api-app-keys/). This provides the capability to customize the experience and provide the appropriate context for better observability. You can change [ddtags](https://docs.datadoghq.com/getting_started/tagging/), the hostname, and service parameters to reflect your organization, team, environment, or application structure. These parameters will appear as facets helping with filtering the logs.

Note: Datadog supports log ingestion pipelines that allow it to better parse logs. In order for the MongoDB log pipeline to work, your *ddsource* must be set to `mongodb.atlas.app.services`.

![Select the MongoDB as a source in Datadog](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/select_mongodo_source_datadog_c9abd3a85a.png)

## Viewing the logs in Datadog

Once log forwarding is configured, your Atlas App Services logs will appear in the Datadog Logs module.

![Datadog Logs](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/datadog_logs_73ed6659cd.png)

You can click on an individual log entry to see the detailed view:

![Datadog Event Attributes](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/datadog_event_attributes_d4d8228ed3.png)

## Conclusion

In this blog, we showed how to configure log forwarding for Atlas App Services logs. If you would like to try configuring log forwarding yourself, sign up for a [14-day free trial of Datadog](https://www.datadoghq.com/free-datadog-trial/) if you don’t already have an account. To try Atlas App Services in AWS Marketplace, sign up for a [free account](https://aws.amazon.com/marketplace/pp/prodview-pp445qepfdy34).

With the Log Forwarding feature, it's now possible to send Atlas Application Services logs to Datadog. This blog outlines the configuration steps necessary as well as strategies for customizing the view to suit the need.

Babu Srinivasan

Content by Babu Srinivasan

Stream Data Into MongoDB Atlas Using AWS Glue

Integrate Atlas Application Services Logs into Datadog on AWS