Venkatesh Shanbhag

Partner Solution Architect

Stanimira Vlaeva

Developer Advocate

On 1st October 2022, MongoDB and Google announced a set of open source [Dataflow templates](https://cloud.google.com/blog/products/data-analytics/mongodb-atlas-and-bigquery-dataflow-templates) for moving data between MongoDB and BigQuery to run analyses on BigQuery using BQML and to bring back inferences to MongoDB. Three templates were introduced as part of this release, including the MongoDB to BigQuery CDC (change data capture) template. 

![MongoDB change stream](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/change_stream_27de6ed2ac.png)

This template requires users to run the change stream on MongoDB, which will monitor inserts and updates on the collection. These changes will be captured and pushed to a [Pub/Sub topic](https://cloud.google.com/pubsub/docs/overview). The CDC template will create a job to read the data from the topic and get the changes, apply the transformation, and write the changes to BigQuery. The transformations will vary based on the user input while running the Dataflow job.

Alternatively, you can use a native Pub/Sub capability to set up a data pipeline between your MongoDB cluster and BigQuery. The Pub/Sub BigQuery subscription writes messages to an existing BigQuery table as they are received. Without the BigQuery subscription type, you need a pull or push subscription and a subscriber (such as Dataflow) that reads messages and writes them to BigQuery. 

This article explains how to set up the BigQuery subscription to process data read from a MongoDB change stream. As a prerequisite, you’ll need a MongoDB Atlas cluster.

> To set up a free tier cluster, you can register to MongoDB either from [Google Cloud Marketplace](https://console.cloud.google.com/marketplace/product/mongodb/mdb-atlas-self-service) or from the [registration page](https://www.mongodb.com/cloud/atlas/register). Follow the steps in the MongoDB documentation to configure the database user and network settings for your cluster. 

On Google Cloud, we will create a Pub/Sub topic, a BigQuery dataset, and a table before creating the BigQuery subscription.

## Create a BigQuery dataset


We’ll start by creating a new dataset for BigQuery in [Google Cloud console](https://console.cloud.google.com/bigquery).

Then, add a new table in your dataset. Define it with a name of your choice and the following schema:


| Field name | Type |
| --- | --- |
| id | STRING |
| source_data | STRING |
| Timestamp | STRING |

![Create a new dataset and table in BigQuery](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/source_and_destination_fb83305dc6.png)

## Configure Google Cloud Pub/Sub

Next, we’ll configure a Pub/Sub schema and topic to ingest the messages from our MongoDB change stream. Then, we’ll create a subscription to write the received messages to the BigQuery table we just created.

For this section, we’ll use the Google Cloud Pub/Sub API. Before proceeding, make sure you have enabled the API for your project.

### Define a Pub/Sub schema 

From the Cloud Pub/Sub UI, Navigate to _[Create Schema](https://console.cloud.google.com/cloudpubsub/schema/create)_.

Provide an appropriate identifier, such as “mdb-to-bq-schema,” to your schema. Then, select “Avro” for the type. Finally, add the following definition to match the fields from your BigQuery table:

```json
{
 "type" : "record",
 "name" : "Avro",
 "fields" : [
   {
     "name" : "id",
     "type" : "string"
   },
   {
     "name" : "source_data",
     "type" : "string"
   },
   {
     "name" : "Timestamp",
     "type" : "string"
   }
 ]
}
```

![Create a Cloud Pub/Sub schema](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_schema_7db8b97369.png)

### Create a Pub/Sub topic

From the sidebar, navigate to “Topics” and click on Create a topic. 

Give your topic an identifier, such as “MongoDBCDC.” Enable the Use a schema field and select the schema that you just created. Leave the rest of the parameters to default and click on _Create Topic_.

![Create a Cloud Pub/Sub topic](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_topic_4948d0ebc9.png)

### Subscribe to topic and write to BigQuery

From inside the topic, click on _Create new subscription_. Configure your subscription in the following way:

- Provide a subscription ID — for example, “mdb-cdc.”
- Define the Delivery type to _Write to BigQuery_.
- Select your BigQuery dataset from the dropdown.
- Provide the name of the table you created in the BigQuery dataset.
- Enable _Use topic schema_.

You need to have a `bigquery.dataEditor` role on your service account to create a Pub/Sub BigQuery subscription. To grant access using the [`bq` command line tool](https://cloud.google.com/bigquery/docs/bq-command-line-tool), run the following command:

```sh
bq add-iam-policy-binding \
    --member="serviceAccount:service<project-number>@gcp-sa-pubsub.iam.gserviceaccount.com" \
    --role=roles/bigquery.dataEditor \
    -t "<dataset>.<table>"
```


Keep the other fields as default and click on _Create subscription_.

![Create a Cloud Pub/Sub subscription](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_subscription_f693420e69.png)

## Set up a change stream on a MongoDB cluster

Finally, we’ll set up a change stream that listens for new documents inserted in our MongoDB cluster. 

We’ll use Node.js but you can adapt the code to a programming language of your choice. Check out the [Google Cloud documentation](https://cloud.google.com/pubsub/docs/publisher#using-schema) for more Pub/Sub examples using a variety of languages. You can find the source code of this example in the dedicated [GitHub repository](https://github.com/mongodb-developer/MongoDB-Change-Stream-to-Google-Cloud-PubSub).

First, set up a new Node.js project and install the following dependencies.

```sh
npm install mongodb @google-cloud/pubsub avro-js
```

Then, add an Avro schema, matching the one we created in Google Cloud Pub/Sub:

**./document-message.avsc**
```json
{
 "type": "record",
 "name": "DocumentMessage",
 "fields": [
   {
     "name": "id",
     "type": "string"
   },
   {
     "name": "source_data",
     "type": "string"
   },
   {
     "name": "Timestamp",
     "type": "string"
   }
 ]
}
```

Then create a new JavaScript module — `index.mjs`. Start by importing the required libraries and setting up your MongoDB connection string and your Pub/Sub topic name. If you don’t already have a MongoDB cluster, you can create one for free in [MongoDB Atlas](https://www.mongodb.com/cloud/atlas/register).

**./index.mjs**
```js
import { MongoClient } from 'mongodb';
import { PubSub } from '@google-cloud/pubsub';
import avro from 'avro-js';
import fs from 'fs';
 
const MONGODB_URI = '<mongodb-connection-string>';
const PUB_SUB_TOPIC = 'projects/<project-name>/topics/<topic-name>';
```

After this, we can connect to our MongoDB instance and set up a [change stream event listener](https://www.mongodb.com/docs/manual/reference/method/db.collection.watch/). Using an aggregation pipeline, we’ll watch only for “insert” events on the specified collection. We’ll also define a 60-second timeout before closing the change stream.

**./index.mjs**
```js
let mongodbClient;
try {
   mongodbClient = new MongoClient(MONGODB_URI);
   await monitorCollectionForInserts(mongodbClient, 'my-database', 'my-collection');
} finally {
   mongodbClient.close();
}
 
async function monitorCollectionForInserts(client, databaseName, collectionName, timeInMs) {
   const collection = client.db(databaseName).collection(collectionName);
   // An aggregation pipeline that matches on new documents in the collection.
   const pipeline = [ { $match: { operationType: 'insert' } } ];
   const changeStream = collection.watch(pipeline);
 
   changeStream.on('change', event => {
       const document = event.fullDocument;
       publishDocumentAsMessage(document, PUB_SUB_TOPIC);
   });
 
   await closeChangeStream(timeInMs, changeStream);
}
 
function closeChangeStream(timeInMs = 60000, changeStream) {
   return new Promise((resolve) => {
       setTimeout(() => {
           console.log('Closing the change stream');
           changeStream.close();
           resolve();
       }, timeInMs)
   })
};
```


Finally, we’ll define the `publishDocumentAsMessage()` function that will:

1. Transform every MongoDB document received through the change stream.
1. Convert it to the data buffer following the Avro schema.
1. Publish it to the Pub/Sub topic in Google Cloud.

```js
async function publishDocumentAsMessage(document, topicName) {
   const pubSubClient = new PubSub();
   const topic = pubSubClient.topic(topicName);
 
   const definition = fs.readFileSync('./document-message.avsc').toString();
   const type = avro.parse(definition);
 
   const message = {
       id: document?._id?.toString(),
       source_data: JSON.stringify(document),
       Timestamp: new Date().toISOString(),
   };
 
   const dataBuffer = Buffer.from(type.toString(message));
   try {
       const messageId = await topic.publishMessage({ data: dataBuffer });
       console.log(`Avro record ${messageId} published.`);
   } catch(error) {
       console.error(error);
   }
}
```

Run the file to start the change stream listener:

```sh
node ./index.mjs
```

Insert a new document in your MongoDB collection to watch it go through the data pipeline and appear in your BigQuery table!

## Summary

There are multiple ways to load the change stream data from MongoDB to BigQuery and we have shown how to use the BigQuery subscription on Pub/Sub. The change streams from MongoDB are monitored, captured, and later written to a Pub/Sub topic using Java libraries.

The data is then written to BigQuery using BigQuery subscription. The datatype for the BigQuery table is set using Pub/Sub schema. Thus, the change stream data can be captured and written to BigQuery using the BigQuery subscription capability of Pub/Sub.

## Further reading

1. A data pipeline for [MongoDB Atlas and BigQuery](https://cloud.google.com/blog/products/data-analytics/mongodb-atlas-and-bigquery-dataflow-templates) using Dataflow.
1. [Setup your first MongoDB cluster](https://www.mongodb.com/docs/atlas/billing/gcp-self-serve-marketplace/) using Google Marketplace.
1. Run analytics using BigQuery using [BigQuery ML](https://cloud.google.com/bigquery-ml/docs).
1. [How to publish](https://cloud.google.com/pubsub/docs/publisher#using-schema) a message to a topic with schema.

Learn how to set up a data pipeline from your MongoDB database to BigQuery using change streams and Google Cloud Pub/Sub.

Create a Data Pipeline for MongoDB Change Stream Using Pub/Sub BigQuery Subscription

Partner

Abirami Sukumaran

Developer Advocate, Google

## Plea and the Pledge: Truly Serverless

As modern application developers, we’re juggling many priorities: performance, flexibility, usability, security, reliability, and maintainability. On top of that, we’re handling dependencies, configuration, and deployment of multiple components in multiple environments and sometimes multiple repositories as well. And then we have to keep things secure and simple. Ah, the nightmare!

This is the reason we love serverless computing. Serverless allows developers to focus on the thing they like to do the most—development—and leave the rest of the attributes, including infrastructure and maintenance, to the platform offerings. 

In this read, we’re going to see how Cloud Run and MongoDB come together to enable a completely serverless MEAN stack application development experience. We'll learn how to build a serverless MEAN application with [Cloud Run](https://cloud.google.com/run/) and [MongoDB Atlas](https://www.mongodb.com/atlas), the multi-cloud developer data platform by MongoDB.

### Containerized deployments with Cloud Run

All serverless platform offer exciting capabilities: 

* Event-driven function (not a hard requirement though)
* No-infrastructure maintenance
* Usage-based pricing
* Auto-scaling capabilities

Cloud Run stands out of the league by enabling us to: 

* Package code in multiple stateless containers that are request-aware and invoke it via HTTP requests
* Only be charged for the exact resources you use
* Support any programming language or any operating system library of your choice, or any binary

Check this link for more features in full [context](https://cloud.google.com/run/?utm_source=google&utm_medium=cpc&utm_campaign=japac-IN-all-en-dr-skws-all-pkws-trial-b-dr-1009882&utm_content=text-ad-none-none-DEV_c-CRE_529584319423-ADGP_Hybrid%20%7C%20SKWS%20-%20PHR%20%7C%20Txt%20~%20Compute%20~%20Cloud%20Run_Global%20gap%20analysis-KWID_43700065767178853-aud-970366092687%3Akwd-677335471899&userloc_1007768-network_g&utm_term=KW_cloud%20run&gclid=Cj0KCQjw1tGUBhDXARIsAIJx01lcrrpCE3MLqe4k3MiQdd8ibvGmXduL9kLXELPjodjSu7Gipc5ODYwaAlBkEALw_wcB&gclsrc=aw.ds).

However, many serverless models overlook the fact that traditional databases are not managed. You need to manually provision infrastructure (vertical scaling) or add more servers (horizontal scaling) to scale the database. This introduces a bottleneck in your serverless architecture and can lead to performance issues.

### Deploy a serverless database with MongoDB Atlas

MongoDB launched serverless instances, a new fully managed, serverless database deployment in Atlas to solve this problem. With serverless instances you never have to think about infrastructure — simply deploy your database and it will scale up and down seamlessly based on demand — requiring no hands-on management. And the best part, you will only be charged for the operations you run. To make our architecture truly serverless, we'll combine Cloud Run and MongoDB Atlas capabilities.

## What's the MEAN stack?

The [MEAN stack](https://www.mongodb.com/mean-stack) is a technology stack for building full-stack web applications entirely with JavaScript and JSON. The MEAN stack is composed of four main components—MongoDB, Express, Angular, and Node.js.

* **MongoDB** is responsible for data storage. 
* **Express.js** is a Node.js web application framework for building APIs.
* **Angular** is a client-side JavaScript platform.
* **Node.js** is a server-side JavaScript runtime environment. The server uses the MongoDB Node.js driver to connect to the database and retrieve and store data. 

![MEAN stack](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/mean_stack_e791a7354e.PNG)

## Steps for deploying truly serverless MEAN stack apps with Cloud Run and MongoDB

In the following sections, we’ll provision a new MongoDB serverless instance, connect a MEAN stack web application to it, and finally, deploy the application to Cloud Run.

### 1. Create the database

Before you begin, get started with [MongoDB Atlas on Google Cloud](https://console.cloud.google.com/marketplace/product/mongodb/mdb-atlas-self-service).

![MongoDB Atlas on Google Cloud subscription page](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/mongodb_atlas_on_google_cloud_e3bcc0b537.JPG)

Once you sign up, click the “Build a Database” button to create a new serverless instance. Select the following configuration:

![Create new serverless instance in MongoDB Atlas configuration page
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_new_serverless_instance_9134772726.JPG)

Once your serverless instance is provisioned, you should see it up and running.

![Database deployments page in MongoDB Atlas
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/database_deployments_page_5309d13fb4.JPG)

Click on the “Connect” button to add a connection IP address and a database user.

For this blog post, we’ll use the “Allow Access from Anywhere” setting. MongoDB Atlas comes with a set of security and access features. You can learn more about them in the [security features documentation article](https://www.mongodb.com/docs/atlas/setup-cluster-security/).

Use credentials of your choice for the database username and password. Once these steps are complete, you should see the following:

![Screen showing 'You are ready to connect' in MongoDB Atlas
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/ready_to_connect_e06352f53b.JPG)

Proceed by clicking on the “Choose a connection method” button and then selecting “Connect your application”.

![Screen showing the connection string for MongoDB Atlas
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/connection_string_mongodb_atlas_c28ea136ce.JPG)

Copy the connection string you see and replace the password with your own. We’ll use that string to connect to our database in the following sections.

### 2. Set up a Cloud Run project

First, sign in to [Cloud Console](http://console.cloud.google.com/), create a new project, or reuse an existing one.

Remember the Project Id for the project you created. Below is an image from [https://codelabs.developers.google.com/codelabs/cloud-run-hello#1](https://codelabs.developers.google.com/codelabs/cloud-run-hello#1) that shows how to create a new project in Google Cloud.

!['Create a new project' configuration page in Google Cloud. The project id is highlighted.
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_new_project_605bbeec53.JPG)

Then, enable Cloud Run API from [Cloud Shell](https://cloud.google.com/shell):

* Activate Cloud Shell from the Cloud Console. Simply click Activate Cloud Shell.

![Google Cloud Shell Editor loading page
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/google_cloud_shell_editor_18236232e8.JPG)

* Use the below command:

*gcloud services enable run.googleapis.com*

We will be using Cloud Shell and [Cloud Shell Editor](https://cloud.google.com/shell/docs/editor-overview#:~:text=Cloud%20Shell%20comes%20with%20a,and%20is%20based%20on%20Theia.) for code references. To access Cloud Shell Editor, click Open Editor from the Cloud Shell Terminal:

![Google Cloud Shell Code Editor
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/google_cloud_shell_code_editor_51ed2f16ca.JPG)

Finally, we need to clone the MEAN stack project we’ll be deploying. 

We’ll deploy an employee management web application. The REST API is built with Express and Node.js; the web interface, with Angular; and the data will be stored in the MongoDB Atlas instance we created earlier.

Clone the project repository by executing the following command in the Cloud Shell Terminal:

`git clone` [https://github.com/mongodb-developer/mean-stack-example.git](https://github.com/mongodb-developer/mean-stack-example.git)

In the following sections, we will deploy a couple of services—one for the Express REST API and one for the Angular web application. 

### 3. Deploy the Express and Node.js REST API

First, we’ll deploy a Cloud Run service for the Express REST API. 

The most important file for our deployment is the Docker configuration file. Let’s take a look at it:

**mean-stack-example/server/Dockerfile**

```
FROM node:17-slim
 
WORKDIR /usr/app
COPY ./ /usr/app
 
# Install dependencies and build the project.
RUN npm install
RUN npm run build
 
# Run the web service on container startup.
CMD ["node", "dist/server.js"]
```

The configuration sets up Node.js, and copies and builds the project. When the container starts, the command “node dist/server.js” starts the service.

To start a new Cloud Run deployment, click on the Cloud Run icon on the left sidebar:

![Select the 'Cloud Run' icon from the left sidebar
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/select_new_cloud_run_e2c48b457a.JPG)

Then, click on the Deploy to Cloud Run icon:

![Select the 'Deploy to Cloud Run' icon from the opened pane
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/select_deploy_to_cloud_run_c90b1dc8f6.JPG)

Fill in the service configuration as follows:

* Service name: node-express-api
* Deployment platform: Cloud Run (fully managed)
* Region: Select a region close to your database region to reduce latency
* Authentication: Allow unauthenticated invocations

Under Revision Settings, click on Show Advanced Settings to expand them:

* Container port: 5200
* Environment variables. Add the following key-value pair and make sure you add the connection string for your own MongoDB Atlas deployment:

`ATLAS_URI:mongodb+srv:/<username>:<password>@sandbox.pv0l7.mongodb.net/meanStackExample?retryWrites=true&w=majority`

For the Build environment, select Cloud Build.

Finally, in the Build Settings section, select:

* Builder: Docker
* Docker: mean-stack-example/server/Dockerfile

Click the Deploy button and then Show Detailed Logs to follow the deployment of your first Cloud Run service!

After the build has completed, you should see the URL of the deployed service:

![Screenshot displaying the message 'Deployment completed successfully!' and the deployment URL.
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/deployment_completed_successfully_31bcb13b6f.JPG)

Open the URL and append ‘/employees’ to the end. You should see an empty array because currently, there are no documents in the database. Let’s deploy the user interface so we can add some!

### 4. Deploy the Angular web application

Our Angular application is in the client directory. To deploy it, we’ll use the Nginx server and Docker.

> Just a thought, there is also an option to use Firebase Hosting for your [Angular application deployment](https://www.youtube.com/watch?v=jVPHsolXgLk) as you can serve your content to a CDN (content delivery network) directly.

Let’s take a look at the configuration files:

**mean-stack-example/client/nginx.conf**

```
events{}
 
http {
 
   include /etc/nginx/mime.types;
 
   server {
       listen 8080;
       server_name 0.0.0.0;
       root /usr/share/nginx/html;
       index index.html;
 
       location / {
           try_files $uri $uri/ /index.html;
       }
   }
}
```

In the Nginx configuration, we specify the default port—8080, and the starting file—`index.html`.

**mean-stack-example/client/Dockerfile**

```
FROM node:17-slim AS build
 
WORKDIR /usr/src/app
COPY package.json package-lock.json ./
 
# Install dependencies and copy them to the container
RUN npm install
COPY . .
 
# Build the Angular application for production
RUN npm run build --prod
 
# Configure the nginx web server
FROM nginx:1.17.1-alpine
COPY nginx.conf /etc/nginx/nginx.conf
COPY --from=build /usr/src/app/dist/client /usr/share/nginx/html
 
# Run the web service on container startup.
CMD ["nginx", "-g", "daemon off;"]
```

In the Docker configuration, we install Node.js dependencies and build the project. Then, we copy the built files to the container, configure, and start the Nginx service.

Finally, we need to configure the URL to the REST API so that our client application can send requests to it. Since we’re only using the URL in a single file in the project, we’ll hardcode the URL. Alternatively, you can attach the environment variable to the window object and access it from there.

**mean-stack-example/client/src/app/employee.service.ts**

```
@Injectable({
 providedIn: 'root'
})
export class EmployeeService {
 // Replace with the URL of your REST API
 private url = 'https://node-express-api-vsktparjta-uc.a.run.app'; 
…
```

We’re ready to deploy to Cloud Run! Start a new deployment with the following configuration settings:

* Service Settings: Create a service
* Service name: angular-web-app
* Deployment platform: Cloud Run (fully managed)
* Authentication: Allow unauthenticated invocations

For the Build environment, select Cloud Build.

Finally, in the Build Settings section, select:

* Builder: Docker
* Docker: mean-stack-example/client/Dockerfile

Click that Deploy button again and watch the logs as your app is shipped to the cloud! When the deployment is complete, you should see the URL for the client app:

![Screenshot displaying the message 'Deployment completed successfully!' and the deployment URL for the Angular service.
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/deployment_completed_successfully_notice_6602368f53.JPG)

Open the URL, and play with your application!

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/employees_demo_1_da0a6342ba.gif)

### Command shell alternative for build and deploy

The steps covered above can alternatively be implemented from Command Shell as below:

Step 1: Create the new project directory named “mean-stack-example” either from the Code Editor or Cloud Shell Command (Terminal):

*mkdir mean-stack-demo
cd mean-stack-demo*

Step 2: Clone project repo and make necessary changes in the configuration and variables, same as mentioned in the previous section.

Step 3: Build your container image using Cloud build by running the command in Cloud Shell:

*gcloud builds submit --tag gcr.io/$GOOGLECLOUDPROJECT/mean-stack-demo*

$GOOGLE_CLOUD_PROJECT is an environment variable containing your Google Cloud project ID when running in Cloud Shell.

Step 4: Test it locally by running: 
docker run -d -p 8080:8080 gcr.io/$GOOGLE_CLOUD_PROJECT/mean-stack-demo 
and by clicking Web Preview, Preview on port 8080.

Step 5: Run the following command to deploy your containerized app to Cloud Run:

*gcloud run deploy mean-stack-demo --image 
gcr.io/$GOOGLECLOUDPROJECT/mean-stack-demo --platform managed --region us-central1 --allow-unauthenticated --update-env-vars DBHOST=$DB_HOST*

a. –allow-unauthenticated will let the service be reached without authentication.

b. –platform-managed means you are requesting the fully managed environment and not the Kubernetes one via Anthos.
 
c. –update-env-vars expects the MongoDB Connection String to be passed on to the environment variable DBHOST.
Hang on until the section on Env variable and Docker for Continuous Deployment for Secrets and Connection URI management.

d. When the deployment is done, you should see the deployed service URL in the command line.

e. When you hit the service URL, you should see your web page on the browser and the logs in the Cloud Logging Logs Explorer page. 

![Service logs displayed in the Cloud Logging Logs Explorer page
](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/services_logs_2085273e8a.JPG)

### 5. Environment variables and Docker for continuous deployment

If you’re looking to automate the process of building and deploying across multiple containers, services, or components, storing these configurations in the repo is not only cumbersome but also a security threat. 

1. For ease of cross-environment continuous deployment and to avoid security vulnerabilities caused by leaking credential information, we can choose to pass variables at build/deploy/up time.
    
    *--update-env-vars* allows you to set the environment variable to a value that is passed only at run time. In our example, the variable DBHOST is assigned the value of $DB_HOST. which is set as *DB_HOST = ‘<<ENCODED CONNECTION URI>>’*.

    Please note that unencoded symbols in Connection URI (username, password) will result in connection issues with MongoDB. For example, if you have a $ in the password or username, replace it with %24 in the encoded Connection URI.

2. Alternatively, you can also pass configuration variables as env variables at build time into docker-compose (*docker-compose.yml*). By passing configuration variables and credentials, we avoid credential leakage and automate deployment securely and continuously across multiple environments, users, and applications.

## Conclusion

MongoDB Atlas with Cloud Run makes for a truly serverless MEAN stack solution, and for those looking to build an application with a serverless option to run in a stateless container, Cloud Run is your best bet. 

## Before you go…

Now that you have learnt how to deploy a simple MEAN stack application on Cloud Run and MongoDB Atlas, why don’t you take it one step further with your favorite client-server use case? Reference the below resources for more inspiration:

* Cloud Run HelloWorld: [https://codelabs.developers.google.com/codelabs/cloud-run-hello#4](https://codelabs.developers.google.com/codelabs/cloud-run-hello#4)
* MongoDB - MEAN Stack: [https://www.mongodb.com/languages/mean-stack-tutorial](https://www.mongodb.com/languages/mean-stack-tutorial)

If you have any comments or questions, feel free to reach out to us online: [Abirami Sukumaran](http://www.linkedin.com/in/abiramisukumaran) and [Stanimira Vlaeva](https://twitter.com/StanimiraVlaeva).

In this blog, we'll see how Cloud Run and MongoDB come together to enable a completely serverless MEAN stack application development experience.

Serverless MEAN Stack Applications with Cloud Run and MongoDB Atlas

Community

Ado Kukic

In this article, we’ll explore the [Mongoose](https://mongoosejs.com/) library for MongoDB. Mongoose is a Object Data Modeling (ODM) library for MongoDB distributed as an npm package. We'll compare and contrast Mongoose to using the native [MongoDB Node.js driver](https://docs.mongodb.com/drivers/node/current/) together with [MongoDB Schema Validation](https://docs.mongodb.com/manual/core/schema-validation/).

We’ll see how the MongoDB Schema Validation helps us enforce a database schema, while still allowing for great flexibility when needed. Finally, we’ll see if the additional features that Mongoose provides are worth the overhead of introducing a third-party library into our applications.

## What is Mongoose?

Mongoose is a Node.js-based Object Data Modeling (ODM) library for MongoDB. It is akin to an Object Relational Mapper (ORM) such as [SQLAlchemy](https://www.sqlalchemy.org/) for traditional SQL databases. The problem that Mongoose aims to solve is allowing developers to enforce a specific schema at the application layer. In addition to enforcing a schema, Mongoose also offers a variety of hooks, model validation, and other features aimed at making it easier to work with MongoDB.

## What is MongoDB Schema Validation?

[MongoDB Schema Validation](https://docs.mongodb.com/manual/core/schema-validation/) makes it possible to easily enforce a schema against your MongoDB database, while maintaining a high degree of flexibility, giving you the best of both worlds. In the past, the only way to enforce a schema against a MongoDB collection was to do it at the application level using an ODM like Mongoose, but that posed significant challenges for developers.

## Getting Started

If you want to follow along with this tutorial and play around with schema validations but don't have a MongoDB instance set up, you can set up a [free MongoDB Atlas cluster here](https://mongodb.com/atlas).

## Object Data Modeling in MongoDB

A huge benefit of using a NoSQL database like MongoDB is that you are not constrained to a rigid data model. You can add or remove fields, nest data multiple layers deep, and have a truly flexible data model that meets your needs today and can adapt to your ever-changing needs tomorrow. But being too flexible can also be a challenge. If there is no consensus on what the data model should look like, and every document in a collection contains vastly different fields, you're going to have a bad time.

### Mongoose Schema and Model

On one end of the spectrum, we have ODM's like Mongoose, which from the get-go force us into a semi-rigid schema. With Mongoose, you would define a `Schema` object in your application code that maps to a collection in your MongoDB database. The `Schema` object defines the structure of the documents in your collection. Then, you need to create a `Model` object out of the schema. The model is used to interact with the collection.

For example, let's say we're building a blog and want to represent a blog post. We would first define a schema and then create an accompanying Mongoose model:

``` javascript
const blog = new Schema({
   title: String,
   slug: String,
   published: Boolean,
   content: String,
   tags: [String],
   comments: [{
       user: String,
       content: String,
       votes: Number
   }]
});
 
const Blog = mongoose.model('Blog', blog);
```

### Executing Operations on MongoDB with Mongoose

Once we have a Mongoose model defined, we could run queries for fetching,updating, and deleting data against a MongoDB collection that alignswith the Mongoose model. With the above model, we could do things like:

``` javascript
// Create a new blog post
const article = new Blog({
   title: 'Awesome Post!',
   slug: 'awesome-post',
   published: true,
   content: 'This is the best post ever',
   tags: ['featured', 'announcement'],
});
 
// Insert the article in our MongoDB database
article.save();
 
// Find a single blog post
Blog.findOne({}, (err, post) => {
   console.log(post);
});
```

### Mongoose vs MongoDB Node.js Driver: A Comparison

The benefit of using Mongoose is that we have a schema to work against in our application code and an explicit relationship between our MongoDB documents and the Mongoose models within our application. The downside is that we can only create blog posts and they have to follow the above defined schema. If we change our Mongoose schema, we are changing the relationship completely, and if you're going through rapid development, this can greatly slow you down.

The other downside is that this relationship between the schema and model only exists within the confines of our Node.js application. Our MongoDB database is not aware of the relationship, it just inserts or retrieves data it is asked for without any sort of validation. In the event that we used a different programming language to interact with our database, all the constraints and models we defined in Mongoose would be worthless.

On the other hand, if we decided to use just the [MongoDB Node.js](https://mongodb.github.io/node-mongodb-native/3.5/) driver, we could
run queries against any collection in our database, or create new ones on the fly. The MongoDB Node.js driver does not have concepts of object data modeling or mapping.

We simply write queries against the database and collection we wish to work with to accomplish the business goals. If we wanted to insert a new blog post in our collection, we could simply execute a command like so:

``` javascript
db.collection('posts').insertOne({
   title: 'Better Post!',
   slug: 'a-better-post',
   published: true,
   author: 'Ado Kukic',
   content: 'This is an even better post',
   tags: ['featured'],
});
```

This `insertOne()` operation would run just fine using the Node.js Driver. If we tried to save this data using our Mongoose `Blog` model, it would fail, because we don't have an `author` property defined in our Blog Mongoose model.

Just because the Node.js driver doesn't have the concept of a model, does not mean we couldn't create models to represent our MongoDB data at the application level. We could just as easily create a generic model or use a library such as [objectmodel](https://objectmodel.js.org/). We could create a `Blog` model like so:

``` javascript
function Blog(post) {
   this.title = post.title;
   this.slug = post.slug;
   ...
}
```

We could then use this model in conjunction with our MongoDB Node.js driver, giving us both the flexibility of using the model, but not being constrained by it.

``` javascript
db.collection('posts').findOne({}).then((err, post) => {
   let article = new Blog(post);
});
```

In this scenario, our MongoDB database is still blissfully unaware of our Blog model at the application level, but our developers can work with it, add specific methods and helpers to the model, and would know that this model is only meant to be used within the confines of our Node.js application. Next, let's explore schema validation.

## Adding Schema Validation

We can choose between two different ways of adding schema validation to our MongoDB collections. The first is to use application-level validators, which are defined in the Mongoose schemas. The second is to use MongoDB schema validation, which is defined in the MongoDB collection itself. The huge difference is that native MongoDB schema validation is applied at the database level. Let's see why that matters by exploring both methods.

### Schema Validation with Mongoose

When it comes to schema validation, Mongoose enforces it at the application layer as we've seen in the previous section. It does this in two ways.

First, by defining our model, we are explicitly telling our Node.js application what fields and data types we'll allow to be inserted into a specific collection. For example, our Mongoose Blog schema defines a `title` property of type `String`. If we were to try and insert a blog post with a `title` property that was an array, it would fail. Anything outside of the defined fields, will also not be inserted in the database.

Second, we further validate that the data in the defined fields matches our defined set of criteria. For example, we can expand on our Blog model by adding specific validators such as requiring certain fields, ensuring a minimum or maximum length for a specific field, or coming up with our custom logic even. Let's see how this looks with Mongoose. In our code we would simply expand on the property and add our validators:

``` javascript
const blog = new Schema({
   title: {
       type: String,
       required: true,
   },
   slug: {
       type: String,
       required: true,
   },
   published: Boolean,
   content: {
       type: String,
       required: true,
       minlength: 250
   },
   ...
});
 
const Blog = mongoose.model('Blog', blog);
```

Mongoose takes care of model definition and schema validation in one fell swoop. The downside though is still the same. These rules only apply at the application layer and MongoDB itself is none the wiser.

The MongoDB Node.js driver itself does not have mechanisms for inserting or managing validations, and it shouldn't. We can define schema validation rules for our MongoDB database using the [MongoDB Shell](https://docs.mongodb.com/mongodb-shell/#mongodb-binary-bin.mongosh) or [Compass](https://www.mongodb.com/products/compass).

We can create a schema validation when creating our collection or after the fact on an existing collection. Since we've been working with this blog idea as our example, we'll add our schema validations to it. I will use Compass and [MongoDB Atlas](https://mongodb.com/atlas). For a great resource on how to programmatically add schema validations, check out this [series](https://www.mongodb.com/blog/post/json-schema-validation--locking-down-your-model-the-smart-way).

> If you want to follow along with this tutorial and play around with
> schema validations but don't have a MongoDB instance set up, you can
> set up a [free MongoDB Atlas cluster here](https://mongodb.com/atlas).

Create a collection called `posts` and let's insert our two documents that we've been working with. The documents are:

``` javascript
[{"title":"Better Post!","slug":"a-better-post","published":true,"author":"Ado Kukic","content":"This is an even better post","tags":["featured"]}, {"_id":{"$oid":"5e714da7f3a665d9804e6506"},"title":"Awesome Post","slug":"awesome-post","published":true,"content":"This is an awesome post","tags":["featured","announcement"]}]
```

Now, within the Compass UI, I will navigate to the **Validation** tab. As expected, there are currently no validation rules in place, meaning our database will accept any document as long as it is valid BSON. Hit the **Add a Rule** button and you'll see a user interface for creating your own validation rules.

![Valid Document Schema](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/schema-validation/blog-valid-schema.png "Valid Document Schema")

By default, there are no rules, so any document will be marked as passing. Let's add a rule to require the `author` property. It will look like this:

``` javascript
{
  $jsonSchema: {
    bsonType: "object",
    required: [ "author" ]
  }
}
```

Now we'll see that our initial post, that does not have an `author` field has failed validation, while the post that does have the `author` field is good to go.

![Invalid Document Schema](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/schema-validation/blog-invalid-schema.png "Invalid Document Schema")

We can go further and add validations to individual fields as well. Say for SEO purposes we wanted all the titles of the blog posts to be a minimum of 20 characters and have a maximum length of 80 characters. We can represent that like this:

``` javascript
{
  $jsonSchema: {
    bsonType: "object",
    required: [ "tags" ],
    properties: {
      title: {
        type: "string",
        minLength: 20,
        maxLength: 80
      }
    }
  }
}
```

Now if we try to insert a document into our `posts` collection either via the Node.js Driver or via Compass, we will get an error.

![Validation Error](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/schema-validation/blog-validation-error.png)

There are many more rules and validations you can add. Check out the full list [here](https://docs.mongodb.com/manual/reference/operator/query/jsonSchema/#json-schema). For a more advanced guided approach, check out the articles on [schema](https://www.mongodb.com/blog/post/json-schema-validation--checking-your-arrays) validation with arrays and [dependencies](https://www.mongodb.com/blog/post/json-schema-validation--dependencies-you-can-depend-on).

### Expanding on Schema Validation

With Mongoose, our data model and schema are the basis for our interactions with MongoDB. MongoDB itself is not aware of any of these constraints, Mongoose takes the role of judge, jury, and executioner on what queries can be executed and what happens with them.

But with MongoDB native schema validation, we have additional flexibility. When we implement a schema, validation on existing documents does not happen automatically. Validation is only done on updates and inserts. If we wanted to leave existing documents alone though, we could change the `validationLevel` to only validate new documents inserted in the database.

Additionally, with schema validations done at the MongoDB database level, we can choose to still insert documents that fail validation. The `validationAction` option allows us to determine what happens if a query fails validation. By default, it is set to `error`, but we can change it to `warn` if we want the insert to still occur. Now instead of an insert or update erroring out, it would simply warn the user that the operation failed validation.

And finally, if we needed to, we can bypass document validation altogether by passing the `bypassDocumentValidation` option with our query. To show you how this works, let's say we wanted to insert just a `title` in our `posts` collection and we didn't want any other data. If we tried to just do this...

``` javascript
db.collection('posts').insertOne({ title: 'Awesome' });
```

... we would get an error saying that document validation failed. But if we wanted to skip document validation for this insert, we would simply do this:

``` javascript
db.collection('posts').insertOne(
   { title: 'Awesome' },
   { bypassDocumentValidation: true }
);
```

This would not be possible with Mongoose. MongoDB schema validation is more in line with the entire philosophy of MongoDB where the focus is on a flexible design schema that is quickly and easily adaptable to your use cases.

## Populate and Lookup

The final area where I would like to compare Mongoose and the Node.js MongoDB driver is its support for pseudo-joins. Both Mongoose and the native Node.js driver support the ability to combine documents from multiple collections in the same database, similar to a join in traditional relational databases.

The Mongoose approach is called **Populate**. It allows developers to create data models that can reference each other and then, with a simple API, request data from multiple collections. For our example, let's expand on the blog post and add a new collection for users.

``` javascript
const user = new Schema({
   name: String,
   email: String
});
 
const blog = new Schema({
   title: String,
   slug: String,
   published: Boolean,
   content: String,
   tags: [String],
   comments: [{
       user: { Schema.Types.ObjectId, ref: 'User' },
       content: String,
       votes: Number
   }]
});
 
const User = mongoose.model('User', user);
const Blog = mongoose.model('Blog', blog);
```

What we did above was we created a new model and schema to represent users leaving comments on blog posts. When a user leaves a comment, instead of storing information on them, we would just store that user’s `_id`. So, an update operation to add a new comment to our post may look something like this:

``` javascript
Blog.updateOne({
   comments: [{ user: "12345", content: "Great Post!!!" }]
});
```

This is assuming that we have a user in our `User` collection with the `_id` of `12345`. Now, if we wanted to **populate** our `user` property when we do a query—and instead of just returning the `_id` return the entire document—we could do:

``` javascript
Blog.
   findOne({}).
   populate('comments.user').
   exec(function (err, post) {
       console.log(post.comments[0].user.name) // Name of user for 1st comment
   });
```

Populate coupled with Mongoose data modeling can be very powerful, especially if you're coming from a relational database background. The drawback though is the amount of magic going on under the hood to make this happen. Mongoose would make two separate queries to accomplish this task and if you're joining multiple collections, operations can quickly slow down.

The other issue is that the populate concept only exists at the application layer. So while this does work, relying on it for your database management can come back to bite you in the future.

MongoDB as of version 3.2 introduced a new operation called `$lookup` that allows to developers to essentially do a left outer join on collections within a single MongoDB database. If we wanted to populate the user information using the Node.js driver, we could create an aggregation pipeline to do it. Our starting point using the `$lookup` operator could look like this:

``` javascript
db.collection('posts').aggregate([
  {
    '$lookup': {
      'from': 'users', 
      'localField': 'comments.user', 
      'foreignField': '_id', 
      'as': 'users'
    }
  }, {}
], (err, post) => {
    console.log(post.users); //This would contain an array of users
});
```

We could further create an additional step in our aggregation pipeline to replace the user information in the `comments` field with the users data, but that's a bit out of the scope of this article. If you wish to learn more about how aggregation pipelines work with MongoDB, check out the [aggregation docs](https://docs.mongodb.com/manual/aggregation/).

## Final Thoughts: Do I Really Need Mongoose?

Both Mongoose and the MongoDB Node.js driver support similar functionality. While Mongoose does make MongoDB development familiar to someone who may be completely new, it does perform a lot of magic under the hood that could have unintended consequences in the future.

I personally believe that you don't need an ODM to be successful with MongoDB. I am also not a huge fan of ORMs in the relational database world. While they make initial dive into a technology feel familiar, they abstract away a lot of the power of a database.

Developers have a lot of choices to make when it comes to building applications. In this article, we looked at the differences between using an ODM versus the native driver and showed that the difference between the two is not that big. Using an ODM like Mongoose can make development feel familiar but forces you into a rigid design, which is an anti-pattern when considering building with MongoDB.

The MongoDB Node.js driver works natively with your MongoDB database to give you the best and most flexible development experience. It allows the database to do what it's best at while allowing your application to focus on what it's best at, and that's probably not managing data models.

Learn why using an Object Data Modeling library may not be the best choice when building MongoDB apps with Node.js.

MongoDB & Mongoose: Compatibility and Comparison

Pavel Duchovny

Lead Developer Advocate

Remix is a new and exciting javascript web framework. Together with the MongoDB Atlas Data API and Atlas Search it can form powerful web applications. A guided tour will show you how to leverage both technologies together. 

Next Gen Web Apps with Remix and MongoDB Atlas Data API

Maxime Beugnet

Senior Developer Advocate

Data Federation is a MongoDB Atlas feature that allows you to query data from disparate sources such as:

* Atlas databases.
* Atlas Data Lake.
* HTTP APIs.
* AWS S3 buckets.

In this tutorial, I will show you how to access your archived documents in S3 **and** your documents in your MongoDB Atlas cluster with a **single** MQL query.

![MongoDB Atlas Data Federation](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/how-to/atlas-data-lake/federated-queries.png)

This feature is really amazing because it allows you to have easy access to your archived data in S3 along with your "hot" data in your Atlas cluster. This could help you prevent your Atlas clusters from growing in size indefinitely and reduce your costs drastically. It also makes it easier to gain new insights by easily querying data residing in S3 and exposing it to your real-time app.

Finally, I will show you how to use the new version of the [$out aggregation pipeline stage](https://docs.mongodb.com/datalake/reference/pipeline/out) to write documents from a MongoDB Atlas cluster into an AWS S3 bucket.

## Prerequisites

In order to follow along this tutorial, you need to:

* Create a [MongoDB Atlas cluster](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster/). ✅ Already have an AWS account? Atlas supports paying for usage via the AWS Marketplace (AWS MP) without any upfront commitment — simply <a href="https://aws.amazon.com/marketplace/pp/prodview-pp445qepfdy34?trk=38088f03-4928-4b00-9a39-86781df6ab9b&sc_channel=el" target="_blank" rel="noreferrer">sign up for MongoDB Atlas via AWS Marketplace.<a>
* Create a user in the **Database Access** menu.
* Add your IP address in the Network Access List in the **Network Access** menu.
* Have Python 3 with `pymongo` and `dnspython` libs installed.

### Configure your S3 bucket and AWS account

Log into your AWS account and create an S3 bucket. Choose a region close to your Atlas deployment to minimize data latency. The scripts in this tutorial use a bucket called `cold-data-mongodb` in the region `eu-west-1`. If you use a different name or select another region, make sure to reflect that in the Python code you’ll see in the tutorial. 

Then, install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and configure it to access your AWS account. If you need help setting it up, refer to the [AWS documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html).

### Prepare the dataset

To illustrate how `$out` and federated queries work, I will use an overly simple dataset to keep things as easy as possible to understand. Our database “test” will have a single collection, “orders,” representing orders placed in an online store. Each order document will have a “created” field of type “Date.” We’ll use that field to archive older orders, moving them from the Atlas cluster to S3.

I’ve written a Python script that inserts the required data in the Atlas cluster. You can get the script, along with the rest of the code we’ll use in the tutorial, from GitHub:

```
git clone https://github.com/mongodb-developer/data-lake-tutorial.git
```

Then, go back to Atlas to locate the connection string for your cluster. Click on “Connect” and then “Connect your application.” Copy the connection string and paste it in the `insert_data.py` script you just downloaded from GitHub. Don’t forget to replace the `<username>` and `<password>` placeholders with the credentials of your database user:

**insert_data.py**
```python
from pymongo import MongoClient
from datetime import datetime

client = MongoClient('mongodb+srv://<username>:<password>@m0.lbtrerw.mongodb.net/')
…
```

Finally, install the required libraries and run the script:

```
pip3 install -r requirements.txt
python3 insert_data.py
```

Now that we have a “massive” collection of orders, we can consider archiving the oldest orders to an S3 bucket. Let's imagine that once a month is over, we can archive all the orders from the previous month. We’ll create one JSON file in S3 for all the orders created during the previous month.

We’ll transfer these orders to S3 using the [aggregation pipeline stage $out](https://docs.mongodb.com/datalake/reference/pipeline/out).

But first, we need to configure Atlas Data Federation correctly.

## Configure Data Federation
Navigate to “Data Federation” from the side menu in Atlas and then click “set up manually” in the "create new federated database" dropdown in the top right corner of the UI.

On the left, we see a panel with the data sources (we don’t have any yet), and on the right are the “virtual” databases and collections of the federated instance.

### Configure the Atlas cluster as a data source

Let’s add the first data source — the orders from our Atlas cluster. Click “Add Data Sources,” select “Atlas Cluster,” and then select your cluster and database.

Click “Next” and you’ll see the “test.orders” collection as a data source. Click on the “test.orders” row, drag it underneath the “VirtualCollection0,” and drop it there as a data source.

### Configure the S3 bucket as a data source

Next, we’ll connect our S3 bucket. Click on “Add Data Sources” again and this time, select Amazon S3. Click “Next” and follow the instructions to create and authorize a new AWS IAM role. We need to execute a couple of commands with the AWS CLI. Make sure you’ve installed and linked the CLI to your AWS account before that. If you’re facing any issues, check out the [AWS CLI troubleshooting page](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-troubleshooting.html).

Once you’ve authorized the IAM role, you’ll be prompted for the name of your S3 bucket and the access policy. Since we'll be writing files to our bucket, we need to choose “Read and write.”

You can also configure a prefix. If you do, Data Federation will only search for files in directories starting with the specified prefix. In this tutorial, we want to access files in the root directory of the bucket, so we’ll leave this field empty.

![S3 bucket configuration in Atlas Data Federation](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/s3_bucket_configuration_6d308356ea.png)

After that, we need to execute a couple more AWS CLI commands to make sure the IAM role has permissions for the S3 bucket. When you’re finished, click “Next.”

Finally, we’ll be prompted to define a path to the data we want to access in the bucket. To keep things simple, we’ll use a wildcard configuration allowing us to access all files. Set `s3://cold-data-mongodb/*` as the path and `any value (*)` as the data type of the file. 

![Configuration of file path in the cold-data-mongodb bucket](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/configuration_of_file_path_1c99fd6ca0.png)

Data Federation also allows you to create partitions and parse fields from the filenames in your bucket. This can optimize the performance of your queries by traversing only relevant files and directories. To find out more, check out the [Data Federation docs](https://www.mongodb.com/docs/atlas/data-federation/config/path-syntax-examples/).

Once we’ve added the S3 bucket data, we can drag it over to the virtual collection as a data source.

### Rename the virtual database and collection

The names “VirtualDatabase0” and “VirtualCollection0” don’t feel appropriate for our data. Let’s rename them to “test” and “orders” respectively to match the data in the Atlas cluster.

![Federated instance configuration](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/federated_instance_configuration_46b4216632.png)

### Verify the JSON configuration

Finally, to make sure that our setup is correct, we can switch to the JSON view in the top right corner, right next to the “Save” button. Your configuration, except for the project ID and the cluster name, should be identical to this:

```json
{
  "databases": [
    {
      "name": "test",
      "collections": [
        {
          "name": "orders",
          "dataSources": [
            {
              "storeName": "M0",
              "database": "test",
              "collection": "orders"
            },
            {
              "storeName": "cold-data-mongodb",
              "path": "/*"
            }
          ]
        }
      ],
      "views": []
    }
  ],
  "stores": [
    {
      "name": "M0",
      "provider": "atlas",
      "clusterName": "M0",
      "projectId": "<your-project-id>"
    },
    {
      "name": "cold-data-mongodb",
      "provider": "s3",
      "bucket": "cold-data-mongodb",
      "prefix": "",
      "delimiter": "/"
    }
  ]
}
```

Once you've verified everything looks good, click the “Save” button. If your AWS IAM role is configured correctly, you’ll see your newly configured federated instance. We’re now ready to connect to it!

## Archive cold data to S3 with $out

Let's now collect the URI we are going to use to connect to Atlas Data Federation.

Click on the “Connect” button, and then “Connect your application.” Copy the connection string as we’ll need it in just a minute.

Now let's use Python to execute our aggregation pipeline and archive the two orders from May 2020 in our S3 bucket.

``` python
from datetime import datetime

from pymongo import MongoClient

client = MongoClient('<YOUR_FEDERATED_INSTANCE_URI>')
db = client.get_database('test')
coll = db.get_collection('orders')

start_date = datetime(2020, 5, 1)  # May 1st
end_date = datetime(2020, 6, 1)  # June 1st

pipeline = [
    {
        '$match': {
            'created': {
                '$gte': start_date,
                '$lt': end_date
            }
        }
    },
    {
        '$out': {
            's3': {
                'bucket': 'cold-data-mongodb',
                'region': 'eu-west-1',
                'filename': start_date.isoformat('T', 'milliseconds') + 'Z-' + end_date.isoformat('T', 'milliseconds') + 'Z',
                'format': {'name': 'json', 'maxFileSize': '200MiB'}
            }
        }
    }
]

coll.aggregate(pipeline)
print('Archive created!')
```
Once you replace the connection string with your own, execute the script:

```
python3 archive.py
```

And now we can confirm that our archive was created correctly in our S3 bucket:

!["file in the S3 bucket"](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/how-to/atlas-data-lake/s3-bucket.png)

### Delete the “cold” data from Atlas

Now that our orders are safe in S3, I can delete these two orders from my Atlas cluster. Let's use Python again. This time, we need to use the URI from our Atlas cluster because the Atlas Data Federation URI doesn't allow this kind of operation.

``` python
from datetime import datetime

from pymongo import MongoClient

client = MongoClient('<YOUR_ATLAS_URI>')
db = client.get_database('test')
coll = db.get_collection('orders')

start_date = datetime(2020, 5, 1)  # May 1st
end_date = datetime(2020, 6, 1)  # June 1st
query = {
    'created': {
        '$gte': start_date,
        '$lt': end_date
    }
}

result = coll.delete_many(query)
print('Deleted', result.deleted_count, 'orders.')
```

Let's run this code:

``` none
python3 remove.py

```

Now let's double-check what we have in S3. Here is the content of the S3 file I downloaded:

``` json
{"_id":{"$numberDouble":"1.0"},"created":{"$date":{"$numberLong":"1590796800000"}},"items":[{"$numberDouble":"1.0"},{"$numberDouble":"3.0"}],"price":{"$numberDouble":"20.0"}}
{"_id":{"$numberDouble":"2.0"},"created":{"$date":{"$numberLong":"1590883200000"}},"items":[{"$numberDouble":"2.0"},{"$numberDouble":"3.0"}],"price":{"$numberDouble":"25.0"}}
```

And here is what's left in my MongoDB Atlas cluster.

![Documents left in MongoDB Atlas cluster](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/test_orders_3a7c09a082.png)

### Federated queries

As mentioned above already, with Data Federation, you can query data stored across Atlas and S3 simultaneously. This allows you to retain easy access to 100% of your data. We actually already did that when we ran the aggregation pipeline with the `$out` stage.

Let's verify this one last time with Python:

``` python
from pymongo import MongoClient

client = MongoClient('<YOUR_FEDERATED_INSTANCE_URI>')
db = client.get_database('test')
coll = db.get_collection('orders')

print('All the docs from S3 + Atlas:')
docs = coll.find()
for d in docs:
    print(d)

pipeline = [
    {
        '$group': {
            '_id': None,
            'total_price': {
                '$sum': '$price'
            }
        }
    }, {
        '$project': {
            '_id': 0
        }
    }
]

print('\nI can also run an aggregation.')
print(coll.aggregate(pipeline).next())
```

Execute the script with:

```bash
python3 federated_queries.py
```

Here is the output:

``` none
All the docs from S3 + Atlas:
{'_id': 1.0, 'created': datetime.datetime(2020, 5, 30, 0, 0), 'items': [1.0, 3.0], 'price': 20.0}
{'_id': 2.0, 'created': datetime.datetime(2020, 5, 31, 0, 0), 'items': [2.0, 3.0], 'price': 25.0}
{'_id': 3.0, 'created': datetime.datetime(2020, 6, 1, 0, 0), 'items': [1.0, 3.0], 'price': 20.0}
{'_id': 4.0, 'created': datetime.datetime(2020, 6, 2, 0, 0), 'items': [1.0, 2.0], 'price': 15.0}

I can also run an aggregation:
{'total_price': 80.0}
```

## Wrap up

If you have a lot of infrequently accessed data in your Atlas cluster but you still need to be able to query it and access it easily once you've archived it to S3, creating a federated instance will help you save tons of money. If you're looking for an automated way to archive your data from Atlas clusters to fully-managed S3 storage, then check out our new [Atlas Online Archive feature](https://docs.atlas.mongodb.com/online-archive/manage-online-archive/)!

Storage on S3 is a lot cheaper than scaling up your MongoDB Atlas cluster because your cluster is full of cold data and needs more RAM and storage size to operate correctly.

All the Python code is available in this [Github repository](https://github.com/mongodb-developer/data-lake-tutorial).

Please let us know on Twitter if you liked this blog post: [@MBeugnet](https://twitter.com/MBeugnet) and [@StanimiraVlaeva](https://twitter.com/StanimiraVlaeva).

If you have questions, please head to our [developer community website](https://www.mongodb.com/community/forums/) where the MongoDB engineers and the MongoDB community will give you a hand.

Learn how to use MongoDB Atlas Data Federation to query data from Atlas databases and AWS S3 and archive cold data to S3 with $out.

MongoDB Atlas Data Federation Tutorial: Federated Queries and $out to AWS S3

We're using BigQuery, Vertex AI, and MongoDB Atlas to predict a categorical variable using a Supervised Machine Learning Model created with AutoML.

Movie Score Prediction with BigQuery, Vertex AI, and MongoDB Atlas

In an era of widespread digitalization, businesses operating in critical sectors such as healthcare, banking and finance, and government face an ever-increasing threat of data breaches and cyber-attacks. Ensuring the security of data is no longer just a matter of compliance but has become a top priority for businesses to safeguard their reputation, customer trust, and financial stability. However, maintaining the privacy and security of sensitive data while still enabling seamless access to services within a virtual private cloud (VPC) is a complex challenge that requires a robust solution. That’s where MongoDB’s Private Service Connect (PSC) on Google Cloud comes in. As a cloud networking solution, it provides secure access to services within a VPC using private IP addresses. PSC is also a powerful tool to protect businesses from the ever-evolving threat landscape of data security. 

## What is PSC (Private Service Connect)?

PSC simplifies how services are being securely and privately consumed. It allows easy implementation of private endpoints for the service consumers to connect privately to service producers across organizations and eliminates the need for virtual private cloud peering. The effort needed to set up private connectivity between MongoDB and Google consumer project is reduced with the PSC.

MongoDB announced the [support](https://www.mongodb.com/blog/post/announcing-google-private-service-connect-psc-integration-mongodb-atlas) for Google Cloud [Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect) (PSC) in November 2021. PSC was added as a new option to access MongoDB securely from Google Cloud without exposing the customer traffic to the public internet. With PSC, customers will be able to achieve one-way communication with MongoDB. In this article, we are going to introduce the new features of PSC and MongoDB integration.

## PSC Interconnect support

Connecting MongoDB from the on-prem machines is made easy using PSC Interconnect support. PSC Interconnect allows traffic from on-prem devices to reach PSC endpoints in the same region as the Interconnect. This is also a transparent update with no API changes.

There are no additional actions required by the customer to start using their Interconnect with PSC. Once Interconnect support has been rolled out to the customer project, then traffic from the Interconnect will be able to reach PSC endpoints and in turn access the data from MongoDB using service attachments.

## Google Cloud multi-region support

Private Service Connect now provides multi-region support for MongoDB Atlas clusters, enabling customers to connect to MongoDB instances in different regions securely. With this feature, customers can ensure high availability even in case of a regional failover. To achieve this, customers need to set up the service attachments in all the regions that the cluster will have its nodes on. Each of these service attachments are in turn connected to Google Cloud service endpoints.

![A connection made from us-east1 will be able to connect to region us-west1 in case of the primary failover from region us-east1 to us-west1.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/connection_9d618950d6.png)

## MongoDB multi-cloud support

Customers who have their deployment on multiple regions spread across multiple clouds can now utilize MongoDB PSC to connect to the Google Cloud nodes in their deployment. The additional requirement is to set up the private link for the other nodes to make sure that the connection could be made to the other nodes from their respective cloud targets. 

## Wrap-up

In conclusion, Private Service Connect has come a long way from its initial release. Now, PSC on MongoDB supports connection from on-prem using Interconnect and also connects to multiple regions across MongoDB clusters spread across Google Cloud regions or multi-cloud clusters securely using Global access.

1. Learn how to set up PSC multi region for MongoDB Atlas with [codelabs tutorials](https://codelabs.developers.google.com/codelabs/psc-mongo-globalaccess#0).
2. You can subscribe to MongoDB Atlas using [Google Cloud Marketplace](https://console.cloud.google.com/marketplace/product/mongodb/mdb-atlas-self-service).
3. You can [sign up](https://mongodb.com/cloud/atlas/register?utm_campaign=devrel&utm_source=cross-post&utm_medium=cta&utm_content=psc_announcement&utm_term=stanimira.vlaeva) for MongoDB using the registration page.
4. Learn more about [Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect).
5. Read the [PSC announcement for MongoDB](https://mongodb.com/blog/post/announcing-google-private-service-connect-psc-integration-mongodb-atlas?utm_campaign=devrel&utm_source=cross-post&utm_medium=cta&utm_content=psc_integration&utm_term=stanimira.vlaeva).

PSC is a cloud networking solution that provides secure access to services within a VPC. Read about the newly announced support for PSC Interconnect and Global access for MongoDB.

Introducing PSC Interconnect and Global Access for MongoDB Atlas

DJ Walker



<div class="introduction">

The real power of a cloud-hosted, fully managed service like [MongoDB Atlas](https://www.mongodb.com/cloud/atlas) is that you can create whole new database deployment architectures automatically, using the services API. Getting to the [MongoDB Atlas Administration API](https://docs.atlas.mongodb.com/api/) is relatively simple and, once unlocked, it opens up a massive opportunity to integrate and automate the management of database deployments from creation to deletion. The API itself is an extensive REST API. There's role-based access control and you can have [user or app-specific credentials](https://docs.atlas.mongodb.com/configure-api-access/) to access it.

There is one tiny thing that can trip people up though. The credentials have to be passed over using the [digest authentication mechanism](https://en.wikipedia.org/wiki/Digest_access_authentication), not the more common basic authentication or using an issued token. Digest authentication, at its simplest, waits to get an HTTP `401 Unauthorized` response from the web endpoint. That response comes with data and the client then sends an encrypted form of the username and password as a digest and the server works with that.

And that's why we’re here today: to show you how to do that with the least fuss in Python, Node, and Ruby. In each example, we'll try and access the base URL of the Atlas Administration API which returns a JSON document about the underlying applications name, build and other facts.
You can find all code samples in the dedicated [Github repository](https://github.com/mongodb-developer/atlas-admin-api-digest-auth).

## Setup

To use the Atlas Administration API, you need… a MongoDB Atlas cluster! If you don’t have one already, follow the [Get Started with Atlas](https://www.mongodb.com/docs/atlas/getting-started/) guide to create your first cluster.

The next requirement is the organization API key. You can set it up in two steps:

[Create an API key](https://www.mongodb.com/docs/atlas/configure-api-access/#std-label-create-org-api-key) in your Atlas organization. Make sure the key has the Organization Owner permission.
Add your IP address to the [API Access List for the API key](https://www.mongodb.com/docs/atlas/configure-api-access/#add-an-api-access-list-entry).

Then, open a new terminal and export the following environment variables, where `ATLAS_USER` is your public key and `ATLAS_USER_KEY` is your private key.

```
export ATLAS_USER=<public_key>
export ATLAS_USER_KEY=<private_key>
```

You’re all set up! Let’s see how we can use the Admin API with Python, Node, and Ruby.

</div>

<div class="content">

## Python

We start with the simplest and most self-contained example: Python.

In the Python version, we lean on the `requests` library for most of the heavy lifting. We can install it with `pip`:

``` bash
python -m pip install requests
```

The implementation of the digest authentication itself is the following:

``` python
import os
import requests
from requests.auth import HTTPDigestAuth
import pprint

base_url = "https://cloud.mongodb.com/api/atlas/v1.0/"
auth = HTTPDigestAuth(
   os.environ["ATLAS_USER"],
   os.environ["ATLAS_USER_KEY"]
)

response = requests.get(base_url, auth = auth)
pprint.pprint(response.json())
```

As well as importing `requests`, we also bring in `HTTPDigestAuth` from requests' `auth` module to handle digest authentication. The `os` import is just there so we can get the environment variables `ATLAS_USER` and `ATLAS_USER_KEY` as credentials, and the `pprint` import is just to format our results.

The critical part is the addition of `auth = HTTPDigestAuth(...)` to the `requests.get()` call. This installs the code needed to respond to the server when it asks for the digest.

If we now run this program...

![Screenshot of the terminal emulator after the execution of the request script for Python. The printed message shows that the request was successful.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/atlaspython_tin8y89s18_02b8a26fbe.png)

…we have our API response.


## Node.js

For Node.js, we’ll take advantage of the `urllib` package which supports digest authentication.

``` bash
npm install urllib
```

The code for the Node.js HTTP request is the following:

``` javascript
const urllib = require('urllib');

const baseUrl = 'https://cloud.mongodb.com/api/atlas/v1.0/';
const { ATLAS_USER, ATLAS_USER_KEY } = process.env;
const options = {
  digestAuth: `${ATLAS_USER}:${ATLAS_USER_KEY}`,
};

urllib.request(baseUrl, options, (error, data, response) => {
  if (error || response.statusCode !== 200) {
    console.error(`Error: ${error}`);
    console.error(`Status code: ${response.statusCode}`);
  } else {
    console.log(JSON.parse(data));
  }
});
```

Taking it from the top… we first require and import the `urllib` package. Then, we extract the `ATLAS_USER` and `ATLAS_USER_KEY` variables from the process environment and use them to construct the authentication key. Finally, we send the request and handle the response in the passed callback. 

And we’re ready to run:

![Screenshot of the terminal emulator after the execution of the request script for Node.js. The printed message shows that the request was successful.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/atlasnode_8rd944aaw7_048b949048.png)

On to our final language...

## Ruby

HTTParty is a widely used Gem which is used by the Ruby and Rails community to perform HTTP operations. It also, luckily, supports digest authentication. So, to get the party started:

``` bash
gem install httparty
```

There are two ways to use HTTParty. One is creating an object which abstracts the calls away while the other is just directly calling methods on HTTParty itself. For brevity, we'll do the latter. Here's the code:

``` ruby
require 'httparty'
require 'json'

base_url = 'https://cloud.mongodb.com/api/atlas/v1.0/'
options = {
  :digest_auth => {
    :username=>ENV['ATLAS_USER'],
    :password=>ENV['ATLAS_USER_KEY']
  }
}

result = HTTParty.get(base_url, options)

pp JSON.parse(result.body())
```

We require the HTTParty and JSON gems first. We then create a dictionary with our username and key, mapped for HTTParty's authentication, and set a variable to hold the base URL. We're ready to do our GET request now, and in the `options` (the second parameter of the GET request), we pass `:digest_auth=>auth` to switch on the digest support. We wrap up by JSON parsing the resulting body and pretty printing that. Put it all together and run it and we get:

![Screenshot of the terminal emulator after the execution of the request script for Ruby. The printed message shows that the request was successful.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/atlasruby_e82kmt50uy_d1e1a11ad7.png)

</div>

<div class="summary">

## Next Stop - The API

In this article, we learned how to call the [MongoDB Atlas Administration API](https://docs.atlas.mongodb.com/api/) using digest authentication. We took advantage of the vast library ecosystems of Python, Node.js, and Ruby, and used the following open-source community libraries:

Requests for Python
urllib for JavaScript
httparty for Ruby

If your project requires it, you can implement digest authentication yourself by following the [official specification](https://datatracker.ietf.org/doc/html/rfc7616). You can draw inspiration from the implementations in the aforementioned libraries.

Additionally, you can find all code samples from the article in Github.

With the authentication taken care of, just remember to be fastidious with your API key security and make sure you revoke unused keys. You can now move on to explore the API itself. Start in the [documentation](https://docs.atlas.mongodb.com/api/) and see what you can automate today.


</div>


Learn how to use digest authentication for the MongoDB Atlas Administration API from Python, Node.js, and Ruby.