GIANT Stories at MongoDB

How to Use MongoDB Atlas in API Integration Services

Aleksey Savateyev

Cloud

MongoDB Atlas, our fully-managed MongoDB as a service, allows organizations to take advantage of the database without having to worry about the operational overhead of managing a distributed system. For developers that seek to integrate the underlying database programmatically with tools or development platforms they're already using, Atlas also provides a full-featured API. We covered how to work with the MongoDB Atlas API from the command line in a recent blog post; today we're going to explore using the API with 3rd party API integration services, sometimes referred to as online workflows or serverless API frameworks.

We have partnered with a number of vendors in this space to allow users to take advantage of Atlas and MongoDB features from within their favored development environments. This blog will provide overviews and walkthroughs of 2 integrations with API integration platforms: Stamplay and Built.io.

Getting started with MongoDB Atlas

To get started, you will need a MongoDB Atlas account. If you don’t have one already, feel free to sign up for one here.

MongoDB Atlas Sign Up or Log In

Once you have an account, you will see the main "MongoDB Atlas Clusters" screen which we will ignore as we will be programmatically creating our clusters as shown below.

MongoDB Atlas Clusters

Click on Settings>Account and copy the User Name in the Profile section (note this may or may not be set to your email address):

MongoDB Atlas Settings

Click on Settings>Public API Access and click on the Generate button:

MongoDB Atlas API Access

Assign a name to the key. After the key is generated, be sure to jot it down because this is the only time that the UI will display it in full:

MongoDB Atlas API Access

The user name and API key values are all you need to connect your API integration tools to MongoDB Atlas.

Stamplay integration with MongoDB Atlas

Stamplay is an integration platform for creating new services by combining existing ones, often with little code required. It allows users to integrate, mix, and match APIs to create new services and automate processes thanks to 80+ cloud connectors and out of the box APIs for data storage, webhooks, NodeJS runtime, user authentication and hosting. Stamplay allows users to trigger workflows either based on events in your MongoDB Atlas cluster or with devops meta-scripts.

The following walkthrough describes the easiest way to use Stamplay to spin up a new MongoDB Atlas cluster, add an IP to the IP whitelist, create new database users and the database itself in the cluster, and finally creating a Slack notification when the cluster is ready. Stamplay supports a wide range of other popular APIs such as Sendgrid, Twilio, Box, and more.

Getting started with Stamplay

Visit stamplay.com to get started:

Get started with Stamplay

After you have signed up and logged in, you will be able to view your projects:

Log In to Stamplay

Click the [+] button and select “USE A BLUEPRINT” to use a sample MongoDB Atlas cluster deployment blueprint with Slack notifications:

Sample Atlas cluster deployment blue with Slack notification

In the following screen, you can see a number of blueprints, including the MongoDB autopilot API, which allows you to create a MongoDB cluster in Atlas and get a notification when it's available in Slack. Select it by clicking the “GET IT” button associated:

Create a MongoDB cluster in Atlas

![Name project](https://webassets.mongodb.com/_com_assets/cms/Stamplay_ProjectName-ikvtvpkdye.png "Name your project")

As you can see, Stamplay assigns your project a new unique endpoint with a public URL. In the next screen, follow along in the wizard to complete your blueprint. Name your project and click “START”:

MongoDB Atlas autopilot API

Click “NEXT” and paste the previously saved user name and API key into the following fields:

Enter user namename and API key

Then click “CONNECT” and if everything was configured correctly, the connection with MongoDB Atlas will be established. Click “CONNECT” in the Slack section and choose your Slack account in the pop-up (or select the "Create a new team" link and then choose a new team to sign up for one"). After signing in with Slack, click the Authorize button to let Stamplay access your Slack account.

Select your Slack Account

Click “NEXT”.

MongoDB cluster configuration in Stamplay

Now it’s time to configure the cluster when you trigger this API workflow. You can configure anything that is available via the Atlas API like disk size, backup, encryption, cluster region and so on.

Deploying MongoDB cluster in Atlas

NOTE: You need to select the Group ID of the MongoDB Atlas group where you’d like the new cluster to be deployed (you can find it in the Settings->Group Settings screen in your Atlas UI).

After that you’ll only need to select a Slack channel where the notification will be sent after the cluster is created. Click “NEXT” and you’ll get to the final screen with the completed API endpoint information.

Completed API endpoint information

Testing Stamplay workflow

To test the workflow, simply click on the provided link and review the response:

JSON response containing the result of the API requests

If everything went according to plan, you should see the JSON response containing the result of all the API requests made against MongoDB Atlas and be able to observe your cluster being created in the Atlas UI itself:

![MongoDB Atlas UI](https://webassets.mongodb.com/_com_assets/cms/MongoDB_Atlas_cluster_creation-7al3u2l5an.png "MongoDB Atlas UI')

How MongoDB Atlas integration works in Stamplay

The API endpoint that we’ve just created is powered by a workflow that chains together a series of MongoDB Atlas actions with the Webhook core component.

The workflow is triggered every time the Webhook called autopilot catches something. The workflow expects to receive two parameters on the Webhook endpoint: a clustername and an IP address. These are used in the following steps of the flow to be able to dynamically pass the name of cluster we’re creating and add an IP address to the whitelist.

Webhook workflow

The workflow leverages the Reply with JSON action from the Webhook component and can be used to create microservice workflows that combine or update data across multiple cloud services with a single API.

The sync=true parameter that you can see in the API URL provided at the end of the walkthrough tells the system that the Webhook should wait for execution to complete before returning a response to the client. You can read more about building APIs with Stamplay’s Flows and Webhooks here.

The workflow we created above also uses Stamplay’s error handling capabilities to manage potential failures in Atlas API calls to Atlas — such as trying to create a cluster with a duplicate name — to further improve the developer productivity with the Atlas API.

Built.io integration with MongoDB Atlas

MongoDB has also partnered with Built.io, which provides its own API-first platform to developers. Its main product, Built.io Flow, is an integration Platform-as-a-Service that helps IT professionals and developers unify disparate IT systems.

The following is a quick guide on how to spin up a new cluster in MongoDB Atlas and email its status with Built.io Flow. This same workflow can be used as the foundation for many other integrations, including Cisco Spark, PagerDuty, Slack, and others.

Getting started with Built.io Flow

To get started with Built.io, you will need a Built.io Flow Enterprise account. If you don’t have an existing account, sign up for a free trial account here.

At the end of this process, you should have constructed a flow that looks like this:

Develop a MongoDB-based application with Built.io

Starting from scratch, drag the following actions on to your blank canvas organized in a similar layout as shown above:

  1. Create Cluster
  2. Add Group Whitelist Entry
  3. Create Database User
  4. Send an Email

Connect the actions as displayed in the image above.

MongoDB cluster configuration in Built.io

After connecting everything, edit the Create Cluster action. The following screen is an example of the information you’ll need to input in order to set up the Create Cluster action:

Create MongoDB Cluster

The first thing in the edit window is the Connect to MongoDB Atlas section. Choose Add New and the following screen should pop up. Input your MongoDB Atlas username and then input your API key:

![Connecting to MongoDB Atlas](https://webassets.mongodb.com/_com_assets/cms/ConnectingtoMongoDBAtlas-ftexontsmo.png "Connect to MongoDB Atlas")

Once you’ve completed adding the connection, get the Group ID from your MongoDB Atlas installation and input it in the Group ID field. Input the Instance Size, the Provider Name, and the Region Name you’d prefer for your new cluster. Be sure to examine all of the fields and their descriptions to customize your new cluster appropriately.

The second step is to enter in the appropriate information for whitelisting an IP Address (or CIDR block) to enable access to your MongoDB cluster. In this case, you’ll need to do a few things:

  1. First, you’ll need to click on Show optional fields to display all of the options.
  2. Second, click inside the Group ID box to grab the mouse focus.
  3. Third, notice the Input section on the right-hand side of the edit window. Click on groupId to place the groupId from the newly formed cluster into the Group ID field here.
  4. Last, go ahead and enter the IP Address or CIDR Block you’d like to whitelist for access to your cluster.

Add group whitelist entry

The third step is to create a new user for your MongoDB database:

  1. As before, pull the Group ID information directly from the newly created cluster.
  2. Then go through and enter all the required information.
  3. Be careful: If you end up changing "Is Editable" to false then you will be unable to edit or delete the created user.
  4. Be sure to record the password as you will never be able to retrieve it from the MongoDB Atlas API.

Create a new user for your MongoDB database

The final step is to send out a confirmation email that everything has been done. This particular action is sent from Built.io’s servers, so it does not require any authentication on your part:

  1. Make sure to show the optional fields if you want to specify whether to send the email via HTML or plain text.
  2. Choose the email address and subject you’d like.
  3. In the Body section, you can click on username from the Create Database User response and name from the Create Cluster response to place both of those in the email.
  4. Press Done.

Send a confirmation email

If you have followed the steps correctly, your MongoDB Atlas workflow is now fully configured.

Testing Built.io workflow

Now that you’re done with your MongoDB Atlas workflow, you can execute it any time you want to by pressing the Play button in the top right corner of the window.

Testing Built.io workflow

There are some other interesting things you can do with Built.io Flow Enterprise. In this particular workflow, you may want to consider looking at the triggers by pressing the Settings icon over the Play action on the canvas. The very first trigger you’ll see available is the Webhook trigger. If you select it and press Save, you’ll set your workflow up to be triggered via a URL. You can use this URL in your scripts or anywhere else that accepts a URL for a web hook.

Another interesting trigger to explore is the PagerDuty trigger. Using a MongoDB Atlas integration with PagerDuty, you can have your Flow execute automatically every time a PagerDuty alert goes out. This can allow you to automate updating a cluster every time you get a low disk space alert from PagerDuty, for example.

Conclusion

Using the MongoDB Atlas API is simple. You can code against it in the programming language of your choice or you can take advantage of one of the modern API integration frameworks available today, such as Built.io or Stamplay, to increase your productivity without sacrificing any of the benefits of MongoDB Atlas.

What was demonstrated here uses only a small portion of the Atlas API. The majority of Atlas API functions, such as working with clusters, users, alerts, and whitelists, are supported both in Built.io Flow and Stamplay and can be leveraged to rapidly create even more comprehensive and sophisticated applications.

Want to know more about MongoDB Atlas? Check out MongoDB Atlas API Documentation and M034 course at MongoDB University!


About the Author – Aleksey Savateyev

Aleksey Savateyev is a Senior Solutions Architect at MongoDB, based in Silicon Valley and focusing on maximizing the value customers get from their investments in MongoDB and MongoDB Atlas.

CIDR Subnet Selection for MongoDB Atlas

Jay Gordon

Technical, Cloud

One of the best features of MongoDB Atlas is the ability to peer your host VPC on your own Amazon Web Services (AWS) account to your Atlas VPC. VPC peering provides you with the ability to use the private IP range of your hosts and MongoDB Atlas cluster. This allows you to reduce your network exposure and improve security of your data. If you chose to use peering there are some considerations you should think about first in selecting the right IP block for your private traffic.

NOTE - As of the writing of this post, AWS standards require both VPCs to be located in the same AWS region. Example: You can peer us-east-1 VPCs to other us-east-1 VPCs, but you cannot peer a us-east-1 VPC to a us-west-2 VPC.

Host VPC

The host VPC is where you configure the systems that your application will use to connect to your MongoDB Atlas cluster. AWS provides your account with a default VPC for your hosts You may need to modify the default VPC or create a new one to work alongside MongoDB Atlas. Regardless of your use case, it's important to ensure a few basics when configuring your host VPC:

  • Host VPC must be in the same region as your Atlas Cluster

  • Use a RFC-1918 private IP Range

MongoDB Atlas requires your host VPC follow the RFC-1918 standard for creating private ranges. The Internet Assigned Numbers Authority (IANA) has reserved the following three blocks of the IP address space for private internets:

10.0.0.0 - 10.255.255.255 (10/8 prefix)
172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
192.168.0.0 - 192.168.255.255 (192.168/16 prefix)

  • Don't overlap your ranges!

The point of peering is to permit two private IP ranges to work in conjunction to keep your network traffic off the public internet. This will require you to use separate private IP ranges that do not conflict.

AWS standard states the following in their "Invalid VPC Peering" document:

"You cannot create a VPC peering connection between VPCs with matching or overlapping IPv4 CIDR blocks."

Cannot create a VPC peering connection between VPCs with matching or overlapping IPv4 CIDR blocks

MongoDB Atlas VPC

When you create a group in MongoDB Atlas, by default we provide you with an AWS VPC which you can only modify before launching your first cluster. Groups with an existing cluster CANNOT MODIFY their VPC CIDR block– this is to comply with the AWS requirement for peering. By default we create a VPC with IP range 192.168.248.0/21. To specify your IP block prior to configuring peering and launching your cluster, follow these steps:

  1. Sign up for MongoDB Atlas and ensure your payment method is completed.
  2. Click on the SECURITY tab, then select PEERING. You should see a page such as this which shows you that you have not launched a cluster yet: Create new peering connection
  3. Click on the New Peering Connection button. You will be given a new "Peering Connection" window to add your peering details. At the bottom of this page you'll see a section to modify "Your Atlas VPC" Modify your Atlas VPC
  4. If you would like to specify a different IP range, you may use one of the RFC-1918 ranges with the appropriate subnet and enter it here. It's extremely important to ensure that you choose two distinct RFC-1918 ranges. These two cannot overlap their subnets: Choose two distinct RFC-1918 ranges
  5. Click on the INITIATE PEERING button and follow the directions to add the appropriate subnet ranges.

Conclusion

Using peering ensures that your database traffic remains off the public network. This provides you with a much more secure solution allowing you to easily scale up and down without specifying IP addresses each time, and reduces costs on transporting your data from server to server. At any time if you run into problems with this, our support team is always available by clicking the SUPPORT icon in the lower left of your window. Our support team is happy to assist in ensuring your peering connection is properly configured.


About the Author - Jay Gordon

Jay is a Technical Account Manager with MongoDB and is available via our chat to discuss MongoDB Cloud Products at https://cloud.mongodb.com. One of the best features of MongoDB Atlas is the ability to peer your host VPC on your own Amazon Web Services (AWS) account to your Atlas VPC. VPC peering provides you with the ability to use the private IP range of your hosts and MongoDB Atlas cluster. This allows you to reduce your network exposure and improve security of your data. If you chose to use peering there are some considerations you should think about first in selecting the right IP block for your private traffic.

NOTE - As of the writing of this post, AWS standards require both VPCs to be located in the same AWS region. Example: You can peer us-east-1 VPCs to other us-east-1 VPCs, but you cannot peer a us-east-1 VPC to a us-west-2 VPC.

Host VPC

The host VPC is where you configure the systems that your application will use to connect to your MongoDB Atlas cluster. AWS provides your account with a default VPC for your hosts You may need to modify the default VPC or create a new one to work alongside MongoDB Atlas. Regardless of your use case, it's important to ensure a few basics when configuring your host VPC:

  • Host VPC must be in the same region as your Atlas Cluster

  • Use a RFC-1918 private IP Range

MongoDB Atlas requires your host VPC follow the RFC-1918 standard for creating private ranges. The Internet Assigned Numbers Authority (IANA) has reserved the following three blocks of the IP address space for private internets:

10.0.0.0 - 10.255.255.255 (10/8 prefix)
172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
192.168.0.0 - 192.168.255.255 (192.168/16 prefix)

  • Don't overlap your ranges!

The point of peering is to permit two private IP ranges to work in conjunction to keep your network traffic off the public internet. This will require you to use separate private IP ranges that do not conflict.

AWS standard states the following in their "Invalid VPC Peering" document:

"You cannot create a VPC peering connection between VPCs with matching or overlapping IPv4 CIDR blocks."

Cannot create a VPC peering connection between VPCs with matching or overlapping IPv4 CIDR blocks

MongoDB Atlas VPC

When you create a group in MongoDB Atlas, by default we provide you with an AWS VPC which you can only modify before launching your first cluster. Groups with an existing cluster CANNOT MODIFY their VPC CIDR block– this is to comply with the AWS requirement for peering. By default we create a VPC with IP range 192.168.248.0/21. To specify your IP block prior to configuring peering and launching your cluster, follow these steps:

  1. Sign up for MongoDB Atlas and ensure your payment method is completed.
  2. Click on the SECURITY tab, then select PEERING. You should see a page such as this which shows you that you have not launched a cluster yet: Create new peering connection
  3. Click on the New Peering Connection button. You will be given a new "Peering Connection" window to add your peering details. At the bottom of this page you'll see a section to modify "Your Atlas VPC" Modify your Atlas VPC
  4. If you would like to specify a different IP range, you may use one of the RFC-1918 ranges with the appropriate subnet and enter it here. It's extremely important to ensure that you choose two distinct RFC-1918 ranges. These two cannot overlap their subnets: Choose two distinct RFC-1918 ranges
  5. Click on the INITIATE PEERING button and follow the directions to add the appropriate subnet ranges.

Conclusion

Using peering ensures that your database traffic remains off the public network. This provides you with a much more secure solution allowing you to easily scale up and down without specifying IP addresses each time, and reduces costs on transporting your data from server to server. At any time if you run into problems with this, our support team is always available by clicking the SUPPORT icon in the lower left of your window. Our support team is happy to assist in ensuring your peering connection is properly configured.


About the Author - Jay Gordon

Jay is a Technical Account Manager with MongoDB and is available via our chat to discuss MongoDB Cloud Products at https://cloud.mongodb.com.

Programmatic API Management of your MongoDB Atlas Database Clusters

Jay Gordon

Technical, Cloud, API

MongoDB Atlas has a full featured API which allows our users to programmatically create, modify and delete their clusters. We'll go over some basics on how to do this using curl in our bash shell. The goal of this post is to show you how to configure API access and use the basic features available.

Pre-Requisites:

  1. Access to a bash shell on your computer or a VM. (Mac or Linux is fine)
  2. curl (ssl support is required)
  3. MongoDB Atlas Cluster Group - Sign up at https://mongodb.com/atlas
  4. Basic understanding of how to utilize APIs.

API Configuration:

First, log into your MongoDB Atlas Control Panel https://cloud.mongodb.com/user/login

MongoDB Atlas Control Panel

Now navigate to the "Settings" section on the left menu. You'll see under "Personal Settings" a section for "Public API Access", click this and you'll then be presented with options to create an API key and then set a whitelist of IP addresses you are able to access the API from with your authenticated key.

Create API Key

Create the Key

First, we will generate an API key. Click the +GENERATE button at the top of the section next to "API Keys" and provide a description. I will use "test-jay.gordon" but there's no specific information required; this is for your own labeling.

Confirm API Key, MongoDB Atlas

Once the key is generated you'll receive a box as above with the key provided. Save this, it will not be repeated for you again. If you are using git, do not check this into to a public repository for security reasons.

Whitelist IP

In this situation, I'll just want to allow "ALL" (0.0.0.0/0) since I am using this temporarily, however if you have a specific subnet of servers that you expect to invoke these functions from, you'll want to whitelist these particular IPs.

Click the ADD+ button next to the API Whitelist section and enter any IP you would like to whitelist with the appropriate CIDR annotation. You can do this multiple times for multiple ranges or IP addresses.

Add Whitelist Entry

Click confirm and then we will be ready to start working with our API now that we have the Key and our whitelist configured:

Public API Access

As you can see, we obscure the full key to ensure it is secure. Protect this resource always, if you no longer require it you may DISABLE or DELETE it by clicking the gear icon under "Actions":

Disable or delete API keys

Create a Cluster:

We'll now use the command prompt on our local computer to do the following tasks:

  1. Create a JSON file with our configuration requirements for our cluster
  2. Write a small curl http POST against the API to create the cluster
  3. Write a small curl http GET against the API to view our cluster

If you review [the spec for our MongoDB Atlas Clusters API](https://docs.atlas.mongodb.com/reference/api/clusters/ "the spec for our MongoDB Atlas Clusters API') you'll see the basics of how to create a curl command for creating a cluster:

POST /api/atlas/v1.0/groups/GROUP-ID/clusters

You must include the following fields and their values when creating a new cluster:

  • name
  • providerSettings.instanceSizeName
  • providerSettings.providerName
  • providerSettings.regionName
  • backupEnabled

All other editable fields are optional. If you do not specify an optional field, MongoDB Atlas uses the field’s current default value. To view the current default values used for new clusters: open the Atlas interface; click the button to add a cluster; view the default selections; close the window without saving changes.

I want to make this easy for myself for the future, so I am going to place these configuration requirements in a simple JSON formatted text file:

{
  "name" : "DataStore",
  "numShards" : 1,
  "replicationFactor" : 3,
  "providerSettings" : {
    "providerName" : "AWS",
    "regionName" : "US_EAST_1",
    "instanceSizeName" : "M30",
    "diskIOPS" : 120,
    "encryptEBSVolume" : false
  },
  "diskSizeGB" : 40,
  "backupEnabled" : true
}

As you can see I have placed the required values for the cluster's name, replication factor, shard total, name of our hosting provider, the region within AWS, the instance size, the disk size, the total amount of IOPS and backups enabled. I saved the file as "json_out.txt" for myself, but you can call it whatever you like. Now let's build our curl statement and execute it. We require two more bits of information to finalize this.

  1. Your username for MongoDB Atlas
  2. Your Atlas group ID. Your username is what you authenticated with when you first logged in to the MongoDB Atlas control panel. The Group ID can be found by going to "Settings" and clicking "Group Settings" on the left side of the page. At the very top, you should see a GROUP ID string:

Group settings

The string above is an example of my group, so be sure to get yours! Now we can build our curl and get our cluster built. Feel free to type this in a editor or just in the bash shell.I prefer to do so in vim so I can save it and modify if I make a mistake.
Now let's look at a few parts of our curl command:

curl -i -u "jay.gordon:$APIKEY" --digest -H "Content-Type: application/json" -X POST "https://cloud.mongodb.com/api/atlas/v1.0/groups/575ece95e4b0ec4f28db42ca/clusters" --data @json_out.txt
  • curl: this is the binary that's running the command for us
  • "jay.gordon:$APIKEY": here i added APIKEY as a variable as to not share it with you, but you may replace it with the key you received earlier (feel free to create a variable in bash if you'd like)
  • https URI: this is the uri we are executing our API against, we've provided our group ID as to ensure we've done this against the right resource
  • --data @json_out.txt: the JSON file containing the requirements for how our cluster will be built.

We're ready to execute the command, let's build our MongoDB Atlas M30 cluster named DataStore with 40 GB of disk, backups enabled, IOPS of 120 and 3 replicas in total.

bash-3.2$ curl -i -u "jay.gordon:$APIKEY" --digest -H "Content-Type: application/json" -X POST "https://cloud.mongodb.com/api/atlas/v1.0/groups/575ece95e4b0ec4f28db42ca/clusters" --data @json_out.txt

Looks like it worked! Here's our output:

HTTP/1.1 201 Created
Date: Fri, 09 Dec 2016 16:41:16 GMT
Content-Length: 506
Content-Type: application/json
{
  "backupEnabled": true,
  "diskSizeGB": 40,
  "groupId": "575ece95e4b0ec4f28db42ca",
  "links": [
    {
      "href": "https:\/\/cloud.mongodb.com\/api\/atlas\/v1.0\/groups\/575ece95e4b0ec4f28db42ca\/clusters\/DataStore",
      "rel": "self"
    }
  ],
  "mongoDBMajorVersion": "3.2",
  "mongoDBVersion": "3.2.11",
  "mongoURIUpdated": "2016-12-09T16:41:16Z",
  "name": "DataStore",
  "numShards": 1,
  "providerSettings": {
    "providerName": "AWS",
    "diskIOPS": 120,
    "encryptEBSVolume": false,
    "instanceSizeName": "M30",
    "regionName": "US_EAST_1"
  },
  "replicationFactor": 3,
  "stateName": "CREATING"
}

Now allow 5-10 minutes for your cluster to build. Once you've finished you can then list your available clusters via a simple GET. There's no need to POST anything here:

curl -i -u "jay.gordon:$APIKEY" --digest "[https://cloud.m575ece95e4b0ec4f28db42ca/clusters](https://cloud.m575ece95e4b0ec4f28db42ca/clusters)"

HTTP/1.1 200 OK
Date: Fri, 09 Dec 2016 16:45:23 GMT
Content-Length: 678
Content-Type: application/json
{
  "links": [
    {
      "href": "https:\/\/cloud.mongodb.com\/api\/atlas\/v1.0\/groups\/575ece95e4b0ec4f28db42ca\/clusters?pageNum=1&itemsPerPage=100",
      "rel": "self"
    }
  ],
  "results": [
    {
      "backupEnabled": true,
      "diskSizeGB": 40,
      "groupId": "575ece95e4b0ec4f28db42ca",
      "links": [
        {
          "href": "https:\/\/cloud.mongodb.com\/api\/atlas\/v1.0\/groups\/575ece95e4b0ec4f28db42ca\/clusters\/DataStore",
          "rel": "self"
        }
      ],
      "mongoDBMajorVersion": "3.2",
      "mongoDBVersion": "3.2.11",
      "mongoURIUpdated": "2016-12-09T16:41:16Z",
      "name": "DataStore",
      "numShards": 1,
      "providerSettings": {
        "providerName": "AWS",
        "diskIOPS": 120,
        "encryptEBSVolume": false,
        "instanceSizeName": "M30",
        "regionName": "US_EAST_1"
      },
      "replicationFactor": 3,
      "stateName": "CREATING"
    }
  ],
  "totalCount": 1
}

We can see it's creating and even programmatically add some delay into our script if we want to then load data once the cluster has finished building

Delete a Cluster:

We know how to list our clusters, so we can use the API to easily destroy the cluster when we are done with them using another curl POST:

bash-3.2$ curl -i -u "jay.gordon:$APIKEY" --digest -X DELETE 
"[https://cloud.mongodb.com/api/atlas/v1.0/groups/575ece95e4b0ec4f28db42ca/clusters/DataStore](https://cloud.mongodb.com/api/atlas/v1.0/groups/575ece95e4b0ec4f28db42ca/clusters/DataStore)"

Here's our output showing we've authenticated and completed our delete process:

HTTP/1.1 202 Accepted
Date: Fri, 09 Dec 2016 16:54:57 GMT
Content-Length: 2
Content-Type: application/json

Even More Resources:

You are not limited to just creating and deleting your cluster. You can use the API to modify other aspects such as the disk size or users while a cluster has already been created. You can review all of our API resources by going to the MongoDB Atlas API Documentation. Want even more MongoDB Atlas information? Check out M034 at MongoDB University for videos on how to use Atlas and start spinning up new MongoDB Clusters!

About the Author - Jay Gordon

Jay is a Technical Account Manager with MongoDB and is available via our chat to discuss MongoDB Cloud Products at https://cloud.mongodb.com.

Questions we got at AWS re:Invent

Last week, the MongoDB team had the pleasure of attending the world’s foremost cloud computing conference. At AWS re:Invent in Las Vegas, we heard our CTO explain how Thermo Fisher reduced mass spectrometry experiment times from days to minutes, we announced the general availability of MongoDB 3.4, and we formally introduced MongoDB Atlas to the AWS community.

Here are some of the most popular questions we were asked last week:

“I’m new to cloud technology. Tell me…”

Where does my application server live and where does my database live?
Depending on your architecture, your app server may run on a different machine or in a different data center. For most applications, latency can be significantly reduced by spinning up your app server and database in regions closest to your user traffic. For example, MongoDB Atlas can currently run in any of 4 supported Amazon regions, with more being added soon!

What’s the easiest way to migrate to the cloud?
As Amazon dramatically revealed in their keynote this year, cloud migration is a very hot topic. Short of uploading all of your data to an Amazon 18-wheeler, there are many options for migration. Depending on your availability requirements, this can be as simple as creating a copy of your data in the cloud and re-routing traffic to the cloud-hosted database. For more mission critical deployments, migration can be achieved with no downtime. We also provide tailored consulting services to help you create a migration path.

“I’m new to MongoDB. Can you explain…”

What are secondary indexes and how do I use them?
Indexes are powerful tools that let you significantly improve query performance by avoiding full collection scans. Secondary indexes can be added to highly-queried fields (or sets of fields) to extend these performance gains to additional queries, aggregations, or sorts. Adding too many indexes can unnecessarily harm read/write performance, so we recommending using MongoDB Compass to identify which indexes are being used and where additional indexes would be most beneficial.

What are the benefits of MongoDB’s native aggregation functionality?
MongoDB’s aggregation framework gives you the capability to run advanced analytics processing right inside the database, rather than in your application. Since MongoDB 2.2, the aggregation framework has been significantly expanded to include many of the rich aggregation features you’d expect from a relational database.

Does MongoDB offer support?
Yes. Creating a MongoDB Atlas account means you have access to assistance with the UI and support for any connectivity issues. You also have our global site reliability engineering team focused on monitoring and ensuring availability of the service.

For support of the underlying database software, we recommend a MongoDB Enterprise Advanced or MongoDB Professional subscription. We also offer a subscription designed specifically for MongoDB Atlas customers. These subscriptions include access to our support engineers 24 hours a day, 365 days a year. Our support team not only helps you troubleshoot specific questions, but can also guide you through upgrades and suggest optimizations. Support plans are also available for applications in development. Learn more about MongoDB support.

“MongoDB Atlas looks great! Tell me more about…”

How secure is MongoDB Atlas? Is my data encrypted?
MongoDB Atlas is secure by default. Each MongoDB Atlas group is provisioned into its own AWS Virtual Private Cloud, isolating your data and underlying systems from other MongoDB Atlas users. You can even directly peer your MongoDB Atlas VPC with your application servers deployed to AWS using private IP addresses.

We also configure network encryption and access control by default, and IP whitelists (enforced with AWS Security Groups) let you limit access to a specific range of IP addresses. MongoDB engineers automatically apply security patches as soon as they are released.

Finally, you can elect to encrypt your storage volumes at no additional cost using using AWS's encrypted EBS volumes.

How does MongoDB Atlas handle backups?
Backups can be configured optionally and are billed by the size of the data being backed up. Backups in MongoDB Atlas are continuous and maintained seconds behind the production cluster, meaning you can conduct point-in-time restores.

MongoDB Atlas backups are stored in MongoDB’s private data centers for added redundancy.

“I’m thinking about using DynamoDB…”

How does MongoDB Atlas compare?
There are a few features in MongoDB Atlas that our users find valuable:

  1. Rich query language: The MongoDB Query language gives you the querying capabilities you’d expect from a SQL-based database with the flexibility and performance of NoSQL.
  2. Native aggregation: MongoDB’s aggregation framework gives you the capability to run advanced analytics processing right inside the database, rather than in your application.
  3. Secondary indexing support: Support for secondary indexes allows you to extend the performance gains of traditional indexing to additional fields or field sets.
  4. Robust tooling: MongoDB was released in 2009 and, in addition to the vast ecosystem of tooling that lets you support, monitor, and visualize your deployment with confidence, MongoDB Atlas is built using the expertise gained from years of building enterprise-grade products for the most complex, mission critical deployments.
  5. Choice of infrastructure vendor (coming soon!): With upcoming support for Google Cloud Platform and Microsoft Azure, you’ll be able to choose where your deployment lives based on your app requirements.

“I’m currently using a 3rd party hosted MongoDB service...”

How does MongoDB Atlas compare?
MongoDB Atlas is built and supported by the team who builds the database. This means you get access to the latest security patches, major releases, and features when they’re released—not a year later!

MongoDB Atlas is also the most cost-effective way to run MongoDB 3.2 or 3.4 with our WiredTiger storage engine. With hourly billing and independently-tuneable memory, storage size, and disk I/O, you only pay for what you need.

All MongoDB Atlas clusters are hosted in isolated, secure VPCs which mean you aren’t sharing bandwidth with other databases within the same virtual machine.

--

Download MongoDB Atlas Best Practice Guide


Thermo Fisher moves into the cloud with MongoDB Atlas & AWS

Biotechnology giant uses MongoDB Atlas and an assortment of AWS technologies and services to reduce experiment times from days to minutes.

Thermo Fisher (NYSE: TMO) is moving its applications to the public cloud as part of a larger Thermo Fisher Cloud initiative with the help of offerings such as MongoDB Atlas and Amazon Web Services. Last week, our CTO & Cofounder Eliot Horowitz presented at AWS re:Invent with Thermo Fisher Senior Software Architect Joseph Fluckiger on some of the transformative benefits they’re seeing internally and across customers. This recap will cover Joseph’s portion of the presentation.

Joseph started by telling the audience that Thermo Fisher is maybe the largest company they’d never heard of. Thermo Fisher employs over 51,000 people across 50 countries, with over $17 billion in revenues in 2015. Formed a decade ago through the merger of Thermo Electron & Fisher Scientific, it is one of the leading companies in the world in the genetic testing and precision laboratory equipment markets.

The Thermo Fisher Cloud is a new offering built on Amazon Web Services consisting of 35 applications supported by over 150 Thermo Fisher developers. It allows customers to streamline their experimentation, processing, and collaboration workflows, fundamentally changing how researchers and scientists work. It serves 10,000 unique customers and stores over 1.3 million experiments, making it one of the largest cloud platforms for the scientific community. For internal teams, Thermo Fisher Cloud has also streamlined development workflows, allowing developers to share more code and create a consistent user experience by taking advantage of a microservices architecture built on AWS.

One of the precision laboratory instruments the company produces is a mass spectrometer, which works by taking a sample, bombarding it with electrons, and separating the ions by accelerating the sample and subjecting it to an electric or magnetic field. Atoms within the sample are then sorted by mass and charge and matched to known values to help customers figure out the exact composition of the sample in question. Joseph’s team develops the software powering these machines.

Thermo Fisher mass spectrometers are used to:

  • Detect pesticides & pollutants — anything that’s bad for you
  • Identify organic molecules on extraplanetary missions
  • Process samples from athletes to look for performance-enhancing substances
  • Drive product authenticity tests
  • And more

Thermo Fisher mass spectrometers

*Mass Spectrometry Results*

MS Instrument Connect

*MS Instrument Connect*

During the presentation, Joseph showed off one application in the Thermo Fisher Cloud called MS Instrument Connect, which allows customers to see the status of their spectrometry instruments with live experiment results from any mobile device or browser. No longer does a scientist have to sit at the instrument to monitor an ongoing experiment. MS Instrument Connect also allows Thermo Fisher customers to easily query across instruments and get utilization statistics. Supporting MS Instrument Connect and marshalling data back and forth is a MongoDB cluster deployed in MongoDB Atlas, our hosted database as a service.

Joseph shared that MongoDB is being used across multiple projects in Thermo Fisher and the Thermo Fisher Cloud, including Instrument Connect, which was originally deployed on DynamoDB. Other notable applications include the Thermo Fisher Online Store (which was migrated from Oracle), Ion Reporter (which was migrated from PostgreSQL), and BioPharma Finder (which is being migrated from SQL Lite).

Thermo Fisher apps moving to MongoDB

*Thermo Fisher apps moving to MongoDB*

To support scientific experiments, Thermo Fisher needed a database that could easily handle a wide variety of fast-changing data and allow its customers to slice and dice their data in many different ways. Experiment data is also very large; each experiment produces millions of “rows” of data. When explaining why MongoDB was chosen for such a wide variety of use cases across the organization, Joseph called the database a “swiss army knife” and cited the following characteristics:

  • High performance
  • High flexibility
  • Ability to improve developer productivity
  • Ability to be deployed in any environment, cloud or on premises

What really got the audience’s attention was a segment where Joseph compared incumbent databases that Thermo Fisher had been using with MongoDB.

MongoDB compared to MySQL (Aurora)

If I were to reduce my slides down to one, this would be that slide,” Joseph stated, “This is absolutely phenomenal. What we did was we inserted data into MongoDB & Aurora and with only 1 line of code, we were able to beat the performance of MySQL.

Inserting data: MongoDB vs. MySQL

*"If I were to reduce my slides down to one, this would be that slide.”*

In additional to delivering 6x higher performance with 40x less code, MongoDB also helped reduce the schema complexity of the app.

MongoDB compared to SQL Lite

For the mass spectrometry application used in performance enhancing drug testing, Thermo Fisher rewrote the data layer from SQL Lite to MongoDB and reduced their code by a factor of about 3.5.

MongoDB compared to DynamoDB MongoDB compared to MongoDB

Joseph then compared MongoDB to DynamoDB, stating that while both databases are great and easy to deploy, MongoDB offers a more powerful query language for richer queries to be run and allows for much simpler schema evolution. He also reminded the audience that MongoDB can be run in any environment while DynamoDB can only be run on AWS.

Finally, Joseph showed an architecture diagram showing how MongoDB is being used with several AWS technologies and services (including AWS Lambda, Docker, & Apache Spark) to parallelize algorithms and significantly reduce experiment processing times.

Parallel data processing

*Reducing experiment times from days to minutes*

He concluded his presentation by explaining why Thermo Fisher is pushing applications to MongoDB Atlas, citing its ease of use, the seamless migration process, and how there has been no downtime, even when reconfiguring the cluster. The company began testing MongoDB Atlas around its release date in early July and began launching production applications on the service in September. With the time Thermo Fisher team is saving by using MongoDB Atlas (that would have otherwise been spent on writing and optimizing their data layer), they’re able to invest more time in improving their algorithms, their customer experience, and their processing infrastructure.

Anytime I can use a service like MongoDB Atlas, I’m going to take that so that we at Thermo Fisher can focus on what we’re good at, which is being the leader in serving science.

To view Joseph & Eliot’s AWS re:Invent presentation in its entirety, click here.

--

Get started with MongoDB Atlas today!


How to Evaluate MongoDB 3.4 Using Your Existing Deployment in MongoDB Atlas

Andrew Davidson

Cloud

MongoDB 3.4 was just released with support for graph processing, multi-language collations, read-only views, faceted navigation, powerful new aggregation operators, numerous performance improvements, and more. MongoDB 3.4 is available in the MongoDB Atlas database service today, so you can quickly and easily deploy new clusters and try it out. But what’s the best way to evaluate MongoDB 3.4 against an existing production workload you have running in MongoDB Atlas? As with any software upgrade, you should first test your application.

MongoDB Atlas automates spinning up new production-ready MongoDB clusters, manages backups and restores, and automates on-demand upgrades of MongoDB from 3.2 to 3.4 at any time. By combining these capabilities, you can very easily:

  1. Spin up a test environment from a backup
  2. Upgrade the test environment from MongoDB 3.2 to MongoDB 3.4
  3. Confirm that your testing and performance suite pass on MongoDB 3.4

Specifically, in this tutorial we’re going to:

  1. Start with a production MongoDB Atlas cluster (“Cluster0”) running MongoDB 3.2 with the Atlas backup service enabled.
  2. Spin up a new MongoDB Atlas test cluster, initially on MongoDB 3.2, (“cluster34upgrade”) with sufficient storage to restore the production cluster’s backup.
  3. Restore the most recent backup snapshot from the production cluster into the test cluster.
  4. Upgrade the test cluster from MongoDB 3.2 to MongoDB 3.4
  5. Execute our test suite against the MongoDB 3.4 test cluster.
  6. If all goes well, we can upgrade our production cluster to MongoDB 3.4 and terminate our test cluster. This upgrade will be made in-place, and will be completed without application downtime.

1. Start with a MongoDB Atlas cluster (“Cluster0”) running MongoDB 3.2 and with the Atlas backup service enabled.

For purposes of this tutorial, we have pre-loaded this cluster and have it under active load.

MongoDB Atlas cluster (“Cluster0”) running MongoDB 3.2 and with the Atlas backup service enabled

** 2. Create a new MongoDB Atlas cluster, initially on MongoDB 3.2, (“cluster34upgrade”) with enough storage to restore the production cluster’s backup.**

We’ll follow the “Add new cluster” flow and deploy an M20 instance with 40GB of storage, just like our production cluster in this example. Note that for your live production deployments, we do recommend that you run on a MongoDB Atlas M30 or higher.

Create a new MongoDB 3.4 Atlas Cluster

After the cluster deploys, we’ll see the following:

MongoDB Atlas cluster after deployment

3. Restore the most recent backup snapshot from the production cluster into the test upgrade cluster.

We’ll navigate into the “Backup” tab and find the backup for our production cluster.

Backup for production cluster

Click into the Options menu (“...”) and select “Restore”.

Choose the most recent snapshot.

Select “Restore my snapshot” to restore it into our test MongoDB cluster.

Select the test upgrade cluster as the destination for the restore.

Now the automated restore into the test cluster will be performed. Depending on the size of the data files being restored, this operation could take some time.

Automated restore into test cluster

After the restore finishes, we’ll see the following:

Test cluster restore complete

4. Upgrade the test upgrade cluster from MongoDB 3.2 to MongoDB 3.4

Now we can initiate our testing procedures to understand the implications at the application tier during the online upgrade process. To perform the upgrade, we’ll select “Configuration” on the new test cluster which we’ve just restored our backup into:

New test cluster configuration

Next, we’ll click “Change version” and select “MongoDB 3.4 with WiredTiger”

Test cluster 3.4 upgrade

After selecting MongoDB 3.4, we’ll see the important notice that downgrading from MongoDB 3.4 to 3.2 will not be possible. This is one of the reasons why performing a test cluster upgrade is highly recommended.

Cluster 3.4 upgrade

After deploying we’ll see the blue bar during the upgrade:

Cluster 3.4 deployment - blue part appears

And quickly we’ll have an upgraded cluster running on MongoDB 3.4:

Upgraded cluster running on MongoDB 3.4

5. Execute our test suite against the MongoDB 3.4 test upgrade cluster.

We will go ahead and confirm connectivity of all of the clients that utilize our cluster and put the new cluster through our entire test and performance suite if it wasn't already running since before initiating the upgrade.

Test suite against MongoDB 3.4 test upgrade cluster

We’ll either confirm that we’re ready for MongoDB 3.4, or potentially unearth areas for tuning or necessary driver upgrades, depending on our usage.

6. If our tests pass, we can upgrade our production cluster to MongoDB 3.4 and terminate our test cluster.

We’ll click “Configure” on our production cluster

Configure production cluster

Because MongoDB Atlas is built on replica sets, the upgrade process performs a rolling restart of the cluster. As a result, the production application remains available and online.

And we can terminate our test cluster if we no longer want to maintain it for testing.

Next steps

If you’re an existing MongoDB Atlas user on MongoDB 3.2, we encourage you to review compatibility changes in MongoDB 3.4 and the full release notes.

You can then follow the process outlined above to restore a backup to a new cluster and upgrade it to MongoDB 3.4.

If you’re new to MongoDB Atlas, you can easily deploy your first MongoDB 3.4 cluster:

Just click the "Build a new cluster" button.

Then select MongoDB 3.4 in the build cluster form.

Deploy MongoDB 3.4 cluster

Listen to Eliot Horowitz on the Future of the Database

Peter Zawistowicz

Cloud

"The main motivation for people trying out MongoDB and adopting MongoDB really came around from developers wanting to be more productive."

Six years after MongoDB was open sourced, we’re still thinking about how to empower software engineers by making app development more efficient and productive. Our CTO and co-founder, Eliot Horowitz, recently sat down with Jeff Meyerson, host of Software Engineering Daily, to talk about how the evolution of MongoDB and its ecosystem has been propelled by the goal of developer productivity.

MongoDB is best known for its JSON-based documents and Eliot explains that this data model provides a "fundamentally easier data structure for developers to work with. It more naturally suits the way programming languages work and thee way people think. No one thinks about breaking things up into rows and columns but they do think of things as structures."

{
'_id' : 1,
    'name' : { 'first' : 'John', 'last' : 'Backus' },
    'contribs' : [ 'Fortran', 'ALGOL', 'Backus-Naur Form', 'FP' ],
    'awards' : [
        {
            'award' : 'W.W. McDowell Award',
            'year' : 1967,
            'by' : 'IEEE Computer Society'
        }, {
            'award' : 'Draper Prize',
            'year' : 1993,
            'by' : 'National Academy of Engineering'
        }
    ]
}
*An example JSON document*

By basing all interactions with data on the document model the creators of MongoDB made it easier for them to store and work with data, and therefore easier to get more value out of it.

Reducing friction for developers doesn't just reduce developer headaches, it also has a direct impact on the bottom line. Since the 1980s hardware and infrastructure costs have fallen, the value of the individual developer has soared. Ensuring individual engineers are productive is critical to today’s businesses.

This is the story that Eliot and MongoDB have been telling for years, but it's particularly interesting to hear Eliot discuss how MongoDB has evolved alongside two other major trends in software engineering: cloud computing and service-oriented architectures (and, by extension, microservices).

Not coincidentally, both of these paradigms are also rooted in unburdening the individual developer. Cloud computing reduces things like lengthy infrastructure provisioning times whereas microservices decouple application logic to allow for faster iteration and feature development. As Eliot points out, it also fundamentally changes the way apps are built as developers are able to use third party services in place of coding necessary functionality from scratch.

Listen in to Eliot's conversation with Jeff as, in addition to talking about the evolution of MongoDB, Eliot talks about the future of the database as well as how we use our own products internally in a hybrid cloud configuration.

If you’re interested in listening to Jeff’s other conversations around the software landscape, Software Engineering Daily comprises hours of fascinating technical content and many of our own engineers are already avid listeners! I hope you'll listen in as this episode kicks off MongoDB’s first podcast partnership. We’re looking forward to engaging with you through this medium. As always, please give us suggestions for new ways to contribute to the ever-growing MongoDB community!

Listen to Eliot on the Software Engineering Daily podcast

Can't listen? You can view the transcript here.

MongoDB 3.4 is available in Atlas and Cloud Manager

MongoDB

Cloud

Concurrent with the release of MongoDB 3.4.0 yesterday, we are happy to announce that Atlas clusters can be upgraded to MongoDB 3.4 immediately, if desired (you can stay on 3.2, and we’ll keep you up to date on that, too!).

https://webassets.mongodb.com/_com_assets/blog/tblr/tumblr_ohgkb0ENB31sdaytmo1_1280.png

Additionally, you can now add 3.4 to your version manager in Cloud Manager.

We here in the Cloud team at MongoDB are super excited about the release of 3.4, and we hope you are too.

Developing a Facebook Chatbot with AWS Lambda and MongoDB Atlas

This post is part of our Road to re:Invent series. In the weeks leading up to AWS re:Invent in Las Vegas this November, we'll be posting about a number of topics related to running MongoDB in the public cloud.

Road to re:Invent

Introduction

While microservices have been the hot trend over the past couple of years, serverless architectures have been gaining momentum by providing a new way to build scalable, responsive and cost effective applications. Serverless computing frees developers from the traditional cost and effort of building applications by automatically provisioning servers and storage, maintaining infrastructure, upgrading software, and only charging for consumed resources. More insight into serverless computing can be found in this whitepaper.

Amazon’s serverless computing platform, AWS Lambda, lets you run code without provisioning and running servers. MongoDB Atlas is Hosted MongoDB as a Service. MongoDB Atlas provides all the features of the database without the heavy operational lifting. Developers no longer need to worry about operational tasks such as provisioning, configuration, patching, upgrades, backups, and failure recovery. In addition, MongoDB Atlas offers elastic scalability, either by scaling up on a range of instance sizes or scaling out with automatic sharding, all with no application downtime. Together, AWS Lambda and MongoDB Atlas allow developers to spend more time developing code and less time managing the infrastructure.

Learn how to easily integrate an AWS Lambda Node.js function with a MongoDB database in this tutorial.

To demonstrate the power of serverless computing and managed database as a service, I’ll use this blog post to show you how to develop a Facebook chatbot that responds to weather requests and stores the message information in MongoDB Atlas.

Facebook chatbot

Setting Up MongoDB Atlas

MongoDB Atlas provides multiple size options for instances. Within an instance class, there is also the ability to customize storage capacity and storage speed, as well as to use encrypted storage volumes. The number of virtual CPUs (vCPUs) – where a vCPU is a shared physical core or one hyperthread – increases as the instance class grows larger.

The M10, M20, and M30 instances are excellent for development and testing purposes, but for production it is recommended to use instances higher than M30. The base options for instances are:

  • M0 - Variable RAM, 512 MB Storage
  • M10 – 2 GB RAM, 10 GB Storage, 1 vCPU
  • M20 – 4 GB RAM, 20 GB Storage, 2 vCPUs
  • M30 – 8 GB RAM, 40 GB Storage, 2 vCPUs
  • M40 – 16 GB RAM, 80 GB Storage, 4 vCPUs
  • M50 – 32 GB RAM, 160 GB Storage, 8 vCPUs
  • M60 – 64 GB RAM, 320 GB Storage, 16 vCPUs
  • M100 – 160 GB RAM, 1000 GB Storage, 40 vCPUs

Register with MongoDB Atlas and use the intuitive user interface to select the instance size, region, and features you need.

Connecting MongoDB Atlas to AWS Lambda

Important note: VPC Peering is not available with MongoDB Atlas free tier (M0). If you use an M0 cluster, allow any IP to connect to your M0 cluster and switch directly to the Set up AWS Lambda section.

MongoDB Atlas enables VPC (Virtual Private Cloud) peering, which allows you to easily create a private networking connection between your application servers and backend database. Traffic is routed between the VPCs using private IP addresses. Instances in either VPC can communicate with each other as if they are within the same network. Note, VPC peering requires that both VPCs be in the same region. Below is an architecture diagram of how to connect MongoDB Atlas to AWS Lambda and route traffic to the Internet.

AWS VPC Peering Architecture
Figure 1: AWS Peering Architecture Architecture

For our example, a Network Address Translation (NAT) Gateway and Internet Gateway (IGW) is needed as the Lambda function will require internet access to query data from the Yahoo weather API. The Yahoo weather API will be used to query real-time weather data from the chatbot. The Lambda function we will create resides in the private subnet of our VPC. Because the subnet is private, the IP addresses assigned to the Lambda function cannot be used in public. To solve this issue, a NAT Gateway can be used to translate private IP addresses to public, and vice versa. An IGW is also needed to provide access to the internet.

The first step is to set up an Elastic IP address, which will be the static IP address of your Lambda functions to the outside world. Go to Services->VPC->Elastic IPs, and allocate a new Elastic IP address.

Elastic IP address set up

Next we will create a new VPC, which you will attach to your Lambda function.

Go to Services->VPC->Start VPC Wizard.

After clicking VPC wizard, select VPC with Public and Private Subnets.

Let’s configure our VPC. Give the VPC a name (e.g., “Chatbot App VPC”), select an IP CIDR block, choose an Availability Zone, and select the Elastic IP you created in the previous step. Note, the IP CIDR that you select for your VPC, must not overlap with the Atlas IP CIDR. Click Create VPC to set up your VPC. The AWS VPC wizard will automatically set up the NAT and IGW.

You should see the VPC you created in the VPC dashboard.

Go to the Subnets tab to see if your private and public subnets have been set up correctly.

Click on the Private Subnet and go to the Route Table tab in the lower window. You should see the NAT gateway set to 0.0.0.0/0, which means that messages sent to IPs outside of the private subnet will be routed to the NAT gateway.

Next, let's check the public subnet to see if it’s configured correctly. Select Public subnet and the Route Table tab in the lower window. You should see 0.0.0.0/0 connected to your IGW. The IGW will enable outside internet traffic to be routed to your Lambda functions.

Now, the final step is initiating a VPC peering connection between MongoDB Atlas and your Lambda VPC. Log in to MongoDB Atlas, and go to Clusters->Security->Peering->New Peering Connection.

After successfully initiating the peering connection, you will see the Status of the peering connection as Waiting for Approval.

Go back to AWS and select Services->VPC->Peering Connections. Select the VPC peering connection. You should see the connection request pending. Go to Actions and select Accept Request.

Once the request is accepted, you should see the connection status as active.

We will now verify that the routing is set up correctly. Go to the Route Table of the Private Subnet in the VPC you just set up. In this example, it is rtb-58911e3e. You will need to modify the Main Route Table (see Figure 1) to add the VPC Peering connection. This will allow traffic to be routed to MongoDB Atlas.

Go to the Routes tab and select Edit->Add another route. In the Destination field, add your Atlas CIDR block, which you can find in the Clusters->Security tab of the MongoDB Atlas web console:

Click in the Target field. A dropdown list will appear, where you should see the peering connection you just created. Select it and click Save.

Now that the VPC peering connection is established between the MongoDB Atlas and AWS Lambda VPCs, let’s set up our AWS Lambda function.

Set Up AWS Lambda

Now that our MongoDB Atlas cluster is connected to AWS Lambda, let’s develop our Lambda function. Go to Services->Lambda->Create Lambda Function. Select your runtime environment (here it’s Node.js 4.3), and select the hello-world starter function.

Select API Gateway in the box next to the Lambda symbol and click Next.

Create your API name, select dev as the deployment stage, and Open as the security. Then click Next.

In the next step, make these changes to the following fields:

  • Name: Provide a name for your function – for example, lambda-messenger-chatbot
  • Handler: Leave as is (index.handler)
  • Role: Create a basic execution role and use it (or use an existing role that has permissions to execute Lambda functions)
  • Timeout: Change to 10 seconds. This is not necessary but will give the Lambda function more time to spin up its container on initialization (if needed)
  • VPC: Select the VPC you created in the previous step
  • Subnet: Select the private subnet for the VPC (don’t worry about adding other subnets for now)
  • Security Groups: the default security group is fine for now

Press Next, review and create your new Lambda function.

In the code editor of your Lambda function, paste the following code snippet and press the Save button:

'use strict';

var VERIFY_TOKEN = "mongodb_atlas_token";

exports.handler = (event, context, callback) => {
  var method = event.context["http-method"];
  // process GET request
  if(method === "GET"){
    var queryParams = event.params.querystring;
    var rVerifyToken = queryParams['hub.verify_token']
    if (rVerifyToken === VERIFY_TOKEN) {
      var challenge = queryParams['hub.challenge']
      callback(null, parseInt(challenge))
    }else{
      callback(null, 'Error, wrong validation token');
    }   
  }
};
 

This is the piece of code we'll need later on to set up the Facebook webhook to our Lambda function.

Set Up AWS API Gateway

Next, we will need to set up the API gateway for our Lambda function. The API gateway will let you create, manage, and host a RESTful API to expose your Lambda functions to Facebook messenger. The API gateway acts as an abstraction layer to map application requests to the format your integration endpoint is expecting to receive. For our example, the endpoint will be our Lambda function.

Go to Services->API Gateway->[your Lambda function]->Resources->ANY.

Click on Integration Request. This will configure the API Gateway to properly integrate Facebook with your backend application (AWS Lambda). We will set the integration endpoint to lambda-messenger-bot, which is the name I chose for our Lambda function.

Uncheck Use Lambda Proxy Integration and navigate to the Body Mapping Templates section.

Select When there are no templates defined as the Request body passthrough option and add a new template called application/json. Don't select any value in the Generate template section, add the code below and press Save:

##  See http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html
##  This template will pass through all parameters including path, querystring, header, stage variables, and context through to the integration endpoint via the body/payload
#set($allParams = $input.params())
{
"body-json" : $input.json('$'),
"params" : {
#foreach($type in $allParams.keySet())
    #set($params = $allParams.get($type))
"$type" : {
    #foreach($paramName in $params.keySet())
    "$paramName" : "$util.escapeJavaScript($params.get($paramName))"
        #if($foreach.hasNext),#end
    #end
}
    #if($foreach.hasNext),#end
#end
},
"stage-variables" : {
#foreach($key in $stageVariables.keySet())
"$key" : "$util.escapeJavaScript($stageVariables.get($key))"
    #if($foreach.hasNext),#end
#end
},
"context" : {
    "account-id" : "$context.identity.accountId",
    "api-id" : "$context.apiId",
    "api-key" : "$context.identity.apiKey",
    "authorizer-principal-id" : "$context.authorizer.principalId",
    "caller" : "$context.identity.caller",
    "cognito-authentication-provider" : "$context.identity.cognitoAuthenticationProvider",
    "cognito-authentication-type" : "$context.identity.cognitoAuthenticationType",
    "cognito-identity-id" : "$context.identity.cognitoIdentityId",
    "cognito-identity-pool-id" : "$context.identity.cognitoIdentityPoolId",
    "http-method" : "$context.httpMethod",
    "stage" : "$context.stage",
    "source-ip" : "$context.identity.sourceIp",
    "user" : "$context.identity.user",
    "user-agent" : "$context.identity.userAgent",
    "user-arn" : "$context.identity.userArn",
    "request-id" : "$context.requestId",
    "resource-id" : "$context.resourceId",
    "resource-path" : "$context.resourcePath"
    }
}

The mapping template will structure the Facebook response in the desired format specified by the application/json template. The Lambda function will then extract information from the response and return the required output to the chatbot user. For more information on AWS mapping templates, see the AWS documentation.

Go back to Services->API Gateway->[your Lambda function]->Resources->ANY and select Method Request. In the Settings section, make sure NONE is selected in the Authorization dropdown list. If not, change it NONE and press the small Update button.

Go back to the Actions button for your API gateway and select Deploy API to make your API gateway accessible by the internet. Your API gateway is ready to go.

Set Up Facebook Messenger

Facebook makes it possible to use Facebook Messenger as the user interface for your chatbot. For our chatbot example, we will use Messenger as the UI. To create a Facebook page and Facebook app, go to the Facebook App Getting Started Guide to set up your Facebook components.

To connect your Facebook App to AWS Lambda you will need to go back to your API gateway. Go to your Lambda function and find the API endpoint URL (obscured in the picture below).

Go back to your Facebook App page and in the Add Product page, click on the Get Started button next to the Messenger section. Scroll down and in the Webhooks section, press the Setup webhooks button. A New Page Subscription page window should pop up. Enter your API endpoint URL in the Callback URL text box and in the Verify Token text box, enter a token name that you will use in your Lambda verification code (e.g. mongodb_atlas_token). As the Facebook docs explain, your code should look for the Verify Token and respond with the challenge sent in the verification request. Last, select the messages and messaging_postbacks subscription fields.

Press the Verify and Save button to start the validation process. If everything went well, the Webhooks section should show up again and you should see a Complete confirmation in green:

In the Webhooks section, click on Select a Page to select a page you already created. If you don't have any page on Facebook yet, you will first need to create a Facebook page. Once you have selected an existing page and press the Subscribe button.

Scroll up and in the Token Generation section, select the same page you selected above to generate a page token.

The first time you want to complete that action, Facebook might pop up a consent page to request your approval to grant your Facebook application some necessary page-related permissions. Press the Continue as [your name] button and the OK button to approve these permissions. Facebook generates a page token which you should copy and paste into a separate document. We will need it when we complete the configuration of our Lambda function.

Connect Facebook Messenger UI to AWS Lambda Function

We will now connect the Facebook Messenger UI to AWS Lambda and begin sending weather queries through the chatbot. Below is the index.js code for our Lambda function. The index.js file will be packaged into a compressed archive file later on and loaded to our AWS Lambda function.

"use strict";

var assert = require("assert");
var https = require("https");
var request = require("request");
var MongoClient = require("mongodb").MongoClient;

var facebookPageToken = process.env["PAGE_TOKEN"];
var VERIFY_TOKEN = "mongodb_atlas_token";
var mongoDbUri = process.env["MONGODB_ATLAS_CLUSTER_URI"];

let cachedDb = null;

exports.handler = (event, context, callback) => {
  context.callbackWaitsForEmptyEventLoop = false;

  var httpMethod;

  if (event.context != undefined) {
    httpMethod = event.context["http-method"];
  } else {
    //used to test with lambda-local
    httpMethod = "PUT";
  }

  // process GET request (for Facebook validation)
  if (httpMethod === "GET") {
    console.log("In Get if loop");
    var queryParams = event.params.querystring;
    var rVerifyToken = queryParams["hub.verify_token"];
    if (rVerifyToken === VERIFY_TOKEN) {
      var challenge = queryParams["hub.challenge"];
      callback(null, parseInt(challenge));
    } else {
      callback(null, "Error, wrong validation token");
    }
  } else {
    // process POST request (Facebook chat messages)
    var messageEntries = event["body-json"].entry;
    console.log("message entries are " + JSON.stringify(messageEntries));
    for (var entryIndex in messageEntries) {
      var messageEntry = messageEntries[entryIndex].messaging;
      for (var messageIndex in messageEntry) {
        var messageEnvelope = messageEntry[messageIndex];
        var sender = messageEnvelope.sender.id;
        if (messageEnvelope.message && messageEnvelope.message.text) {
          var onlyStoreinAtlas = false;
          if (
            messageEnvelope.message.is_echo &&
            messageEnvelope.message.is_echo == true
          ) {
            console.log("only store in Atlas");
            onlyStoreinAtlas = true;
          }
          if (!onlyStoreinAtlas) {
            var location = messageEnvelope.message.text;
            var weatherEndpoint =
              "https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in%20(select%20woeid%20from%20geo.places(1)%20where%20text%3D%22" +
              location +
              "%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys";
            request(
              {
                url: weatherEndpoint,
                json: true
              },
              function(error, response, body) {
                try {
                  var condition = body.query.results.channel.item.condition;
                  var response =
                    "Today's temperature in " +
                    location +
                    " is " +
                    condition.temp +
                    ". The weather is " +
                    condition.text +
                    ".";
                  console.log(
                    "The response to send to Facebook is: " + response
                  );
                  sendTextMessage(sender, response);
                  storeInMongoDB(messageEnvelope, callback);
                } catch (err) {
                  console.error(
                    "error while sending a text message or storing in MongoDB: ",
                    err
                  );
                  sendTextMessage(sender, "There was an error.");
                }
              }
            );
          } else {
            storeInMongoDB(messageEnvelope, callback);
          }
        } else {
          process.exit();
        }
      }
    }
  }
};

function sendTextMessage(senderFbId, text) {
  var json = {
    recipient: { id: senderFbId },
    message: { text: text }
  };
  var body = JSON.stringify(json);
  var path = "/v2.6/me/messages?access_token=" + facebookPageToken;
  var options = {
    host: "graph.facebook.com",
    path: path,
    method: "POST",
    headers: { "Content-Type": "application/json" }
  };
  var callback = function(response) {
    var str = "";
    response.on("data", function(chunk) {
      str += chunk;
    });
    response.on("end", function() {});
  };
  var req = https.request(options, callback);
  req.on("error", function(e) {
    console.log("problem with request: " + e);
  });

  req.write(body);
  req.end();
}

function storeInMongoDB(messageEnvelope, callback) {
  if (cachedDb && cachedDb.serverConfig.isConnected()) {
    sendToAtlas(cachedDb, messageEnvelope, callback);
  } else {
    console.log(`=> connecting to database ${mongoDbUri}`);
    MongoClient.connect(mongoDbUri, function(err, db) {
      assert.equal(null, err);
      cachedDb = db;
      sendToAtlas(db, messageEnvelope, callback);
    });
  }
}

function sendToAtlas(db, message, callback) {
  db.collection("records").insertOne({
    facebook: {
      messageEnvelope: message
    }
  }, function(err, result) {
    if (err != null) {
      console.error("an error occurred in sendToAtlas", err);
      callback(null, JSON.stringify(err));
    } else {
      var message = `Inserted a message into Atlas with id: ${result.insertedId}`;
      console.log(message);
      callback(null, message);
    }
  });
}

We are passing the MongoDB Atlas connection string (or URI) and Facebook page token as environment variables so we'll configure them in our Lambda function later on.

For now, clone this GitHub repository and open the README file to find the instructions to deploy and complete the configuration of your Lambda function.

Save your Lambda function and navigate to your Facebook Page chat window to verify that your function works as expected. Bring up the Messenger window and enter the name of a city of your choice (such as New York, Paris or Mumbai).

Store Message History in MongoDB Atlas

AWS Lambda functions are stateless; thus, if you require data persistence with your application you will need to store that data in a database. For our chatbot, we will save message information (text, senderID, recipientID) to MongoDB Atlas (if you look at the code carefully, you will notice that the response with the weather information comes back to the Lambda function and is also stored in MongoDB Atlas).

Before writing data to the database, we will first need to connect to MongoDB Atlas. Note that this code is already included in the index.js file.

function storeInMongoDB(messageEnvelope, callback) {
  if (cachedDb && cachedDb.serverConfig.isConnected()) {
    sendToAtlas(cachedDb, messageEnvelope, callback);
  } else {
    console.log(`=> connecting to database ${mongoDbUri}`);
    MongoClient.connect(mongoDbUri, function(err, db) {
      assert.equal(null, err);
      cachedDb = db;
      sendToAtlas(db, messageEnvelope, callback);
    });
  }
}

sendToAtlas will write chatbot message information to your MongoDB Atlas cluster.

function sendToAtlas(db, message, callback) {
  db.collection("records").insertOne({
    facebook: {
      messageEnvelope: message
    }
  }, function(err, result) {
    if (err != null) {
      console.error("an error occurred in sendToAtlas", err);
      callback(null, JSON.stringify(err));
    } else {
      var message = `Inserted a message into Atlas with id: ${result.insertedId}`;
      console.log(message);
      callback(null, message);
    }
  });
}

Note that the storeInMongoDB and sendToAtlas methods implement MongoDB's recommended performance optimizations for AWS Lambda and MongoDB Atlas, including not closing the database connection so that it can be reused in subsequent calls to the Lambda function.

The Lambda input contains the message text, timestamp, senderID and recipientID, all of which will be written to your MongoDB Atlas cluster. Here is a sample document as stored in MongoDB:

{
  "_id": ObjectId("58124a83c976d50001f5faaa"),
  "facebook": {
    "message": {
      "sender": {
        "id": "1158763944211613"
      },
      "recipient": {
        "id": "129293977535005"
      },
      "timestamp": 1477593723519,
      "message": {
        "mid": "mid.1477593723519:81a0d4ea34",
        "seq": 420,
        "text": "San Francisco"
      }
    }
  }
}

If you'd like to see the documents as they are stored in your MongoDB Atlas database, download MongoDB Compass, connect to your Atlas cluster and visualize the documents in your fbchats collection:

"MongoDB Compass"

Note that we're storing both the message as typed by the user, as well as the response sent back by our Lambda function (which comes back to the Lambda function as noted above).

Using MongoDB Atlas with other AWS Services

In this blog, we demonstrated how to build a Facebook chatbot, using MongoDB Atlas and AWS Lambda. MongoDB Atlas can also be used as the persistent data store with many other AWS services, such as Elastic Beanstalk and Kinesis. To learn more about developing an application with AWS Elastic Beanstalk and MongoDB Atlas, read Develop & Deploy a Node.js App to AWS Elastic Beanstalk & MongoDB Atlas.

To learn how to orchestrate Lambda functions and build serverless workflows, read Integrating MongoDB Atlas, Twilio, and AWS Simple Email Service with AWS Step Functions.

For information on developing an application with AWS Kinesis and MongoDB Atlas, read Processing Data Streams with Amazon Kinesis and MongoDB Atlas.

To learn how to use your favorite language or framework with MongoDB Atlas, read Using MongoDB Atlas From Your Favorite Language or Framework.

About the Author - Raphael Londner

Raphael Londner is a Principal Developer Advocate at MongoDB, focused on cloud technologies such as Amazon Web Services, Microsoft Azure and Google Cloud Engine. Previously he was a developer advocate at Okta as well as a startup entrepreneur in the identity management space. You can follow him on Twitter at @rlondner.

MongoDB Atlas Customer Spotlight: Checkr

MongoDB Atlas continues to explode in popularity amongst our users. One of the big reasons MongoDB has grown to be the fourth largest database in the world is the desire by users to share their successes on implementation.

Checkr is a company that provides an API that assists businesses in performing modern and compliant background checks. Their discovery and eventual migration to Atlas was recently shared with the internet by Sašo Matejina of the Checkr team.

Checkr

The blog post "MongoDB cluster migration with zero downtime" provides a detailed idea of why they decided to make this change to Atlas and even includes additional software written in Go by Sašo and his team. I got a chance to catch up with Sašo recently to discuss his blog post and some other thoughts on his migration, open source software and MongoDB Atlas.

Jay: How long have you been working with MongoDB personally and at Checkr?

Saso: I started using MongoDB in 2010 right after it shifted its model to open source. So for the last couple of years I used, deployed and managed MongoDB at different scales. Checkr was built using MongoDB from the start and we are actively using it with our microservices.

Jay: What was your most compelling reason to switch to Atlas?

Sašo: We started evaluating the product a couple of months ago, and we were satisfied with easy management and possibility to scale our clusters with a click of a button, flexibility on storage and securing our data with on-rest encryption.

Jay: Talk to me about your support experience a bit. Did you need to reach out to our team? What was it like?

Sašo: We were quite active on support channels as our migration required some special permissions that were not available in Atlas at the time. Support was helpful, and we like that the Atlas team listened to us and made the necessary product changes.

Jay: What's your favorite feature of MongoDB Atlas?

Sašo: I would go with ease of management and scaling. But there are a lot of other cool features like access control with support for teams, alerting on host type. But there are also some features we would like to see in the future like slow query log and index management.

Jay: How do you feel about the performance you see with Atlas?

Sašo: Performance is good, and web interface is very responsive.

Jay: You wrote a wrapper to sync additional data after your data import, was this your first open source project?

Sašo: I've done and contributed to open source before. We are all using open source every day and it's important and it feels good to give something back to the community.

Jay: Would you recommend MongoDB Atlas as an alternative to any existing MongoDB as a services that exist?

Sašo: I would recommend it because I think it provides a good product with competitive pricing.

Great to hear about another successful migration. Tell me more about yours! Email us at cloud-tam@mongodb.com and share your success!

About the Author - Jay Gordon

Jay is a Technical Account Manager with MongoDB and is available via our chat to discuss MongoDB Cloud Products at https://cloud.mongodb.com.