GIANT Stories at MongoDB

Calling the MongoDB Atlas API - How to do it from Go

After last week's article on how to access the Atlas API with Node, Python, and Ruby, I was asked why didn't I mention Go (among other languages). Well, no need to worry. Here's an extra slab of Atlas API access in Go.

Calling the MongoDB Atlas API - How to do it from Node, Python, and Ruby

The real power of a cloud-hosted, fully managed service like MongoDB Atlas is that you can create whole new database deployment architectures automatically, using the services API. Getting to the MongoDB Atlas API is relatively simple and, once unlocked, it opens up a massive opportunity to integrate and automate the management of database deployments from creation to deletion. The API itself is an extensive REST API, there's role-based access control and you can have user or app-specific credentials to access it.

There is one tiny thing that can trip people up though. The credentials have to be passed over using the digest authentication mechanism, not the more common basic authentication or using an issued token. Digest authentication, at its simplest, waits to get an HTTP 401 (not authorized) from the web endpoint. That response comes with data and the client then sends an encrypted form of the username and password as a digest and the server works with that.

Creating a Data Enabled API in 10 Minutes with MongoDB Stitch

Creating an API that exposes data doesn’t have to be complicated. With MongoDB Stitch, you can create a data enabled endpoint in about 10 minutes or less.

At the heart of the entire process is MongoDB Stitch’s Services. There are several from which to choose and to create a data enabled endpoint, you’ll choose the HTTP Service with a Webhook.

Adding a Stitch Service

When you create an HTTP Service, you’re enabling access to this service from Stitch’s serverless functions in the form of an object called context.services. More on that later when we create a serverless function attached to this service.

Name and add the service and you’ll then get to create an “Incoming Webhook”. This is the process that will be contacted when your clients request data of your API.

Call the webhook whatever you like, and set the parameters as you see below:

We’ll create this API to respond with results to GET requests. Next up, you’ll get to create the logic in a function that will be executed whenever your API is contacted with a GET request.

Defining the Function

Before we modify this script to return data, let’s take a look at the Settings tab — this is where you’ll find the URL where your clients will reach your API.

That’s it — you’ve configured your API. It’s not going to do anything interesting. In fact, the default responds to requests with “Hello World”. Let’s add some data.

Assuming we have a database called mydatabase and a collection of contact data called mycollection, let’s write a function for our service:

Creating a function to return data from a collection in MongoDB Stitch

And here’s the source:

exports = function(payload) {
 const mongodb = context.services.get(“mongodb-atlas”);
 const mycollection = mongodb.db(“mydatabase”).collection(“mycollection”);
 return mycollection.find({}).toArray();
};

This exposes all documents in the database whenever a client calls the webhook URL associated with our HTTP Service. That’s it.

Let’s use Postman to show how this works. Grab your API Endpoint URL from the service settings screen. Mine is as follows — yours will differ.


https://webhooks.mongodb-stitch.com/api/client/v2.0/app/devrel-mrmrq/service/api/incoming_webhook/webhook0

Paste that into the GET URL field and hit Send, you should see something similar to the following:

Check out the GitHub Repository to review the code and try it yourself and review the screencast where I create a data enabled API in 10 Minutes with MongoDB Stitch.

Want to try this for yourself? Sign up for a free MongoDB Atlas account. Looking to leverage an API for integration with MongoDB? Read Andrew Morgan’s article on Building a REST API with MongoDB Stitch.

Additional Resources:

How To Pause and Resume Atlas Clusters

Last week we showed you how to list the resources associated with your MongoDB Atlas environment via a simple Python program. Let’s extend this program this week with a more useful feature, the ability to pause and resume clusters. We can use the Atlas Management API to do this via the “Pause Cluster” menu entry.

Pause a Cluster in MongoDB Atlas

However, when we pause a cluster the Atlas environment will restart the cluster after seven days. Also, both pausing and resuming require a login, navigation etc. Basically, it’s a drag to do this regularly. If you are running clusters for development they are rarely required late at night or at weekends.

It would be great to have a simple script to pause and resume these clusters using the project ID and cluster name. Then we could run this script in crontab or our own favorite scheduling program and pause and resume clusters on a defined schedule. We have rewritten the py-atlas-list.py script to do exactly that.

The extended py-atlas-list.py script allows you to both list resources and pause and/or resume clusters using their project ID and cluster name.

$ python py-atlas-cluster.py -h
usage: py-atlas-cluster.py [-h] [--username USERNAME] [--apikey APIKEY]
                           [--project_id PROJECT_ID] [--org_id ORG_ID]
                           [--pause PAUSE_CLUSTER_NAME]
                           [--resume RESUME_CLUSTER_NAME] [--list]

optional arguments:
  -h, --help            show this help message and exit
  --username USERNAME   MongoDB Atlas username
  --apikey APIKEY       MongoDB Atlas API key
  --project_id PROJECT_ID
                        specify project for cluster that is to be paused
  --org_id ORG_ID       specify an organisation to limit what is listed
  --pause PAUSE_CLUSTER_NAME
                        pause named cluster in project specified by
                        --project_id
  --resume RESUME_CLUSTER_NAME
                        resume named cluster in project specified by
                        --project_id
  --list                List of the complete org hierarchy
$

To pause a cluster just run:

$ python py-atlas-cluster.py --list --org_id XXXXXXXXXXXXXXXXXXXX175c
 1. Org  : 'Open Data at MongoDB',   id=XXXXXXXXXXXXXXXXXXXX175c
  1. Proj : 'JD Stitch Demos',        id=XXXXXXXXXXXXXXXXXXXXcb08
   1. cluster: 'stitch',                 id=XXXXXXXXXXXXXXXXXXXX5697 paused=True
  2. Proj : 'MUGAlyser',              id=XXXXXXXXXXXXXXXXXXXX9bab
   1. cluster: 'MUGAlyser',              id=XXXXXXXXXXXXXXXXXXXXbfba paused=False
  3. Proj : 'Open Data',              id=XXXXXXXXXXXXXXXXXXXX8010
   1. cluster: 'Utility',                id=XXXXXXXXXXXXXXXXXXXX1a03 paused=True
   2. cluster: 'MOT',                    id=XXXXXXXXXXXXXXXXXXXX94dd paused=False
   3. cluster: 'Foodapedia',             id=XXXXXXXXXXXXXXXXXXXX9fbf paused=False
   4. cluster: 'UKPropertyPrices',       id=XXXXXXXXXXXXXXXXXXXX7ac5 paused=False
   5. cluster: 'New-York-Taxi',          id=XXXXXXXXXXXXXXXXXXXXa18a paused=False
   6. cluster: 'demodata',               id=XXXXXXXXXXXXXXXXXXXX2cf8 paused=False
(We have hidden the real resource IDs behind X’s).

To get the project ID look for the id field for the Proj entry. To get the cluster name just look for the string in quotes after the cluster identifier. We have highlighted the project ID and the cluster name we are going to use.

Now to pause the cluster just run:

$ python py-atlas-cluster.py --project_id XXXXXXXXXXXXXXXXXXXX9bab --pause MUGAlyser
Pausing cluster: 'MUGAlyser'
$

To resume a cluster just use the --resume argument instead of the --pause argument. Want to pause or resume more than one cluster in a single project? You can, just by adding multiple --pause or --resume arguments.

Now, you just need to add this script to your favourite scheduler. Note for this example I have already set the environment variables ATLAS_USERNAME and ATLAS_APIKEY so we don’t need to pass them in on the command line.

Now go save some money on your development clusters. Your boss will thank you!

Programmatic API Management of your MongoDB Atlas Database Clusters

Jay Gordon

Technical, Cloud, API

MongoDB Atlas has a full featured API which allows our users to programmatically create, modify and delete their clusters. We'll go over some basics on how to do this using curl in our bash shell. The goal of this post is to show you how to configure API access and use the basic features available.

Pre-Requisites:

  1. Access to a bash shell on your computer or a VM. (Mac or Linux is fine)
  2. curl (ssl support is required)
  3. MongoDB Atlas Cluster Group - Sign up at https://mongodb.com/atlas
  4. Basic understanding of how to utilize APIs.

API Configuration:

First, log into your MongoDB Atlas Control Panel https://cloud.mongodb.com/user/login

MongoDB Atlas Control Panel

Now navigate to the "Settings" section on the left menu. You'll see under "Personal Settings" a section for "Public API Access", click this and you'll then be presented with options to create an API key and then set a whitelist of IP addresses you are able to access the API from with your authenticated key.

Create API Key

Create the Key

First, we will generate an API key. Click the +GENERATE button at the top of the section next to "API Keys" and provide a description. I will use "test-jay.gordon" but there's no specific information required; this is for your own labeling.

Confirm API Key, MongoDB Atlas

Once the key is generated you'll receive a box as above with the key provided. Save this, it will not be repeated for you again. If you are using git, do not check this into to a public repository for security reasons.

Whitelist IP

In this situation, I'll just want to allow "ALL" (0.0.0.0/0) since I am using this temporarily, however if you have a specific subnet of servers that you expect to invoke these functions from, you'll want to whitelist these particular IPs.

Click the ADD+ button next to the API Whitelist section and enter any IP you would like to whitelist with the appropriate CIDR annotation. You can do this multiple times for multiple ranges or IP addresses.

Add Whitelist Entry

Click confirm and then we will be ready to start working with our API now that we have the Key and our whitelist configured:

Public API Access

As you can see, we obscure the full key to ensure it is secure. Protect this resource always, if you no longer require it you may DISABLE or DELETE it by clicking the gear icon under "Actions":

Disable or delete API keys

Create a Cluster:

We'll now use the command prompt on our local computer to do the following tasks:

  1. Create a JSON file with our configuration requirements for our cluster
  2. Write a small curl http POST against the API to create the cluster
  3. Write a small curl http GET against the API to view our cluster

If you review [the spec for our MongoDB Atlas Clusters API](https://docs.atlas.mongodb.com/reference/api/clusters/ "the spec for our MongoDB Atlas Clusters API') you'll see the basics of how to create a curl command for creating a cluster:

POST /api/atlas/v1.0/groups/GROUP-ID/clusters

You must include the following fields and their values when creating a new cluster:

  • name
  • providerSettings.instanceSizeName
  • providerSettings.providerName
  • providerSettings.regionName
  • backupEnabled

All other editable fields are optional. If you do not specify an optional field, MongoDB Atlas uses the field’s current default value. To view the current default values used for new clusters: open the Atlas interface; click the button to add a cluster; view the default selections; close the window without saving changes.

I want to make this easy for myself for the future, so I am going to place these configuration requirements in a simple JSON formatted text file:

{
  "name" : "DataStore",
  "numShards" : 1,
  "replicationFactor" : 3,
  "providerSettings" : {
    "providerName" : "AWS",
    "regionName" : "US_EAST_1",
    "instanceSizeName" : "M30",
    "diskIOPS" : 120,
    "encryptEBSVolume" : false
  },
  "diskSizeGB" : 40,
  "backupEnabled" : true
}

As you can see I have placed the required values for the cluster's name, replication factor, shard total, name of our hosting provider, the region within AWS, the instance size, the disk size, the total amount of IOPS and backups enabled. I saved the file as "json_out.txt" for myself, but you can call it whatever you like. Now let's build our curl statement and execute it. We require two more bits of information to finalize this.

  1. Your username for MongoDB Atlas
  2. Your Atlas group ID. Your username is what you authenticated with when you first logged in to the MongoDB Atlas control panel. The Group ID can be found by going to "Settings" and clicking "Group Settings" on the left side of the page. At the very top, you should see a GROUP ID string:

Group settings

The string above is an example of my group, so be sure to get yours! Now we can build our curl and get our cluster built. Feel free to type this in a editor or just in the bash shell.I prefer to do so in vim so I can save it and modify if I make a mistake.
Now let's look at a few parts of our curl command:

curl -i -u "jay.gordon:$APIKEY" --digest -H "Content-Type: application/json" -X POST "https://cloud.mongodb.com/api/atlas/v1.0/groups/575ece95e4b0ec4f28db42ca/clusters" --data @json_out.txt
  • curl: this is the binary that's running the command for us
  • "jay.gordon:$APIKEY": here i added APIKEY as a variable as to not share it with you, but you may replace it with the key you received earlier (feel free to create a variable in bash if you'd like)
  • https URI: this is the uri we are executing our API against, we've provided our group ID as to ensure we've done this against the right resource
  • --data @json_out.txt: the JSON file containing the requirements for how our cluster will be built.

We're ready to execute the command, let's build our MongoDB Atlas M30 cluster named DataStore with 40 GB of disk, backups enabled, IOPS of 120 and 3 replicas in total.

bash-3.2$ curl -i -u "jay.gordon:$APIKEY" --digest -H "Content-Type: application/json" -X POST "https://cloud.mongodb.com/api/atlas/v1.0/groups/575ece95e4b0ec4f28db42ca/clusters" --data @json_out.txt

Looks like it worked! Here's our output:

HTTP/1.1 201 Created
Date: Fri, 09 Dec 2016 16:41:16 GMT
Content-Length: 506
Content-Type: application/json
{
  "backupEnabled": true,
  "diskSizeGB": 40,
  "groupId": "575ece95e4b0ec4f28db42ca",
  "links": [
    {
      "href": "https:\/\/cloud.mongodb.com\/api\/atlas\/v1.0\/groups\/575ece95e4b0ec4f28db42ca\/clusters\/DataStore",
      "rel": "self"
    }
  ],
  "mongoDBMajorVersion": "3.2",
  "mongoDBVersion": "3.2.11",
  "mongoURIUpdated": "2016-12-09T16:41:16Z",
  "name": "DataStore",
  "numShards": 1,
  "providerSettings": {
    "providerName": "AWS",
    "diskIOPS": 120,
    "encryptEBSVolume": false,
    "instanceSizeName": "M30",
    "regionName": "US_EAST_1"
  },
  "replicationFactor": 3,
  "stateName": "CREATING"
}

Now allow 5-10 minutes for your cluster to build. Once you've finished you can then list your available clusters via a simple GET. There's no need to POST anything here:

curl -i -u "jay.gordon:$APIKEY" --digest "[https://cloud.m575ece95e4b0ec4f28db42ca/clusters](https://cloud.m575ece95e4b0ec4f28db42ca/clusters)"

HTTP/1.1 200 OK
Date: Fri, 09 Dec 2016 16:45:23 GMT
Content-Length: 678
Content-Type: application/json
{
  "links": [
    {
      "href": "https:\/\/cloud.mongodb.com\/api\/atlas\/v1.0\/groups\/575ece95e4b0ec4f28db42ca\/clusters?pageNum=1&itemsPerPage=100",
      "rel": "self"
    }
  ],
  "results": [
    {
      "backupEnabled": true,
      "diskSizeGB": 40,
      "groupId": "575ece95e4b0ec4f28db42ca",
      "links": [
        {
          "href": "https:\/\/cloud.mongodb.com\/api\/atlas\/v1.0\/groups\/575ece95e4b0ec4f28db42ca\/clusters\/DataStore",
          "rel": "self"
        }
      ],
      "mongoDBMajorVersion": "3.2",
      "mongoDBVersion": "3.2.11",
      "mongoURIUpdated": "2016-12-09T16:41:16Z",
      "name": "DataStore",
      "numShards": 1,
      "providerSettings": {
        "providerName": "AWS",
        "diskIOPS": 120,
        "encryptEBSVolume": false,
        "instanceSizeName": "M30",
        "regionName": "US_EAST_1"
      },
      "replicationFactor": 3,
      "stateName": "CREATING"
    }
  ],
  "totalCount": 1
}

We can see it's creating and even programmatically add some delay into our script if we want to then load data once the cluster has finished building

Delete a Cluster:

We know how to list our clusters, so we can use the API to easily destroy the cluster when we are done with them using another curl POST:

bash-3.2$ curl -i -u "jay.gordon:$APIKEY" --digest -X DELETE 
"[https://cloud.mongodb.com/api/atlas/v1.0/groups/575ece95e4b0ec4f28db42ca/clusters/DataStore](https://cloud.mongodb.com/api/atlas/v1.0/groups/575ece95e4b0ec4f28db42ca/clusters/DataStore)"

Here's our output showing we've authenticated and completed our delete process:

HTTP/1.1 202 Accepted
Date: Fri, 09 Dec 2016 16:54:57 GMT
Content-Length: 2
Content-Type: application/json

Even More Resources:

You are not limited to just creating and deleting your cluster. You can use the API to modify other aspects such as the disk size or users while a cluster has already been created. You can review all of our API resources by going to the MongoDB Atlas API Documentation. Want even more MongoDB Atlas information? Check out M034 at MongoDB University for videos on how to use Atlas and start spinning up new MongoDB Clusters!

About the Author - Jay Gordon

Jay is a Technical Account Manager with MongoDB and is available via our chat to discuss MongoDB Cloud Products at https://cloud.mongodb.com.