Atlas Cluster Automation Using Scheduled Triggers
Brian LeonardPublished Jan 08, 2020 • Updated Oct 12, 2022
Rate this tutorial
Every action you can take in the Atlas user interface is backed by a corresponding Administration API, which allows you to easily bring automation to your Atlas deployments. Some of the more common forms of Atlas automation occur on a schedule, such as pausing a cluster that’s only used for testing in the evening and resuming the cluster again in the morning.
Having an API to automate Atlas actions is great, but you’re still on the hook for writing the script that calls the API, finding a place to host the script, and setting up the job to call the script on your desired schedule. This is where Atlas Scheduled Triggers come to the rescue.
In this article I will show you how a Scheduled Trigger can be used to easily incorporate automation into your environment. In addition to pausing and unpausing a cluster, I’ll similarly show how cluster scale up and down events could also be placed on a schedule. Both of these activities allow you to save on costs for when you either don’t need the cluster (paused), or don’t need it to support peak workloads (scale down).
Three example scheduled triggers are provided in this solution. Each trigger has an associated trigger function. The bulk of the work is handled by the modifyCluster function, which as the name implies is a generic function for making modifications to a cluster. It's a wrapper around the Atlas Update Configuration of One Cluster Admin API.
In order to call the Atlas Administrative APIs, you'll first need an API Key with the Organization Owner role. API Keys are created in the Access Manager. At the Organization level (not the Project level), select Access Manager from the menu on the left:
Then select the API Keys tab.
Create a new key, giving it a good description. Assign the key Organization Owner permissions, which will allow it to manage any of the projects in the organization.
Click Next and make a note of your Private Key:
Let's limit who can use our API key by adding an access list. In our case, the API key is going to be used by a Trigger which is a component of Atlas App Services. You will find the list of IP addresses used by App Services in the documentation under Firewall Configuration. Note, each IP address must be added individually.
Since this solution works across your entire Atlas organization, I like to host it in its own dedicated Atlas Project.
We will host our trigger in an Atlas App Services Application. To begin, just click the App Services tab:
You'll see that App Services offers a bunch of templates to get you started. For this use case, just select the first option to Build your own App:
You'll then be presented with options to link a data source, name your application and choose a deployment model. The current iteration of this utility doesn't use a data source, so you can ignore that step (a free cluster for you regardless). You can also leave the deployment model at its default (Global), unless you want to limit the application to a specific region.
I've named the application Automation App:
From here, you have the option to simply import the Realm application and adjust any of the functions to fit your needs. If you prefer to build the application from scratch, skip to the next section.
The extract has a dependency on the API Secret Key, thus the import will fail if it is not configured beforehand.
Use the Values menu on the left to Create a Secret named AtlasPrivateKeySecret containing your private key (the secret is not in quotes):
The Realm CLI is available on npm. To install the Realm CLI on your system, ensure that you have Node.js installed and then run the following command in your shell:
Download and extract the AutomationApp.zip.
To configure your app with realm-cli, you must log in to Atlas using your API keys:
Select the App Settings menu and copy your Application ID:
Run the following realm-cli push command from the directory where you extracted the export:
After the import, replace the
AtlasPublicKeywith your API public key value.
The imported application includes 3 self-explanatory sample scheduled triggers:
The 3 triggers have 3 associated Functions. The pauseClustersTrigger and resumeClustersTrigger function supply a set of projects and clusters to pause, so these need to be adjusted to fit your needs:
All 3 trigger functions call the modifyCluster function, where the bulk of the work is done.
In addition, you'll find two utility functions, getProjectClusters and getProjects. These functions are not utilized in this solution, but are provided for reference if you wanted to further automate these processes (that is, removing the hard coded project IDs and cluster names in the trigger functions):
Now that you have reviewed the draft, as a final step go ahead and deploy the Realm application.
To understand what's included in the application, here are the steps to build it yourself from scratch.
The functions we need to create will call the Atlas Administration APIs, so we need to store our API Public and Private Keys, which we will do using Values & Secrets. The sample code I provide references these values as AtlasPublicKey and AtlasPrivateKey, so use those same names unless you want to change the code where they’re referenced.
You'll find Values under the BUILD menu:
First, create a Value for your public key (note, the key is in quotes):
Create a Secret containing your private key (the secret is not in quotes):
The Secret cannot be accessed directly, so create a second Value that links to the secret:
We need to note the IDs of the projects that have clusters we want to automate. Click the 3 dots in the upper left corner of the UI to open the Project Settings:
Under which you’ll find your Project ID:
I will create two functions, a generic function to modify a cluster and a trigger function to iterate over the clusters to be paused.
You'll find Functions under the BUILD menu:
I’m only demonstrating a couple of things you can do with cluster automation, but the sky is really limitless. The following modifyCluster function is a generic wrapper around the Update Configuration of One Cluster API for calling the API from App Services (or Node.js for that matter).
Create a New Function named modifyCluster. Set the function to Private as it will only be called by our trigger. The other default settings are fine:
Switch to the Function Editor tab and paste the following code:
To test this function, you need to supply an API key, an API secret, a project Id, an associated cluster name to modify, and a payload containing the modifications you'd like to make. In our case it's simply setting the paused property.
Note: By default, the Console supplies 'Hello world!' when test running a function, so my function code tests for that input and provides some default values for easy testing.
Press the Run button to see the results, which will appear in the Result window:
And you should find you cluster being resumed (or paused):
This function will be called by a trigger. As it's not possible to pass parameters to a scheduled trigger, it uses a hard-coded list of project Ids and associated cluster names to pause. Ideally these values would be stored in a collection with a nice UI to manage all of this, but that's a job for another day :-).
In the appendix of this article, I provide functions that will get all projects and clusters in the organization. That would create a truly dynamic operation that would pause all clusters. You could then alternatively refactor the code to use an exclude list instead of an allow list.
The ability to pause and resume a cluster is supported by the Update Configuration of One Cluster API. To begin, select Triggers from the menu on the left:
And add a Trigger.
Set the Trigger Type to Scheduled and the name to pauseClusters:
As for the schedule, you have the full power of CRON Expressions at your fingertips. For this exercise, let’s assume we want to pause the cluster every evening at 6pm. Select Advanced and set the CRON schedule to
0 22 * * *.
Note, the time is in GMT, so adjust accordingly for your timezone. As this cluster is running in US East, I’m going to add 4 hours:
Check the Next Events window to validate the job will run when you desire.
The final step is to select the function for the trigger to execute. Select the pauseClustersTrigger function.
And Save the trigger.
The final step is to REVIEW DRAFT & DEPLOY.
You could opt to manually resume the cluster(s) as it’s needed. But for completeness, let’s assume we want the cluster(s) to automatically resume at 8am US East every weekday morning.
Duplicate the pauseClustersTrigger function to a new function named resumeClustersTriggger
At a minimum, edit the function code setting paused to false. You could also adjust the projectIDs and clusterNames to a subset of projects to resume:
Then add a new scheduled trigger named resumeClusters. Set the CRON schedule to:
0 12 * * 1-5. The Next Events validates for us this is exactly what we want:
It’s not uncommon to have workloads that are more demanding during certain hours of the day or days of the week. Rather than running your cluster to support peak capacity, you can use this same approach to schedule your cluster to scale up and down as your workload requires it.
NOTE: Atlas Clusters already support Auto-Scaling, which may very well suit your needs. The approach described here will let you definitively control when your cluster scales up and down.
Let’s say we want to scale up our cluster every day at 9am before our store opens for business.
Add a new function named scaleClusterUpTrigger. Here’s the function code. It’s very similar to before, except the body’s been changed to alter the provider settings:
Then add a scheduled trigger named scaleClusterUp. Set the CRON schedule to:
0 13 * * *.
Scaling a cluster back down would simply be another trigger, scheduled to run when you want, using the same code above, setting the instanceSizeName to whatever you desire.
And that’s it. I hope you find this beneficial. You should be able to use the techniques described here to easily call any MongoDB Atlas Admin API endpoint from Atlas App Services.
This standalone function can be test run from the App Services console to see the list of all the projects in your organization. You could also call it from other functions to get a list of projects:
Another example function that will return the cluster details for a provided project.
Note, to test this function, you need to supply a projectId. By default, the Console supplies ‘Hello world!’, so I test for that input and provide some default values for easy testing.