EventGet 50% off your ticket to MongoDB.local NYC on May 2. Use code Web50!Learn more >>
MongoDB Developer
Atlas
plus
Sign in to follow topics
MongoDB Developer Centerchevron-right
Developer Topicschevron-right
Productschevron-right
Atlaschevron-right

Visualize MongoDB Atlas Database Audit Logs

Fuat Sungur11 min read • Published Oct 06, 2022 • Updated Jun 12, 2023
Atlas
Facebook Icontwitter iconlinkedin icon
Web-ready data lake file
Rate this tutorial
star-empty
star-empty
star-empty
star-empty
star-empty
MongoDB Atlas has advanced security capabilities, and audit logs are one of them. Simply put, enabling audit logs in an Atlas cluster allows you to track what happened in the database by whom and when.
In this blog post, I’ll walk you through how you can visualize MongoDB Atlas Database Audit Logs with MongoDB Atlas Charts.

High level architecture

Steps of Visualizing MongoDB Atlas Audit Logs
  1. In Atlas App Services Values, Atlas Admin API public and private keys and AWS API access key id and secret access have been defined.
  2. aws-sdk node package has been added as a dependency to Atlas Functions.
  3. Atlas Data Federation has been configured to query the data in a cloud storage - Amazon S3 of Microsoft Blob Storage bucket.
  4. Atlas Function retrieves both Atlas Admin API and AWS API credentials.
  5. Atlas Function calls the Atlas Admin API with the credentials and other relevant parameters (time interval for the audit logs) and fetches the compressed audit logs.
  6. Atlas Function uploads the compressed audit logs as a zip file into a cloud object storage bucket where Atlas has read access.
  7. Atlas Charts visualize the data in S3 through Atlas Data Federation.

Prerequisites

The following items must be completed before working on the steps.
  • Provision an Atlas cluster where the tier is at least M10. The reason for this is auditing is not supported by free (M0) and shared tier (M2, M5) clusters.
  • You need to set up database auditing in the Atlas cluster where you want to track activities.
    • Under the Security section on the left hand menu on the main dashboard, select Advanced. Then toggle Database Auditing and click Audit Filter Settings.
  • For the sake of simplicity, check All actions to be tracked in Audit Configuration as shown in the below screenshot. Actions to be logged in Audit
  • If you don’t have your own load generator to generate load in the database in order to visualize through MongoDB Charts later, you can review this load generator in the Github repository of this blog post.
  • Create an app in Atlas App Services that will implement our functions inside it. If you haven’t created an app in Atlas App Services before, please follow this tutorial.
  • Create an AWS account along with the following credentials — AWS Access Key and AWS Secret Access Secret.
  • Set an AWS IAM Role that has privilege to write into the cloud object storage bucket.
    • Later, Atlas will assume this role to make write operations inside S3 bucket.

Step 1: configuring credentials

Atlas Admin API allows you to automate your Atlas environment. With a REST client, you can execute a wide variety of management activities such as retrieving audit logs.
In order to utilize Atlas Admin API, we need to create keys and use these keys later in Atlas Functions. Follow the instructions to create an API key for your project.

Creating app services values and app services secrets

After you’ve successfully created public and private keys for the Atlas project, we can store the Atlas Admin API keys and AWS credentials in App Services Values and Secrets.
App Services Values and App Services Secrets are static, server-side constants that you can access or link to from other components of your application.
In the following part, we’ll create four App Services Values and two App Services Secrets to manage both MongoDB Atlas Admin API and AWS credentials. In order to create App Services Values and Secrets, navigate your App Services app, and on the left hand menu, select Values. This will bring you to a page showing the secrets and values available in your App Services app.

Setting up Atlas Admin API credentials

In this section, we’ll create two App Services Values and one App Services Secrets to store Atlas Admin API Credentials.
Value 1: AtlasAdminAPIPublicKey
This Atlas App Services value keeps the value of the public key of Atlas Admin API. Values should be wrapped in double quotes as shown in the following example.
Public Key of Atlas Admin API
Secret 1: AtlasAdminAPIPrivateKey
This Atlas App Services Secret keeps the value of the private key of Atlas Admin API. You should not wrap the secret in quotation marks.
Private Key (Secret) of Atlas Admin API
Value 2: AtlasAdminAPIPrivateKeyLinkToSecret
We can’t directly access secrets in our Atlas Functions. That’s why we have to create a new value and link it to the secret containing our private key.
Linking to the Secret for Atlas Admin API
Until now, we’ve defined necessary App Services Values and Atlas App Services Secrets to access Atlas Admin API from an App Services App.
In order to access our S3 bucket, we need to utilize AWS SDK. Therefore, we need to do a similar configuration for AWS SDK keys.

Setting up AWS credentials

In this section, we’ll create two App Services Values and one App Services Secret to store AWS Credentials. Learn how to get your AWS Credentials.
Value 3: AWSAccessKeyId
This Atlas App Services Value keeps the value of the access key id of AWS SDK.
Access Key of AWS
Secret 2: AWSSecretAccessKey
This Atlas App Services Secret keeps the value of the secret access key of AWS SDK.
Private Key (Secret Access Key Id) of AWS Credentials
Value 4: AWSSecretAccessKeyLinkToSecret
This Atlas App Services Value keeps the link of Atlas App Services Secret that keeps the secret key of AWS SDK.
Linking to the Secret for AWS Credentials
And after you have all these values and secrets as shown below, you can deploy the changes to make it permanent.
App Services Values and App Services Secrets

Step 2: adding an external dependency

An external dependency is an external library that includes logic you'd rather not implement yourself, such as string parsing, convenience functions for array manipulations, and data structure or algorithm implementations. You can upload external dependencies from the npm repository to App Services and then import those libraries into your functions with a require('external-module') statement.
In order to work with AWS S3, we will add the official aws-sdk npm package.
In your App Services app, on the left-side menu, navigate to Functions. And then, navigate to the Dependencies pane in this page.
In order to work with AWS S3, we will add the official aws-sdk npm package.
In your App Services app, on the left-side menu, navigate to Functions. And then, navigate to the Dependencies pane in this page.
Atlas Functions Dependencies
Click Add Dependency.
Dependency Adding
Provide aws-sdk as the package name and keep the package version empty. That will install the latest version of aws-sdk node package.
Available Dependent Packages
Now, the aws-sdk package is ready to be used in our Atlas App Services App.

Step 3: configuring Atlas Data Federation to consume cloud object storage data through MongoDB Query Language (MQL)

In this tutorial, we’ll not go through all the steps to create a federated database instance in Atlas. Please check out our Atlas Data Federation resources to go through all steps to create Atlas Data Federated Instance.
As an output of this step, we’d expect a ready Federated Database Instance as shown below.
Federated Database Instances
I have already added the S3 bucket (the name of the bucket is fuat-sungur-bucket) that I own into this Federated Database Instance as a data source and I created the collection auditlogscollection inside the database auditlogs in this Federated Database Instance.
Defining Data Source and database (virtual) and collections (virtual) in the federated database instance
Now, if I have the files in this S3 bucket (fuat-sungur-bucket), I’ll be able to query it using the MongoDB aggregation framework or Atlas SQL.

Step 4: creating an Atlas function to retrieve credentials from Atlas App Services Values and Secrets

Let’s create an Atlas function, give it the name RetrieveAndUploadAuditLogs, and choose System for authentication.
Naming and Choosing Authentication Options for an Atlas Function
We also provide the following piece of code in the Function Editor and Run the function. We’ll see the credentials have been printed out in the console.

Step 5: retrieving audit logs in the Atlas function

We now continue to enhance our existing Atlas function, RetrieveAndUploadAuditLogs. Now, we’ll execute the HTTP/S request to retrieve audit logs into the Atlas function.
Following piece of code generates an HTTP GET request,calls the relevant Atlas Admin API resource to retrieve audit logs within 1440 minutes, and converts this compressed audit data to the Buffer class in JavaScript.

Step 6: uploading audit data into the S3 bucket

Until now, in our Atlas function, we retrieved the audit logs based on the given interval, and now we’ll upload this data into the S3 bucket as a zip file.
Firstly, we import aws-sdk NodeJS library and then configure the credentials for AWS S3. We have already retrieved the AWS credentials from App Services Values and App Services Secrets and assigned those into function variables.
After that, we configure S3-related parameters, bucket name, key (folder and filename), and body (actual payload that is our audit zip file stored in a Buffer Javascript data type). And finally, we run our upload command (S3.putObject()).
Here you can find the entire function code:
After we run the Atlas function, we can check out the S3 bucket and verify that the compressed audit file has been uploaded.
A folder in an S3 bucket where we store the audit logs
You can find the entire code of the Atlas functions in the dedicated Github repository.

Step 7: visualizing audit data in MongoDB Charts

First, we need to add our Federated Database Instance that we created in Step 4 into our Charts application that we created in the prerequisites section as a data source. This Federated Database Instance allows us to run queries with the MongoDB aggregation framework on the data that is in the cloud object storage (that is S3, in this case).
Before doing anything with Atlas Charts, let’s connect to Federated Database Instance and query the audit logs to make sure we have already established the data pipeline correctly.
Now, we can get a record from the auditlogscollection.
Let’s check the audit log of an update operation.
If we are able to see some records, that’s great. Now we can build our dashboard in Atlas Charts.
A dashboard in Atlas Charts displays data regarding MongoDB Atlas Auditing
You can import this dashboard into your Charts application. You might need to configure the data source name for the Charts for your convenience. In the given dashboard, the datasource was a collection with the name auditlogscollection in the database auditlogs in the Atlas Federated Database Instance with the name FederatedDatabaseInstance0, as shown below.
Available data sources in Atlas Charts

Caveats

The following topics can be considered for more effective and efficient audit log analysis.
  • You could retrieve logs from all the hosts rather than one node.
    • Therefore you can track the data modifications even in the case of primary node failures.
  • You might consider tracking only the relevant activities rather than tracking all the activities in the database. Tracking all the activities in the database might impact the performance.
  • You can schedules your triggers.
    • The Atlas function in the example runs once manually, but it can be scheduled via Atlas scheduled triggers.
      • Then, date intervals (start and end time) for the audit logs for each execution need to be calculated properly.
  • You could improve read efficiency.
    • You might consider partitioning in the S3 bucket by most frequently filtered fields. For further information, please check the docs on optimizing query performance.

Summary

MongoDB Atlas is not only a database as a service platform to run your MongoDB workloads. It also provides a wide variety of components to build end-to-end solutions. In this blog post, we explored some of the capabilities of Atlas such as App Services, Atlas Charts, and Atlas Data Federation, and observed how we utilized them to build a real-world scenario.
Questions on this tutorial? Thoughts, comments? Join the conversation over at the MongoDB Community Forums!

Facebook Icontwitter iconlinkedin icon
Rate this tutorial
star-empty
star-empty
star-empty
star-empty
star-empty
Related
Tutorial

Instant GraphQL APIs for MongoDB with Grafbase


Oct 12, 2023 | 7 min read
Article

Build a Newsletter Website With the MongoDB Data Platform


Sep 23, 2022 | 9 min read
Article

How to Easily Pause and Resume MongoDB Atlas Clusters


Sep 23, 2022 | 5 min read
Tutorial

Utilizing Collection Globbing and Provenance in Data Federation


Jun 28, 2023 | 5 min read
Table of Contents