BlogAtlas Vector Search voted most loved vector database in 2024 Retool State of AI reportLearn more >>
MongoDB Developer
Sign in to follow topics
MongoDB Developer Centerchevron-right
Developer Topicschevron-right

Using AWS IAM Authentication with the MongoDB Connector for Apache Kafka

Robert Walters, Jagadish Nallapaneni4 min read • Published Jul 01, 2024 • Updated Jul 01, 2024
Facebook Icontwitter iconlinkedin icon
Rate this tutorial
AWS Identity and Access Management (IAM) is a widely used service that enables developers to control secure access to AWS services and resources. A key benefit of AWS IAM is its ability to make secure authenticated connections without the client having to pass sensitive passwords.
As of version 1.13, the MongoDB Connector for Apache Kafka (Kafka Connector) now supports authentication via AWS IAM. AWS IAM support has been one of the most commonly requested features for the Kafka Connector, and in this tutorial, we will walk through how to use it.
To achieve support for AWS IAM, we added support for overriding the MongoClient interface allowing a developer to provide a CustomCredentialProvider. While this provider can be used for any authentication method, this tutorial will focus on building an AWS IAM credential provider.
To illustrate this, we will walk through a simple configuration of an AWS EC2 instance installed with Apache Kafka. This instance will use the MongoDB Connector for Apache Kafka to sink data from a Kafka topic to an Atlas cluster, and authenticate using AWS IAM.


If you wish to follow along in this article, make sure you have the following:
  • An AWS IAM role, “KafkaBlogAtlasRole”. This role will later be mapped as an Atlas database user.
  • MongoDB Atlas cluster. If you do not have an Atlas cluster, learn how to get started with Atlas..
  • AWS EC2 instance
    • Install Apache Kafka.
    • Configure the EC2 instance with an AWS IAM role, “KafkaConnectBlogRole.”
    • Add a policy to KafkaConnectBlogRole that allows “AssumeRole” on the KafkaBlogAtlasRole.
Example policy:
  • Java Development Kit version 8 or later
  • A database user in the desired MongoDB Atlas cluster
    • Configure the database user with AWS IAM and use the AWS Role ARN for KafkaBlogAtlasRole. The Atlas database user should have access to read/write to the desired collection.
MongoDB Atlas edit user dialog showing AWS IAM authentication method
At this point, we are ready to start building the custom authentication provider.

Getting started

Starting with version 1.13 of the MongoDB Connector for Apache Kafka, we now have the ability to plug in a custom authentication provider. In this article, we will create a custom authentication provider in Java, compile it, and add it to the lib directory of your Kafka deployment. The provider we will create will support AWS IAM authentication.

Building the custom authentication component

To start, in your favorite Java editor, create a project and build the following JAR that contains this source:
Example pom.xml file
Note: We need to reference MongoDB Connector for Apache Kafka version 1.13 or later. In this example, the connector was downloaded from the Kafka Connector GitHub repository and stored in the local Maven repository.
Compile and place this SampleAssumeRole.jar file in the Kafka libs directory in your Kafka Connect deployment. You are now ready to configure your MongoDB Connector for Apache Kafka to use this custom authentication provider.

Configure the MongoDB Connector for Apache Kafka

To support the custom provider in version 1.13, the following configuration options were added to the connector:
Configuration ParameterDescription
mongo.custom.auth.mechanism.enableDetermines if the connector should use the provider class; default is false
mongo.custom.auth.mechanism.providerClassJava class entry point for the custom provider
mongodbaws.auth.mechanism.roleArnRole that has IAM access to the database
In this example, the custom authentication component will use AWS IAM AssumeRole to authenticate with MongoDB Atlas. Here is an example of this configuration connecting to a MongoDB Atlas cluster.
Notice that the roleArn is the ARN (Amazon Resource Name) of the AWS IAM role that is used for Atlas users, KafkaBlogAtlasRole. At runtime, since Kafka Connect is running under the AWS IAM role of the EC2 instance, KafkaConnectBlogRole, and this role has a policy that allows AssumeRole on the KafkaBlogAtlasRole, the service can connect to Atlas without storing any credentials!
You are now ready to send data to Kafka and watch it arrive in a MongoDB collection using the MongoDB Connector for Apache Kafka configured with AWS IAM authentication to MongoDB Atlas.


In this article, we discussed the requirements needed to support a custom authentication provider such as AWS IAM authentication. Version 1.13 of the Kafka Connector introduced new parameters that define which class to use within the custom provider JAR file. This JAR file includes the logic for the custom authentication.
Note that while adding AWS IAM authentication capability natively to the Kafka Connector is possible (it is just software, after all), doing so would not only add a version dependency on the AWS IAM SDK, but it would also significantly increase the size of the connector, making it impractical for all other users who do not wish to use this authentication method. For these reasons, writing the custom credential provider is the solution for integrating the Kafka Connector with authentication methods such as AWS IAM.
Read the MongoDB Connector for Apache Kafka documentation to learn more. Questions? Join us in the MongoDB Developer Community.
Top Comments in Forums
There are no comments on this article yet.
Start the Conversation

Facebook Icontwitter iconlinkedin icon
Rate this tutorial

Deploying MongoDB Across Multiple Kubernetes Clusters With MongoDBMulti

Sep 05, 2023 | 11 min read

MongoDB Podcast Interview with Connectors and Translators Team

Sep 23, 2022 | 16 min

Mastering MongoDB Ops Manager on Kubernetes

Jan 13, 2023 | 7 min read

How to Get Started with MongoDB Atlas Stream Processing and the HashiCorp Terraform MongoDB Atlas Provider

May 20, 2024 | 5 min read
Table of Contents
  • Prerequisites