Unable to connect to Atlas-Hosted MongoDB via Confluent connector hosted in MSK

Working with Data Connectors & Integrations
, , ,
1

I am attempting to use the official MongoDB kafka connector https://www.mongodb.com/docs/kafka-connector/current/sink-connector/ as a SINK to move data from an AWS hosted MSK cluster into an ATLAS hosted mongodb.

I’ve configured the client in a way that I THINK should work but I’m getting issues

[Worker-00cbe7dbb4d2fb6dc] [2022-04-27 13:30:21,449] INFO Exception in monitor thread while connecting to server [mydb].mongodb.net:27017 (org.mongodb.driver.cluster:76)
[Worker-00cbe7dbb4d2fb6dc] com.mongodb.MongoSocketOpenException: Exception opening socket
[Worker-00cbe7dbb4d2fb6dc] 	at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70)

I’m wondering if anyone out there has successfully done this and if there is a trick somewhere that I’m missing.

This is a development level proof of concept at the moment, so I’ve got a fully-open kafka solution in AWS, the mongodb in atlas has a 0.0.0.0/0 setting so it should be open.

My connection.uri uses a mongodb+srv link

I am able to successfully connect to the mongo instance if I attempt to connect via mongosh from within an EC2 instance on the same AWS cluster, but the MSK Connect workers in AWS are failing.

Would love to hear any insights from others who have successfully done this in the past.

1 Like
2

Hey
I’ve just run into the exact same problem, where you able to resolve it in the end?

3

Just in case someone else stumbles across this I’ll share the answer

You have to create a Private Endpoint in Atlas then a VPC endpoint in AWS.

There is a section in this guide that explains how to set up the private endpoint: Integrating MongoDB with Amazon Managed Streaming for Apache Kafka (MSK) | MongoDB

For me even though we had a peering connection set up between AWS and Atlas with all of the correct routes etc, we still had to go via this private endpoint option, it’s a limitation of MSK I believe

4

For VPC Peering to work, you’ll need NAT GW configured. It doesn’t work with IGW.