Unable to get MongoDB source connector to talk to azure event hub

Working with Data Connectors & Integrations
, , ,
1

I have setup a kafka connect docker image that is successfully configured to use Azure event hub as the broker. I was able to verify that connect was configured correctly when I saw the new event hubs created by my connect instance. In order for my kafka connect instance to talk to Azure, I had to set its configuration with these settings:

ENV CONNECT_BOOTSTRAP_SERVERS="weavix.servicebus.windows.net:9093"
ENV CONNECT_SECURITY_PROTOCOL="SASL_SSL"
ENV CONNECT_SASL_MECHANISM="PLAIN"
ENV CONNECT_SASL_JAAS_CONFIG="<my connection string>";"

My MongoDB cluster is hosted by Mongo.

I installed mongodb/kafka-connect-mongodb:1.11.0 connector and sent this payload for a new connector:

{

{
    "name": "weavix-dev-master-mongodb-source",
    "config": {
        "tasks.max": "1",
        "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
        "connection.uri": "mongodb+srv://<user>:<password>@<my db>.mezxs.mongodb.net/test?authSource=admin&replicaSet=atlas-8mkql5-shard-0&readPreference=primary&ssl=true",
        "database": "master",
        "topic.prefix": "weavix-dev",
        "collection": "accounts",
        "topic.separator": "-",
        "startup.mode": "copy_existing"
    }
}

The connection string I use is usable with MongoCompass.

Unfortunately, I am getting in the logs is a repetitive set of messages:

[2023-10-09 21:09:16,558] INFO [weavix-dev-master-mongodb-source|task-0] [Producer clientId=connector-producer-weavix-dev-master-mongodb-source-0] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:937)
[2023-10-09 21:09:16,558] INFO [weavix-dev-master-mongodb-source|task-0] [Producer clientId=connector-producer-weavix-dev-master-mongodb-source-0] Cancelled in-flight API_VERSIONS request with correlation id 4112 due to node -1 being disconnected (elapsed time since creation: 52ms, elapsed time since send: 52ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient:341)
[2023-10-09 21:09:16,558] WARN [weavix-dev-master-mongodb-source|task-0] [Producer clientId=connector-producer-weavix-dev-master-mongodb-source-0] Bootstrap broker weavix.servicebus.windows.net:9093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient:1065)

It looks like a networking issue with the connector but I don’t see any settings for the connector for setting up broker authentication. Anybody have any suggestions?

2

When I review the logs when I add the connector, I see this:

2023-10-09 16:34:16 [2023-10-09 21:34:16,296] INFO [weavix-dev-master-mongodb-source|task-0] ProducerConfig values:
2023-10-09 16:34:16 acks = -1
2023-10-09 16:34:16 batch.size = 16384
2023-10-09 16:34:16 bootstrap.servers = [weavix.servicebus.windows.net:9093]
…
2023-10-09 16:34:16 sasl.client.callback.handler.class = null
2023-10-09 16:34:16 sasl.jaas.config = null
2023-10-09 16:34:16 sasl.kerberos.kinit.cmd = /usr/bin/kinit
2023-10-09 16:34:16 sasl.kerberos.min.time.before.relogin = 60000
2023-10-09 16:34:16 sasl.kerberos.service.name = null
2023-10-09 16:34:16 sasl.kerberos.ticket.renew.jitter = 0.05
2023-10-09 16:34:16 sasl.kerberos.ticket.renew.window.factor = 0.8
2023-10-09 16:34:16 sasl.login.callback.handler.class = null
2023-10-09 16:34:16 sasl.login.class = null
2023-10-09 16:34:16 sasl.login.connect.timeout.ms = null
2023-10-09 16:34:16 sasl.login.read.timeout.ms = null
2023-10-09 16:34:16 sasl.login.refresh.buffer.seconds = 300
2023-10-09 16:34:16 sasl.login.refresh.min.period.seconds = 60
2023-10-09 16:34:16 sasl.login.refresh.window.factor = 0.8
2023-10-09 16:34:16 sasl.login.refresh.window.jitter = 0.05
2023-10-09 16:34:16 sasl.login.retry.backoff.max.ms = 10000
2023-10-09 16:34:16 sasl.login.retry.backoff.ms = 100
2023-10-09 16:34:16 sasl.mechanism = GSSAPI
2023-10-09 16:34:16 security.protocol = PLAINTEXT
2023-10-09 16:34:16 security.providers = null
2023-10-09 16:34:16 send.buffer.bytes = 131072

These messages seems to indicate to me that the connector’s producer may not be configured correctly, alas I don’t see a way to set its configuration.