Using File ConfigProvider with Mongodb Kafka Connector

Working with Data Connectors & Integrations
#1

Hi

We are trying to avoid exposing secrets of the Mongodb connection url using the File config provider. We have the connection url stored in a secrets.properties file.

In the connector configuration

We added the following lines in connect-distributed.properties file

config.providers=file
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider

And in the connector configuration we add the following line

{
  "name": "MarketPlaceSourceConnector",
  "config": {
    "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
    "pipeline": "[{\"$match\":{\"$or\":[{\"operationType\":\"insert\"},{\"operationType\":\"update\"}]}}]",
    "database": "markk",
    "publish.full.document.only": "false",
    "change.stream.full.document": "updateLookup",
    "tasks.max": "1",
    "connection.url": "${file:/opt/secrets.properties:mongodb-connection-uri}",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "collection": "marketplace",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter"
  }
}

But the config failed to load with a timeout to connect.

#2

I have issue on same topic . If I try to set the source connector configuration with wrong password in the file specified it does give authentication error so file permission is not an issue but when i provide correct password it does give following error

Source Connector Configuration:-
{“name”: “mongo-source-passwd”,
“config”: {
“connector.class”: “com.mongodb.kafka.connect.MongoSourceConnector”,
“connection.uri”: “${file:/data/kafka-poc/db-connect.properties:db-url}”,
“database”: “kafka-poc-source”,
“collection”: “other1”,
“pipeline”: “[]”,
“prefix”: “”
}
}

With wrong password in data/kafka-poc/db-connect.properties (POST Source connector Configuration throws following error) :-
{“error_code”:400,“message”:"Connector configuration is invalid and contains the following 1 error(s):\nInvalid user permissions authentication failed. Exception authenticating MongoCredential{mechanism=SCRAM-SHA-1, userName=‘kafka_admin’, source=‘admin’, password=, mechanismProperties=}\nYou can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate"}

With correct Password (POST Source connector Configuration throws following error) :-
{“error_code”:500,“message”:“Could not read properties from file /data/kafka-poc/db-connect.properties”}

File has this entry below (removed password and server name):-
db-url=mongodb://kafka_admin:@mongodb server27017/?authSource=admin

#3

FYI , nothing wrong in the configuration above , when i restarted the KAFKA connect on all nodes which was part of a cluster it fixed the issue. I did restarted the KAFKA connect on one node when I did create MongoDB Source connector instance as explained above . Still not sure why we have to restart all KAFKA workers if new MongoDB source connector instance is added as it does not make sense to every time restart KAFKA workers if new MongoDB source/sink instance is added.