In Kafka, the "Key" does not match the "Id" when updating the document in MongoDB

Working with Data Connectors & Integrations
,
1

We are trying to take all the records from MongoDB to Kafka using the com.mongodb.kafka.connect.MongoSourceConnector. The settings are used for connector as follows:

{
    "name": "mongo-source",
    "config": {
        "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
        "connection.uri": "mongodb:***:27017/?authSource=admin&replicaSet=myMongoCluster&authMechanism=SCRAM-SHA-256",
        "database": "someDb",
        "collection": "someCollection",
        "output.format.value":"json",
        "output.format.key":"json",
        "key.converter.schemas.enable":"false",
        "value.converter.schemas.enable":"false",
        "key.converter":"org.apache.kafka.connect.storage.StringConverter",
        "value.converter":"org.apache.kafka.connect.storage.StringConverter",
        "publish.full.document.only": "true",
        "change.stream.full.document":"updateLookup",
        "copy.existing": "true"
    }
}

When all documents are initially uploaded from MongoDB to Kafka, the “key” corresponds to the “id” from Mongo document:

{"_id": {"_id": {"$oid": "5e54fd0fbb5b5a7d35737232"}, "copyingData": true}}

But when a document in MongoDB is updated, an update with a different “key” gets into Kafka:

{"_id": {"_data": "82627B2EF6000000022B022C0100296E5A1004A47945EC361D42A083988C14D982069C46645F696400645F0FED2B3A35686E505F5ECA0004"}}

Thus, the consumer cannot identify the initially uploaded document and update for it.

Please help me find which settings on the Kafka, Connector or MongoDB side are responsible for this and how I can change the “Key” in Kafka to the same as during the initial upload.

1 Like
2

I am having the same issue, did you find any solution to this?