Using kafka connect to grab data from a newly uploaded document and update an existing one

[Dylan_Campopiano]

I have a a mongo db that stores user information as a document in a collection and periodically receives a document in a different collection with a value that needs to be added to one of the user documents.

is it a good idea to use Kafka connect to facilitate this?

I can grab the data from the update doc easily but sinking it to its proper field has been harder.

[MaBeuLux88 (Maxime Beugnet)]

Hi @Dylan_Campopiano and welcome in the MongoDB Community !

If you are using MongoDB Atlas, I would just use an Atlas Trigger in the Realm App Services. It’s like 3 or 4 lines of code.

If not, then I think you have to use Kafka to retrieve the docs as they arrive in your collection and setup a Kafka consumer so you can consume these documents and send an update command to MongoDB to update the related document in the other collection.

I think Kafka connector are just capable of reading or writing entire docs into a collection. Not deal with specific tasks like "get just this field and the _id and send an update to collection xyz".

Cheers,
Maxime.