Using kafka connect to grab data from a newly uploaded document and update an existing one

I have a a mongo db that stores user information as a document in a collection and periodically receives a document in a different collection with a value that needs to be added to one of the user documents.

is it a good idea to use Kafka connect to facilitate this?

I can grab the data from the update doc easily but sinking it to its proper field has been harder.

Hi @Dylan_Campopiano and welcome in the MongoDB Community :muscle: !

If you are using MongoDB Atlas, I would just use an Atlas Trigger in the Realm App Services. It’s like 3 or 4 lines of code.

If not, then I think you have to use Kafka to retrieve the docs as they arrive in your collection and setup a Kafka consumer so you can consume these documents and send an update command to MongoDB to update the related document in the other collection.

I think Kafka connector are just capable of reading or writing entire docs into a collection. Not deal with specific tasks like "get just this field and the _id and send an update to collection xyz".


This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.