I want to store Customer details in MongoDB. Customer document has various fields/embedded documents e.g. Demographics, Address, Phone, Email etc. I will be streaming them as events to Kafka Topics… As Avro supports different schema types, I want to leverage that to show different events types in same Customer Profile topic. My question is when I do Change Data capture from MongoDB to Coarse grained Kafka Topic, will it send full document each time or only the fields/embedded document that have changed?
It sends a change stream event https://docs.mongodb.com/kafka-connector/master/kafka-source/#change-stream-event-document-format/. There is a parameter called “publish.full.document.only” that when set to true will put the full document in the event.