Sync mongodb with elastic

Kafka Connect is a native Kafka service that is built for integrating heterogenous data like MongoDB within Kafka. You could write your own consumer but you’ll have to deal with initial loading of data and keeping track of topic offsets to protect yourself in case of errors. In the end why bother coding all that, just use the service that is already designed for this purpose. The MongoDB Connector for Apache Kafka is the official connector supporting Kafka Connect.

In your scenario, simply use the MongoDB connector as a source and point to your MongoDB cluster. Then use the ElasticSearch Sink connector to pull data from the kafka topic to Elastic.

1 Like