Multiple kafka topic for one mongodb collection using kafka connector

I have a collection upon which i want to enable mongo-kafka connector to push changes to kafka.
I want to push 2 types of events on 2 separate topics: snapshot and delta where the 1st topic is just the snapshot of record(comes from “fullDocument” key in stream event) and the second is what has changed(comes from “updateDescription”). I need these 2 separately because of the way “fullDocument” works where we may miss updates in between because of concurrent writes.
Is there a way to achieve this in mongo connector?

You can create two instance of the connector that point to the same namespace and define the pipeline configuration accordingly.

An example of the pipeline parameter is something like:

[{"$match":{ "$and": [{"fullDocument.type":"temp"},{"fullDocument.value":{"$gte":100}}] }}]

Here I am matching where the type field is “temp” and the “value” field contains a number greater than or equal to 100.

1 Like

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.