Hi team,
I am using MongoDb Kafka Source connector to produce the existing document in Atlas DB into the Kafka topic. I want to exclude a collection from the database from which the documents are fetched.
I set the pipeline option as:
[{“$match”: {“ns.coll”: {“$regex”: /^(?!collection_to_be_excluded).*/}}}]
along with copy.existing
set to true
. But I am not seeing any existing records from the allowed collections.
Using pipeline works fine when not copying existing data. So the newly added records in the allowed collections will be produced into the Kafka topic.
I tried setting copy.existing.pipeline
similar to pipeline value above, but still no record is getting into the topic.
How can I filter the collections for existing documents?
Many thanks.
cc: @Robert_Walters