We are encountering a problem in our kafka connect setup where connect tasks silently go down without reporting any error. We do not realize that the task has gone down unless the oplog change stream is lost and we start getting errors “Resume of change stream was not possible, as the resume point may no longer be in the oplog”. Does anyone have any insights on this and how can we debug this?
We are using mongo db kafka connect version 1.10 and our kafka connect version is 7.3.
We run more than 20 connectors in a single kafka connect pod and this issue happens randomly in any one connector once in a while and leads to data loss for us.
We also seeing weird behaviour which is as follows
8/22/2024 17:33 we did a offset for one of the task with copy existing : true post which it started copying existing data to kafka topic.
8/22/2024 02:05:55 : Last message received in kafka topic, it has copy existing set as true.
8/22/2024 02:06:01 : Kafka connect threw the first error that it cannot resume change stream since token is not valid
Our hypothesis here is :
- Since we did copy existing true, task paused reading from oplog and started reading from mongo. The process took time since task deals with large number of documents. (20M in current topic)
- When it finished copying data it tried to resume change stream but by the time oplog rolled over and it threw an exception and went down.
This contradicts with following facts
- Mongo op log duration in current production is 24 hours but here everything happened from task starting to failing within 9 hours.
- In past we have seen whenever copy existing is set to true, the task starts reading documents from mongo in an unthrottled way (as fast as it can in it’s memory, this is the reason we run kafka connect on large compute nodes) but here we did not see a huge jump and max memory just went upto 2.5GB as opposed to 20-25GB we had seen on experiments in our dev env.
Would really appreciate help on this topic if anyone has any insights