Mongo Kafka Connector poll.await.time.ms Not Working

Hi,

I am working on Kafka Mongo Source Connector and as per business requirement I need to poll for data in MongoDB for every 30 min interval, so for this I am using this property: “poll.await.time.ms”, but it is not working, when I insert/update data it immediately loads in Kafka topic, is there any way I could configure it to poll for specified time and then load data in Kafka topic.
Can anyone please suggest me any solution.

Thank you!

Hey @sanket_kokne,

The “poll.await.time.ms” property in the Kafka Mongo Source Connector controls the amount of time that the connector waits before polling for new data from MongoDB. However, it does not guarantee that the connector will only load data into Kafka every 30 minutes.

If you want to load data into Kafka every 30 minutes, you could use a scheduler to run the Kafka Mongo Source Connector every 30 minutes. One way to do this is to use a cron job. Here’s an example of how you could set up a cron job to run the Kafka Mongo Source Connector every 30 minutes:

*/30 * * * * /path/to/kafka-mongo-source-connector.sh

This cron job will run the “kafka-mongo-source-connector.sh” script every 30 minutes. You would need to replace “/path/to/kafka-mongo-source-connector.sh” with the path to the script that runs the Kafka Mongo Source Connector.

Another option is to modify the Kafka Mongo Source Connector code to implement a custom polling interval. This would require more development work, but it would give you more control over the data loading process. You could modify the “poll” method in the connector code to wait for the specified interval before loading data into Kafka. However, I would recommend using a scheduler like cron instead, as it is a simpler and more reliable solution.

Hi @Brock ,

Thank you so much for your help