Batch.size & poll.max.batch.size

Working with Data Connectors & Integrations
2

Hi @Luan,

Great question!

  • poll.max.batch.size is the maximum number of records the source connector will wait for before publishing the data on the topic.
  • poll.await.time.ms is the maximum amount of time the source connector will wait before publishing the data on the topic.

So data is published to the topic when either of those limits is reached.

The source connector uses a change stream cursor underneath, which also can be configured.

  • batch.size configures the MongoDB cursor and specifies the maximum number of change events to return in each batch of the response from the MongoDB cluster. The default is 0 meaning it uses the servers default.

So the poll. configurations are to do with how often the connector should pass data to the topic and the batch.size is to do with the maximum amount of data to retrieve from MongoDB.

I hope that helps,

Ross

3 Likes