Heartbeat topic configuration for Kafka Source Connector

Working with Data Connectors & Integrations
1

Hi.

We are considering using heartbeat mechanism to prevent the loss of the resume token when reading changes from a collection.

Our current scenario involves 6 collections being polled, each by one separate connector.

So far, we haven’t found any documentation about the heartbeat topic parameters. Any advise on how to choose, topic retention, topic partition number, compaction… or any other relevant heartbeat topic configs?

Also, given our scenario of several connectors, is it safe to share the heartbeat topic or should we create a separate topic per connector?

Thanks

5 Likes
2

Would love to get some guidance on this as well. We also have the multiple connectors configuration and am not sure if we should create one topic per connector, or if it’s safe for them to share.

1 Like
3

We are facing the very same issue too and can’t find any answer:

  1. Can a heartbeat topic be shared among multiple connectors?
  2. Are there any recommendations about topic retention, partition number or compaction?

It would be highly appreciated if anyone would be able to answer those questions.
Thanks a lot in advance!