I am running Apache Kafka on my Windows machine with two Kafka-Connect-Workers(Port 8083, 8084) and three partitions(replication of one).
My issue is that I am able to see the fail-over to other Kafka-Connect worker whenever I shutdown one of them but load balancing is not happening because the number of tasks is always ONE.
I am using Official MongoDB-Kafka-Connector as Source(ChangeStream) with tasks.max=6.
I tried updating MongoDB with multiple threads so that it could push more data into Kafka-Connect and may perhaps make Kafka-Connect create more tasks. Even under higher volume of data, tasks count remain one.
API always shows one task: “http://localhost:8083/connectors/mongodb-connector/status” :
Response:
{
"name":"mongodb-connector",
"connector": {
"state":"RUNNING",
"worker_id":"<ipaddress>:8083"
}
"tasks": [
{
"id": 0,
"state": "RUNNING"
"worker_id": "<ipaddress>:8083"
}
],
"type": "source"
}
Am I missing something here? Why more tasks are not created?