After fail-over, the message is not sent to kafka

Messages are not sent to kafka topic after failover. I’m using the mongodb source connector.

I checked in connect log, it seems that the old primary (mongodb1) automatically connects to the new primary (mongodb2) after server down.
However, even though the connector status is running, no message is sent to the topic. No errors have occurred.

Task restart, connector restart, pause, resume… No matter what I do, the message was not sent to the topic. Instead, I found that when the old primary(mongodb1) starts up, messages are sent to the topic again. I hope that when the new primary is elected, messages will continue to be sent to the topic.

What setting is required for mongodb or mongodb source connector? Should I do additional setting for this?

"config":{
   "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
   "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": true,
    "value.converter.schemas.enable": true,
    "connection.uri": "mongodb://mongoadm:mongoadm@mongodb1:27017,mongodb2:27017,mongodb3:27017/?replicaSet=rs01",
    "topic.prefix" : "metaDB",
    "database":"metaDB",
    "tasks.max":1,
    "poll.max.batch.size":1000,
    "poll.await.time.ms" :5000,
    "collection": "conntest",
    "copy.existing" : true,
    "batch.size" : 0,
    "change.stream.full.document":"updateLookup"
	}

It might be by design as change streams will pause notifications in this node failure scenario

1 Like

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.