Hi everyone,
We’re encountering the following error after some time while running the source connector for one of the collections in our database.
[2021-01-23 18:40:08,030] WARN [Producer clientId=connector-producer-source_transaction_existing_true_xy1-0] Got error produce response with correlation id 57021 on topic-partition mongo.primary.transaction-1, retrying (2147482595 attempts left). Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)
[2021-01-23 18:40:08,030] WARN [Producer clientId=connector-producer-source_transaction_existing_true_xy1-0] Received invalid metadata error in produce request on partition mongo.primary.transaction-1 due to org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received… Going to request metadata update now (org.apache.kafka.clients.producer.internals.Sender)
Connector config:
{
“connector.class”:“com.mongodb.kafka.connect.MongoSourceConnector”,
“publish.full.document.only”:“false”,
“output.schema.infer.value”:“false”,
“output.format.value”:“json”,
“change.stream.full.document”:“updateLookup”,
“collection”:“transaction”,
“output.format.key”:“json”,
“pipeline”:"[{ $unset: “lineItems” }, { $unset: “termsAndConditions” }, { $unset: “comments” }]",
“database”:“primary”,
“topic.prefix”:“mongo”,
“output.json.formatter”:“com.mongodb.kafka.connect.source.json.formatter.DefaultJson”,
“poll.await.time.ms”:“500”,
“name”:“source_transaction_7”,
“connection.uri”:“mongodb://[hidden]”,
“copy.existing”:“false”,
“value.converter”:“org.apache.kafka.connect.json.JsonConverter”,
“key.converter”:“org.apache.kafka.connect.json.JsonConverter”
}
To get it started again, we have to recreate the connector but that loses the offset and we have to manually recover the missing transactions. It’s worth mentioning that the connection is through Azure EventHub.
We tried to get help everywhere, but we couldn’t find anything that helps. We are running the same connector in our Dev and Staging environments where we are not seeing this issue.
Any help would be greatly appreciated.