Hello,
Faced the following behavior while using connector 1.6.1:
Need to publish messages in topic as a diffs, according to documentation it were a common connector use case some time ago. But now, according to docs, for this case the following parameter must be used:
"change.stream.full.document" : "default"
The
default
setting returns the differences between the original document and the updated document.
Created connector with following configuration:
{
"name" : "msc-connector-test",
"config" : {
"batch.size" : "8192",
"connection.uri" : "***",
"connector.class" : "com.mongodb.kafka.connect.MongoSourceConnector",
"copy.existing" : "false",
"collection" : "***",
"database" : "***",
"heartbeat.interval.ms" : "5000",
"heartbeat.topic.name" : "__msc_heartbeat-topic",
"key.converter" : "org.apache.kafka.connect.storage.StringConverter",
"key.converter.schemas.enable" : "false",
"name" : "msc-connector-test",
"output.json.formatter" : "com.mongodb.kafka.connect.source.json.formatter.SimplifiedJson",
"change.stream.full.document" : "default",
"topic.creation.default.partitions" : "3",
"topic.creation.default.replication.factor" : "3",
"transforms" : "dropPrefix",
"transforms.dropPrefix.regex" : "(.*)<db_name>(.*)",
"transforms.dropPrefix.replacement" : "msc-topic-test",
"transforms.dropPrefix.type" : "org.apache.kafka.connect.transforms.RegexRouter",
"value.converter" : "org.apache.kafka.connect.storage.StringConverter",
"value.converter.schemas.enable" : "false"
}
}
But published messages have following format:
{
"_id" : {
"_data" : "***"
},
"operationType" : "replace",
"clusterTime" : {
"$timestamp" : {
"t" : 1646896094,
"i" : 1
}
},
"fullDocument" : {
***
},
"ns" : {
"db" : "***",
"coll" : "***"
},
"documentKey" : {
"_id" : "***"
}
}
There’s no diffs in messages, just an updated fullDocument.
Is this an expected behavior since 1.6 update or something wrong or missed in connector configuration?