Hi I’m new to the community and kafka connect.
I’m trying to configure MongoDB source connector as below to produce kafka event message with payload also in json.
For e.g.)
{
"name": "mongo-native-cdc-test-connector",
"config": {
"name":"mongo-native-cdc-test-connector",
"connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
"connection.uri":"<connection url>",
"startup.mode":"latest",
"output.format.key":"json",
"output.format.value":"json",
"key.converter.schema.enabled":false,
"value.converter.schema.enabled":false,
"key.converter":"org.apache.kafka.connect.json.JsonConverter",
"value.converter":"org.apache.kafka.connect.json.JsonConverter"
}
}
However this produces payload in json string such as
{
"schema": {
"type": "string",
"optional": false
},
"payload": "{\"_id\": {\"_data\": \"***\"}, \"operationType\": \"insert\", \"clusterTime\": {\"$timestamp\": {\"t\": 1677466433, \"i\": 1}}, \"fullDocument\": {\"_id\": {\"$oid\": \"***\"}, \"a\": \"name12\", \"b\": \"value12\"}, \"ns\": {\"db\": \"***\", \"coll\": \"***\"}, \"documentKey\": {\"a\": \"name12\", \"_id\": {\"$oid\": \"***\"}}}"
}
Weirdly, I found out that when I use String converter for value Converter, it produces the payload in json as follows.
{
"_id": {
"_data": "***"
},
"operationType": "insert",
"clusterTime": {
"$timestamp": {
"t": 1677811331,
"i": 1
}
},
"fullDocument": {
"_id": {
"$oid": "***"
},
"a": "name21",
"b": "value21"
},
"ns": {
"db": "***",
"coll": "***"
},
"documentKey": {
"a": "name21",
"_id": {
"$oid": "***"
}
}
so I was wondering if this is the correct behavior? I assumed that in order to produce json format i need to use jsonConverter and now i’m really confused what’s going on under the hood… or am I missing something…?
Any reply regarding this would be much appreciated! Thanks in advance.