Docs Menu

Docs Home → MongoDB Kafka Connector

What's New

On this page

  • What's New in 1.11
  • What's New in 1.10.1
  • What's New in 1.10
  • What's New in 1.9.1
  • What's New in 1.9
  • What's New in 1.8.1
  • What's New in 1.8
  • What's New in 1.7
  • What's New in 1.6.1
  • What's New in 1.6
  • What's New in 1.5
  • What's New in 1.4
  • What's New in 1.3
  • What's New in 1.2
  • What's New in 1.1
  • What's New in 1.0

Learn what's new by version:

What's New in 1.11

  • Added support for regular expressions in the topic.namespace.map property. To learn more about this feature and see an example of its use, see the Regular Expressions usage example in the Topic Naming page.

  • Added support for setting a custom delete write model strategy by using the delete.writemodel.strategy configuration property. To learn more, see Sink Connector Write Model Strategies.

  • Added the UpdateOneDefaultStrategy write model strategy. To learn more, see the list of Write Model Strategies.

  • Added the change.stream.document.key.as.key source connector configuration property. When set to true, the connector adds keys of the deleted documents to the tombstone events. When set to false, the connector uses the resume token as the source key for the tombstone events.

    Because this property is set to true by default, this might be a breaking change for some users. To learn more, see the list of Change Stream Properties.

  • DDL events from Debezium are recorded as no-ops and no longer cause an error.

What's New in 1.10.1

Important

Upgrade to Version 1.10.1

Version 1.9 introduced a bug related to MongoSourceTask.start that can cause a resource leak on both the connector side and the server side .

Upgrade to version 1.10.1 if you are using version 1.9 or 1.10 of the connector.

  • Fixed a resource leak related to MongoSourceTask.start that was introduced in version 1.9.

What's New in 1.10

  • Added the connector name to JMX monitoring metrics.

  • Added support for SSL by creating the following configuration options:

    • connection.ssl.truststore

    • connection.ssl.truststorePassword

    • connection.ssl.keystore

    • connection.ssl.keystorePassword

  • Ensured the driver parses config values from config providers before validating them.

What's New in 1.9.1

  • Corrected the behavior of schema inference for documents in nested arrays.

What's New in 1.9

  • Introduced the startup.mode=timestamp setting that allows you to start a Change Stream at a specific timestamp by setting the new startup.mode.timestamp.start.at.operation.time property.

  • Deprecated the copy.existing property and all copy.existing.* properties. You should use startup.mode=copy_existing and startup.mode.copy.existing.* properties to configure the copy existing feature.

  • Introduced the change.stream.full.document.before.change setting that allows you to access and configure the pre-image of an update operation in the change stream event document.

  • Improved schema inference for nested documents contained in arrays.

  • Introduced the publish.full.document.only.tombstones.on.delete setting that configures the connector to send tombstone events when documents are deleted. This setting only applies when publish.full.document.only is true.

  • Added MongoDB server exception information to dead letter queue messages.

What's New in 1.8.1

  • Corrected the type returned by getAttribute() and getAttributes() method calls in JMX MBeans to Attribute.

What's New in 1.8

Sink Connector

  • Added support for the Debezium MongoDB change stream CDC handler. You can now configure the connector to listen for events produced by this handler.

What's New in 1.7

  • Updated the MongoDB Java driver dependency to version 4.5

Sink Connector

  • Added dead letter queue error reports in the event the connector experiences bulk write errors

  • Added support for unordered bulk writes with the bulk.write.ordered configuration property

  • Added warning when attempting to use a Change Data Capture (CDC) handler with a post processor

  • Removed support for the max.num.retries configuration property

  • Removed support for the retries.defer.timeout configuration property

Important

Disable Retries Through Connection URI

To disable retries, specify the retryWrites=false option in your MongoDB connection URI.

The following configuration, which contains a placeholder MongoDB connection URI, disables retries:

connection.uri=mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/?replicaSet=myRepl&retryWrites=false

To learn more about connecting the MongoDB Kafka Connector to MongoDB, see the Connect to MongoDB guide.

To learn more about connection URI options, see the Connection Options guide in the MongoDB Java driver documentation.

Source Connector

Bug Fixes

  • Fixed Avro schema union validation

What's New in 1.6.1

  • Updated MongoDB Java driver dependency to 4.3.1 in the combined JARs

Bug Fixes

  • Fixed connection validator user privilege check

  • Fixed a bug in UuidProvidedIn[Key|Value]Strategy classes that prevented them from loading

What's New in 1.6

  • Added support for Stable API to force the server to run operations with behavior compatible with the specified API version

    Note

    Starting from February 2022, the Versioned API is known the Stable API. All concepts and features remain the same with this naming change.

  • Added error handling properties for the sink connector and source connector that can override the Kafka Connect framework's error handling behavior

  • Added mongo-kafka-connect-<version>-confluent.jar, which contains the connector and all dependencies needed to run it on the Confluent Platform

Sink Connector

Source Connector

  • No new changes, additions or improvements

Bug Fixes

  • Corrected the behavior of LazyBsonDocument#clone to respond to any changes made once unwrapped

  • Fixed the timestamp integer overflow in the Source Connector

  • Updated to enable recovery when calling the getMore() method in the Source Connector

  • Updated to enable recovery from broken change stream due to event sizes that are greater than 16 MB in the Source Connector

What's New in 1.5

  • Updated the MongoDB Java driver dependency to version 4.2

Sink Connector

  • Added the DeleteOneBusinessKeyStrategy write strategy to remove records from a topic

  • Added support for handling errant records that cause problems when processing them

  • Added support for Qlik Replicate Change Data Capture (CDC) to process event streams

Source Connector

  • Replaced BsonDocument with RawBsonDocument

  • Improved the copy.existing namespace handling

  • Improved the error messages for invalid pipeline operators

  • Improved the efficiency of heartbeats by making them tombstone messages

Bug Fixes

  • Corrected the inferred schema naming conventions

  • Updated to ensure that schemas can be backwards compatible

  • Fixed the Sink validation issue with topics.regex

  • Fixed the Sink NPE issue when using with Confluent Connect 6.1.0

  • Updated to ensure that the change stream cursor closes so it only reports errors that exist

  • Changed to include or exclude the _id field for a projection only if it's explicitly added

What's New in 1.4

  • Updated the MongoDB Java Driver to version 4.1

Sink Connector

  • Added support for Change Data Capture (CDC) based on MongoDB change stream events

  • Added the NamespaceMapper interface to allow for dynamic namespace mapping

Source Connector

  • Added the TopicMapper interface to allow topic mapping

Bug Fixes

  • Changed the top-level inferred schema to be mandatory

  • Fixed a validation issue and synthetic configuration property in the Sink Connector

  • Corrected general exception logging

  • Updated to clone the LazyBsonDocument instead of the unwrapped BsonDocument

What's New in 1.3

  • Added automated integration testing for the latest Kafka Connector and Confluent Platform versions to ensure compatibility

Sink Connector

  • Added support for records that contain Bson byte types

  • Added support for the errors.tolerance property

  • Changed max.num.retries default to 1

  • Improved the error messages for business key errors

  • Improved the error handling for List and JSON array configuration options

  • Updated to use the dot notation for filters in key update strategies

Source Connector

  • Added support to output a key or value as a Bson byte type

  • Added support for schema and custom Avro schema definitions

  • Added support for dead letter queue and the errors.tolerance property

  • Added configurations for the following formatters:

    • DefaultJson

    • ExtendedJson

    • SimplifiedJson

  • Added configuration for copy.existing.pipeline to allow you to use indexes during the copying process

  • Added configuration for copy.existing.namespace.regex to allow you to copy the filtering of namespaces

  • Added configuration for offset.partition.name to allow for custom partitioning naming strategies

  • Updated to validate that the fullDocument field is a document

  • Updated to sanitize the connection string in the offset partition map to improve maintenance of the connection.uri, database, and collection parameters

  • Updated to disable publishing a source record without a topic name

Bug Fixes

  • Stopped MongoDB 3.6 from copying existing issues when the collection didn't exist in the Source Connector

What's New in 1.2

Sink Connector

Important

We deprecated the following post processors:

  • BlacklistKeyProjector

  • BlacklistValueProjector

  • WhitelistKeyProjector

  • WhitelistValueProjector

If you are using one of these post processors, use the respective one instead for future compatibility:

  • BlockListKeyProjector

  • BlockListValueProjector,

  • AllowListKeyProjector

  • AllowListValueProjector

  • Added configurations for the following properties:

    • document.id.strategy.overwrite.existing

    • UuidStrategy output types

    • document.id.strategy.partial.value.projection.type

    • document.id.strategy.partial.value.projection.list

    • document.id.strategy.partial.key.projection.type

    • document.id.strategy.partial.key.projection.list

    • UuidProvidedInKeyStrategy

    • UuidProvidedInValueStrategy

  • Added the UpdateOneBusinessKeyTimestampStrategy post processor

  • Added built-in support for parallelism and scalable data copying by assigning topic partitions to tasks

Source Connector

  • Improved the error messaging for missing resume tokens

Bug Fixes

  • Removed failures with the MongoCopyDataManager when the source database does not exist

  • Fixed the copying the existing resumability error in the Source Connector

What's New in 1.1

Sink Connector

  • Added support for the topics.regex property

  • Updated to ignore unused source record key or value fields

  • Added validation for the connection using MongoSinkConnector.validate

Source Connector

  • Added validation for the connection using MongoSourceConnector.validate

Bug Fixes

  • Removed the "Unrecognized field: startAfter" error for resuming a change stream in the Source Connector

What's New in 1.0

The initial GA release.

←  MongoDB Kafka ConnectorQuick Start →

On this page

Share Feedback