Docs Menu

Docs HomeMongoDB Kafka Connector

What's New

On this page

  • What's New in 1.7
  • What's New in 1.6.1
  • What's New in 1.6
  • What's New in 1.5
  • What's New in 1.4
  • What's New in 1.3
  • What's New in 1.2
  • What's New in 1.1
  • What's New in 1.0

Learn what's new by version:

  • Updated the MongoDB Java driver dependency to version 4.5

  • Added dead letter queue error reports in the event the connector experiences bulk write errors

  • Added support for unordered bulk writes with the bulk.write.ordered configuration property

  • Added warning when attempting to use a Change Data Capture (CDC) handler with a post processor

  • Removed support for the max.num.retries configuration property

  • Removed support for the retries.defer.timeout configuration property

Important

Disable Retries Through Connection URI

To disable retries, specify the retryWrites=false option in your MongoDB connection URI.

The following configuration, which contains a placeholder MongoDB connection URI, disables retries:

connection.uri=mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/?replicaSet=myRepl&retryWrites=false

To learn more about connecting the MongoDB Kafka Connector to MongoDB, see the Connect to MongoDB guide.

To learn more about connection URI options, see the Connection Options guide in the MongoDB Java driver documentation.

  • Fixed Avro schema union validation

  • Updated MongoDB Java driver dependency to 4.3.1 in the combined JARs

  • Fixed connection validator user privilege check

  • Fixed a bug in UuidProvidedIn[Key|Value]Strategy classes that prevented them from loading

  • Added support for Stable API to force the server to run operations with behavior compatible with the specified API version

    Note

    Starting from February 2022, the Versioned API is known the Stable API. All concepts and features remain the same with this naming change.

  • Added error handling properties for the sink connector and source connector that can override the Kafka Connect framework's error handling behavior

  • Added mongo-kafka-connect-<version>-confluent.jar, which contains the connector and all dependencies needed to run it on the Confluent Platform

  • No new changes, additions or improvements

  • Corrected the behavior of LazyBsonDocument#clone to respond to any changes made once unwrapped

  • Fixed the timestamp integer overflow in the Source Connector

  • Updated to enable recovery when calling the getMore() method in the Source Connector

  • Updated to enable recovery from broken change stream due to event sizes that are greater than 16 MB in the Source Connector

  • Updated the MongoDB Java driver dependency to version 4.2

  • Added the DeleteOneBusinessKeyStrategy write strategy to remove records from a topic

  • Added support for handling errant records that cause problems when processing them

  • Added support for Qlik Replicate Change Data Capture (CDC) to process event streams

  • Replaced BsonDocument with RawBsonDocument

  • Improved the copy.existing namespace handling

  • Improved the error messages for invalid pipeline operators

  • Improved the efficiency of heartbeats by making them tombstone messages

  • Corrected the inferred schema naming conventions

  • Updated to ensure that schemas can be backwards compatible

  • Fixed the Sink validation issue with topics.regex

  • Fixed the Sink NPE issue when using with Confluent Connect 6.1.0

  • Updated to ensure that the change stream cursor closes so it only reports errors that exist

  • Changed to include or exclude the _id field for a projection only if it's explicitly added

  • Updated the MongoDB Java Driver to version 4.1

  • Added support for Change Data Capture (CDC) based on MongoDB change stream events

  • Added the NamespaceMapper interface to allow for dynamic namespace mapping

  • Added the TopicMapper interface to allow topic mapping

  • Changed the top-level inferred schema to be mandatory

  • Fixed a validation issue and synthetic configuration property in the Sink Connector

  • Corrected general exception logging

  • Updated to clone the LazyBsonDocument instead of the unwrapped BsonDocument

  • Added automated integration testing for the latest MongoDB Kafka Connector and Confluent Platform versions to ensure compatibility

  • Added support for records that contain Bson byte types

  • Added support for the errors.tolerance property

  • Changed max.num.retries default to 1

  • Improved the error messages for business key errors

  • Improved the error handling for List and JSON array configuration options

  • Updated to use the dot notation for filters in key update strategies

  • Added support to output a key or value as a Bson byte type

  • Added support for schema and custom Avro schema definitions

  • Added support for dead letter queue and the errors.tolerance property

  • Added configurations for the following formatters:

    • DefaultJson

    • ExtendedJson

    • SimplifiedJson

  • Added configuration for copy.existing.pipeline to allow you to use indexes during the copying process

  • Added configuration for copy.existing.namespace.regex to allow you to copy the filtering of namespaces

  • Added configuration for offset.partition.name to allow for custom partitioning naming strategies

  • Updated to validate that the fullDocument field is a document

  • Updated to sanitize the connection string in the offset partition map to improve maintenance of the connection.uri, database, and collection parameters

  • Updated to disable publishing a source record without a topic name

  • Stopped MongoDB 3.6 from copying existing issues when the collection didn't exist in the Source Connector

Important

We deprecated the following post processors:

  • BlacklistKeyProjector

  • BlacklistValueProjector

  • WhitelistKeyProjector

  • WhitelistValueProjector

If you are using one of these post processors, use the respective one instead for future compatibility:

  • BlockListKeyProjector

  • BlockListValueProjector,

  • AllowListKeyProjector

  • AllowListValueProjector

  • Added configurations for the following properties:

    • document.id.strategy.overwrite.existing

    • UuidStrategy output types

    • document.id.strategy.partial.value.projection.type

    • document.id.strategy.partial.value.projection.list

    • document.id.strategy.partial.key.projection.type

    • document.id.strategy.partial.key.projection.list

    • UuidProvidedInKeyStrategy

    • UuidProvidedInValueStrategy

  • Added the UpdateOneBusinessKeyTimestampStrategy post processor

  • Added built-in support for parallelism and scalable data copying by assigning topic partitions to tasks

  • Improved the error messaging for missing resume tokens

  • Removed failures with the MongoCopyDataManager when the source database does not exist

  • Fixed the copying the existing resumability error in the Source Connector

  • Added support for the topics.regex property

  • Updated to ignore unused source record key or value fields

  • Added validation for the connection using MongoSinkConnector.validate

  • Added validation for the connection using MongoSourceConnector.validate

  • Removed the "Unrecognized field: startAfter" error for resuming a change stream in the Source Connector

The initial GA release.

←  MongoDB Kafka ConnectorQuick Start →