What's New
On this page
Learn what's new by version:
- Version 1.7
- Version 1.6.1
- Version 1.6
- Version 1.5
- Version 1.4
- Version 1.3
- Version 1.2
- Version 1.1
- Version 1.0
What's New in 1.7
- Updated the MongoDB Java driver dependency to version 4.5
Sink Connector
- Added dead letter queue error reports in the event the connector experiences bulk write errors
- Added support for unordered bulk writes with the
bulk.write.ordered
configuration property - Added warning when attempting to use a Change Data Capture (CDC) handler with a post processor
- Removed support for the
max.num.retries
configuration property - Removed support for the
retries.defer.timeout
configuration property
To disable retries, specify the retryWrites=false
option
in your MongoDB connection URI.
The following configuration, which contains a placeholder MongoDB connection URI, disables retries:
connection.uri=mongodb://mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/?replicaSet=myRepl&retryWrites=false
To learn more about connecting the MongoDB Kafka Connector to MongoDB, see the Connect to MongoDB guide.
To learn more about connection URI options, see the Connection Options guide in the MongoDB Java driver documentation.
Source Connector
- Added support for user-defined topic separators with the
topic.separator
configuration property - Added support for the
allow disk use
field of the MongoDB Query API in the copy existing aggregation with the
copy.existing.allow.disk.use
configuration property - Added support for Avro schema namespaces
in the
output.schema.value
andoutput.schema.key
configuration properties
Bug Fixes
- Fixed Avro schema union validation
What's New in 1.6.1
- Updated MongoDB Java driver dependency to 4.3.1 in the combined JARs
Bug Fixes
- Fixed connection validator user privilege check
- Fixed a bug in
UuidProvidedIn[Key|Value]Strategy
classes that prevented them from loading
What's New in 1.6
Added support for Stable API to force the server to run operations with behavior compatible with the specified API version
NoteStarting from Feburary 2022, the Versioned API is known the Stable API. All concepts and features remain the same with this naming change.
- Added error handling properties for the sink connector and source connector that can override the Kafka Connect framework's error handling behavior
- Added
mongo-kafka-connect-<version>-confluent.jar
, which contains the connector and all dependencies needed to run it on the Confluent Platform
Sink Connector
- Added support for automatic time-series collection creation in MongoDB 5.0 to efficiently store sequences of measurements over a period of time. Learn how to configure connectors to Migrate an Existing Collection to a Time Series Collection.
- Improved the error logging for bulk write exceptions
Source Connector
- No new changes, additions or improvements
Bug Fixes
- Corrected the behavior of
LazyBsonDocument#clone
to respond to any changes made once unwrapped - Fixed the timestamp integer overflow in the Source Connector
- Updated to enable recovery when calling the
getMore()
method in the Source Connector - Updated to enable recovery from broken change stream due to event sizes that are greater than 16 MB in the Source Connector
What's New in 1.5
- Updated the MongoDB Java driver dependency to version 4.2
Sink Connector
- Added the
DeleteOneBusinessKeyStrategy
write strategy to remove records from a topic - Added support for handling errant records that cause problems when processing them
- Added support for Qlik Replicate Change Data Capture (CDC) to process event streams
Source Connector
- Replaced
BsonDocument
withRawBsonDocument
- Improved the
copy.existing
namespace handling - Improved the error messages for invalid pipeline operators
- Improved the efficiency of heartbeats by making them tombstone messages
Bug Fixes
- Corrected the inferred schema naming conventions
- Updated to ensure that schemas can be backwards compatible
- Fixed the Sink validation issue with
topics.regex
- Fixed the Sink NPE issue when using with Confluent Connect 6.1.0
- Updated to ensure that the change stream cursor closes so it only reports errors that exist
- Changed to include or exclude the
_id
field for a projection only if it's explicitly added
What's New in 1.4
- Updated the MongoDB Java Driver to version 4.1
Sink Connector
- Added support for Change Data Capture (CDC) based on MongoDB change stream events
- Added the
NamespaceMapper
interface to allow for dynamic namespace mapping
Source Connector
- Added the
TopicMapper
interface to allow topic mapping
Bug Fixes
- Changed the top-level inferred schema to be mandatory
- Fixed a validation issue and synthetic configuration property in the Sink Connector
- Corrected general exception logging
- Updated to clone the
LazyBsonDocument
instead of the unwrappedBsonDocument
What's New in 1.3
- Added automated integration testing for the latest Kafka Connector and Confluent Platform versions to ensure compatibility
Sink Connector
- Added support for records that contain
Bson
byte types - Added support for the
errors.tolerance
property - Changed
max.num.retries
default to1
- Improved the error messages for business key errors
- Improved the error handling for
List
and JSON array configuration options - Updated to use the dot notation for filters in key update strategies
Source Connector
- Added support to output a key or value as a
Bson
byte type - Added support for schema and custom Avro schema definitions
- Added support for dead letter queue and the
errors.tolerance
property Added configurations for the following formatters:
DefaultJson
ExtendedJson
SimplifiedJson
- Added configuration for
copy.existing.pipeline
to allow you to use indexes during the copying process - Added configuration for
copy.existing.namespace.regex
to allow you to copy the filtering of namespaces - Added configuration for
offset.partition.name
to allow for custom partitioning naming strategies - Updated to validate that the
fullDocument
field is a document - Updated to sanitize the connection string in the offset partition map to
improve maintenance of the
connection.uri
,database
, andcollection
parameters - Updated to disable publishing a source record without a topic name
Bug Fixes
- Stopped MongoDB 3.6 from copying existing issues when the collection didn't exist in the Source Connector
What's New in 1.2
Sink Connector
We deprecated the following post processors:
BlacklistKeyProjector
BlacklistValueProjector
WhitelistKeyProjector
WhitelistValueProjector
If you are using one of these post processors, use the respective one instead for future compatibility:
BlockListKeyProjector
BlockListValueProjector
,AllowListKeyProjector
AllowListValueProjector
Added configurations for the following properties:
document.id.strategy.overwrite.existing
UuidStrategy
output typesdocument.id.strategy.partial.value.projection.type
document.id.strategy.partial.value.projection.list
document.id.strategy.partial.key.projection.type
document.id.strategy.partial.key.projection.list
UuidProvidedInKeyStrategy
UuidProvidedInValueStrategy
- Added the
UpdateOneBusinessKeyTimestampStrategy
post processor - Added built-in support for parallelism and scalable data copying by assigning topic partitions to tasks
Source Connector
- Improved the error messaging for missing resume tokens
Bug Fixes
- Removed failures with the
MongoCopyDataManager
when the source database does not exist - Fixed the copying the existing resumability error in the Source Connector
What's New in 1.1
Sink Connector
- Added support for the
topics.regex
property - Updated to ignore unused source record key or value fields
- Added validation for the connection using
MongoSìnkConnector.validate
Source Connector
- Added validation for the connection using
MongoSourceConnector.validate
Bug Fixes
- Removed the
"Unrecognized field: startAfter"
error for resuming a change stream in the Source Connector
What's New in 1.0
The initial GA release.