Docs Menu
Docs Home
/
MongoDB Kafka Connector
/ /

Change Stream Properties

On this page

  • Overview
  • Settings

Use the following configuration settings to specify aggregation pipelines for change streams and read preferences for change stream cursors when working with the MongoDB Kafka source connector.

For a list of source connector configuration settings organized by category, see the guide on Source Connector Configuration Properties.

Name
Description
pipeline
Type: string

Description:
An array of aggregation pipelines to run in your change stream. You must configure this setting for the change stream event document, not the fullDocument field.

Example

[{"$match": { "$and": [{"operationType": "insert"}, {"fullDocument.eventId": 1404 }] } }]

Tip

Additional Examples

Default: "[]"
Accepted Values: Valid aggregation pipeline stage
change.stream.full.document
Type: string

Description:
Determines what values your change stream returns on update operations.
The default setting returns the differences between the original document and the updated document.
The updateLookup setting returns the differences between the original document and updated document as well as a copy of the entire updated document at a point in time after the update.
The whenAvailable setting returns the updated document, if available.
The required setting returns the updated document and raises an error if it is not available.

Tip

For more information on how this change stream option works, see the MongoDB server manual guide on Lookup Full Document for Update Operations.


Default: ""
Accepted Values: "", "updateLookup", "whenAvailable", or "required"
change.stream.full.document.before.change
Type: string

Description:
Configures the document pre-image your change stream returns on update operations. The pre-image is not available for source records published while copying existing data, and the pre-image configuration has no effect on copying.

Tip

To learn how to configure a collection to enable pre-images, see the Server manual entry on pre- and post-images.

The default setting suppresses the document pre-image.
The whenAvailable setting returns the document pre-image if it's available, before it was replaced, updated, or deleted.
The required setting returns the document pre-image and raises an error if it is not available.

Default: ""
Accepted Values: "" or "whenAvailable" or "required"
publish.full.document.only
Type: boolean

Description:
Whether to return only the fullDocument field from the change stream event document produced by any update event. The fullDocument field contains the most current version of the updated document. To learn more about the fullDocument field, see the update Event in the Server manual.
When set to true, the connector overrides the change.stream.full.document setting and sets it to updateLookup so that the fullDocument field contains updated documents.
Default: false
Accepted Values: true or false
publish.full.document.only.tombstone.on.delete
Type: boolean

Description:
Whether to return tombstone events when documents are deleted. Tombstone events contain the keys of deleted documents with null values. This setting applies only when publish.full.document.only is true.
Default: false
Accepted Values: true or false
change.stream.document.key.as.key
Type: boolean

Description:
Whether to use the document key for the source record key if the document key is present.
When set to true, the connector adds keys of the deleted documents to the tombstone events. When set to false, the connector uses the resume token as the source key for the tombstone events.
Default: true
Accepted Values: true or false
collation
Type: string

Description:
A JSON collation document that specifies language-specific ordering rules that MongoDB applies to the documents returned by the change stream.

Default: ""
Accepted Values: A valid collation JSON document
batch.size
Type: int

Description:
The change stream cursor batch size.

Default: 0
Accepted Values: An integer
poll.await.time.ms
Type: long

Description:
The maximum amount of time in milliseconds that the server waits for new data changes to report to the change stream cursor before returning an empty batch.

Default: 5000
Accepted Values: An integer
poll.max.batch.size
Type: int

Description:
Maximum number of documents to read in a single batch when polling a change stream cursor for new data. You can use this setting to limit the amount of data buffered internally in the connector.

Default: 1000
Accepted Values: An integer

Back

Kafka Topic

On this page