Docs Menu

Connector Message Processing Properties

On this page

  • Overview
  • Settings

Use the settings on this page to configure the message processing behavior of the sink connector including the following:

  • Message batch size
  • Rate limits
  • Number of parallel tasks

For a list of sink connector configuration settings organized by category, see the guide on Sink Connector Configuration Properties.

Name
Description
max.batch.size
Type: int

Description:
Maximum number of sink records to batch together for processing.

Consider the batch that contains the following records:
[ 1, 2, 3, 4, 5 ]
When set to 0, the connector performs a single bulk write for the entire batch.

When set to 1, the connector performs one bulk write for each record in the batch, for a total of five bulk writes as shown in the following example:
[1], [2], [3], [4], [5]
Default: 0
Accepted Values: An integer
bulk.write.ordered
Type: boolean

Description:
Whether the connector writes a batch of records as an ordered or unordered bulk write operation. When set to true, the default value, the connector writes a batch of records as an ordered bulk write operation.
To learn more about ordered and unordered bulk write operations, see Bulk Write Operations in the MongoDB Java driver documentation.
Tip
Performance Benefits

If you can write your data to MongoDB in any order, use unordered bulk write operations. Unordered bulk writes provide the following benefits:

  • MongoDB can execute unordered bulk writes in parallel
  • If you configured your connector to tolerate errors, unordered bulk writes help you lose less data.

To learn more about the performance benefits of unordered bulk write operations, see Unordered Operations in the MongoDB Manual.


Default: true
Accepted Values: true or false
rate.limiting.every.n
Type: int

Description:
Number of batches of records the sink connector processes in order to trigger the rate limiting timeout. A value of 0 means no rate limiting.

Default: 0
Accepted Values: An integer
rate.limiting.timeout
Type: int

Description:
How long (in milliseconds) to wait before the sink connector should resume processing after reaching the rate limiting threshold.

Default: 0
Accepted Values: An integer
tasks.max
Type: int

Description:
The maximum number of tasks to create for this connector. The connector may create fewer than the maximum tasks specified if it cannot handle the level of parallelism you specify.
Important
Multiple Tasks May Process Messages Out of Order

If you specify a value greater than 1, the connector enables parallel processing of the tasks. If your topic has multiple partition logs, which enables the connector to read from the topic in parallel, the tasks may process the messages out of order.

Default: 1
Accepted Values: An integer
←  Kafka Topic PropertiesConnector Error Handling Properties →

On this page

Give Feedback
© 2022 MongoDB, Inc.

About

  • Careers
  • Investor Relations
  • Legal Notices
  • Privacy Notices
  • Security Information
  • Trust Center
© 2022 MongoDB, Inc.