Docs Menu

Docs HomeMongoDB Kafka Connector

Monitoring

On this page

Learn how to observe the behavior of your MongoDB source or sink connector through monitoring. Monitoring is the process of getting information about the activities a running program performs for use in an application or an application performance management library.

To learn how monitoring works in the connector and how to use it, see the Use Cases section.

To view an example that shows how to monitor a running connector, see the Example - Monitor the Quick Start section.

To view a list of all metrics produced by MongoDB source and sink connectors, see the Available Metrics section.

This section describes use cases for monitoring MongoDB source and sink connectors, and how you can use the metrics your connector provides to satisfy those use cases.

Tip

Computed Values

To learn what types of metrics the connector provides and when you must implement logic to compute a value, see Types of Metrics.

The following table describes some use cases for monitoring the MongoDB sink connector and the metrics the sink connector provides to satisfy those use cases:

Use Case
Metrics to Use
You need to know if a component of your pipeline is falling behind.
Use the latest-kafka-time-difference-ms metric. This metric indicates the interval of time between when a record arrived in a Kafka topic and when your connector received that record. If the value of this metric is increasing, it signals that there may be a problem with Apache Kafka or MongoDB.
You need to know the total number of records your connector wrote to MongoDB.
Use the records metric.
You need to know the total number of write errors your connector encountered when attempting to write to MongoDB.
Use the batch-writes-failed metric.
You need to know if your MongoDB performance is getting slower over time.

Use the in-task-put-duration-ms metric to initally diagnose a slowdown.

Use the following metrics to further diagnose any issues:

  • batch-writes-successful-duration-over-<number>-ms

  • batch-writes-failed-duration-over-<number>-ms

  • processing-phase-duration-over-<number>-ms

You need to find a bottleneck in how Kafka Connect and your MongoDB sink connector write Apache Kafka records to MongoDB.

Compare the values of the following metrics:

  • in-task-put-duration-ms

  • in-connect-framework-duration-ms

You can find descriptions of all MongoDB sink connector metrics in the Sink Connector JMX Metrics section.

The following table describes some use cases for monitoring the MongoDB source connector and the metrics the source connector provides to satisfy those use cases:

Use Case
Metrics to Use
You need to know if a component of your pipeline is falling behind.
Use the latest-mongodb-time-difference-secs metric. This metric indicates how old the most recent change stream event your connector processed is. If this metric is increasing, it signals that there may be a problem with Apache Kafka or MongoDB.
You need to know the total number of change stream events your source connector has processed.
Use the records metric.
You need to know the percentage of records your connector received but failed to write to Apache Kafka.

Perform the following calculation with the records, records-filtered, and records-acknowledged metrics:

(records - (records-acknowledged + records-filtered)) / records
You need to know the average size of the documents your connector has processed.

Perform the following calculation with the mongodb-bytes-read and records metrics:

mongodb-bytes-read / records

To learn how to calculate the average size of records over a span of time, see mongodb-bytes-read.

You need to find a bottleneck in how Kafka Connect and your MongoDB source connector write MongoDB documents to Apache Kafka.

Compare the values of the following metrics:

  • in-task-poll-duration-ms

  • in-connect-framework-duration-ms

You need to know if your MongoDB performance is getting slower over time.

Use the in-task-poll-duration-ms metric to initially diagnose a slowdown.

Use the following metrics to further diagnose any issues:

  • initial-commands-successful-duration-over-<number>-ms

  • initial-commands-failed-duration-over-<number>-ms

  • getmore-commands-successful-duration-over-<number>-ms

  • getmore-commands-failed-duration-over-<number>-ms

You can find descriptions of all MongoDB source connector metrics in the Source Connector JMX Metrics section.

The MongoDB Kafka Connector uses Java Management Extensions (JMX) to enable monitoring. JMX is a technology included in the Java Platform, Standard Edition that provides tools to monitor applications and devices. You can view the metrics produced by the connector with any JMX console, such as JConsole.

The MongoDB Kafka Connector provides metrics for individual tasks. Tasks are classes instantiated by Kafka Connect that copy data to and from datastores and Apache Kafka. The names and responsibilities of the two types of tasks in Kafka Connect are as follows:

  • A source task copies data from a data store to Apache Kafka.

  • A sink task copies data from Apache Kafka to a data store.

A sink connector configures one or more sink tasks. A source connector configures one or more source tasks.

To learn more about JMX, see the following resources from Oracle:

To learn more about tasks and connectors in Kafka Connect, see the following resources:

The MongoDB Kafka Connector uses Kafka Connect's metrics infrastructure to serve metrics. To read the metrics produced by your connector, enable JMX in your Kafka Connect deployment.

To learn how to enable JMX for a Kafka Connect instance running on your host machine, see the Official Kafka Documentation.

To learn how to enable JMX for a containerized Kafka Connect deployment, see Kafka Monitoring and Metrics Using JMX with Docker.

The connector provides metrics related to the following types of quantities:

  • The number of times an event has occurred in total for a connector task

  • The value related to the most recent occurrence of an event

For some use cases, you must perform extra computations with the metrics the connector provides. For example, you must compute the following values from provided metrics:

  • The rate of change of a metric

  • The value of a metric over a span of time

  • The difference between one metric and another metric

To view some examples of computed metrics, see the Use Cases section.

The MongoDB Kafka Connector and Kafka Connect both produce metrics for MongoDB connector tasks.

Both sets of metrics provide information about how your tasks interact with Kafka Connect, but only the MongoDB Kafka Connector metrics provide information about how your tasks interact with MongoDB.

The MongoDB Kafka Connector produces metrics under the following JMX paths:

  • com.mongodb.kafka.connect.sink-task-metrics.sink-task-<monitonically increasing number>

  • com.mongodb.kafka.connect.source-task-metrics.source-task-<monitonically increasing number>

  • com.mongodb.kafka.connect.source-task-metrics.source-task-change-stream-<monitonically increasing number>

  • com.mongodb.kafka.connect.source-task-metrics.source-task-copy-existing-<monitonically increasing number>

Kafka Connect produces metrics under the following JMX paths:

  • kafka.connect.sink-task-metrics.<connector-name>

  • kafka.connect.source-task-metrics.<connector-name>

  • kafka.connect.connector-task-metrics.<connector-name>

To relate Kafka Connect metrics to MongoDB Kafka Connector metrics, you must remember the order in which you added your connectors to Kafka Connect.

Note

Naming Conflicts

If the MongoDB Kafka Connector ever encounters a naming conflict when it attempts to register an MBean on a JMX path, the MongoDB Kafka Connector adds a version suffix to the MBean.

For example, if the connector tries to register an MBean under the path com.mongodb.kafka.connect.sink-task-metrics.sink-task-0 and is unable to do so, it attempts to register the MBean under com.mongodb.kafka.connect.sink-task-metrics.sink-task-0-v1.

Assume you add a single MongoDB source connector named my-source-connector to your deployment.

The MongoDB source connector writes metrics to the following JMX path:

  • com.mongodb.kafka.connect.sink-task-metrics.sink-task-0

Kafka Connect writes metrics for this task under the following path:

  • kafka.connect.sink-task-metrics.my-source-connector

The sample environment provided in the Quick Start exposes metrics on your host machine at the URI localhost:35000.

To view these metrics with JConsole, perform the following actions:

1

JConsole is part of the Java Platform, Standard Edition. To download JConsole, download the Java SE Development Kit from Oracle.

2

Follow the Quick Start until the Send the Contents of a Document through Your Connectors step.

3

Run the following command from your command line to start JConsole:

jconsole
4
1

Enter the URI localhost:35000 into the Remote Process text input box in the JConsole interface.

2
3
5
1
2

Notice that the com.mongodb.kafka.connect.sink-task-metrics.sink-task-0.records attribute has a value of 0. This value indicates that your sink task has not recieved any records from Apache Kafka.

3

Continue the Quick Start until, but not through, the Remove the Sandbox step.

4

Navigate back to the MBeans tab in JConsole. The com.mongodb.kafka.connect.sink-task-metrics.sink-task-0.records attribute should now have a value of 1.

6

To stop and remove the Quick Start environment, follow the Remove the Sandbox step of the Quick Start.

Use the attributes in the tables in this section to monitor the behavior of your source and sink connectors through Java Management Extensions (JMX).

Tip

JMX Attributes

JMX represents an individual metric as an attribute of an MBean. To learn more about attributes and MBeans, see the Standard MBeans Tutorial from Oracle.

Note

Poll and Put Methods

A MongoDB source connector task has a poll() method to retrieve documents from MongoDB and send them to Apache Kafka. A MongoDB sink connector task has a put() method to retrieve documents from Apache Kafka and send them to MongoDB.

To learn more about poll() and put() methods, see the following resources:

Attribute Name
Description
records
The total number of Kafka records a MongoDB sink task received.
records-successful
The total number of Kafka records a MongoDB sink task successfully wrote to MongoDB.
records-failed
The total number of Kafka records a MongoDB sink task failed to write to MongoDB.
latest-kafka-time-difference-ms
The number of milliseconds of the most recent time difference recorded between a MongoDB sink task and Kafka. This value is calculated by subtracting the current time of the connector's clock and the timestamp of the last record the task received.
in-task-put
The total number of times the Kafka Connect framework executed the put() method of the MongoDB sink task.
in-task-put-duration-ms
The total number of milliseconds the Kafka Connect framework spent executing the put() method of a MongoDB sink task.
in-task-put-duration-over-1-ms
The total number of MongoDB sink task put() method executions with a duration that exceeded 1 millisecond.
in-task-put-duration-over-10-ms
The total number of MongoDB sink task put() method executions with a duration that exceeded 10 milliseconds.
in-task-put-duration-over-100-ms
The total number of MongoDB sink task put() method executions with a duration that exceeded 100 milliseconds.
in-task-put-duration-over-1000-ms
The total number of MongoDB sink task put() method executions with a duration that exceeded 1000 milliseconds.
in-task-put-duration-over-10000-ms
The total number of MongoDB sink task put() method executions with a duration that exceeded 10000 milliseconds.
in-connect-framework
The total number of times code in the Kafka Connect framework executed after the first invocation of the put() method of the MongoDB sink task.
in-connect-framework-duration-ms
The total number of milliseconds spent executing code in the Kafka Connect framework since the framework first invoked the put() method of the MongoDB sink task. This metric does not count time executing code in the MongoDB sink task towards the total.
in-connect-framework-duration-over-1-ms
The total number of times code in the Kafka Connect framework executed for a duration that exceeded 1 millisecond.
in-connect-framework-duration-over-10-ms
The total number of times code in the Kafka Connect framework executed for a duration that exceeded 10 milliseconds.
in-connect-framework-duration-over-100-ms
The total number of times code in the Kafka Connect framework executed for a duration that exceeded 100 milliseconds.
in-connect-framework-duration-over-1000-ms
The total number of times code in the Kafka Connect framework executed for a duration that exceeded 1000 milliseconds.
in-connect-framework-duration-over-10000-ms
The total number of times code in the Kafka Connect framework executed for a duration that exceeded 10000 milliseconds.
processing-phases
The total number of times a MongoDB sink task executed the processing phase on a batch of records from Kafka. The processing phase of a MongoDB sink task is the set of actions that starts after records are obtained from Kafka and ends before records are written to MongoDB.
processing-phases-duration-ms
The total number of milliseconds a MongoDB sink task spent processing records before writing them to MongoDB.
processing-phases-duration-over-1-ms
The total number of MongoDB sink task processing phase executions with a duration that exceeded 1 millisecond.
processing-phases-duration-over-10-ms
The total number of MongoDB sink task processing phase executions with a duration that exceeded 10 milliseconds.
processing-phases-duration-over-100-ms
The total number of MongoDB sink task processing phase executions with a duration that exceeded 100 milliseconds.
processing-phases-duration-over-1000-ms
The total number of MongoDB sink task processing phase executions with a duration that exceeded 1000 milliseconds.
processing-phases-duration-over-10000-ms
The total number of MongoDB sink task processing phase executions with a duration that exceeded 10000 milliseconds.
batch-writes-successful
The total number of batches a MongoDB sink task successfully wrote to the MongoDB server.
batch-writes-successful-duration-ms
The total number of milliseconds a MongoDB sink task spent successfully writing to the MongoDB server.
batch-writes-successful-duration-over-1-ms
The total number of successful batch writes performed by the MongoDB sink task with a duration that exceeded 1 millisecond.
batch-writes-successful-duration-over-10-ms
The total number of successful batch writes performed by the MongoDB sink task with a duration that exceeded 10 milliseconds.
batch-writes-successful-duration-over-100-ms
The total number of successful batch writes performed by the MongoDB sink task with a duration that exceeded 100 milliseconds.
batch-writes-successful-duration-over-1000-ms
The total number of successful batch writes performed by the MongoDB sink task with a duration that exceeded 1000 milliseconds.
batch-writes-successful-duration-over-10000-ms
The total number of successful batch writes performed by the MongoDB sink task with a duration that exceeded 10000 milliseconds.
batch-writes-failed
The total number of batches a MongoDB sink task failed to write to the MongoDB server.
batch-writes-failed-duration-ms
The total number of milliseconds a MongoDB sink task spent unsuccessfully attempting to write batches to the MongoDB server.
batch-writes-failed-duration-over-1-ms
The total number of failed batch writes attempted by the MongoDB sink task with a duration that exceeded 1 millisecond.
batch-writes-failed-duration-over-10-ms
The total number of failed batch writes attempted by the MongoDB sink task with a duration that exceeded 10 milliseconds.
batch-writes-failed-duration-over-100-ms
The total number of failed batch writes attempted by the MongoDB sink task with a duration that exceeded 100 milliseconds.
batch-writes-failed-duration-over-1000-ms
The total number of failed batch writes attempted by the MongoDB sink task with a duration that exceeded 1000 milliseconds.
batch-writes-failed-duration-over-10000-ms
The total number of failed batch writes attempted by the MongoDB sink task with a duration that exceeded 10000 milliseconds.

Note

Initial Commands and getMore Commands

Some metrics for source connector tasks distinguish between initial commands and getMore commands. An initial command is a find or aggregate command sent to a MongoDB instance that retrieves the first set of documents in a client-side MongoDB cursor. The getMore command is the MongoDB command that fetches the subsequent sets of documents in a cursor.

To learn more about getMore commands, see the getMore page.

Attribute Name
Description
records
The total number of records a MongoDB source task passed to the Kafka Connect framework.
records-filtered
The number of records a MongoDB source task passed to the Kafka Connect framework that were then filtered by the framework. A filtered record is not written to Kafka.
records-acknowledged
The total number of records a MongoDB source task passed to the Kafka Connect framework that were then successfully written to Kafka.
mongodb-bytes-read

The total number of bytes a MongoDB source task read from the MongoDB server.

Tip

Average Record Size Over a Span of Time

To calculate the average size of the records your sink connector processed over a span of time, perform the following actions:

  1. Determine the change in the value of the mongodb-bytes-read attribute for a span of time.

  2. Determine the change in the value of the records attribute for the same span of time you used for the preceding step.

  3. Divide the change in the value of the mongodb-bytes-read attribute by the change in the value of the records attribute.

latest-mongodb-time-difference-secs
The number of seconds of the most recent time difference recorded between a MongoDB server and the post-batch resume token held by a MongoDB source task. This value is calculated by subtracting the timestamp of the task's post-batch resume token from the operationTime value of the most recent successful MongoDB command executed by the task.
in-task-poll
The total number of times the Kafka Connect framework executed the poll() method of a MongoDB source task.
in-task-poll-duration-ms
The total number of milliseconds the Kafka Connect framework spent executing the poll() method of a MongoDB source task.
in-task-poll-duration-over-1-ms
The total number of MongoDB source task poll() method executions with a duration that exceeded 1 millisecond.
in-task-poll-duration-over-10-ms
The total number of MongoDB source task poll() method executions with a duration that exceeded 10 milliseconds.
in-task-poll-duration-over-100-ms
The total number of MongoDB source task poll() method executions with a duration that exceeded 100 milliseconds.
in-task-poll-duration-over-1000-ms
The total number of MongoDB source task poll() method executions with a duration that exceeded 1000 milliseconds.
in-task-poll-duration-over-10000-ms
The total number of MongoDB source task poll() method executions with a duration that exceeded 10000 milliseconds.
in-connect-framework
The total number of times code in the Kafka Connect framework executed after the first invocation of the poll() method of the MongoDB source task.
in-connect-framework-duration-ms
The total number of milliseconds spent executing code in the Kafka Connect framework since the framework first invoked the poll() method of the MongoDB source task. This metric does not count time executing code in the MongoDB sink task towards the total.
in-connect-framework-duration-over-1-ms
The total number of times code in the Kafka Connect framework executed for a duration that exceeded 1 millisecond.
in-connect-framework-duration-over-10-ms
The total number of times code in the Kafka Connect framework executed for a duration that exceeded 10 milliseconds.
in-connect-framework-duration-over-100-ms
The total number of times code in the Kafka Connect framework executed for a duration that exceeded 100 milliseconds.
in-connect-framework-duration-over-1000-ms
The total number of times code in the Kafka Connect framework executed for a duration that exceeded 1000 milliseconds.
in-connect-framework-duration-over-10000-ms
The total number of times code in the Kafka Connect framework executed for a duration that exceeded 10000 milliseconds.
initial-commands-successful
The total number of initial commands issued by a MongoDB source task that succeeded. An initial command is a find or aggregate command sent to a MongoDB server that retrieves the first set of documents in a cursor. A getMore command is not an initial command.
initial-commands-successful-duration-ms
The total number of milliseconds a MongoDB source task spent executing initial commands that succeeded.
initial-commands-successful-duration-over-1-ms
The total number of successful initial commands issued by a MongoDB source task with a duration that exceeded 1 millisecond.
initial-commands-successful-duration-over-10-ms
The total number of successful initial commands issued by a MongoDB source task with a duration that exceeded 10 milliseconds.
initial-commands-successful-duration-over-100-ms
The total number of successful initial commands issued by a MongoDB source task with a duration that exceeded 100 milliseconds.
initial-commands-successful-duration-over-1000-ms
The total number of successful initial commands issued by a MongoDB source task with a duration that exceeded 1000 milliseconds.
initial-commands-successful-duration-over-10000-ms
The total number of successful initial commands issued by a MongoDB source task with a duration that exceeded 10000 milliseconds.
getmore-commands-successful
The total number of getMore commands issued by a MongoDB source task that succeeded.
getmore-commands-successful-duration-ms
The total number of milliseconds a MongoDB source task spent executing getMore commands that succeeded.
getmore-commands-successful-duration-over-1-ms
The total number of successful getMore commands issued by a MongoDB source task with a duration that exceeded 1 millisecond.
getmore-commands-successful-duration-over-10-ms
The total number of successful getMore commands issued by a MongoDB source task with a duration that exceeded 10 milliseconds.
getmore-commands-successful-duration-over-100-ms
The total number of successful getMore commands issued by a MongoDB source task with a duration that exceeded 100 milliseconds.
getmore-commands-successful-duration-over-1000-ms
The total number of successful getMore commands issued by a MongoDB source task with a duration that exceeded 1000 milliseconds.
getmore-commands-successful-duration-over-10000-ms
The total number of successful getMore commands issued by a MongoDB source task with a duration that exceeded 10000 milliseconds.
initial-commands-failed
The total number of initial commands issued by a MongoDB source task that failed. An initial command is a find or aggregate command sent to a MongoDB server that retrieves the first set of documents in a cursor. A getMore command is not an initial command.
initial-commands-failed-duration-ms
The total number of milliseconds a MongoDB source task spent unsuccessfully attempting to issue initial commands to the MongoDB server.
initial-commands-failed-duration-over-1-ms
The total number of failed initial commands issued by a MongoDB source task with a duration that exceeded 1 millisecond.
initial-commands-failed-duration-over-10-ms
The total number of failed initial commands issued by a MongoDB source task with a duration that exceeded 10 milliseconds.
initial-commands-failed-duration-over-100-ms
The total number of failed initial commands issued by a MongoDB source task with a duration that exceeded 100 milliseconds.
initial-commands-failed-duration-over-1000-ms
The total number of failed initial commands issued by a MongoDB source task with a duration that exceeded 1000 milliseconds.
initial-commands-failed-duration-over-10000-ms
The total number of failed initial commands issued by a MongoDB source task with a duration that exceeded 10000 milliseconds.
getmore-commands-failed
The total number of getMore commands issued by a MongoDB source task that failed.
getmore-commands-failed-duration-ms
The total number of milliseconds a MongoDB source task spent unsuccessfully attempting to issue getMore commands to the MongoDB server.
getmore-commands-failed-duration-over-1-ms
The total number of failed getMore commands issued by a MongoDB source task with a duration that exceeded 1 millisecond.
getmore-commands-failed-duration-over-10-ms
The total number of failed getMore commands issued by a MongoDB source task with a duration that exceeded 10 milliseconds.
getmore-commands-failed-duration-over-100-ms
The total number of failed getMore commands issued by a MongoDB source task with a duration that exceeded 100 milliseconds.
getmore-commands-failed-duration-over-1000-ms
The total number of failed getMore commands issued by a MongoDB source task with a duration that exceeded 1000 milliseconds.
getmore-commands-failed-duration-over-10000-ms
The total number of failed getMore commands issued by a MongoDB source task with a duration that exceeded 10000 milliseconds.
←  MongoDB AWS-based AuthenticationMigrate from Kafka Connect MongoDB →
Give Feedback
© 2022 MongoDB, Inc.

About

  • Careers
  • Investor Relations
  • Legal Notices
  • Privacy Notices
  • Security Information
  • Trust Center
© 2022 MongoDB, Inc.