Question:
I’m working with the Kafka MongoDB connector to stream data from Kafka into a MongoDB time-series collection. My setup involves processing data from an API and sinking it into MongoDB using the Kafka connector. When attempting to sink new messages, I’m encountering an error I can’t resolve:
Cannot perform a non-multi update on a time-series collection
Setup:
- My messages are in the following JSON format:
{
"metadata": {...},
"timestamp": 1689949371,
"readings": {...}
}
- I’ve ensured I’m only inserting new records and not trying to update existing ones.
- The connector configurations are set up to handle this appropriately (previously shared in detail).
What I’ve tried:
- Confirmed I’m only trying to insert new records.
- Reviewed MongoDB community responses - most refer to issues updating time-series collections, but I’m inserting new records.
- Checked connector configurations for potential implicit update behaviors.
Issue:
Despite the above, when I try to sink messages, I’m getting the aforementioned error. The connector seems to think it’s an update operation, even though I’m only sinking new messages.
Question:
Why might the Kafka MongoDB connector perceive these insertions as updates? Has anyone else experienced this, especially with time-series collections in MongoDB, and how did you resolve it?
Any insights or pointers would be greatly appreciated.