ChangeStreamHistoryLost - how to prevent

Working with Data Connectors & Integrations
1

I have a heavily used mongo db where I’m using the kafka connector to push this data to Redpanda. Far too often I get exceptions like this, which Mongo has documented how to resolve/work around but the docs mention that its usually is the result of either a paused connector or an infrequently updated namespace. The database is busy enough to not belelive it’s the latter, but also we arent pausing these connectors. So what are other causes of this?

{"ts":"2023-12-22 15:05:02,739","level":"INFO","thread":"task-thread-production-email-mongos-0","ctx":"[production-email-mongos|task-0] ","class":"com.mongodb.kafka.connect.source.StartedMongoSourceTask:605","msg":"An exception occurred when trying to get the next item from the Change Stream"}
com.mongodb.MongoQueryException: Query failed with error code 286 with name 'ChangeStreamHistoryLost' and error message 'Error on remote shard mongo-email-shard-1-b.us-central1-c.c.indigo-lotus-415.internal:27018 :: caused by :: Executor error during getMore :: caused by :: Resume of change stream was not possible, as the resume point may no longer be in the oplog.' on server mongos-email-0.us-central1-c.c.indigo-lotus-415.internal:27018

Since I’m not doing the things this note says typically cause this, what else could be causing it?

2

Is the OpLog size on the shards adequate for the load?

Check the OpLog Window at the time(s) of the observed error. And resize if this is too small, 24-72hrs of oplog is generally a good size but can vary widely due to operation requirement and constraints.

https://www.mongodb.com/docs/manual/tutorial/change-oplog-size/

3

I’m sure I don’t understand all the inner workings, but would that seem to imply that we’re not able to consume the changes in the oplog fast enough? So we’re falling further and further behind until the oplog rolls and then throws this error? Would love to understand this better if anyone is able to explain.

4

We also sometimes get this on a brand new connector that’s set to startup.mode: latest which I’m really having a hard time wrapping my mind around.