I am using MongoDB change streams for real-time synchronization to another database. To ensure continuity after a pause, I utilize the resumeToken
to resume the stream from where it was paused. However, I am encountering significant issues during the resumption process:
- Cluster CPU and Memory Spikes: Upon resuming the change stream, there is a large spike in the CPU and memory usage of my MongoDB cluster.
- Full Collection Oplog Scan: MongoDB appears to scan the entire oplog during the resume operation, even when the
resumeToken
is used. This results in significant performance degradation.
Attached is a screenshot from the MongoDB Atlas Profiler showing that 32.3 million documents were scanned in a single operation with only 2 documents returned, indicating an inefficient operation.
Attached is the profiler screenshot showing the inefficiency in the operation:
- Docs Examined: 32.3M
- Docs Returned: 2
- Execution Time: 3.09 min
Questions:
- Why does MongoDB scan the entire oplog collection when resuming the stream using a valid
resumeToken
? - Can change streams be configured to resume in a chunk-wise or incremental manner to avoid full oplog scans?