I’m currently preparing to migrate a subset of data from a single-node replica set on a self-managed server to a managed 3-node replica set cloud instance on IBM Cloud (both Mongo 4.0). My plan is to use Change Streams to capture events from the old DB, and propagate them onto the new one. I would like to start the Change Streams app and capture all events in the past 24 hours, as well as new events going forward.
I currently am able to successfully capture changes only when I launch my Change Stream Node app without the “startAtOperationTime” option. Meaning I can get current changes forward at the time of launch, but not the ones in the past I’m trying to grab.
When I query my replica set, I can see that my opLog goes back more than 10 days. And when I look at entries in the replica set, I can see timestamps from specific operations. For instance, here’s a test insert operation.
Since I can confirm the timestamp of that operation, I’ve been passing it exactly how it appears to my db.watch() function like so.
const pastTimestamp = new Timestamp(1626715191,0)
const changeStream = oldEcomm.watch({startAtOperationTime:pastTimestamp})
However I get the following error each time.
MongoError: Resume of change stream was not possible, as the resume point may no longer be in the oplog.
operationTime: Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1626718848 },
ok: 0,
code: 280,
codeName: 'ChangeStreamFatalError',
'$clusterTime': {
clusterTime: Timestamp { _bsontype: 'Timestamp', low_: 1, high_: 1626718848 },
signature: {
hash: Binary {
_bsontype: 'Binary',
sub_type: 0,
position: 20,
buffer: Buffer(20) [Uint8Array] [
205, 98, 57, 27, 102, 145,
95, 15, 149, 110, 130, 150,
109, 131, 5, 172, 9, 95,
126, 246
]
},
keyId: Long { _bsontype: 'Long', low_: 3, high_: 1625867199 }
}
}
}
What I can’t figure out is how the Change Streams app can’t “resume” the stream considering I just confirmed that the opLog is inclusive of that exact timestamp.
Any advice is really appreciated, thanks.