Change Stream On Restored Oplog

Ops and Admin Replication
1

How does the change stream return events on the restored oplog collections.
Using DB version 4.2 when the oplog.rs is queried there are events returns but on the same oplog the change stream does not give the insert events which are existing in the oplog.rs collection.
The dump restore with oplog replay is done to have copy of production data on staging environment. Why are only insert events missing in change stream when a restore is done.

3

Hi @Sajida_Begum welcome to the community!

Are you missing events from the change stream, or does the restored oplog itself missing events?

I tend to think that if the restored oplog is missing things, it’s either 1) events that happens between the backup and the restore would naturally be missing, and 2) maybe the backup was not properly made.

More information would probably needed here:

  1. What’s your exact MongoDB version?
  2. Can you show us the change stream definition
  3. Can you show us some example of missing events both in the changestream and in the oplog
  4. How did you create the backup, and how did you restore it

Best regards
Kevin

4

Hi @kevinadi, Thank you for your response.

The restored oplog does have the all the events. nothing is missed.

Before we start the backup. we do write lock on the replica set so that no new event is added at the time of backup and once the backup is complete the write lock is removed so that events get replicated correctly. so that when the next backup is taken from the timestamp it has all the events from the last time backup was done.

We use mongodump and mongorestore with oplogReplay option. we do this on daily basis to have the data replicated on staging environment.

To specify the oplog.rs does have the insert event after the restore is done. when the change stream is started with option startAtOpertionTime, and the MongoBSON Timestamp is in the range of oplog replication, except the insert event all other events are been notified. This happens on the data where the restore is done. But if the insert event is done real time the there is no issue.

The DB version used is : 4.2.22
Hope have shared enough information on the issue.

NOTE : Have tried to check this with upgraded version 4.4. There is no issue with any of the events listening on the restored data. The same oplogReplay process is done on 4.4 as well. So what could be the issue when the restore is done on version 4.2 .

Thanks in Advance,
Sajeeda

5

Hi @Sajida_Begum sorry for the delayed response.

If I understand correctly, the restored oplog has all the events, right? That means that the oplog was 100% restored with no missing data? However you’re missing some insert events during the backup event from the change stream’s point of view. Is this correct?

One thing I can think of is that the startAtOpertionTime parameter somehow didn’t include the events of interest. Perhaps you can tweak this value to an earlier timestamp and see if this is still an issue?

If this is still an issue for you, could you attach some reproduction script, or a reproduction method so we can replicate what you’re seeing exactly?

Best regards
Kevin

7

Hi @kevinadi,

Thank you for response.
Yes, your understanding is correct. I did check with changing the timestamp for startAtOperationTime but that did not solve the problem.

Anyway we got our database upgraded to 4.4 which has solved the problem.

Steps i did to reproduce the issue on 4.2 :

  1. On a vagrant machine created a DB say testdb1 with a collection which has around 10 documents where all were inserted first and few entries were updated. the oplog which was created had all insert and update events.
  2. Took dump of oplog and did a restore on a new DB day testdb2.
  3. Now start watching events on collection in new testdb2. this results in missing insert events.

Best Regards,
Sajeeda Begum