Mongorestore - error applying oplog

Hi, we are having to migrate a large database that is in use. So we have a new database stack and I take an initial DB dump of the data (~500GB) using mongodump

mongodump --host=localhost --port=27017 --oplog --gzip --username=admin --authenticationDatabase=admin --out=/data/db/backup/mongodb-dump-date \+\%Y\%m\%d_\%H\%M\%S

At the same time, I output the timestamp when that dump finishes so I can use that in the incremental backup later on.

use local{$natural:-1}).limit(1).next().ts

I then scp that dump over to the new host and I can restore it without isseu (takes 2.5hrs)

mongorestore --oplogReplay --port=27017 --username=admin --authenticationDatabase=admin --gzip /data/backup/mongodb-dump-20200917_201959

Then, since the old DB is still in use, the plan is to take an incremental backup from the point of the last full dump (using that timestamp above). So if I wait a few hours and run that it looks like this

mongodump --host=localhost --port=27017 -d local -c --username=admin --authenticationDatabase=admin --gzip --out=/data/db/backup/mongodb-dump-increment-date \+\%Y\%m\%d_\%H\%M\%S --query ‘{ “ts” : { $gt : Timestamp(1600376683, 359) } }’ 2>&1 | tee -a /data/db/backup/incremental.log

So when I then scp this incremental backup over and load it, I get the mentioned duplicate key errors.

Restoring the incremental dump:

mongorestore --oplogReplay --port=27017 --username=admin --authenticationDatabase=admin --gzip /data/db/backup/mongodb-dump-increment-20200922_163727

And then I get the errors:

Failed: restore error: error applying oplog: applyOps: E11000 duplicate key error collection: countly.app_crashusers5d72120aab9bc2004a137120 index: group_1_uid_1 dup key: { : null, : null }

I am using --oplogReplay. Is there another option to run on mongorestore to avoid these errors?


Is there any update on it? Facing same issue


I am also facing a similar issue. Can anyone share some knowledge on this topic. Will be very useful.