Hi, we are having to migrate a large database that is in use. So we have a new database stack and I take an initial DB dump of the data (~500GB) using mongodump
mongodump --host=localhost --port=27017 --oplog --gzip --username=admin --authenticationDatabase=admin --out=/data/db/backup/mongodb-dump-
date \+\%Y\%m\%d_\%H\%M\%S
At the same time, I output the timestamp when that dump finishes so I can use that in the incremental backup later on.
use local
db.oplog.rs.find().sort({$natural:-1}).limit(1).next().ts
I then scp that dump over to the new host and I can restore it without isseu (takes 2.5hrs)
mongorestore --oplogReplay --port=27017 --username=admin --authenticationDatabase=admin --gzip /data/backup/mongodb-dump-20200917_201959
Then, since the old DB is still in use, the plan is to take an incremental backup from the point of the last full dump (using that timestamp above). So if I wait a few hours and run that it looks like this
mongodump --host=localhost --port=27017 -d local -c oplog.rs --username=admin --authenticationDatabase=admin --gzip --out=/data/db/backup/mongodb-dump-increment-
date \+\%Y\%m\%d_\%H\%M\%S--query ‘{ “ts” : { $gt : Timestamp(1600376683, 359) } }’ 2>&1 | tee -a /data/db/backup/incremental.log
So when I then scp this incremental backup over and load it, I get the mentioned duplicate key errors.
Restoring the incremental dump:
mongorestore --oplogReplay --port=27017 --username=admin --authenticationDatabase=admin --gzip /data/db/backup/mongodb-dump-increment-20200922_163727
And then I get the errors:
Failed: restore error: error applying oplog: applyOps: E11000 duplicate key error collection: countly.app_crashusers5d72120aab9bc2004a137120 index: group_1_uid_1 dup key: { : null, : null }
I am using --oplogReplay. Is there another option to run on mongorestore to avoid these errors?
thanks