Mongorestore: opLogReplay E11000 duplicate key error

Working with Data Developer Tools
, , , ,
#1

Hello Team,

Mongo Server: v6.0.4
Operating System: Amazon Linux2
Mode: 3 Node Replicaset

Background:
We create nightly snapshots of EBS Volume of /data mount location of our primary replicaset.

Our goal is to be able to perform a point in time restore of the Mongo database from T-1 FileSysytem Snapshot copy. The database receives multiple updates/deletions/inserts for the records.

Steps we have done so far:

  1. Create new EC2 instance with mongo configuration and attached the Volume created from T-1 day EBS snapshot and mounted on /data
  2. Started the Mongo server as standalone on new EC2 instance.
  3. Created oplog dump from existing running replicaset cluster using below command
mongodump -u admin --authenticationDatabase=admin -h <exisiting_running_host_ip> -d local -c oplog.rs -o oplogDumpDir/
  1. Found the oplog entry till which we want to replay the oplog by executing the below command in existing running replcaset cluster.
use local;
db.oplog.rs.find({op:"i"}).sort({$natural: -1}).limit(10);

  1. Copied the opLog timestamp t and i values from the existing replicaset
    example: ts: Timestamp({ t: 1689689931, i: 7 }),

  2. Now; we want to replay the transactions in new EC2 instance until the above timestamp.

mongorestore -v -u root --noIndexRestore --oplogReplay --drop --oplogLimit 1689689931:7 oplogRecoveryDir/

We are receiving the below error

Failed: restore error: error applying oplog: applyOps: (DuplicateKey) E11000 duplicate key error collection: test.test2 index: test-id_1_local-date_1_type_1 dup key: { test-id: ObjectId('6454a05b0389d90393268baf'), local-date: 20220915, type: "asdada" }

I have given a read to MongoDB community forum posts and JIRA tickets regarding the same; but unable to find any solution.

I have tried the mongorestore with different parameters as well (–noIndexRestore --maintainInsertionOrder --keepIndexVersion ) but; still no solution.

We would appreciate for help and direction to move forward.

#3

I was able to resolve the issue and perform the PITR of mongo 3 node replicaset cluster.

Adding the steps in-case someone else stumbles across this post.

After restoring the T-1 EBS Snapshot of /data mount point in new EC2 instance.
…

  1. Get the latest oplog entry in the new EC2 instance launched from T-1 Snapshot
>use local
>db.oplog.rs.find({op:"i"}).sort({$natural: -1}).limit(1);

# Sample Output for ts value
ts: Timestamp({ t: 1689748934, i: 3 }),
  1. Take oplog dump from existing Running Mongo cluster.
    Our oplLog contains entries of last 57 hours; so I do not know why we were having Duplicate key errors despite the fact the oplog should be idempotent. And replaying the oplog again and again should be fine.
mongodump -u username --authenticationDatabase=admin -h secondary-node-ip-existing-cluster -d local -c oplog.rs --query '{"ts": {"$gt": {"$timestamp": {"t": 1689748934, "i": 3}}}}' -o oplogDumpDir/
  1. Perform renaming of oplog file
mv oplogDumpDir/local/oplog.rs.bson oplog.bson
rm -rf oplogDumpDir/local
  1. Perform Replay of opLog on new EC2 instance until the bad transaction
mongorestore  --authenticationDatabase=admin -u root -h new-ec2-instance-ip  --oplogReplay --oplogLimit 1689837570:1 oplogDumpDir/
  1. Wait for the completion of the process.

We verified the restoration using one document that gets updated very frequently.

Two things that we observed:
a. The transactions were replayed but the oplog entry in the recovered instance was not updated. I am not sure if this is Mongo Server behaviour or not.

b. It took nearly 12 hours just to replay the 14GB oplog on t3.large EC2 instance with /data EBS Volume of gp2 type with 1200 IOPS with no active user connections. This is too long for production environment.

While searching for solutions; we stumbled across

pbm

tool as well. It looks promising, but we did not research much into that implementation.

#4