Mongorestore using OplogReplay

Hi Team,

I have dropped a collection from my TestDB (from both Stage and Prod Server), and I wanted to restore it back using OplogReplay by taking backup of oplog.rs collection. But it is throwing the below error.
Can you please help?

Note: I am using AKS cluster.

Syntax:
mongorestore --host=xxxxxxxx --port 27017 --oplogReplay --oplogLimit 1644392307 --authenticationDatabase admin -u xxxxxx -p xxxx /var/lib/mongodb/local/oplog.rs.bson

Stage Server Error:

Failed: restore error: error applying oplog: applyOps: (Unauthorized) not authorized on admin to execute command { applyOps: [ { ts: Timestamp(1641288700, 2), t: 1, h: null, v: 2, op: “c”, ns: “config.$cmd”, o: { create: “system.indexBuilds”, idIndex: { v: 2, key: { _id: 1 }, name: “id” } } } ], lsid: { id: UUID(“a3a053a6-ef03-44e9-ad18-69d44040f577”) }, $clusterTime: { clusterTime: Timestamp(1644407312, 1), signature: { hash: BinData(0, 5219E1B914EF211962681D46B336D3864642C59F), keyId: 7049281289794355202 } }, $db: “admin”, $readPreference: { mode: “primaryPreferred” } }
2022-02-09T11:48:32.786+0000 0 document(s) restored successfully. 0 document(s) failed to restore.

Production Server Error:

Now we are getting error in Stage server, I am not sure why Duplicate Key error is coming for a table which is dropped.

2022-02-09T12:22:03.295+0000 skipping applying the config.system.sessions namespace in applyOps
2022-02-09T12:22:03.296+0000 skipping applying the config.system.sessions namespace in applyOps
2022-02-09T12:22:03.296+0000 skipping applying the config.transactions namespace in applyOps
2022-02-09T12:22:03.398+0000 oplog 694MB
2022-02-09T12:22:03.403+0000 Failed: restore error: error handling transaction oplog entry: error applying transaction op: applyOps: (DuplicateKey) E11000 duplicate key error collection: DataUniverseStg.Hierarchy index: HierachyIDandDUPKI dup key: { HierarchyID: 1343, DU_PKI: 15 }
2022-02-09T12:22:03.403+0000 0 document(s) restored successfully. 0 document(s) failed to restore.

Hi Team,

Any suggestion?

Thanks,
Rana

Check this jira ticket.It suggests to use additional parameter
https://jira.mongodb.org/browse/TOOLS-2041

Similar error I am also getting. during restore - with latest version of mongodb tools 100.6.1
mongorestore --authenticationDatabase admin --port 27017 -u admin --oplogReplay /tmp/new/logs_0/20230208015236_20230208015524/local/oplog.rs.bson --oplogLimit 1675821494:0 -p
2023-02-08T07:29:32.814+0000 checking for collection data in /tmp/new/logs_0/20230208015236_20230208015524/local/oplog.rs.bson
2023-02-08T07:29:32.814+0000 replaying oplog
2023-02-08T07:29:32.815+0000 Failed: restore error: error applying oplog: applyOps: (Unauthorized) not authorized on admin to execute command { applyOps: [ { ts: Timestamp(1675821194, 1), t: 1, h: null, v: 2, op: “c”, ns: “config.$cmd”, o: { create: “system.sessions”, idIndex: { v: 2, key: { _id: 1 }, name: “id” } } } ], lsid: { id: UUID(“1da48df1-7268-4bf2-a1bb-6455005d68f3”) }, $clusterTime: { clusterTime: Timestamp(1675841367, 2), signature: { hash: BinData(0, 23BDCA774453874BA680500BE149D0764CC5811C), keyId: 7197629778025775108 } }, $db: “admin”, $readPreference: { mode: “primaryPreferred” } }
2023-02-08T07:29:32.815+0000 0 document(s) restored successfully. 0 document(s) failed to restore.

Appears to be privileges issue
What role your user is having?

1 Like

My user is root role user -855WB43P44:PRIMARY> show users
{
“_id” : “admin.admin”,
“userId” : UUID(“cc00c914-0020-47f4-8ca4-d1e0fe11f227”),
“user” : “admin”,
“db” : “admin”,
“roles” : [
{
“role” : “root”,
“db” : “admin”
}
],

Root does not have privs to run this command
Applyops is an internal command
You have to create customrole
Check these links

1 Like

Thanks for sharing let me try it out, however to me ‘root’ role is super user roles right, we should not need any additional role.
for me this looks to be bug in extension to similar issue raised earlier - https://jira.mongodb.org/browse/TOOLS-2952

It kind of is, if you’re dealing with databases and collections. However, applyOps is an internal MongoDB command, and thus require a special system-level privilege.

From https://www.mongodb.com/docs/database-tools/mongorestore/#required-access

To run with --oplogReplay, create a user-defined role that has anyAction on anyResource.

Grant only to users who must run mongorestore with --oplogReplay.

Hope this helps

Best regards
Kevin

3 Likes

As per document - super user ‘root’ roles has combined privileges over below which includes restore as well -

root

Provides access to the operations and all the resources of the following roles combined:

Please refer to the link shared by Kevin.Restore has access to all non system related objects but when you are accessing system related objects/running internal commands additional privileges need to be given
That’s why you need to create a custom role with access any object and grant it to the user who is performing the restore

2 Likes

I agree restore role alone might have to give additional privileges, however this is about super user -root role whom should be able to do every thing without any restriction, did you check this link -https://www.mongodb.com/docs/manual/reference/built-in-roles/#mongodb-authrole-root

Did you mention here that with root role also you need extra privilege.

The role called root in MongoDB terms is not the same as root in UNIX terms. In MongoDB, it is not “superuser”.

See https://www.mongodb.com/docs/manual/reference/built-in-roles/

root

Provides access to the operations and all the resources of the following roles combined:

It’s basically allows you to do data operations across all databases and all collections, but not system objects and system operations. Thus it’s not a superuser in the traditional UNIX convention.

There is another role that’s basically superuser, but no user should be given this role according to the documentation:

__system

MongoDB assigns this role to user objects that represent cluster members, such as replica set members and mongos instances. The role entitles its holder to take any action against any object in the database.

Do not assign this role to user objects representing applications or human administrators, other than in exceptional circumstances.

If you need access to all actions on all resources, for example to run applyOps commands, do not assign this role. Instead, create a user-defined role that grants anyAction on anyResource and ensure that only the users who need access to these operations have this access.

I’d like to go back to the earlier error that is being discussed:

Failed: restore error: error applying oplog: applyOps: (Unauthorized) not authorized on admin to execute command

The error is saying that you need applyOps privilege to execute an oplog apply operation. This was answered earlier:

Best regards
Kevin

5 Likes

Thanks for detailed clarification , so we can be assured that with ‘anyAction’ on ‘anyResource’, we wont hit any further issues like below -

2023-02-09T17:56:49.621+0000 Failed: restore error: error applying oplog: applyOps: (Unauthorized) not authorized on admin to execute command { applyOps: [ { ts: Timestamp(1675883519, 1), t: 11, h: null, v: 2, op: “c”, ns: “admin.$cmd”, o: { create: “system.roles”, idIndex: { v: 2, key:

And we can be assured that with these roles - we wont hit issue which were reported in this bug# and there are few open items still still as followup to this bug
https://jira.mongodb.org/browse/TOOLS-3203

Or do you suggest to use - __system role to resolve every issues universally.
Thanks again for helping on this.

This is a totally different issue, as far as I can tell. Firstly, it’s a mongodump issue, not mongorestore, and secondly it concerns the config database of a sharded cluster.

The applyOps permission issue you’re seeing is about executing a system level command that manipulates the oplog: a dangerous and potentially irreversible destructive operation if done by accident, hence the need for a special permission.

No, I would follow the documentation’s recommendation to not use this role and instead create a new user-defined role (anyAction on anyResource). Similar to UNIX, running everything as root and doing chmod 777 * when you’re seeing permission issues is usually not the right answer :slight_smile:

Hope this helps

Best regards
Kevin

2 Likes

Yes, I agree with input.
I feel this jira ticket https://jira.mongodb.org/browse/TOOLS-2952 and ‘anyAction’ on ‘anyResource’ privileges are tightly coupled.
In jira we are skipping system collections like session/cache/transaction etc at both mongodump and restore both, as per code changes .
And at the same time we are also asking users to give additional privileges if there are some system collections which are not skipped in above jira.

Any ways - We will go with document you have shared where additional privileges are mentioned to be used for restore.
Thanks again for all your inputs , we can close this thread - Also if you agree that skipping jira and additional permission are relevant then further followup can be done with Dev team.

I have admin user with below privilege : anyAction on anyResource and still restore is failing with below error -

replaying oplog
2023-06-22T08:15:12.152-0700 Failed: restore error: error applying oplog: applyOps: (Location40528) Direct writes against config.transactions cannot be performed using a transaction or on a session.

I am on mongdb 4.4.22 version.
This was never the case earlier while using mongo shell, recently we have started using mongosh.