Why sharding cluster did not permit db.fsyncUnlock() command

I created new sharded cluster v7.0.12 with one shard and config replica.

I try to implement file system snapshot backup but get stuck with lock unlock command at mongos context.

fsyncLock() work fine as expected.

[direct: mongos] admin> db.getSiblingDB("admin").fsyncLock()
{
  numFiles: 1,
  all: {
    raw: {
      'rs1/backup-poc-rs1-0-fsn1:27017,backup-poc-rs1-1-nbg1:27017,backup-poc-rs1-2-hel1:27017': {
        info: 'now locked against writes, use db.fsyncUnlock() to unlock',
        lockCount: Long('1'),
        seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',
        ok: 1
      },
      'cfg/backup-poc-cfg-0-fsn1:27017,backup-poc-cfg-1-nbg1:27017,backup-poc-cfg-2-hel1:27017': {
        info: 'now locked against writes, use db.fsyncUnlock() to unlock',
        lockCount: Long('1'),
        seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',
        ok: 1
      }
    }
  },
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1723557836, i: 1 }),
    signature: {
      hash: Binary.createFromBase64('DIr2nxYdYZypvYVSU6P4ULm+4dA=', 0),
      keyId: Long('7402590685452304406')
    }
  },
  operationTime: Timestamp({ t: 1723557836, i: 1 })
}

But db.getSibling(“admin”).fsyncUnlock() did not permited.

[direct: mongos] admin> db.fsyncUnlock()
MongoServerError[Unauthorized]: not authorized on admin to execute command { fsyncUnlock: 1, lsid: { id: UUID("062761fa-34ae-4723-a93b-1bb9a97cbcca") }, $clusterTime: { clusterTime: Timestamp(1723557836, 1), signature: { hash: BinData(0, 0C8AF69F161D619CA9BD855253A3F850B9BEE1D0), keyId: 7402590685452304406 } }, $db: "admin" }

User that run both command have next roles
- “clusterAdmin”
- “dbAdminAnyDatabase”
- “root”
- “hostManager”
that should be enough according manual https://www.mongodb.com/docs/manual/reference/built-in-roles/#mongodb-authrole-hostManager

How can be solve this puzzle?

Best regards
Roman Sereda

Hi @Roman_Sereda1 and welcome to the community!
curious, are you sure you have authenticated yourself?
With root privileges, you should be able to safely perform the operation.

Best Regards

Yes. I check this first and set of real permission as second.

db.fsyncLock() command allowed in same user session

Hi @Roman_Sereda1,
I just did a test and personally it works correctly for me:

[direct: mongos] test> db.getSiblingDB("admin").fsyncLock()
{
 numFiles: 1,
 all: {
   raw: {
     'myShard_0/replicadue.mongodb.int:27000,replicatre.mongodb.int:27000,replicauno.mongodb.int:27000': {
       info: 'now locked against writes, use db.fsyncUnlock() to unlock',
       lockCount: Long('1'),
       seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',
       ok: 1
     },
     'myShard_1/replicadue.mongodb.int:27001,replicatre.mongodb.int:27001,replicauno.mongodb.int:27001': {
       info: 'now locked against writes, use db.fsyncUnlock() to unlock',
       lockCount: Long('1'),
       seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',
       ok: 1
     },
     'configRS/replicadue.mongodb.int:27020,replicatre.mongodb.int:27020,replicauno.mongodb.int:27020': {
       info: 'now locked against writes, use db.fsyncUnlock() to unlock',
       lockCount: Long('1'),
       seeAlso: 'http://dochub.mongodb.org/core/fsynccommand',
       ok: 1
     }
   }
 },
 ok: 1,
 '$clusterTime': {
   clusterTime: Timestamp({ t: 1722613341, i: 7 }),
   signature: {
     hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
     keyId: Long('0')
   }
 },
 operationTime: Timestamp({ t: 1722613341, i: 7 })
}
[direct: mongos] test> db.getSiblingDB("admin").fsyncUnlock()
{
 raw: {
   'configRS/replicadue.mongodb.int:27020,replicatre.mongodb.int:27020,replicauno.mongodb.int:27020': { info: 'fsyncUnlock completed', lockCount: Long('0'), ok: 1 },
   'myShard_1/replicadue.mongodb.int:27001,replicatre.mongodb.int:27001,replicauno.mongodb.int:27001': { info: 'fsyncUnlock completed', lockCount: Long('0'), ok: 1 },
   'myShard_0/replicadue.mongodb.int:27000,replicatre.mongodb.int:27000,replicauno.mongodb.int:27000': { info: 'fsyncUnlock completed', lockCount: Long('0'), ok: 1 }
 },
 ok: 1,
 '$clusterTime': {
   clusterTime: Timestamp({ t: 1722613341, i: 16 }),
   signature: {
     hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0),
     keyId: Long('0')
   }
 },
 operationTime: Timestamp({ t: 1722613341, i: 16 })
}

Same problem here. Do you have find any solution ?

I did not catch root of problem. I avoid this operation and use network restriction to manage income session to prevent write operation at maintenance event time. Problem is reproducible at my environment.

Hello everyone, same problem here.
Running MongoDB 6.0.21 from TAR archive (on Debian 11 in case it matters),
fsyncLock() worked correctly via mongos, but I cannot unlock the db:

superadmin@admin || REMOTE => db.fsyncUnlock()
MongoServerError[Unauthorized]: not authorized on admin to execute command { fsyncUnlock: 1, lsid: { id: UUID("1e47fd83-9d0f-4085-a067-3ef9124df008") }, $clusterTime: { clusterTime: Timestamp(1746603456, 1), signature: { hash: BinData(0, 80792718E59049D81365EAC9C57D8A3F1AF64A66), keyId: 7501307811511402521 } }, $db: "admin" }

The user I’m using to issue the command has the current roles:

db.runCommand({ connectionStatus: 1 })
{
  authInfo: {
    authenticatedUsers: [ { user: 'superadmin', db: 'admin' } ],
    authenticatedUserRoles: [
      { role: 'clusterAdmin', db: 'admin' },
      { role: 'dbAdminAnyDatabase', db: 'admin' },
      { role: 'hostManager', db: 'admin' },
      { role: 'readWriteAnyDatabase', db: 'admin' },
      { role: 'root', db: 'admin' },
      { role: 'userAdminAnyDatabase', db: 'admin' }
    ]
  },
  ok: 1,
  .....
}

I’ve successfully unlocked the database by running db.fsyncUnlock() on the primary nodes of the 2 shards,
but according to MongoDB documentation db.fsyncUnlock() - Database Manual v8.0 - MongoDB Docs
it should be possible to lock and unlock connecting via mongos:
Starting in MongoDB 7.1 (also available starting in 7.0.2, 6.0.11, and 5.0.22) the db.fsyncLock() and db.fsyncUnlock() methods can run on mongos to lock and unlock a sharded cluster.

Is this a bug?

It would be nice to have the db.fsyncUnlock() properly working via mongos to ease the backup process.
Thank you for any hint,
_
Fabio

Hello Fabio, sorry to resurrect this thread,
which roles the user you used had? At the moment I have:

authenticatedUserRoles: [
      { role: 'clusterAdmin', db: 'admin' },
      { role: 'dbAdminAnyDatabase', db: 'admin' },
      { role: 'dbOwner', db: 'admin' },
      { role: 'hostManager', db: 'admin' },
      { role: 'readWriteAnyDatabase', db: 'admin' },
      { role: 'root', db: 'admin' },
      { role: 'userAdminAnyDatabase', db: 'admin' }

but I still get:

db.getSiblingDB("admin").fsyncUnlock()

MongoServerError[Unauthorized]: not authorized on admin to execute command { fsyncUnlock: 1, lsid: { id: UUID("f914c7ea-de33-4f81-ac62-763d02b979d3") }, $clusterTime: { clusterTime: Timestamp(1746605884, 1), signature: { hash: BinData(0, ADD7CB273715DF53828EC7D8D04CEE1E24F9B3C9), keyId: 7501307811511402521 } }, $db: "admin" }

Hi @Fabio_Venturi,
When I did the test, I used a user with root role , so it’s quite strange what happens to you.

Best Regards

1 Like

Hi @Fabio_Ramohitaj thank you very much for your reply.
It’s really weird indeed, I’ve the same problem with both mongodb 6.0.21 and 7.0.20 (haven’t tried 8 though).

Furthermore if I run this via pymongo:

client = MongoClient(MONGO_URI)
admin = client.admin
result = admin.command("fsync", lock=False, maxTimeMS=10000)

I got ok: 1.0

{'numFiles': 1, 'all': {'raw': {'shard0/mongodb-restore-shard01.nonprod:27018,mongodb-restore-shard02.nonprod:27018,mongodb-restore-shard03.nonprod:27018': {'numFiles': 1,
 'ok': 1.0}, 'config/mongodb-restore-config1.nonprod:27019,mongodb-restore-config2.nonprod:27019,mongodb-restore-config3.nonprod:27019': {'numFiles': 1, 'ok': 1.0}, 'shard
1/mongodb-restore-shard11.nonprod:27018,mongodb-restore-shard12.nonprod:27018,mongodb-restore-shard13.nonprod:27018': {'numFiles': 1, 'ok': 1.0}}}, 'ok': 1.0, '$clusterTim
e': {'clusterTime': Timestamp(1746702209, 2), 'signature': {'hash': b'n\xf4YF\x82\xdfJ\xd5]\x1a\xc0N\x1e\x99\x8f\xf9\x1a\x1b\xcaw', 'keyId': 7501702647854923798}}, 'operationTime': Timestamp(1746
702209, 2)}

but if I run the aggregation pipeline showed in the documentation

db.getSiblingDB("admin").aggregate( [
   {  $currentOp: { } },
   {  $facet: {
      "locked": [
         { $match: { $and: [
            { fsyncLock: { $exists: true } }
         ] } }],
       "unlocked": [
         { $match: { fsyncLock: { $exists: false } } }
       ]
   } },
   {  $project: {
      "fsyncLocked": { $gt: [ { $size: "$locked" }, 0 ] },
      "fsyncUnlocked": { $gt: [ { $size: "$unlocked" }, 0 ] }
   } }
 ] )

I still get that there is a lock :man_shrugging:

[ { fsyncLocked: true, fsyncUnlocked: false } ]

I start suspecting I did something wrong while creating the admin during cluster bootstrap, but i don’t know what. For now I’ll just connect to the replica set and lift the lock directly on the primary nodes.
Thank you again
_
Fabio