Explain steps moveChunk operation in config db changelog

Ops and Admin Sharding
1

Hi,
I’m investigating the performance of the slow balancer in my cluster and some steps are clearly taking longer than others, however, I can’t find anywhere an explanation of what each step of moveChunk.from and moveChunk.to means, could somebody explain or provide documentation?

What I do is on the config DB use the following query db.changelog.find() and such example responses are returned:

  • moveChunk.from
{
    _id: 'myserver:27017-2022-12-20T05:28:58.272-08:00-63a1b89a2cccc07e737b297',
    server: 'myserver:27017',
    shard: 'shardSrc',
    clientAddr: '',
    time: ISODate("2022-12-20T12:28:58.272Z"),
    what: 'moveChunk.from',
    ns: 'mydb.mycol',
    details: {
      'step 1 of 6': 0,
      'step 2 of 6': 7,
      'step 3 of 6': 115,
      'step 4 of 6': 2182,
      'step 5 of 6': 14,
      'step 6 of 6': 8625,
      min: { _id: Long("-8078899907612850299") },
      max: { _id: Long("-8078890509510611981") },
      to: 'shardDst',
      from: 'shardSrc',
      note: 'success'
    }
  }
  • moveChunk.to
  {
    _id: 'myserver:27017-2022-12-20T05:28:56.578-08:00-63a1b898asdaae95291a541a',
    server: 'myserver:27017',
    shard: 'shardDst',
    clientAddr: '',
    time: ISODate("2022-12-20T12:28:56.578Z"),
    what: 'moveChunk.to',
    ns: 'mydb.mycol',
    details: {
      'step 1 of 8': 2,
      'step 2 of 8': 1455,
      'step 3 of 8': 4,
      'step 4 of 8': 719,
      'step 5 of 8': 0,
      'step 6 of 8': 11,
      'step 7 of 8': 0,
      'step 8 of 8': 6935,
      min: { _id: Long("-8078899907612850299") },
      max: { _id: Long("-8078890509510611981") },
      to: 'shardDst',
      from: 'shardSrc',
      note: 'success'
    }
  }

Thanks for your time

2

@Bartosz_Dabrowski1 All moveChunk steps mentioned at mongo/README.md at master · mongodb/mongo · GitHub
Seems cloning & commit steps running for a longer duration. What is the version of your mongo? Do you see any latencies, or replication lag in source, destination & config servers?

1 Like
3

@K.V Although I am not the original poster, I am running into a similar issue on MongoDB 6.0, where RemoveShard takes 30 minutes, even with no databases written to the cluster. (3 shards, one config server) somehow, chunk migration is taking a long time.

(I have played around with removing and adding shards, so I apologise if the field to and from are confusing.)
Specifically here are my values for moveChunk.to:

    _id: ‘xxx-bc66fe-2:27017-2023-10-24T13:46:57.676+00:00-6537cad16a36f38b3c1d01fa',
    server: ‘xxx-bc66fe-2:27017',
    shard: 'shard-two',
    clientAddr: '',
    time: ISODate("2023-10-24T13:46:57.676Z"),
    what: 'moveChunk.to',
    ns: 'config.system.sessions',
    details: {
      'step 1 of 8': 1,
      'step 2 of 8': 0,
      'step 3 of 8': 1,
      'step 4 of 8': 1,
      'step 5 of 8': 0,
      'step 6 of 8': 11,
      'step 7 of 8': 0,
      'step 8 of 8': 15,
      min: { _id: { id: new UUID("17000000-0000-0000-0000-000000000000") } },
      max: { _id: { id: new UUID("17400000-0000-0000-0000-000000000000") } },
      to: 'shard-two',
      from: 'shard-one',
      note: 'success'
    }
  }

and moveChunk.from:

  {
    _id: ‘xxx-bc66fe-2:27017-2023-10-24T14:06:33.851+00:00-6537cf696a36f38b3c1d8453',
    server: ‘xxx-bc66fe-2:27017',
    shard: 'shard-two',
    clientAddr: '',
    time: ISODate("2023-10-24T14:06:33.851Z"),
    what: 'moveChunk.from',
    ns: 'config.system.sessions',
    details: {
      'step 1 of 6': 0,
      'step 2 of 6': 4,
      'step 3 of 6': 10,
      'step 4 of 6': 7,
      'step 5 of 6': 15,
      'step 6 of 6': 26,
      min: { _id: { id: new UUID("56000000-0000-0000-0000-000000000000") } },
      max: { _id: { id: new UUID("56400000-0000-0000-0000-000000000000") } },
      to: 'shard-three',
      from: 'shard-two',
      note: 'success'
    }
  }

looking at top on each shard (and config server), I see low consumption from mongod and mongos processes.

Can you help me identify which steps these are? And what issues they might correspond with? I had a hard time identifying which step corresponds to which action via the link you sent.

4

More info _waitForDelete is unset in my charm, so I find it odd that the cleanup phase might be blocking here. When I did some digging in the docs I got a little more confused.

From docs: "By default, the balancer does not wait for the on-going migration’s delete phase to complete before starting the next chunk migration. "

But from the docs “MongoDB can perform parallel data migrations, but a shard can participate in at most one migration at a time.”

So when a single shard is draining, I dont understand how " the balancer does not wait for the on-going migration’s delete phase to complete" if "a shard can participate in at most one migration at a time.”

5

I am now getting closer to understanding the issue, as I have now been able to consistently re-create the issue. The issue occurs when I remove the shard that hosts the chunks for the config database.

From sh.status():

    database: { _id: 'config', primary: 'config', partitioned: true },
    collections: {
      'config.system.sessions': {
        shardKey: { _id: 1 },
        unique: false,
        balancing: true,
        chunkMetadata: [ { shard: 'shard-two', nChunks: 1024 } ],
        chunks: [
          'too many chunks to print, use verbose if you want to force print'
        ],
        tags: []
      }

From logs of mongod on the shard that is getting removed:

2023-10-25T11:49:21Z charmed-mongodb.mongod[4753]: {"t":{"$date":"2023-10-25T11:49:21.835+00:00"},"s":"I",  "c":"COMMAND",  "id":51803,   "ctx":"conn179","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"_shardsvrMoveRange":"config.system.sessions","toShard":"shard-one","min":{"_id":{"id":{"$uuid":"83400000-0000-0000-0000-000000000000"}}},"waitForDelete":false,"epoch":{"$oid":"6538e3c125ae3b07d28d5eaf"},"fromShard":"shard-three","maxChunkSizeBytes":200000,"forceJumbo":2,"secondaryThrottle":false,"writeConcern":{"w":1,"wtimeout":0},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1698234561,"i":1}},"signature":{"hash":{"$binary":{"base64":"BDUe5gGly3g4iPDSiuCwtTRJTLU=","subType":"0"}},"keyId":7293828730399490051}},"$configTime":{"$timestamp":{"t":1698234561,"i":1}},"$topologyTime":{"$timestamp":{"t":1698233824,"i":2}},"mayBypassWriteBlocking":false,"$db":"admin"},"numYields":0,"reslen":236,"locks":{},"writeConcern":{"w":1,"wtimeout":0,"provenance":"clientSupplied"},"remote":"10.18.246.22:60792","protocol":"op_msg","durationMillis":536}}

I looked at a similar post on the forum and saw a message from @Doug_Duncan suggesting looking at how shard_key is defined. From what I can tell the shard key for this collection is shardKey: { _id: 1 }, I still am new to shard keys and need to read up on the significance of this. But perhaps that might be the issue

6

So now I have a new question, I’ve read about config.system.sessions and since this is setup + monitored by mongod/mongos it makes me wonder two things:

  1. should I change the shard key for it
  2. is 20-30 minutes for it to drain reasonable

From my understanding it takes 20-30 minutes to drain this collection when it is on a single shard. 10ish minutes when it is balanced.