I am now getting closer to understanding the issue, as I have now been able to consistently re-create the issue. The issue occurs when I remove the shard that hosts the chunks for the config database.
From sh.status():
database: { _id: 'config', primary: 'config', partitioned: true },
collections: {
'config.system.sessions': {
shardKey: { _id: 1 },
unique: false,
balancing: true,
chunkMetadata: [ { shard: 'shard-two', nChunks: 1024 } ],
chunks: [
'too many chunks to print, use verbose if you want to force print'
],
tags: []
}
From logs of mongod on the shard that is getting removed:
2023-10-25T11:49:21Z charmed-mongodb.mongod[4753]: {"t":{"$date":"2023-10-25T11:49:21.835+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"conn179","msg":"Slow query","attr":{"type":"command","ns":"admin.$cmd","command":{"_shardsvrMoveRange":"config.system.sessions","toShard":"shard-one","min":{"_id":{"id":{"$uuid":"83400000-0000-0000-0000-000000000000"}}},"waitForDelete":false,"epoch":{"$oid":"6538e3c125ae3b07d28d5eaf"},"fromShard":"shard-three","maxChunkSizeBytes":200000,"forceJumbo":2,"secondaryThrottle":false,"writeConcern":{"w":1,"wtimeout":0},"$clusterTime":{"clusterTime":{"$timestamp":{"t":1698234561,"i":1}},"signature":{"hash":{"$binary":{"base64":"BDUe5gGly3g4iPDSiuCwtTRJTLU=","subType":"0"}},"keyId":7293828730399490051}},"$configTime":{"$timestamp":{"t":1698234561,"i":1}},"$topologyTime":{"$timestamp":{"t":1698233824,"i":2}},"mayBypassWriteBlocking":false,"$db":"admin"},"numYields":0,"reslen":236,"locks":{},"writeConcern":{"w":1,"wtimeout":0,"provenance":"clientSupplied"},"remote":"10.18.246.22:60792","protocol":"op_msg","durationMillis":536}}
I looked at a similar post on the forum and saw a message from @Doug_Duncan suggesting looking at how shard_key is defined. From what I can tell the shard key for this collection is shardKey: { _id: 1 }, I still am new to shard keys and need to read up on the significance of this. But perhaps that might be the issue