Single Shard failure bringing entire system down

Ops and Admin Sharding
, ,
3

This query will target all shards of the collection, this is why this particular query will fail.

Any shards chunks will still be available, but if a query need to query the shard that is not available then the query will fail. Also if the unavailable shard is also the primary shard for a database then all the unsharded collections of that database will be unavailable too.

This behaviour is described in sharding / high availability

Given this sharded cluster:

sh.status() (click to expand) 
 sh.status()
shardingVersion
{ _id: 1, clusterId: ObjectId('65e5f0b19111036bf95cc8a6') }
---
shards
[
  {
    _id: 'rs0',
    host: 'rs0/mongo-0-a:27018',
    state: 1,
    topologyTime: Timestamp({ t: 1709568222, i: 2 })
  },
  {
    _id: 'rs1',
    host: 'rs1/mongo-1-a:27018',
    state: 1,
    topologyTime: Timestamp({ t: 1709568229, i: 2 })
  }
]
---
active mongoses
[ { '7.0.6': 1 } ]
---
autosplit
{ 'Currently enabled': 'yes' }
---
balancer
{
  'Currently enabled': 'yes',
  'Currently running': 'no',
  'Failed balancer rounds in last 5 attempts': 0,
  'Migration Results for the last 24 hours': { '1': 'Success' }
}
---
databases
[
  {
    database: {
      _id: 'anotherDB',
      primary: 'rs0',
      partitioned: false,
      version: {
        uuid: UUID('583c1ef7-36bf-487d-8f21-1ce9819d7091'),
        timestamp: Timestamp({ t: 1709603393, i: 10015 }),
        lastMod: 2
      }
    },
    collections: {}
  },
  {
    database: { _id: 'config', primary: 'config', partitioned: true },
    collections: {
      'config.system.sessions': {
        shardKey: { _id: 1 },
        unique: false,
        balancing: true,
        chunkMetadata: [ { shard: 'rs0', nChunks: 1 } ],
        chunks: [
          { min: { _id: MinKey() }, max: { _id: MaxKey() }, 'on shard': 'rs0', 'last modified': Timestamp({ t: 1, i: 0 }) }
        ],
        tags: []
      }
    }
  },
  {
    database: {
      _id: 'test',
      primary: 'rs1',
      partitioned: false,
      version: {
        uuid: UUID('4494c13a-cfe4-43a3-a071-81b0529fffe1'),
        timestamp: Timestamp({ t: 1709568305, i: 1 }),
        lastMod: 1
      }
    },
    collections: {
      'test.foo': {
        shardKey: { code: 1 },
        unique: false,
        balancing: true,
        chunkMetadata: [ { shard: 'rs0', nChunks: 1 }, { shard: 'rs1', nChunks: 1 } ],
        chunks: [
          { min: { code: MinKey() }, max: { code: 519 }, 'on shard': 'rs0', 'last modified': Timestamp({ t: 2, i: 0 }) },
          { min: { code: 519 }, max: { code: MaxKey() }, 'on shard': 'rs1', 'last modified': Timestamp({ t: 2, i: 1 }) }
        ],
        tags: []
      }
    }
  }
]

If shard rs1 is unavailable then:

  1. Any query on test.foo where code is 519 to MaxKey will fail. (Chunks are on rs1)

    example 
    [direct: mongos] test> db.getSiblingDB('test').foo.aggregate([{$sortByCount:'$code'}])
MongoServerError[FailedToSatisfyReadPreference]: Could not find host matching read preference { mode: "primary" } for set rs1
    example 
     [direct: mongos] test> db.getSiblingDB('test').foo.aggregate([{$match:{code:{$gte:519}}},{$sortByCount:'$code'}])
 MongoServerError[FailedToSatisfyReadPreference]: Could not find host matching read preference { mode: "primary" } for set rs1

  2. Any query on unsharded collections on the test database will fail. (rs1 is the primary shard for test

    example 
    [direct: mongos] test> db.getSiblingDB('test').bar.find({})
MongoServerError[FailedToSatisfyReadPreference]: Encountered non-retryable error during query :: caused by :: Could not find host matching read preference { mode: "primary" } for set rs1

  3. Queries on anotherDB will succeed. (rs0 )is the primary shard for anotherDB

    example 
    [direct: mongos] test> db.getSiblingDB('anotherDB').bar.aggregate({$sortByCount:'$f'})
[
  { _id: 0, count: 3334 },
  { _id: 1, count: 3333 },
  { _id: 2, count: 3333 }
]

  4. Queries on test.foo where code is from MinKey and less than 519 will succeed.

    example 
    [direct: mongos] test> db.getSiblingDB('test').foo.aggregate([{$match:{code:{$lte:518}}},{$sortByCount:'$code'}])
[
 { _id: 226, count: 11 }, { _id: 506, count: 10 },
 { _id: 208, count: 10 }, { _id: 156, count: 10 },
 { _id: 69, count: 10 },  { _id: 416, count: 10 },
 { _id: 132, count: 10 }, { _id: 9, count: 10 },
 { _id: 348, count: 10 }, { _id: 65, count: 10 },
 { _id: 323, count: 10 }, { _id: 141, count: 10 },
 { _id: 263, count: 10 }, { _id: 148, count: 10 },
 { _id: 499, count: 10 }, { _id: 288, count: 10 },
 { _id: 268, count: 10 }, { _id: 22, count: 10 },
 { _id: 442, count: 10 }, { _id: 374, count: 10 }
]
Type "it" for more