Number of insert by second not increase with 2 shards

MongoDB Atlas
,
1

Hello everybody.
Last week we created an account of Mongodb Atlas Service. We are doing do some stress tests to see the performance of one part of our system using Mongo.

Context:

  • Mongo version 5.0.18
  • Test 1
    Resources M30 (azure) with 1 shards
  • Test 2
    Resources M30 (azure) with 2 shards

How the test works?
We are processing in parallel 40 items and then we wait 1s to process 40 items more. The total items to process is 1600.

Every item is not going to be a register in the database, one sigle item will create more than 100 registers inside the database.

The process we followed to create the sharded collections:
1- sh.enableSharding(“db-name”)
2- db.createCollection(‘collection-name’)
3- db.getCollection(“collection-name”).createIndex({ PartitionKey: “hashed” });
4- sh.shardCollection(“db-name.collection name”, { PartitionKey: “hashed” });
5- sh.status() to see if the collections are well created and distributed in the 2 shards.

After that we run a script in our system that creates several registers inside the collections but the time to process these data doesn’t dicrease if we add a new shard.
What we are doing wrong? Why is the total time of the process not dicreasing with one more shard?
Here you can see some data from sh.status after we run our script:

"db-name.collection2": {
    shardKey: { PartitionKey: "hashed" },
    unique: false,
    balancing: true,
    chunkMetadata: [
      { shard: "atlas-7dz9ng-shard-0", nChunks: 2 },
      { shard: "atlas-7dz9ng-shard-1", nChunks: 2 },
    ],
    chunks: [
      {
        min: { PartitionKey: MinKey() },
        max: { PartitionKey: -4611686018427388000 },
        "on shard": "atlas-7dz9ng-shard-0",
        "last modified": Timestamp({ t: 1, i: 0 }),
      },
      {
        min: { PartitionKey: -4611686018427388000 },
        max: { PartitionKey: 0 },
        "on shard": "atlas-7dz9ng-shard-0",
        "last modified": Timestamp({ t: 1, i: 1 }),
      },
      {
        min: { PartitionKey: 0 },
        max: { PartitionKey: 4611686018427388000 },
        "on shard": "atlas-7dz9ng-shard-1",
        "last modified": Timestamp({ t: 1, i: 2 }),
      },
      {
        min: { PartitionKey: 4611686018427388000 },
        max: { PartitionKey: MaxKey() },
        "on shard": "atlas-7dz9ng-shard-1",
        "last modified": Timestamp({ t: 1, i: 3 }),
      },
    ],
    tags: [],
  },
  "db-name.collection2": {
    shardKey: { PartitionKey: "hashed" },
    unique: false,
    balancing: true,
    chunkMetadata: [
      { shard: "atlas-7dz9ng-shard-0", nChunks: 2 },
      { shard: "atlas-7dz9ng-shard-1", nChunks: 2 },
    ],
    chunks: [
      {
        min: { PartitionKey: MinKey() },
        max: { PartitionKey: -4611686018427388000 },
        "on shard": "atlas-7dz9ng-shard-0",
        "last modified": Timestamp({ t: 1, i: 0 }),
      },
      {
        min: { PartitionKey: -4611686018427388000 },
        max: { PartitionKey: 0 },
        "on shard": "atlas-7dz9ng-shard-0",
        "last modified": Timestamp({ t: 1, i: 1 }),
      },
      {
        min: { PartitionKey: 0 },
        max: { PartitionKey: 4611686018427388000 },
        "on shard": "atlas-7dz9ng-shard-1",
        "last modified": Timestamp({ t: 1, i: 2 }),
      },
      {
        min: { PartitionKey: 4611686018427388000 },
        max: { PartitionKey: MaxKey() },
        "on shard": "atlas-7dz9ng-shard-1",
        "last modified": Timestamp({ t: 1, i: 3 }),
      },
    ],
    tags: [],
  }

Thank you for your time

2

May be, just may be, your bottleneck is not the server. May be your test driver is too slow compared to the server.

Doubling the number of highway lines will not help traffic if everyone transit via exit 35 which only have one line.

So 1600 / 40 = 400, so you are sleeping 400s, so your total time cannot be less than 6.67 minutes.

What is the load on the servers with your single shard setup?

What is the load on the test driver machine?

What do you mean by register? Does 100 registers per item and 1600 items end up being 160000 documents? What is the size of each document? I would really really be surprised if 160000 inserts on M30 even require sharding.

When do you call shardCollection(), before or after the inserts?

You left out so many details about your setup that there is not much more that we can say.

3

Thanks for your response, finally investigating we found the problem. The problem is on the client where is running the app. We have incresed the reources for the app service adding more cpu and ram and after that is increasing the number of inserts per second.

PD: Before to see that the problem was in the app service of the app we have been testing the perfomance with mongoose because we thought that the problem was that they didn’t open engough connections to parallize the operations.

1 Like