Each shard has a member with unbalanced disk usage

Hello , MongoDB Team.

Our Sharded Cluster has 3 shards and each shard contains 3 members.
But , specific 1 member of each shard tends to show high disk usage.

(MongoDB binary : Community Server)

And, it seems not to be associated which is Primary or Secondary.

Here is disk usage on each shard :

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

< Shard 1>

  • Member 1-1 (Primary)

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 3.1T 2.9T 140G 96% /home

  • Member 1-2 (Secondary)

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 3.1T 2.3T 754G 76% /home

  • Member 1-3 (Secondary)

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 3.1T 2.3T 754G 76% /home

< Shard 2>

  • Member 2-1 (Primary)

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 3.1T 2.3T 776G 75% /home

  • Member 2-2 (Secondary)

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 3.1T 2.9T 157G 95% /home

  • Member 2-3 (Secondary)

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 3.1T 2.3T 773G 75% /home

< Shard 3>

  • Member 3-1 (Primary)

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 3.1T 2.9T 154G 96% /home

  • Member 3-2 (Secondary)

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 3.1T 2.3T 758G 76% /home

  • Member 3-3 (Secondary)

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 3.1T 2.3T 759G 76% /home

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

To ask investigation , I was seeking where this difference was made.
Total usage of user-made database was almost the same but database ‘local’ was not.

specific only one member of a shard shows high usage on ‘local’ DB.
I think this was the reason

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

< Shard 1> ## user-made DB : noteDB_v2

  • Member 1-1 (Primary)

654G local
2.2T noteDB_v2

  • Member 1-2 (Secondary)

38G local
2.2T noteDB_v2

  • Member 1-3 (Secondary)

38G local
2.2T noteDB_v2

< Shard 2>

  • Member 2-1 (Secondary)

38G local
2.2T noteDB_v2

  • Member 2-2 (Primary)

655G local
2.2T noteDB_v2

  • Member 2-3 (Secondary)

38G local
2.2T noteDB_v2

< Shard 3>

  • Member 3-1 (Primary)

646G local
2.2T noteDB_v2

  • Member 3-2 (Secondary)

38G local
2.2T noteDB_v2

  • Member 3-3 (Secondary)

39G local
2.2T noteDB_v2

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

I need to know why it has happened
and need to get the solution.

(
. Cluster Establishment :
3 Routers
3 Config server members
3 Shard (3 members each / No arbiter (PSS))

. Version : MongoDB 4.4.22 (Community Server)

I asked this via MongoDB Customer Support first, but they told me any questions with Commnunity server will be proceeded here .

)

Thank you.

Hi @Jae-woong_Lee

Check the collection size for each collection. Its likely the Oplog but find out for certain.

This version is now end of life.

Upgrading to a current supported version is recommended.

1 Like