One Shard Getting High CPU

We are having 5 shard cluster in one shard the CPU is reaching maximum & Remaining 4 shards are good, How can i find the high CPU usage in this shard and how can we balance it
to other shard to decrease the CPU usage of the particular shard

Hi @dev123_dev123 and welcome in the MongoDB Community :muscle: !

This isn’t a good sign. :confused:
It probably means you have something unbalanced in your cluster.

I’d first run a mongostat and mongotop on each shard (==replica set) in your sharded cluster to check that they are all roughly similar. If your shard keys are correct, your work load should be shared evenly across the different shards.
With mongotop for example if you see that there is a clear difference on that particular shard VS the other shard on a specific collection, at least now you know which collection(s) is(are) causing the problem.

I would also run sh.status() to check that you don’t have jumbo chunks and that the chunk repartition is even across the 5 shards for all the sharded collections.

While you are on the result of this command, I would double check all the shard keys and make sure that they all follow the good practices - especially that none of them is growing monotonically. This would result in a single shard receiving all the insert operations for that particular collection which isn’t very scalable.

I hope this helps.
Don’t hesitate to provide the output of these commands if you can’t find the problem.

Cheers,
Maxime.

Hi, is there any method we can check how that CPU is getting used? We meet a similar problem that traffic causes all shards CPU to get high but only 1 gets to 90%+ and the rest of them stay at 60%.
We are actively debug what might go wrong, but wondered if we have more tools and monitors than sh.status() on the shard.

You will need system level tool to check cou usage. Eg top.

Even say your data is evenly distributed some keys may be hot than ithers and /or disk usage or network traffic differs.

Or maybe caused by other processing on the same node

Have you tried running mongotop & mongostat on this Shard (i.e. replica set) and compared the values with other shards?
Mongotop will show you the collections that are using the more resources per seconds. It’s a good way to identify a collection that is using a bad shard key (growing monotonically) for example.