Balancing performance after adding a shard

Ops and Admin Sharding
,
1

Hello,
can a single-core performance of primary config server be a bottleneck of chunk migration after adding an additional shard?

Context:
I have a 4-shard (3 nodes each, plus 3 config server) 4.4.1 cluster. Each node has about 3TB compressed (zstd) data, ~15TB uncompressed. I have recently added a new (5th) shard due to DB growth. After a few hours I calculated that balancing will take long weeks or even months. I also found that mongod process on primary config server uses about 1 full CPU core (on a 4-core) system and seems to be a bottleneck since I/O (including network) and CPU usage on other nodes is rather low. Is the balancer process single-threaded or it just doesn’t want to use the whole CPU? Can I speed it up by adding more CPU cores?
Is there any other way to speed the balancing up?

Without that I may have disk space issues on existing nodes since I won’t be able to release disk space fast enough.

Thanks
Andrzej

2

Bump, after almost 2 years and adding new shards I always have the same problem of balancer performance but it hits me even harder (the performance is even lower).
Is there ANY way to speed up the balancer?

3

Hey Andrzej,

The balancer is currently single-threaded in order to reduce the impact of chunk migrations on ongoing workloads. With that said, depending on your machine sizes, you can be bottlenecked on RAM, CPU or IOPS. We are actively working on speeding up the addShard process and will get back to you once the improvements are delivered.

Thanks for your input!

Garaudy

4

Hi @Garaudy_Etienne ,
I am also having production issues with this, after adding a new shard, the performance is hit very hard.

  1. CPU goes up, being a bottleneck
  2. the find() queries seem to very slow (maybe due to the chunk migration, with the re-indexing being re-calculated?), making data not usable. For example, a query taking a few millisecond time on a regular base, takes minutes to finish while the shard addition takes place.

Is their any way to handle this until the addShard optimizations are done in the next versions?

Currently using mongodb 6.0.6.

5

Try running the balancer in non peak window only?

Migrating data of course consume resources amd may impact performance at busy time

6

Thank you @Kobe_W .
Unfortunately, I see that the balancer window should be used when:

particularly when your data set grows slowly and a migration can impact performance

In my case the data grows very quickly (~5K IOPS).

Also, seeing the following desclaimer:

The balancer window must be sufficient to complete the migration of all data inserted during the day.

And am not really sure what the length of the window should be in my case, or what should I do if I don’t find the right window.

Is there any other option we can use? or can we make it less impactful in general? (without a window)

7

Try searching mongo doc. There are some available params to tune it.

Eg

8

If your io is really high, also consider upgrading your hardware.

9

There is an option of resharding twice, that would be much faster and less impactful on your system. You would need to be on MongoDB 5.0 or later and make sure you have enough spare disk storage capacity. Please read this article to understand the concept behind it.

  • Let’s say your shard key is userId:1. You would reshard to the hashed version of your shard key, so userId:"hashed" then reshard again back to your original shard key of userId:1.

  • Let’s say your shard key is state:1,city:1. You would reshard to the hashed version of your shard key, so state:"hashed",city:1 then reshard again back to your original shard key of state:1,city:1.

Depending on how much data you have per shard, each resharding could take anywhere from a day to a week. This works much faster because resharding writes to all shards in parallel instead of migrating data one chunk at-a-time.

The upside is that it’s much faster than chunk balancing/migration and has almost no impact on your workload (since you simply drop the old collection), meaning you can run it 24/7. But you must ensure that you have enough spare disk space for it. (Please read the resharding documentation)

Since you are resharding to the hashed version of your shard key and then back, you DO NOT need to rewrite your application’s queries to use both the current shard key and the new shard key. You can simply reshard twice without any changes to your application.

10

Hi @Garaudy_Etienne ,
Unfortunately, I don’t meet the requirements of 1.2x free disk space to do this, and part of the reason for the shard expansion is due to low disk space, so the data can move out to other shards, so it will lower the load and disk usage.