I have a 10 shard ( with 3 replicaset) cluster with very heavy traffic. Storage almost 1.5TB per shard in 2000 databases
Since these servers were added at different points on time , the shards are not balanced.
I am using hash sharding
If I start balancer the activity is so intense that my replication stops working across replica sets.
Since these servers have lot of queries on replicas I cannot afford to have replication breaking. I am forced to stop balancers and keep them off
If I manually do a move chunk ( one key at a time ) using sh.movechunk() then this works
So I plan to scriptize this
- Query the large shards and get the keys by directly connecting to mongod
- Distribute these docs using sh.movechunk
- Once some kind of balance is achieved then start balancer
Is that a good idea ?
Does mongodb balancer have some option to run slowly, so that my servers are not “overwhelmed” ? That would be ideal