I’m planning a very large implementation of MongoDb. It will be hundreds of TB and maybe even over a PB. There is no need for global distribution, it will all reside in a single data center.
My question is, would it be better to have many small shards or fewer large shards? To put it another way, should I consider hundreds of bare metal boxes, one for each RS node or would it be better to have thousands of containerized nodes?
Hello @Tim_Heikell
without deeper knowledge of your use case it is difficult to be precise. Since you have quite a bit of data, it most likely will be better to scale horizontal to a higher number of shards. The mongos will distribute the queries on the individual machines and only the final state will take place on the mongos or the primary shard, depending on your query. This way you will have smaller amounts of data to be processed on the indiviual machines. I am assuming here that you have a good shard key (high cardinality and frequency, non monotonic increasing) for targeted queries.
I also would highly recommend to split your data on at least two, better three data centers!
Cheers,
Michael
I’m using a hashed index on the _id for my shard key but many queries will be using indexes which do not include the _id. I expect that scaling out makes the most sense.
Thanks for the advice.
Tim
Hi Tim,
If your common queries aren’t using the shard key they will be scatter-gather (directed to all shards in the cluster). This won’t be ideal for performance if you have hundreds of shards to query. If your goal is to efficiently scale out, it would be better to have range-based shard keys that support your common queries and allow for targeted queries. For more information see Read Operations to Sharded Clusters.
An important consideration with capacity planning is your working set of actively used data and indexes. If your use case has a large amount of historical data that is infrequently accessed, the desired ratio of storage to RAM will be different than working with the same data set in memory. Scalability has several dimensions (cluster scale, performance scale, and data scale) which will affect your deployment decisions. For some examples of different scaling scenarios, see: MongoDB at Scale.
Depending on your use case, you may also want to consider zone sharding with tiered resources for varying SLA or SLO.
I’d recommend getting Professional Advice to plan a deployment of this size. There are many considerations, and an experienced consultant can provide better advice with a more holistic understanding of your requirements. Some decisions affecting scalability (such as shard key selection) are more difficult to course correct once you have a significant amount of production data.
Regards,
Stennie
Hello @Tim_Heikell
@Stennie_X knows it all, so please go with his suggestions. Re-reading this post one thing came up in my mind: With MongoDB 4.4, currently in beta, you will have compound hashed shard keys. This will enable you to choose a shard key which enables targeted queries and also, by “adding” a hashed key you gain high cardinality. But, as Stennie suggested, when you build such a large setup it probably pays off fast the get professional advice on board.
Cheers,
Michael
Hi Stennie.
In my case I have different properties I need to query for and I don’t think I have any way to get around my queries being scatter-gather. Additionally, although I have a lot of data that is mostly stale, there is no way for me to know which data is stale. Any of it can become relevant at any time. The good news is that the queries will be executed by background services so ms responses aren’t critical.
There are clearly too many factors for a non-expert like myself to find the best model so I think professional advice is needed.
Thanks.
Tim