This is how looks my main collection :
STORAGE SIZE: 248.29GB
TOTAL DOCUMENTS: 3564898
INDEXES TOTAL SIZE: 4.74GB
When I do a simple documentCounts({}) without any filter (so no index required) it takes about 90 minutes.
The cpu usage of the shards does not exceed 10%
Is it a normal behavior and can I make it faster ?
How have suggested @steevej you can add on your query a condition on the _id field, coz MongoDB create a unique index on the _id field during the creation of a collection.
So the possible way to improve the performance are:
Hi @ABot,
i think yes coz (for the first time), the function countDocument(), perform behind an aggregate’ s operation, checking the condition for all documents in the collection.
I need the exact count, and sometimes with a filter. But I have issue with or without filter even with basic indexes.
Yes the amount of document increase minute by minute
The weird thing is that everything is working well on my previous setup with a single mongod server:
v 4.4.1
64Gb RAM (but mongod WTcache limited to 1Gb because the rest of the memory is used for other tasks)
16 Cpu
On my new setup it’s a sharded cluster but with more memory allocated:
6Gb WTcache per instance
v 6.0.1
4 Cpu but they are almost not used when I do the count query
Maybe it’s a disk speed problem ? I will do a speed test later
Even if I do the query in a loop it’s always long, like nothing goes in the cache.
I tried to restart the instances, and perform the count query, to force the good index to be set in memory but no success.
I have 18 indexes on this collection, maybe it’s too much (Index total size is less than 5GB so it should fit in memory).
In my mind is does not seems to be a hardware issue since the CPU is at least not moving, and the mem should be enough for the indexes.
But thank you Steeve, I will check the doc you sent to see if I missed something
Do you mean that each machines is 16GB with 4CPU and that you have a total of 7 machines, that 2 PS per shard, 2 PS for config server and 1 machine for mongos.
or
Do you mean that you are running 6 data bearing instances and mongos on a single 16GB machine?