Some queries which are normally fast are very slow sometimes, yet the execution plan is IXSCAN and there are locks that the query had to wait according to the profiler. Someone at Stack Overflow suggested looking at the globalLock section of serverStatus but I don’t know how to interpret the data. There are very few users reading from the database at the same time so MongoDb should be able to handle the load.
This is the output of serverStatus when the query is slow
Having IXSCAN is a good start, but it doesn’t mean it has an efficient index to support your queries. Analyze the logs first and also determine if you have an undersized resources. Keyhole can help on the performance analytics.
It’s difficult to say what’s going on in the server using a single snapshot of serverStatus. Ideally a series of serverStatus output captured during a period of time could show a more complete picture. However this is not a trivial troubleshooting effort and would require tooling, understanding of how WiredTiger interfaces with MongoDB, and a lot of time and patience
Having said that, from your description, it sounds like your hardware is struggling to meet demand. IXSCAN won’t help much in an overburdened machine due to various reasons, e.g. the working set is too large for the amount of RAM, slow disks, multiple queries that requires a change in cache content which results in many loading/unloading of cache contents, among many.
If it’s possible, the low hanging fruit is to try to increase the RAM size of your deployment and see if it improves the situation. I would next try to hunt for inefficient queries to understand if the underlying cause was simply not enough hardware, or something else.