I am facing the issue of intermittent spikes in response latencies while fetching data in a sharded cluster. Data is always being queried via indexed fields.
Also seeing high memory usage on secondary boxes (memoryUsage:99% loadAvg:2.5) at exact same time as the latencies.
Client: Morphia Client with ReadPreference = PRIMARY, ReadConcern = Local, WriteConcern Default. As per my understanding secondaries should not affect the response if ReadConcern is Local.
Is there any correlation here?
MongoDB version 4.2.2
mongod instance running on 8 core, 40 GB RAM machine, wiredtiger cache 11 GB.
Can anybody suggest what might be the issue or how to debug this?
Unfortunately, we don’t have enough information to diagnose the problem. Memory usage is unlikely to be related unless you’re experiencing cache pressure (in fact, 99% memory usage could simply be filesystem cache if you’re checking with free -m).
Latency can be caused by a lot of things. It includes the network initiation, network round trip, query execution, and client-side deserialization. Do you have any measurements available to isolate the issue? Generally, intermittent spikes are network based. If the spikes are more regular, it may be a cron job. Are you instantiating a lot of connections all at once when you’re seeing the spikes? It could be CPU exhaustion.
Ultimately, there are so many things it could be that we’ll need a lot more information. Are you using Atlas? The profiler and performance adviser are invaluable tools for helping to diagnose performance problems with the mongod instance itself.