Slow Query Performance issue

Ops and Admin Replication
, ,
#1

Hi MongoDB Community,

We are observing performance issue in our replica-set having 29 nodes. Application use secondary read preference and read concern "majority " or “linearizable”. Find query running against GridFS .GridFs collection is indexed properly as per mongoDB recommendation .Find query has different execution time and varies from 0 ms to ~700ms(Attached for more details “QueryExecutions_Mtool”).Query using index and explain plan is always same. “queryHash:6DAB46EC” is always same but different execution times.

Server capacity is same for all the nodes as build based on same AMI.

We tired to diagnosed but not able to find-out root cause .Kindly help to diagnose the issue. We have attached the required details ,If any other details required please let us know.

Thanks in advanced .
Satvant Singh

Connection_String_sample.txt (1.0 KB)
Disk_stats.txt (11.5 KB)
Mongod_Configuration_file.txt (648 Bytes)
MongoStat.txt (51.4 KB)
QUERY.txt (948 Bytes)
QueryExecutions_Mtool.txt (9.5 KB)
ServerConfiguration.txt (233 Bytes)
WiredTIger_Cache_Stats_from_high_utilized_nodes.txt (29.1 KB)

#3

Hi @satvantsingh

There are some things I’d like to know further:

We are observing performance issue in our replica-set having 29 nodes.

What are the output of rs.conf() and rs.status()?

Application use secondary read preference and read concern "majority " or “linearizable”.

The attached query contains the linearizable query that took 675ms. Do you have an example of this same query that takes much less time? Also, read concern linearizable is always performed against the primary, and only for a single document. Which query type do you have the most issue with? The secondary reads or the linearizable reads?

Server capacity is same for all the nodes as build based on same AMI.

What’s the server spec you’re using (RAM size, disk IOPS, EC2 type, etc.)?

Some additional questions:

  • What’s your exact MongoDB version?
  • Do you notice a pattern when the query become slow or fast? Are they slower on certain time of days, or with x number of clients, for example?
  • Are the disks using provisioned IOPS, or are they burstable types?
  • Could you post more example queries with their varying execution times?

Best regards
Kevin

1 Like