There are some things I’d like to know further:
We are observing performance issue in our replica-set having 29 nodes.
What are the output of rs.conf() and rs.status()?
Application use secondary read preference and read concern "majority " or “linearizable”.
The attached query contains the linearizable query that took 675ms. Do you have an example of this same query that takes much less time? Also, read concern linearizable is always performed against the primary, and only for a single document. Which query type do you have the most issue with? The secondary reads or the linearizable reads?
Server capacity is same for all the nodes as build based on same AMI.
What’s the server spec you’re using (RAM size, disk IOPS, EC2 type, etc.)?
Some additional questions:
- What’s your exact MongoDB version?
- Do you notice a pattern when the query become slow or fast? Are they slower on certain time of days, or with x number of clients, for example?
- Are the disks using provisioned IOPS, or are they burstable types?
- Could you post more example queries with their varying execution times?
Best regards
Kevin