"FETCH" step in secondary replica is slower than primary replica

Other MongoDB Topics
, , , ,
1

We have the following setup:

  • 3 replicas
  • 1 primary
  • 2 secondaries

Our read preference is set to secondary, for some reason the “FETCH” step in the secondary is way slower than the “FETCH” step in primary (1.5s vs 250ms).

As soon as I set my readPreference to primary, everything became faster.

  • All replicas share the same hardware.
  • All replicas are using the same indexes.
  • None of the replicas is set to readOnly (although I’m not sure if it makes any difference)

Here are some extra observations:

  • Secondary Cache details:
{
  "Cache Size (GB)": 27,
  "Current Usage (GB)": 22,
  "Pages Requested": {
    "low": -769666212,
    "high": 157,
    "unsigned": false
  },
  "Pages Read from Disk": {
    "low": 1196761264,
    "high": 4,
    "unsigned": false
  },
  "Cache Hit Ratio %": "97.29",
  "Cache Overflow": false
}
  • Primary Cache details:
{
  "Cache Size (GB)": 27,
  "Current Usage (GB)": 22,
  "Pages Requested": {
    "low": -1505669711,
    "high": 416,
    "unsigned": false
  },
  "Pages Read from Disk": {
    "low": 1655481745,
    "high": 3,
    "unsigned": false
  },
  "Cache Hit Ratio %": "99.19",
  "Cache Overflow": false
}

Here are my questions:

  • What are commands that I can run that could help me figure out the reason behind the slowness?
  • I’m reading that it is preferred to use primary or nearest instead of secondary, would I need to configure any other options?
  • I see a slight difference in the “Cache Hit Ratio %”, is this significant and could be the cause behind the slowness?

I’m happy to share more information, or even run reviewed mongosh commands that could lead to figuring out the root cause and understanding how MongoDB operates.

Thank you!