Hi. We’ve been experiencing the problem that when we add a node to a replica set (mongo4.0) from backup files, the application with readPreference=secondary often encounters latency spikes for a period time.
I know cache of the new node is cold, and the reads may be served from disk.
Is this problem solved in mongo 5.0 ? I know there’s Hedged Reads, but not sure whether it can solve the spike problem when a new node is added ?
Maybe we can gradually send requests to the new node when the connections between mongos and mongod are spawned (SpecificPool::spawnConnections) ?
If you’re restoring from a backup, the new node will have to catch up to the primary. What does the CPU and IOWAIT look like on said node until the latency spikes resolve?
It could be that the node is under load trying to replay the writes to the primary during replication, saturating disk, and at the same time you are trying to load working set into memory to serve reads.
I’d try to restore from as hot as a backup as possible to limit replication time.
Then I would use the touch command, which will load data from disk while preventing reads to that machine until it is ready: touch — MongoDB Manual
1 Like