Hi mongoldb forum,
I hope i am posting this in the write place.
We have been attempting to upgrade our two replica sets from 3.6.15 to 4.0.27 a while now and have been running into issues with the servers running the 4.0.27 binary. Each replica set has 3 members, the primary and two secondaries.
Upgrading the 1st replica set was easy and works as intended it does however not have to same amount of load as the primary replica set.
When updating the primary replica when it receives some queries that take longer to complete (I am talking aggregation pipelines that can take up to 30 minutes to complete) replication lag will start to continuously rise while that operation is in progress. We cannot see what introduced in version 4.0 could be causing this since we never do not have that problem with servers running version 3.6.15.
All servers in each replica set are running the exact same hardware which are amazon EC2 instances.
Any indication of where we have gone wrong would be greatly appreciated.