Looks like everything is completely overloaded and the secondaries can’t keep up with the primary.
This is honestly VERY concerning.
Reduce the workload or increase the server capacities ASAP. RAM + CPU + IOPS.
The fact that the primary can’t keep up means that it cannot acknowledge write operation with writeConcern w>1 and this also means that their is an additional cache pressure on the primary which can’t move the majority-commit point forward.
I would first stabilize this system before considering upgrading anything. You need a health cluster to consider doing a rolling upgrade a new version.
In this current state, if the primary goes down, one secondary will be elected (I guess) and it’s going to rollback 0.73hrs worth of data. (The most up-to-date node should win the election).
Find out first which ressources is saturated on the secondaries and work on them one by one until they can catch up with the primary. Once you are confident than one of them can be elected primary, you can work on the primary. If you are using the default readPreference and writeConcern, the primary should be doing the reads and the writes while the secondary should only be doing the writes. So in theory they should be less overloaded than the primary.
Cheers,
Maxime.