Hi @Justin_Sayre welcome to the community!
If you’re just starting on your MongoDB journey, I highly recommend the free courses available at the MongoDB University. They cover materials from beginners to more advanced topics.
Having said that, I will go ahead and just jump into the deep end with your questions
The secondary has fallen out of sync with the primary and I cannot catch it up. It appears the oplog size is not large enough.
To determine your oplog’s length in time, you can use
db.printReplicationInfo(). Note that the interpretation of this command assumes a steady state of writing, so it could show less time if your cluster is busy, or show more time if it’s less busy.
Once a secondary fell off the oplog, the only way to recover it is to do a resync.
If I understand things correctly, I need to extend the oplog size on secondary before primary (with DB restarts)
That scenario is assuming the secondary is still functioning. I believe it is not in your case. Thus the procedure outlined in Change the Size of the Oplog doesn’t really apply to you since you’re gonna rebuild a new secondary anyway (with an appropriately sized oplog from the start). At this point, you just need to resize the primary’s oplog.
Am I better off at this point removing the secondary from the cluster so that if the primary fails, the now-defunct secondary does not try to take over?
A defunct secondary will never try to take over (marked by its status of anything other than
SECONDARY in the
rs.status() output). Only when a node having a status of
SECONDARY will it be able to take over as primary, since that status means that it’s up, ready to take over, and is following the primary’s write closely.
In your situation, I would assess what is the appropriate oplog size for your workload. This can be done using simulations of your production workload, so that this situation doesn’t repeat itself in the future.
I would also consider deploying a primary-secondary-secondary setup instead of primary-secondary-arbiter. Having two secondaries vastly increase the chances of the whole set having zero downtime in the face of failures. It will also help with zero downtime maintenance, since with two secondaries you can do an effective rolling maintenance/upgrade.
Another point is, if your current setup have a different hardware spec between primary/secondaries, I would consider making them all the same, since in a replica set, all nodes have an equal chance of becoming primary (assuming default setup of voting/priority). Unless you have very specific needs and reasons, I would not change the default voting/priority settings, since it will interfere with High Availability guarantees a replica set gives you, and making failure scenarios more complex.