Thanks a lot Max and Mark for your detailed replies. And Stennie too.!
Asya, I agree with your team’s thoughts as you are the experts in this area.
To make this thread complete, also wanted to share more details on our thoughts and reasoning around our proposal.
The failover and High Availability is maintained in general by a set of servers outside of the actual data nodes. An example could be Sentinel monitoring for Redis, Or Zookeeper monitoring for Solr/Kafka etc.
The difference with Mongo is that the data nodes also participate in the voting which is different than the systems mentioned above.
We have a high availability setup of 3 nodes, that generally monitors the data nodes as mentioned above.
So we thought we would fit in the arbiters as well into the three nodes of High availability cluster, to keep the installation consistent in our High Availability cluster nodes.
But we realized late that Mongo doesn’t work this way as the data nodes (primary and secondary) also participate in voting process.
As you guys suggested, we will trade-off the write concern to the deploying of additional hardware. We will have the decision made on this by next week and will keep you posted on the decision progress.
Many thanks again for all your help and guidance on this. Truly insightful every feedback.
Regds,
Sivakumar