Hello everyone,
it is planned to use MongoDB in a new project. Will be the Main site and disaster recovery (DR) site for it. So, 2 nodes will be on Main and 1 node will be on DR.
If any 1 node goes down, the replica set continues working. That sounds good. But if the Main site goes down, DR remains in read-only mode. Not so good.
Is there a procedure to manually transition MongoDB DR instance to read-write mode to continue work and then merge new changes into the main site when it will be available?
No.
Essentially once you execute the procedure to reconfigure a secondary it diverges. Things can get… messy after that if precautions are not taken. The best way to recover is to then add the nodes from the primary site back as new members and sync from the node in the secondary site.
I like a 5 node cluster for this configuration with one of those nodes being an arbiter in a third datacenter and using member priority to keen the primary in the main site during normal operation. An arbiter can be modest in specs compared to the data nodes. This allows for network partitioning to occur, primary move to the secondary site, and recovery back to the main afterwards(given an appropriately sized oplog).automatically.
Thanks you! The solution using 3 datacenters looks great but I am not sure about operator of planned system has that possibility.
I understand that there is many reasons to get unavailable primary site. It can be only connectivity issue, then mongo cluster on primary site will work the same way as if DR site goes down. In this case parallel changes are possible if a node in DR site will be available for write.
Or it can be a nature cataclysm that will brake all of primary site and secondary will remain single source of truth.
So, disaster recovery procedure shall include a decision, can we bring up a primary site pretty fast and remain DR site in read-only mode or we shall discard data on primary site, build a new cluster on DR site and then add primary site nodes to it and after finishing of synchronisation exclude two additional nodes from DR site.
How it looks? Are there good practices to get back to work until primary site is available for a replica set across 2 datacenters?