If we have two shards across two DCs then if LD5 DC ( primary) is totally lost, can we restore mongo cluster data using ARK DC(secondary) without any data inconsistency issues?
Using mongo 6.x community
Not able to find my older posts, but i would say, the best way is always to “not to lose the primary DC”.
Why? backup is always done at a time in the past. (e.g. point in time backup), Then to recover the writes issued after the point, you need to use oplog entries. it’s very difficult/challenging to apply those changes so that all the data on all the nodes are consistent and update to date.
Thanks @Kobe_W !
So, you mean to say it’s not possible to recover the cluster using just secondary DC?
btw another question, can secondary DC nodes join/be part of new cluster if I restore primary DC nodes of the cluster from the backup?
it’s possible, but likely with data loss.
Generally yes. Before backing up the cluster, balancing should be turned off so that config data is consistent with shard data. Then after restoring, though shard data may not be 100% consistent, config data should be fine.
data loss is not an issue and with write majority this can be avoided I guess. My concern is with inconsistency of config server not in sync with shard server. is that possible?
oh, need to correct myself. To recover with a backup, likely you get a data loss. To recover from a new election, as long as some working nodes have the most recent data, it will be fine. primary is down, but writes are still in secondary, then secondary is elected during a failover, then the old-primary rejoins and starts syncing. no data loss.
This shouldn’t happen during normal operations, as long as the connection between config servers and shard nodes is working.