MongoDB Backup for Sharded Clusters
A MongoDB backup strategy is necessary to protect your mission critical data. Backing up a distributed database system, such as a MongoDB sharded cluster, presents an additional level of complexity because each shard runs somewhat independently.
Backing up any system requires a high level of attention to detail. Backups may not be fascinating, but they have to be right.
MongoDB Management Service (MMS)
moves the complexity of backing up a sharded system from you to us, while still providing confidence that the backups are reliable. To understand the benefits of MMS, let’s go over the alternative of backing up sharded MongoDB cluster yourself.
You Can Do It Yourself
To obtain a completely consistent point-in-time snapshot across the cluster, you will need to stop writes for a short time, essentially bringing the database offline. Once you do that, you can use any of the standard backup methods that MongoDB supports, including
mongodump
or
file system snapshots
.
Most applications don’t allow stopping all writes to the database, so this way of getting a backup is a non-starter. Instead, most do-it-yourself solutions settle for approximate system wide snapshots.
So just to be clear, if you are doing it yourself, you are probably settling for something other than a consistent snapshot. Even then, it’s not simple.
Best practices are to have each shard be a replica set with at least three nodes. One node will be primary, and there will be two secondaries. Alternatively, you might have two nodes and an arbiter in each replica set.
Here is how to get a mostly-consistent snapshot:
Stop the cluster balancer
Take a backup up of the config database using mongodump.
Lock one secondary node of each shard by issuing db.fsyncLock(). Do this as close to simultaneously as is feasible.
Take backups of each shard in the cluster, preferably using a file system snapshot tool. Unlock each secondary as it completes.
These steps are just an overview. The procedure is explained in-depth in the
MongoDB docs
.
MongoDB Backups with MMS
With MMS, you install the backup agent, turn the system on and you get consistent snapshots of the sharded cluster every six hours. That’s it.
Ok, so how does it work? If you’re familiar with MMS, you’ll know that the MMS Backup Agent tails the replication oplog to provide a continuous backup of your data.
MMS uses that oplog to maintain what is essentially a secondary node of every shard in your cluster. By default, we construct complete snapshots every six hours. For your replica sets, we can provide point-in-time recovery by replaying the oplog to the moment between snapshots that you specify.
For sharded clusters, the backup agent pauses the balancer every six hours and inserts a token across all shards and config servers. MMS is always watching for those tokens. When MMS sees the token, it knows that it’s time to cut a consistent snapshot of the sharded system. The oplogs are applied to all replica sets in the cluster until the point in which the token was inserted, providing a consistent snapshot across the cluster.
In addition to getting a complete snapshot every six hours, you can get more granular restore points by configuring checkpoints. During a checkpoint, a token is inserted to mark that moment. A snapshot is not created, but MMS can build restore images to that point. You can configure checkpoints at 15, 30, or 60 minute intervals.
Let’s talk about the quality of the snapshot itself. It’s not a moment in time to the same degree you could obtain by turning off writes to your cluster. But, it’s better than the the technique outlined above in the do-it-yourself section. The tokens are inserted sequentially into every shard. The snapshots that result represent a view of the database system that an observer might witness standing at any given mongos.
Using MMS, not only do you get a higher quality snapshot than you are likely getting yourself, you are getting it without the hassle of orchestrating your own snapshots. MMS moves the complexity of snapshot creation from your cluster to ours.
You can start
backing up MongoDB now by creating an MMS account
.
April 21, 2014