Selecting AWS Storage for MongoDB Deployments: Ephemeral vs. EBS
The excitement around AWS re:Invent put us in a blogging mood. So, what better to talk about than how to select high performance storage for running MongoDB on EC2?
A Brief Refresher
MongoDB processes operations on data in memory. That is, for a document to be accessed that document must be present in the in-memory cache. Any request for a document that is not currently in memory is first read from disk and loaded into memory. The document is retained in the cache once it has been loaded, and subsequent operations on the document will be much faster as access to disk won't be necessary.
A critical factor in the performance of the system is understanding the size of your application’s working set. The working set is the portion of your data that is frequently accessed. In some applications this can be a small fraction of your total database, whereas in others it may actually be your entire database. If your working set exceeds available memory, documents will repeatedly be read from disk. This can lead to saturated disk and severe performance degradation. You may either scale your system to meet the load by one, or a combination of, the following options.
- Upgrading the instance type to one with more RAM
- Increase the IO bandwidth of the individual instance (e.g. increase PIOPs)
- Increase the capacity of the database by sharding (effectively increasing RAM and IO bandwidth as an aggregate across the set of sharded nodes)
Selecting which combination of these strategies is best depends on your specific workload and their respective costs. For the purposes of this post, we're going to focus on the factors that matter when deciding between Amazon's two storage options: Amazon Elastic Block Store (EBS) versus EC2 Instance Store.
Elastic Block Store or Instance Store? Or, both?
EBS volumes are persistent block-level storage volumes that are attached to EC2 instances. EBS volumes are replicated within their availability zone and are durable if the EC2 instance to which they are attached should fail or terminate. That is, if the EC2 instance on which MongoDB is running should fail, the EBS volume and the data it contains is recoverable.
EC2 instance stores (as known as ephemeral storage) are disks that are physically attached to the host instance. They have better performance, decreased cost, and decreased complexity. Instance stores have lower latencies since the requests do not need to go over the network, and several instance types come with multiple attached volumes that can be RAIDed for greater throughput and capacity. However, when an instance is terminated the ephemeral drive and its data is lost. Fortunately, MongoDB provides high availability through replication. As long as your MongoDB deployment is running as a properly configured replica set, distributed across multiple availability zones, the risk of losing data from an individual instance failure is mitigated.
This doesn't mean that EBS volumes aren't valuable. Ephemeral storage isn’t suited for a stand alone deployment of MongoDB. While such a deployment may not require high database availability itself, an instance failure shouldn't mean that you've unnecessarily lost your data as well. Even in the case that you've configured a replica set using ephemeral storage, you may configure that at least one of the secondaries in the set use EBS volumes as an added assurance of data persistence, just be sure that the EBS volumes are provisioned to match the write load of the primary lest they fall behind in replication. Secondaries configured in this way are often used exclusively for backups and may not be capable of handling a full combined load of reads and writes. In that case be sure to configure this secondary with a priority of 0 to prevent this instance becoming a primary. Also, use replica set tagging to isolate the node from production load.
Using Elastic Block Store: Avoid Noisy Neighbors!
When using EBS volumes in support of a MongoDB deployment, we advise using Provisioned IOPs (PIOPs). Consistent reliable IO performance is critically important in maintaining the performance of the database. PIOPs is an optional but critical feature of EBS that gives your volume reliable and predictable performance. EBS volumes not using PIOPs are subject to noisy neighbor issues and can drop performance in unpredictable and non-deterministic ways. The loss of underlying storage performance will cause issues at higher levels, including the database, and performance of the overall system will degrade as result. Debugging and diagnosing the cause of performance dips can be very difficult if the performance of the storage layer is not consistent.
Dedicated Throughput: EBS Optimized Instances
Network performance between the EC2 instance and its EBS volume is equally important. We suggest using EBS optimized instances to assure consistent bandwidth to the EBS volume. EBS optimized instances have dedicated throughput to EBS volumes that might otherwise be under contention from noisy neighbors. EBS optimization is available for some instance types for an additional hourly rate. However, some instance types have EBS optimization enabled by default.
RAIDing Volumes
You may also RAID your EBS volumes for performance or redundancy. For example, a set of striped EBS volumes, each configured with 16,000 PIOPs, would deliver the performance of a single volume configured to 32,000 PIOPs. However, be sure that the total throughput of the combined volumes does not exceed the throughput of the EC2 instance to which they are attached. The high bandwidth of the EBS volumes can be wasted on an instance that can't keep up.
You'll also need to put special consideration to your backup strategy when using RAIDed volumes. Proper backup of the database requires that a consistent snapshot be made across the set of EBS volumes. This consistent snapshot can either be generated by an Logical Volume Manager (LVM) or by the file system the volume is running under. The procedure for generating snapshots will vary based on how the EC2 instance is configured and what storage engine you are using in MongoDB. Details on the process are available here. If this seems daunting, don’t worry. Consider using MongoDB's Cloud Manager. It's the easiest way to get safe backups for any deployment of MongoDB. And while I don't mean to explicitly plug a MongoDB product, it’s a mistake to go into production without a reliable backup strategy. Make it easy on yourself to deploy safely.
SSD vs. Magnetic
There is another storage option available that I haven't yet mentioned: Magnetic volumes. AWS magnetic volumes are spinning disk drives and are the least expensive storage option in AWS. However, magnetic volumes are really best suited for sequential access patterns. MongoDB induces random IO patterns that can be very tough for a spinning disk to serve. Despite the low cost of magnetic volumes, SSD backed volumes provide much better value for users of MongoDB and are the recommended choice.
Next Steps
Amazon AWS provides many options for instance types and storage. Selecting the right combination depends on your specific workload. We’ve tackled some of the best practices and considerations, but there are many more learning resources available. Please read our white paper on MongoDB & AWS:
About the Author - Bryan Reinero
Bryan Reinero is US Developer Advocate at MongoDB fostering understanding and engagement in the community. Previously Bryan was a Senior Consulting Engineer at MongoDB, helping users optimize MongoDB for scale and performance and a contributor to the Java Driver for MongoDB.
Earlier, Bryan was Software Engineering Manager at Valueclick, building and managing large scale marketing applications for advertising, retargeting, real-time bidding and campaign optimization. Earlier still, Bryan specialized in software for embedded systems at Ricoh Corporation and developed data analysis and signal processing software at the Experimental Physics Branch of Ames Research Center.