Selecting AWS Storage for MongoDB Deployments: Ephemeral vs. EBS

Bryan Reinero
October 7, 2015 | Updated: June 3, 2020
#Technical

The excitement around AWS re:Invent put us in a blogging mood. So, what better to talk about than how to select high performance storage for running MongoDB on EC2?

A Brief Refresher

MongoDB processes operations on data in memory. That is, for a document to be accessed that document must be present in the in-memory cache. Any request for a document that is not currently in memory is first read from disk and loaded into memory. The document is retained in the cache once it has been loaded, and subsequent operations on the document will be much faster as access to disk won't be necessary.

A critical factor in the performance of the system is understanding the size of your application’s working set. The working set is the portion of your data that is frequently accessed. In some applications this can be a small fraction of your total database, whereas in others it may actually be your entire database. If your working set exceeds available memory, documents will repeatedly be read from disk. This can lead to saturated disk and severe performance degradation. You may either scale your system to meet the load by one, or a combination of, the following options.

  • Upgrading the instance type to one with more RAM
  • Increase the IO bandwidth of the individual instance (e.g. increase PIOPs)
  • Increase the capacity of the database by sharding (effectively increasing RAM and IO bandwidth as an aggregate across the set of sharded nodes)

Selecting which combination of these strategies is best depends on your specific workload and their respective costs. For the purposes of this post, we're going to focus on the factors that matter when deciding between Amazon's two storage options: Amazon Elastic Block Store (EBS) versus EC2 Instance Store.

Elastic Block Store or Instance Store? Or, both?

EBS volumes are persistent block-level storage volumes that are attached to EC2 instances. EBS volumes are replicated within their availability zone and are durable if the EC2 instance to which they are attached should fail or terminate. That is, if the EC2 instance on which MongoDB is running should fail, the EBS volume and the data it contains is recoverable.

EC2 instance stores (as known as ephemeral storage) are disks that are physically attached to the host instance. They have better performance, decreased cost, and decreased complexity. Instance stores have lower latencies since the requests do not need to go over the network, and several instance types come with multiple attached volumes that can be RAIDed for greater throughput and capacity. However, when an instance is terminated the ephemeral drive and its data is lost. Fortunately, MongoDB provides high availability through replication. As long as your MongoDB deployment is running as a properly configured replica set, distributed across multiple availability zones, the risk of losing data from an individual instance failure is mitigated.

This doesn't mean that EBS volumes aren't valuable. Ephemeral storage isn’t suited for a stand alone deployment of MongoDB. While such a deployment may not require high database availability itself, an instance failure shouldn't mean that you've unnecessarily lost your data as well. Even in the case that you've configured a replica set using ephemeral storage, you may configure that at least one of the secondaries in the set use EBS volumes as an added assurance of data persistence, just be sure that the EBS volumes are provisioned to match the write load of the primary lest they fall behind in replication. Secondaries configured in this way are often used exclusively for backups and may not be capable of handling a full combined load of reads and writes. In that case be sure to configure this secondary with a priority of 0 to prevent this instance becoming a primary. Also, use replica set tagging to isolate the node from production load.

Using Elastic Block Store: Avoid Noisy Neighbors!

When using EBS volumes in support of a MongoDB deployment, we advise using Provisioned IOPs (PIOPs). Consistent reliable IO performance is critically important in maintaining the performance of the database. PIOPs is an optional but critical feature of EBS that gives your volume reliable and predictable performance. EBS volumes not using PIOPs are subject to noisy neighbor issues and can drop performance in unpredictable and non-deterministic ways. The loss of underlying storage performance will cause issues at higher levels, including the database, and performance of the overall system will degrade as result. Debugging and diagnosing the cause of performance dips can be very difficult if the performance of the storage layer is not consistent.

Dedicated Throughput: EBS Optimized Instances

Network performance between the EC2 instance and its EBS volume is equally important. We suggest using EBS optimized instances to assure consistent bandwidth to the EBS volume. EBS optimized instances have dedicated throughput to EBS volumes that might otherwise be under contention from noisy neighbors. EBS optimization is available for some instance types for an additional hourly rate. However, some instance types have EBS optimization enabled by default.

RAIDing Volumes

You may also RAID your EBS volumes for performance or redundancy. For example, a set of striped EBS volumes, each configured with 16,000 PIOPs, would deliver the performance of a single volume configured to 32,000 PIOPs. However, be sure that the total throughput of the combined volumes does not exceed the throughput of the EC2 instance to which they are attached. The high bandwidth of the EBS volumes can be wasted on an instance that can't keep up.

You'll also need to put special consideration to your backup strategy when using RAIDed volumes. Proper backup of the database requires that a consistent snapshot be made across the set of EBS volumes. This consistent snapshot can either be generated by an Logical Volume Manager (LVM) or by the file system the volume is running under. The procedure for generating snapshots will vary based on how the EC2 instance is configured and what storage engine you are using in MongoDB. Details on the process are available here. If this seems daunting, don’t worry. Consider using MongoDB's Cloud Manager. It's the easiest way to get safe backups for any deployment of MongoDB. And while I don't mean to explicitly plug a MongoDB product, it’s a mistake to go into production without a reliable backup strategy. Make it easy on yourself to deploy safely.

SSD vs. Magnetic

There is another storage option available that I haven't yet mentioned: Magnetic volumes. AWS magnetic volumes are spinning disk drives and are the least expensive storage option in AWS. However, magnetic volumes are really best suited for sequential access patterns. MongoDB induces random IO patterns that can be very tough for a spinning disk to serve. Despite the low cost of magnetic volumes, SSD backed volumes provide much better value for users of MongoDB and are the recommended choice.

Next Steps

Amazon AWS provides many options for instance types and storage. Selecting the right combination depends on your specific workload. We’ve tackled some of the best practices and considerations, but there are many more learning resources available. Please read our white paper on MongoDB & AWS:

Learn more about MongoDB & AWS

About the Author - Bryan Reinero

Bryan Reinero is US Developer Advocate at MongoDB fostering understanding and engagement in the community. Previously Bryan was a Senior Consulting Engineer at MongoDB, helping users optimize MongoDB for scale and performance and a contributor to the Java Driver for MongoDB.

Earlier, Bryan was Software Engineering Manager at Valueclick, building and managing large scale marketing applications for advertising, retargeting, real-time bidding and campaign optimization. Earlier still, Bryan specialized in software for embedded systems at Ricoh Corporation and developed data analysis and signal processing software at the Experimental Physics Branch of Ames Research Center.

← Previous

How Cloud Manager Adds a Replica Set Member Safely

The recent release of Cloud Manager has changed the way we add new members to a replica set with Automation. Here’s how we’ve changed it, and why. Previously, if you asked Cloud Manager to add a replica set member for you with a particular vote and priority setting, we would simply add a new member with that priority and vote setting. Now we do something a little different, and a little safer. With this release, we save the vote and priority settings, but we don’t set them immediately. Instead, we start with vote and priority set to 0 until the initial sync has completed. One benefit of this procedure is that it means that any write concern you use will not be affected by the new node until it is ready to receive replicated writes. Another reason is to prevent the node in initial sync from vetoing a legitimate election. Once the initial sync is complete, we set the priority and votes to the desired levels.

October 7, 2015
Next →

Congratulations to the 2023 APAC Innovation Award Winners

I’m thrilled to announce the nine winners of the 2023 MongoDB APAC Innovation Awards . The MongoDB Innovation Awards honor projects and people who dream big. They celebrate the groundbreaking use of data to build compelling applications and the creativity of professionals expanding the limits of technology with MongoDB. This year, we have broken the awards down regionally to celebrate organizations in APAC, from startups to industry-leading enterprises, across a wide variety of industries, who are delivering big results. We are delighted to announce the winners below: 2023 MongoDB APAC Innovation Award Winners: Positive Impact Open Government Products Open Government Products (OGP) is an in-house team of engineers, designers, and product managers, who is a part of the Singapore Government, and is responsible for building technologies for the public good. OGP used MongoDB’s developer data platform, MongoDB Atlas to create its digital form builder, FormSG. Used by the Singapore government and public healthcare institutions, FormSG securely collects data from residents and businesses and helps public officers to create digital government forms in minutes. It eliminates the use of paper forms and the manual process of transcribing physical documents, which had raised concerns around data privacy and protection. During the pandemic, FormSG enabled public officers to collect more than 100,000 daily temperature declarations nationwide. Today, FormSG has served more than 120,000 public officers from 155 agencies and it has created more than 500,000 digital forms to help the government collect data on travel and health declarations by visitors to the country, applications for COVID-19 swab tests, and applications for financial assistance. Organization Transformation Bendigo and Adelaide Bank Bendigo and Adelaide Bank is one of Australia’s largest banks, with around 7,000 employees helping more than 2.2 million customers achieve their financial goals. The bank has been on a multi-year journey of transformation using MongoDB's developer data platform to improve efficiency and deliver a better customer experience as they fulfill their vision to become Australia’s bank of choice. Recently, the cloud team launched Ready-Set-MongoDB (or RSM). This event-driven framework allows developers to streamline the consumption of internal or external APIs, and applies data transformations and storage automatically within a MongoDB collection of their choice. Using MongoDB Atlas Search, the bank also enabled developers to gain insights across its multi-cloud deployments, identifying cost savings, and providing inventory information to account owners and technical stakeholders. Within the first 18 months of launching these programmes, the automation had saved the organization more than 1,100 developers days. It also helped reduce human involvement, removed stale data, and allowed engineers to focus on the things that matter. The development of Ready-Set-MongoDB is ongoing and improving, as new Bendigo multi-cloud challenges arise and new MongoDB products are released. The application is a perfect representation of how Bendigo's Technology Department is using modern technology, rapid development, and innovation-led problem solving to drive organizational transformation. Heroes in Health Redcliffe Lifetech Private Limited Over the last few years, Redcliffe Labs has become India's fastest growing technology-driven diagnostics service provider. Redcliffe Labs is on a mission to serve 500 Million Indians by 2030 with fusion of technology and world- class laboratories. The company already serves thousands of people daily, with more than 73 labs and close to 1500 walk-in centers across 180 cities. Redcliffe Labs has relied on MongoDB Atlas’ flexible document model to power its innovative Smart Health Report, a patient resource that provides a number of indicators and trackers to gauge holistic health. The MongoDB developer data platform's best in class security, compliance, and privacy controls allows Redcliffe's team to confidently handle even the most sensitive patient data. MongoDB Atlas takes care of many of the traditional database management challenges, which means that developers can spend their time building diagnostics for patients, rather than managing databases. Redcliffe Labs is focusing on incorporating next-generation technologies in the diagnostics space with an AI platform that will make Interactive Diagnostics reports, Advanced Health Profiling and more detailed Diagnostics and Health Alerts. Industry Disruptor Cathay Pacific Cathay Pacific , Hong Kong’s home carrier operating in more than 60 destinations worldwide, has been on an impressive journey to become one of the very first airlines to create a truly paperless flight deck. Until recently, a flight from Hong Kong to New York would require a crew to review more than 150 pages of finely printed text and charts before their flight and make ongoing updates throughout the trip. In 2019, Cathay Pacific conducted the first zero paper flight, removing 50kg of manuals, charts, maps, and flight briefing paperwork. They achieved this enormous feat with the help of one seamless and highly customized iPad application: Flight Folder. Built on MongoDB Atlas, Flight Folder is designed to improve the pilot briefing experience. MongoDB helped consolidate dozens of different information sources into one place, and made it possible for flight crews to easily share their experiences with others. It also included a digital refueling feature that helps crews become much more efficient with fueling strategies – saving significant flight time and costs. The use of MongoDB Device Sync enables seamless syncing and no data loss even when the app goes on- and offline mid-flight. Since the Flight Folder launch, Cathay Pacific has completed more than 340,000 flights with full digital integration in the flight deck. In addition to the greatly improved flight crew experience, flight times have been reduced, and digital refueling saves eight minutes of ground time on average. All these efficiencies have helped the company avoid the release of 15,000 tons of carbon. From Batch to Real-Time Adani Digital Labs Adani Digital Labs is the India-based digital innovation arm of the larger Adani group. The lab’s team's mission is to create one single platform – a SuperApp called AdaniOne – to empower a billion stories in India. To address several use cases and the huge scale that will be required by the superapp, the Adani Digital team selected MongoDB Atlas as its the main transactional database that will further enhance the application. A key component of the app is how it can bring together disparate data in order to provide a single view of activity across the application. In the first process, developers had taken out the data in batches and sent it to their database However, this was too slow and unpredictable as far as business requirements are concerned. Also, the consolidated view of customer history, orders, inventory, and supply chain network updates was likely to impact their customer's ability to generate revenue. Therefore, in order to find a better solution, Adani Digital Labs built a more modern architecture in line with MongoDB. Using MongoDB's Change Streams and the data platform's native Kafka connector, they created an event-based architecture that pushes the data out in real-time for analysis. Adani Digital Labs is still in the early phases of the SuperApp's rollout and collaborating with MongoDB as its developer data platform continues to help the firm to grow and deliver insights in real time. Industry 4.0 Dongwha Founded in 1948, the Dongwha Group has evolved from a singular focus on the wood and timber industry into a global leader across a number of sectors including building materials, chemicals and media. As part of its wider digital transformation strategy, Dongwha required smarter factories that would improve and optimize their production efficiency. Dongwha built an innovative Smart Factory Software platform that collects and analyzes data to enhance quality and production management capabilities. Originally, the platform was built with the community version of MongoDB. However, in order to scale and adapt, the team recently migrated to MongoDB Atlas in the cloud. This enabled them to store large volumes in the fastest and most secure way, optimize their solution for time series data, and make it easy to run machine learning across their data. Dongwha completed the migration seamlessly, without any disruption or downtime to their factories, and it has now been launched across five different sites. Over the last year, the application has significantly increased its availability and reliability while performance has improved by as much as 6x . As they look to the future, Dongwha plans to roll out the software to more of its international factories. Digital Native myBillBook India is home to more than 60 million small and medium-sized businesses (SMBs) but only a small portion of those SMBs are taking advantage of digitization and many still operate using pen and paper. In addition, many businesses in India still struggle with fluctuations in internet services, outages, and latency. FloBiz is on a mission to change that with myBillBook , a one-stop solution that helps SMBs create professional invoices, manage stock, collect payments, automate reminders through smart banking, engage with their customers, manage staff attendance and payroll and generate more than 25 business reports for accounting and decision making. The app is also mobile-first, so businesses can access them from their mobile devices and allows users to manage billing and inventory in both online and offline environments. The myBillbook app is powered by MongoDB Atlas, providing the flexible and scalable foundation for the business to do everything from building new features to performing complex analytical queries. In addition, MongoDB Realm, the mobile database within the data platform, supports offline usage and syncing to ensure there is never data loss or functionality for users due to poor internet connection. Because of its success in supporting customers with business critical operations, more than 6.5 million business owners in India are now using myBillbook for their billing, accounting, collection and business growth. Customer Focused KASIKORN Business-Technology Group Established in 1945, Kasikornbank (KBank) is one of the largest and oldest banks in Thailand. Their mission is to strive towards service excellence and empower every customer’s life and business. One of KBank’s subsidiaries, KASIKORN Business-Technology Group (KBTG) , developed a mobile banking application – MAKE by KBank. MongoDB Atlas’ flexibility and ease of development enabled MAKE’s development team to choose the best type of database for its tasks, to automate data tiering with Atlas Online Archive, and to reduce hours spent on operational maintenance. With more time to focus on delivering new innovations to customers, they created unique features like Cloud Pocket which can allocate funds into unlimited customizable pockets for separate usage. They also built Pop Pay, a feature that allows users to easily search for nearby friends and transfer money by clicking their profile picture as well as “Expense Summary" a spending analysis services that helps inform and manage users’ financial habits. As of January 2023, MAKE has acquired more than 1 million users, and increased the number of transactions in MAKE from 900,000 to more than 7.5 million in a span of one year. Massive Scale China Mobile China Mobile provides mobile voice and multimedia services via its nationwide mobile telecommunications network across mainland China and Hong Kong. It is the world's largest mobile network operator by total number of subscribers. The telecommunications leader is using MongoDB to support one of its largest and most critical push services, which sends out billing details to more than 1 billion users every month. Prior to MongoDB, the tech team relied on Oracle, but as the user numbers increased, performance degraded. Despite large investments, it was still taking too long to do basic requests like finalize and deliver bills to users. In 2019, after comprehensive testing, China Mobile migrated to MongoDB. By taking advantage of MongoDB's native sharding, they were able to improve performance by 80% and go from 50 Oracle machines, to just 12 machines for the same workload. The service now handles all current requirements and is set up to scale with future growth. With the support of MongoDB, China Mobile is growing steadily,with more than 168 million monthly users and has one of the highest customer satisfaction scores in the China Mobile group.

March 30, 2023