Visualizing Performance Issues with MMS

MongoDB
September 20, 2013
#Releases

This is a guest post by Albert Engelbrecht, Web App Developer at SuretyBonds.com.

“LMS is down.” The dreaded phrase came back again, meaning the office is twiddling its thumbs as the sales lead system becomes unresponsive. I open my SSH connection, run the MongoDB startup script and say “It’s back up, Brad.”

I get alert emails on IO usage on the servers nightly; the swap file is killing the disk. For some reason, our single MongoDB instance shuts down a few times a day. Logs pointed to the DB running out of memory, which was supported by my slowing database calls and huge IO overhead.

The obvious short-term answer to our out-of-memory errors was to upgrade to a bigger server instance for more swap space, however I knew this wouldn’t improve database performance of any one operation.

There is nothing more painful when trying to debug a hard problem as realizing that you can’t see anything - you are entirely blind. Mongotop and mongostat tools are a bit arcane and don’t give the big picture view. Ultimately my research for MongoDB monitoring led me to try MMS - MongoDB Management Service.

Suddenly there are graphs and charts galore. Everything’s there: non-mapped memory, database size, total memory footprint. I am in hog heaven. Not only can I get DB statistics but with a Munin agent running, it can report back hardware statistics as well. These aren’t your Grandpa’s Munin graphs either, MMS’s graphs are interactive, allowing me to set custom date ranges.

Another neat feature that I use is text and email alerts on a number of different events. Not only does MMS have graphs and alerts, but it also allows me to go through our slower queries and order them by response time. Instead of sifting through profiling data using queries inside the mongo shell, now I can easily view it all on one page. You know how many queries I rewrote? Probably a million, maybe two.

Take this for example:

Lead.find( {'$where' : "['fresh', 'new'].indexOf(this.status) > -1"} )
// This runs JavaScript to figure out if the status wasn't "fresh" or "new"
 
Lead.find( {"status" : { "$nin" : [ "fresh", "new" ] } } )
// Much more performant

This fun little query runs JavaScript to find every record in the collection to find the ones that don’t have two statuses. It runs every time someone hits the front page, even though the information is only shown to admins. Removing the query made a big difference in the non-mapped memory size. Take that interns, for you shall impact my DB performance no longer!

At the end of the day, upgrading the DB server and doing major refactoring worked out well. Our MongoDB instance hasn’t gone down since then, plus I find myself regularly coming back MMS to see how our newer queries are working out. It’s also good to know when we grow large enough to require sharding, MMS can seamlessly scale. If you happen to be wondering just how well your DB health is, I encourage you to check out MMS Monitoring. 10gen, er, MongoDB Inc. did a great job, plus it’s free!

When not waxing poetic about code smell, Albert works on web applications for SuretyBonds.com, a nationwide surety bonding agency.

← Previous

Installing Cloud Manager Monitoring in Less than 5 Minutes

Editor's note: This post was edited on June 24, 2015 to reflect the relaunch of MongoDB Management Service to MongoDB Cloud Manager. Learn more here . MongoDB Cloud Manager provides monitoring with metrics specifically tuned for your MongoDB deployment. In this short tutorial, we’ll walk you through how you can set up Monitoring for your existing deployment quickly. Step 1: Create a Cloud Manager account and Begin Setup Go to cloud.mongodb.com and click the “Sign up for free” button. Enter in your details, and click “Create Account”. Choose a group name. Click “Begin Setup” and choose “Manage Existing Deployment”. Step 2: Install Automation Agent Follow the instructions to install the Automation Agent. Make sure to install the Automation Agent on a server that can connect to your entire MongoDB deployment. One the agent is running, click “Verify Agent”. When the agents are verified, you can go on to the next step. Step 3: Install Monitoring Agent The Monitoring and Backup Agents will next be downloaded to your machine and started. The Monitoring Agent will be used to collect health and performance metrics from your MongoDB deployment. A Backup Agent will also be installed, but will lie dormant until you choose to enable our Backup feature. Once both agents are running and are successfully verified, click continue to proceed. Step 4: Import your Deployment for Monitoring Enter your host’s details such as hostname and port that a primary, secondary, or mongos process is running on, and optionally auth mechanism. Click continue, and we will wait for the ping from your Monitoring Agent. Once we get a ping back from your host your deployment should be found.,Click continue and review that the processes that you want to monitor are shown. Step 5 (Optional): Add Automation on Each Remaining Server You've already connected your database and set it up to be monitored by MongoDB Cloud Manager. You can take take your Cloud Manager integration a step further and manage your entire deployment with automation. To do this, you will be required to install an Automation Agent on each remaining server in your deployment. If you don’t wish to do this, simply click “No, Just Monitor.” Step 6: Enjoy Cloud Manager You will now be placed into deployment view and can see your new configuration! If you are interested in importing your existing deployment to Automation, check out our blog post on Import Existing for Automation . If you are interested in using Backup, visit the docs on setting up Backup . For detailed instructions and installation troubleshooting, visit the docs . Ready to get started? Create a free Cloud Manager account now: Try Cloud Manager Today

September 19, 2013

Next →

Data Resilience with MongoDB Atlas

Data is the central currency in today's digital economy. Studies have shown that 43% of companies that experience major data loss incidents are unable to resume business operations. A range of scenarios can lead to data loss, yet within the realm of database technology, they typically fall under three main categories: catastrophic technical malfunctions, human error, and cyber attacks. A data loss event due to a catastrophic breakdown, human error, or cyber attack is not a matter of if, but a matter of when it will occur. Hence, businesses need to focus on how to avoid and minimize the effects as much as possible. Failure to effectively address these risks can lead to extended periods of downtime of a few hours or even a few weeks following an incident. The average cost of cyberattacks is a surprising $4.45 million, with some attacks costing in the hundreds of millions. Reputational harm is harder to quantify but no doubt real and substantial. The specific industry you're in might be subject to regulatory frameworks designed to counter cyber attacks. Businesses that are subject to regulatory regimes must maintain compliance with these requirements. This can determine the configuration of your disaster recovery approach. In this blog post, we'll explain the key disaster recovery (DR) capabilities available with MongoDB Atlas . We'll also cover the core responsibilities and strategies for data resilience including remediation, and recovery objectives (RTO/RPO). Planning for data resilience in Atlas Data resilience is not a one-size-fits-all proposition, which is why we offer a range of choices in Atlas for a comprehensive strategy. Our sensible defaults ensure you're automatically safeguarded, while also offering a variety of choices to precisely align with the needs of each individual application. When formulating a disaster recovery plan, organizations commonly begin by assessing their recovery point objective (RPO) and recovery time objective (RTO). The RPO specifies the amount of data the business can tolerate losing during an incident, while the RTO indicates the speed of recovery. Since not all data carries the same urgency, analyzing the RPO and RTO on a per-application basis is important. For instance, critical customer data might have specific demands compared to clickstream analytics. The criteria for RTO, RPO, and the length of time you need to retain backups will influence the financial and performance implications of maintaining backups. With MongoDB Atlas, we provide standard protective measures by default, with customizable options for tailoring protection to the service level agreements specified by the RPO and RTO in your DR plan. These are enhanced by additional features that can be leveraged to achieve greater levels of availability and durability for your most vital tasks. These features can be grouped into two main categories: prevention and recovery. Backup, granular recovery, and resilience There are many built-in features that are designed to prevent disasters from ever happening in the first place. Some key features and capabilities that enable a comprehensive prevention strategy include multi-region and multi-cloud clusters , encryption at rest , Queryable Encryption , cluster termination safeguards , backup compliance protocols , and the capability to test resilience . (We will discuss the features in-depth in part two of this series.) While prevention might satisfy the resilience needs of certain applications, different applications may demand greater resilience against failures based on the business requirements of data protection and disaster recovery. MongoDB provides comprehensive management of data backups, including the geographic distribution of backups across multiple regions, and the ability to prevent backups from being deleted, all through an automated retention schedule. Recovery capabilities are aimed at supporting RTO and minimizing data loss and include continuous cloud backups with point-in-time recovery. Atlas cloud backups utilize the native snapshot feature of your cluster's cloud service provider, ensuring backup storage is kept separate from your MongoDB Atlas instances. Backups are essentially snapshots that capture the condition of your database cluster at a specific moment. They serve as a safeguard in case data is lost or becomes corrupted. For M10+ clusters, you have the option of utilizing Atlas Cloud Backups, which leverage the cluster's cloud service provider for storing backups in a localized manner. Atlas comes with strong default backup retention of 12 months out of the box. You also have the option to customize snapshot and retention schedules, including the time of day for snapshots, the frequency at which snapshots are taken over time, and retention duration. Another important feature is continuous cloud backup with point-in-time recovery, which enables you to restore data to the moment just before any incident or disruption, such as a cyber attack. To ensure your backups are regionally redundant and you can still restore even if the primary region that your backups are in is down, MongoDB Atlas offers the ability to copy these critical backups, with the point-in-time data, to any secondary region available from your cloud provider in Atlas. For the most stringent regulations, or for businesses that want to ensure backups are available even after a bad actor or cyber attack, MongoDB Atlas can ensure that no user, regardless of role, can ever delete a backup before a predefined protected retention period with the Backup Compliance Policy. Whatever your regulatory obligations or business needs are, MongoDB Atlas provides the flexibility to tailor your backup settings for requirements. Crucially, this ensures you can recover quickly, minimizing data loss and meeting your RPO in the event of a disaster recovery scenario. When properly configured, testing has shown that Atlas can quickly recover to the exact timestamp before a disaster or failure event, giving you a zero-minute RPO and RTO of less than 15 minutes when utilizing optimized restores. Recovery times can vary due to cloud provider disk warming and which point in time you are restoring to. So, it is important to also test this regularly. This means that regardless of your regulatory or business requirements, MongoDB Atlas allows you to configure your backups to ensure that you can meet your recovery requirements and, most importantly, recover with precision and speed to ensure that your data loss is minimal and your recovery point objectives are met should you experience a recovery event. Conclusion As regulations and business needs continue to evolve, and cyber-attacks become more sophisticated and varied, creating and implementing a data resilience strategy can be simple and manageable. MongoDB Atlas comes equipped with built-in measures that deliver robust data resilience at the database layer, ensuring your ability to both avoid incidents and promptly restore operations with minimal data loss if an incident does occur. Furthermore, setting up and overseeing additional advanced data resilience features is straightforward, with automation driven by a pre-configured policy that operates seamlessly at any scale. This streamlined approach supports compliance without the need for manual interventions, all within the MongoDB Atlas platform. For more information on the data resilience and disaster recovery features in MongoDB Atlas, download the Data Resilience Strategy with MongoDB Atlas whitepaper. To get started on Atlas today, we invite you to launch a free tier today .

October 3, 2023