Send Your MMS Alerts to the Right On-Call Engineer with PagerDuty
When you set up alerts in MongoDB Management Service , you now have the option to send them to PagerDuty . Like MMS, PagerDuty helps you increase application uptime and identify performance issues. With this new integration, you can set up multiple on-call schedules in PagerDuty to rotate your MMS alerts to different members of the team. MongoDB Management Service tracks dozens of different MongoDB-specific metrics, providing you with visibility into the performance of you MongoDB system. Using MMS, you can create alerts when particular statistics are out of range. While previously you were limited to setting email and mobile alerts to individual members of the team, you can now rotate those alerts around your team, giving everyone time off by easily scheduling on-call rotations. And with PagerDuty, there are automated escalations so you can be sure that you’ll never miss an alert. Looking for best practices on setting alerting thresholds in MMS? Check out our recent blog post and presentation on the top 5 recommended alerts to keep your deployment on track.
Backup vs. Replication - Why Do You Need Both?
Sometimes when MongoDB users evaluate Cloud Manager , they question whether they need a backup if they are already using replication. Doesn’t replication already protect their data sufficiently? The answer can be reduced to a single distinction: replication is for availability and backups are for disaster recovery. Availability is a measure of percentage of time an application is working and accessible to end users at full capacity. ^1 Over time, infrastructures always suffer predictable failures that can impact this availability. All drives eventually fail, often without warning. So do power supplies, network cards, and other individual machine components. Sometimes your server runs out of space on its disk, at which point you’ll need to take it offline so you can install a bigger one. These issues impact cloud infrastructures as well, although in different ways. For example, you might decide you need to re-provision a machine with twice as much RAM. All these issues would lead to your system being unavailable. To insulate infrastructures from these completely inevitable events, we build redundancy into our systems. If a component becomes unavailable, a standby system takes over immediately and transparently. The MongoDB feature that enables redundancy is called Replication . For those not familiar, when you organize MongoDB into a replica set, all members of the set stay in sync. All database writes go to the primary member of the replica set and are quickly synced to all secondary members. In the event a primary node fails, an election takes place between the remaining members and a new primary is chosen. Members can also be removed from the set for maintenance, and reintegrated seamlessly. Replication ensures that in the event of a node failure, your application will still be available. Failover occurs automatically within seconds and with few exceptions will be invisible to end users. As a result, individual machines can fail, or be serviced, without impacting your availability. Disaster recovery, in contrast, is about dealing with events that are far less predictable, and which by their nature defeat the protections offered by redundancy. ^2 These events fall into two categories. The first category is human error and the second is catastrophic failure. In the category of human error, you have application bugs, deliberate hacking and accidental deletion or corruption of all data on the primary node. In all those cases, the errors introduced to the primary will propagate automatically to all members of your replica set, often within seconds! Given that human error is just as guaranteed as disk failure, this alone is enough reason to have backups. In the category of catastrophic failure, this includes scenarios that permanently destroy all members of your replica set. If you keep all replica set servers in the same data center, a fire that destroys the servers would qualify. So would a disgruntled employee who goes out and deliberately deletes the data. For these lower probability events, you want a relatively inexpensive solution that is well isolated from your production system. The MongoDB feature that enables protection against disasters is Backup . All backup systems share the characteristic that they offer a snapshot of your system at a past moment in time. That this snapshot is forever frozen is a critical feature of the backup. Backup snapshots should be stored far from your production system, far both physically, logically and administratively. Restoring the backup snapshot rolls back the clock on the event that caused the loss of all data across your replica set. Taking backups of MongoDB systems is incredibly easy, so you have no excuse for not developing a comprehensive backup strategy. MongoDB Cloud Manager , the hosted backup solution from MongoDB, Inc., provides a number of extra benefits. Cloud Manager further reduces the window of data loss vulnerability by continually recording your replica set oplog and providing point-in-time recovery for the most recent 24 hours (in addition to keeping the point-in-time snapshots). This is the holy grail for backup: recovery to an arbitrary moment in time before the error occurred. Cloud Manager is also separate from Amazon Web Services, a popular place to put production data. That means that failures that affect AWS will be uncorrelated to failures that might affect the Cloud Manager infrastructure (and we have our own levels of redundancy). That’s good for any insurance policy. Restoring from backup (your own, or Cloud Manager) is not instantaneous. You incur some downtime. But because backup covers catastrophes, this is acceptable. If you are restoring from backup, your application is already so broken that the cost of the downtime is less than the cost of running continuously with bad or missing data. To sum up: you want backup to cover the events that you hope to never see happen. (If it’s happening often, you need a new process.) Replication, on the other hand, offers fault tolerance against events that are fairly common and must be addressed without the user being aware of the event. Long story short, you need backup and replication. They address a different set of risks. More Information Replication in MongoDB MongoDB Backup Strategies Backing up with Cloud Manager Read More about Disaster Recovery
Now Available: Two-Factor Authentication for MMS Backup via Google Authenticator
Today we are pleased to announce Google Authenticator as an alternative to SMS for two-factor authentication in MongoDB Management Service (MMS) . MMS requires two-factor authentication to ensure that only authorized users can access sensitive data and perform certain operations, such as pulling a restore. Users receive a code via SMS to verify their identity and enter that same code into MMS to be authenticated. The SMS-based authentication is powered by Twilio and works smoothly for the vast majority of users. There are, however, some edge cases. Some of our customers reside in areas where SMS service is unreliable or signals are weak. Offering Google Authenticator as an alternative gives those customers peace of mind that they will be able to access their backups quickly and securely in a data emergency. We built this feature based on your feedback and will continue to add the features and functionality that you want. Please keep the requests coming! To learn more about backing up MongoDB with MMS, visit mms.mongodb.com .
New Cloud Manager Backup Prices
Editor's note: This post was edited on June 23, 2015 to reflect the change from MongoDB Management Service to MongoDB Cloud Manager. Learn more here . Protecting your mission critical MongoDB data has never been easier or more affordable. MongoDB Cloud Manager provides continuous, online backup with point-in-time recovery of replica sets and consistent snapshots of sharded clusters with limited performance impact. The service is designed and managed by the team that created MongoDB, so you can rest assured that you always have a reliable backup. Lower Cloud Manager Backup Prices We are pleased to announce that we are simplifying prices for Cloud Manager. Here is the price breakdown: Monitoring and Automation $39 / server* per month or $379 / server* per year (save $89 per year) *A “server” means a single virtual machine or physical server running MongoDB that stores data. Config servers, pure arbiters, and servers only running mongos routers do not count. Try it free for 30 days Backup First 1GB per replica set free and then $2.50 / GB / month or $30 / GB / year prepaid Estimating Your Costs and Getting Started Curious how much backing up MongoDB with Cloud Manager will cost you? Create a Cloud Manager account and install Cloud Manager Monitoring on your deployment if you haven't done so already. Visit the Backup page within the Cloud Manager app. At the bottom of the page you will see a pricing calculator that will estimate your monthly cost. From the Backup page, you can get started with the service Should you have any questions about Cloud Manager Backup, feel free to contact us .
Introducing the MongoDB Management Service Activity Feed
Now it’s even easier to let MongoDB Management Service (MMS) alert you of performance issues. When you log into MMS to monitor your servers, you'll see that we’ve combined alerts and events into a single activity feed. We've also added several new features for an overall improved user experience. The Activity tab in MMS now provides you with a consolidated view of events as well as open alerts. From the Activity feed, you can acknowledge open alerts and see your recent events, such as host restarts, host additions or elections. You’ll also know if you have any open alerts when the bell icon is highlighted on the top of your screen. Within Alert Settings, you can create new alerts across a variety of metrics. You can also target alerts for specific hosts. For example, you can be alerted when lock percentage exceeds 20 for hosts with hostnames that contain “prod.” The alert can be sent to your entire MMS group as well as to a specified email address or phone number. Once created, you’ll see your new alert in Alert Settings. From that page, you can easily edit, clone, disable or delete the alert that you created. With these improvements in place, we expect MMS users to take greater advantage of the alerting capabilities in MMS. If you need guidance on the best alerts to keep your deployment on track, here are our top 5 recommended alerts to get you started or you can join our upcoming webinar on the same topic . And if you haven’t started monitoring MongoDB with MMS, you can create your free account at mms.mongodb.com . Special thanks Dan Spinosa at Shelby.tv and the other MMS users that helped test and provide feedback on the activity feed!
How We Implemented Freemium for MMS Backup Using Stripe
We recently launched a free usage tier for MMS Backup , our online backup service for MongoDB. We bill for backups on a monthly basis via credit card using Stripe , a MongoDB-powered service that makes it easy for developers to build applications that accept payments on the web. We wanted to offer a free usage tier so that customers could sample our backup service. To accomplish that, we decided to make any bill less than $5 complimentary. $5 would be enough to back up roughly 5GB with 100MB of oplog data. Once the bill exceeded $5, the customer would be charged the regular pay-as-you-go rate. Our backup service uses Stripe’s subscription billing capabilities. We couldn’t find any documentation or case studies on companies using Stripe to offer a freemium product. With some help from Stripe’s engineering team (thanks Michael Schade!), we were able to find a solution that satisfied our requirements. We investigated a few approaches before settling on using webhooks . At the completion of an invoice period, Stripe creates the invoice for the customer. We then receive an webhook for an "invoice created" event . When we receive this webhook notification from Stripe, we inspect the invoice. If the total bill is less than $5, we apply a negative invoice item in the amount of the invoice. This approach enabled us to quickly launch our free usage tier and we hope that it will be helpful to others looking to do the same. For more information, check out Stripe for your next app or start backing up your MongoDB deployment with MongoDB Management Service .
Do you monitor MongoDB with MMS? Tell your story for the chance to win a Nexus 5.
Are you one of the 20,000 MongoDB users monitoring their servers with MongoDB Management Service ? If so, you’ve probably gained valuable experience and expertise on the critical metrics to watch, performance tuning and operations best practices. We want you to share those experiences with the broader MongoDB community. To that end, we’re organizing a blogging contest this month to encourage you to tell those stories. Here’s how you can participate. Rules Publish a blog post of at least 250 words about MMS, with the goal of educating the MongoDB community on monitoring best practices Write for a technical audience - our readers are engineers, DBAs, SysAdmins, and DevOps professionals Submit your post to email@example.com by November 4, 2013 Topic Ideas How you debugged a performance issue using MMS Explain specific metrics or graphs in MMS and what you look for Describe the alerts you have set up in MMS and how you determined the thresholds Review your MMS dashboard and why you selected the charts on it The rewards Since MMS users love to measure everything, the first 20 entrants will receive a Fitbit Flex wireless activity and sleep wristband All entrants will receive a MongoDB swag pack that includes a MongoDB coffee mug, stickers and other goodies Great posts will be tweeted from @mongodb (over 45,000 followers) We will choose one winner to receive a Nexus 5, which has no shortage of very cool sensors and measuring devices Blogging Tips & Tricks Check out the HTML5Rock Writing Tips for suggestions on writing a great technical blog post Don’t stress about length. While most posts are in the 500 word range, we’ve seen great blogs that were short and sweet and others that were quite lengthy. Have something to say, but don’t have a blog? Try posting on your company blog or use a simple blogging platform like Medium or Tumblr . Feel free to post any questions in the comments section. We look forward to seeing what you come up with!
Start Backing Up MongoDB for Free -- Free Tier in MMS Backup Now Available
We’re pleased to introduce a free usage tier to MMS Backup , our online backup service designed for MongoDB . The free tier will make it easier for new users to get started with the service. MMS offers point-in-time recovery for replica sets and consistent snapshots for sharded systems. It is also one of the lowest overhead ways to backup MongoDB, continually reading the oplog and transferring the data encrypted to the backup service. The free tier will be available to new users during their first 12 months using MMS Backup. In that period, any monthly bill under $5 is free. What does $5 get you? It’s roughly enough to back up a 2GB database with 500MB of oplog data churn per month. With the free tier in place, we expect MongoDB users to take advantage of this offering in a variety of ways. The free tier can be used to sample the service for a larger, mission critical application. You can also leverage MMS during development before your dataset grows. And the free tier is perfect for side projects that you want to safeguard but don’t have the time to invest in building a backup strategy. Getting started with MMS Backup is easy: Create an account at mms.mongodb.com Install monitoring on your deployment (monitoring is free) Go to the Backup tab to accept legal terms and enter credit card details Install an agent in your environment and start an initial sync to our datacenters Rest assured knowing that your backups are reliable and safe For more information on MMS, visit mms.mongodb.com . For specific questions about the free tier and eligibility, please visit the FAQ .
Installing Cloud Manager Monitoring in Less than 5 Minutes
Editor's note: This post was edited on June 24, 2015 to reflect the relaunch of MongoDB Management Service to MongoDB Cloud Manager. Learn more here . MongoDB Cloud Manager provides monitoring with metrics specifically tuned for your MongoDB deployment. In this short tutorial, we’ll walk you through how you can set up Monitoring for your existing deployment quickly. Step 1: Create a Cloud Manager account and Begin Setup Go to cloud.mongodb.com and click the “Sign up for free” button. Enter in your details, and click “Create Account”. Choose a group name. Click “Begin Setup” and choose “Manage Existing Deployment”. Step 2: Install Automation Agent Follow the instructions to install the Automation Agent. Make sure to install the Automation Agent on a server that can connect to your entire MongoDB deployment. One the agent is running, click “Verify Agent”. When the agents are verified, you can go on to the next step. Step 3: Install Monitoring Agent The Monitoring and Backup Agents will next be downloaded to your machine and started. The Monitoring Agent will be used to collect health and performance metrics from your MongoDB deployment. A Backup Agent will also be installed, but will lie dormant until you choose to enable our Backup feature. Once both agents are running and are successfully verified, click continue to proceed. Step 4: Import your Deployment for Monitoring Enter your host’s details such as hostname and port that a primary, secondary, or mongos process is running on, and optionally auth mechanism. Click continue, and we will wait for the ping from your Monitoring Agent. Once we get a ping back from your host your deployment should be found.,Click continue and review that the processes that you want to monitor are shown. Step 5 (Optional): Add Automation on Each Remaining Server You've already connected your database and set it up to be monitored by MongoDB Cloud Manager. You can take take your Cloud Manager integration a step further and manage your entire deployment with automation. To do this, you will be required to install an Automation Agent on each remaining server in your deployment. If you don’t wish to do this, simply click “No, Just Monitor.” Step 6: Enjoy Cloud Manager You will now be placed into deployment view and can see your new configuration! If you are interested in importing your existing deployment to Automation, check out our blog post on Import Existing for Automation . If you are interested in using Backup, visit the docs on setting up Backup . For detailed instructions and installation troubleshooting, visit the docs . Ready to get started? Create a free Cloud Manager account now: Try Cloud Manager Today
How PublishThis Tames Its Big Data Workload with MongoDB Management Service (MMS)
Database efficiency matters to PublishThis . Providing the most up-to-date content for thousands of always-on feeds to customers like Overstock.com, Sporting News, SAP, and Cox Media is critical, and to do that requires careful monitoring and tuning of the performance of their MongoDB cluster. I recently spoke with Dr. Chris Lane, the Director of Data Services at PublishThis, about how he optimizes his MongoDB deployment. PublishThis’ platform crawls the internet for the latest content and scores it based on its patent-pending relevancy engine so their customers can scale their real-time publishing efforts. PublishThis’ crawler indexes content from 300,000 of the web’s best sources daily, pulling data from automated feeds, social media, and other sources on topics from sports to politics to entertainment to technology . The crawler collects huge amounts of semi-structured data from the web, which is then stored in MongoDB. Finding Performance Bottlenecks... Before Customers Do To optimize the performance of their database cluster, Chris uses MongoDB Management Service (MMS) . MMS Monitoring provides valuable insight into their infrastructure as he manages a multi-terabyte deployment. The PublishThis crawler database is a sharded system, with data spread across multiple servers. Earlier in their platform development, Chris would run the balancer from 00:00 to 06:00 every day. With MMS monitoring the system, the team noticed that the lock percent and queues went up on the system when the balancer kicked in. With that insight, Chris checked the logs and noticed the moveChunk command was taking a very long time -- sometimes as long as 9 hours! Chris realized that he needed a faster storage system like a SAN capable of handling large numbers of IOPS. As a quick fix while waiting for the delivery of the SAN, Chris reduced their chunk size. This allowed the balancer to make progress while it ran and minimized overall impact on the system by breaking the work into smaller, more digestible chunks. This improved performance significantly. Once the SAN arrived, he continued to run with the reduced chunk size based on the excellent performance he observed in MMS. “MMS helps PublishThis deliver the web’s best content at real-time scale by making it easy to manage processing massive, always-on volume with our industry-leading relevancy engine.” – Dr. Chris Lane, Director of Data Services - PublishThis Monitoring Tips From Chris New to monitoring with MMS ? Chris offers the following advice to those getting started with MMS: Set up simple dashboards with just a few key metrics. MMS provides a lot of data, so create a dashboards with the actionable metrics you understand. The 'Events' tab is very helpful. You can see everything that's happened with each replica set in one place, such as new hosts or elections. Run at least two agents on VMs on physically different hosts. If something happens with one VM (or, worst case, the host), you won't lose monitoring. Haven’t started monitoring with MMS yet? It’s free. Create your account at mms.mongodb.com -- you’ll be able to visualize performance of your MongoDB system within minutes.
10 Things You Didn’t Know About MMS
Editor's note: This post was edited on June 23, 2015 to reflect the change from MongoDB Management Service to MongoDB Cloud Manager. Learn more here . Whether you are new to MongoDB Cloud Manager or an experienced user, there are probably some features that you may not have explored. Here are a few MMS tips, tricks and new features to help you get the most out of MMS. 10. Highlight Chart to Zoom In; Double Click to Reset Get a closer look at any chart by highlighting the section of the chart data that you want to view. You can highlight horizontally or vertically to zoom in. To reset your view, double click on the graph. 9. Read only users This is a recent addition to the Cloud Manager feature set. You can now create users with read only access to your Cloud Manager group. 8. Database versions turn red when you need to upgrade On an older version of MongoDB or one of the agents? Cloud Manager will let you know that you’re not running the most up-to-date version by making your MongoDB or agent version number red. 7. Munin plugin Cloud Manager provides support for collecting and charting hardware statistics collected with Munin by installing the munin-node package . 6. Profiler Data You can enable transmission of database profiling results to Cloud Manager by clicking the clock icon on the Hosts page. 5. Exclude collections from your backup You can tune your backup costs by excluding non-critical databases and collections from your backup. Click on the pencil icon next to the name of your replica set and enter the names of the databases or collections that you wish to exclude. 4. Custom Alerts via Email or SMS You can set up alerts around custom metrics (e.g. lock percent above 30% or replication lag greater than 240 seconds) and have them delivered via email or text message. You can also set the delivery requirements (e.g. send every 20 minutes or delay 5 minutes). 3. Export or Share Charts There are lots of ways to export data to share with the members of your team: You can create a permalink, email the chart, or export to PDF or PNG. 2. Backup Pricing Calculator If you haven’t decided whether to use Cloud Manager Backup and want to get an idea of what your costs might be, you can use the pricing calculator to estimate a monthly price. Not seeing all of your replica sets? Click on the “Estimate Costs” button for a full calculator. 1. Monitoring is Free Monitoring with Cloud Manager is completely free. You’re minutes away from seeing the performance of your MongoDB cluster. Get started at cloud.mongodb.com .
Slides and Recording from Performance Tuning and Monitoring with MMS Webinar
Nicholas Tang, who leads 10gen’s North American support team, recently presented on Performance Tuning and Monitoring Using MongoDB Management Service (MMS) in a live webcast. In the session, Nicholas explained what MMS is, why you should use it, how you can set it up and how to use it for performance tuning. He then gave some examples of real-world scenarios where he worked with customers to use MMS to diagnose and debug performance issues using MMS. The video and slides are now available online below. To start monitoring your MongoDB deployment today, sign up for a free MMS account at mms.10gen.com .