Visualizing Performance Issues with MMS

This is a guest post by Albert Engelbrecht, Web App Developer at SuretyBonds.com.

“LMS is down.” The dreaded phrase came back again, meaning the office is twiddling its thumbs as the sales lead system becomes unresponsive. I open my SSH connection, run the MongoDB startup script and say “It’s back up, Brad.”

I get alert emails on IO usage on the servers nightly; the swap file is killing the disk. For some reason, our single MongoDB instance shuts down a few times a day. Logs pointed to the DB running out of memory, which was supported by my slowing database calls and huge IO overhead.

The obvious short-term answer to our out-of-memory errors was to upgrade to a bigger server instance for more swap space, however I knew this wouldn’t improve database performance of any one operation.

There is nothing more painful when trying to debug a hard problem as realizing that you can’t see anything - you are entirely blind. Mongotop and mongostat tools are a bit arcane and don’t give the big picture view. Ultimately my research for MongoDB monitoring led me to try MMS - MongoDB Management Service.

Suddenly there are graphs and charts galore. Everything’s there: non-mapped memory, database size, total memory footprint. I am in hog heaven. Not only can I get DB statistics but with a Munin agent running, it can report back hardware statistics as well. These aren’t your Grandpa’s Munin graphs either, MMS’s graphs are interactive, allowing me to set custom date ranges.

Another neat feature that I use is text and email alerts on a number of different events. Not only does MMS have graphs and alerts, but it also allows me to go through our slower queries and order them by response time. Instead of sifting through profiling data using queries inside the mongo shell, now I can easily view it all on one page. You know how many queries I rewrote? Probably a million, maybe two.

Take this for example:

Lead.find( {'$where' : "['fresh', 'new'].indexOf(this.status) > -1"} )
// This runs JavaScript to figure out if the status wasn't "fresh" or "new"
 
Lead.find( {"status" : { "$nin" : [ "fresh", "new" ] } } )
// Much more performant

This fun little query runs JavaScript to find every record in the collection to find the ones that don’t have two statuses. It runs every time someone hits the front page, even though the information is only shown to admins. Removing the query made a big difference in the non-mapped memory size. Take that interns, for you shall impact my DB performance no longer!

At the end of the day, upgrading the DB server and doing major refactoring worked out well. Our MongoDB instance hasn’t gone down since then, plus I find myself regularly coming back MMS to see how our newer queries are working out. It’s also good to know when we grow large enough to require sharding, MMS can seamlessly scale. If you happen to be wondering just how well your DB health is, I encourage you to check out MMS Monitoring. 10gen, er, MongoDB Inc. did a great job, plus it’s free!

When not waxing poetic about code smell, Albert works on web applications for SuretyBonds.com, a nationwide surety bonding agency.