MongoQP: MongoDB Slow Query Profiler

Jeremy Mikola
December 5, 2012 | Updated: February 17, 2023
#Releases

Two times a year 10gen’s Drivers and Innovations team gather together for a face to face meeting to work together and setting goals for the upcoming six months. This year the team broke up into teams for an evening hackathon. MongoQP, a query profiler, was one of the hacks presented by Jeremy Mikola, PHP Evangelist at 10gen.

Logging slow queries is essential for any database application, and MongoDB makes doing so relatively painless with its database profiler. Unfortunately, making sense of the system.profile collection and tying its contents back to your application requires a bit more effort. The heart of mongoqp (Mongo Query Profiler) is a bit of map/reduce JS that aggregates those queries by their BSON skeleton (i.e. keys preserved, but values removed). With queries reduced to their bare structure, any of their statistics can be aggregated, such as average query time, index scans, counts, etc.

As a fan of Genghis, a single-file MongoDB admin app, I originally intended to contribute a new UI with the profiler results, but one night was not enough time to wrap my head around Backbone.js and develop the query aggregation. Instead, I whipped up a quick frontend using the Silex PHP micro-framework. But with the hack day deadline no longer looming, there should be plenty of time to get this functionality ported over to Genghis. Additionally, the map/reduce JS may also show up in Tyler Brock’s mongo-hacker shell enhancement package.

While presenting mongoqp to my co-workers, I also learned about Dan Crosta’s professor, which already provides many of the features I hoped to implement, such as incremental data collection. I think there is still a benefit to developing the JS innards of mongoqp and getting its functionality ported over to other projects, but I would definitely encourage you to check out professor if you’d like a stand-alone query profile viewer.

Contributions welcome through Github.

← Previous

Lessons Learnt Building mongoengine

Recently, I attended both Pycon UK and Pycon Ireland to talk about the lessons I have learnt while maintaining mongoengine . The conferences were both excellent and surprisingly different. Pycon UK had quite an “unconference” feel, with some exciting sprint rooms - I wish I had more time as by all reports the educational jam was inspirational. Pycon Ireland in contrast felt more slick with booths from DemonWare, Amazon and Facebook. If you can, I’d advise going to both conferences as they really complement each other. Going to python conferences its great to see ever increasing numbers of developers adopting MongoDB , its such a good fit for python! Also, I can’t tell you how great it is to have people come up and thank you and 10gen for creating such a great product. As a 10gen engineer one of our core goals is to build upon the power and ease of use of mongoDB. Back to the matter at hand, here’s a quick overview of the talk. Please check out the slides for all the detail I glazed over here and for the extra keen you can even watch the video as well! Lessons learnt building mongoengine Dive in! If a project is on Github its a license to dive in. Don’t be afraid to, but do be a good citizen and feedback changes to the root repository so others can benefit. Metaclasses Turns out they aren’t magical! In fact how we use them in mongoengine allows us to change the class and add metadata at declaration time. This really useful trick makes creating instances of classes quick and easy. Straying from the path Mongoengine uses a Django style query syntax. This is bad in that its different from the native pymongo syntax. But its also good as you we add some sweet syntactic sugar. Not all ideas are good In mongoDB the native save operation overwrites the whole document. This catches new users out. So in mongonengine save tracks changes to the document and uses atomic updates . The jury is out whether or not this is a good thing™. Managing a community I’ve never done it before and its not always easy but it has been worth it. My top tip is to check out the github network graph and ask committers to forks to add pull requests back upstream. In cases where people aren’t willing to do so add a ticket with a link to the feature you’ve spotted downstream. An added benefit of bubbling up these useful features is that it will bring more contributors into the core project. Don’t be afraid to ask for help Cue me asking for help on any aspect of mongoengine, be it the core, the website or framework support! Please get in contact Slides The video is available on the PyCon Ireland channel .

November 29, 2012

Next →

Welcoming the Grainite Team to MongoDB: Accelerating Atlas Stream Processing

Tens of thousands of customers and millions of developers rely on MongoDB Atlas to run business-critical applications because of its flexible document data model and ability to work with virtually any type of data using a single, intuitive developer data platform that can run on all major cloud providers. Earlier this year, we announced that we would enable developers to apply the benefits of the document model and a unified interface to streaming data with Atlas Stream Processing . In a short amount of time, we have seen a strong response from developers eager to use this capability to simplify building event-driven applications. To accelerate the development of our streaming offering, we are thrilled to announce that the team behind Grainite — a streaming application platform, backed by Sequoia Capital and Menlo Ventures, that makes it easy to read, process, store, and query events in real-time — has now joined MongoDB. Grainite was founded by world-class technologists Ashish Kumar, who spent over ten years running structured storage products and other high-scale infrastructure projects at Google, and Abhishek Chauhan, who previously served as CTO of Cloud Networking at Citrix. Ashish and Abhishek bring with them a team of talented engineers to MongoDB who will help accelerate the development of Atlas Stream Processing and the release of new capabilities that will make it even easier for customers to process and analyze streaming data for real-time application experiences. “We’re excited to be joining a company that shares our vision for making it effortless to build event-driven applications,” said Kumar. “We’re looking forward to continuing to develop a world-class streaming platform that will make it easier for developers to incorporate real-time data into their applications and becoming part of an organization that can significantly amplify the reach and impact of event-driven applications for developers and end users.” We look forward to welcoming the Grainite team to MongoDB and sharing updates on Atlas Stream Processing and our broader streaming roadmap in the near future.

September 28, 2023