Leaf in the Wild: Haymarket Migrates from MySQL to MongoDB, Achieving 8x Higher Platform Efficiency
October 27, 2015 | Updated: December 8, 2016
Leaf in the Wild posts highlight real world MongoDB deployments. Read other stories about how companies are using MongoDB for their mission-critical projects.
Classified Ads and New Analytics Hub Deployed to MongoDB for Faster Time to Market, Lower Costs and Embrace of Cloud
The media industry is undergoing a fundamental transformation as the move from print to web and mobile accelerates. I met with Peter Dignan, Platform Director at Haymarket Media Group to discuss how the use of modern database platforms enables media companies to embrace new channels and engage with a global audience.
Please start by telling us a little bit about your company. Haymarket Media Group is one of the world’s largest privately owned media companies, headquartered in London. We have over 70 brands delivered over print, web, events and mobile channels targeting the consumer, business, and professional sectors. Our properties are pretty diverse, including Asian Investor, Autocar, Autosport, Brand Republic, Campaign, Management Today, Marketing, Media Week, PistonHeads, The Clothes Show, and What Car.
I head the platform team at PistonHeads.com, the UK’s biggest online motoring community.
Please describe how you use MongoDB. We have multiple projects within PistonHeads.com. We started out with MongoDB powering our classified advertising product catalog back in 2011. We then deployed MongoDB as the backend database to our Single Sign On (SSO) systems that enable secure access to the site.
Beyond PistonHeads.com, many other Haymarket divisions use MongoDB today:
- Autosport has migrated from MySQL to MongoDB to handle traffic spikes around deadline times for its Grand Prix Predictor game. It is also using the ability to scale MongoDB quickly to handle mass results processing, which in turn has driven a better user experience, more clicks, and higher revenue.
- Our business to business properties are using MongoDB to store page layouts, working with Elasticsearch to assemble dynamic content.
- Our medical journals track advert views, and run aggregation queries to generate reports. This allow us to provide more reliable statistics for our advertisers, and in turn increase our CPM (cost per thousand impressions).
What were you using before MongoDB? We built our first classified ads system with ASP.NET web framework, backed by MySQL. As our ad volume grew, we started to hit scalability challenges, so we did what many relational database users do in this situation – we added a caching layer (Squid in our case) in an attempt to avoid queries ever having to hit the database. It was a sticking plaster approach. It temporarily brought us more headroom, but as traffic continued to grow we were soon hitting the limits again.
By 2011, we decided to explore our options by evaluating multiple NoSQL databases as part of a replatforming exercise. Not only did we want to increase scalability, we also wanted to personalize the user experience, and so we needed something much more sophisticated than a simple cache. During our evaluations, we found the memory mapped storage engine design of MongoDB gave us much higher performance, and when we coupled it with the Solr search engine, we were able to build a highly functional “pseudo”-caching layer. MySQL remained in place as the system of record.
We provisioned the infrastructure on Amazon Web Services and began load testing. And that gave us a nasty shock when we discovered we would need a minimum of 24 web servers to handle our expected load. MySQL was proving the bottleneck due to slow connections with the web tier. As a result, we postponed the launch and went back to the design, moving more ad content into MongoDB, including page elements, reference data, dropdowns, and static information.
As a result of replacing MySQL with MongoDB, we were able to reduce the size of the web tier from 24 to just 3 servers, significantly reducing our cost and operational complexity, while also boosting performance and reliability.
From that initial 2012 launch, we have since moved all our catalog data into MongoDB. Performance is much higher and we are less I/O bound than when we were using MySQL. We live by the mantra that “if you hit disk, you are dead!”.
Can you describe your MongoDB deployment? Today we run MongoDB across two replica sets with three nodes each, distributed across multiple Amazon Web Services availability zones for disaster resilience. Each replica set is also configured with a hidden member, against which we perform backups. Each MongoDB node is provisioned onto Linux-based AWS R3 xlarge instances with SSDs. Our applications connect to MongoDB via the .NET driver.
MongoDB stores around 650,000 adverts per month, in addition to associated engagement statistics. So far this year, that totals 50 million individual documents.
We use AWS CloudFormation to build our environments, and New Relic for monitoring.
What best practices can you share with the MongoDB community? Firstly, profile your queries to understand access patterns to the database, and design your data model from there. MongoDB offers a rich array of secondary indexes, which are incredibly powerful in supporting complex queries. But it isn’t magic – you need to invest the time to understand your queries and optimize your schema.
Secondly, use SSDs. We’ve had major performance gains from moving up to SSD-equipped instances.
Thirdly, go along to your local MongoDB User Group (MUG), especially as you are getting started with MongoDB. There is a wealth of expertise on hand, from both the organizers and the attendees. This really helped to accelerate our project in the early days.
How are you measuring the impact of MongoDB on your business? Across multiple dimensions:
- Time to market. We love the dynamic schema and JSON document model. It enables rapid development and feature evolution. We also love the performance we get out of the box with MongoDB. We’ve actually been able to reduce the amount of performance testing we do as we build our apps, which gives us more time to implement new features demanded by the business.
- Cost savings. Consolidating our web tier as a result of migrating from MySQL to MongoDB has driven significant infrastructure simplification.
- Embracing new platforms. As we have replatformed more of our infrastructure to the cloud, MongoDB has made the transition much simpler. It’s a better architectural fit to cloud platforms than relational databases ever were.
Do you have plans to use MongoDB for other applications?
MongoDB is now our default database. We wouldn’t think of using anything else.
What excites you about the MongoDB roadmap? MongoDB 3.2 looks pretty interesting. We are especially looking forward to the BI Connector demonstrated at MongoDB World. The business regularly requests custom, ad-hoc reports, so being able to connect a visualization tool to MongoDB and run standard SQL queries against our collections will help us to develop new reports faster, and with less development effort.
Peter, thanks for sharing your experiences with the MongoDB community.
If you are hitting the scalability limits of a relational database, read our RDBMS to MongoDB migration guide.
About the Author - Mat Keep
Mat is a director within the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.