Jake McInteer

2 results

Cloud Strides at MongoDB.local Auckland

MongoDB.local Auckland was held earlier this week, and from the testimonies shared by our customers and partners to the announcements made by our global executives one thing is clear: there’s never been a greater need for strong cloud strategies to support data-driven businesses. Kiwi companies of all sizes, whether serving the New Zealand market or reaching a global audience, are avid users of cloud services and leading technologies such as MongoDB. Recent commitments from Microsoft, Amazon, and Google to build and run cloud regions in New Zealand will unlock a wide array of new use cases, especially when it comes to making it simpler to build and deploy cloud projects that need to meet growing data sovereignty requirements. Senior Account Executive Jake McInteer kicking off MongoDB.local Auckland Speaking at the event, Ben Cefalo (VP of Product Management, MongoDB) further announced MongoDB’s intention to fully support each of the cloud providers’ regions on MongoDB Atlas . This will bring a local presence for MongoDB’s leading cloud database platform which is already being used widely around the world and in New Zealand. It’s a great example of MongoDB’s commitment to meet customers where they are, and our multi-cloud strategy to be available in as many regions around the world as possible. MongoDB already counts a number of large enterprises as customers in New Zealand, as well as a growing number of public sector customers. One data sovereignty-sensitive customer is planning to deploy a major system in Microsoft's NZ North cloud region once it opens (slated for next year by Microsoft). They have been running development and test workloads out of Azure's New South Wales data centre, but are planning for Production go-live in New Zealand due to the sensitive nature of the data. MongoDB's advanced multi-cloud and multi-region features in Atlas have streamlined that department's development lifecycle and helped keep the project on track. MongoDB has been increasing its capabilities dedicated to the public sector, and we are expecting more exciting projects in this space in the coming years. Having recently completed the IRAP assessment in Australia also brings the added safety net for regulated industries in New Zealand that MongoDB is built for security, resilience and compliance. MongoDB’s New Zealand customer base also boasts some of New Zealand’s fastest-moving and best startups and is spread right across the country - with LuminPDF down south , Marsello in Wellington , and Figured in Auckland , just to name a few. At the MongoDB.local Auckland event, there were also presentations from a number of customers such as Canva, Cloudize, and LuminPDF. Michael Pearson (Software Engineer from Canva) presented his team’s challenges and triumphs operating at a massive scale - supporting 110 million monthly active users who collaborate across more than 15 billion designs, a scale very few companies and platforms have reached globally. It’s a great example of how MongoDB Atlas supports businesses as they scale to a global audience. Michael Pearson, Software Engineer from Canva We also heard from MongoDB partners such as Microsoft who presented their plans to open the New Zealand North Azure Cloud Region. This was a great example, alongside AWS and GCP’s commitments, of global tech platforms investing in the NZ Tech ecosystem - giant shoulders on which platforms like MongoDB Atlas can build upon. I also particularly enjoyed Erich Kuba (Founder at Cloudize) presenting on his experience trying to scale a spam filtering business on a relational database technology. His business got so big that it broke his database and ultimately his business failed - it was a pertinent reminder of the criticality of the work software engineers do and the importance that our technology choices take on. Overall MongoDB.local Auckland was an awesome event, showcasing the best of MongoDB and our community here in New Zealand - we can’t wait to see it back again next year. Interested in going to a MongoDB.local event? See if one is coming to a city near you !

August 17, 2023

Video: Canva's Lessons From Scaling MongoDB Atlas to 10 Billion Documents Across 100 Nodes

Running complex, global, and mission-critical infrastructure at scale is difficult, and anyone who has done it for any length of time usually has a few gnarly lessons to share. At MongoDB World in June 2022, we were lucky enough to feature someone who had done just that. Michael Pearson , software engineering team lead at Canva , gave a talk titled “10 Billion Documents: How Canva Scaled MongoDB to 100 Nodes.” I’ve had the pleasure of working alongside Pearson and his team for almost a year now, and his presentation focused on some of the massive challenges (and lessons) they’ve faced over the last two years as they have scaled into tens of terabytes of data and tens of billions of documents. I’m writing this blog to give a few highlights, but I’d recommend everyone check the original talk in full: A tricky problem For the uninitiated, Canva is a visual communication platform that empowers its users to design anything and publish anywhere. Or, as Pearson explained in his talk, “Canva is a really simple way to create beautiful designs and presentations.” Canva’s mission is to empower the world to design, and more than 85 million people in over 190 countries use the platform every month. As you can imagine, this presents a huge data challenge — and opportunity. Canva holds more than 10 billion designs and receives up to 30,000 document requests per second. The success of the platform comes down to providing a fantastic user experience every time, and to do that they need to present their customers with the right data at the right time. “This could be a really tricky problem for a database platform, particularly for a business based in Sydney with many users on the other side of the world,” said Pearson. MongoDB Atlas supports the Document Service, which enables opening, creating, updating, or deleting any design on Canva. The Document Service is critical for every single user — if the Document Service is down, then Canva’s users can’t design. But before we get too far into things, we should probably start with why Canva started using MongoDB in the first place. Flexibility to support rapidly changing requirements Michael Pearson, software engineering team lead at Canva. “Canva was launched to the world back in 2013, when MongoDB was very new to the scene,” explains Pearson. “I'm not sure if there were any other databases that would have been up for the challenge.” From those earliest days, MongoDB's flexible document model was the perfect fit for Canva's increasingly complex designs and document types. “The flexibility that MongoDB gave us in those early days was instrumental to our success. As the Canva platform evolved, we were throwing new schema and new features at it. MongoDB would just handle it.” Its continued innovation and problem-solving means MongoDB remains as valuable to us today as it was in 2012. Michael Pearson, software engineering team lead at Canva At the same time, it was essential that Canva’s engineering team was focused on building Canva, rather than time spent managing the data platform. With that in mind, Canva chose to run MongoDB as a service. After trying out multiple options, they went with MongoLabs and, in 2019, following MongoDB's acquisition of MongoLabs, Canva migrated onto MongoDB Atlas , running on AWS, where they remain to this day. Ten years of relative bliss “Before 2021, we had a very hands-off approach to how we used MongoDB,” said Pearson. “MongoDB just handled it. We didn't have to think about it at all." That's incredible, right? Think about it — for nearly a decade the team barely had to think about their data layer and could spend their time working on new features and making the actual product better for its millions of users around the world. It's what every developer wants. Eventually, though, Canva’s own success created certain challenges around scaling. With the stratospheric increase in growth, the load on the Document Service also continued to increase. MongoDB’s ability to scale horizontally through the use of sharding was critical to overcoming initial scale challenges, something that traditional database management systems would have struggled to achieve, said Pearson. With MongoDB, sharding is distributed or partitioned across multiple machines — useful when no single machine can handle large workloads. In due course, though, some attributes of Canva’s workload presented a new challenge. Said Pearson: “We were unique in that we have one cluster with one collection with a million chunks. Our documents are fairly large, given our product has evolved over the years and we put more and more stuff into our documents.” Or, Canva does many updates to relatively large documents, and by mid-2021 the surge in traffic was causing issues. “Our peak traffic caused three main problems: inability to run the balancer, latency issues, and a disk usage pretty much at capacity,” Pearson explained. “A really ineffective cache caused a really high write load to our cluster. This was causing downstream failures." Pearson discussed some of the tactical solutions the company took. “Disabling the balancer immediately brought us back to service, but now we knew that there was something wrong with that cluster and we couldn’t operate without the balancer,” said Pearson. “We also noticed that the number of chunks in our cluster had skyrocketed, from around 400,000 to just over a million.” Getting to the root of the problem The professional services team at MongoDB discovered that “metadata updates were causing anywhere from a one-second to five-minute stalls in the cluster.” Going from 400,000 chunks to a million chunks, at the rate of a minute of each change, was not optimal. There were three things to address with that cluster: reduce the number of chunks, reduce that disk contention, and reduce the size of documents. “With regard to reducing the number of chunks, we just took any contiguous chunks on a shard and merged them unconditionally,” said Pearson. “This was tooling built in collaboration with MongoDB.” After three months of merging chunks, Canva saw massive improvements in its cluster’s performance. A failure rate during reboot of around 4% dwindled to less than 1% during maintenance operations. Further, to address latency spikes and full-disk capacity, the team formulated a six-step plan to move from network-based storage volumes to locally attached disks. This has proved a huge success. “We were able to run the balancer. Our large spikes in latency were pretty much all gone, and our disk usage was almost at zero,” Pearson said. He continued: "The key takeaway for me is that sharding is great, but it's never a silver bullet. I don't think we would have caught these issues so quickly without such a thorough incident review process and such a close working relationship with MongoDB." What was learned? After presenting all of that information, Pearson closed out the presentation with a few key lessons. For anyone interested in running infrastructure at a massive scale, they are simple and worth taking note of: Take advantage of the flexible document model to accelerate your pace of development. Ensure chunks are distributed uniformly across the cluster in a consistent size. Maintain a thorough incident review process and include your trusted partners (such as MongoDB). Reliability is an essential part of Canva’s engineering practice, and prolonged service disruptions were particularly upsetting not only for engineers but for Canva’s global users. Pearson is glad to report that Canva has seen a turnaround in the number of incidents impacting its Document Service. This has freed the document team to shift focus back to shipping features and ensuring every user has a flawless experience using Canva. Interested in joining Canva as it pursues its mission to empower the world to design? Canva is looking for a software engineer to join its Core Data team. Want to take advantage of the flexible document model to accelerate your pace of development? Learn more about MongoDB Atlas .

August 2, 2022