On Thursday MongoDB core committer Eliot Horowitz presented to the New York MongoDB User Group on the latest features in v2.0. The event was hosted by Sailthru, a MongoDB-powered startup doing intelligent email marketing. The meetup was announced Monday night and within a day was oversubscribed. After the presentation, we all went out for drinks to celebrate the release.
The NY MUG has over 1,000 members and meets monthly. There are also MongoDB user groups in San Francisco, Washington DC, London, Boston, Japan, and more. And if there isn’t a MongoDB meetup in your city, we’re happy to support you if you would like to start one.
In case you missed the meetup, video and photos are posted below. Enjoy!
Cache Reheating - Not to be Ignored
An important aspect to keep in mind with databases is the cost of cache reheating after a server restart. Consider the following diagram which shows several cache servers (e.g., memcached) in front of a database server. This sort of setup is common and can work quite well when appropriate; it removes read load from the database and allows more RAM to be utilized for scaling (when the database doesn’t scale horizontally). But what happens if all the cache servers restart at the same time, say, on a power glitch in a data center? We then have a cache reheating scenario. After the bounce the full load of all read requests will hit the database server (for a while) given the caches are empty. The server won’t be able to handle it (that’s why the cache servers were there in the first place). Now if the reheat time is short, this is not a big problem : we did go down after all. But if reheat takes a long time, it’s a big problem. Imagine 20 cache servers with 64GB RAM each. 1.2 terabytes of data must be queried from the database to be fully reheated! Even without a cache server, the same issue exists for almost all databases. Imagine a server restart on a system with 64GB RAM. If loading 100MB/sec, reheat will take 10 minutes. However, if the queries are coming in randomly, it could take much much longer – only with sequential I/O can we get anywhere near that speed. One could just sequentially read data in and fill the cache. However for databases much larger than RAM, loading in the right portion is difficult. “Hot” data could be at very different locations in the terabyte(s) of on disk data. A very nice attribute of the MongoDB storage engine is its use of memory-mapped files. In this model the cache is the operating system’s file system cache. Restart the mongod process, and there is no reheat issue at all. Some databases use their own page cache which causes a reheat scenario even on just a process restart. Of course on a full server reboot MongoDB must reheat too. A few points to keep in mind: Think about reheating and how you will operationally handle it if a scenario involving it occurs. For schedules OS maintenance restart the server during off peak hours to minimize the load during reheat. Restarting the mongod process is safe with respect to reheating. Remounting a volume likely loses all file system cache info for the volume. On a server restart, copy datafiles to /dev/null to force reheating to be sequential and thus much faster. This can be done even if the mongod process is already running. If the database is larger than RAM, copy only the newest datafiles (ones with the highest numbers); while this isn’t perfect, the latest files likely contain the largest percentage of frequently used data.
MongoDB and AWS: How a decade-old collaboration got even better in 2022
Developers select MongoDB because it makes building with data for almost any class of application easy and fast for them. They select Amazon Web Services (AWS) because it offers a comprehensive and broadly adopted cloud platform, offering more than 200 fully featured services. Bringing together MongoDB Atlas on AWS helps developers build and ship higher quality applications faster and scale them further. MongoDB has collaborated with AWS for close to a decade now, but 2022 has seen dramatic growth in both the quantity and quality of our joint activities, resulting in a strategic collaboration agreement announced earlier this year. Our collaboration spans joint product engineering and integration so MongoDB Atlas is a first-party service on AWS, and also extends to making it easy for customers to procure MongoDB Atlas on AWS. In 2022, we have worked more closely together than ever before. In this post, we'll cover what we've achieved, and how our customers benefit. If at any point you want to stop reading about the partnership and experience it in action, we invite you to get started for free with MongoDB's fully managed, pay-as-you-go listing on the AWS Marketplace . Delivering an outstanding customer experience Since re:Invent 2021, MongoDB and AWS have jointly seen an explosion in customer success, with MongoDB for Startups becoming one of the most widely used offerings in the AWS Activate program after we launched in July. And, since launching in the AWS Marketplace with pay-as-you-go pricing in December 2021, MongoDB Atlas has become one of the most popular self-service listings, with well over 1,000 customers. More broadly, we've seen our AWS Marketplace business show triple-digit growth through significant, mutual investments across engineering, sales, and marketing. We've also found great success working with AWS' Workload Migration and Proof of Concept programs, helping many new customers accelerate their migration to MongoDB Atlas on AWS over the past 12 months. Additionally, while MongoDB works closely with AWS across the globe, we devoted increased attention to Europe this past year, resulting in a considerable increase in customer adoption. As a result, AWS named us their AWS Marketplace Partner of the Year - EMEA in November 2022. One way that we've helped to accelerate such customer success is by making it easier to procure MongoDB Atlas on AWS. Over the past year, MongoDB and AWS have significantly simplified the purchasing experience for customers. We did this across a few key areas. One thing customers love about buying through AWS Marketplace is how seamless it makes the purchasing experience. However, historically this has been slowed somewhat for MongoDB customers by the need to agree to separate legal terms. Starting in November 2022, however, all Atlas on AWS customers purchasing through the AWS Marketplace Self Service listing use AWS Marketplace’s Standard Contract for Marketplace (SCMP) terms and conditions rather than MongoDB Cloud Terms of Service, thereby further reducing friction to getting productive, faster, with MongoDB. Close product collaboration Behind these improvements to our joint purchasing experience were significant improvements to how MongoDB Atlas integrates with key AWS services. MongoDB has long worked seamlessly with core AWS services such as Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3), and more recently has collaborated with AWS to ensure tight integration with AWS container services like Amazon Elastic Kubernetes Service (Amazon EKS) and Amazon Elastic Container Service (Amazon ECS), AWS serverless technologies like AWS Lambda, Amazon Eventbridge, and AWS Fargate; and edge computing services like AWS Wavelength . Over the past year, however, we've delved more deeply into AWS machine learning services (Amazon Comprehend, Amazon Kendra, Amazon Lex, etc.), AWS AppSync, Amazon Forecast, AWS Elastic Beanstalk, and more. In addition to direct integrations with AWS services, we made it simpler for customers to use MongoDB with important joint partners such as Datadog, Databricks, and Confluent. For Datadog, we improved MongoDB Atlas App Service to support forwarding logs on AWS to Datadog, thereby improving observability through real-time log analytics. With Databricks, we announced MongoDB as a data source within a Databricks notebook, thereby offering data practitioners an easier, more curated experience for connecting Databricks to MongoDB Atlas data. And with Confluent, we strengthened our integrations to help developers easily build robust, reactive data pipelines that stream events between applications and services in real time. Through innovations to the purchasing process and the product experience, we've helped make thousands of customers successful running MongoDB on AWS. Some joint customers, like Unqork , are upending entire industries with innovative approaches to technology and business. Others, like Volvo's Connected Solutions business , rely on MongoDB and AWS to scale their fleet management solution from tens of millions to billions of daily events. Other recent customers include Verizon , Marsello , GLS , and Shopline . Get started with MongoDB Atlas on AWS You needn't take our word for it, however. With just a few clicks — and no risk — you can get started for free with MongoDB Atlas on AWS . There's no upfront commitment, and if you choose to continue to build with MongoDB on AWS, you only pay for what you use.