GIANT Stories at MongoDB

Introducing the Best Database for Modern Applications

The announcements we made today at MongoDB World 2018 represent a significant milestone in the evolution of MongoDB, making it the database of choice for all modern applications. Broadly speaking, there are three reasons for this:

  1. The document data model – presenting you the best way to work with data.
  2. It’s distributed by design – allowing you to intelligently put data where you want it.
  3. A unified experience that gives you the freedom to run anywhere – allowing you to future-proof your work and eliminate vendor lock-in.

There is a ton of new stuff, and so I wanted to give you a summary of what I covered during my keynote, with links to key resources so you can learn more.

Best Way to Work with Data

Today we released MongoDB Server 4.0 for General Availability. The highlight of the release is multi-document ACID transactions, which we previewed back in February with a beta program that attracted thousands of members of the community, putting transactions through their paces and providing invaluable feedback to the engineering team. We’ve implemented transactions so they feel just like the transactions you are familiar with from relational databases. They enforce snapshot isolation to provide a consistent view of data, and all-or-nothing execution to maintain data integrity. And while the document model means multi-document transactions aren’t necessary for most operations, with them it’s even easier for you to address a complete range of use cases with MongoDB.

It’s no secret how much I love MongoDB’s aggregation framework. Building queries stage-by-stage, checking your output as you go is by far a better way to write your most complicated queries than dealing with a monolithic snarl of SQL. To make that workflow even better, we’ve enhanced MongoDB Compass with the aggregation pipeline builder, which provides stage-by-stage, real-time feedback on the documents flowing through your pipelines. It’s easier than ever to deploy sophisticated processing pipelines that transform, aggregate, and analyze your data, all from the simple and intuitive MongoDB Compass GUI. You can then export the pipelines, and any other queries you create in Compass, to the native code of your preferred programming language. Server 4.0 also adds type conversions to the aggregation pipeline. With the new $convert operator you can transform mixed data types into standardized, cleansed formats natively within the database, preparing it for BI and machine learning, while eliminating costly, slow, and fragile ETL processes.

Extending the tools you can use to work with data managed by the server, we announced the public beta of MongoDB Charts, which provides the fastest and easiest way to get insights into your operational data, in real time. With Charts, you can create and share visualisations of your MongoDB data, using a document-native interface, without needing to move it into other systems or leverage third-party tools.

Documents and MongoDB’s query language are the best way to work with data, and to bring that power out of the datacenter and into the hands of app developers, MongoDB Stitch, which is GA as of today, provides two of its four services: QueryAnywhere and Functions. Using the authentication and declarative access control rules of Stitch QueryAnywhere, we can end the horrid practice of implementing shadow query languages in REST on top of application servers that just turn those REST calls into real query languages. With a native SDK, developers can make use of the full power of MongoDB from mobile and JavaScript applications, while Stitch makes sure that the right permissions are observed. Stitch Functions, JavaScript functions that execute with full access to application context, let developers compose their business logic with access to Atlas and calls to external services. With these two services, it’s easy to build complete applications without standing up a single application server.

Intelligently Put Data Where you Want It

As a distributed system, MongoDB enables you to spread data out across a cluster of nodes for resilience, scalability, and workload isolation. Unlike other distributed databases that randomly spray data around a cluster, MongoDB allows you to define controls that place data on specific nodes, for example in a specific region for low latency reads and writes, and for compliance with new privacy regulations.

The new Global Clusters introduced to MongoDB Atlas allow you to deploy a geographically distributed, fully managed database that provides low latency writes and reads to users anywhere, with data placement controls for regulatory compliance. We also announced Atlas Enterprise, offering new security controls including LDAP integration, the encrypted storage engine with bring-your-own key management, and database-level auditing. Organizations can now also use databases managed by MongoDB Atlas to build HIPAA-compliant applications under an executed Business Associate Agreement (BAA) with MongoDB, Inc. With these now announcements, MongoDB Atlas is the most secure cloud database service available anywhere.

Coding in a distributed world also means that the traditional means of responding to events in a database are no longer viable. So Stitch Triggers, also GA today, makes it possible by building on the Change Streams introduced in MongoDB 3.6. When you create a trigger, Stitch manages a change stream on your behalf, providing real-time notifications to Stitch Functions, which can react in all the ways functions can, from updating analytics rollup collections, to sending email or text messages, or kicking off other external services like Kafka or Kinesis.

Freedom to Run Anywhere

Whether you want to consume your database as a service in MongoDB Atlas, or manage it yourself on your own infrastructure, the announcements today make that even easier. We deliver a data platform that runs the same everywhere, that leverages the benefits of multi-cloud strategy with no lock-in, and is available in 50+ regions across the major cloud providers.

If you want to run MongoDB yourself, then we have released our new free MongoDB monitoring cloud service. The service is available to all MongoDB users, without needing to install an agent, navigate a paywall, or complete a registration form. You will be able to see the metrics and topology about your environment from the moment free monitoring is enabled. You can enable free monitoring easily using the MongoDB shell, MongoDB Compass, or by starting the mongod process with the new db.enableFreeMonitoring() command line option, and you can opt out at any time.

We’re seeing more DevOps teams leveraging the power of containerization and technologies like Kubernetes and Red Hat OpenShift to manage containerized clusters. Today we announced beta of the new MongoDB Enterprise Operator for Kubernetes, enabling you to deploy and manage MongoDB clusters from within the Kubernetes API, without having to connect separately to Ops Manager. You can learn more by reading our Red Hat OpenShift and MongoDB blog, and checking out the repository on GitHub.

Announced today, MongoDB Mobile takes MongoDB to a new frontier. Available in beta, MongoDB Mobile extends your ability to put data where you need it, all the way out to the edge of the network on IoT assets and iOS and Android mobile devices. MongoDB Mobile provides a single database, query language, and the intuitive Stitch SDK that runs consistently for data held on mobile clients, through to the backend server.

MongoDB Mobile provides the power and flexibility of MongoDB in a compact form that is power- and performance-aware with a low disk and memory footprint. It supports 64 bit iOS and Android operating systems and is easily embedded into mobile and IoT devices for fast and reliable local storage of JSON documents. With secondary indexing, access to the full MongoDB query language and aggregations, users can query data any way they want. With local reads and writes, MongoDB Mobile lets you build the fastest, most reactive apps. And Stitch Mobile Sync, which is in private beta now, will automatically synchronize data changes between data held locally and your backend database, helping resolve any conflicts – even after the device has been offline. The beta program is open now, and you can sign up for access on the MongoDB Mobile product page.

What’s Next?

So as you can see, that’s a ton of stuff. The announcements today represent our biggest set of releases yet, and we’re incredibly excited to get it into your hands and see what amazing things you do with them. Head over to our MongoDB World 2018 announcements page for more resources on each of these new products and services.

Announcing MongoDB 3.2

Eliot Horowitz
November 03, 2015
Company

Today we announced the release of MongoDB 3.2, the first since I declared a new era of maturity for our product with 3.0. This release continues to make the case for MongoDB as the default database for an organization, by meeting the needs of more mission critical use cases, and by meeting the needs of more roles within an organization.

MongoDB 3.2 is a giant release, and you can read the release notes to see the full list of new features. I’m proud of each feature individually, but moreover, I’m proud of how MongoDB’s engineering team continues to increase its bandwidth. When we were starting out, we had to round-robin between development domains, focusing in turn on whatever was the highest priority, whether it was improving performance, extending the query language, building out our scaling features, or enhancing operational ease. This release demonstrates how our team is firing on all cylinders, able to make progress on many fronts at once. Let’s take a tour...

We continued to push the boundaries of the document database model with features like document validation and partial indexes. We introduced huge new components, like the in-memory and encrypted storage engines, and the BI connector. We made infrastructure improvements, such as config servers as replica sets; under-the-hood improvements, such as replacing the JavaScript engine (more on that in another post); and we continued to refine our tools, giving mongodump and mongorestore the ability to compress, pipe, and stream data over a network.

The work we’ve done on our core feature set, including an enormous expansion of our aggregation framework and improvements to geo and text indexing, would alone have been a significant release in previous years, but this year we did them alongside the above.

We also put significant effort into housekeeping, bringing consistency in CRUD semantics between the new API and the shell, and we added fit and finish, such as optional case and diacritic sensitivity in text search.

In Ops Manager land, we added file system backups as an option, a new query profiler, index suggestions, and the ability to do a rolling index build across a replica set..

Huge new features, brand new tools, performance tuning, edge cases filed smooth, and some tidying up -- we did it all this year. And we’re still just getting started.


Learn more about MongoDB 3.2, register for our upcoming webinar:
Find out what's new

Additional Resources


About the Author - Eliot Horowitz

Eliot Horowitz is CTO and Co-Founder of MongoDB. Eliot is one of the core MongoDB kernel committers. Previously, he was Co-Founder and CTO of ShopWiki. Eliot developed the crawling and data extraction algorithm that is the core of its innovative technology. He has quickly become one of Silicon Alley's up and coming entrepreneurs and was selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 nationwide in 2006. Earlier, Eliot was a software developer in the R&D group at DoubleClick (acquired by Google for $3.1 billion). Eliot received a BS in Computer Science from Brown University.

Revisiting $lookup

Eliot Horowitz
October 29, 2015
Technical

Two weeks ago I announced that the new aggregation pipeline stage $lookup (a limited left-outer join operator) would be a feature available only in MongoDB Enterprise Advanced Server. Our rationale was that we viewed $lookup as an enabler of other Enterprise features and we were concerned that widespread availability of $lookup in the Community Edition would lead to users treating MongoDB like a relational database.

Since that announcement, we've heard from users who disagree with our decision. They believe that $lookup should be core to a document database, and available to anyone developing against it.

Dealing with that feedback has been tricky. No serious complaints about the way we sort features into the Enterprise versus Community edition have ever come up before. It required us to have long, careful discussions, in which we had to examine the criteria we use to decide what features belong in a community edition as opposed to a paid version. No simple, clear-cut principles are available -- we have tried very hard to distill them, but none of our attempts stand up to scrutiny.

Nonetheless, one thing is clear: this surprised our users unpleasantly, which is something we never want to do. We hear you. $lookup is going to be a community feature. Finding the principle that makes sense of this decision (and which can guide and explain future choices) is important to us, but not as important as the confidence of our community.

We’re still concerned that $lookup can be misused to treat MongoDB like a relational database. But instead of limiting its availability, we’re going to help developers know when its use is appropriate, and when it’s an anti-pattern. In the coming months, we will go beyond the existing documentation to provide clear, strong guidance in this area.

MongoDB is a commercial enterprise, but it’s also a community effort, and we will never forget that we owe our success to our users. Your passion and feedback help us make MongoDB a better product with every release. Thank you to everyone who made us look twice at $lookup.


Learn more about the new features coming in MongoDB 3.2, register for our webinar:
What's New in MongoDB 3.2

About the Author - Eliot Horowitz

Eliot Horowitz is CTO and Co-Founder of MongoDB. Eliot is one of the core MongoDB kernel committers. Previously, he was Co-Founder and CTO of ShopWiki. Eliot developed the crawling and data extraction algorithm that is the core of its innovative technology. He has quickly become one of Silicon Alley's up and coming entrepreneurs and was selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 nationwide in 2006. Earlier, Eliot was a software developer in the R&D group at DoubleClick (acquired by Google for $3.1 billion). Eliot received a BS in Computer Science from Brown University.

MongoDB on Gartner’s Magic Quadrant for Operational Database Management Systems 2015

Eliot Horowitz
October 13, 2015
Business

This week Gartner released its 2015 Magic Quadrant for Operational Database Management Systems. In this report MongoDB is recognized as a Leader, putting us in a coveted category of companies with both the ability to execute and completeness of vision.

The recognition of our current position is exciting, but what I find most encouraging is what history now says about our progress. MongoDB has been recognized in this Magic Quadrant since Gartner released the first Operational DBMS report in 2013, when we appeared as a Niche Player: significant, but wanting improvement in both execution and completeness of vision. In 2014 we again appeared in the Magic Quadrant, but this time positioned as a “Challenger”: strong on execution, but we still had work to do in terms of making our vision clear.

While I might not agree with Gartner that we didn’t have a strong vision, their analysis of our trajectory matches mine. When we started MongoDB in 2007, it was to prove that a document-oriented database could alleviate a host of problems that plague application developers. While there was a lot of early validation of that initial concept, we still needed years of building and evolving our product to fully realize it. One way we know we delivered was to see Gartner acknowledge that improvement in execution.

Today we once again find our hard work validated with an acknowledgment that our vision is now far more complete and well articulated. This is exactly the trend I see for MongoDB as well. MongoDB has grown to become a mature, organized, and multi-faceted team, full of talented engineers, able to focus on delivering near-term improvements to MongoDB while at the same time steering our product towards a long-term vision. This versatility is timely, because over the past few years, conversations with our users have taught us far more than we could have known at the outset, about the nature of data itself, and its place in application development. Out of this, we have refined a truly far-reaching vision:

  • The database is at the heart of modern applications
  • Databases should provide both high performance and high-level functionality -- you don’t have to choose one or the other
  • Documents are the right foundation, as they serve the needs of a wide range of applications, and can also function as the basis for other models, like key value.
  • A unified and tunable database framework is far better than a portfolio of specialized products, for both development and operations
  • Distributed, scaled, cloud infrastructures will be the default, so managing them must be simple

Furthermore, the choices we made, that were radical and untested when we started out, are becoming standard across databases, and the technologies based on them. Our vision drives more than just our product, it drives the industry as a whole. It’s a great time for MongoDB; we’ve got so many great clients, a great roadmap, and a great team capable of delivering it.

Read the press release

About the Author - Eliot Horowitz

Eliot Horowitz is CTO and Co-Founder of MongoDB. Eliot is one of the core MongoDB kernel committers. Previously, he was Co-Founder and CTO of ShopWiki. Eliot developed the crawling and data extraction algorithm that is the core of its innovative technology. He has quickly become one of Silicon Alley's up and coming entrepreneurs and was selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 nationwide in 2006. Earlier, Eliot was a software developer in the R&D group at DoubleClick (acquired by Google for $3.1 billion). Eliot received a BS in Computer Science from Brown University.

Thoughts on our new feature, $lookup

Eliot Horowitz
September 30, 2015
Technical

Editors note: the content in this blog was revised in a follow-up post, Revisiting $lookup.

One MongoDB 3.2 feature we demoed at MongoDB World in June was $lookup. $lookup is an aggregation pipeline stage that lets you insert data from a different collection into your pipeline. This is effectively a left outer join. $lookup will only be included in MongoDB Enterprise Server, and we’d like to give some background as to why and how we have decided this.

When we decided that MongoDB would be a document-based database, we did so for pragmatic technical reasons. The relational database model has several theoretical merits, but also many real-world limitations, chiefly the need to join data together across tables. Joins impede performance, inhibit scaling, and introduce substantial technical and cognitive overhead into all but textbook examples. In the real world, people adopt complicated workarounds in order to get good performance for modern applications: any large, performant, RDBMS-backed application will incorporate ad hoc denormalization, materialized views, external caching layers, and so on.

It was these limitations that motivated us to look for a data model that would both benefit developers building applications and also perform well at scale. We believe that documents are a natural, general-purpose data model that suits most applications people want to create today, and that, with careful schema design, allows for applications that are simultaneously more performant, more easily understood, and more scalable than applications that must rely on frequent joins.

Using a document model does not mean that joins aren't useful. There's no question that when data is being used for reporting, business intelligence, and analytics, joins are effective. Many of our users, having built fast, flexible, and feature-rich applications with MongoDB, tell us they now have so much application data in MongoDB that they'd like to use their existing BI tools to derive more business-value from it.

The full integration with these tools is not limited to $lookup, but also includes MongoDB Connector for BI, which translates SQL queries into Mongo Query Language queries. Our integration with BI and visualization tools can add tremendous value to companies looking for insights into their data, but it requires an outsized amount of time and testing to ensure smooth interoperation with these other costly products. As such, it is offered as part of our MongoDB Enterprise Advanced subscription. Our philosophy continues to be that documents are the right model for application development.

If you'd like to try out this feature, it's available in the 3.1.8 Enterprise Server build.


About the Author - Eliot Horowitz

Eliot Horowitz is CTO and Co-Founder of MongoDB. Eliot is one of the core MongoDB kernel committers. Previously, he was Co-Founder and CTO of ShopWiki. Eliot developed the crawling and data extraction algorithm that is the core of its innovative technology. He has quickly become one of Silicon Alley's up and coming entrepreneurs and was selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 nationwide in 2006. Earlier, Eliot was a software developer in the R&D group at DoubleClick (acquired by Google for $3.1 billion). Eliot received a BS in Computer Science from Brown University.

MongoDB Security Best Practices

Editor's note: This post was edited on June 23, 2015 to reflect the change from MongoDB Management Service to MongoDB Cloud Manager. Learn more here.

MongoDB takes security very seriously. Recently a team of German researchers discovered unsecured instances of MongoDB running openly on the internet. Readers who are concerned about access to their systems are reminded of the following resources:

  • The most popular installer for MongoDB (RPM) limits network access to localhost by default.
  • Security is addressed in detail in our Security Manual. The Security Checklist discusses limiting network exposure. Note that the method to do this will vary significantly depending on where the service is hosted (AWS, Azure, locally, etc).
  • Additionally, users of MongoDB Cloud Manager can enable alerts to detect if their deployment is internet exposed (see figure below).
  • A discussion on security is provided in two parts. Part 1 covers Design and Configuration. Part II covers 10 mistakes that can compromise your database.
  • We encourage users who have experienced a security incident for MongoDB to create a vulnerability report.

Create a new alert to notify of host being exposed to the public internet.


If you are interested in learning more about security best practices watch our on demand webinar.
Securing your MongoDB deployment

About the Author - Eliot Horowitz

Eliot Horowitz is CTO and Co-Founder of MongoDB. Eliot is one of the core MongoDB kernel committers. Previously, he was Co-Founder and CTO of ShopWiki. Eliot developed the crawling and data extraction algorithm that is the core of its innovative technology. He has quickly become one of Silicon Alley's up and coming entrepreneurs and was selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 nationwide in 2006. Earlier, Eliot was a software developer in the R&D group at DoubleClick (acquired by Google for $3.1 billion). Eliot received a BS in Computer Science from Brown University.

Announcing MongoDB 3.0

Eliot Horowitz
February 02, 2015
Company

Today we are announcing MongoDB 3.0. This release marks the beginning of a new phase in which we build on an increasingly mature foundation to deliver a database so powerful, flexible, and easy to manage that it can be the new DBMS standard for any team, in any industry.

MongoDB 3.0 brings with it massive improvements to performance and scalability, enabled by comprehensive improvements in the storage layer. We have built in the WiredTiger storage engine, an incredible technology with a distinguished pedigree. WiredTiger was engineered with latch-free, non-blocking algorithms to take advantage of trends in modern hardware, like large on-chip caches and heavily threaded architectures. By drawing on both academic research and their extensive commercial experience, the WiredTiger team built a storage engine that can underpin the next 20 years of data storage applications.

With WiredTiger, MongoDB 3.0 introduces document-level concurrency control, so performance remains fast and predictable under concurrent, write-intensive workloads. Transparent on-disk compression reduces storage requirements by up to 80%, and a choice of compression algorithms means that developers can tailor the performance/space trade-off to suit the needs of particular components in their applications.

MongoDB's original storage engine has been enhanced, including collection-level concurrency control and more efficient journaling. Now called MMAPv1, the default storage engine is a binary drop in that provides more efficient and concurrent handling of workloads.

We are currently running the production infrastructure of MongoDB Management Service on MongoDB 3.0 replica sets, mixing both MMAPv1 and WiredTiger storage engines. This type of flexibility is possible because MongoDB 3.0’s pluggable storage engine API addresses the reality that there is no single storage engine perfect for all applications. Through this architecture, MongoDB provides an easy to use, high-level layer for application development, durability, and horizontal scale, while allowing lower-level storage engines to offer solutions engineered for specialized use cases.

The storage engine API means that MongoDB can thrive in any environment, such as in-memory, encrypted, HDFS, or hardware-optimized, to name but a few. In fact, MongoDB can now be used as a laboratory for storage engine development. Delegating higher-level concerns to MongoDB lets storage engine developers focus on their domain of expertise, and lets a massive installation base of MongoDB users take the new engine for a spin just by adding a replica set member.

As with all infrastructure, MongoDB needs to be managed. We know this can be a challenge at large scale, so with MongoDB 3.0, we are giving enterprises Ops Manager, an on-premise product that makes MongoDB a first-class citizen of the mature operations ecosystem. Ops Manager makes routine tasks push-button, eliminating the vast majority of work that operations teams must do to manage MongoDB. It doesn't try to replace existing automation or provisioning tools, rather, it supports and supplements them, so ops can focus on architecture, not plumbing. Ops Manager's API means it can be integrated into existing tooling for monitoring, backup and management.

MongoDB 3.0 is a landmark release. It brings comprehensive improvements in performance, flexibility, ease of management, and cost to operate. MongoDB has always made it fast and easy to iterate -- from prototype to production to maturity, keeping pace in the most agile environments. Taken as a whole, the changes in 3.0 remove bottlenecks and obviate workarounds, suiting MongoDB to an even wider variety of applications, and providing teams the confidence to run it in more mission critical situations. MongoDB 3.0 is a true contender for the title of "default database" in any organization.

This is just the beginning. We will strengthen the case for MongoDB as your default database with every iteration. We will continue to surface advanced features of WiredTiger such as Log Structured Merge Trees. We will continue to push the envelope in data interaction semantics, by implementing a transaction system for the distributed document model. And we will iterate aggressively on Ops Manager, to make sure that operating MongoDB is as effortless as it is to develop with.

MongoDB 3.0 will be generally available in March, when we finish putting it through its paces. Stay tuned for our latest release candidate, we would love it if you would try it out and give us feedback.


If you’re considering updating your version of MongoDB, take a look at our Major Version Upgrade consulting services:

Upgrade with Confidence