GIANT Stories at MongoDB

Ten Years On, Still Going Strong - The MongoDB-User Community

This February is the ten-year anniversary of the creation of the mongodb-user group, a group which has been and still is the backbone of the MongoDB community. From its early days as a founder-supported channel for help and feedback, it has evolved through various forms from a nexus for support question to now, where it is a space where the MongoDB user community can share their knowledge - and that's a lot of knowledge with thirty-four thousand topics. With twenty-two thousand users, it's also where MongoDB Inc employees can make the fastest connections with that community.

The Future Will Be Documented

MongoDB was born out of frustration from using relational databases not designed for today's modern applications. After 40 years of no real alternatives, we pioneered a new way to work with data -- the MongoDB document model and the associated query language.

MongoDB in 2018

Welcome to that time of the year when we look back at what we've delivered to our users and community in the past year. Here's MongoDB's 2018, month by month - it's been a great year so let's start with...

January

The year started with the announcement of a new Go driver in development. Go is a popular language, used widely including inside MongoDB Inc, and we decided to start developing an official driver.

MongoDB now released under the Server Side Public License

Eliot Horowitz

Company

MongoDB has created a new software license called the Server Side Public License, or SSPL. The SSPL clarifies the conditions for making MongoDB publicly available as a service, to ensure we can continue to invest in building MongoDB for our users rather than in costly litigation over enforcing the AGPL. All subsequent versions and patch releases to prior versions of MongoDB made after October 16th, 2018 will be issued under the new SSPL.

Software as a service is one of the most significant and far-reaching paradigm shifts the technology industry has ever seen, and the market is moving to adopt it rapidly. This change is motivated by one simple truth: it is inefficient to operate software at scale when you just want to make use of its capabilities.

This should be a time of incredible opportunity for open source. The revenue generated by a service can be a great source of funding for open source projects, far greater than what has historically been available. The reality, however, is that once an open source project becomes interesting, it is too easy for large cloud vendors to capture most of the value while contributing little or nothing back to the community. As a result, smaller companies are understandably unwilling to wager their existence against the strategic interests of the large cloud vendors, and most new software is being written as closed source.

The best current solution for an open source company is to license their software under the AGPL, which requires a management stack used to operate that software as a service to be made available under the terms of the AGPL. This approach was believed to be good enough, as most people understood their obligations to comply with AGPL. However, as AGPL-licensed software like MongoDB has become more popular, organizations like the international cloud providers have begun to test the boundaries of this license. We would prefer to avoid litigation to defend the AGPL but instead devote our time to build great products for the community and our customers.

Since we own 100% of the copyright of MongoDB, one option available to us was to convert our open source license to a closed source license. We chose not to because we fundamentally believe in open source software. We believe an open source approach leads to more valuable, robust, and secure software, and it directly enables a stronger community and better products. We also could have licensed most of the code for the MongoDB server as AGPL while applying a closed license to some critical files. We chose not to do that because we believe that a uniform licensing approach, where all the code in a repository shares a single license, makes it easier to understand the obligations of using that code, leading to a stronger development community.

The community needs an updated open source license that builds on the spirit of the AGPL, but makes explicit the conditions for providing the licensed software as a service.

The SSPL is designed to make sure that companies who do run a publicly available MongoDB (or any software subject to the SSPL) as a service are giving back to the community.

It should be noted that the new license maintains all of the same freedoms the community has always had with MongoDB under the AGPL - they are free to use, review, modify, and redistribute the source code. We based the SSPL on GPL v3, with one new section that sets out an explicit condition for offering the licensed software as a service. We believe that the SSPL meets the standards for an open source license and are working to have it approved by the OSI.

Obviously, this new license helps our business, and that is one of the primary reasons we made this decision, but it is also critical for the MongoDB community. Over the past decade MongoDB has invested approximately $300M in research and development to offer an open source database to everyone, and this change protects our ability to invest aggressively in R&D to drive further innovation and value for the community.

I am convinced that the SSPL will have a positive effect on the open source community by encouraging increased investment into open source server-side software. As always, we are listening closely for your feedback, and we are eager to work with the community to drive towards a more open future.

Resources

Make Data-driven Decisions using MongoDB and Looker

Seth Payne

Company, BI

Today we are announcing a partnership and integration with Looker, a leader in modern, user-friendly business intelligence solutions. Like most BI tools, Looker works well with relational data, so the integration with MongoDB is made possible through the MongoDB Connector for Business Intelligence.

Using a simple, yet highly functional and powerful interface, Looker can ingest data from a number of different data sources empowering business users to create and manage reports and dashboards and make data-driven decisions. Now that MongoDB data can be queried by Looker directly, this removes the need to perform any ETL operations on MongoDB data.

Adding MongoDB as a data source is simple and can be accomplished through Looker’s admin interface for connections. The simplest way to get data into Looker is to use MongoDB Atlas, our DBaaS offering. Alternatively, customers can also connect to on-premises of self-managed instances. Due to MongoDB’s distributed architecture, you can connect to a dedicated secondary replica assigned to analytics and report away without impacting operational workloads running against other nodes in the cluster.

By combining the flexibility of MongoDB, along with Looker’s simple, intuitive, yet powerful interface, business users can quickly and easily gain insights from their MongoDB data. Users can create and manage dashboards, schedule reports, and even share these with colleagues. With Looker and MongoDB, business users can better understand data, identify trends, and make timely decisions based on solid analysis.

To learn more about MongoDB’s integration with Looker, please stop by our booth at Looker JOIN 2018 this week in San Francisco. There you will see hands-on demonstrations of the integration and examples of reports and dashboards that can be created using MongoDB and Looker together.

MongoDB wins 2018 Customer Impact award at SpringOne Platform!

Sarah Branfman

Cloud, Events, Company

MongoDB has won the Independent Software Vendor 2018 Pivotal Partner Award for Customer Impact , at the PivotalSpringOne Platform summit. This award recognizes partners that have delivered technology contributing to notable customer success.

We are humbled and honored!

In a world with endless options, the most sophisticated and demanding organizations choose to run their business on MongoDB. This is not a coincidence. MongoDB constantly pushes the pace of innovation to addresses the most challenging problems for our customers and strategically partners with leaders like Pivotal to deliver robust solutions to accelerate development and success.

We launched MongoDB for Pivotal Cloud FoundryⓇ (PCF) earlier this year and already have amazing and very public success. For example, Merrill Corporate launched a new category for M&A professionals with MongoDB, Pivotal and Microsoft. Our joint solution yielded 20x faster Deployment, client-identified bugs fixed in hours, 25% increase in sales and a true transformation into a product led technology organization!

Staying close to customer requirements and focused on customer success - that’s why it’s now easier than ever to rapidly deploy MongoDB powered applications on Pivotal Cloud Foundry by abstracting complexities around ensuring a consistent, predictable, and secure underlying infrastructure that can scale. MongoDB will continue with frequent updates and releases as we evolve our product inline with customer needs. Customers can download a BOSH deployed tile for Pivotal Application ServiceⓇ (PAS) and we are thrilled to announce the beta of MongoDB for the Pivotal Container ServiceⓇ (PKS) as well!

Learn more about the tile in these great posts by my colleagues: On Demand MongoDB Enterprise Server on Pivotal Cloud Foundry and MongoDB Enterprise Server for Pivotal Cloud Foundry goes GA

For those with us in Washington D.C. for the SpringOne Platform conference, we have two amazing sessions for you:

Join MongoDB’s Jeff Yemin (Lead Engineer, Database Engineering) and Pivotal’s Christoph Strobl (Software Engineer) for ‘Next Generation MongoDB: Sessions, Streams, Transactions’ and Diana Esteves (MongoDB Senior Engineer) for MongoDB + CredHub = Secure By Default Data Services on PCF and stop by our meeting room to say hello!

We’re honored to have received this prestigious award from Pivotal and look forward to continued success for our joint customers as MongoDB and Pivotal help tackle their biggest challenges!

16 Cities in 5 Months: The MongoDB team is coming to an AWS Summit near you

As our community of users continues to grow and become more diverse, we want to ensure all of our customers are fully equipped to be successful on MongoDB Atlas. To that end, we have partnered with AWS, committing to 16 of their regional Summits. These 16 events span 13 different countries and expect to draw thousands of members of the AWS and MongoDB communities.

Stack Overflow Research of 100,000 Developers Finds MongoDB is the Most Wanted Database

Michael Lynn

Company

It’s well established that developers want to work with a database that offers flexibility, versatility, and ease of use. Scalability and reliability certainly don’t hurt either. For an increasing number of developers, MongoDB is the desired solution to meet all these requirements. And for the second year in a row, this has been validated by the opinions of those who matter most — developers themselves.

Today, we’re excited to announce that, for the second year in a row, MongoDB is the most wanted database in the Stack Overflow Developer Survey 2018, the world’s largest developer survey with over 100,000 respondents.

Steady Increases in Popularity

Since its debut, MongoDB’s popularity among developers has steadily increased. This popularity has been driven primarily by the platform’s ease of use and flexibility; more recently, the release of the fully managed database service MongoDB Atlas has made it easier than ever to run the database in any of the major cloud platforms.

MongoDB continues to innovate, listening and reacting to the demands of developers from small startups to the largest of enterprises. New and exciting features to the platform include change streams, which push real-time data updates to downstream applications, retryable writes, which enhance reliability without increasing complexity, and the announcement of multi-document ACID transactions, coming in version 4.0 later this year.

Why MongoDB?

MongoDB’s document model offers developers massive increases in productivity by allowing developers to store data in a way that is consistent with how they think and create applications. Additionally, MongoDB’s native replication and horizontal partitioning give developers the confidence and freedom to concentrate on differentiating code without having to be concerned about challenges associated with data locality, reliability and scalability. Lastly, MongoDB offers the freedom to run consistently all the way from local development environments to the largest mainframe deployments. The database can be deployed on-premises, in a hybrid cloud, or in any of the public clouds.

How can you benefit from this news?

Hiring managers seeking talented developers may do well to review some of the additional statistics included in today’s release. One interesting statistic can be found in the chart entitled “How Developers Assess Potential Jobs”. If you are a hiring manager and you’re looking to attract top talent, considering the most wanted database platform might be a way to enhance the attractiveness of your opportunities.

Javascript and Python developers reviewing the Stack Overflow report will be encouraged as these development environments continue to grow in popularity. Popularity of web application stacks like MEAN or MERN have provided Javascript developers with frameworks for building applications quickly.

Not using Javascript or Python? MongoDB offers full support for these and all popular development languages. Want to use Golang with MongoDB? A fully supported MongoDB Golang driver has reached Alpha stage in development, giving even more options to developers to build with the latest and most popular languages.

In summary

Developers are the new Kingmakers and their opinions are vitally important. The results of the Stack Overflow survey are an exciting validation of MongoDB’s efforts to build a resilient, flexible, and easy-to-use platform wanted by developers the world over.


Are you looking to learn more?

MongoDB University offers free courses for beginners, developers, database administrators, and operations personnel. Get online education about MongoDB Atlas, Compass and other products to help developers build great modern applications.

Want to build right away with MongoDB? Get started today with a 512 MB database managed by MongoDB Atlas here.

MongoDB Drops ACID

MongoDB 4.0 will add support for multi-document transactions, making it the only database to combine the speed, flexibility, and power of the document model with ACID data integrity guarantees. Through snapshot isolation, transactions provide a globally consistent view of data, and enforce all-or-nothing execution to maintain data integrity.

Transactions in MongoDB will feel just like transactions developers are familiar with from relational databases. They will be multi-statement, with similar syntax (e.g. start_transaction and commit_transaction), making them familiar to anyone with prior transaction experience. The changes to MongoDB that enable multi-document transactions will not impact performance for workloads that do not require them. In MongoDB 4.0, which will be released this summer*, transactions will work across a single replica set, and MongoDB 4.2* will support transactions across a sharded deployment.

Because documents can bring together related data that would otherwise be modeled across separate parent-child tables in a relational schema, MongoDB’s atomic single-document operations already provide transaction semantics that meet the data integrity needs of the majority of applications. But multi-document transactions will make it easier than ever for developers to address a complete range of use-cases, while for many, simply knowing that they are available will provide critical peace of mind. With MongoDB 4.0, you’ll be able to rely on transactional integrity, regardless of how you model your data.

The imminent arrival of transactions is the culmination of a multi-year engineering effort, beginning over 3 years ago with the integration of the WiredTiger storage engine. We’ve laid the groundwork in almost every part of the server – from the storage layer itself, to the replication consensus protocol, to the sharding architecture. We’ve built out fine-grained consistency and durability guarantees, introduced a global logical clock, refactored cluster metadata management, and more. We’ve also exposed all of these enhancements through APIs that are fully consumable by our drivers. We’re now about 85% of the way through the backlog of features that enable transactions, as this diagram summarizes:

You can read more about our drive to multi-document transactions here. And if you can’t wait to take transactions for a spin, we’d love to have you join our beta program; all the details are at http://mongodb.com/transactions/.


About the Author

Eliot Horowitz is the CTO and Co-Founder of MongoDB. He wrote the core code base for MongoDB starting in 2007, and subsequently built the engineering and product teams. Today, Eliot oversees those teams, and continues to drive technology innovations at MongoDB. Prior to MongoDB, Eliot co-founded and built ShopWiki, a groundbreaking online retail search engine. He built its technology, its team, and presided over its private sale in 2010. Before that, Eliot was a software developer in the R&D group at DoubleClick.

Eliot is on the board of the NY Tech Talent Pipeline. In 2006, he was selected as one of BusinessWeek’s Top 25 Entrepreneurs Under Age 25, and in 2015 was named to the Business Insider “Under 35 and Crushing it” list. He was also recently named to Crain’s NY Business 40 Under 40 Class of 2017 list. Eliot received a B.S. in Computer Science from Brown University.


* Safe Harbour Statement

This post contains “forward-looking statements” within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934, as amended. Such forward-looking statements are subject to a number of risks, uncertainties, assumptions and other factors that could cause actual results and the timing of certain events to differ materially from future results expressed or implied by the forward-looking statements. Factors that could cause or contribute to such differences include, but are not limited to, those identified our filings with the Securities and Exchange Commission. You should not rely upon forward-looking statements as predictions of future events. Furthermore, such forward-looking statements speak only as of the date of this presentation.

In particular, the development, release, and timing of any features or functionality described for MongoDB products remains at MongoDB’s sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality. Except as required by law, we undertake no obligation to update any forward-looking statements to reflect events or circumstances after the date of such statements.

MongoDB’s Drive to Multi-Document Transactions

Transactions are important. Any database needs to offer transactional guarantees to enforce data integrity. But they don’t all do it in the same way – different database technologies take different approaches:

  • Relational databases model an entity’s data across multiple rows and parent-child tables, and so transactions need to span those rows and tables.
  • With subdocuments and arrays, document databases allow related data to be unified hierarchically inside a single data structure. The document can be updated with an atomic operation, giving it the same data integrity guarantees as a multi-table transaction in a relational database.

Because of this fundamental difference in data modeling, MongoDB’s existing atomicity guarantees are able to meet the data integrity needs of most applications. In fact, we estimate 80%-90% of applications don’t need multi-document transactions at all. However, there are some legitimate use cases and workloads where transactions across multiple documents are needed. In those cases, without transactions, a developer would have to implement complex logic on their own in the application layer. Also, some developers and DBAs have been conditioned by 40 years of relational data modeling to assume multi-table/document transactions are a requirement for any database, irrespective of the data model they are built upon. Others are concerned that while multi-document transactions aren’t needed by their apps today, they might be in the future and they don’t want to outgrow their database.

And so, the addition of multi-document ACID transactions makes it easier than ever for developers to address a complete range of use-cases on MongoDB.

As one can imagine, multi-document transactions are a much more complex thing to build in a distributed database than in a monolithic, scale-up database. In fact, we have been working on bringing multi-document transactions to MongoDB as part of a massive multi-year engineering investment. We have made enhancements to practically every part of the system – the storage layer itself, our replication consensus protocol, sharding architecture, consistency and durability guarantees, the introduction of a global logical clock, and refactored cluster metadata management and more. And we’ve exposed all of these enhancements through APIs that are fully consumable by our drivers.

The figure below represents the evolution of these enhancements as well as the work in progress to enable multi-document transactions. As you can see, we are nearly done.

In MongoDB 4.0, coming in summer 2018*, multi-document transactions will work across a replica set. We will extend support for transactions across a sharded deployment in the following release.

Importantly, the green boxes highlight all of the critical dependencies to transactions that have already been delivered over the past 3 years. And, frankly, that was the hardest part of the project – how to balance building the stepping stones we needed to get to transactions with delivering useful features to our users straightaway to improve their development experience along this journey. Wherever we could, we built components that suited both goals. For example, the introduction of the global logical clock and timestamps in the storage layer enforces consistent time across every operation in a distributed cluster. These enhancements are needed for transactions in order to provide snapshot isolation, but they also allowed us to implement change stream resumability and causal consistency in MongoDB 3.6, which are immediately valuable on their own. Change streams enable developers to build reactive applications that can view, filter, and act on data changes as they occur in the database in real-time, and recover from transient failures. Causal consistency allows developers to maintain the benefits of strong data consistency with “read your own write” guarantees, while taking advantage of scalability and availability of our intelligent distributed data platform.

The global logical clock is just one example. A selection of other key enhancements along the way illustrates how our engineering team deliberately laid the groundwork for transactions in such a way that we consistently surfaced additional benefits to our users:

  • The acquisition of WiredTiger Inc. and integration of its storage engine way back in MongoDB 3.0 brought massive scalability gains with document level concurrency control and compression to MongoDB. And with MVCC support, it also provided the storage layer foundations for transactions coming in MongoDB 4.0.
  • In MongoDB 3.2, the enhanced consensus protocol allowed for faster and more deterministic recovery from the failure or network partition of the primary replica set member, along with stricter durability guarantees for writes. These enhancements were immediately useful to MongoDB users then, and they are also essential capabilities for transactions.
  • The introduction of readConcern in 3.2 allowed applications to specify read isolation level on a per operation basis, providing powerful and granular consistency controls.
  • Logical sessions in MongoDB 3.6 gave our users causal consistency and retryable writes, but as a foundation for transactions, they provide MongoDB the ability to coordinate client and server operations across the nodes of a distributed cluster, managing the execution context for each statement in a transaction.
  • Similarly, retryable writes, implemented in MongoDB 3.6, simplify the development of applications in the face of elections (or other transient failures) while the server enforces at most once processing semantics.
  • Replica set point in time reads in 4.0 are essential for transactional consistency, but it’s also highly valuable to regular read operations that don’t need to be executed in a transaction. With this feature, reads will only show a view of the data that is consistent at the point the find() operation starts, irrespective of which replica serves the read, or what data has been modified by concurrent operations.

The number of remaining pieces on the roadmap to transactions is small. Once complete, multi-document distributed transactions will provide a globally consistent view of data (both in replica set and sharded deployments) through snapshot isolation and maintain all-or-nothing guarantees in cases of node failures. This will greatly simplify your application code. After all, MongoDB’s job is to take hard problems and solve them for as many developers as possible, so that you can focus on adding value to your applications and not dealing with the underlying plumbing.

We’re really excited about the release of multi-document transactions, and what they will allow you to build with MongoDB going forward. You should view our multi-document transactions page to learn more, and we invite you to sign up for the beta program so that you can start to put all of the work we’ve done through its paces.


* Safe Harbour Statement

This post contains “forward-looking statements” within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934, as amended. Such forward-looking statements are subject to a number of risks, uncertainties, assumptions and other factors that could cause actual results and the timing of certain events to differ materially from future results expressed or implied by the forward-looking statements. Factors that could cause or contribute to such differences include, but are not limited to, those identified our filings with the Securities and Exchange Commission. You should not rely upon forward-looking statements as predictions of future events. Furthermore, such forward-looking statements speak only as of the date of this presentation.

In particular, the development, release, and timing of any features or functionality described for MongoDB products remains at MongoDB’s sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality. Except as required by law, we undertake no obligation to update any forward-looking statements to reflect events or circumstances after the date of such statements.