GIANT Stories at MongoDB

MongoDB and Realm make it easy to work with data, together

The best thing in the world is when someone just gets you, and you get them, because when you share a vision of the world like that, you can do incredible things together. That’s exactly the case with MongoDB and Realm, so I’m very happy to report that MongoDB has just entered into an agreement to acquire Realm).

Documents Are Everywhere

Over the past decade, following MongoDB’s lead, a raft of new document databases have been introduced and legacy databases have added document capabilities. In 2017, Microsoft layered an API for MongoDB on top of Cosmos DB (at the time called “DocumentDB”, but no longer), and recently Amazon released DocumentDB, which presents a subset of the MongoDB query language atop their Aurora technology. The document model, and the MongoDB API in particular, is flourishing.

MongoDB Drops ACID

MongoDB 4.0 will add support for multi-document transactions, making it the only database to combine the speed, flexibility, and power of the document model with ACID data integrity guarantees. Through snapshot isolation, transactions provide a globally consistent view of data, and enforce all-or-nothing execution to maintain data integrity.

Transactions in MongoDB will feel just like transactions developers are familiar with from relational databases. They will be multi-statement, with similar syntax (e.g. start_transaction and commit_transaction), making them familiar to anyone with prior transaction experience. The changes to MongoDB that enable multi-document transactions will not impact performance for workloads that do not require them. In MongoDB 4.0, which will be released this summer*, transactions will work across a single replica set, and MongoDB 4.2* will support transactions across a sharded deployment.

Because documents can bring together related data that would otherwise be modeled across separate parent-child tables in a relational schema, MongoDB’s atomic single-document operations already provide transaction semantics that meet the data integrity needs of the majority of applications. But multi-document transactions will make it easier than ever for developers to address a complete range of use-cases, while for many, simply knowing that they are available will provide critical peace of mind. With MongoDB 4.0, you’ll be able to rely on transactional integrity, regardless of how you model your data.

The imminent arrival of transactions is the culmination of a multi-year engineering effort, beginning over 3 years ago with the integration of the WiredTiger storage engine. We’ve laid the groundwork in almost every part of the server – from the storage layer itself, to the replication consensus protocol, to the sharding architecture. We’ve built out fine-grained consistency and durability guarantees, introduced a global logical clock, refactored cluster metadata management, and more. We’ve also exposed all of these enhancements through APIs that are fully consumable by our drivers. We’re now about 85% of the way through the backlog of features that enable transactions, as this diagram summarizes:

You can read more about our drive to multi-document transactions here. And if you can’t wait to take transactions for a spin, we’d love to have you join our beta program; all the details are at http://mongodb.com/transactions/.


About the Author

Eliot Horowitz is the CTO and Co-Founder of MongoDB. He wrote the core code base for MongoDB starting in 2007, and subsequently built the engineering and product teams. Today, Eliot oversees those teams, and continues to drive technology innovations at MongoDB. Prior to MongoDB, Eliot co-founded and built ShopWiki, a groundbreaking online retail search engine. He built its technology, its team, and presided over its private sale in 2010. Before that, Eliot was a software developer in the R&D group at DoubleClick.

Eliot is on the board of the NY Tech Talent Pipeline. In 2006, he was selected as one of BusinessWeek’s Top 25 Entrepreneurs Under Age 25, and in 2015 was named to the Business Insider “Under 35 and Crushing it” list. He was also recently named to Crain’s NY Business 40 Under 40 Class of 2017 list. Eliot received a B.S. in Computer Science from Brown University.


* Safe Harbour Statement

This post contains “forward-looking statements” within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934, as amended. Such forward-looking statements are subject to a number of risks, uncertainties, assumptions and other factors that could cause actual results and the timing of certain events to differ materially from future results expressed or implied by the forward-looking statements. Factors that could cause or contribute to such differences include, but are not limited to, those identified our filings with the Securities and Exchange Commission. You should not rely upon forward-looking statements as predictions of future events. Furthermore, such forward-looking statements speak only as of the date of this presentation.

In particular, the development, release, and timing of any features or functionality described for MongoDB products remains at MongoDB’s sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality. Except as required by law, we undertake no obligation to update any forward-looking statements to reflect events or circumstances after the date of such statements.

Announcing MongoDB 3.6

Today we’re announcing the release of MongoDB 3.6, the latest milestone in our efforts to give developers the most productive general-purpose data management systems for modern applications. 3.6 makes it easier than ever to work with data in the most natural, efficient, and frictionless way possible.

Allowing developers to persist rich, nested data without flattening it out is one of MongoDB’s defining strengths. Documents can model any kind of data: key value, graph, and relational data sets are as much at home in documents as heterogeneous, nested structures are. With MongoDB Server 3.6, we’re making the query language much more powerful, with new array update operators that allow you to specify in-place updates to specific, matching array items, at any depth of nesting. Extensions to the $lookup aggregation stage now allow uncorrelated subqueries and multiple matching conditions, so referencing and joining documents in complex combinations can be handled in the database.

Modern applications need to respond instantaneously to changes, providing notifications to their users and interfaces that update live. To enable this, MongoDB 3.6 introduces Change Streams, which applications can use to get real-time notification of updates to collection data. Just a few lines of code yield a tailable cursor that continuously delivers notifications on updates to the watched collection.

A key characteristic of robust systems is that they handle network outages gracefully, but the defensive coding to handle them can represent a significant burden on developers. MongoDB 3.6 alleviates that burden with Retryable Writes, a new feature ensuring that writes are performed exactly once, even in the face of outages.

MongoDB 3.6 also concludes our march to uniformly secure defaults, with localhost only binding implemented directly in the core server. Since 2.6.0, this behavior has been the default configuration in the most popular packaged versions of MongoDB, but the server itself did not default to localhost only binding, because many customer infrastructures use border security and relied on the previously permissive behavior. As of this release, MongoDB Server itself will default to refusing all connections unless they are from a whitelisted IP.

The flexibility of documents is fully compatible with data validation, and MongoDB 3.6 improves on its previous capabilities with the introduction of JSON Schema. Using JSON Schema, you can specify (in JSON), on a per-collection basis, exactly what qualifies as a valid document, such as the types a field can have, whether it is required, and if a document permits fields not listed in the spec. With MongoDB 3.6, schema isn’t a straightjacket, it’s framework of validation you can tune to exactly the degree you need.

MongoDB Atlas has seen an incredible collection of improvements in 3.6. Cross-Region Clusters provide a whole new level of high-availability at the click of a button, whether you run Atlas on AWS, Google Cloud Platform, or Azure. We’ve also introduced two new instance classes, M2 and M5. These low-cost, shared tenancy instances bridge the gap between the free tier and the fully dedicated tiers, and are perfect for the development phase of a project between toy and production.

On the operations side, Atlas and Ops Manager have both gained a Data Explorer for quickly digging into the data in your database, and a Performance Advisor, which analyzes the queries running in your cluster and finds opportunities for improvement, like missing indexes to add.

The BI Connector, introduced last year, has been completely re-written for MongoDB 3.6 making it run much faster and easier to manage. The BI connector 2.0 is much better at translating SQL queries into MongoDB’s native aggregation framework, so it can push much more work directly down to MongoDB, instead of having to do it itself in memory.

With over a hundred new and updated features, MongoDB 3.6 is our biggest release yet, and we’re incredibly excited to get it into your hands and see what amazing things you do with it.

Learn more

About the Author

Eliot Horowitz is the CTO and Co-Founder of MongoDB. He wrote the core code base for MongoDB starting in 2007, and subsequently built the engineering and product teams. Today, Eliot oversees those teams, and continues to drive technology innovations at MongoDB. Prior to MongoDB, Eliot co-founded and built ShopWiki, a groundbreaking online retail search engine. He built its technology, it’s team, and presided over its private sale in 2010. Before that, Eliot was a software developer in the R&D group at DoubleClick.

Eliot is on the board of the NY Tech Talent Pipeline. In 2006, he was selected as one of BusinessWeek’s Top 25 Entrepreneurs Under Age 25, and in 2015 was named to the Business Insider “Under 35 and Crushing it” list. He was also recently named to Crain’s NY Business 40 Under 40 Class of 2017 list. Eliot received a B.S. in Computer Science from Brown University.

A Founder’s Reflections on 10 Years of MongoDB

Today is the 10 year gitaversary of mongodb, and it is also the day that we become a publicly traded company. A coincidence, but fun regardless.

When Dwight and I started working on the project that would become MongoDB ten years ago, we definitely did not imagine that we would be where we are today. We only had our belief that we could make developers more productive. MongoDB was born out of our frustration using tabular databases in large, complex production deployments. We set out to build a database that we would want to use, so that whenever developers wanted to build an application, they could focus on the application, not on working around the database.

Databases are large and complex pieces of software. We would never have gotten to where we are today without an absolutely tremendous team, incredible customers, and a community that both supports us and holds us accountable to be as good as we can be.

While thanking each and every person who helped us would be impossible, what I can do is promise to continue making MongoDB the best it can be, so that you can focus on building great applications, and leave the data management nonsense to us.


About the Author

Eliot Horowitz is the CTO and Co-Founder of MongoDB. He wrote the core code base for MongoDB starting in 2007, and subsequently built the engineering and product teams. Today, Eliot oversees those teams, and continues to drive technology innovations at MongoDB. Prior to MongoDB, Eliot co-founded and built ShopWiki, a groundbreaking online retail search engine. He built its technology, it’s team, and presided over its private sale in 2010. Before that, Eliot was a software developer in the R&D group at DoubleClick.

Eliot is on the board of the NY Tech Talent Pipeline. In 2006, he was selected as one of BusinessWeek’s Top 25 Entrepreneurs Under Age 25, and in 2015 was named to the Business Insider “Under 35 and Crushing it” list. He was also recently named to Crain’s NY Business 40 Under 40 Class of 2017 list. Eliot received a B.S. in Computer Science from Brown University.

Announcing MongoDB Stitch: A Backend as a Service for MongoDB

Yesterday, we announced the release of MongoDB Stitch, a backend as a service for MongoDB designed for the modern application development paradigm. As with MongoDB Server, MongoDB Stitch was born to ameliorate the drudgery associated with building applications and replace it with the joy of creativity. What we did with the database 10 years ago, we now hope to do at the heart of application logic.


Back in 2007, MongoDB was born from 10gen, the platform as a service that Dwight Merriman and I created to solve a very similar problem, in a very different environment. Back then there were no services. The state of the art in application development was a Ruby on Rails app deployed to Heroku, and if you were really cutting-edge you'd be storing data in S3. Back then, we were trying to solve things like image thumbnailing and load balancing and rapid scaling… oh, and data storage... in such a way that developers could just write *their code* and not worry about the rest.

Speaking of which, how did people do, for example, image thumbnailing back then? Well, we all did the same thing: we installed ImageMagick on a bunch of servers and wrote some wrapper code. Of course, no-one would dream of doing that today. Today there are services for that sort of thing! In fact, we now have a virtual embarrassment of riches in the array of services available to handle everything from image OCR to credit card payments to geo-position mapping. But this glut of functionality has created a new pain point for developers.

A modern application developer needs to do three things above all:

1) support CRUD operations with data;

2) specify access control rules on their data; and

3) connect services together, and to their application, whether they are third-party services providing commodity functionality or proprietary microservices.

Unfortunately, each of these three things, until now, have required developers to create kloc upon kloc of the same time-draining, boilerplate code. Even when everything goes right with that code it's a drain on productivity, and when things go wrong, it becomes a liability ranging from costly to disastrous. All just to realize the promise of cloud computing and a service oriented architecture. This is exactly the kind of aggravation and inefficiency that MongoDB was founded to eliminate.

MongoDB Stitch does that by providing exactly the three things a modern application requires, as a simple layer adjacent to a MongoDB cluster. MongoDB Stitch provides a:

1) REST API to MongoDB, letting client code interact directly with the databases;

2) configuration based access control system, providing a flexible and powerful way to express precisely which users can perform what operations on what data; and

3) uniform, document-centric mechanism to connect services with custom application code.

And that's it. MongoDB Stitch can be used alongside existing code or to stand up brand new applications. Applications can do all the standard CRUD against MongoDB, but with complete assurance that clients can only access data to the exact extent to which they’re supposed to. Developers can compose services with MongoDB data operations into pipelines, meaning text messages routed from Twilio can become documents that can flow to MongoDB where they can be stored and continue on to S3 to be served via http. And none of it requires glue code beyond the bare minimum required to name and connect the services.

MongoDB Stitch is the next phase of MongoDB's mission to erase the routine tasks facing developers that lead to tedious, error-prone, and undifferentiated work. We're thrilled to finally have a chance to put the next piece of the puzzle in place.

P.S. I’m so thrilled that when I finally got to announce Stitch on stage at MongoDB World, I couldn’t help but show it off myself. If you want to see the code I demoed, it’s up on GitHub here.



About the Author - Eliot Horowitz

Eliot Horowitz is CTO and Co-Founder of MongoDB. Eliot is one of the core MongoDB kernel committers. Previously, he was Co-Founder and CTO of ShopWiki. Eliot developed the crawling and data extraction algorithm that is the core of its innovative technology. He has quickly become one of Silicon Alley's up and coming entrepreneurs and was selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 nationwide in 2006. Earlier, Eliot was a software developer in the R&D group at DoubleClick (acquired by Google for $3.1 billion). Eliot received a BS in Computer Science from Brown University.

Announcing Free Tier and Live Migration Tool for MongoDB Atlas

Last June we introduced MongoDB Atlas, the database as a service for MongoDB. Atlas is designed in accordance with all of the best practices for managing MongoDB, so using it is like getting a professional MongoDB Ops team on your side. It is the easiest and most cost effective way to run MongoDB in the cloud, and it is already helping thousands of teams -- from innovative startups like Bond to established industry leaders like eHarmony and Thermo Fisher -- to build apps more efficiently by making database management as easy as possible.

We’re incredibly excited by the success our customers have had with Atlas so far, and today I’d like to share some updates to the service that will make it even easier to get started with Atlas.

Making Atlas data migrations simple with MongoMirror

It’s a cinch to spin up a MongoDB cluster with Atlas, but if you’re already running an application, you still have to migrate data, which until now has been a manual process. Today we’re introducing a new utility called MongoMirror that automates that process. MongoMirror will live migrate data to MongoDB Atlas from any pre-existing MongoDB 3.0+ replica set, making it even easier to get your existing applications migrated to Atlas.

Get MongoDB in the cloud for free with the new M0 tier

We’re also making it easier than ever to experiment with a real cloud environment for MongoDB. The new “M0” cluster type is a free cluster, ideal for learning MongoDB or building a prototype. Like our existing cluster types, the M0 tier has optimal security, availability, and managed upgrades by default.

More to come

The M0 tier and MongoMirror remove even more barriers between developers and execution of their ideas. Now you can get started with MongoDB Atlas for free, migrate without downtime, and scale up as you need, completely seamlessly. In the coming months, we’ll be bringing MongoDB Atlas to the Google Compute Engine and Microsoft Azure, and we’re actively working on even more tools to seamlessly migrate existing workloads to MongoDB Atlas, so stay tuned.

Try MongoDB Atlas for yourself



About the Author - Eliot Horowitz

Eliot Horowitz is CTO and Co-Founder of MongoDB. Eliot is one of the core MongoDB kernel committers. Previously, he was Co-Founder and CTO of ShopWiki. Eliot developed the crawling and data extraction algorithm that is the core of its innovative technology. He has quickly become one of Silicon Alley's up and coming entrepreneurs and was selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 nationwide in 2006. Earlier, Eliot was a software developer in the R&D group at DoubleClick (acquired by Google for $3.1 billion). Eliot received a BS in Computer Science from Brown University.

Announcing MongoDB 3.4

Today we are announcing MongoDB 3.4, another milestone in our march to being the default database for modern applications. 3.4 makes MongoDB more flexible than ever, allowing developers to consolidate even more use cases into their MongoDB deployment, even as we continue to mature the platform and its ecosystem.

MongoDB was created to make it easy for developers to work with their data, beginning with introducing the document model itself. Documents are the best rudimentary unit for a data store, because they let you represent any kind of data, and embody their structure however best suits your use case. Whether that means deep or shallow nesting (or no nesting), documents can handle it. The key is being able to add many types of queries and algorithms to the data.

MongoDB 3.4 adds a stage to the aggregation pipeline that enables faceted search, greatly simplifying the query load for applications that browse and explore that data. It also adds operators to power graph queries. As we continue to add query features, users can consolidate more uses cases, instead of bloating their application footprint with a proliferation of specialized data stores.

Just because it’s easy to work with data in MongoDB, it doesn’t mean we can’t make it easier. In 3.4, the aggregation pipeline continues to mature, with more operators and expressions, enhancing string handling, allowing more sophisticated use of array elements, testing fields for type, and support for branching. Financial calculations are made simple with the addition of a Decimal data type.

I think it was John Donne who said: “No database is an island,” but whoever said it, they were very right. A database has to work as the heart of an ecosystem, and in 3.4, we continue to build that thriving ecosystem. Connecting MongoDB to the outside world is better than ever. MongoDB 3.4 introduces a ground-up rewrite of the BI connector, which improves performance, simplifies installation and configuration, and supports Windows. 3.4 also includes an update for our Apache Spark connector, with support for the Spark 2.0.

We’ve also extended the platforms that MongoDB runs on, including ARM-64, and IBM’s POWER8 and zSeries platforms.

MongoDB Compass is growing up with 3.4. It has new ways to depict data, such as the map view for geographic data, and it has become a data manipulation and performance tuning tool as well. In 3.4, Compass offers visual plan explanations, real-time stats, CRUD operations and index creation, so now you can identify, diagnose, and fix performance and data problems all from within Compass.

Of course, MongoDB 3.4 is supported by our trifecta of enterprise-grade ops management platforms: Ops Manager, Cloud Manager, and MongoDB Atlas, each of which add new features with this release. Ops Manager, for example, has improved its monitoring with built in telemetry gathering tailored to each deployment platform, and now allows ops teams to create server pools to serve up database-as-a-service to internal teams. Atlas introduces Virtual Private Cloud (VPC) Peering, allowing teams to use convenient private IPs to talk to their MongoDB service from within their AWS VPC.

There’s a ton more than I can fit into a blog post. That’s what release notes are for. But I shouldn’t leave out a few highlights, like: tunable consistency control for replica sets, including linearizable reads; collations for queries and indexes; and read-only views, which enable us to bring field level security to apps handling regulated data.

We’re incredibly excited to ship MongoDB 3.4 to you, so it can help your data serve you, not the other way around. Our approach is to build a database that can handle any kind of data, and the capabilities to query that data however you need to.

Learn more about MongoDB 3.4, register for our upcoming webinar:

Find out what's new



About the Author - Eliot Horowitz

Eliot Horowitz is CTO and Co-Founder of MongoDB. Eliot is one of the core MongoDB kernel committers. Previously, he was Co-Founder and CTO of ShopWiki. Eliot developed the crawling and data extraction algorithm that is the core of its innovative technology. He has quickly become one of Silicon Alley's up and coming entrepreneurs and was selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 nationwide in 2006. Earlier, Eliot was a software developer in the R&D group at DoubleClick (acquired by Google for $3.1 billion). Eliot received a BS in Computer Science from Brown University.

Announcing MongoDB Atlas: Database as a Service for MongoDB

Today we're announcing MongoDB Atlas, the simplest, most robust, and most cost effective way to run MongoDB in the Cloud. Atlas is a database as a service that makes running MongoDB almost effortless, whether you run a single replica set, or a sharded cluster hosting a hundred terabytes.

Simple:

Using Atlas is simplicity itself. There are no servers to set up, configure, or manage, no backups to schedule, no need to set up monitoring or look for security vulnerabilities. If a server goes down in the night, the system takes care of it for you. And if something cannot be fixed automatically, our 24/7 response team is paged, not you. When you need to add more storage, run on bigger hardware, or add more shards, it's a few clicks in the UI, and Atlas takes care of the rest.

Robust:

MongoDB Atlas is built upon the components of our current cloud offerings, which have been responsible for managing clusters for years. The monitoring component was first released 4 years ago. Atlas comes with the most advanced backup solution for MongoDB, first released three years ago. It supports point in time recovery, powerful configuration options, and has proven its stability over years of backing up the most demanding MongoDB workloads. The brain behind managing your Atlas clusters, Automation, has been available for two years, delivering reliable cluster creation, rolling upgrades, and configuration changes for thousands of MongoDB deployments in production. So while Atlas itself is new, all of the components that comprise Atlas have been battled tested for many years.

Elastic Pricing:

One of the nice features of cloud products in general is elastic pricing, and we wanted to provide the same convenience for our database as a service. With Atlas you pay for exactly what you use, and we bill your credit card on a monthly basis. You can spin up a cluster for a few hours to kick the tires for just a few dollars, and run your largest, most mission-critical workloads with confidence and predictable pricing.

Other Options to Manage MongoDB:

MongoDB Atlas is the absolute best way to run MongoDB in the cloud. However if you do not want to run entirely in the cloud, we offer other solutions that make managing MongoDB in your chosen environment easy. If you are running a hybrid environment you should look at MongoDB Cloud Manager, and if you are fully On-Prem you should try out MongoDB Ops Manager. All three management products share the same powerful components, assuring you the same peace of mind no matter what environment you run in. Wherever you want to deploy MongoDB, we’ve got you covered.

Today MongoDB Atlas is available for production on AWS, and will be launched with our strategic partners, Microsoft Azure and Google Cloud Platform in the future.

MongoDB has always been about getting the database out of the way and making it easy for teams to develop their apps, and now Atlas makes doing so in the cloud easier than ever.

Register below for our upcoming webinar covering MongoDB Atlas.

Register now: Introducing MongoDB Atlas

About the Author - Eliot Horowitz

Eliot is CTO and Co-Founder of MongoDB. He is one of the core MongoDB kernel committers. Previously, he was Co-Founder and CTO of ShopWiki. Eliot developed the crawling and data extraction algorithm that is the core of its innovative technology. He has quickly become one of Silicon Alley's up and coming entrepreneurs and was selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 nationwide in 2006. Earlier, Eliot was a software developer in the R&D group at DoubleClick (acquired by Google for $3.1 billion). Eliot received a BS in Computer Science from Brown University.