Revisiting $lookup

Eliot Horowitz


Two weeks ago I announced that the new aggregation pipeline stage $lookup (a limited left-outer join operator) would be a feature available only in MongoDB Enterprise Advanced Server. Our rationale was that we viewed $lookup as an enabler of other Enterprise features and we were concerned that widespread availability of $lookup in the Community Edition would lead to users treating MongoDB like a relational database.

Since that announcement, we've heard from users who disagree with our decision. They believe that $lookup should be core to a document database, and available to anyone developing against it.

Dealing with that feedback has been tricky. No serious complaints about the way we sort features into the Enterprise versus Community edition have ever come up before. It required us to have long, careful discussions, in which we had to examine the criteria we use to decide what features belong in a community edition as opposed to a paid version. No simple, clear-cut principles are available -- we have tried very hard to distill them, but none of our attempts stand up to scrutiny.

Nonetheless, one thing is clear: this surprised our users unpleasantly, which is something we never want to do. We hear you. $lookup is going to be a community feature. Finding the principle that makes sense of this decision (and which can guide and explain future choices) is important to us, but not as important as the confidence of our community.

We’re still concerned that $lookup can be misused to treat MongoDB like a relational database. But instead of limiting its availability, we’re going to help developers know when its use is appropriate, and when it’s an anti-pattern. In the coming months, we will go beyond the existing documentation to provide clear, strong guidance in this area.

MongoDB is a commercial enterprise, but it’s also a community effort, and we will never forget that we owe our success to our users. Your passion and feedback help us make MongoDB a better product with every release. Thank you to everyone who made us look twice at $lookup.

Learn more about the new features coming in MongoDB 3.2, register for our webinar:
What's New in MongoDB 3.2

About the Author - Eliot Horowitz

Eliot Horowitz is CTO and Co-Founder of MongoDB. Eliot is one of the core MongoDB kernel committers. Previously, he was Co-Founder and CTO of ShopWiki. Eliot developed the crawling and data extraction algorithm that is the core of its innovative technology. He has quickly become one of Silicon Alley's up and coming entrepreneurs and was selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 nationwide in 2006. Earlier, Eliot was a software developer in the R&D group at DoubleClick (acquired by Google for $3.1 billion). Eliot received a BS in Computer Science from Brown University.