The Rise of the Strategic Developer
The work of developers is sometimes seen as tactical in nature. In other words, developers are not often asked to produce strategy. Rather, they are expected to execute against strategy, manifesting digital experiences that are defined by the “business.” But that is changing. With the automation of many time-consuming tasks -- from database administration to coding itself -- developers are now able to spend more time on higher value work, like understanding market needs or identifying strategic problems to solve. And just as the value of their work increases, so too does the value of their opinions. As a result, many developers are evolving, from coders with their heads-down in the corporate trenches to highly strategic visionaries of the digital experiences that define brands. “I think the very definition of ‘developer’ is expanding,” says Stephen “Stennie” Steneker, an engineering manager on the Developer Relations team at MongoDB. “It’s not just programmers anymore. It’s anyone who builds something.” Stennie notes that the learning curve needed to build something is flattening. Fast. He points to an emerging category of low code tools like Zapier, which allows people to stitch web apps together without having to write scripts or set up APIs. “People with no formal software engineering experience can build complex automated workflows to solve business problems. That’s a strategic developer.” Many other traditional developer tasks are being automated as well. At MongoDB, for example, we pride ourselves on removing the most time-consuming, low-value work of database administration. And of course, services like GitHub Copilot are automating the act of coding itself. So what does this all mean for developers? A few things: First, move to higher ground. In describing one of the potential outcomes of GitHub Copilot, Microsoft CTO Kevin Scott said, ““It may very well be one of those things that makes programming itself more approachable.” When the barriers to entry for a particular line of work start falling, standing still is not an option. It’s time to up your strategic game by offering insight and suggestions on new digital experiences that advance the objectives of the business. Second, accept more responsibility. A strategic developer is someone who can conceive, articulate, and execute an idea. That also means you are accountable for the success or failure of that idea. And as Stennie reminded me, “There are more ways than ever before to measure the success of a developer’s work.” And third, never stop skilling. Developers with narrow or limited skill sets will never add strategic value, and they will always be vulnerable to replacement. Like software itself, developers need to constantly evolve and improve, expanding both hard and soft skills. How do you see the role of the developer evolving? Any advice for those that aspire to more strategic roles within their organizations? Reach out and let me know what you think at @MarkLovesTech .
4 Common Misperceptions about MongoDB
One year ago, in the middle of the pandemic, Dev Ittycheria, the CEO of MongoDB, brought me on as Chief Technology Officer. Frankly, I thought I knew everything about databases and MongoDB. After all, I’d been in the database business for 32 years already. I’d been on MongoDB’s Board of Directors and used the products extensively. And of course I’d done my due diligence, met the leadership team, and analyzed earnings reports and product roadmaps. Even with all that knowledge, this past year as MongoDB’s CTO has taught me that many of my preconceived notions were just plain wrong. This made me wonder how many other people might also have the wrong impression about this company. And this blog is my attempt to set those perceptions straight by sharing my four major revelations of the last year. My first revelation is that MongoDB is not trying to become this generation’s relational database. For years I assumed that MongoDB basically wanted to be a better, more modern version of Oracle when it grew up. In other words, compete with the huge footprint of Oracle and other commercial RDBMSs that have been the industry archetype for so long. I was way off. The whole point of MongoDB is to leave all those forms of archaic, legacy database technology in the historical dust. This was never supposed to be an evolution, but instead a revolution. Our founders not only envisioned the world's fastest and most scalable persistent store, but also one that would be programmed and operated differently. The combination of embedded documents and structures combined with automatic high availability and almost-infinite distribution capability all add up to a fundamentally different way of working with data, building applications, and running those applications in production. Oracle and (SQL*Server, etc) still hang their hats on E.F. Codd’s 51-year old vision of rows and columns. To obtain high availability and distribution of data, you need add ons, options packages, baling wire and duct tape. And you need a lot of database administrators. Not cheap. Even after all that, you’re still trailing the technological edge. This is how wrong I was. Our durable competitive advantages over these legacy data stores make competing with those products almost irrelevant. We instead focus on the modern needs of modern developers building modern applications. These developers need to create their own competitive advantage through language-native development, reliable deployments to production, and lightning fast iteration. And the world is noticing; just check out the falling slope of Oracle and SQL*Server and the rising slope of MongoDB on the db-engines website. Which brings me to my second revelation: MongoDB was built for developers, by developers. I always knew that MongoDB was exceedingly fast and easy to program against. One time while I was bored in a meeting (yes, it happens here as well!), I built an Atlas database, loaded it with 350MB of data, downloaded and learned our Compass data discovery tool, built-in analytics aggregation pipelines, and our Charts package, and embedded live charts in a web page. This took me all of 19 minutes, end to end. To build something like that for engineers , it just has to be built by engineers , ones that are free to focus on all the rough edges that creep into products as features are added. I was first exposed to software planning and management over 40 years ago, and my LinkedIn profile shows a pretty diverse tour around the industry. Now, one year in, I can emphatically state that engineering and product at MongoDB are both different and better than any company I’ve ever had the privilege to work at. Our executive leadership gives engineering and product broad brushstokes of goals and desired outcomes, and then we work together to come up with detailed roadmaps, updated quarterly, that meet those goals in the way we think best, with no micromanagement. And we’re not afraid of 3-5 year projects, either. For example, multi-cloud was more than three years in the making. Also unlike any other company I’ve been at, we embrace the creation and re-payment of tech debt, rather than sweeping it under the rug. We do this through giving our product and engineering teams huge amounts of context, delivered with candor and openness. And one more essential thing; we have an empowered program management team that improves processes (including killing them) as fast as we create them. In short, we paint the targets for our teams and let them decide how and when to shoot. They even design the arrows and bows. It’s true bottoms-up engineering. Our engineers feel valued and understood. And that, in turn, empowers them to develop features that make our customers feel valued and understood, like a unified query language, or real-time analytics and charting directly in the console, or multi-region/multi-cloud clusters where all the networking cruft is taken care of for you. And this brings me to my third revelation: MongoDB is built for even the most demanding mission critical applications. Fast? Yes. Easy? Of course. But mission-critical? That’s not how I saw MongoDB when I used Version 2 for a massive student data project 10 years ago. While it was the only possible datastore we could have chosen for the amount of data and the speed of ingestion and processing needed, it was pretty hard to set up and use in a 24 x 365 environment. MongoDB had gotten ahead of itself in the early 2010’s. There was a gap between our capabilities and the expectations of the market. And it was painful. Other databases had had more than 30 years to solidify their systems and operations. We’d had five. But with Version 3 we added a new storage engine, full ACID transactions, and search. We built on it with Version 4. And then again with Version 5, released this week at our .Live conference. I knew about all this progress intellectually of course when I joined, but not viscerally. I came to realize that the security, durability, availability, scalability, and operability our platform offers (of course in addition to all the features that developers love too) was ideal for architecting fast-moving enterprise applications. And I found the proof in our customer list. It reads like a Who’s Who of major global banks, retailers, and telecommunications companies, running core systems like payments, IoT applications, content management, and real-time analytics. They use our database, data lake, analytics, search, and mobile products across their entire businesses, in every major cloud, on-premises, and on their laptops. And that leads me to my fourth and final revelation. MongoDB is no longer just a database. Of course, the database is still the core. But MongoDB now provides an enterprise-class, mission-critical application data platform. A cohesive, integrated suite of offerings capable of managing modern data requirements across even the most sprawling digital estates, and scaling to meet the level of any company’s ambition, without sacrificing speed or security. Since the day I was first introduced to MongoDB’s products, I’ve had tremendous respect and admiration for the teams and their work. After all, I’m a developer, first and foremost. And it always felt like they “got” me. But had I known then what I know now, I would have jumped on this train a long time ago. In fact, I might have camped out on their doorstep with my resume in hand. And who knows? Maybe a bunch of people reading this will do just that, and have their own revelations about how fulfilling and exciting it can be to be at a great company, with a great culture, producing great products. I’ll write another letter a year from now, and let you know how it’s going then. In the meantime, please reach out to me here, or at @MarkLovesTech .
몽고DB에 대한 4가지 오해
일년 전 코로나19가 한창일 때 몽고DB CEO 데브 이티체리아가 나를 CTO(최고 기술 책임자)로 채용했다. 솔직히 나는 데이터베이스와 몽고DB에 대해 이미 모든 것을 알고 있다고 생각했다. 데이터 베이스 업계에서 일한지도 벌써 32년이나 됐고, 몽고DB 이사회에 참여하면서 몽고DB 제품을 두루 사용해왔기 때문이다. 물론 실사도 진행했고 경영진과 미팅을 가졌으며 실적 보고서와 제품 로드맵을 분석해왔다. 이렇게 지식을 쌓았음에도 불구하고 지난 한 해 동안 몽고DB에서 CTO로 일하면서 나는 내가 얼마나 잘못된 선입견을 갖고 있었는지를 깨닫게 됐다. 그리고 나처럼 몽고DB에 대해 오해하고 있는 사람이 많이 있을 것이라는 생각에 이르렀다. 이에 지난 일 년간 내가 알게 된 4가지 사실을 이 블로그를 통해 공유하여 사람들의 인식을 바로잡고자 한다. 첫째, 몽고DB의 비전은 현 세대의 관계형 데이터베이스가 되는 것이 아니라는 것이다. 나는 수년 전부터 몽고DB가 오라클보다 한층 개선된 최신 버전을 추구한다고 생각해왔다. 다시 말해, 몽고DB가 오라클의 광범위한 입지와 더불어 업계 전형으로 오랜 기간 군림해왔던 여타 상용 RDBMS와 경쟁하려는 듯 보였다. 하지만 이런 내 추측은 완전히 벗겨 갔다. 몽고DB의 궁극적인 목표는 모든 유형의 구식 레거시 데이터 베이스 기술에서 완전히 탈피하는 것이었다. 그리고 이러한 목표를 진화가 아닌 혁명으로 달성하고자 했다. 몽고DB의 설립자들은 세계 최고의 속도와 확정성을 갖춘 퍼시스턴트 저장소에서 한 걸음 더 나아가, 다른 방식으로 프로그래밍 및 운영이 가능한 저장소를 꿈꿨다. 내장된 문서와 구조가 고가용성을 갖춘 무제한에 가까운 자동 배포 기능과 결합하면 데이터 작업, 애플리케이션 구축, 그리고 프로덕션 단계에서의 애플리케이션 실행 방식이 근본적으로 달라지게 된다. 오라클과 SQL*서버 등은 아직도 열과 행이라는 51년 된 E.F 코드의 비전에 의존하고 있다. 데이터의 고가용성과 배포 능력을 확보하기 위해서는 추가 기능을 비롯해 옵션 패키지, 베일링 와이어, 그리고 덕트 테이프가 필요하다. 여기에 데이터베이스 관리자도 상당수 투입해야 하기에 만만치 않은 비용이 들게 된다. 그럼에도 불구하고 많은 이들이 여전히 기술적 우위를 추구하고 있다. 나도 이렇게 잘못 생각하고 있었다. 레거시 데이터 저장소를 능가하는 몽고DB의 지속 가능한 경쟁 우위를 고려할 때, 레거시 데이터 저장소 제품과 경쟁한다는 건 사실상 무의미하다. 대신 우리는 최신 애플리케이션을 구축하는 오늘날 개발자들의 최신 요구사항에 초점을 맞춘다. 지금의 개발자들은 고유 언어 개발, 프로덕션 환경으로의 안정적인 배치, 초고속 반복 작업을 통해 자신만의 경쟁 우위를 창출해야 한다. 그리고 전 세계가 이를 주목하고 있다. db 엔진 사이트 만 하더라도, 오라클과 SQL*서버는 하락세에 접어들고 있는데 반해 몽고DB는 상승세를 타고 있는 것을 알 수 있다. 여기서 내가 두 번째로 깨달은 점이 있다. 몽고DB는 개발자가 직접 개발자를 위해 만든 제품이라는 것이다. 나는 몽고DB가 굉장히 빠르고 프로그래밍 하기도 매우 쉽다는 것을 익히 알고 있었다. 어느 날은 회의가 너무 지루해서 (몽고DB에서도 이런 일은 일어난다!) 아틀라스 데이터베이스를 구축해 350MB의 데이터를 로드했다. 그리고 몽고DB의 콤파스 데이터 디스커버리 툴, 내장형 분석 집계 파이프라인, 몽고DB 차트 패키지, 웹페이지의 내장 라이브 차트를 다운로드해 공부했다. 시작부터 끝까지 총 19분이 걸렸다. 이러한 환경을 엔지니어를 위해 개발하려면 제품에 기능을 추가할 때 발생하는 소소한 오류에 자유롭게 집중할 수 있는 엔지니어들이 개발해야 한다. 40여 년 전, 소프트웨어 기획과 관리를 처음 접하게 되었는데 내 링크드인 계정을 보면 이 쪽 업계에서 아주 다양한 일들을 해왔다는 것을 알 수 있을 것이다. CTO직을 맡은 지 일년이 지난 지금, 나는 우리의 엔지니어링 기술과 제품이 내가 그 간 일했던 다른 기업들의 제품과 다르며 월등하다고 장담할 수 있다. 몽고DB의 경영진은 엔지니어링과 제품 부서에게 우리의 목표와 원하는 결과를 대략적으로 설명한 다음 협력을 통해 세부 로드맵을 마련하고, 또 로드맵을 분기별로 업데이트하며, 사소한 것까지 관리할 필요 없이 우리의 목표를 가장 잘 충족하는 로드맵을 마련한다. 3~5년짜리 프로젝트를 한다고 하더라도 두려울 게 없다. 일례로, 멀티 클라우드를 개발하는 데는 3년 이상이 걸렸다. 과거 일했던 다른 기업들과는 달리, 몽고DB는 기술 부채를 비밀로 덮어두기 보다는 기술 부채의 생성과 상환에 적극 대응하고 있다. 우리는 제품 팀과 엔지니어링 팀에게 이러한 상황 전체를 가감없이 솔직하게, 그리고 공개적으로 알린다. 한 가지 더 중요한 사실은, 프로세스를 생성하는 즉시 개선할 수 있는(프로세스 삭제 포함) 역량 있는 프로그램 관리 팀이 있다는 것이다. 간략히 말해, 우리는 실무팀에게 목표를 제시하고, 목표를 달성하는 방식과 시기를 주체적으로 결정하게 한다. 실무 팀은 이 과정에서 자신들이 사용할 툴까지도 직접 설계한다. 진정한 상향식 엔지니어링인 것이다. 몽고DB의 엔지니어들은 스스로 가치 있고, 경영진으로부터 이해 받고 있다고 느낀다. 이를 통해 엔지니어들은 고객에게도 이러한 감정을 불러일으킬 수 있는 다양한 기능을 만들 수 있다. 이를테면 통합 쿼리 언어나 콘솔 기반의 실시간 분석 및 차트 작성 또는 모든 네트워킹 크러프트를 사용자 대신 처리하는 멀티 리전/멀티 클라우드 클러스터 기능 등이 있다. 그리고 여기서 세 번째로 알게 된 사실은, 몽고DB가 가장 까다로운 미션 크리티컬 애플리케이션에도 사용할 수 있도록 설계됐다는 점이다. 속도가 빠르냐고? 그렇다. 사용 편의성은? 당연히 편리하다. 그렇다면 미션 크리티컬은? 10년 전에 진행된 대규모의 학생 데이터 프로젝트에서 버전 2를 사용할 때까지만 해도 몽고DB를 이렇게 생각하진 않았다. 몽고DB는 작업에 필요한 데이터의 양과 데이터 수집 및 처리 속도를 위해 선택할 수 있는 유일한 데이터 저장소였지만, 24시간 상시 가동 환경에서 구축하여 사용하기에는 턱없이 부족했다. 몽고DB는 2010년대 초에 자체 역량을 넘어섰는데, 우리의 역량과 시장의 기대 사이에는 차이가 있었다. 이 점이 매우 힘들었다. 다른 데이터베이스는 시스템과 운영 방식을 강화하기까지 30년 이상이 걸렸다. 하지만 몽고DB는 이를 단 5년만에 해낸 것이다. 버전 3에는 새로운 저장 엔진, 풀 ACID 트랜잭션, 그리고 검색 기능을 추가했다. 이를 기반으로 버전 4를 만들었다. 그후 버전 5가 MongoDB.Live 2021 컨퍼런스에서 정식 출시됐다.(2021년 7월) 몽고DB에 입사할 당시 나는 이러한 상황을 알고 있었지만 실질적으로 체감하진 못했다. 그러다가 몽고DB 플랫폼이 제공하는 보안, 내구성, 가용성, 확장성, 운용성(그 외에도 개발자들이 선호하는 모든 기능)이 빠르게 변화하는 엔터프라이즈 애플리케이션을 설계하는 데 이상적이라는 것을 알게 되었다. 몽고DB 고객 목록에서 그 증거를 확인할 수 있었다. 우리의 고객 목록은 마치 결제, IoT 애플리케이션, 콘텐츠 관리, 실시간 분석 등 핵심 시스템을 운영하는 주요 글로벌 은행, 소매업체, 통신사의 명사록과 같았다. 이들은 자사의 모든 사업, 모든 주요 클라우드, 사내 현장, 노트북 등에서 몽고DB 데이터베이스, 데이터 레이크, 분석, 검색, 모바일 제품을 사용하고 있다. 여기서 네번째로 알게 된 마지막 사실은, 몽고DB가 단순한 데이터베이스가 아니라는 점이다. 물론 데이터베이스가 몽고DB의 핵심 사업이긴 하다. 하지만 몽고DB는 이제 엔터프라이즈급의 미션 크리티컬 애플리케이션 데이터 플랫폼을 제공하고 있다. 최대 규모의 디지털 단지에서도 최신 데이터 요건을 관리하고 속도나 보안을 저해하지 않으면서 회사가 추구하는 수준으로 확장 가능한 통합 제품군인 것이다. 나는 몽고DB 제품을 처음 접한 날부터 몽고DB 팀과 이들이 만든 제품에 감탄을 금치못했고 엄청난 존경심을 갖게 됐다. 어쨌건 나는 누가 뭐라 해도 개발자이다. 또 항상 개발자들은 나를 “이해한다”는 느낌이 들었다. 하지만 지금 알고 있는 사실을 그때 알았더라면 진작에 몽고DB에 입사했을 것이다. 이력서를 손에 들고 몽고DB 건물 입구에서 밤을 새고 기다렸을 수도 있다. 누가 알랴? 이 글을 읽는 많은 사람들이 그렇게 할지? 훌륭한 기업 문화를 갖춘 기업에서 훌륭한 제품을 만드는 것이 얼마나 보람 있고 흥분되는 일인지 깨달으면서 말이다. 1년 후에 다시 한번 글을 올려 그 때의 상황을 전할 생각이다. 그때까지 이 포스팅에 댓글을 남기거나 MarkLovesTech 로 연락 주기 바란다.
Launched Today: MongoDB 5.0, Serverless Atlas, and the Evolution of our Application Data Platform
Today we welcome you to our annual MongoDB .Live developer conference. Through our keynote and conference sessions we'll show you all the improvements, new features, and exciting things we've been working on since last year’s conference. What I want to do in this blog post is provide you with a summary of what we are announcing, and resources to help you learn more. While it's easy to focus on what we are announcing at this year's event, we actually started out on this journey 12 years ago by releasing the world’s most intuitive and productive database technology to develop with — MongoDB. And we believe the applications of the NEXT 10 YEARS will be built on data architectures that continue to optimize for the developer experience, allowing teams like yours to innovate at speed and scale. So how are we building on this vision? Today I am incredibly proud to announce three big things: The General Availability (GA) of MongoDB 5.0, the latest generation of our core database. It includes native support for time series workloads, new ways to future-proof your applications, multi-cloud privacy controls, along with a host of other improvements and new features. The preview release of serverless instances on MongoDB Atlas, which makes it even easier for development teams who don’t want to think about capacity management at all to get the database resources they need quickly and efficiently. Major enhancements to Atlas Data Lake, Atlas Search, and Realm Sync, which allow engineering teams to reduce architectural complexity and get more value out of their data by leveraging a unified application data platform. MongoDB 5.0 GA MongoDB 5.0 is the latest generation of the database most wanted by developers . Our new release makes it even easier to support a broader range of workloads, introduces new ways of future-proofing your apps, and further enhances privacy and security. This major jump in version number from MongoDB 4.4 – our prior GA version – to 5.0 reflects a new era for MongoDB's release cadence: We want to get new features and improvements into your hands faster. Starting with MongoDB 5.0, we will be publishing new Rapid Releases every quarter, which will roll up into Major Releases once a year for those of you that want to maintain the existing annual upgrade cadence. You can learn more about the new MongoDB release cadence from our blog post published last October. Digging into MongoDB 5.0, here is what’s new and improved: Native Time Series Designed for IoT and financial analytics, our new time series collections, clustered indexing, and window functions make it easier, faster, and lower cost to build and run time series applications, and to enrich your enterprise data with time series measurements. MongoDB automatically optimizes your schema for high storage efficiency, low latency queries, and real-time analytics against temporal data. Running your time series applications on our application data platform eliminates the time and the complexity of having to stitch together multiple technologies yourself. You can manage the entire time series data lifecycle in MongoDB – from ingestion, storage, querying, real-time analysis, and visualization through to online archiving or automatic expiration as data ages. Time series collections can sit right alongside regular collections in your MongoDB database, making it really easy to combine time series data with your enterprise data within a single versatile, flexible database – using a single query API to power almost any class of workload. Our new time-series collections blog post gives you everything you need to get started. Future-proof with the Versioned API and Live Resharding Starting with MongoDB 5.0, the Versioned API future-proofs your applications. You can fearlessly upgrade to the latest MongoDB releases without the risk of introducing backward-breaking changes that require application-side rework. Using the new versioned API decouples your app lifecycle from the database lifecycle, so you only need to update your application when you want to introduce new functionality, not when you upgrade the database. Future-proofing doesn’t end with the Versioned API. MongoDB 5.0 also introduces Live Resharding which allows you to easily change the shard key for your collections on demand – with no database downtime – as your workload grows and evolves. The way I like to think about this is that we’ve extended the flexibility the document model has always given you down to how you distribute your data. So as things change, MongoDB adapts without expensive schema or sharding migrations. Next-Gen Privacy & Security MongoDB’s unique Client-Side Field Level Encryption now extends some of the strongest data privacy controls available anywhere to multi-cloud databases. And with the ability in 5.0 to reconfigure your audit log filters and rotate x509 certificates without downtime you maintain a strict security posture with no interruption to your applications. Run MongoDB 5.0 Anywhere MongoDB 5.0 is available today as a fully-managed service in Atlas . You can of course also download and run MongoDB 5.0 on your own infrastructure, either with the community edition of MongoDB, or with MongoDB Enterprise Advanced . The Enterprise Advanced offering provides sophisticated operational tooling via Ops Manager, advanced security controls, proactive 24x7 support, and more. MongoDB Ops Manager 5.0 enhancements include: Support for the automation, monitoring, and backup/restore of MongoDB 5.0 deployments. Improved load performance with parallelized client-side restores. A quick start experience for deploying MongoDB in Kubernetes with Ops Manager. And lastly, a guided Atlas migration experience that walks users through provisioning a migration host to push data from their existing environment into the fully managed Atlas cloud service. You can learn more about MongoDB 5.0 from our What’s New guide . New to MongoDB Atlas — Serverless Instances (Preview) We want developers to be able to build MongoDB applications without having to think about database infrastructure or capacity management. With serverless instances on MongoDB Atlas, now available in Preview, you can automatically get the database resources you need based on your workload demand. It’s really simple: the only decision you need to make is the cloud region hosting your data. After that, you’ll get an on-demand database endpoint that dynamically adapts to your application traffic. Serverless instances will support the latest MongoDB 5.0 GA release, Versioned API, and upcoming Rapid Releases so you never have to worry about backwards compatibility or upgrades. Pay only for reads and writes your application performs and the storage resources you use (up to 1TB of storage in preview) and leave capacity management to MongoDB Atlas’s best-in-class automation. We invite you to try it out today with a new or existing Atlas account. And the Preview release is just the beginning – we will be working with partners such as Vercel and Netlify to deliver an integrated serverless development experience in the coming months. In the longer term, we will continue to evolve our cloud-native backend architecture to abstract and automate even more infrastructure decisions and optimizations to deliver the best database experience on the market. The New MongoDB Shell GA The new MongoDB Shell has been redesigned from the ground up to provide a modern command-line experience with enhanced usability features and powerful scripting environment. It makes it even easier for users to interact and manage their MongoDB data platform, from running simple queries to scripting admin operations. A great user experience, even on a command-line tool, should always be a major consideration. With the new MongoDB Shell we have introduced syntax highlighting, intelligent auto-complete, contextual help and useful error messages creating an intuitive, interactive experience for MongoDB users. Check out this blog post for more information. MongoDB Charts and Atlas Data Lake: Better Together MongoDB Charts intuitive UI and ability to quickly create & share charts and graphs of JSON data is now integrated with Atlas Data Lake . You can now easily visualize JSON data stored in Amazon AWS S3 without any data movement, duplication or transformation. Furthermore, you can run Atlas Data Lake’s federated query to blend data across multiple Atlas databases and AWS S3, and visualize the results with Charts. By adding Atlas Data Lake as a data source in Charts, you can discover deeper, more meaningful insights in real time. Check out this blog post for more information. Atlas Search — More Relevance Features It’s incredibly important for modern applications to deliver fast and relevant search functionality: it powers discoverability and personalization of content, which in turn drives user engagement and retention. Atlas Search , which delivers powerful full-text search functionality without the need for a separate search engine, has several new capabilities for building rich end user experiences. We’ve recently added support for function scoring, which allows teams to apply mathematical formulas on fields within documents to influence their relevance, such as popularity or distance — e.g. closer restaurants with more or better reviews will show up higher in a list of results. In addition, you can now define collections of synonyms for a particular search index. By associating semantically equivalent terms with each other, you can respond to a wider range of user-initiated queries in your applications. Realm Realm lets you have simple, powerful local persistence on mobile phones, tablets and IoT devices like Raspberry Pi. The Realm SDKs provide a set of APIs that let developers store and interact with native objects directly, reducing the amount of code required as there is no need for ORMs or learning cryptic database syntax. In addition, we made MongoDB Realm Sync generally available earlier this year, making it easy to synchronize data between local storage on your devices and MongoDB Atlas on the backend. No need to worry about networking code or dealing with conflict resolution as we handle all of that for you. Today, we’re excited to announce support for Unity. You can now use Realm to store your game data, like scores and player state, and sync it automatically across devices. Realm's support for Unity is now Generally Available and ready for production workloads. We're also investing in support for more cross-platform frameworks — the Kotlin Multiplatform and Flutter/Dart SDKs are now both available in Alpha. And finally, the team is working towards Realm Flexible Sync, a new way to synchronize data with more granular control. Flexible Sync will allow you to — Build applications that respond dynamically to user's needs. Let your end users decide what data they need, and when. Use more precise permissions that can adapt over time. Check out this dedicated blog on our upcoming plans for Flexible Sync to learn more. Getting Started With everything we announced today, you can imagine it was a packed keynote! And there is so much more that we didn’t cover. You can get all of the highlights from our new announcements page where you will also find all the resources you need to get started.
The Foundations of IT Modernization
Lately, I’ve been thinking a lot about the term “modernization.” It’s a broad term that means different things to different people. To some, modernization means migrating legacy systems to the cloud. To others, it means rewriting applications, containerizing, or embracing microservices architecture. And to others still, modernization is synonymous with an equally amorphous (and ubiquitous) term: digital transformation. However you define it, modernization is all the rage right now. IDC says these investments are growing at a compound annual growth rate of 15.5%, and will reach $6.8 trillion by 2023 . (yeah, trillion, with a ‘t’). This frenzy of spending on technology, services, and skills is intended to bring aging systems and business processes up to date. In many cases, these investments are urgent and necessary, as companies of all shapes and sizes, in every industry, must accelerate their pace of innovation in order to survive. But the work of modernization is complex, costly, and technically challenging. It’s like renovating every room in a sprawling estate, while you’re still living in it. It’s hard to even know where to start. To that end, I can’t help but think about the words of the CIO of a $30 billion insurance company who had already been on a modernization journey for years. He said: “We tried everything to accelerate innovation...but, in the end, it was our data platform that was holding us back.” In other words, they were spending millions to fix up their estate, adding radiant heat, smart speakers, and a state-of-the-art home theater. But they were building on top of foundations that were first poured when disco was new. (I’m looking at you, relational databases, first conceived and implemented in the 1970s). In the digital economy, companies succeed or fail based on how fast they innovate. More often than not, that innovation takes the form of software and services, which in turn create value by storing, manipulating, and querying data. And what do you use to store, manipulate, and query all that data? Your application data platform. Years ago, that just meant ‘database with some scripts around it.’ Those days are gone. Now, an application data platform has to supply speed, governance, security, availability, and more. So let’s get back my modernization metaphor. You can’t build new solid things on top of creaky, unstable, old things. We all know the old things I’m talking about; databases that make you structure your data in a way that isn’t natural, languages written to be so precise to the computer that they are inscrutable to developers, ‘roach motel’ storage systems that don’t store things in modern, open formats. So if you want to modernize your infrastructure, or modernize your applications, or modernize the way you build software, shouldn’t you first modernize your data platform? Sure, it’s hard to renovate your house. But this is where you live. And if you want that house to last, make sure it’s built on solid foundations. What does that mean? It means that it’s not enough to design and build the right apps. If you want to be truly modern, look at how you input and output your data, how you query, manipulate, and store it, and how you program against it. Get those things right, and you dramatically increase your pace of innovation. No matter where you are in your own modernization journey, it’s not too late to do this. Don’t believe it? Hit me back on Twitter at @MarkLovesTech and I’ll show you how.
The Difference Between R and D
I used to believe that Research and Development (R&D) departments should work in lockstep with Product teams so that they can stay focused on commercially-viable innovations. After all, not every innovation has a market, and not every business has resources to bet on future markets. All of that changed when I met Dr. Michael Cahill, the head of MongoDB Labs in Australia. Michael came to MongoDB through our acquisition of WiredTiger back in 2014, an open source storage engine company he co-founded with Keith Bostic. He holds a PhD in Computer Science and a history of breakthrough innovation. He also has an enlightened point of view on the role of research in any technology company. “Researchers need time and space to pursue the theoretical,” he told me. “We want them to come up with crazy ideas, with much longer time horizons.” Michael is referring to the fundamentally different mindsets required of researchers versus developers. Our developers are focused on new products or features that can make an impact in the next 3-4 quarters. Our researchers are thinking about solving problems that have the potential to reshape entire markets for decades. Big difference. Funding this kind of innovation is challenging for the MBA set, and measuring the ROI of basic research is notoriously difficult. Progress can seem slow and difficult to quantify. Our researchers occupy a space that straddles art and science, industry and academia. They spend a lot of time reading, thinking, and tinkering. Ideas are freely shared, cultivated, iterated, and sometimes abandoned. This is the price of disruptive innovation. In fact, MongoDB would never exist if our founders had set out to simply improve upon relational databases. Instead, they wanted to invent an entirely new way to manage data. It was an ambitious idea that required long-term thinking. An idea that despite MongoDB’s current success, is still only in its infancy. An idea so humongous, Michael Cahill may have even called it “crazy.” Don’t get me wrong. The work of MongoDB Labs is firmly grounded in MongoDB’s core strategy: to constantly improve the way data is stored, accessed, secured and processed. Document databases are only the first act of this play. And I’m certain the next act is being written as we speak, by Michael and his team. Have a different approach to R&D? Think my ideas are “crazy”? Let me hear about it on Twitter at @MarkLovesTech
The Innovation Tax: How much are unproductive and unhappy developers costing you?
I am not someone who believes that developers should be coddled. And I don’t subscribe to a culture of entitlement for developers, or any other part of an organization for that matter, including the C-suite. We are all professional adults operating in the real world. We should treat each other like grownups, regardless of role or responsibility. From the coder to the financial analyst to the sales rep, we all bring our unique value to the company. So execs like me need to strive to understand, appreciate, and foster the critical skills every team member brings to the table. Let’s start with developers, one of my favorite cohorts. We’ve all heard the now overused adage of the digital age: “Every company is becoming a software company.” What this trope is trying to convey is that innovation in the digital space - application development - is a major force in driving new business creation and competitive advantage. The speed with which a new application can be deployed, coupled with the quantity of innovative features in it, is a direct lever on the success of a business. If applications are the currency of the new economy, then development teams are the market makers. In my experience, however, despite the relentless strategic emphasis on speed and innovation in the digital economy, these teams continue to be misunderstood, mismanaged, and marginalized inside both large and small companies. It’s not rational. Worse, it’s incredibly costly. I think about this as a tax on the amount of innovation that a company can produce. Companies pay this tax when they fail to understand the nature of the work developers do, or provide a safe and productive environment for them to do it. And if you don’t get that right, you’re not going to be in this game for very long. Though I don’t write any production code these days, at heart, I’m still a developer. And MongoDB leads hundreds of developers spread across tens of teams, so I’m constantly exposed to developer issues. Over the course of my career, I’ve learned a few things about how – and how not – to cultivate a productive culture for developers. This will be an ongoing discussion, for sure. But to get things started, here are a few things to think about if you’re trying to reduce the “Innovation Tax” your currently paying: Give your developers business context Don’t insult the intelligence or maturity of your developers. They can – and must – understand the business rationale for their work. In fact, painting the strategic target for developers will result in a better work product as they align their key decisions in the architecture and design experience of your software. Once they understand the business context, they’ll find better ways of achieving it bottoms-up than any tops-down leader, even a CTO like me, possibly can. Respect tech debt — and pay down the principal In my experience, the single biggest source of low morale among developers is the combination of too much tech debt and management’s dismissal of it. Taking on some debt to get a release out is fine - if you do it knowingly and pay down the principal later. But leaders who don’t pay attention to mounting debt demonstrate in a very visceral way to developers that they’ve become Gantt-chart leaders, and lost touch with their ethos of engineering. Developers don’t do well with cognitive dissonance, so when you tell them to build the next great thing on top of a dumpster fire, you lose credibility, they lose patience, and your company loses money as the pace of innovation slows to a crawl. Understand what your developers are really doing I could talk about this one for days, but the bottom line is that if leaders don’t understand how developers spend their time, they have no business leading the teams. It’s easy to just focus on new features, but you must acknowledge and address the fact that adjacent work like maintaining databases or a legacy staging environment is pure drudgery that provides no innovation value, costs a fortune in developer time, and saps morale. Listen to the developers when they say that they need to revamp an adjacent or dependent system to understand why it’s important. Remove OKRs and vanity metrics Top-down innovation is an oxymoron. You have to trust that developers want nothing more than to see their work come to life. The more management tells them how to do their job – through objectives and key results or any other key performance indicators – the more they limit the scope of innovation. Paint the target, then get out of the way. Align your goals This goes back to providing business context. Leaders and developers need to believe they are working together toward the same goal. An oppositional relationship takes developers out of flow, and you can lose a whole day of productivity from a single negative interaction. Again, I’m not advocating coddling; developers have their part to play in the complex recipe that builds a successful company, just like everybody else. But for that to work, you must align business, technical, and organizational goals, and build honest and transparent relationships with your devs. Like I said, I could riff on this topic for many more days (or posts). And keep in mind, mismanaging developers is just one form of innovation tax. I’ll be exploring other hidden levies in this space over the coming months. But hopefully this starter list of dos and don’ts gets the conversation going. Please feel free to add to it (or subtract from it) on Twitter at @MarkLovesTech .
The Evolution of Data
It used to be so simple. Not that long ago, the universe of corporate data was a fraction of its current size. We fit most of it into rigid little rows and columns. And we didn’t ask all that much of it: some transaction processing, a few charts and graphs, a little business intelligence. I’m exaggerating, of course. We’ve been pushing the boundaries of data processing since at least 1964, when SABRE, the world’s first airline passenger system, launched on two IBM mainframes and 1,500 terminals, processing an average of one transaction per second. But it’s not hyperbole to say that today’s data would barely recognize its early ancestors. First of all, there is more of it – 59 zettabytes and counting. Second, the very definition of data has changed dramatically, expanding well beyond payroll records and stock prices to include diverse data types like weblogs, fraud scores, maps, and fingerprints. But perhaps the biggest change has been the role data plays in the enterprise. Data has always been used to inform business strategy. But today, data often is the business strategy. Consider this: 20 years ago, there was no such thing as a Chief Data Officer. Today? Almost two-thirds of Fortune 1000 companies have one. Why? Because we are asking more of our data than ever before. And rightly so. In the digital economy, every company competes on the basis of insight-driven innovation. More and more, those innovations take the form of software built around clever algorithms. And the raw material to both craft and execute those algorithms is data. All of which means the ability to efficiently and effectively manage data -- at speed -- is a strategic imperative for any company, in any industry. And yet, even as the volume, variety, and strategic importance of data has rapidly evolved over the last two decades, many enterprises haven’t changed how they manage it. Of course, I’m talking about the continued use of legacy relational databases, which are too rigid and don’t scale well enough to handle the demands of modern application development. Solving this problem was the entire reason for the “NoSQL” movement in the late 2000s, and MongoDB’s invention of the document-oriented database in the first place. But I’m talking about something bigger; a longer-term trend that demands a fresh look at the way we work with data. Our customers are telling us that the fundamental requirements of their various data sets aren’t just changing, they’re converging. This is a surprise, and a reversal of the trend of siloization and specialized tools for the last 50 years. Let’s take a step back: For decades, enterprises have maintained systems of record and systems of engagement. Systems of record are foundational, mission-critical, sources of truth that are accessed primarily by internal programs and users. Systems of engagement are the digital interfaces with which customers and employees interact. And recently we have seen the addition of systems of insight, which combine data from various sources to inform decision making across the enterprise. For a long time, each system lived on different computers, had different data management requirements, and were funded by different departments. But that is changing. With the hard and fast divisions between back office and front office dissolving, we now need all of our data systems to do everything. They need to be both fast and accurate. They need to be both accessible and secure. They need to handle both transactions and analytics. In particular, with the rise of model training and inference, a different kind of analytics is arriving; one where it is programs that are asking the systems of insight questions and reacting to them in real time, rather than humans asking questions and then writing programs to implement them. This is a fundamental shift; so fundamental that you could liken it to the change from the IBM 7090s that powered SABRE to those that (will?) power SKYNET. This “convergence” of data requirements is both a challenge and an opportunity. Just like document databases enabled us to rethink how data was accessed and stored, convergence is forcing us to rethink the systems we use to manage data across the enterprise yet again. Companies across the industry, from Snowflake to Databricks to MongoDB, and every cloud provider, are working to provide the systems that let companies get more value from their data, using microservices-based networks or programs that drive informed, real-time decision making. Interestingly, this comes at a time when most companies are undergoing radical digital transformation projects in order to become innovation-powered, software-driven, and cloud-based. In other words, even though everyone is already quite busy, there has never been a better time to think beyond the database, and architect an actual “data platform” that can process, store, secure, and analyze data in real-time, across all the relevant data sets - either without copying the data or making such copying invisible. Over the coming months, I’ll be sharing more about what these data platforms look like and how they support the creation of modern applications. I’ll also try to peer into the foggy future with you, looking at the constraints and freedoms of the modern enterprise, and predicting what products will be built to address that matrix. But for now, I’m interested in hearing what you’re seeing under the hood of your data(base) estate. Are you experiencing a similar convergence of data requirements? If so, is it factored into your digital transformation strategy? And if not, what changes are you seeing in the way you use data across the enterprise? Please reach out to me here or on Twitter, at @MarkLovesTech - because I really do. Thanks for reading - may the data odds be in your favor until we chat again. Mark