Drowning in Data: Why It's Time to End the Healthcare Data Lake
From digital check-ins, to connected devices and telehealth programs, patients expect the benefits of a more digitized healthcare experience. At the same time, they’re also demanding a more personalized approach from healthcare providers. This duality - the need to provide a more convenient experience with one that’s more tailored to the patient - is fueling a wave of technology modernization efforts and the replacement of monolithic legacy IT systems. With limited re-use outside of the context they were built for and a reliance on nightly batch processing, legacy IT systems fail to deliver the services healthcare IT teams need or provide the experiences patients demand. Modernization should come with a move to microservices that can be used by multiple applications, agile teams that embrace domain driven design principles, and event busses like Kafka to deliver real-time data and functionality to users. While this transformation is occurring, there’s an 800lb gorilla not being widely addressed. Analytics. What the healthcare industry doesn’t want to talk about, is how costly analytics has become; the people, the software, the infrastructure, and particularly how difficult it is to move data in and out of data lakes and warehouses. It's hindering the industry’s ability to deliver insights to patients and providers in a timely and efficient manner. And yet, so many organizations are modernizing their analytics data warehouses and data lakes with an approach that simply updates the underlying technology. It’s a lift-and-shift effort of tremendous scale and cost, but one that is not addressing the underlying issues preventing the speedy delivery of meaningful insights. Drowning in data: A 1980s model in the 2020s While the business application landscape has changed, healthcare is still clinging to the same 1980’s paradigm when it comes to analytics data. It started by physically moving all the data from transactional systems into a single data warehouse or data lake (or worse, both), so as not to disrupt the performance of business applications by executing analytics queries against the transactional database. Eventually, as data warehouses had enough relational tables and data in them, queries began to slow down, and even time-out before delivering results to end users. This gave rise to data marts, yet another database to copy the warehouse data into, using a star schema model to return query results more efficiently than in the relational warehouse. In the last and current iteration of analytics data platforms, warehouses and data marts became augmented, and were even replaced in some cases, with data lakes. Technologies like Hadoop promised a panacea where all sorts of structured and unstructured data could be stored, and where queries against massive datasets could be executed. In reality it turned out to be a costly distraction, and one that did not make an organization's data easier to work with, or provide real-time data insights. Hence why it earned the nickname “data jail”. It was hard to load data into, and even harder to get data out of. New technology, same challenges While Hadoop and other technologies did not last long, they hung around just long enough to negatively alter the trajectory of many analytics shops, which are now investing heavily in migrating away from Hadoop, to cloud-based platforms. But, are these cloud alternatives solving the challenges of the Hadoop era? Can your organization rapidly experiment, innovate and serve up data insights from your data lake? Can you go from an idea to delivery in days? Or, is it weeks, months even? Despite the significant amounts of time, money and people required to load data into these behemoth cloud data stores, they still exhibit the same challenges as their Hadoop-era predecessors. They are difficult to load and even more difficult to make changes to. They can never realistically offer real-time or even near-real-time processing, the response time that patients and providers expect. Worse, they contain so much data, that making sense of it is a task often left to either a sophisticated add-on like AWS HealthLake, or specialized data engineering and data science teams. To add to this, the cloud based analytics systems are typically managed by a single team that’s responsible for collecting, understanding and storing data from all of the different domains within an organization. This is what we like to call a modernized monolith, the pairing of updated technology with a failure to fundamentally address or improve the overall limitations or constraints of a system or process. It’s an outdated and inefficient approach that’s simply been “lifted and shifted” from one technology to another. Many data lake implementations take a modernized monolithic approach which, like their predecessors, results in a bottleneck and difficulty in getting information out, once it goes in. In a world where data is at the center of every innovative business, and real-time analytics is top-of-mind for executives, product owners and architects alike, most data lakes don’t deliver. Transforming your organization into a data-driven enterprise requires a more agile approach to managing and working with ever-growing sums of data. The rise of the operational data layer — an ODS renaissance To provide meaningful insights to patients in a timely and efficient manner, two very important things need to happen. Healthcare organizations need to overcome the limitations of legacy systems, and they need to make sense of a lot of very complex data. A lift-and-shift approach migrating data into a data lake will not solve these problems. In addition, it’s not feasible or advisable to spend tens, or even hundreds of millions of dollars to replace legacy systems as a precursor to a digital engagement strategy. The competition will leap-frog you before your efforts are even half complete. So, what can be done? Can your organization make better sense of its data, and at the same time mitigate the issues legacy systems impose? Can this be done without a herculean effort? The answer is yes. The solution is an operational data layer (ODL) , formerly known as the operational data store. It’s a method that’s been tried and tested by major corporations, and is the underlying technology that powers many of the apps you interact with on your phone. An ODL lets you build new features without existing system limitations. It lets you summarize, analyze, and respond to data events, in real-time. It helps you migrate from legacy systems, without incurring the cost and complexity of replacing legacy systems. It can give your teams the speed and agility that working against a data lake will simply never have. Data lakes and warehouses have their place, and the kinds of long-term data insights and data science benefits that can be gleaned from them are significant. The challenge, however, is reacting in real-time, and serving those insights to patients, quickly. An ODL strategy offers the best, most cost and time efficient approach to mitigate legacy system issues, without the pain of replacing legacy systems. Investing in an ODL strategy will both solve your legacy modernization dilemma, and it will help you deliver real-time data and analytics at the speed of an agile software delivery team. MongoDB is an ideal ODL provider . Not only does it have the underlying, flexible document-based database, but it is also an application data platform, empowering your developers to focus on building features, not managing databases and data. If you’re interested in learning about how MongoDB has enabled organizations large and small to successfully implement ODL strategies and tackle other burning healthcare issues, click here .
MongoDB & Bosch: Discussion on AIoT
For more than a decade, the digital transformation of industry has been focused on the technologies that make up the Internet of Things (IoT). As AI and machine learning tech mature, a new field has appeared that combines these trends: AIoT, the Artificial Intelligence of Things, which applies AI to the data collected by IoT devices. Among the firms pioneering this space is the engineering and industrial giant Bosch, which has long been a leader in IoT. The move to AIoT has allowed for Bosch to create smart products that either have intelligence built in, or have “swarm intelligence” in their back end that allows for the collection of data that is used to improve the products. In April 2021, Mark Porter, CTO at MongoDB, and Dirk Slama, VP of Co-innovation and IT/IoT Alliances at Bosch, sat down to discuss AIoT. Their conversation saw them touch on what MongoDB and Bosch are working on around AIoT, and where they see the future of AIoT heading. Bosch’s new focus on AIoT has enhanced their need for a flexible, modern data platform such as MongoDB. IoT devices collect enormous amounts of data; as Bosch adds sensors and new types of data to their products, MongoDB has allowed it to adapt quickly without having to go through a schema redesign whenever they need to implement a change to their products. As part of their efforts to progress AIoT technology, Bosch and other companies recently created the AIoT User Group , an initiative open to anyone. The group’s goal is to bring end users working on AIoT business and use cases together with technology experts to share best practices around AIoT solutions. This co-creation approach allows for the rapid utilization of best practices to try out and develop new ideas and technologies. Porter and Slama’s conversation covered many AIoT topics — and a glimpse at the technology’s next steps. For instance, Slama wants to see agility added to AIoT without losing control. In AIoT, there are many features that must be perfect on day one; but there are also a lot of features where you want to continuously improve system performance, which requires an agile approach. For Mark Porter and Dirk Slama's full conversation, check out the video below!
The Rise of the Strategic Developer
The work of developers is sometimes seen as tactical in nature. In other words, developers are not often asked to produce strategy. Rather, they are expected to execute against strategy, manifesting digital experiences that are defined by the “business.” But that is changing. With the automation of many time-consuming tasks -- from database administration to coding itself -- developers are now able to spend more time on higher value work, like understanding market needs or identifying strategic problems to solve. And just as the value of their work increases, so too does the value of their opinions. As a result, many developers are evolving, from coders with their heads-down in the corporate trenches to highly strategic visionaries of the digital experiences that define brands. “I think the very definition of ‘developer’ is expanding,” says Stephen “Stennie” Steneker, an engineering manager on the Developer Relations team at MongoDB. “It’s not just programmers anymore. It’s anyone who builds something.” Stennie notes that the learning curve needed to build something is flattening. Fast. He points to an emerging category of low code tools like Zapier, which allows people to stitch web apps together without having to write scripts or set up APIs. “People with no formal software engineering experience can build complex automated workflows to solve business problems. That’s a strategic developer.” Many other traditional developer tasks are being automated as well. At MongoDB, for example, we pride ourselves on removing the most time-consuming, low-value work of database administration. And of course, services like GitHub Copilot are automating the act of coding itself. So what does this all mean for developers? A few things: First, move to higher ground. In describing one of the potential outcomes of GitHub Copilot, Microsoft CTO Kevin Scott said, ““It may very well be one of those things that makes programming itself more approachable.” When the barriers to entry for a particular line of work start falling, standing still is not an option. It’s time to up your strategic game by offering insight and suggestions on new digital experiences that advance the objectives of the business. Second, accept more responsibility. A strategic developer is someone who can conceive, articulate, and execute an idea. That also means you are accountable for the success or failure of that idea. And as Stennie reminded me, “There are more ways than ever before to measure the success of a developer’s work.” And third, never stop skilling. Developers with narrow or limited skill sets will never add strategic value, and they will always be vulnerable to replacement. Like software itself, developers need to constantly evolve and improve, expanding both hard and soft skills. How do you see the role of the developer evolving? Any advice for those that aspire to more strategic roles within their organizations? Reach out and let me know what you think at @MarkLovesTech .
몽고DB에 대한 4가지 오해
일년 전 코로나19가 한창일 때 몽고DB CEO 데브 이티체리아가 나를 CTO(최고 기술 책임자)로 채용했다. 솔직히 나는 데이터베이스와 몽고DB에 대해 이미 모든 것을 알고 있다고 생각했다. 데이터 베이스 업계에서 일한지도 벌써 32년이나 됐고, 몽고DB 이사회에 참여하면서 몽고DB 제품을 두루 사용해왔기 때문이다. 물론 실사도 진행했고 경영진과 미팅을 가졌으며 실적 보고서와 제품 로드맵을 분석해왔다. 이렇게 지식을 쌓았음에도 불구하고 지난 한 해 동안 몽고DB에서 CTO로 일하면서 나는 내가 얼마나 잘못된 선입견을 갖고 있었는지를 깨닫게 됐다. 그리고 나처럼 몽고DB에 대해 오해하고 있는 사람이 많이 있을 것이라는 생각에 이르렀다. 이에 지난 일 년간 내가 알게 된 4가지 사실을 이 블로그를 통해 공유하여 사람들의 인식을 바로잡고자 한다. 첫째, 몽고DB의 비전은 현 세대의 관계형 데이터베이스가 되는 것이 아니라는 것이다. 나는 수년 전부터 몽고DB가 오라클보다 한층 개선된 최신 버전을 추구한다고 생각해왔다. 다시 말해, 몽고DB가 오라클의 광범위한 입지와 더불어 업계 전형으로 오랜 기간 군림해왔던 여타 상용 RDBMS와 경쟁하려는 듯 보였다. 하지만 이런 내 추측은 완전히 벗겨 갔다. 몽고DB의 궁극적인 목표는 모든 유형의 구식 레거시 데이터 베이스 기술에서 완전히 탈피하는 것이었다. 그리고 이러한 목표를 진화가 아닌 혁명으로 달성하고자 했다. 몽고DB의 설립자들은 세계 최고의 속도와 확정성을 갖춘 퍼시스턴트 저장소에서 한 걸음 더 나아가, 다른 방식으로 프로그래밍 및 운영이 가능한 저장소를 꿈꿨다. 내장된 문서와 구조가 고가용성을 갖춘 무제한에 가까운 자동 배포 기능과 결합하면 데이터 작업, 애플리케이션 구축, 그리고 프로덕션 단계에서의 애플리케이션 실행 방식이 근본적으로 달라지게 된다. 오라클과 SQL*서버 등은 아직도 열과 행이라는 51년 된 E.F 코드의 비전에 의존하고 있다. 데이터의 고가용성과 배포 능력을 확보하기 위해서는 추가 기능을 비롯해 옵션 패키지, 베일링 와이어, 그리고 덕트 테이프가 필요하다. 여기에 데이터베이스 관리자도 상당수 투입해야 하기에 만만치 않은 비용이 들게 된다. 그럼에도 불구하고 많은 이들이 여전히 기술적 우위를 추구하고 있다. 나도 이렇게 잘못 생각하고 있었다. 레거시 데이터 저장소를 능가하는 몽고DB의 지속 가능한 경쟁 우위를 고려할 때, 레거시 데이터 저장소 제품과 경쟁한다는 건 사실상 무의미하다. 대신 우리는 최신 애플리케이션을 구축하는 오늘날 개발자들의 최신 요구사항에 초점을 맞춘다. 지금의 개발자들은 고유 언어 개발, 프로덕션 환경으로의 안정적인 배치, 초고속 반복 작업을 통해 자신만의 경쟁 우위를 창출해야 한다. 그리고 전 세계가 이를 주목하고 있다. db 엔진 사이트 만 하더라도, 오라클과 SQL*서버는 하락세에 접어들고 있는데 반해 몽고DB는 상승세를 타고 있는 것을 알 수 있다. 여기서 내가 두 번째로 깨달은 점이 있다. 몽고DB는 개발자가 직접 개발자를 위해 만든 제품이라는 것이다. 나는 몽고DB가 굉장히 빠르고 프로그래밍 하기도 매우 쉽다는 것을 익히 알고 있었다. 어느 날은 회의가 너무 지루해서 (몽고DB에서도 이런 일은 일어난다!) 아틀라스 데이터베이스를 구축해 350MB의 데이터를 로드했다. 그리고 몽고DB의 콤파스 데이터 디스커버리 툴, 내장형 분석 집계 파이프라인, 몽고DB 차트 패키지, 웹페이지의 내장 라이브 차트를 다운로드해 공부했다. 시작부터 끝까지 총 19분이 걸렸다. 이러한 환경을 엔지니어를 위해 개발하려면 제품에 기능을 추가할 때 발생하는 소소한 오류에 자유롭게 집중할 수 있는 엔지니어들이 개발해야 한다. 40여 년 전, 소프트웨어 기획과 관리를 처음 접하게 되었는데 내 링크드인 계정을 보면 이 쪽 업계에서 아주 다양한 일들을 해왔다는 것을 알 수 있을 것이다. CTO직을 맡은 지 일년이 지난 지금, 나는 우리의 엔지니어링 기술과 제품이 내가 그 간 일했던 다른 기업들의 제품과 다르며 월등하다고 장담할 수 있다. 몽고DB의 경영진은 엔지니어링과 제품 부서에게 우리의 목표와 원하는 결과를 대략적으로 설명한 다음 협력을 통해 세부 로드맵을 마련하고, 또 로드맵을 분기별로 업데이트하며, 사소한 것까지 관리할 필요 없이 우리의 목표를 가장 잘 충족하는 로드맵을 마련한다. 3~5년짜리 프로젝트를 한다고 하더라도 두려울 게 없다. 일례로, 멀티 클라우드를 개발하는 데는 3년 이상이 걸렸다. 과거 일했던 다른 기업들과는 달리, 몽고DB는 기술 부채를 비밀로 덮어두기 보다는 기술 부채의 생성과 상환에 적극 대응하고 있다. 우리는 제품 팀과 엔지니어링 팀에게 이러한 상황 전체를 가감없이 솔직하게, 그리고 공개적으로 알린다. 한 가지 더 중요한 사실은, 프로세스를 생성하는 즉시 개선할 수 있는(프로세스 삭제 포함) 역량 있는 프로그램 관리 팀이 있다는 것이다. 간략히 말해, 우리는 실무팀에게 목표를 제시하고, 목표를 달성하는 방식과 시기를 주체적으로 결정하게 한다. 실무 팀은 이 과정에서 자신들이 사용할 툴까지도 직접 설계한다. 진정한 상향식 엔지니어링인 것이다. 몽고DB의 엔지니어들은 스스로 가치 있고, 경영진으로부터 이해 받고 있다고 느낀다. 이를 통해 엔지니어들은 고객에게도 이러한 감정을 불러일으킬 수 있는 다양한 기능을 만들 수 있다. 이를테면 통합 쿼리 언어나 콘솔 기반의 실시간 분석 및 차트 작성 또는 모든 네트워킹 크러프트를 사용자 대신 처리하는 멀티 리전/멀티 클라우드 클러스터 기능 등이 있다. 그리고 여기서 세 번째로 알게 된 사실은, 몽고DB가 가장 까다로운 미션 크리티컬 애플리케이션에도 사용할 수 있도록 설계됐다는 점이다. 속도가 빠르냐고? 그렇다. 사용 편의성은? 당연히 편리하다. 그렇다면 미션 크리티컬은? 10년 전에 진행된 대규모의 학생 데이터 프로젝트에서 버전 2를 사용할 때까지만 해도 몽고DB를 이렇게 생각하진 않았다. 몽고DB는 작업에 필요한 데이터의 양과 데이터 수집 및 처리 속도를 위해 선택할 수 있는 유일한 데이터 저장소였지만, 24시간 상시 가동 환경에서 구축하여 사용하기에는 턱없이 부족했다. 몽고DB는 2010년대 초에 자체 역량을 넘어섰는데, 우리의 역량과 시장의 기대 사이에는 차이가 있었다. 이 점이 매우 힘들었다. 다른 데이터베이스는 시스템과 운영 방식을 강화하기까지 30년 이상이 걸렸다. 하지만 몽고DB는 이를 단 5년만에 해낸 것이다. 버전 3에는 새로운 저장 엔진, 풀 ACID 트랜잭션, 그리고 검색 기능을 추가했다. 이를 기반으로 버전 4를 만들었다. 그후 버전 5가 MongoDB.Live 2021 컨퍼런스에서 정식 출시됐다.(2021년 7월) 몽고DB에 입사할 당시 나는 이러한 상황을 알고 있었지만 실질적으로 체감하진 못했다. 그러다가 몽고DB 플랫폼이 제공하는 보안, 내구성, 가용성, 확장성, 운용성(그 외에도 개발자들이 선호하는 모든 기능)이 빠르게 변화하는 엔터프라이즈 애플리케이션을 설계하는 데 이상적이라는 것을 알게 되었다. 몽고DB 고객 목록에서 그 증거를 확인할 수 있었다. 우리의 고객 목록은 마치 결제, IoT 애플리케이션, 콘텐츠 관리, 실시간 분석 등 핵심 시스템을 운영하는 주요 글로벌 은행, 소매업체, 통신사의 명사록과 같았다. 이들은 자사의 모든 사업, 모든 주요 클라우드, 사내 현장, 노트북 등에서 몽고DB 데이터베이스, 데이터 레이크, 분석, 검색, 모바일 제품을 사용하고 있다. 여기서 네번째로 알게 된 마지막 사실은, 몽고DB가 단순한 데이터베이스가 아니라는 점이다. 물론 데이터베이스가 몽고DB의 핵심 사업이긴 하다. 하지만 몽고DB는 이제 엔터프라이즈급의 미션 크리티컬 애플리케이션 데이터 플랫폼을 제공하고 있다. 최대 규모의 디지털 단지에서도 최신 데이터 요건을 관리하고 속도나 보안을 저해하지 않으면서 회사가 추구하는 수준으로 확장 가능한 통합 제품군인 것이다. 나는 몽고DB 제품을 처음 접한 날부터 몽고DB 팀과 이들이 만든 제품에 감탄을 금치못했고 엄청난 존경심을 갖게 됐다. 어쨌건 나는 누가 뭐라 해도 개발자이다. 또 항상 개발자들은 나를 “이해한다”는 느낌이 들었다. 하지만 지금 알고 있는 사실을 그때 알았더라면 진작에 몽고DB에 입사했을 것이다. 이력서를 손에 들고 몽고DB 건물 입구에서 밤을 새고 기다렸을 수도 있다. 누가 알랴? 이 글을 읽는 많은 사람들이 그렇게 할지? 훌륭한 기업 문화를 갖춘 기업에서 훌륭한 제품을 만드는 것이 얼마나 보람 있고 흥분되는 일인지 깨달으면서 말이다. 1년 후에 다시 한번 글을 올려 그 때의 상황을 전할 생각이다. 그때까지 이 포스팅에 댓글을 남기거나 MarkLovesTech 로 연락 주기 바란다.
The Foundations of IT Modernization
Lately, I’ve been thinking a lot about the term “modernization.” It’s a broad term that means different things to different people. To some, modernization means migrating legacy systems to the cloud. To others, it means rewriting applications, containerizing, or embracing microservices architecture. And to others still, modernization is synonymous with an equally amorphous (and ubiquitous) term: digital transformation. However you define it, modernization is all the rage right now. IDC says these investments are growing at a compound annual growth rate of 15.5%, and will reach $6.8 trillion by 2023 . (yeah, trillion, with a ‘t’). This frenzy of spending on technology, services, and skills is intended to bring aging systems and business processes up to date. In many cases, these investments are urgent and necessary, as companies of all shapes and sizes, in every industry, must accelerate their pace of innovation in order to survive. But the work of modernization is complex, costly, and technically challenging. It’s like renovating every room in a sprawling estate, while you’re still living in it. It’s hard to even know where to start. To that end, I can’t help but think about the words of the CIO of a $30 billion insurance company who had already been on a modernization journey for years. He said: “We tried everything to accelerate innovation...but, in the end, it was our data platform that was holding us back.” In other words, they were spending millions to fix up their estate, adding radiant heat, smart speakers, and a state-of-the-art home theater. But they were building on top of foundations that were first poured when disco was new. (I’m looking at you, relational databases, first conceived and implemented in the 1970s). In the digital economy, companies succeed or fail based on how fast they innovate. More often than not, that innovation takes the form of software and services, which in turn create value by storing, manipulating, and querying data. And what do you use to store, manipulate, and query all that data? Your application data platform. Years ago, that just meant ‘database with some scripts around it.’ Those days are gone. Now, an application data platform has to supply speed, governance, security, availability, and more. So let’s get back my modernization metaphor. You can’t build new solid things on top of creaky, unstable, old things. We all know the old things I’m talking about; databases that make you structure your data in a way that isn’t natural, languages written to be so precise to the computer that they are inscrutable to developers, ‘roach motel’ storage systems that don’t store things in modern, open formats. So if you want to modernize your infrastructure, or modernize your applications, or modernize the way you build software, shouldn’t you first modernize your data platform? Sure, it’s hard to renovate your house. But this is where you live. And if you want that house to last, make sure it’s built on solid foundations. What does that mean? It means that it’s not enough to design and build the right apps. If you want to be truly modern, look at how you input and output your data, how you query, manipulate, and store it, and how you program against it. Get those things right, and you dramatically increase your pace of innovation. No matter where you are in your own modernization journey, it’s not too late to do this. Don’t believe it? Hit me back on Twitter at @MarkLovesTech and I’ll show you how.
The Difference Between R and D
I used to believe that Research and Development (R&D) departments should work in lockstep with Product teams so that they can stay focused on commercially-viable innovations. After all, not every innovation has a market, and not every business has resources to bet on future markets. All of that changed when I met Dr. Michael Cahill, the head of MongoDB Labs in Australia. Michael came to MongoDB through our acquisition of WiredTiger back in 2014, an open source storage engine company he co-founded with Keith Bostic. He holds a PhD in Computer Science and a history of breakthrough innovation. He also has an enlightened point of view on the role of research in any technology company. “Researchers need time and space to pursue the theoretical,” he told me. “We want them to come up with crazy ideas, with much longer time horizons.” Michael is referring to the fundamentally different mindsets required of researchers versus developers. Our developers are focused on new products or features that can make an impact in the next 3-4 quarters. Our researchers are thinking about solving problems that have the potential to reshape entire markets for decades. Big difference. Funding this kind of innovation is challenging for the MBA set, and measuring the ROI of basic research is notoriously difficult. Progress can seem slow and difficult to quantify. Our researchers occupy a space that straddles art and science, industry and academia. They spend a lot of time reading, thinking, and tinkering. Ideas are freely shared, cultivated, iterated, and sometimes abandoned. This is the price of disruptive innovation. In fact, MongoDB would never exist if our founders had set out to simply improve upon relational databases. Instead, they wanted to invent an entirely new way to manage data. It was an ambitious idea that required long-term thinking. An idea that despite MongoDB’s current success, is still only in its infancy. An idea so humongous, Michael Cahill may have even called it “crazy.” Don’t get me wrong. The work of MongoDB Labs is firmly grounded in MongoDB’s core strategy: to constantly improve the way data is stored, accessed, secured and processed. Document databases are only the first act of this play. And I’m certain the next act is being written as we speak, by Michael and his team. Have a different approach to R&D? Think my ideas are “crazy”? Let me hear about it on Twitter at @MarkLovesTech
The Innovation Tax: How much are unproductive and unhappy developers costing you?
I am not someone who believes that developers should be coddled. And I don’t subscribe to a culture of entitlement for developers, or any other part of an organization for that matter, including the C-suite. We are all professional adults operating in the real world. We should treat each other like grownups, regardless of role or responsibility. From the coder to the financial analyst to the sales rep, we all bring our unique value to the company. So execs like me need to strive to understand, appreciate, and foster the critical skills every team member brings to the table. Let’s start with developers, one of my favorite cohorts. We’ve all heard the now overused adage of the digital age: “Every company is becoming a software company.” What this trope is trying to convey is that innovation in the digital space - application development - is a major force in driving new business creation and competitive advantage. The speed with which a new application can be deployed, coupled with the quantity of innovative features in it, is a direct lever on the success of a business. If applications are the currency of the new economy, then development teams are the market makers. In my experience, however, despite the relentless strategic emphasis on speed and innovation in the digital economy, these teams continue to be misunderstood, mismanaged, and marginalized inside both large and small companies. It’s not rational. Worse, it’s incredibly costly. I think about this as a tax on the amount of innovation that a company can produce. Companies pay this tax when they fail to understand the nature of the work developers do, or provide a safe and productive environment for them to do it. And if you don’t get that right, you’re not going to be in this game for very long. Though I don’t write any production code these days, at heart, I’m still a developer. And MongoDB leads hundreds of developers spread across tens of teams, so I’m constantly exposed to developer issues. Over the course of my career, I’ve learned a few things about how – and how not – to cultivate a productive culture for developers. This will be an ongoing discussion, for sure. But to get things started, here are a few things to think about if you’re trying to reduce the “Innovation Tax” your currently paying: Give your developers business context Don’t insult the intelligence or maturity of your developers. They can – and must – understand the business rationale for their work. In fact, painting the strategic target for developers will result in a better work product as they align their key decisions in the architecture and design experience of your software. Once they understand the business context, they’ll find better ways of achieving it bottoms-up than any tops-down leader, even a CTO like me, possibly can. Respect tech debt — and pay down the principal In my experience, the single biggest source of low morale among developers is the combination of too much tech debt and management’s dismissal of it. Taking on some debt to get a release out is fine - if you do it knowingly and pay down the principal later. But leaders who don’t pay attention to mounting debt demonstrate in a very visceral way to developers that they’ve become Gantt-chart leaders, and lost touch with their ethos of engineering. Developers don’t do well with cognitive dissonance, so when you tell them to build the next great thing on top of a dumpster fire, you lose credibility, they lose patience, and your company loses money as the pace of innovation slows to a crawl. Understand what your developers are really doing I could talk about this one for days, but the bottom line is that if leaders don’t understand how developers spend their time, they have no business leading the teams. It’s easy to just focus on new features, but you must acknowledge and address the fact that adjacent work like maintaining databases or a legacy staging environment is pure drudgery that provides no innovation value, costs a fortune in developer time, and saps morale. Listen to the developers when they say that they need to revamp an adjacent or dependent system to understand why it’s important. Remove OKRs and vanity metrics Top-down innovation is an oxymoron. You have to trust that developers want nothing more than to see their work come to life. The more management tells them how to do their job – through objectives and key results or any other key performance indicators – the more they limit the scope of innovation. Paint the target, then get out of the way. Align your goals This goes back to providing business context. Leaders and developers need to believe they are working together toward the same goal. An oppositional relationship takes developers out of flow, and you can lose a whole day of productivity from a single negative interaction. Again, I’m not advocating coddling; developers have their part to play in the complex recipe that builds a successful company, just like everybody else. But for that to work, you must align business, technical, and organizational goals, and build honest and transparent relationships with your devs. Like I said, I could riff on this topic for many more days (or posts). And keep in mind, mismanaging developers is just one form of innovation tax. I’ll be exploring other hidden levies in this space over the coming months. But hopefully this starter list of dos and don’ts gets the conversation going. Please feel free to add to it (or subtract from it) on Twitter at @MarkLovesTech .
The Evolution of Data
It used to be so simple. Not that long ago, the universe of corporate data was a fraction of its current size. We fit most of it into rigid little rows and columns. And we didn’t ask all that much of it: some transaction processing, a few charts and graphs, a little business intelligence. I’m exaggerating, of course. We’ve been pushing the boundaries of data processing since at least 1964, when SABRE, the world’s first airline passenger system, launched on two IBM mainframes and 1,500 terminals, processing an average of one transaction per second. But it’s not hyperbole to say that today’s data would barely recognize its early ancestors. First of all, there is more of it – 59 zettabytes and counting. Second, the very definition of data has changed dramatically, expanding well beyond payroll records and stock prices to include diverse data types like weblogs, fraud scores, maps, and fingerprints. But perhaps the biggest change has been the role data plays in the enterprise. Data has always been used to inform business strategy. But today, data often is the business strategy. Consider this: 20 years ago, there was no such thing as a Chief Data Officer. Today? Almost two-thirds of Fortune 1000 companies have one. Why? Because we are asking more of our data than ever before. And rightly so. In the digital economy, every company competes on the basis of insight-driven innovation. More and more, those innovations take the form of software built around clever algorithms. And the raw material to both craft and execute those algorithms is data. All of which means the ability to efficiently and effectively manage data -- at speed -- is a strategic imperative for any company, in any industry. And yet, even as the volume, variety, and strategic importance of data has rapidly evolved over the last two decades, many enterprises haven’t changed how they manage it. Of course, I’m talking about the continued use of legacy relational databases, which are too rigid and don’t scale well enough to handle the demands of modern application development. Solving this problem was the entire reason for the “NoSQL” movement in the late 2000s, and MongoDB’s invention of the document-oriented database in the first place. But I’m talking about something bigger; a longer-term trend that demands a fresh look at the way we work with data. Our customers are telling us that the fundamental requirements of their various data sets aren’t just changing, they’re converging. This is a surprise, and a reversal of the trend of siloization and specialized tools for the last 50 years. Let’s take a step back: For decades, enterprises have maintained systems of record and systems of engagement. Systems of record are foundational, mission-critical, sources of truth that are accessed primarily by internal programs and users. Systems of engagement are the digital interfaces with which customers and employees interact. And recently we have seen the addition of systems of insight, which combine data from various sources to inform decision making across the enterprise. For a long time, each system lived on different computers, had different data management requirements, and were funded by different departments. But that is changing. With the hard and fast divisions between back office and front office dissolving, we now need all of our data systems to do everything. They need to be both fast and accurate. They need to be both accessible and secure. They need to handle both transactions and analytics. In particular, with the rise of model training and inference, a different kind of analytics is arriving; one where it is programs that are asking the systems of insight questions and reacting to them in real time, rather than humans asking questions and then writing programs to implement them. This is a fundamental shift; so fundamental that you could liken it to the change from the IBM 7090s that powered SABRE to those that (will?) power SKYNET. This “convergence” of data requirements is both a challenge and an opportunity. Just like document databases enabled us to rethink how data was accessed and stored, convergence is forcing us to rethink the systems we use to manage data across the enterprise yet again. Companies across the industry, from Snowflake to Databricks to MongoDB, and every cloud provider, are working to provide the systems that let companies get more value from their data, using microservices-based networks or programs that drive informed, real-time decision making. Interestingly, this comes at a time when most companies are undergoing radical digital transformation projects in order to become innovation-powered, software-driven, and cloud-based. In other words, even though everyone is already quite busy, there has never been a better time to think beyond the database, and architect an actual “data platform” that can process, store, secure, and analyze data in real-time, across all the relevant data sets - either without copying the data or making such copying invisible. Over the coming months, I’ll be sharing more about what these data platforms look like and how they support the creation of modern applications. I’ll also try to peer into the foggy future with you, looking at the constraints and freedoms of the modern enterprise, and predicting what products will be built to address that matrix. But for now, I’m interested in hearing what you’re seeing under the hood of your data(base) estate. Are you experiencing a similar convergence of data requirements? If so, is it factored into your digital transformation strategy? And if not, what changes are you seeing in the way you use data across the enterprise? Please reach out to me here or on Twitter, at @MarkLovesTech - because I really do. Thanks for reading - may the data odds be in your favor until we chat again. Mark