The DBA's future as the Database Adviser
With such a large shift towards NoSQL technologies like MongoDB, there is a lot of discussion about the changing role of the Database Administrator (DBA). Many go so far as to say DBAs are no longer needed , an idea driven by new database capabilities that focus on agility with a dynamic schema instead of a fixed schema and features that make operational management easier in areas like high availability and horizontal scaling. Based on my experience working with hundreds of customers to implement MongoDB in their organizations, there is still room for the DBA, even if their everyday tasks might take less time with MongoDB. First, let's talk about the parts of their role that stay the same. There still needs to be someone to set up back-up/recovery processes, handle capacity planning, run maintenance tasks (e.g. upgrades), diagnose issues, do configuration management, and set up replication and sharding. In enterprises, often separate operational teams handle security, monitoring, and diagnosing common issues but that could be part of the DBA's role in some firms as well. Notice I left out schema management; in MongoDB, the implementation of the schema is NOT predefined but is rather determined at the application level and the structure of the object is stored in the application team’s favorite programming language. This is incredibly valuable when business units request new features or data be added to the application. The application developer can simply add it in, and no change needs to be made in the database, enabling rapid agility. In this case, the DBA is no longer the middleman, and can focus on keeping the database up and running. But in some cases, application changes are more complex, and development teams need a database expert to consult about schema design and its impact on performance, maintainability, and other factors. In this case, the role of the DBA transforms into the DB Adviser. In this role, the DB Advisor would work closely with the application development team that implements the schema. The DBA provides the process and due diligence to manage the pace of change rather than enforce the limitations of the relational schema. Implementing the schema in the application might be disconcerting to some DBAs and others out there who rightfully worry about application developers making uninformed decisions and bringing down an application. Letting go of schema management might be a difficult step, but DBAs rely on application developers to make decisions in their application code on a regular basis. Creating a poorly constructed algorithm will bring the system to a crawl and querying fields in unexpected ways would cause performance problems as well. Is choosing schema design in the database so much more responsibility than any other development decision? The DBA should still guide developers, but without the enormous overhead of having to always update a relationship database schema to simply add a field! Your schema should be as dynamic as your business: agile with the optimal amount of control. The DBA has always been the person accountable to ensure the reliability and performance of the database alongside development teams. However, in today's market, the DBA will continue performing most of the activities they do in the former relational-only world, but will advise in other areas to allow their business groups to innovate and iterate faster than their competition. In this shift, the DBAs can take on new roles to help their businesses achieve more, faster. See how you can help your organizations get faster, better and leaner with MongoDB .
Living in the post-transactional database future
Given that we’ve spent decades building applications around relational databases, it’s not surprising that the first response to the introduction of NoSQL databases like MongoDB is sometimes “Why?” Developers aren’t usually the ones asking this question, because they love the approachability and flexibility MongoDB gives them. But DBAs who have built their careers on managing heavy RDBMS infrastructure? They’re harder to please. 10gen president Max Schireson estimates that 60 percent of the world’s databases are operational in nature, which is MongoDB’s market. Of those use cases, most of them are ripe for a non-relational approach. The database market is rapidly changing, and very much up for grabs. Or as Redmonk analyst James Governor puts it , “The idea that everything is relational? Those days are gone.” As useful as relational databases are (and they’re very useful for a certain class of application), they are losing relevance in a world where complex transactions are more the exception, less the rule. In fact, I’d argue that over time, the majority of application software that developers write will be in use cases that are better fits for MongoDB and other NoSQL technology, not legacy RDBMS. That’s the future. What about now? Arguably, many of the applications being built today are already post-transaction, ripe for MongoDB and poor fits for RDBMS. Consider: Amazon: its systems that process order transactions (RDBMS) are largely “done” and “stable”. Amazon’s current development is largely focusing on how to provide better search and recommendations or how to adapt prices on the fly (NoSQL). Netflix: the vast majority of it engineering is focusing on recommending better movies to you (NoSQL), not processing your monthly bill (RDBMS). Square: the easy part is processing the credit card (RDBMS). The hard part is making it location aware, so it knows where you are and what you’re buying (NoSQL). It’s easy, but erroneous, to pigeon-hole these examples as representative of an anomalous minority of enterprises. Yes, these companies represent the cutting edge of both business and technology. But no, they are not alone in building these sorts of applications. For every early-adopter Netflix there’s a sizable, growing population of mainstream companies in media (e.g., The Guardian ), finance (e.g., Intuit ), or other verticals that are looking to turn technology into a revenue-driving asset, and not simply something that helps keep the lights on and payrolls running. When what we built were websites, RDBMS worked great. But today, we’re building applications that are mobile, social, involve high volume data feeds, incorporate predictive analytics, etc. These modern applications? They don’t fit RDBMS. Andy Oliver lists 10 things never to do with a relational database , but the list is much longer, and growing. MongoDB is empowering the next generation of applications: post-transactional applications that rely on bigger data sets that move much faster than an RDBMS can handle. Yes, there will remain a relatively small sphere of applications unsuitable for MongoDB (including applications with a heavy emphasis on complex transactions), but the big needs going forward like search, log analysis, media repositories, recommendation engines, high-frequency trading, etc.? Those functions that really help a company innovate and grow revenue? They’re best done with MongoDB. Of course, given RDBMS’ multi-decade legacy, it’s natural for developers to try to force RDBMS to work for a given business problem. Take log analysis, for example. Oliver writes: Log analysis : …[T]urn on the log analysis features of Hadoop or RHQ/JBossON for a small cluster of servers. Set the log level and log capture to anything other than ERROR. Do something more complex and life will be very bad. See, this kind of somewhat unstructured data analysis is exactly what MapReduce à la Hadoop and languages like PIG are for. It’s unfortunate that the major monitoring tools are RDBMS-specific — they really don’t need transactions, and low latency is job No. 1. For forward-looking organizations, they already realize that MongoDB is an excellent fit for log management, which is why we see more and more enterprises turning to MongoDB for this purpose. I expect this to continue. As MongoDB continues to enrich its functionality , the universe of applications for which it is not merely applicable, but also better , will continue to expand, even as the universe of applications for which RDBMS is optimal will decline. Indeed, we’re already living in a post-transactional world. Some people just don’t know it yet. (Or, as William Gibson would say, “The future is already here – it’s just not very evenly distributed.”) Posted by Matt Asay , vice president of Corporate Strategy, with significant help from my inestimable colleague, Jared Rosoff . Tagged with: NoSQL, MongoDB, RDBMS, relational, James Governor, Redmonk, log analysis, Andy Oliver, transactions, Netflix, Amazon, Square, operational database, DBA