MongoDB Q&A: What's the deal with non-relational databases and Agile software development?
January 11, 2019 | Updated: February 1, 2019
If you are new to MongoDB, you probably have a lot of questions. I know, because I'm new as well. I'm lucky enough to have direct access to the experts at MongoDB, and I've been asking them a LOT of questions. In this series, I'll share answers to my questions.
Throughout my career as a software engineer, I’ve been interested in how Agile principles can improve software development. I first became interested in Agile software development in undergrad and then focused on distributed pair programming for my master’s thesis. This led to my career at IBM as a software engineer where I worked as a web developer, tester, and automation specialist while applying Agile principles. I shared my expertise while serving on the IBM Agile Leadership Team.
When I joined MongoDB, I kept hearing people talk about MongoDB being built for Agile development. So I decided to dig in and explore: Why are people talking about non-relational databases and Agile software development as if they go hand-in-hand?
Let's start with the Agile Manifesto:
Manifesto for Agile Software Development
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
© 2001 Beck, K., Beedle, M., Bennekum, A., Cockburn, A., Cunningham, W., Fowler, M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., Kern, J., Marick, B., Martin, R., Mellor, S., Schwaber, K., Sutherland, J. and Thomas, D. - this declaration may be freely copied in any form, but only in its entirety through this notice.
I wondered: what do the tenets of the Agile Manifesto have to do with the underlying database technology you use for building your application? When the Agile Manifesto focuses on how you build your application, why does it matter what you use to build your application?
After doing some digging, I discovered the underlying database can actually make a massive impact on how agile your team is. In this post, I'll focus on two of the Agile Manifesto tenets.
One of the main tenets of Agile software development is valuing getting your software working more than writing documentation.
When using a relational database, you'll typically spend a lot of time upfront planning your database schema because schema changes are difficult and can require you to take down your database (and therefore your application). Your planning will likely require you to think through all of your use cases to understand what data you will need to store and the best way to store it. Sounds a bit like waterfall planning, doesn't it?
Many non-relational databases (including MongoDB) have flexible schemas. You can begin storing data in the database immediately if you'd like. You can change the data you're storing and create schema validation rules at any point in time without taking down the database, so you don't have to know all of your use cases right away. Non-relational databases give you the flexibility to focus on getting your application working without doing a lot of upfront database planning.
Translating your app's code to your data model
In relational databases, you'll usually create a normalized, tabular schema, which does not typically map well to object-oriented programming. For example, the data for a single object like a user will likely be broken into multiple tables like Credentials, Contact_Information, and Connections. You'll probably spend a decent amount of time diagramming (isn't creating a massive ER Diagram fun?) and debating your schema until your team agrees on a finalized schema.
One of the nice features of document-based, non-relational databases is that an object in your code will typically map to a single document in your database without having to be broken down into multiple documents. Continuing with the example above, a user object that keeps track of a user's credentials, contact information, and connections could be modeled in a single User document or record by using rich embedded structures such as subdocuments and arrays. This simplifies your database design (and therefore documentation) and can help you build and modify your app quickly.
Responding to change
Another main tenet of Agile software development is valuing responding to change over following a plan. As we develop our apps, we learn new information and requirements, we (hopefully) fail fast, and we pivot. Sticking to an initial set of specifications is rare.
Storing new types of information
When you use a relational database, the data you store must follow the schema rules. Let's say you discovered mid-development that it would be helpful to store a new piece of information in an existing table to support a new use case. To do this, you would add a new column to a table, take down the database, add the field (which would create empty field values for every row in the table), and bring the database back up. It's not a simple process, and you might have a Database Administrator who is the gatekeeper and implementer of changes like these.
When you use a non-relational database, you have much more flexibility in your data. Let's examine the case above where you decide to store a new piece of information in your database to support a new use case. Because the documents or records in non-relational databases can be polymorphic (meaning they can take on many shapes), your application could simply begin sending the new information to the database. Old documents and records would not have the new field, which is completely fine. You don't have to explicitly tell the database that you're adding a new field--you can just start using it. Your application developers are empowered to easily make changes to the data being stored themselves.
One catch I should mention: when you need to update your query patterns in order to support new or updated requirements, many non-relational databases require you to remodel your data. Your data must be modeled based on how you will query it. However, MongoDB gives you the flexibility to query your data however you'd like--little to no remodeling required. MongoDB's rich query language, flexible indexing capabilities, and aggregation pipeline allow you to query your data almost any way you want.
Scaling your database
Ideally, your app is going to make it big. When it does, will your database be ready for the load?
Typically, you scale relational databases vertically by increasing the capacity of a single server. You might do this by increasing the server's RAM, CPU, or storage space. Servers can get expensive or be unavailable due to technology limitations or Cloud-based provider limitations.
Non-relational databases can typically scale horizontally, which means that you can scale by adding more servers and spreading the load and data across them. The servers don't have to be top-of-the-line servers, which can be much more cost effective. Plus, when your app makes it big, you can keep adding servers without worrying about running out of space.
After talking with my teammates and doing a lot of reading, I get it: Agile software development and non-relational databases do go hand-in-hand. If you'd like to learn more about how MongoDB is architected to support modern, Agile development, check out the MongoDB Architecture Guide.
If you have questions you'd like me to answer in this blog series or you have a story of how using a non-relational database has enabled your team to be more agile, I'd love to hear! Drop a comment below.