Empower Modern App Developers with Document Databases

Andrew Davidson
June 5, 2023

Across industries, business success depends on a company’s ability to deliver new digital experiences through software. The speed at which a company can develop and deploy a new application with innovative features is a direct lever on business outcomes. Given the vital role developers play in the success of your business, it stands to reason that equipping them with the tools to maximize their productivity is in your best interest.

Unfortunately, many organizations are unaware of the tax they’re placing on their development teams by using a relational database. While the relational database has been a bedrock for data-driven applications for 50 years, it was developed in an era before the internet and is a poor fit as the foundation of today’s web and mobile applications.

Document databases, which have emerged over the past decade, have cemented themselves as the most popular and widely used alternative to the tabular model found in traditional relational databases. Document databases have become so powerful that even relational databases are trying to emulate them. Built around JavaScript Object Notation (JSON)–like documents, document databases are intuitive for developers to use. Instead of the rigid row-and-column structure of the relational model, document databases map documents directly to objects in code, which is how coders naturally think of and work with data.

Let’s break down the key advantages to document databases in building modern applications. We’ll see why the document model’s flexibility eliminates the complex intergroup dependencies that have traditionally slowed developers down.

The limitations of the relational database model

Relational databases add complexity to a developer’s workload, severely hampering the velocity of work. The rigid row-and-column structure creates a mismatch between the way developers think of code and data, and how they need to store it. Additionally, while the relational model was fine in an age when most applications used a small pre-set of attributes such as last names, ZIP codes, and state abbreviations, the majority of data collected by organizations today is rich in structure. We have given names and sometimes preferred names. We have unique attributes that are relevant to only some of us: For example, people with PhDs have dissertation topics, sports lovers have favorite sports, and our families come in every conceivable shape and size. This richly structured data reflects how we actually think about the real world, and it’s very difficult to flatten, store, analyze, or query using rows and columns.

With relational databases, developers can feel stuck in quicksand with changes to their applications requiring them to carefully collaborate with experts like database administrators (DBAs) who help them translate their schemas and queries to underlying relational data models to ensure that indexing strategies are appropriately employed. The layer of indirection increases cognitive load, is hard to reason about, and slows everything down. More than half of application changes require database schema modifications. Those database modifications take longer to complete than the application changes they are designed to support. You can quickly see why these complicated efforts severely slow the delivery of new software features into production.

Enabling development of modern apps with document databases

With the birth of the internet and the proliferation of mobile and web apps, developers’ roles evolved. The emergence of robust development frameworks, which abstracted away underlying complexity, and the rise of DevOps led organizations to consolidate developer functions. The new generation of full-stack developers wanted databases that better addressed their applications’ requirements and their ways of working with data.

The founders of MongoDB recognized a need for a modern database solution while at the adtech giant DoubleClick in 2007. They still were unable to scale to the 400,000 transactions per second the business required due to the constraints of the relational model. These challenges inspired them to create a new, modern, general-purpose database. This database could address the shortcomings of the relational data model and offer a solution that developers actually wanted to use. The result was a horizontally scalable, document-based NoSQL database called MongoDB. The document database model in general, and MongoDB more specifically, addresses the limitations of relational databases in several notable ways:

Intuitive data model: The documents at the center of document databases have a universal data format. JSON is a lightweight, language-independent, and human-readable format that has become a widely used standard for storing and exchanging data. These documents map directly to data structures in popular programming languages so there is no need for the additional mapping layer often used with relational databases. Because data that is accessed together is stored together, there is less code to write; developers don’t need to decompose data across tables or run joins.

Flexible schema: These JSON-like documents are flexible. Each document can have its own fields, and there’s no need to pre-define the schema in the database. It can be modified at any time. That flexibility enhances developer agility.

Meeting user expectations while simplifying application architectures

The most innovative applications we use in our daily lives — think Netflix and Instagram — have raised user expectations for what every application should be. Today we expect applications to be:

Highly responsive
Able to deliver relevant information
Optimized for mobile devices
Secure
Powered by real-time insights
Continuously improved

Meeting those expectations can be extremely challenging, especially for developers using relational databases. A typical data infrastructure built around a legacy relational database can trap your development team in overly complex and siloed architectures. Document databases, on the other hand, can simplify application architectures. Documents are a superset of all other data models, so developers can store and work with a variety of data types. Development teams can accommodate most of their use cases in a single data model and database

The document data model can help your developers overcome the limitations of the relational model while improving their productivity and velocity. Allowing them to minimize the undifferentiated work of maintaining their infrastructure and to focus on meeting demanding user expectations.

As a result, they can deliver better, more innovative applications faster than before.

Click here to read the original article published on The New Stack.

← Previous

The Guide to openEHR Schema Modeling with MongoDB

Disclosure: This article features valuable insights from MongoDB's experts, customers, or partners to further an understanding of MongoDB in specific industry use cases. Please note that while MongoDB does not validate the accuracy of the text and statements, this resource aims to provide practical knowledge for your reference. The openEHR specification is a widely used standard for storing and managing electronic health records (EHRs). It offers a structured way of organizing clinical data that makes it easy to query and analyze. However, implementing the schema modeling and querying of openEHR data can present unique challenges. In this blog post, we will explore the intricacies of openEHR schema modeling, some challenges, and outline potential solutions that we can implement using MongoDB. Understanding openEHR specification and interoperability standards Before diving into the complexities of openEHR schema modeling and querying, let’s first understand the openEHR specification and how it relates to other interoperability standards such as HL7 FHIR. These standards work together to enable seamless healthcare information exchange across various systems and applications. A clear understanding of these standards will provide a solid foundation for addressing the challenges we’ll encounter. openEHR (open Electronic Health Record) is an open-source standard for the representation and management of electronic health records (EHRs). It is designed to provide a flexible and interoperable framework for the collection, storage, retrieval, and exchange of health data, regardless of the system or application used to generate or consume it. Key building blocks of openEHR include: Archetypes: Archetypes are structured, reusable models that define the content and structure of clinical information. An example is a Vital Signs Archetype which defines the structure and constraints for capturing vital signs measurements. Properties may include elements such as temperature, heart rate, blood pressure, and oxygen saturation. Templates: Templates, on the other hand, are derived from archetypes and are used to define specific subsets of clinical information. An example is an Adult Vital Signs Template — a template derived from the Vital Signs Archetype and customized for adult patients. It includes a subset of vital sign elements specifically relevant to adult patients, such as temperature, heart rate, and blood pressure. Compositions: A composition is an instance of a template that contains actual clinical data. An example is Patient Encounter Composition — a composition that represents a patient encounter, containing various clinical measurements and data. Properties include sections for patient demographics, symptoms, diagnoses, procedures, medications, and vital signs. Within the composition, the vital signs section would follow the structure defined by the Adult Vital Signs Template, containing actual vital signs measurements for a specific encounter. Figure 1: openEHR - Archetype & template openEHR & HL7 FHIR While openEHR defines an “information model” for modeling and persisting data in EMR systems, FHIR is an open-source standard for healthcare information exchange across EMRs and other systems. The diagram below can help you visualize how these standards and technologies work together in healthcare systems. FIgure 2: openEHR, FHIR, & HL7 openEHR schema modeling challenges openEHR schema modeling poses several challenges due to its complex hierarchical structure and the need to handle diverse data types. Additionally, querying this data can be difficult due to the complex relationships between the different data elements. Many of the end user queries are at composition level and each composition is made up of hundreds of fields. This is typically overcome by creating multiple sets of indexes often resulting in performance bottlenecks. Let’s examine some of the typical query patterns and demonstrate how these pose challenges for schema modeling and querying: Complex Hierarchical Structure: The openEHR specification is built on a complex hierarchical structure that represents various healthcare concepts and their relationships. Translating this structure into a database schema can be challenging, as traditional relational databases may struggle to handle the dynamic and nested nature of openEHR data. Finding an efficient and flexible way to model this complex structure is essential for ensuring accurate representation and easy retrieval of data. Data Versioning and Evolution: openEHR supports the concept of versioning and evolution, allowing for changes and updates to healthcare records over time. Modeling and querying evolving data can be complex, as it requires maintaining the history of changes and accommodating different versions of the schema. Ensuring data consistency, efficient versioning, and the ability to query historical data are critical considerations in openEHR schema modeling. Performance and Scalability: Healthcare systems generate a vast amount of data, and efficient querying of openEHR records is crucial for timely analysis and decision-making. Designing a schema that allows for fast and scalable querying is a challenge, particularly when dealing with large datasets and complex query patterns. Optimizing query performance, indexing strategies, and data partitioning techniques are essential for ensuring a responsive and scalable system. Query Patterns: openEHR data is queried based on various patterns, such as retrieving patient records, searching for specific diagnoses, or aggregating data for statistical analysis. Each query pattern may have different performance requirements and may involve traversing complex relationships within the hierarchical structure. Designing an efficient schema that can handle these query patterns and provide fast and accurate results is a significant challenge in openEHR schema modeling. Learn more about how MongoDB works with any healthcare data standard in our whitepaper, What is Radical Interoperability Typical openEHR schema model As you see in the below hierarchical model, the complexity of the openEHR specification can make it challenging to model and query clinical data. In the subsequent sections, we will explore potential solutions for openEHR schema modeling, including the Attribute Pattern and the Flat Hierarchy Pattern. Figure 3: Part of Archetype data captured in JSON format Archetype query language Archetype Query Language (AQL) is a query language specifically designed for querying clinical data stored in openEHR-based electronic health record systems. It provides a standardized and powerful way to retrieve specific clinical information from structured data using archetypes and templates. AQL enables clinicians, researchers, and developers to express complex queries, filter data based on clinical criteria, and retrieve meaningful information for analysis and decision support. For example to “get the latest 5 abnormal blood pressure values that were recorded in a health encounter for a specific patient”, you can write an AQL as follows: As you can see above in the highly nested and hierarchical schema model, while being a flexible and extensible approach to representing clinical data, presents unique challenges in storage, retrieval, and performance. Even a small volume of 250k documents in this format requires a significant amount of storage space for handling. In addition, given the variable nature of nesting hierarchy, typical patterns for indexing are highly inefficient. We will explore possible solution options using MongoDB schema modeling patterns. Addressing openEHR schema modeling challenges with MongoDB MongoDB provides flexible and powerful features that can help address the challenges of openEHR schema modeling and querying. Attribute pattern with standard index One possible solution is to use the attribute pattern modeling style, simplifying the schema model and making it more predictable. This approach allows for efficient indexing, enabling fast retrieval of data. The query for our example scenario to “get the latest 5 abnormal blood pressure values that were recorded in a health encounter for a specific patient” can be simplified as follows. The above query is highly efficient as evaluated against a collection of 5 MM document size. Additionally, the storage is significantly improved taking 833 MB compressed storage space for 5 MM documents, as well as efficient retrieval using index. However the index size is significantly large for this pattern — about 3.5 GB for the 5 MM document size – which may be a blocker for larger data sets. Can we improve on this? What are our options? Learn more about how MongoDB works with any healthcare data standard in our whitepaper, What is Radical Interoperability Flat hierarchy model with wildcard search An alternative approach to addressing challenges discussed above is by leveraging a flattened document model and using wildcard index. The same openEHR schema model can be represented as follows: By using a flattened document data model, the complex hierarchical structure of openEHR schemas can be simplified for storage and retrieval efficiency. While this simplifies the schema model, there are a few considerations like the dimensions need to be moved to the application / configuration layer, like the body temperature is celsius, for example. In addition the query on the above document model is simplified as below. To make the query efficient, we create a wild-card index as follows. With the above index in place, the query is significantly more efficient as we can see in the explain plan below. When comparing the storage size to openEHR standard spec, the flat hierarchy spec model only takes 839 MB compressed storage for 5 MM documents. However, the wildcard index is significantly heavy. For the above volume of documents, the index size is only 705 MB. Can we improve upon this further? Flat hierarchy model with Atlas Search MongoDB Atlas Search brings the power of Apache Lucene to MQL. You can simplify the process of indexing and search significantly. You start by creating an Atlas Search index as follows: Atlas takes care of the various steps leading up to making a Lucene search index available. Once the index is in place, you can query using $search opearator. The query we have seen in above solution options can be constructed as follows. The data storage is identical to the flat hierarchy schema, with only the index built using Atlas search. The search index size is similar to the wildcard index. However the search index provides significantly greater functionalities and capabilities. Keep in mind that this capability is only available on cloud and with MongoDB Atlas . We have seen multiple solution options as detailed above. You can choose one that suits your application as well as infrastructure requirements. Design Validation Typical process of evaluating various possible schema models and their efficiencies will require multiple iterations including NFR validation on production scale of data volume. This helps us to: Better understand fit of MongoDB to the particular use case Understand MongoDB sizing expectations Demonstrate MongoDB performance Understand performance of specific queries Fine tune MongoDB schema for our needs Current tools only replicate a sample data or generate random data and are not suitable for the above purposes as the indexes get skewed and performance of queries is not close to real world. Completely random data also does not provide an accurate view of the MongoDB sizing that will be required. Test data generator Test data generator and Performance testing tools are a solution accelerator from PeerIslands that helps customers generate large volume, customizable, close-to-real-world test data with specific customer schemas. We have used the test data generator to generate a 5 Million documents data set for both attribute and flat hierarchy design patterns. The test data generator takes a configuration file and quickly generates large volumes of data, as shown in the flat hierarchy schema model below. Generating test data for the original openEHR spec requires a more complex configuration as the one below. Results and comparison We generated 5MM data items for each type of design pattern (attribute and flat hierarchy). For base openEHR spec, we generated 250k items given the size and time requirements. The figure below provides a comparison between the data and index sizes for each design approach: In addition the data and index sizes for Atlas Search is shown below for a 214k dataset. The attribute pattern performs significantly better than the base openEHR spec. However it is limited by its index size. The flat hierarchy data model performs the best overall with a wildcard index. When you can use cloud, you have the option of using Atlas Search. While the overall index size is not any better than the wildcard index on the flat hierarchy collection, the Lucene search index provides a significantly expanded feature set for query and retrieval. Final thoughts The MongoDB document data model provides a powerful and intuitive approach to structuring and interacting with healthcare data (read openEHR), which is often complex and variable. It closely aligns with how you think and code, allowing you to store and retrieve data of any shape and form. The powerful query engine and indexing capabilities further enhance its versatility, enabling you to develop complex query patterns and optimize performance for your specific application requirements. Choosing the appropriate modeling approach depends on the specific requirements of the application, query patterns, and performance considerations. Both the Attribute Pattern and the Flat Hierarchy models offer viable solutions for openEHR data storage in MongoDB, providing flexibility and performance optimizations tailored to different use cases. Additionally, MongoDB Atlas Search introduces powerful search capabilities for enhanced query and retrieval functionalities. While we looked into schema modeling and querying solutions for openEHR data in MongoDB, there are other topics that are of interest while developing production-scale applications and environments where you would like both engineers and operations teams to be more productive. We will be exploring the following topics in a future blog. Simplifying openEHR queries: A DSL based approach to convert existing AQL to MQL and using Generative AI to build MQL based on natural language prompts. Strategies for organizing openEHR data such as Multi-tenancy by clinic and horizontal scaling using sharding. Generate production-scale data volume persisted in a Sharded MongoDB cluster. Run NFR validation on dedicated production grade infrastructure and compare performance of various approaches discussed. We hope this comprehensive guide has provided valuable insights into openEHR schema modeling and querying challenges, as well as potential solutions using MongoDB. Learn more about how MongoDB works with any healthcare data standard in our whitepaper, What is Radical Interoperability References: Introduction to openEHR HL7 FHIR & openEHR Choosing the standard that is right for you MongoDB Wildcard Index

June 5, 2023

Next →

Collaborating to Build AI Apps: MongoDB and Partners at Google Cloud Next '24

From April 9 to April 11, Las Vegas became the center of the tech world, as Google Cloud Next '24 took over the Mandalay Bay Convention Center—and the convention’s spotlight shined brightest on gen AI. Check out our AI resource page to learn more about building AI-powered apps with MongoDB. Between MongoDB’s big announcements with Google Cloud (which included an expanded collaboration to enhance building, scaling, and deploying GenAI applications using MongoDB Atlas Vector Search and Vertex AI ), industry sessions, and customer meetings, we offered in-booth lightning talks with leaders from four MongoDB partners—LangChain, LlamaIndex, Patronus AI, and Unstructured—who shared valuable insights and best practices with developers who want to embed AI into their existing applications or build new-generation apps powered by AI. Developing next-generation AI applications involves several challenges, including handling complex data sources, incorporating structured and unstructured data, and mitigating scalability and performance issues in processing and analyzing them. The lightning talks at Google Cloud Next ‘24 addressed some of these critical topics and presented practical solutions. One of the most popular sessions was from Harrison Chase , co-founder and CEO at LangChain , an open-source framework for building applications based on large language models (LLMs). Harrison provided tips on fixing your retrieval-augmented generation (RAG) pipeline when it fails, addressing the most common pitfalls of fact retrieval, non-semantic components, conflicting information, and other failure modes. Harrison recommended developers use LangChain templates for MongoDB Atlas to deploy RAG applications quickly. Meanwhile, LlamaIndex —an orchestration framework that integrates private and public data for building applications using LLMs—was represented by Simon Suo , co-founder and CTO, who discussed the complexities of advanced document RAG and the importance of using good data to perform better retrieval and parsing. He also highlighted MongoDB’s partnership with LlamaIndex, allowing for ingesting data into the MongoDB Atlas Vector database and retrieving the index from MongoDB Atlas via LlamaParse and LlamaCloud . Guillaume Nozière - Patronus AI Andrew Zane - Unstructured Amidst so many booths, activities, and competing programming, a range of developers from across industries showed up to these insightful sessions, where they could engage with experts, ask questions, and network in a casual setting. They also learned how our AI partners and MongoDB work together to offer complementary solutions to create a seamless gen AI development experience. We are grateful for LangChain, LlamaIndex, Patronus AI, and Unstructured's ongoing partnership. We look forward to expanding our collaboration to help our joint customers build the next generation of AI applications. To learn more about building AI-powered apps with MongoDB, check out our AI Resources Hub and stop by our Partner Ecosystem Catalog to read about our integrations with these and other AI partners.

April 23, 2024