This Week in MongoDB Sep 23-29

Amy Berman
September 23, 2013 | Updated: May 22, 2015

Here's what's going on in the MongoDB community this week:

Learn More

Upcoming Events:

Upcoming Webinars:

Online Education:

Public Training:

User Groups and Events this Week:

← Previous

Visualizing Performance Issues with MMS

This is a guest post by Albert Engelbrecht, Web App Developer at SuretyBonds.com . “LMS is down.” The dreaded phrase came back again, meaning the office is twiddling its thumbs as the sales lead system becomes unresponsive. I open my SSH connection, run the MongoDB startup script and say “It’s back up, Brad.” I get alert emails on IO usage on the servers nightly; the swap file is killing the disk. For some reason, our single MongoDB instance shuts down a few times a day. Logs pointed to the DB running out of memory, which was supported by my slowing database calls and huge IO overhead. The obvious short-term answer to our out-of-memory errors was to upgrade to a bigger server instance for more swap space, however I knew this wouldn’t improve database performance of any one operation. There is nothing more painful when trying to debug a hard problem as realizing that you can’t see anything - you are entirely blind. Mongotop and mongostat tools are a bit arcane and don’t give the big picture view. Ultimately my research for MongoDB monitoring led me to try MMS - MongoDB Management Service . Suddenly there are graphs and charts galore. Everything’s there: non-mapped memory, database size, total memory footprint. I am in hog heaven. Not only can I get DB statistics but with a Munin agent running, it can report back hardware statistics as well. These aren’t your Grandpa’s Munin graphs either, MMS’s graphs are interactive, allowing me to set custom date ranges. Another neat feature that I use is text and email alerts on a number of different events. Not only does MMS have graphs and alerts, but it also allows me to go through our slower queries and order them by response time. Instead of sifting through profiling data using queries inside the mongo shell, now I can easily view it all on one page. You know how many queries I rewrote? Probably a million, maybe two. Take this for example: Lead.find( {'$where' : "['fresh', 'new'].indexOf(this.status) > -1"} ) // This runs JavaScript to figure out if the status wasn't "fresh" or "new" Lead.find( {"status" : { "$nin" : [ "fresh", "new" ] } } ) // Much more performant This fun little query runs JavaScript to find every record in the collection to find the ones that don’t have two statuses. It runs every time someone hits the front page, even though the information is only shown to admins. Removing the query made a big difference in the non-mapped memory size. Take that interns, for you shall impact my DB performance no longer! At the end of the day, upgrading the DB server and doing major refactoring worked out well. Our MongoDB instance hasn’t gone down since then, plus I find myself regularly coming back MMS to see how our newer queries are working out. It’s also good to know when we grow large enough to require sharding, MMS can seamlessly scale. If you happen to be wondering just how well your DB health is, I encourage you to check out MMS Monitoring . 10gen, er, MongoDB Inc. did a great job, plus it’s free! When not waxing poetic about code smell, Albert works on web applications for SuretyBonds.com , a nationwide surety bonding agency.

September 20, 2013
Next →

How Edenlab Built a High-Load, Low-Code FHIR Server to Deliver Healthcare for 40 Million Plus Patients

The Kodjin FHIR server has speed and scale in its DNA. Edenlab, the Ukrainian company behind Kodjin , built our original FHIR solution to digitize and service the entire Ukrainian national health system. The learnings and technologies from that project informed our development of the Kodjin FHIR server. At Edenlab, we have always been driven by our passion for building solutions that excel in speed and scale. With Kodjin, we have embraced a modern tech stack to deliver unparalleled performance that can handle the demands of large-scale healthcare systems, providing efficient data management and seamless interoperability. Eugene Yesakov, Solution Architect, Author of Kodjin Built for speed and scale While most healthcare projects involve handling large volumes of data, including patient records, medical images, and sensor data, the Kodjin FHIR server is based on a system developed to handle tens of millions of patient records and thousands of requests per second, to ensure timely access and efficient decision-making for a population of over 40 million people. And all of this information had to be processed and exchanged in real-time or near real-time, without delays or bottlenecks. This article will explore some of the architectural decisions the Edenlab team took when building Kodjin, specifically the role MongoDB played in enhancing performance and ensuring scalability. We will examine the benefits of leveraging MongoDB's scalability, flexibility, and robust querying capabilities, as well as its ability to handle the increasing velocity and volume of healthcare data without compromising performance. About Kodjin FHIR server Kodjin is an ONC-certified and HIPAA-compliant FHIR Server that offers hassle-free healthcare data management. It has been designed to meet the growing demands of healthcare projects, allowing for the efficient handling of increasing data volumes and concurrent requests. Its architecture, built on a horizontally scalable microservices approach, utilizes cutting-edge technologies such as the Rust programming language, MongoDB, ElasticSearch, Kafka, and Kubernetes. These technologies enable Kodjin to provide users with a low-code approach while harnessing the full potential of the FHIR specification. A deeper dive into the architecture approach - the role of MongoDB in Kodjin When deciding on the technology stack for the Kodjin FHIR Server, the Edenlab team knew that a document database would be required to serve as a transactional data store. In an FHIR Server, a transactional data store ensures that data operations occur in an atomic and consistent manner, allowing for the integrity and reliability of the data. Document databases are well-suited for this purpose as they provide a flexible schema and allow for storing complex data structures, such as those found in FHIR data. FHIR resources are represented in a hierarchical structure and can be quite intricate, with nested elements and relationships. Document databases, like MongoDB, excel at handling such complex and hierarchical data structures, making them an ideal choice for storing FHIR data. In addition to supporting document storage, the Edenlab team needed the chosen database to provide transactional capabilities for FHIR data operations. FHIR transactions, which encompass a set of related data operations that should either succeed or fail as a whole, are essential for maintaining data consistency and integrity. They can also be used to roll back changes if any part of the transaction fails. MongoDB provides support for multi-document transactions , enabling atomic operations across multiple documents within a single transaction. This aligns well with the transactional requirements of FHIR data and ensures data consistency in Kodjin. Implementation of GridFS as a storage for the terminologies in Terminology service Terminology service plays a vital role in FHIR projects, requiring a reliable and efficient storage solution for terminologies used. Kodjin employs GridFS , a file system within MongoDB designed for storing large files, which makes it ideal to handle terminologies. GridFS offers a convenient way to store and manage terminology files, ensuring easy accessibility and seamless integration within the FHIR ecosystem. By utilizing MongoDB's GridFS, Kodjin ensures efficient storage and retrieval of terminologies, enhancing the overall functionality of the terminology service. Kodjin FHIR server performance To evaluate the efficiency and responsiveness of the Kodjin FHIR server in various scenarios we conducted multiple performance tests using Locust, an open-source load testing tool. One of the performance metrics measured was the retrieval of resources by their unique ids using the GET by ID operation. Kodjin with MongoDB achieved a performance of 1721.8 requests per second (RPS) for this operation. This indicates that the server can efficiently retrieve specific resources, enabling quick access to desired data. The search operation, which involves querying ElasticSearch to obtain the ids of the searched resources and retrieving them from MongoDB, exhibited a performance of 1896.4 RPS. This highlights the effectiveness of polyglot persistence in Kodjin, leveraging ElasticSearch for fast and efficient search queries and MongoDB for resource retrieval. The system demonstrated its ability to process search queries and retrieve relevant results promptly. In terms of resource creation, Kodjin with MongoDB showed a performance of 1405.6 RPS for POST resource operations. This signifies that the system can effectively handle numerous resource-creation requests. The efficient processing and insertion of new resources into the MongoDB database ensure seamless data persistence and scalability. Overall, the performance tests confirm that Kodjin with MongoDB delivers efficient and responsive performance across various FHIR operations. The high RPS values obtained demonstrate the system's capability to handle significant workloads and provide timely access to resources through GET by ID, search, and POST operations. Conclusion Kodjin leverages a modern tech stack including Rust, Kafka, and Kubernetes to deliver the highest levels of performance. At the heart of Kodjin is MongoDB, which serves as a transactional data store. MongoDB's capabilities, such as multi-document transactions and flexible schema, ensure the integrity and consistency of FHIR data operations. The utilization of GridFS within MongoDB ensures efficient storage and retrieval of terminologies, optimizing the functionality of the Terminology service. To experience the power and potential of the Kodjin FHIR server firsthand, we invite you to contact the Edenlab team for a demo. For more information On MongoDB’s work in healthcare, and to understand why the world’s largest healthcare companies trust MongoDB, read our whitepaper on radical interoperability .

May 31, 2023