MongoDB Blogroll: The Best of July 2012
Every month, we’ll be publishing the best community blog posts from the month. Here is the digest for July:
Every month, we’ll be publishing the best community blog posts from the month. Here is the digest for July:
- Mike O'Brien, node.js and python engineer at 10gen, wrote an overview of tips and tricks for using mongo, the MongoDB shell
- Jesse Jiryu Davis published an overview of Motor, the Asynchronous python driver for MongoDB.
- Kristina Chodorow wrote an overview of Shard Tagging, a new feature in the 2.2.0 series.
- Juhi Bhatia, a member of the Pune MongoDB User Group, wrote an article on how she improved on her Schema Design for Entrib’s EMG PaaS.
- Tobias Trelle, organizer of the Dusseldorf MongoDB User group, published a post on using GridFS with Spring Data.
- Rick Copeland wrote an overview of the Ming toolkit for Python with tips on how it can accelerate your development.
- Stephen Bronstein shared some tactics in continuous deployment in his blog post on Feature flipping with MongoDB and Node.js.
- 10gen interns, Samantha Ritter and Kaushal Parikh released their log visualization tool, Edda.
- Database as a service company ObjectRocket launched their blog with the post In Memory MongoDB concurrency testing on ObjectRocket
- Comsysto’s Johannes Brandstetter showcased how to use MongoDB to create a real-time Twitter Heatmap using MongoDB’s capped collections.
Want your blog post to be included in the next update? Tweet it out with the #mongodb hashatag or send it to us directly.
Building Big Data Portal through Liferay and MongoDB integration
This was originally posted to the CIGINEX Blog The CIGNEX Datamatics Big Data Portal is a web based solution which combines the powerful presentation capabilities of a portal such as rich user interfaces, collaboration, and secures access, with centralized & massively scalable data storage as the back end, consisting of a variety of content (Audio, Video, Images, Documents, Metadata) in large volume. We have been providing content management and portal solutions for the past 12 years. While serving our customers we have observed the following: The content is growing big, really big (volume) The unstructured (variety of) content is now becoming a business need for customers to process Customers wanting to capture smart information (meta-data) about the content Need to provide high performance and secure access of content to variety of applications and devices We have integrated two leading Open Source technologies, Liferay and MongoDB to build a powerful, cost effective solution as ...BIG DATA PORTALâ€œ to the ever growing need to manage the vast amount of information available. Big Data is at the foundation of many technologies that are trending today. With years of proven global experience in Open Source, we are excited to be pioneering solutions that solve many of today's growing challenges. CIGNEX Datamatics Big Data Portal is represented in the following diagram. Liferay is the leading Open source portal with a strong community; with 4+ million downloads, with 500,000 deployments worldwide. Liferay is featured as Leader in Gartner's Magic Quadrant for Horizontal Portals. MongoDB is an immensely scalable, NoSQL and agile document-oriented database based on JSON-like document storage with dynamic schemas. MongoDB's flexible data structure, ability to index & query and auto-sharding makes it a strong tool that adapts to changes and reduces the complexity. MongoDB's GridFS enables large binary objects like Images, Video or Audio. Big Data Portal with MongoDB and Liferay provides lower total cost of ownership and higher ROI to the businesses. CIGNEX Datamatics have developed a connector which enables Liferay to manage content in the clustered environment in the MongoDB solution. The architecture diagram is given below: Key Benefits with our solution include: Elimination of high end storage systems such as SAN and Oracle Clusters, which is a huge cost savings Secure access to data Flexibility leading to high performance Simplified Data Management through a single system managing structured and unstructured data Consistent look and feel accessible across myriad gadgets and devices For more details, download the presentation from CIGNEX Datamatics website at, http://www.cignex.com/resources/presentations/psc Munwar Shariff, CTO, CIGNEX Datamatics For more details, contact: munwar at cignex dot com Tagged with: big data, portal, cloud, open source, MongoDB, Mongo, NoSQL, Polyglot persistence, 10gen
How Edenlab Built a High-Load, Low-Code FHIR Server to Deliver Healthcare for 40 Million Plus Patients
The Kodjin FHIR server has speed and scale in its DNA. Edenlab, the Ukrainian company behind Kodjin , built our original FHIR solution to digitize and service the entire Ukrainian national health system. The learnings and technologies from that project informed our development of the Kodjin FHIR server. At Edenlab, we have always been driven by our passion for building solutions that excel in speed and scale. With Kodjin, we have embraced a modern tech stack to deliver unparalleled performance that can handle the demands of large-scale healthcare systems, providing efficient data management and seamless interoperability. Eugene Yesakov, Solution Architect, Author of Kodjin Built for speed and scale While most healthcare projects involve handling large volumes of data, including patient records, medical images, and sensor data, the Kodjin FHIR server is based on a system developed to handle tens of millions of patient records and thousands of requests per second, to ensure timely access and efficient decision-making for a population of over 40 million people. And all of this information had to be processed and exchanged in real-time or near real-time, without delays or bottlenecks. This article will explore some of the architectural decisions the Edenlab team took when building Kodjin, specifically the role MongoDB played in enhancing performance and ensuring scalability. We will examine the benefits of leveraging MongoDB's scalability, flexibility, and robust querying capabilities, as well as its ability to handle the increasing velocity and volume of healthcare data without compromising performance. About Kodjin FHIR server Kodjin is an ONC-certified and HIPAA-compliant FHIR Server that offers hassle-free healthcare data management. It has been designed to meet the growing demands of healthcare projects, allowing for the efficient handling of increasing data volumes and concurrent requests. Its architecture, built on a horizontally scalable microservices approach, utilizes cutting-edge technologies such as the Rust programming language, MongoDB, ElasticSearch, Kafka, and Kubernetes. These technologies enable Kodjin to provide users with a low-code approach while harnessing the full potential of the FHIR specification. A deeper dive into the architecture approach - the role of MongoDB in Kodjin When deciding on the technology stack for the Kodjin FHIR Server, the Edenlab team knew that a document database would be required to serve as a transactional data store. In an FHIR Server, a transactional data store ensures that data operations occur in an atomic and consistent manner, allowing for the integrity and reliability of the data. Document databases are well-suited for this purpose as they provide a flexible schema and allow for storing complex data structures, such as those found in FHIR data. FHIR resources are represented in a hierarchical structure and can be quite intricate, with nested elements and relationships. Document databases, like MongoDB, excel at handling such complex and hierarchical data structures, making them an ideal choice for storing FHIR data. In addition to supporting document storage, the Edenlab team needed the chosen database to provide transactional capabilities for FHIR data operations. FHIR transactions, which encompass a set of related data operations that should either succeed or fail as a whole, are essential for maintaining data consistency and integrity. They can also be used to roll back changes if any part of the transaction fails. MongoDB provides support for multi-document transactions , enabling atomic operations across multiple documents within a single transaction. This aligns well with the transactional requirements of FHIR data and ensures data consistency in Kodjin. Implementation of GridFS as a storage for the terminologies in Terminology service Terminology service plays a vital role in FHIR projects, requiring a reliable and efficient storage solution for terminologies used. Kodjin employs GridFS , a file system within MongoDB designed for storing large files, which makes it ideal to handle terminologies. GridFS offers a convenient way to store and manage terminology files, ensuring easy accessibility and seamless integration within the FHIR ecosystem. By utilizing MongoDB's GridFS, Kodjin ensures efficient storage and retrieval of terminologies, enhancing the overall functionality of the terminology service. Kodjin FHIR server performance To evaluate the efficiency and responsiveness of the Kodjin FHIR server in various scenarios we conducted multiple performance tests using Locust, an open-source load testing tool. One of the performance metrics measured was the retrieval of resources by their unique ids using the GET by ID operation. Kodjin with MongoDB achieved a performance of 1721.8 requests per second (RPS) for this operation. This indicates that the server can efficiently retrieve specific resources, enabling quick access to desired data. The search operation, which involves querying ElasticSearch to obtain the ids of the searched resources and retrieving them from MongoDB, exhibited a performance of 1896.4 RPS. This highlights the effectiveness of polyglot persistence in Kodjin, leveraging ElasticSearch for fast and efficient search queries and MongoDB for resource retrieval. The system demonstrated its ability to process search queries and retrieve relevant results promptly. In terms of resource creation, Kodjin with MongoDB showed a performance of 1405.6 RPS for POST resource operations. This signifies that the system can effectively handle numerous resource-creation requests. The efficient processing and insertion of new resources into the MongoDB database ensure seamless data persistence and scalability. Overall, the performance tests confirm that Kodjin with MongoDB delivers efficient and responsive performance across various FHIR operations. The high RPS values obtained demonstrate the system's capability to handle significant workloads and provide timely access to resources through GET by ID, search, and POST operations. Conclusion Kodjin leverages a modern tech stack including Rust, Kafka, and Kubernetes to deliver the highest levels of performance. At the heart of Kodjin is MongoDB, which serves as a transactional data store. MongoDB's capabilities, such as multi-document transactions and flexible schema, ensure the integrity and consistency of FHIR data operations. The utilization of GridFS within MongoDB ensures efficient storage and retrieval of terminologies, optimizing the functionality of the Terminology service. To experience the power and potential of the Kodjin FHIR server firsthand, we invite you to contact the Edenlab team for a demo. For more information On MongoDB’s work in healthcare, and to understand why the world’s largest healthcare companies trust MongoDB, read our whitepaper on radical interoperability .