MMS Release Notes: Automation and MongoDB 2.8 Prep Work
Another release of MMS is here!
In the realm of automation, today we are releasing the ability to provision on ephemeral SSD drives for all EC2 types.
Within the MMS UI, users can now search by replica set name, which is particularly helpful if you have hundreds of replica sets.
A new monitoring agent (126.96.36.199) and a new backup agent (188.8.131.52) were released - agents will now identify themselves to the MMS servers using the FQDN of the servers on which they are running.
The outlook for 2015 is “busy” – we’re just starting to roll out 2.8 across all of MMS Systems here internally, and we are sure customers are going to love all the new functionality.
On day one, we plan to have functionality for customers to easily and quickly upgrade to MongoDB, including choosing their storage engine. There will also be some changes to performance metrics in the UI - some metrics have been removed, including the old favorite “Lock %”. This is due to the fact that some metrics are no longer supported in MongoDB 2.8, and the metrics supported also depend on the storage engine chosen. 2.8 will also mean new metrics, so stay tuned for a list of those. In summary, this release was primarily pre-release prep work, which is not yet available in the UI.
Leaf in the Wild: Scaling China’s Largest Car Service App with MongoDB
Leaf in the Wild posts highlight real world MongoDB deployments. Read other stories about how companies are using MongoDB for their mission-critical projects. Kuaidi uses MongoDB at the heart of its taxi hailing service, connecting drivers with passengers up to 6 million times a day, and managing nearly half a billion orders. Kuaidi has scaled MongoDB across 4 geographic regions, serving thousands of reads and writes every second. Following his presentation at last month’s MongoDB Day in Beijing, I sat down with Ouyang Kang, Chief Architect at Kuaidi, to learn more about how China’s leading taxi booking application is using MongoDB, and his recommendations for those getting started with the database. Smartphone based taxi-calling and ride-sharing services are growing at an astounding rate – attracting significant investment (and huge company valuations). They are also intensely competitive. The choice of technology will ultimately drive success or failure in the market. In the world’s most populous country – and one suffering the most severe traffic congestion – the importance of using agile and scalable technology for transportation services is magnified. Please start by telling us a little bit about your company. Kuaidi was founded in 2012 and has grown to become Greater China’s largest car service application 1 , attracting investment from Alibaba and Matrix Partners. In just 2 years, we have attracted 100 million users who place up to 6 million ride requests every day via our smartphone app, connecting them to 3 million drivers in more than in 300 cities across China. And we are continuing to grow fast. The goal of Kuaidi Group is to improve the efficiency of urban transportation and the population’s quality of life. We currently operate 2 branded services – Kuaidi Taxi and Kuaidi ONE – which provide taxi and chauffeured limousine services respectively. Our long term plan is to offer services for every facet of passenger transportation combining location-based mobile technologies, data mining of our huge user base and intelligent routing algorithms. Tell us how you use MongoDB. At heart of our taxi booking application is the location based service, and we rely on MongoDB for this. Using MongoDB’s geospatial indexes and queries we can track the location of our drivers in real time, using it to connect users with their closest taxi, and displaying updates directly to the customer’s app. The location data is constantly being updated and queried. We also use MongoDB as an active archive of our order data. Each time a customer requests a taxi, the journey’s start and end points, the driver identity and fare are stored in a single record. We initially built our archive on top of MySQL, but once our order volume exceeded 100 million records, we hit scaling limits. We knew MongoDB scaled, so we migrated the archive to get the cost and performance benefits of horizontal scale out. What other databases do you use? We use Redis for caching and MySQL to store operational customer and order data. We also replicate data from MongoDB and MySQL into Hadoop for data mining and analytics. Did you consider other databases for your app? What made you select MongoDB? We considered three options for our location based service: Relational solutions based on MySQL and Postgres SOLR (for the search element of the application) MongoDB We evaluated each on multiple criteria, including Performance. We measure performance on multiple dimensions: latency, which is critical for good user experience on mobile apps; and speed of real time updates, so we are always working from the freshest data Scalability. We were confident that the service would quickly gain traction, so knowing we could scale our database on demand was paramount Ease-of-Use. We needed to achieve our performance and scalability goals without burdening our developer and operations team with complexity We evaluated all of the options on this criteria, and found MongoDB to be the best choice for us. It met the performance objectives. We found it easy to develop against. What was really important was that it proved easy to deploy and easy to run at scale . Please describe your MongoDB deployment Our MongoDB database is sharded across four geographic regions. A 7-node replica set is deployed in each region (6 data-bearing nodes and an arbiter). This deployment enables us to place data physically closer to local users for low latency access, as well as provide the scalability and resilience our application needs. We cannot tolerate downtime at all. We use Nagios for monitoring the application and database. Geo-Distributed MongoDB Deployment at Kuaidi We are running MongoDB 2.6 with the Java driver. Are there any metrics you can share? Yes. MongoDB is serving 50,000 operations per second (split 80:20 between reads and writes) Our database has grown to just under half a billion documents and continues to scale Do you have plans to use MongoDB for other applications? Our marketing team stores all of its promotions and messaging in MySQL, but is starting to hit scaling limits. As a result, it is not keeping pace with their demands. We are evaluating migrating this to MongoDB as well. What feature of the forthcoming MongoDB 3.0 release are you most looking forward to? It has to be document level concurrency control. As our service continues to grow, we need to scale to keep pace – especially writes. This is something we believe MongoDB 3.0 with its new WiredTiger storage engine will allow us to do. What advice would you give someone who is considering using MongoDB for their next project? Don’t just follow the crowd. Don’t just choose the same technology you have always chosen. There is so much innovation happening today, and the databases of the last decade are not always the right choice. Once you have a short-list of potential technologies, test them with your app, your queries and your data. It is the only way to be sure you are choosing the right technology going forward. Ouyang, thank you for your time, and sharing your experiences with the MongoDB community. Thinking about migrating from a relational database? Read the MongoDB white paper to get started: Migrating from RDBMS to MongoDB 1 Based on market share and transaction volume About the Author - Mat Keep Mat is part of the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies. < Read About Our William Zola Award for Community Excellence
How Edenlab Built a High-Load, Low-Code FHIR Server to Deliver Healthcare for 40 Million Plus Patients
The Kodjin FHIR server has speed and scale in its DNA. Edenlab, the Ukrainian company behind Kodjin , built our original FHIR solution to digitize and service the entire Ukrainian national health system. The learnings and technologies from that project informed our development of the Kodjin FHIR server. At Edenlab, we have always been driven by our passion for building solutions that excel in speed and scale. With Kodjin, we have embraced a modern tech stack to deliver unparalleled performance that can handle the demands of large-scale healthcare systems, providing efficient data management and seamless interoperability. Eugene Yesakov, Solution Architect, Author of Kodjin Built for speed and scale While most healthcare projects involve handling large volumes of data, including patient records, medical images, and sensor data, the Kodjin FHIR server is based on a system developed to handle tens of millions of patient records and thousands of requests per second, to ensure timely access and efficient decision-making for a population of over 40 million people. And all of this information had to be processed and exchanged in real-time or near real-time, without delays or bottlenecks. This article will explore some of the architectural decisions the Edenlab team took when building Kodjin, specifically the role MongoDB played in enhancing performance and ensuring scalability. We will examine the benefits of leveraging MongoDB's scalability, flexibility, and robust querying capabilities, as well as its ability to handle the increasing velocity and volume of healthcare data without compromising performance. About Kodjin FHIR server Kodjin is an ONC-certified and HIPAA-compliant FHIR Server that offers hassle-free healthcare data management. It has been designed to meet the growing demands of healthcare projects, allowing for the efficient handling of increasing data volumes and concurrent requests. Its architecture, built on a horizontally scalable microservices approach, utilizes cutting-edge technologies such as the Rust programming language, MongoDB, ElasticSearch, Kafka, and Kubernetes. These technologies enable Kodjin to provide users with a low-code approach while harnessing the full potential of the FHIR specification. A deeper dive into the architecture approach - the role of MongoDB in Kodjin When deciding on the technology stack for the Kodjin FHIR Server, the Edenlab team knew that a document database would be required to serve as a transactional data store. In an FHIR Server, a transactional data store ensures that data operations occur in an atomic and consistent manner, allowing for the integrity and reliability of the data. Document databases are well-suited for this purpose as they provide a flexible schema and allow for storing complex data structures, such as those found in FHIR data. FHIR resources are represented in a hierarchical structure and can be quite intricate, with nested elements and relationships. Document databases, like MongoDB, excel at handling such complex and hierarchical data structures, making them an ideal choice for storing FHIR data. In addition to supporting document storage, the Edenlab team needed the chosen database to provide transactional capabilities for FHIR data operations. FHIR transactions, which encompass a set of related data operations that should either succeed or fail as a whole, are essential for maintaining data consistency and integrity. They can also be used to roll back changes if any part of the transaction fails. MongoDB provides support for multi-document transactions , enabling atomic operations across multiple documents within a single transaction. This aligns well with the transactional requirements of FHIR data and ensures data consistency in Kodjin. Implementation of GridFS as a storage for the terminologies in Terminology service Terminology service plays a vital role in FHIR projects, requiring a reliable and efficient storage solution for terminologies used. Kodjin employs GridFS , a file system within MongoDB designed for storing large files, which makes it ideal to handle terminologies. GridFS offers a convenient way to store and manage terminology files, ensuring easy accessibility and seamless integration within the FHIR ecosystem. By utilizing MongoDB's GridFS, Kodjin ensures efficient storage and retrieval of terminologies, enhancing the overall functionality of the terminology service. Kodjin FHIR server performance To evaluate the efficiency and responsiveness of the Kodjin FHIR server in various scenarios we conducted multiple performance tests using Locust, an open-source load testing tool. One of the performance metrics measured was the retrieval of resources by their unique ids using the GET by ID operation. Kodjin with MongoDB achieved a performance of 1721.8 requests per second (RPS) for this operation. This indicates that the server can efficiently retrieve specific resources, enabling quick access to desired data. The search operation, which involves querying ElasticSearch to obtain the ids of the searched resources and retrieving them from MongoDB, exhibited a performance of 1896.4 RPS. This highlights the effectiveness of polyglot persistence in Kodjin, leveraging ElasticSearch for fast and efficient search queries and MongoDB for resource retrieval. The system demonstrated its ability to process search queries and retrieve relevant results promptly. In terms of resource creation, Kodjin with MongoDB showed a performance of 1405.6 RPS for POST resource operations. This signifies that the system can effectively handle numerous resource-creation requests. The efficient processing and insertion of new resources into the MongoDB database ensure seamless data persistence and scalability. Overall, the performance tests confirm that Kodjin with MongoDB delivers efficient and responsive performance across various FHIR operations. The high RPS values obtained demonstrate the system's capability to handle significant workloads and provide timely access to resources through GET by ID, search, and POST operations. Conclusion Kodjin leverages a modern tech stack including Rust, Kafka, and Kubernetes to deliver the highest levels of performance. At the heart of Kodjin is MongoDB, which serves as a transactional data store. MongoDB's capabilities, such as multi-document transactions and flexible schema, ensure the integrity and consistency of FHIR data operations. The utilization of GridFS within MongoDB ensures efficient storage and retrieval of terminologies, optimizing the functionality of the Terminology service. To experience the power and potential of the Kodjin FHIR server firsthand, we invite you to contact the Edenlab team for a demo. For more information On MongoDB’s work in healthcare, and to understand why the world’s largest healthcare companies trust MongoDB, read our whitepaper on radical interoperability .