Deploy and Manage MongoDB Atlas from AWS CloudFormation
June 21, 2021
As a premier launch partner for the recent GA announcement of the AWS CloudFormation Public Registry, we’re delighted to share that you can now deploy and manage MongoDB Atlas directly from your AWS environment.
Amazon and MongoDB have been pioneers in the cloud computing space, providing mission critical systems for over a decade. Before MongoDB Atlas was launched in June 2016, tens of thousands of customers were running MongoDB themselves on AWS EC2 instances, and many of them were originally spun up using the legacy MongoDB on the AWS Cloud: Quick Start Reference Deployment. This Quick Start was among the top five most popular guides for AWS and allows users to take advantage of AWS CloudFormation's seamless automation and MongoDB’s flexible data model and expressive query API.
In April 2021, we launched a new AWS Quick Start for MongoDB Atlas, which allows AWS customers to quickly and easily launch a basic MongoDB Atlas deployment from the AWS CLI or console. Now, with the availability of the MongoDB Atlas resource types on the CloudFormation Public Registry, customers have more flexibility over their deployment configurations to better meet their cloud workflows. Let’s walk through how it works.
Setup your AWS account for MongoDB Atlas CloudFormation Support
The first step is to sign up for MongoDB Atlas, if you haven’t done so already. Once you create your account, follow these steps:
- Skip the cluster deployment options
- Go to Billing and add a credit card to your account
- Create an organization-level MongoDB Atlas Programmatic API Key with an IP Access List entry. The key needs
Organization Project Creatorpermissions.
Next, open the AWS console in your browser and navigate to CloudFormation. On the left-side navigation, select the Public extensions option. From there you will be able to find the MongoDB Atlas resource types by selecting the “Resource Types” and “Third Party” options.
For each of the MongoDB::Atlas resource types, click “Activate”, and then follow on screen prompts to complete the process.
Once you have activated the MongoDB Atlas resources in a region, you’re ready to launch apps with MongoDB Atlas directly from your AWS control plane.
Build apps faster with Cloud Automation
Context switching is a hassle for developers. Launching and deploying application stacks with MongoDB Atlas directly from the AWS console is now more seamless than ever. Whether you use the AWS Quick Start deployment guide as a template or create your own MongoDB Atlas CloudFormation templates, you can leverage the latest in cloud automation to reduce the pain of infrastructure provisioning and management.
Try out the new MongoDB Atlas CloudFormation Resources today, and stay tuned for an in depth look at building apps with AWS Lambda and SAM CLI in an upcoming DevHub article!
Accelerate Digital Transformation With MongoDB's Developer Data Platform
Business disruption isn’t new. But it is accelerating to a new, ever-more-disruptive level. That acceleration is being amplified by the forces of our era, from global health crises to political shifts to climate change, among others. These disruptive forces are creating huge changes in corporate behavior and consumer expectations. When considering changing business conditions, whether from adaptation or innovation, it’s worth asking, “Is this something that is going to default back to the way it was before?” In most cases, the answer is "probably not.” Disruption is here to stay. While that can be scary for incumbent enterprises, we believe that disruption is the opposite of stagnation. It is an opportunity for change, adaptation, and movement. In 2020, many companies had to completely rethink how they accomplished their business goals and were pushed to implement new strategies at an accelerated pace. Years of digital transformation were condensed into months. Companies developed new structures and flexible models — internal muscles that will continue serving them as they move forward. The pace at which digital transformation happens is only accelerating. With an increasingly digital economy, we’re seeing: No more barriers to entry. Obstacles to entering the digital economy are essentially gone. For example, a bank considering fintech disruption is no longer just competing with interregional banks. It’s now competing with every mobile-first challenger bank anywhere in the world. Competition is more high stakes than ever. A need to enable sustainable innovation. The most competitive businesses are maniacally focused on removing every obstacle so that their teams can be agile and move quickly in a scalable and sustainable way. An emphasis on data security. As more of the IT landscape moves into the cloud, and as organizations increasingly go global, data security and privacy need to be at the forefront. It’s critical that a company’s customers trust them to be the stewards of their data now and into the future. However, not everyone’s digital However, not everyone’s digital transformation is successful. Many companies can’t keep up with new competitors and sudden market shifts. Your competitive advantage Competitive advantage is now directly tied to how well companies are able to build software around their most important asset: data. Companies have been using commercial software since the early 1970s. What is different now is that their differentiation as a business is tied to the software they build internally. No one is expecting to win in their industry because they use an off-the-shelf CRM product, for example. That is to say: Competitive advantage cannot be bought, it must be built. This is not a new idea. Even the most basic software cannot work without proper storage, retrieval, and use of data. Otherwise, every application would have a beautiful front end but nothing functional behind them. The true art and skill of modern application development is to tap into data and wield it to create true innovation. Customer experience and expectations Almost 15 years into the smartphone and smart device era, consumer and B2B expectations for their digital interactions and experiences are extremely high and exceptionally demanding. Customers expect their digital experiences to: Be highly responsive: Digital experiences must be quick to react to events, actions, and consumer behavior. Deliver relevant information: Modern digital experiences present the most relevant information intelligently, sometimes even predicting what a consumer is searching for before they complete their thought. Embrace mobile first: Mobile is becoming the primary way customers interact with companies. They expect to be able to do everything they would’ve done on a desktop from their mobile devices. Uphold data privacy: Customers expect complete data privacy — and for companies to allow them to take control of their data when requested. Be powered by real-time analytics: Customers expect applications to be smart. In addition to all of the above, consumers expect apps to guide them, assure them, and delight them with rich experiences powered by analytics and delivered in real time. Continuously improve: Customer expectations demand improvements at a faster rate than ever before. Legacy infrastructure: A challenge in digital transformation Companies' ability to deliver on customer expectations is almost entirely reliant on their underlying data infrastructure — it’s the foundation of their entire tech stack. Modern digital experiences demand a modern data infrastructure that addresses how data is stored, processed, and used. Despite companies’ best efforts — and significant spending — it’s estimated that more than 70% of enterprises fail in their digital transformation initiatives . This alarming number can make it appear as though digital improvement is a gamble not worth taking, but this is most definitely not the case. The truth is that though the way companies leverage data to build modern applications has changed, the typical data infrastructure has not kept up with the demands, making working with data the hardest part of building and evolving applications. One key factor is that typical data infrastructures are still built around legacy relational databases. These outdated infrastructures mean: Rigidity. The rigidity of the rows and columns that define relational databases make experimenting and iterating on applications difficult. Anytime a data model or schema needs to be changed as an application evolves or incorporates new types of data, developers must consider dependencies at the data layer, and the brittle nature of relational databases makes such change difficult. Data structure clashes. Relational tabular data structures are at odds with how most developers think and code, which is through object-oriented programming languages. To put it simply, developers do not think in rows and columns, which clash with modern data and the objects developers work with. Automatic failover and horizontal scaling are not natively supported. The essentials that modern applications need, such as automatic failover and support for massive scale on demand, are not natively built into legacy relational databases; these become more obstacles to overcome. With a relational data infrastructure, it’s typical for an organization to have hundreds or thousands of tables built up over years. Having to update or unwind them as it is trying to build or iterate on its applications brings innovation to a crawl — or puts it on pause altogether. As an example, Travelers , a Fortune 500 U.S.-based insurance company, recently attempted to modernize an underwriting application. Their most profitable unit, business insurance, required much faster software delivery and response times. The company attempted to solve this with standard solutions, such as implementing agile development, breaking down monoliths with microservices, and rolling out continuous delivery. Despite their best efforts, however, legacy relational databases held them back. Travelers’ senior director of architecture at the time, Jeff Needham, in reference to their attempts at transformation, said, “At the end, it was still the legacy database. We implemented all these things, but we still had to wait days for a database change to be implemented.” Both Travelers’ failing result and eventual frustration are shared by many organizations that get ensnared in their own data infrastructures. What about NoSQL? For teams that need to deliver more modern application experiences or operate at faster speeds, it might appear that the most obvious path is to add NoSQL datastores as a bandage to address relational shortcomings. But doing so requires ETL (extract, transform, and load data from one source to another), adding more complexity to data management. These teams quickly realize that non-relational or NoSQL databases only excel at a few select objectives, with otherwise limited capabilities (including limited query capabilities and the lack of data consistency guarantees). The truth is that simply adding on a NoSQL database to a data infrastructure, adds needless complexity, drains resources, and really only serves niche use cases. In the end, it’s not just one database being added and requiring management but several — one for graph data and traversals, one for time series, one for key values, and so on. The ever-increasing need to address diversified data workloads means a new managed database for each type of data, creating even more silos. The bottom line is that adding NoSQL to cover what relational databases can’t makes the data environment even more complex than it was before. Beyond operational databases Today, an organization’s application data infrastructure is made up of more than just operational databases. To deliver rich search capabilities, companies often add separate search engine technologies to their environments, putting the onus on their teams to manage the movement of data between systems. To enable low-latency and offline-first application experiences on mobile devices, they often add separate local data storage technologies. Syncing data from devices to the backend becomes another spinning plate for developers to keep up with since it involves complexities such as networking code and conflict resolution. Finally, to create rich analytics-backed application experiences, more often than not organizations use ETL for their data, reformatting it along the way for an entirely separate analytics database. Every step of the way, more time, people, and money goes toward what is now a growing data infrastructure problem — an increasing sprawl of complexity — and eating away at development cycles that could otherwise be spent innovating their products. Spaghetti architecture is a tax on innovation As they try to solve data issues by adding new components, services, or technologies, many companies find themselves trapped in “spaghetti architecture,” meaning overly complex and siloed architectures piled on top of already heavy infrastructures. Each bit of technology has its quirks from operational, security, and performance standpoints, demanding expertise and attention — and making data integration difficult. Moving data between systems requires dedicated people, teams, and money. Massive resources go into dealing with the incredible amount of data duplication. But beyond just cost, development resources must go toward dealing with multiple operational and security models when data is distributed across so many different systems. This makes it incredibly difficult to innovate in a scalable, sustainable way. In fact, this is why many digital transformations fail: inadequate data infrastructures, burning through resources, and “solutions” creating more complexity. And all while, they are falling behind their competitors. We think of all of this as a recurring tax on innovation tied to an ever-growing data infrastructure problem that we call DIRT (data and innovation recurring tax). DIRT is recurring because it never goes away by itself. It’s a 2,000-pound boulder strapped to a team’s back today, tomorrow, and five years from now. It will continue to weigh down teams until they address it head on. Eliminate DIRT DIRT is a real problem, but there are equally real, and realistic, solutions. The most successful and advanced organizations avoid such complexities altogether by building data infrastructures focused on four key guidelines: Doubling developer productivity. Companies’ success depends on their developers’ ability to create industry-leading applications, so these businesses prioritize removing any obstacles to productivity, including rigid data structures, fragmented developer experiences, and backend maintenance. Prioritizing elegant, repeatable architectures. The companies that will win the race toward data integrity understand the cost of bespoke data infrastructures and technologies that only make their production environments more complex. These companies use niche technologies only when absolutely necessary. Intentionally addressing security and data privacy. Successful businesses don’t let data security and privacy become a separate and massive project. They’re able to satisfy sophisticated requirements without compromising simplicity or the developer experience. Leveraging the power of multi-cloud. These companies don’t compromise on deployment flexibility. They’re ahead of data gravity and can deploy a single application globally across multiple regions and multiple clouds without having to rewrite code or spend months in planning. How MongoDB helps MongoDB provides companies with a developer data platform that allows them to move fast and simplify how they build with data for any application. This allows organizations to spend less effort rationalizing their data infrastructure and focus more on innovation and building out their unique differentiation, eliminating DIRT. The document model is flexible and maps to how developers think and code. The flexible document data model makes it easy to model and remodel data as application requirements change, increasing developer productivity by eliminating the headache of rows and columns. Instead, documents map exactly to the objects that developers work with in their code. This is the core insight that MongoDB’s founders had at least a decade ago: Data systems fundamentally need a different data model to be able to match modern development. This is also why MongoDB has become so popular with developers, with more than 265 million downloads to date. MongoDB documents are a superset of all other data models. The MongoDB document model upholds the superset of legacy functions, allowing users to store and work with data of various types and shapes. In contrast to niche databases, it covers the needs of relational data, objects, cache formats, and specialized data such as GIS data types or time series data. Document databases are not just one of many other databases to be used simultaneously. Advanced organizations realize that the document model underpins a broad spectrum of use cases. For example, the simplest documents serve as key-value pairs. Documents can be modeled as the nodes and edges of a graph. Documents are actually more intuitive for modeling relationships with support for embedding and nested structures. The ability to work with diverse varieties of data fits neatly within the document data model, giving MongoDB a concrete foundation to build from. MongoDB features a powerful, expressive, and unified interface. This provides for improved productivity because developers do not need to research, learn, and stay up-to-date on multiple ways to work with data across their different workloads. It’s also much more natural to use than SQL because the developer experience is one that feels like an extension of programming languages. The experience is idiomatic to each programming language and paradigm; developers can view MongoDB documents in their native format and work directly with the data without the need for abstraction layers such as object relational mappers (ORMs), data abstraction layers (DALs), and more — they can simply be removed or retired. Furthermore, multiple different teams working in different programming environments — from C# to Java to C++ — can access the same data at their leisure, allowing simplification and integration of data domains. MongoDB Atlas, a developer data platform MongoDB Atlas is more than just a database. It is a multi-purpose, multi-faceted developer data platform. This means that MongoDB recognizes data comes in a wide variety of formats, needs to be processed, stored, trained (and so on) in a broad variety of ways, and needs to be regulated, audited, encrypted, and secured in a similarly diverse set of ways. Data is one of the most valuable yet complex assets companies have. MongoDB simplifies many different use cases to wield this important asset in an intelligent, beautiful way by offering a unified interface to work with data generated by modern applications. MongoDB brings together two foundational concepts — the document model and a unified query API — in the form of an operational and transactional database. MongoDB’s Atlas offers: A transactional database: MongoDB has the transactional guarantees and data governance capabilities required to not just supplement but actually replace relational databases. Distributed multi-document transactions are fully ACID compliant, making it the transactional database behind core mission-critical applications. Search capabilities: Fully integrated full-text search eliminates the need for separate search engines. The MongoDB platform includes integrated search capabilities, including an extended query API, so developers are not forced to stand up a dedicated search engine to enable application search. All of the operations, including data movement, is handled natively within the MongoDB platform. Mobile readiness: MongoDB Realm’s flexible local datastore includes seamless edge-to-cloud sync for mobile and computing at the edge. MongoDB Realm enables agility for mobile developers through a natural development experience that syncs data from the backend to the front end with minimal code required. Things like conflict resolution, networking code, and plumbing are all handled automatically. Real-time analytics: MongoDB offers real-time analytics with workload isolation and native data visualization. As more organizations design smarter applications with MongoDB, they can call on real-time analytics tied to either machine learning or direct application experiences. Data lake: With MongoDB, developers can run federated queries across operational databases, transactional databases, and cloud object storage. Queries can also be extended across multiple clusters, or even to data sitting outside of MongoDB. MongoDB’s architecture is able to federate and aggregate data for analytical purposes as needed. A sustainable platform: In real-world applications, no capabilities matter if the platform is not secure, resilient, and scalable. Only sustainable frameworks can evolve with changes in the market and demand for the product. Scalability and compliance: Everything at MongoDB is built on a distributed systems architecture, with turnkey global data distribution for data sovereignty and fast access. This is not just for horizontal scaling and linear cost economics as workloads get larger, but it also helps organizations handle data distribution for their global applications, keeping relevant data close to the user but, for example, distributed across different geographic regions as needed to deliver a low-latency experience. MongoDB can also be used to isolate data by country to address data sovereignty regulations. Security: MongoDB holds industry-leading data privacy controls with client-side field level encryption, having built security into the core of the database, whether it's with encrypted storage, role-based access controls, or enterprise-grade auditing. In a world where there are often third-party providers involved, this gives more control to the end customer so they can definitively say that no third-party provider can access sensitive data, preventing full breaches of security. Multi-cloud: With MongoDB, developers have the flexibility to deploy across clouds in nearly 80 regions. Extending their databases across multiple clouds allows developers to leverage the innovative services that may be associated with another provider, build cross-cloud resiliency, and get additional reach without having to stand up separate databases in different regions. This, in turn, allows for a unified developer experience across data workloads, a simpler operational and security model, an automated and transparent data movement between services, and reduction of the dreaded data duplication. Interested in getting started with MongoDB Atlas for your digital transformation? Start for free here or contact us directly.
How Edenlab Built a High-Load, Low-Code FHIR Server to Deliver Healthcare for 40 Million Plus Patients
The Kodjin FHIR server has speed and scale in its DNA. Edenlab, the Ukrainian company behind Kodjin , built our original FHIR solution to digitize and service the entire Ukrainian national health system. The learnings and technologies from that project informed our development of the Kodjin FHIR server. At Edenlab, we have always been driven by our passion for building solutions that excel in speed and scale. With Kodjin, we have embraced a modern tech stack to deliver unparalleled performance that can handle the demands of large-scale healthcare systems, providing efficient data management and seamless interoperability. Eugene Yesakov, Solution Architect, Author of Kodjin Built for speed and scale While most healthcare projects involve handling large volumes of data, including patient records, medical images, and sensor data, the Kodjin FHIR server is based on a system developed to handle tens of millions of patient records and thousands of requests per second, to ensure timely access and efficient decision-making for a population of over 40 million people. And all of this information had to be processed and exchanged in real-time or near real-time, without delays or bottlenecks. This article will explore some of the architectural decisions the Edenlab team took when building Kodjin, specifically the role MongoDB played in enhancing performance and ensuring scalability. We will examine the benefits of leveraging MongoDB's scalability, flexibility, and robust querying capabilities, as well as its ability to handle the increasing velocity and volume of healthcare data without compromising performance. About Kodjin FHIR server Kodjin is an ONC-certified and HIPAA-compliant FHIR Server that offers hassle-free healthcare data management. It has been designed to meet the growing demands of healthcare projects, allowing for the efficient handling of increasing data volumes and concurrent requests. Its architecture, built on a horizontally scalable microservices approach, utilizes cutting-edge technologies such as the Rust programming language, MongoDB, ElasticSearch, Kafka, and Kubernetes. These technologies enable Kodjin to provide users with a low-code approach while harnessing the full potential of the FHIR specification. A deeper dive into the architecture approach - the role of MongoDB in Kodjin When deciding on the technology stack for the Kodjin FHIR Server, the Edenlab team knew that a document database would be required to serve as a transactional data store. In an FHIR Server, a transactional data store ensures that data operations occur in an atomic and consistent manner, allowing for the integrity and reliability of the data. Document databases are well-suited for this purpose as they provide a flexible schema and allow for storing complex data structures, such as those found in FHIR data. FHIR resources are represented in a hierarchical structure and can be quite intricate, with nested elements and relationships. Document databases, like MongoDB, excel at handling such complex and hierarchical data structures, making them an ideal choice for storing FHIR data. In addition to supporting document storage, the Edenlab team needed the chosen database to provide transactional capabilities for FHIR data operations. FHIR transactions, which encompass a set of related data operations that should either succeed or fail as a whole, are essential for maintaining data consistency and integrity. They can also be used to roll back changes if any part of the transaction fails. MongoDB provides support for multi-document transactions , enabling atomic operations across multiple documents within a single transaction. This aligns well with the transactional requirements of FHIR data and ensures data consistency in Kodjin. Implementation of GridFS as a storage for the terminologies in Terminology service Terminology service plays a vital role in FHIR projects, requiring a reliable and efficient storage solution for terminologies used. Kodjin employs GridFS , a file system within MongoDB designed for storing large files, which makes it ideal to handle terminologies. GridFS offers a convenient way to store and manage terminology files, ensuring easy accessibility and seamless integration within the FHIR ecosystem. By utilizing MongoDB's GridFS, Kodjin ensures efficient storage and retrieval of terminologies, enhancing the overall functionality of the terminology service. Kodjin FHIR server performance To evaluate the efficiency and responsiveness of the Kodjin FHIR server in various scenarios we conducted multiple performance tests using Locust, an open-source load testing tool. One of the performance metrics measured was the retrieval of resources by their unique ids using the GET by ID operation. Kodjin with MongoDB achieved a performance of 1721.8 requests per second (RPS) for this operation. This indicates that the server can efficiently retrieve specific resources, enabling quick access to desired data. The search operation, which involves querying ElasticSearch to obtain the ids of the searched resources and retrieving them from MongoDB, exhibited a performance of 1896.4 RPS. This highlights the effectiveness of polyglot persistence in Kodjin, leveraging ElasticSearch for fast and efficient search queries and MongoDB for resource retrieval. The system demonstrated its ability to process search queries and retrieve relevant results promptly. In terms of resource creation, Kodjin with MongoDB showed a performance of 1405.6 RPS for POST resource operations. This signifies that the system can effectively handle numerous resource-creation requests. The efficient processing and insertion of new resources into the MongoDB database ensure seamless data persistence and scalability. Overall, the performance tests confirm that Kodjin with MongoDB delivers efficient and responsive performance across various FHIR operations. The high RPS values obtained demonstrate the system's capability to handle significant workloads and provide timely access to resources through GET by ID, search, and POST operations. Conclusion Kodjin leverages a modern tech stack including Rust, Kafka, and Kubernetes to deliver the highest levels of performance. At the heart of Kodjin is MongoDB, which serves as a transactional data store. MongoDB's capabilities, such as multi-document transactions and flexible schema, ensure the integrity and consistency of FHIR data operations. The utilization of GridFS within MongoDB ensures efficient storage and retrieval of terminologies, optimizing the functionality of the Terminology service. To experience the power and potential of the Kodjin FHIR server firsthand, we invite you to contact the Edenlab team for a demo. For more information On MongoDB’s work in healthcare, and to understand why the world’s largest healthcare companies trust MongoDB, read our whitepaper on radical interoperability .