MongoDB Atlas Integrations for AWS CloudFormation and CDK are now Generally Available

Darshana Paithankar and Zuhair Ahmed
February 28, 2023 | Updated: May 15, 2023
#Atlas

Infrastructure as Code (IaC) tools allows developers to manage and provision infrastructure resources through code, rather than through manual configuration. IaC have empowered developers to apply similar best practices from software development to application instructure deployments. This includes:

  • Automation - helping to ensure repeatable, consistent, and reliable infrastructure deployments

  • Version Control - check in IaC code into GitHub, BitBucket, AWS CodeCommit, or GitLab for improved team collaboration and higher code quality

  • Security - create clear audit trails of each infrastructure modification

  • Disaster Recovery - IaC scripts can be used to quickly recreate infrastructure in the event of availability zone or region outages

  • Cost Savings - prevent overprovisioning and waste of cloud resources

  • Improved Compliance - easier to enforce organizational policies and standards

Today we are doubling down on this commitment and announcing MongoDB Atlas integrations with AWS CloudFormation and Cloud Development Kit (CDK). AWS CloudFormation allows customers to define and provision infrastructure resources using JSON or YAML templates. CloudFormation provides a simple way to manage infrastructure as code and automate the deployment of resources. AWS Cloud Development Kit (CDK) is an open-source software development framework that allows customers to define cloud infrastructure in code and provision it through AWS CloudFormation. It supports multiple programming languages and allows customers to use high-level abstractions to define infrastructure resources. These new integrations are built on top of the Atlas Admin API and allow users to automate infrastructure deployments by making it easy to provision, manage, and control Atlas Infrastructure as Code in the cloud.

MongoDB Atlas & AWS CloudFormation:

To meet developers where they are, we now have multiple ways to get started with MongoDB Atlas using AWS Infrastructure as Code. Each of these allow users to provision, manage, and control Atlas infrastructure as code on AWS:

Option 1: AWS CloudFormation

Customers can begin their journey using Atlas resources directly from the AWS CloudFormation Public Registry. We currently have 33 Atlas Resources and will continue adding more. Examples of available Atlas resources today include: Dedicated Clusters, Serverless Instances, AWS PrivateLink, Cloud Backups, and Encryption at Rest using Customer Key Management. In addition, we have published these resources to 22 (and counting) AWS Regions where MongoDB Atlas is supported today.

Learn how to get started via this quick demo.

Option 2: AWS CDK

After its launch in 2019 as an open source project, AWS CDK has gained immense popularity among the developer community with over a thousand external contributors and more than 1.3 million weekly downloads. AWS CDK abstracts away the low-level details of cloud infrastructure, making it easier for developers to define and manage their infrastructure natively in their programming language of choice. This helps to simplify the deployment process and eliminates context switching. Under the hood, AWS CDK synthesizes CloudFormation templates on your behalf which is then deployed to AWS accounts. In AWS CDK, L1 (Level 1) and L2 (Level 2) constructs refer to two different levels of abstraction for defining infrastructure resources: L1 constructs are lower-level abstractions that provide a one-to-one mapping to AWS CloudFormation resources. They are essentially AWS CloudFormation resources wrapped in code, making them easier to use in a programming context. L2 constructs are higher-level abstractions that provide a more user-friendly and intuitive way to define AWS infrastructure. They are built on top of L1 constructs and provide a simpler and more declarative API for defining resources.

Today we announce MongoDB Atlas availability for AWS CDK in JavaScript and TypeScript, with plans for Python, Java, Go, and .NET support coming later in 2023. Now customers can easily deploy and manage all available Atlas resources by vending AWS CDK applications with prebuilt L1 Constructs. We also have a growing number of L2 and L3 CDK Constructs available. These include Constructs to help users to quickly deploy the core resources they need to get started with MongoDB Atlas on AWS in just a few lines JavaScript or TypeScript (see awscdk-resources-mongodbatlas to learn more). Users can also optionally select to add more advanced networking configurations such as VPC peering and AWS PrivateLink.

Option 3: AWS Partner Solutions (previously AWS Quick Starts)

Instead of manually pulling together multiple Atlas CloudFormation resources, AWS Partner Solutions gives customers access to pre-built CloudFormation templates for both general and specific use cases with MongoDB Atlas. By using AWS Partner Solution templates, customers can save time and effort compared to architecting their deployments from scratch. These were jointly created and incorporate best practices from MongoDB Atlas and AWS. Go to the AWS Partner Solutions Portal to get started.

Start building today!

These MongDB Atlas integrations with AWS CloudFormation are free and open source licensed under Apache License 2.0. Users only pay for underlying Atlas resources created and can get started with Atlas always free tier (M0 clusters).

Getting started today is faster than ever with MongoDB Atlas and AWS CloudFormation. We can’t wait to see what you will build next with this powerful combination!

Learn more about MongoDB Atlas integrations with AWS CloudFormation

← Previous

MongoDB Atlas as the Data Hub for Smart Manufacturing with Microsoft Azure

All the source code used in this project, along with a detailed deployment guide, is available on our public Github page . Manufacturing companies are emerging from the pandemic with a renewed focus on digital transformation and smart factories investment. COVID-19 has heightened the need for Industrial IoT technology and innovation as consumers have moved towards online channels, forcing manufacturers to compete in a digitalized business environment. The manufacturing ecosystem can be viewed as a multi-dimensional grouping of systems designed to support the various business units in manufacturing organizations such as operations, engineering, maintenance, and learning & development functions. Process and equipment data is generated on the shop floor from machines and systems such as SCADA and then stored in a process historian or an operational database. The data originating from shop floor devices are generally structured time series data acquired through regular polling and sampling. Historians provide fast insertion rates of time series data, with capacities that reach up to tens of thousands of PLC tags processed per second. They rely on efficient data compression engines which can either be lossy or lossless. Traditional RDBMS storage comes packaged with the manufacturing software applications such as a Manufacturing Execution System (MES). Relational databases are traditionally common in manufacturing systems and thus the choice of database systems for these manufacturing applications are typically driven by historical preferences. Manufacturing companies have long relied on using several databases and data warehouses to accommodate various transactional and analytical workloads. The strategy of separating operational and analytical systems has worked well so far and has caused least interference with the operational process. However this strategy will not fare well in the near future for two reasons: Manufacturers are generating high volume, variety and veracity data using advanced IIoT platforms to create a more connected product ecosystem. The growth of IIoT data has been rapid and in fact, McKinsey and Company estimates that companies will spend over $175B in IIoT and edge computing hardware by 2025. A traditional manufacturing systems setup necessitates the deployment and maintenance of several technologies including graph databases (for asset digital models and relationships) and time series databases (for time series sensor data) and leads to IT sprawl across the organization. A complex infrastructure causes latency and delays in data access which leads to non-realization of real time insights for improving manufacturing operations. To establish an infrastructure that can enable real time analytics, companies need real time access to data and information to make the right decision in time. Analytics can no longer be a separate process, it needs to be brought into the application. The applications have to be supplied with notifications and alerts instantly. This is where application-driven analytics platforms such as MongoDB Atlas come into picture. We understand that to build smarter and faster applications, we can no longer rely on maintaining separate systems for different transactional and analytical workloads. Moving data between disparate systems takes time and energy and results in longer time to market and slower speed of innovation. Many of our customers start out using MongoDB as an operational database for both new cloud-native services as well as modernized legacy apps. More and more of these clients are now improving customer experience and speeding business insight by adopting application-driven analytics within the MongoDB Atlas platform. They use MongoDB to support use cases in real-time analytics, customer 360, internet of Things (IoT) and mobile applications across all industry sectors. As mentioned before, Manufacturing ecosystem employs a lot of databases just to run production operations. Once IIoT solutions are added to the mix, each solution (shown in yellow in Figure 1) may come with its own database (Time Series, relational, graph etc.) and the number of databases will increase dramatically. With MongoDB Atlas, this IT sprawl can be reduced as multiple use cases can be enabled using MongoDB Atlas (Figure 2). The versatility of the document model to structure data any way the application needs, coupled with an expressive API and indexing that allows you to query data any way you want is a powerful value proposition. The benefits of MongoDB Atlas are amplified by the platform’s versatility to address almost any workload. Atlas combines transactional processing, application-driven analytics, relevance-based search, and mobile edge computing with cloud sync. These capabilities can be applied to almost every type of modern applications being built for the digital economy by developers. Figure 1: IT sprawl with IIoT and analytics solutions deployment in Manufacturing Figure 2: MongoDB Atlas simplifying road to Smart Manufacturing MongoDB and Hyperscalers leading the way for smart manufacturing Manufacturers who are actively investing in digital transformation and IIoT are experiencing an exponential growth in data. All this data offers opportunities for new business models and digital customer experiences. To drive the right outcomes from all this data, manufacturers are setting up scalable infrastructures using Hyperscalers such as Azure, AWS and GCP. These hyperscalers offer a suite of components for efficient, scalable implementation of IIoT platforms. Companies are leveraging these accelerators to quickly build solutions, which help access, organize, and analyze previously untapped data from sensors, devices, and applications. In this article, we are focused on how MongoDB integrates with Microsoft Azure IoT modules and acts as the digital data hub for smart manufacturing use cases. MongoDB and Microsoft have been partners since 2019, but last year it was expanded, enabling developers to build data intensive applications within the Azure marketplace and Azure portal. This enables an enhanced developer experience and allows burn down of their Microsoft Azure Consumption Commitment. The alliance got further boost when Microsoft included MongoDB as a partner in its newly launched Microsoft Intelligent Data Platform Ecosystem . MongoDB Atlas can be deployed in 35 regions in Azure and has seamless integration with most of the Azure Developer services (Azure functions, App services, ADS), Analytics services (Azure Synapse), Data Governance (Microsoft Purview), ETL (ADF) and cross cutting services (AD, KMS, AKS etc.) powering building of innovative solutions. Example scenario: Equipment failure prediction Imagine a manufacturing facility that has sensors installed in their Computer Numerical Control (CNC) machines measuring parameters such as temperature, torque, rotational speed and tool wear. A sensor gateway converts analog sensor data to digital values and pushes it to Azure IoT Edge which acts as a gateway between factory and the Cloud. This data is transmitted to Azure IoT Hub where the IoT Edge is registered as an end device. Once we have the data in the IoT Hub, Azure Stream Analytics can be utilized to filter the data so that only relevant information flows into the MongoDB Atlas Cluster. The connection between Stream Analytics and MongoDB is done via an Azure Function. This filtered sensor data inside MongoDB is used for following purposes: To provide data for machine learning model that will predict the root cause of machine failure based on sensor data. To act as a data store for prediction results that can be utilized by business intelligence tools such as PowerBI using Atlas SQL Interface. To store the trained machine learning model checkpoint in binary encoded format inside a collection. The overall architecture is shown in Figure 3. Figure 3: Overall architecture Workflow: The sensors in the factory are sending time series measurements to Azure IoT Hub. These sensors are measuring for multiple machines: Product Type Air Temperature (°C) Process Temperature (°C) Rotational Speed Torque Tool Wear (min) IoT Hub will feed these sensor data to Azure Stream Analytics, where the data will be filtered and pushed to MongoDB Atlas time series collections. The functionality of Stream Analytics can be extended by implementing machine learning models to do real-time predictive analytics on streaming input data. The prediction results can also be stored in MongoDB in a separate collection. The sensor data contains the device_id field which helps us filter data coming from different machines. As MongoDB is a document database, we do not need to create multiple tables to store this data, in fact we can just use one collection for all the sensor data coming from various devices or machines. Once the data is received in MongoDB, sum and mean values of sensor data will be calculated for the predefined production shift duration and the results will be pushed to MongoDB Atlas Charts for visualization. MongoDB Time series window functions are used in an aggregation pipeline to produce the desired result. When a machine stoppage or breakdown occurs during the course of production, it may lead to downtime because the operator has to find out the cause of the failure before the machine can be put back into production. The sensor data collected from the machines can be used to train a machine learning model that can automatically predict the root cause when a failure occurs and significantly reduce the time spent on manual root cause finding on the shop floor. This can lead to increased availability of machines and thus more production time per shift. To achieve this goal, our first task is to identify the types of failures we want to predict. We can work with the machine owners and operators to identify the most common failure types and note that down. With this important step completed, we can identify the data sources that have relevant data about that failure type. If need be, we can update the Stream Analytics filter as well. Once the right data is identified, we train a Decision Tree Classifier model in Azure Machine Learning and deploy it as a binary value as a separate collection inside MongoDB. Atlas Scheduled Triggers are used to trigger the model (via an Azure Function) and the failure prediction results are written back results into a separate Failures collection in MongoDB. Scheduled triggers’ schedule can be aligned to production schedule so that it only fires when a changeover occurs for example. After a failure is detected, the operator and supervisor needs to be notified immediately. Using App Services, a mobile application is developed to send notifications and alerts to floor supervisor and machine operator once a failure root cause is predicted. Figure 4 shows the mobile app user interface where the user has an option to acknowledge the alert. Thanks to Atlas Device Sync , even when the mobile device is facing unreliable connectivity, the data keeps in sync between Atlas cluster and Realm database in the app. MongoDB’s Realm , is an embedded database technology already used on millions of mobile phones as part of mobile apps as well as infotainment like systems. FIgure 4: Alert app user interface Business benefits of using MongoDB Atlas as smart manufacturing data hub Scalability: MongoDB is a highly scalable document based database that can handle large amounts of structured, semi-structured and unstructured data. Native time series collections are available that help with storing large amounts of data generated by IIoT enabled equipment in a highly compressed manner. Flexibility: MongoDB stores data in a flexible, JSON-like format, which makes it easy to store and query data in a variety of ways. This flexibility makes it well-suited for handling the different data structures needed to store sensor data, ML models and prediction results, all in one database. This removes the need for maintaining separate databases for each type of data reducing IT sprawl in manufacturing organizations. Real-time Analytics: As sensor data comes in, MongoDB aggregation pipelines can help in generating features to be used for machine learning models. Atlas Charts can be set up in minutes to visualize important features and their trends in near real time. BI Analytics: Analysts can use the Atlas SQL interface to access MongoDB data from SQL based tools. This allows them to work with rich, multi-structured documents without defining a schema or flattening data. In a connected factory setting, this can be useful to generate reports for failures over a period of time and comparison between different equipment failures types. Data can be blended from MongoDB along with other sources of data to provide a 360 degree view of production operations. Faster Mobile Application Development: Atlas device sync bidirectionally connects and synchronizes Realm databases inside mobile applications with the MongoDB Atlas backend, leading to faster mobile application development and less time needed for maintenance of deployed applications. Conclusion The MongoDB Atlas developer data platform is designed and engineered to help speed up your journey towards smart manufacturing. It is not just suitable for high speed time series workloads but also for workloads that power mobile applications and BI Dashboards – leading to smarter applications, increased productivity and eventually smarter factories. Learn more All the source code used in this project, along with a detailed deployment guide, is available on our public Github page . To learn more about how MongoDB enables IIoT for our customers, please visit our IIoT use cases page . Get started today with MongoDB Atlas on Azure Marketplace listing .

February 27, 2023
Next →

Five Languages, One Goal: A Developer's Path to Certification Mastery

MongoDB Community Creator Markandey Pathak has become a certified developer in five different programming languages: C#, Java, Node.JS, PHP, and Python. Pursuing multiple certifications equips developers with a diverse skill set, making them invaluable team members. Fluency across different programming languages enables them to foster platform-agnostic solutions and promote adaptability, collaboration, and informed decision-making, which are crucial for success in the global tech landscape. To understand what led Markandey to take on so many certifications while managing a busy and successful career, we spoke with him to gain insights into the challenges and triumphs he faced. What motivated you to pursue certification in multiple programming languages, and how has achieving such a diverse set of skills impacted your career? C was the first programming language I learned, followed by C# and the .NET ecosystem a few years later. Transitioning to a new language like C# after knowing one was straightforward. I then delved into ASP.NET, JAVA, and subsequently PHP. Despite the differing syntax of these languages, I found that fundamental programming concepts remained consistent. This enlightening realization led me to explore JavaScript and, later, Python. Such a diverse skill set made me a go-to resource for many senior leaders seeking insights. This versatility allowed me to transcend categorization based on programming ecosystems in the workplace, evolving my mindset to develop platform-agnostic solutions. I believe in the adage of being a jack of all trades while still mastering one or more. I took on the challenge of discovering MongoDB drivers available for various platforms. I created sample applications to practice basic MongoDB concepts using specific drivers, and soon, everything fell into place effortlessly. What tips or advice would you share with someone who looks up to your achievement and aspires to become a certified developer in multiple languages like C#, Java, Node.JS, PHP, and Python? How can they effectively approach learning and mastering these languages? Before attempting proficiency in MongoDB across multiple languages, it's crucial to prioritize understanding fundamental concepts such as data modeling practices, CRUD operations, and indexes. Mastering MongoDB's shell, MongoSh, is essential to grasp the workings of MongoDB's read and write operations. Following this, individuals should select a programming environment they're most adept in and practice executing MongoDB operations within that ecosystem. Constructing a personal project can aid in practically observing various MongoDB concepts in action. Utilizing resources such as MongoDB Certification Learning Paths , practice tests, and MongoDB Documentation is vital for excelling in certification exams. Additionally, it's advisable to undertake the initial certification in the programming language one feels most comfortable with. Reflection is key; saving or emailing exam scores enables individuals to identify areas needing improvement for future attempts. With proficiency in C#, Java, Node.JS, PHP, and Python, how do you perceive the role of versatility in today's tech industry, especially regarding job opportunities and project flexibility? Programming languages, very much like spoken languages, are merely a medium. The most important thing is knowing what to say. The tech industry depends on problems, and developers seek solutions to them. Once they have a solution, programming languages help make those solutions a reality. It’s not hard to learn different programming languages or even to master them. Knowing the basics of different programming ecosystems can give developers an edge regarding job opportunities. It makes them flexible and enables them to make crucial and informed decisions in choosing the correct tech stack or defining good architecture for solutions. In your experience, how does fluency in multiple languages enhance collaboration and innovation within development teams, particularly in today's globalized tech landscape? Fluency or even practical awareness about programming languages or ecosystems promotes versatility in problem-solving, facilitates cross-functional collaboration, supports agile development, enables integration with legacy systems, fosters global collaboration, reduces dependency, and empowers informed decision-making, all of which are crucial for staying competitive in today's globalized tech landscape. As a MongoDB Community Creator, how do you leverage your expertise in these five languages to contribute to and engage with the broader tech community? What advice would you offer aspiring developers seeking to expand their skill set? I aim to open-source my MongoDB-focused projects across various ecosystems, accompanied by detailed articles outlining their construction. Since these projects were designed with exams in mind, they serve as skill-testing tools for developers and comprehensive guides to the various components comprising certification exams. I advocate for developers to choose a favorite language and compare others to it, as this approach facilitates a quicker and more efficient understanding of concepts. Relating new information to familiar concepts makes learning easier and more effective. The MongoDB Community Advocacy Program is a vibrant global community designed for MongoDB enthusiasts who are passionate about advocating for the platform. Our Community Creators Program welcomes members of all skill levels eager to deepen their involvement in advancing MongoDB's community and technology. We empower our members to expand their expertise, visibility, and leadership by actively engaging with and advocating for MongoDB technologies among users worldwide. Join us and amplify your impact within the MongoDB community! Elevate your career with MongoDB University 's 1,000+ learning assets. Access free courses and hands-on labs, and earn certifications to boost your skills and stand out in tech.

April 24, 2024