Real-time Applications Made Simple with MongoDB and Redpanda

Robert Walters
October 26, 2021

MongoDB has a long history of advocating for simplicity and focusing on making developers more agile and productive. MongoDB first disrupted the database market with the document model, storing data records as BSON (binary representation of JSON documents). This approach to working with data enables developers to easily store and query their data as they use it naturally within their applications. As your data changes, you simply add an attribute to your documents and move on to the next ticket. There is no need to waste time altering tables and constraints when the needs of your application change.

MongoDB is always on the lookout for more ways to make life easier for developers, such as addressing the challenges of working with streaming data. With streaming data, it may take armies of highly skilled operational personnel to build and maintain a production-ready platform (like Apache Kafka). Developers then have to integrate their applications with these streaming data platforms resulting in complex application architectures.

It’s exciting to see technologies like Redpanda seeking to improve developer productivity for working with streaming data. For those unfamiliar with Redpanda, it is a Kafka API compatible streaming platform that works with the entire Kafka ecosystem, such as Kafka-Connect and popular Kafka drivers: librdkafka, kafka-python, and the Apache Kafka Java Client. Redpanda is written in C++ and leverages the RAFT protocol, which makes Apache ZooKeeper irrelevant. Also, its thread-per-core architecture and JVM-free implementation enable performance improvements over other data streaming platforms. On a side note, MongoDB also implements a protocol similar to RAFT for its replica set cluster primary and secondary elections and management.

Both MongoDB and Redpanda share a common goal of simplicity and making complex tasks trivial for the developer. So we decided to show you how to pull together a simple streaming application using both technologies.

The example application (found in this GitHub repository) considers the scenario where stock ticker data is written to a Redpanda and consumed by MongoDB. Once you have the example running, a “stock generator” creates a list of 10 fictitious companies and starts writing ticker data to a Redpanda topic. Kafka Connect service listens for data coming into this topic and “sinks” the data to the MongoDB cluster. Once landed in MongoDB, the application issues an aggregation query to determine the moving averages of the stock securities and updates the UI. MongoDB consumes the ticker data and calculates the average stock price trends using the aggregation framework.

Once you have downloaded the repository, a docker-compose script includes a Node server, Redpanda deployment, Kafka Connect service, and a MongoDB instance. The Kafka Connect image includes the Dockerfile-MongoConnect file to install the MongoDB Connector for Apache Kafka. The Dockerfile-Nodesvr is included in the nodesvr image and it copies the web app code & installs the necessary files via NPM.

There is a run.sh script file that will launch the docker-compose script to launch the containers. To start the demo, simply run this script file via sh run.sh and upon success, you will see a list of the servers and their ports:

The following services are running:

MongoDB server on port 27017
Redpanda on 8082 (Redpanda proxy on 8083)
Kafka Connect on 8083
Node Server on 4000 is hosting the API and homepage

Status of kafka connectors:
sh status.sh

To tear down the environment and stop these services:
docker-compose down -v

Once started, navigate to localhost:4000 in a browser and click the “Start” button. After a few seconds, you will see the sample stock data from 10 fictitious companies with the moving average price.

MongoDB & Redpanda Stock Example

Get started with MongoDB and Redpanda

This example showcases the simplicity of moving data through the Redpanda streaming platform and into MongoDB for processing. Check out these resources to learn more:

← Previous

Driving Innovation with MongoDB Atlas on Google Cloud

MongoDB Atlas and Google Cloud’s global partnership continues to set the standard for cloud modernization across industries. Our joint solution helps customers modernize their database stack when they migrate their infrastructure to the cloud, ultimately boosting developer productivity and reducing total cost of ownership (TCO) while giving customers access to state-of-the-art analytics and the machine learning capabilities of Google Cloud. Our partnership is producing plenty of exciting developments—read on to learn more about our joint awards, latest integrations, and what’s next. Earlier this month on October 12, MongoDB was announced as a Google Cloud Cross-Industry Customer of the Year . On the heels of an expanded, five-year partnership that began earlier in 2021, MongoDB and Google Cloud continue to create excellent customer experiences for our joint customers across the globe. Adopted by clients in the gaming, retail, healthcare, financial services, and automotive industries, among others, MongoDB Atlas on Google Cloud is experiencing exciting year-over-year growth in a variety of innovative fields. After putting much effort into launching joint integrations to become a first-class database within Google Cloud Marketplace, MongoDB was also named Google Cloud Technology Partner of the Year Award - Marketplace —for the 2nd year in a row . Since being made available on Google Cloud Console and Marketplace, MongoDB Atlas on the Marketplace has rapidly become the preferred engagement model for users. MongoDB Atlas on the Marketplace is available as a pay-as-you-go offering, and clients can now use their Google Cloud spending commitments toward Atlas to take advantage of integrated billing and support. The intuitive database structure of MongoDB combined with the cloud computing power and security of Google Cloud enables clients to take a quantum leap forward from their legacy systems. With MongoDB Atlas on Google Cloud, companies gain the power to scale effortlessly, maintain business-critical reliability, and cut costs. So, what's the latest? Today’s users expect a seamless application experience regardless of where they’re located. MongoDB Atlas delivers responsive and reliable applications worldwide alongside continuous support for new regions. Most recently, three new Google Cloud regions for Atlas dedicated clusters were released in Delhi (asia-south2), Melbourne (australia-southeast2), and Warsaw (europe-central2)—for a total of 27 supported regions, allowing customers to deploy fully managed MongoDB databases in every Google Cloud region. These regions expand the geographical coverage not just of Atlas on Google Cloud, but Atlas as a whole. The availability of Atlas in new regions will provide lower-latency access to data, offer more options for disaster recovery globally, and make it easier for customers to satisfy local regulatory and compliance requirements. Companies across the world are benefitting from our global partnership. In Japan, e-commerce analytics company PLAID has migrated to MongoDB Atlas on Google Cloud to automate operational tasks like version upgrades and backups. And in North America, the largest tire distributor on the continent, ATD, now has its own custom event-driven architecture that’s accessible, pluggable, and scalable to support the company’s expanding business horizons. ATD can now scale up for bulk one-time loads and process more than 50 million transactions in less than four hours, which is nearly 5x their real-time transactions per day. In a development that was just announced, MongoDB Atlas now allows users to deploy a serverless database on Google Cloud via serverless instances , available in preview. Serverless computing is becoming increasingly popular among developers due its undeniable benefits—including consumption-based pricing and a service that scales dynamically as needed, which eliminates the need to provision and manage capacity needs. Serverless computing abstracts and automates away many of the lower-level infrastructure decisions that consume precious developer time so they can instead focus on building differentiated features. With serverless instances, users can get started with minimal configuration—simply choose a region and give your database a name and Atlas will provide the resources your app needs. For any company that needs faster insights from data stored all over their organization, Google Cloud introduced Datastream, and MongoDB Atlas was excited to be a launch partner . Datastream is a new serverless change data capture and replication service that allows developers to easily stream data from relational databases like Oracle and MySQL to MongoDB Atlas. By making data more accessible, Datastream and MongoDB Atlas help companies make their data more actionable, and therefore more valuable. Announced in August 2021, Google Cloud’s Private Service Connect (PSC) is now generally available in all Google Cloud regions . By abstracting the underlying networking infrastructure, PSC allows users to create private and secure connections from cloud networks to services like MongoDB. PSC will allow Google Cloud customers to use MongoDB Atlas more easily, while ensuring that the connectivity is private and secure. Connecting MongoDB Atlas with Google Cloud services will make it possible for enterprises to innovate faster. Together, MongoDB and Google Cloud are developing ways for companies to accelerate digital transformation, empower employees, and raise customer satisfaction to new levels. Learn more about what your organization can do with MongoDB and Google Cloud at our first-ever joint Developer Summit on October 28. Sign up now to hear directly from your peers about how they’re using the combined power of MongoDB and Google Cloud, and participate in a live developer Q&A.

October 25, 2021
Next →

Retrieval Augmented Generation for Claim Processing: Combining MongoDB Atlas Vector Search and Large Language Models

Following up on our previous blog, AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases , we’ll pick up the conversation right where we left it. We discussed extensively how Atlas Vector Search can benefit the claim process in insurance and briefly covered Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More One of the biggest challenges for claim adjusters is pulling and aggregating information from disparate systems and diverse data formats. PDFs of policy guidelines might be stored in a content-sharing platform, customer information locked in a legacy CRM, and claim-related pictures and voice reports in yet another tool. All of this data is not just fragmented across siloed sources and hard to find but also in formats that have been historically nearly impossible to index with traditional methods. Over the years, insurance companies have accumulated terabytes of unstructured data in their data stores but have failed to capitalize on the possibility of accessing and leveraging it to uncover business insights, deliver better customer experiences, and streamline operations. Some of our customers even admit they’re not fully aware of all the data in their archives. There’s a tremendous opportunity to leverage this unstructured data to benefit the insurer and its customers. Our image search post covered part of the solution to these challenges, opening the door to working more easily with unstructured data. RAG takes it a step further, integrating Atlas Vector Search and LLMs, thus allowing insurers to go beyond the limitations of baseline foundational models, making them context-aware by feeding them proprietary data. Figure 1 shows how the interaction works in practice: through a chat prompt, we can ask questions to the system, and the LLM returns answers to the user and shows what references it used to retrieve the information contained in the response. Great! We’ve got a nice UI, but how can we build an RAG application? Let’s open the hood and see what’s in it! Figure 1: UI of the claim adjuster RAG-powered chatbot Architecture and flow Before we start building our application, we need to ensure that our data is easily accessible and in one secure place. Operational Data Layers (ODLs) are the recommended pattern for wrangling data to create single views. This post walks the reader through the process of modernizing insurance data models with Relational Migrator, helping insurers migrate off legacy systems to create ODLs. Once the data is organized in our MongoDB collections and ready to be consumed, we can start architecting our solution. Building upon the schema developed in the image search post , we augment our documents by adding a few fields that will allow adjusters to ask more complex questions about the data and solve harder business challenges, such as resolving a claim in a fraction of the time with increased accuracy. Figure 2 shows the resulting document with two highlighted fields, “claimDescription” and its vector representation, “claimDescriptionEmbedding” . We can now create a Vector Search index on this array, a key step to facilitate retrieving the information fed to the LLM. Figure 2: document schema of the claim collection, the highlighted fields are used to retrieve the data that will be passed as context to the LLM Having prepared our data, building the RAG interaction is straightforward; refer to this GitHub repository for the implementation details. Here, we’ll just discuss the high-level architecture and the data flow, as shown in Figure 3 below: The user enters the prompt, a question in natural language. The prompt is vectorized and sent to Atlas Vector Search; similar documents are retrieved. The prompt and the retrieved documents are passed to the LLM as context. The LLM produces an answer to the user (in natural language), considering the context and the prompt. Figure 3: RAG architecture and interaction flow It is important to note how the semantics of the question are preserved throughout the different steps. The reference to “adverse weather” related accidents in the prompt is captured and passed to Atlas Vector Search, which surfaces claim documents whose claim description relates to similar concepts (e.g., rain) without needing to mention them explicitly. Finally, the LLM consumes the relevant documents to produce a context-aware question referencing rain, hail, and fire, as we’d expect based on the user's initial question. So what? To sum it all up, what’s the benefit of combining Atlas Vector Search and LLMs in a Claim Processing RAG application? Speed and accuracy: Having the data centrally organized and ready to be consumed by LLMs, adjusters can find all the necessary information in a fraction of the time. Flexibility: LLMs can answer a wide spectrum of questions, meaning applications require less upfront system design. There is no need to build custom APIs for each piece of information you’re trying to retrieve; just ask the LLM to do it for you. Natural interaction: Applications can be interrogated in plain English without programming skills or system training. Data accessibility: Insurers can finally leverage and explore unstructured data that was previously hard to access. Not just claim processing The same data model and architecture can serve additional personas and use cases within the organization: Customer Service: Operators can quickly pull customer data and answer complex questions without navigating different systems. For example, “Summarize this customer's past interactions,” “What coverages does this customer have?” or “What coverages can I recommend to this customer?” Customer self-service: Simplify your members’ experience by enabling them to ask questions themselves. For example, “My apartment is flooded. Am I covered?” or “How long do windshield repairs take on average?” Underwriting: Underwriters can quickly aggregate and summarize information, providing quotes in a fraction of the time. For example, “Summarize this customer claim history.” “I Am renewing a customer policy. What are the customer's current coverages? Pull everything related to the policy entity/customer. I need to get baseline info. Find relevant underwriting guidelines.” If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: RAG for claim processing GitHub repository From Relational Databases to AI: An Insurance Data Modernization Journey Modernize your insurance data models with MongoDB and Relational Migrator

April 18, 2024