This Week in MongoDB July 8-14

Amy Berman
July 9, 2013 | Updated: May 22, 2015

Here's what's going on in the MongoDB community this week:

Learn More

Upcoming Webinars:

Online Education:

Have something you'd like to share? Let us know

← Previous

Leafblower: Winner of the MongoDB March Madness Hackathon

In March, 10gen hosted a Global Hackathon called MongoDB March Madness, and challenged the community to build tools to support the growing MongoDB community. Leafblower, a project built at the Sydney March Madness Hackathon, was the global winner of the Hackathon Who are rdrkt? Andy Sharman and Josh Stevenson work together on independent projects under their brand rdrkt (pronounced “redirect") and capitalize on their largely overlapping skillset, but unique passions to build cutting-edge web applications. Josh is passionate about flexible, scalable back-end systems and Andy is passionate about accessible, responsive user interfaces. Josh, originally from the United States, now lives and works full-time in Sydney, Australia for Excite Digital Media. He has experience with the LAMP stack, CakePHP, ZendFramework, and MongoDB. He’s also worked on projects which have required his significant expertise with jQuery, HTML5 and CSS3. He’s very active in the Sydney area tech community, regularly attending PHP, MongoDB, WebDirections and WebBlast meetups. Andy, from the UK, also now lives in Sydney and works for Visualjazz Isobar. He has frontend experience with HTML5, Vanilla JavaScript, jQuery, Mootools, and WebGL with Three.js. In addition to that, he has experience with backend languages such as PHP, C#.NET, Node.JS and CMS’ such as Sitecore, Magento, Joomla, and Wordpress. What is Leafblower? Leafblower is a real-time dashboard platform we conceived and programmed for the MongoDB March Madness Hackathon. Its primary focus is on bleeding-edge, up-to-the-second application analytics and monitoring to help campaign managers and system administrators to react quickly to ever-changing trending and traffic spiking. We live in an age where one link on Reddit or a viral post on Facebook can mean the difference between “getting your break" and catastrophic failure of the application’s server infrastructure. Leafblower helps users understand what is happening “right now" in their applications and most importantly their server availability and usage by cutting out as much aggregation as possible and focusing on capturing and reporting data samples as-they-happen. Where did the idea for Leafblower come from? At the start of the Sydney-based MongoDB hackathon, Andy wanted to build some sort of analytics or reporting toolset to help people learn the way Mongo databases function and grow, but it wasn’t until the starting bell rang that Josh had the idea of doing real-time monitoring rather than historical analysis. That’s how Leafblower was born. We went on to win the Sydney hackathon, and decided to really buckle down and clean up what we worked on for the global competition, rather than trying to build out too many extra features. Anyone can look at and download everything needed to install Leafblower on Github How does Leafblower work? After the initial idea for Leafblower was formed, the rest of the project flowed together organically. The platform uses MongoDB as the storage engine to run all the data “blocks" which display statistics intuitively for end-users. Blocks render themselves using information passed to them via a Node.js communication layer which itself connects to a backend API server. This allows many clients to be connected and viewing dashboards without requiring any additional load on the API server and thus the application itself. Leafblower is currently running on a Linux/Apache/MongoDB/PHP/Node.JS system, however, Linux could be swapped out for any OS, and Apache could be swapped out for any web server assuming that the correct version of the required services are installed and configured to play nicely. There is a full list of required software available on our Github page. Leafblower uses well-supported cross-OS services it so you can run an instance with the operating system you’re used to. Rather than trying to create a platform that is generic enough to be useful to all the different applications out there, our goal was to create enough useful Blocks and then put the power to customize or extend block functionality with the people actually using Leafblower to monitor their apps. NoSQL solutions like MongoDB are the perfect fit for projects looking to allow developers to extend our platform with minimal-to-zero prior knowledge of how data is stored and passed around, internally. For example, while some of the elements of the configuration documents in MongoDB are required to function properly, whole portions are completely arbitrary depending on the data needs of a given block. With MongoDB this is no sweat. For us, the advantage is not so much about “No Schema" as is it is about a “Flexible Schema". When will it be ready for a production environment? After completing the early stages of the project and successfully winning the MongoDB Global March Madness Hackathon Josh and Andy have a number of features and services, both short and long-term, to implement. Some of the items on their shortlist are: Users and Groups to which profiles can be assigned Easier setup and installation/configuration for users who want to deploy Leafblower on their own hardware A full suite of Blocks of both a basic and advanced nature, such as API Blocks for Twitter/Facebook/Google which do live social/analytics monitoring Automatic historical data comparison to make it easier to compare current live data with past trends Admin panel enhancements to make users, groups, profiles, and Blocks easier to configure User logins and access groups are our first priority and we expect to include them in the next release of Leafblower on Github. We’re also hoping the community will engage with Leafblower and help us build out our library of standard Blocks. Why did rdrkt use MongoDB? Leafblower is a real time monitoring platform, not a complete self-contained application. In order for it to be flexible enough to monitor all sorts of potential data feeds and apis, we knew a flexible schema was the way to go. For example, let’s look at a document from the blocks collection { "_id" : "geoCheckIns", "type" : "geospacial", "title" : "Visualize checkins as they occur.", "description" : "Block for visualizing the checkins occurring in your app bound by a circle centered at a given lat/lon and radius", "ttl" : 5000, "options" : { "lat" : -33.873651, "lon" : -151.2068896, "radius" : 50, "host" : "localhost", "db" : "demo", "collection" : "checkins" } } In the case of a block, each one will always have the fields _id , type , title , description , ttl and options , however, options is a subdocument and may contain an arbitrary number of fields. In this example, we need to know which database host, database name, a collection name with checkin data to monitor, a center point to draw our great circle and the radius of the circle. From those data points, the block will be able to render a map centered over our point and drop pins onto the map as users check in to your application. This particular block would be useful for someone monitoring their social media campaign as it goes live in a specific city or region. Let’s look at another block type for contrast: { "_id" : "memory", "type" : "system", "title" : "Memory", "description" : "View statistics about the memory usage on your server.", "ttl" : 1000, "options" : { } } In this example, we left options completely empty because we are retrieving the monitoring data from the server the user is currently connected to. Our application still expects an object to exist when we access options, so we maintain the structure of our document despite not needing to read any further data for this type of block. We might expect sysadmins to use a modified version of this block for monitoring servers they administer. This is a good illustration of what we mean when we say “flexible schema". It’s an important distinction to make from “no schema", since there is a clear idea of how the data is structured, we’ve just allowed for flexibility in certain places where advantageous. NoSQL solutions like MongoDB are very attractive to many developers, because they can interact with the database as if they are simply storing and retrieving objects from a datastore. This is a dangerous path to follow, however, because a “set it and forget it" attitude will quickly lead to scaling and performance issues as an application grows. Know the structure of your data and only make changes to that structure incrementally. MongoDB is easy to scale and has a vibrant user community. Leafblower gives back to the community by creating useful tools to showcase MongoDB’s advantages while teaching them how it works. It’s a win for everyone: for ourselves, for MongoDB, and for the IT Community at large. Want more information? Leafblower: Github repo Twitter Angelist

July 1, 2013
Next →

Retrieval Augmented Generation for Claim Processing: Combining MongoDB Atlas Vector Search and Large Language Models

Following up on our previous blog, AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases , we’ll pick up the conversation right where we left it. We discussed extensively how Atlas Vector Search can benefit the claim process in insurance and briefly covered Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More One of the biggest challenges for claim adjusters is pulling and aggregating information from disparate systems and diverse data formats. PDFs of policy guidelines might be stored in a content-sharing platform, customer information locked in a legacy CRM, and claim-related pictures and voice reports in yet another tool. All of this data is not just fragmented across siloed sources and hard to find but also in formats that have been historically nearly impossible to index with traditional methods. Over the years, insurance companies have accumulated terabytes of unstructured data in their data stores but have failed to capitalize on the possibility of accessing and leveraging it to uncover business insights, deliver better customer experiences, and streamline operations. Some of our customers even admit they’re not fully aware of all the data in their archives. There’s a tremendous opportunity to leverage this unstructured data to benefit the insurer and its customers. Our image search post covered part of the solution to these challenges, opening the door to working more easily with unstructured data. RAG takes it a step further, integrating Atlas Vector Search and LLMs, thus allowing insurers to go beyond the limitations of baseline foundational models, making them context-aware by feeding them proprietary data. Figure 1 shows how the interaction works in practice: through a chat prompt, we can ask questions to the system, and the LLM returns answers to the user and shows what references it used to retrieve the information contained in the response. Great! We’ve got a nice UI, but how can we build an RAG application? Let’s open the hood and see what’s in it! Figure 1: UI of the claim adjuster RAG-powered chatbot Architecture and flow Before we start building our application, we need to ensure that our data is easily accessible and in one secure place. Operational Data Layers (ODLs) are the recommended pattern for wrangling data to create single views. This post walks the reader through the process of modernizing insurance data models with Relational Migrator, helping insurers migrate off legacy systems to create ODLs. Once the data is organized in our MongoDB collections and ready to be consumed, we can start architecting our solution. Building upon the schema developed in the image search post , we augment our documents by adding a few fields that will allow adjusters to ask more complex questions about the data and solve harder business challenges, such as resolving a claim in a fraction of the time with increased accuracy. Figure 2 shows the resulting document with two highlighted fields, “claimDescription” and its vector representation, “claimDescriptionEmbedding” . We can now create a Vector Search index on this array, a key step to facilitate retrieving the information fed to the LLM. Figure 2: document schema of the claim collection, the highlighted fields are used to retrieve the data that will be passed as context to the LLM Having prepared our data, building the RAG interaction is straightforward; refer to this GitHub repository for the implementation details. Here, we’ll just discuss the high-level architecture and the data flow, as shown in Figure 3 below: The user enters the prompt, a question in natural language. The prompt is vectorized and sent to Atlas Vector Search; similar documents are retrieved. The prompt and the retrieved documents are passed to the LLM as context. The LLM produces an answer to the user (in natural language), considering the context and the prompt. Figure 3: RAG architecture and interaction flow It is important to note how the semantics of the question are preserved throughout the different steps. The reference to “adverse weather” related accidents in the prompt is captured and passed to Atlas Vector Search, which surfaces claim documents whose claim description relates to similar concepts (e.g., rain) without needing to mention them explicitly. Finally, the LLM consumes the relevant documents to produce a context-aware question referencing rain, hail, and fire, as we’d expect based on the user's initial question. So what? To sum it all up, what’s the benefit of combining Atlas Vector Search and LLMs in a Claim Processing RAG application? Speed and accuracy: Having the data centrally organized and ready to be consumed by LLMs, adjusters can find all the necessary information in a fraction of the time. Flexibility: LLMs can answer a wide spectrum of questions, meaning applications require less upfront system design. There is no need to build custom APIs for each piece of information you’re trying to retrieve; just ask the LLM to do it for you. Natural interaction: Applications can be interrogated in plain English without programming skills or system training. Data accessibility: Insurers can finally leverage and explore unstructured data that was previously hard to access. Not just claim processing The same data model and architecture can serve additional personas and use cases within the organization: Customer Service: Operators can quickly pull customer data and answer complex questions without navigating different systems. For example, “Summarize this customer's past interactions,” “What coverages does this customer have?” or “What coverages can I recommend to this customer?” Customer self-service: Simplify your members’ experience by enabling them to ask questions themselves. For example, “My apartment is flooded. Am I covered?” or “How long do windshield repairs take on average?” Underwriting: Underwriters can quickly aggregate and summarize information, providing quotes in a fraction of the time. For example, “Summarize this customer claim history.” “I Am renewing a customer policy. What are the customer's current coverages? Pull everything related to the policy entity/customer. I need to get baseline info. Find relevant underwriting guidelines.” If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: RAG for claim processing GitHub repository From Relational Databases to AI: An Insurance Data Modernization Journey Modernize your insurance data models with MongoDB and Relational Migrator

April 18, 2024