Closing the Developer Experience Gap: MongoDB World Announcements

Now is a great time to be a software developer or architect. Never have there been so many solutions, vendors, and architectural patterns to choose from as you build new applications and features. But the sheer number of choices creates another puzzle for developers to solve before they can begin to build.

Many of MongoDB’s efforts over the past year have been to help address the needs of the developer communities we serve, and one of the greatest needs we’ve seen in developer communities is improving the experience of being a developer.

At MongoDB World 2022, we announced several tools to help improve that experience and to boost developer velocity:

  • Atlas Data API — A serverless API that lets you easily access your Atlas data from any environment that supports HTTPS requests, including services like AWS Lambda and Google App Services. The Atlas Data API is fully functional upon generation, language-agnostic, and secure from the start.

  • Serverless instances — With MongoDB serverless instances, developers don’t have to worry about scaling up to meet increasing workloads or paying for resources they’re not using if their workload is idle. The serverless model dynamically uses only what it needs — and only charges for what it uses.

  • Atlas CLI — The MongoDB Atlas CLI is a completely new way to access Atlas in a non-GUI-centered environment. CLIs are often the interaction method of choice by developers, especially advanced developers who prefer control and speed over a more visual interface. Our new CLI gives these developers an easier registration experience with nearly instant free tier deployments in Atlas.

  • Time series — We have expanded our data platform so developers can work more easily with time series data in support of IoT use cases, financial analytics, logistics, and more. MongoDB time series makes it faster and lower cost to build and run time series applications by natively supporting the entire time series data lifecycle.

  • Facets in Atlas Search — Categorize data with facets for fast, filtered search results. With facets in Atlas Search, you can index your data to map fields to categories, then quickly update query results based on the ones relevant to your users.

  • Verified Solutions — The MongoDB Verified Solutions program gives developers the confidence to use third-party tools, such as Mongoose, by guaranteeing comprehensive testing of the tools as well as a base level of support from MongoDB Technical Services.

  • Change streams — Change streams enable developers to build real-time, event-driven applications that react to data changes as they happen. This allows them to build more complex features and better end-user experiences.

The paradox of choice for developers

Developers today have no shortage of tools to work with, but the abundance of options is itself a problem. And when there’s little or no central decision-making, developers are forced to figure out how to stitch together a patchwork of technology solutions to create the seamless user experiences that consumers have come to expect.

Developers had fewer choices when applications were built on a three-tier framework composed of a relational database, a J2EE stack, and an app or web server. Since then, however, application development has fragmented into different architectures, SDKs, and cloud services, leaving developers many more patterns to figure out. On top of that, the rise of DevOps has increased the pressure on developers to build and maintain the tools they’re working with, and serious development shops often take pride in building their own toolchains, backends, and databases.

Put it all together — the abundance of choices, the patchwork nature of solutions, the pressure to build and maintain toolchains, and the glue code keeping it all together — and it adds up to more cognitive load, elevated stress levels, and a lengthening of time to value. As Stephen O’Grady from analyst firm RedMonk explains, “Developers are forced to borrow time from writing code and redirect it toward managing the issues associated with highly complex, multifactor developer toolchains held together in places by duct tape and baling wire. This, then, is the developer experience gap.”

Having a lot of options is a good thing — until it’s not. One way we’re working to unwind the paradox of choice is by providing tools that exist in the same form whether in the cloud or on the client — that is, solutions that integrate with the way developers already work. This could mean plugging into a CLI first, abstracting provisioning, simplifying and securing the data layer so developers don’t have to worry about it, and unlocking the creativity of developers with a data model that maps to how data is actually going to be used. We’re also enabling developers to access the tools they need from within MongoDB without having to integrate myriad bolt-on tools (i.e., the paradox of choice).

Building at velocity

The key to unlocking developer productivity, as we see it, is giving developers the building blocks they need to create a whole workload from scratch, or to bring a new workload into their ecosystem — be it time-series, search, or analytics — and have them run on a single platform instead of having to stitch together disparate systems.

Our goal is to bring a modern data layer to modern applications. We want to bring that experience to more and more of what you work on. We know that modern applications have complicated data requirements, but that shouldn’t mean complicated data infrastructure. We want to serve most of your workloads with a single unified platform.

Learn more about MongoDB World 2022 announcements at mongodb.com/new and in these stories:

← Previous

Announcing Atlas Data Federation and Atlas Data Lake

Two years ago, we released the first iteration of Atlas Data Lake . Since then, we’ve helped customers combine data from various storage layers to feed downstream systems. But after years spent studying our customers’ experiences, we realized we hadn’t gone far enough. To truly unleash the genius in all our developers, we needed to add an economical cloud object storage solution with a rich MQL query experience to the world of Atlas. Today, we’re thrilled to announce that our new Atlas Data Federation and Atlas Data Lake offerings do just that. We now offer two complementary services, Atlas Data Federation (our existing query service formerly known as Atlas Data Lake) and our new and improved Atlas Data Lake (a fully managed analytic-oriented storage service). Together, these services (both in preview) provide flexible and versatile options for querying and transforming data across storage services, as well as a MongoDB-native analytic storage solution. With these tools, you can query across multiple clusters, move data into self managed cloud object storage for consumption by downstream services, query a workload-isolated inexpensive copy of cluster data, compare your cluster data across different points in time, and much, much more. In hearing from our customers about their experiences with Atlas Data Lake, we learned where they have struggled, as well as the features they’ve been looking for us to provide. With this in mind, we decided to shift the name of our current query federation service to Atlas Data Federation to better align with how customers see the service and are getting value. We’ve seen many customers benefit from the flexibility of a federated query engine service, including querying data across multiple clusters, databases, and collections, as well as exporting data to third-party systems. We also saw where our customers were struggling with data lakes. We heard them ask for a fully managed storage solution so they could achieve all of their analytic goals within Atlas. Specifically, customers wanted scalable storage that would provide high query performance at a low cost. Our new Data Lake provides a high-performance analytic object storage solution, allowing customers to query historical data with no additional formatting or maintenance work needed on their end. How it works Atlas Data Federation encompasses our existing Data Lake functionality with several new affordances. It continues to deliver the same power that it always has, with increased performance and efficiency. The new Atlas Data Lake will now allow you to create Data Lake pipelines (based on your Atlas Cluster backup schedules) and fields on which you can optimize queries. The service takes the following steps: On the selected schedule, a copy of your collection will be extracted from your Atlas backup with no impact to your cluster. During extraction, we build partition indexes based on the contents of your documents and the fields you’ve selected for optimization. These indexes allow your queries to be optimized by capturing the minimums and maximums (and other stats) of the records in each partition, letting you quickly find the relevant data for your queries. Finally, the underlying data lands in an analytic-oriented format inside of cloud object storage. This minimizes data scanned when you execute a query. Once a pipeline has run and a Data Lake dataset has been created, you can select it as a data source in our new Data Federation query experience. You can either set it as the source for a specific virtual collection in a Federated Database Instance or you can have your Federated Database Instance generate a collection name for each dataset that your pipeline has created. Amazingly, no part of this process will consume compute resources from your cluster — neither the export nor the querying of datasets. These datasets provide workload isolation and consistency for long-running analytic queries, a target for ETL jobs using the powerful $out to S3. This makes it easy to compare the state of your data over time. Advanced though this is, it’s only the beginning of the story. We’re committing to evolving the service, improving performance, adding more sources of data, and building new features. All of this will be based on the feedback you, the user, gives us. We can’t wait to see how you’ll use this powerful new tool and can’t wait to hear what you’d like to see next. Try Atlas Data Lake Today

June 7, 2022
Next →

Retrieval Augmented Generation for Claim Processing: Combining MongoDB Atlas Vector Search and Large Language Models

Following up on our previous blog, AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases , we’ll pick up the conversation right where we left it. We discussed extensively how Atlas Vector Search can benefit the claim process in insurance and briefly covered Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More One of the biggest challenges for claim adjusters is pulling and aggregating information from disparate systems and diverse data formats. PDFs of policy guidelines might be stored in a content-sharing platform, customer information locked in a legacy CRM, and claim-related pictures and voice reports in yet another tool. All of this data is not just fragmented across siloed sources and hard to find but also in formats that have been historically nearly impossible to index with traditional methods. Over the years, insurance companies have accumulated terabytes of unstructured data in their data stores but have failed to capitalize on the possibility of accessing and leveraging it to uncover business insights, deliver better customer experiences, and streamline operations. Some of our customers even admit they’re not fully aware of all the data in their archives. There’s a tremendous opportunity to leverage this unstructured data to benefit the insurer and its customers. Our image search post covered part of the solution to these challenges, opening the door to working more easily with unstructured data. RAG takes it a step further, integrating Atlas Vector Search and LLMs, thus allowing insurers to go beyond the limitations of baseline foundational models, making them context-aware by feeding them proprietary data. Figure 1 shows how the interaction works in practice: through a chat prompt, we can ask questions to the system, and the LLM returns answers to the user and shows what references it used to retrieve the information contained in the response. Great! We’ve got a nice UI, but how can we build an RAG application? Let’s open the hood and see what’s in it! Figure 1: UI of the claim adjuster RAG-powered chatbot Architecture and flow Before we start building our application, we need to ensure that our data is easily accessible and in one secure place. Operational Data Layers (ODLs) are the recommended pattern for wrangling data to create single views. This post walks the reader through the process of modernizing insurance data models with Relational Migrator, helping insurers migrate off legacy systems to create ODLs. Once the data is organized in our MongoDB collections and ready to be consumed, we can start architecting our solution. Building upon the schema developed in the image search post , we augment our documents by adding a few fields that will allow adjusters to ask more complex questions about the data and solve harder business challenges, such as resolving a claim in a fraction of the time with increased accuracy. Figure 2 shows the resulting document with two highlighted fields, “claimDescription” and its vector representation, “claimDescriptionEmbedding” . We can now create a Vector Search index on this array, a key step to facilitate retrieving the information fed to the LLM. Figure 2: document schema of the claim collection, the highlighted fields are used to retrieve the data that will be passed as context to the LLM Having prepared our data, building the RAG interaction is straightforward; refer to this GitHub repository for the implementation details. Here, we’ll just discuss the high-level architecture and the data flow, as shown in Figure 3 below: The user enters the prompt, a question in natural language. The prompt is vectorized and sent to Atlas Vector Search; similar documents are retrieved. The prompt and the retrieved documents are passed to the LLM as context. The LLM produces an answer to the user (in natural language), considering the context and the prompt. Figure 3: RAG architecture and interaction flow It is important to note how the semantics of the question are preserved throughout the different steps. The reference to “adverse weather” related accidents in the prompt is captured and passed to Atlas Vector Search, which surfaces claim documents whose claim description relates to similar concepts (e.g., rain) without needing to mention them explicitly. Finally, the LLM consumes the relevant documents to produce a context-aware question referencing rain, hail, and fire, as we’d expect based on the user's initial question. So what? To sum it all up, what’s the benefit of combining Atlas Vector Search and LLMs in a Claim Processing RAG application? Speed and accuracy: Having the data centrally organized and ready to be consumed by LLMs, adjusters can find all the necessary information in a fraction of the time. Flexibility: LLMs can answer a wide spectrum of questions, meaning applications require less upfront system design. There is no need to build custom APIs for each piece of information you’re trying to retrieve; just ask the LLM to do it for you. Natural interaction: Applications can be interrogated in plain English without programming skills or system training. Data accessibility: Insurers can finally leverage and explore unstructured data that was previously hard to access. Not just claim processing The same data model and architecture can serve additional personas and use cases within the organization: Customer Service: Operators can quickly pull customer data and answer complex questions without navigating different systems. For example, “Summarize this customer's past interactions,” “What coverages does this customer have?” or “What coverages can I recommend to this customer?” Customer self-service: Simplify your members’ experience by enabling them to ask questions themselves. For example, “My apartment is flooded. Am I covered?” or “How long do windshield repairs take on average?” Underwriting: Underwriters can quickly aggregate and summarize information, providing quotes in a fraction of the time. For example, “Summarize this customer claim history.” “I Am renewing a customer policy. What are the customer's current coverages? Pull everything related to the policy entity/customer. I need to get baseline info. Find relevant underwriting guidelines.” If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: RAG for claim processing GitHub repository From Relational Databases to AI: An Insurance Data Modernization Journey Modernize your insurance data models with MongoDB and Relational Migrator

April 18, 2024