Improved Experience for Saved Aggregations and Queries in MongoDB Compass

Tens of thousands of MongoDB users take advantage of MongoDB Compass to query their data and build sophisticated aggregation pipelines. As an easy-to-use GUI, Compass lets you seamlessly connect to and interact with your data, including using our powerful Query API. You just connect to your cluster, navigate to your chosen database and collection, and start building your queries.

Many Compass users want to come back again and again to their best queries — or make a query repeatedly available to all database users — but the experience of working with saved queries and aggregations has created some challenges for users in the past. Previously, saved aggregations and queries were bound to a specific database and collection, making it harder to integrate those saved queries and aggregations into the standard software-development lifecycle. If, for example, you built an aggregation pipeline against a staging database and saved it, you’d still have to build that same pipeline again if you wanted to use it for your production database. Users have also reported difficulty finding their favorites after saving them.

That’s why we’ve released a new-and-improved experience for saved aggregations and queries in MongoDB Compass. It includes a new “My Queries” screen you can navigate to from the left sidebar or from a tab at the top, next to the “Database” and “Performance” tabs.

Once on the “My Queries” screen, you can search across all your saved queries/aggregations and sort or filter by database or collection. And you can apply your saved queries/aggregations across namespaces.

Queries Dashboard

To learn more about working with queries and aggregations in Compass, visit our documentation on the Aggregation Pipeline Builder or queries.

We’re confident this new experience will make it easier than ever to build, save, and reuse your favorite aggregations and queries, and ultimately remove friction with integrating them into the application development process.

Head over to your Compass instance and check it out. (If you’re not yet a Compass user, you can download it for free.) Happy querying!

← Previous

Consider Your Cloud Backup Strategy on World Backup Day, March 31

World Backup Day , marked every March 31, reminds all of us of the risk and vulnerability of the data stored on our devices and systems. Data loss happens every day for a variety of reasons. While human error is the most common cause, ransomware is fast becoming the most expensive one. A big reason is because ransomware criminals have moved on from targeting individuals to attacking businesses, where data is far more sensitive and valuable. Businesses have more to lose if their data is compromised or exposed, and they have deeper pockets to pay larger ransoms. Having a backup copy of data can help a business recover clean copies of data after it’s been corrupted by malware, accidentally deleted, or destroyed by a fire or flood. Backup could also save the day during cloud outages. For businesses, any type of data loss is extremely costly. So having a backup plan as part of an overall disaster recovery strategy is critical for the survival of every business. Backup benefits A backup and disaster recovery strategy is necessary to protect your mission critical data against these types of risks. With such a strategy in place, you'll gain peace of mind knowing that if your data ever becomes accidentally deleted or infected by malware, you'll be able to recover it and avoid the cost and consequences of data loss. You'll also satisfy important regulatory and compliance requirements by demonstrating that you've taken a proactive approach towards data safety and business continuity. Taking regular backups offers other advantages as well. The backups can be used to create new environments for development, staging, or QA without impacting production. This practice enables development teams to quickly and easily test new features, accelerating application development and ensuring smooth product launches. Backup and the cloud factor Back when business systems were mostly on-premises, there was a simple framework for disaster recovery planning: the 3-2-1 backup rule. It essentially recommends keeping three copies of your data, in at least two different form factors, with one of them kept at an offsite or remote location. In practice, this might mean creating a bare metal image of an entire server and storing it on a secondary server in the same server room. In addition, you would also make a copy of all of your server data on magnetic tape and transport it to an offsite, secure location. So that’s three copies of data (the original, the bare metal image, and the tape backup) in two different form factors (disc and tape), with one stored offsite. The 3-2-1 backup rule has worked well for a lot of businesses for decades. But systems have changed. First, barely anyone uses tape anymore. The time it takes to transport a tape backup to where it can be used for disaster recovery is more than most businesses can tolerate. The criticality of systems has shrunk recovery time objectives (RTO) to just hours or minutes. Tape backups are simply not practical for most modern IT environments. But the cloud is. Hyperscale cloud providers can provide service levels that are just as reliable, if not moreso, than traditional on-premises environments. But customers shouldn’t make the mistake of thinking that by deploying in the cloud, they’re off the hook when it comes to planning and implementing a disaster recovery strategy. Businesses can and do lose data in the cloud. And hyperscale cloud providers do experience outages. Businesses must have a backup strategy, but it needs to be updated to factor in the cloud workloads that have become ubiquitous. Cross-cloud data protection An outage at any of the major cloud providers could take your databases offline. Keeping extra backup copies with different cloud providers is a good way to ensure you can still access your data in the event of an outage with a single cloud provider. Recent outages at hyperscale cloud providers have underscored the need for cross-cloud backup. “Most cloud outages are related to software bugs rather than physical catastrophes,” says Chris Shum, Atlas product lead at MongoDB. “We've always protected ourselves against physical catastrophes by distributing across data centers or regions, but no one really protects themselves against the software bug.” Shum says by backing up workloads on different clouds, you could tolerate a cloud provider going down and your database and your application would still be up. Getting around the cloud backup skills gap Data gravity keeps many businesses locked into a single cloud provider. Becoming fluent with a particular cloud provider is a skill to acquire just like anything else. And once you become comfortable with one provider, its hardware configurations, operational tools, and its offerings, it can be hard to venture out to a different provider with different hardware and pricing. But developing flexibility in the cloud is critical if you hope to leverage the best features, functionality, and cost efficiencies from each cloud provider. MongoDB has made it possible to do just that. “With Atlas, we’ve made it our mission to abstract as much of the management away as possible,” Shum says. “It’s all available as a fully managed service. So things like hardware asymmetry between different cloud providers, offerings being different, prices being different, how you’d set up networking infrastructure, all of those things that to you as a consumer might be different, we’ve abstracted it away for you. And because of the abstraction, you are then free to move nodes to whichever cloud provider or region you want.” Data protection with Atlas MongoDB Atlas provides point in time recovery of replica sets and cluster-wide snapshots of sharded clusters. It’s simple to restore to precisely the moment you need, quickly and safely. Backups can be restored automatically to the existing MongoDB Atlas cluster or downloaded to be manually archived or restored on a different infrastructure. MongoDB Atlas provides: Security features to protect access to your data Built in replication for always-on availability, tolerating complete data center failure Backups and point in time recovery to protect against data corruption Fine-grained monitoring to let you know when to scale — additional instances can be provisioned with the push of a button Automated patching and one-click upgrades for new major versions of the database, enabling you to take advantage of the latest and greatest MongoDB features A choice of cloud providers, regions, and billing options Backup wrap-up If you’re still depending on a single cloud provider to keep your critical workloads online and available, consider World Backup Day as your reminder to identify and close the gaps in your cloud disaster recovery strategy. For information on how to back up and restore cluster data in MongoDB read this article in our documentation.

March 31, 2022
Next →

Retrieval Augmented Generation for Claim Processing: Combining MongoDB Atlas Vector Search and Large Language Models

Following up on our previous blog, AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases , we’ll pick up the conversation right where we left it. We discussed extensively how Atlas Vector Search can benefit the claim process in insurance and briefly covered Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More One of the biggest challenges for claim adjusters is pulling and aggregating information from disparate systems and diverse data formats. PDFs of policy guidelines might be stored in a content-sharing platform, customer information locked in a legacy CRM, and claim-related pictures and voice reports in yet another tool. All of this data is not just fragmented across siloed sources and hard to find but also in formats that have been historically nearly impossible to index with traditional methods. Over the years, insurance companies have accumulated terabytes of unstructured data in their data stores but have failed to capitalize on the possibility of accessing and leveraging it to uncover business insights, deliver better customer experiences, and streamline operations. Some of our customers even admit they’re not fully aware of all the data in their archives. There’s a tremendous opportunity to leverage this unstructured data to benefit the insurer and its customers. Our image search post covered part of the solution to these challenges, opening the door to working more easily with unstructured data. RAG takes it a step further, integrating Atlas Vector Search and LLMs, thus allowing insurers to go beyond the limitations of baseline foundational models, making them context-aware by feeding them proprietary data. Figure 1 shows how the interaction works in practice: through a chat prompt, we can ask questions to the system, and the LLM returns answers to the user and shows what references it used to retrieve the information contained in the response. Great! We’ve got a nice UI, but how can we build an RAG application? Let’s open the hood and see what’s in it! Figure 1: UI of the claim adjuster RAG-powered chatbot Architecture and flow Before we start building our application, we need to ensure that our data is easily accessible and in one secure place. Operational Data Layers (ODLs) are the recommended pattern for wrangling data to create single views. This post walks the reader through the process of modernizing insurance data models with Relational Migrator, helping insurers migrate off legacy systems to create ODLs. Once the data is organized in our MongoDB collections and ready to be consumed, we can start architecting our solution. Building upon the schema developed in the image search post , we augment our documents by adding a few fields that will allow adjusters to ask more complex questions about the data and solve harder business challenges, such as resolving a claim in a fraction of the time with increased accuracy. Figure 2 shows the resulting document with two highlighted fields, “claimDescription” and its vector representation, “claimDescriptionEmbedding” . We can now create a Vector Search index on this array, a key step to facilitate retrieving the information fed to the LLM. Figure 2: document schema of the claim collection, the highlighted fields are used to retrieve the data that will be passed as context to the LLM Having prepared our data, building the RAG interaction is straightforward; refer to this GitHub repository for the implementation details. Here, we’ll just discuss the high-level architecture and the data flow, as shown in Figure 3 below: The user enters the prompt, a question in natural language. The prompt is vectorized and sent to Atlas Vector Search; similar documents are retrieved. The prompt and the retrieved documents are passed to the LLM as context. The LLM produces an answer to the user (in natural language), considering the context and the prompt. Figure 3: RAG architecture and interaction flow It is important to note how the semantics of the question are preserved throughout the different steps. The reference to “adverse weather” related accidents in the prompt is captured and passed to Atlas Vector Search, which surfaces claim documents whose claim description relates to similar concepts (e.g., rain) without needing to mention them explicitly. Finally, the LLM consumes the relevant documents to produce a context-aware question referencing rain, hail, and fire, as we’d expect based on the user's initial question. So what? To sum it all up, what’s the benefit of combining Atlas Vector Search and LLMs in a Claim Processing RAG application? Speed and accuracy: Having the data centrally organized and ready to be consumed by LLMs, adjusters can find all the necessary information in a fraction of the time. Flexibility: LLMs can answer a wide spectrum of questions, meaning applications require less upfront system design. There is no need to build custom APIs for each piece of information you’re trying to retrieve; just ask the LLM to do it for you. Natural interaction: Applications can be interrogated in plain English without programming skills or system training. Data accessibility: Insurers can finally leverage and explore unstructured data that was previously hard to access. Not just claim processing The same data model and architecture can serve additional personas and use cases within the organization: Customer Service: Operators can quickly pull customer data and answer complex questions without navigating different systems. For example, “Summarize this customer's past interactions,” “What coverages does this customer have?” or “What coverages can I recommend to this customer?” Customer self-service: Simplify your members’ experience by enabling them to ask questions themselves. For example, “My apartment is flooded. Am I covered?” or “How long do windshield repairs take on average?” Underwriting: Underwriters can quickly aggregate and summarize information, providing quotes in a fraction of the time. For example, “Summarize this customer claim history.” “I Am renewing a customer policy. What are the customer's current coverages? Pull everything related to the policy entity/customer. I need to get baseline info. Find relevant underwriting guidelines.” If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: RAG for claim processing GitHub repository From Relational Databases to AI: An Insurance Data Modernization Journey Modernize your insurance data models with MongoDB and Relational Migrator

April 18, 2024