Search Modernization with Pureinsights and MongoDB Atlas Search

Phil Lewis and Kamran Khan
April 17, 2023

Search engines have transformed in recent years from basic tools used for retrieving data within an organization, searching product catalogs and ecommerce sites, to astonishing cognitive search tools. Cognitive search combines traditional search with knowledge graph and advanced AI technologies like natural language processing (NLP) and machine learning (ML). These technologies provide powerful search and discovery experiences for workplaces, ecommerce platforms, customer support, websites, or any applications where people are looking for information.

Together with Pureinsights, MongoDB Atlas Search helps builders and developers achieve their moonshots for AI-enabled search technologies. Read this blog to learn how MongoDB’s partnership with Pureinsights is bringing futuristic search capabilities to businesses and agencies around the world.

MongoDB Atlas Search + Pureinsights Discovery Platform

The Pureinsights Discovery Platform™ (PDP) enhances traditional search engines and enables a ‘Google-like’ search experience. Built from best-in-class components and services, it uses a modern cloud-based architecture, and incorporates data connectors, content processing, AI services, a search engine (MongoDB Atlas Search) and knowledge graph.

The AI services can include ML, modern transform models and large language models (LLMs). Used together these tools can provide powerful features such as: natural language queries, question answering and vector search/extractive answers. These features go way beyond what a typical enterprise search engine provides, enabling organizations to give their people the search experience they now expect.

PDP also enables developers to ingest content and then process and enrich it to enhance search capabilities. PDP also provides tools to understand the intent of a user’s query so that we can deliver the most relevant, personalized and actionable search results.

Six ways Pureinsights unleashes the full potential of MongoDB Atlas Search

Pureinsights have been working with MongoDB to help clients realize the full potential of Atlas Search. Here is a selection of our search capabilities:

Migration: Help with migrating a legacy search system such as Elasticsearch, Solr or MarkLogic to Atlas Search.
Application build: Search applications can be complex and difficult to implement. Pureinsights can help you design, build and deploy your Atlas Search application and then work with you to maintain its performance.
Modernization: We can use PDP to integrate AI technologies (e.g. ChatGPT) with Atlas Search to enable cognitive search features such as natural language queries, question answering and extractive answers.
Relevancy tuning: Our expertise in search relevancy tuning and search engine scoring helps you develop a repeatable process to continually improve search engine results and relevancy. This proven methodology has helped our clients deliver optimized search experiences.
Data engineering and transformation: Search application results are data-driven. Pureinsights can help you with data transformation, migration and enrichment to ensure the best possible search experience for your users.
Managed services: And finally we can even run your search application for you with our SearchOps fully managed service so you can focus on your core business.

Next steps

Whether you are evaluating MongoDB Atlas or MongoDB Atlas Search, or ready to implement a solution Pureinsights can ensure that your project meets your technical and business goals. If you are already using MongoDB Atlas, then the decision to use Atlas Search should be an easy one. And if you have a separate search system then let us help you migrate to Atlas Search, for Pureinsights, it’s a well-trodden path.

Pureinsights has certified MongoDB engineers who use best practice Agile and Lean principles to deliver successful projects. We can also assist you in realizing the full potential of your existing Atlas Search application or even take you beyond search by utilizing knowledge graphs, ML, and more to develop enterprise search applications that better understand user intent and deliver the information users require.

Additional resources

← Previous

What's New in Atlas Charts: Schedule Dashboard Reports to Share Data with Your Team

Today, we’re introducing an exciting feature addition for teams using Atlas Charts . Charts project owners can now schedule dashboard reports to be sent via email to keep team members informed about key data. This feature has been heavily requested by some of our largest users as there are many use cases where dashboards may be valuable to your team, but you don’t necessarily want to require anyone to do extra work to access and view data. Enter scheduled dashboard reports in Atlas Charts! In any dashboard that your team relies on for regular data review, simply schedule a dashboard report. The new Schedule button can be found at the top right of the dashboard screen: Once you’ve chosen a dashboard from which to create a report, you will see a variety of options letting you customize the content and frequency of your report before you schedule. A report requires basic fields like a name or subject line, recipient list, and optionally, a message for the body of the email. In addition to a link to the dashboard in Charts, you can choose whether to attach an image or PDF for quick reference in the message itself. Finally, you can set a schedule of daily, weekly, monthly, or quarterly delivery. You can also simply send a single email if you have a one-time need to share a report. And once you’ve set everything up, your email will be sent on your defined schedule. As you use scheduled dashboard reports more and more, we created a Reports page where you can manage all reports in your project. Note that if you’re on an free tier, you can try one scheduled report. If you’re on an M2 cluster or higher, you can create up to 100 reports per project. To learn more, please check out our documentation . We’re always listening to feature requests that will enhance using Charts across teams, so if you have any requests or feedback, please share them with us here . Log in to Atlas Charts today to schedule your first report! If you’re new to Atlas Charts, get started today by logging into or signing up for MongoDB Atlas.

April 13, 2023

Next →

Retrieval Augmented Generation for Claim Processing: Combining MongoDB Atlas Vector Search and Large Language Models

Following up on our previous blog, AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases , we’ll pick up the conversation right where we left it. We discussed extensively how Atlas Vector Search can benefit the claim process in insurance and briefly covered Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More One of the biggest challenges for claim adjusters is pulling and aggregating information from disparate systems and diverse data formats. PDFs of policy guidelines might be stored in a content-sharing platform, customer information locked in a legacy CRM, and claim-related pictures and voice reports in yet another tool. All of this data is not just fragmented across siloed sources and hard to find but also in formats that have been historically nearly impossible to index with traditional methods. Over the years, insurance companies have accumulated terabytes of unstructured data in their data stores but have failed to capitalize on the possibility of accessing and leveraging it to uncover business insights, deliver better customer experiences, and streamline operations. Some of our customers even admit they’re not fully aware of all the data in their archives. There’s a tremendous opportunity to leverage this unstructured data to benefit the insurer and its customers. Our image search post covered part of the solution to these challenges, opening the door to working more easily with unstructured data. RAG takes it a step further, integrating Atlas Vector Search and LLMs, thus allowing insurers to go beyond the limitations of baseline foundational models, making them context-aware by feeding them proprietary data. Figure 1 shows how the interaction works in practice: through a chat prompt, we can ask questions to the system, and the LLM returns answers to the user and shows what references it used to retrieve the information contained in the response. Great! We’ve got a nice UI, but how can we build an RAG application? Let’s open the hood and see what’s in it! Figure 1: UI of the claim adjuster RAG-powered chatbot Architecture and flow Before we start building our application, we need to ensure that our data is easily accessible and in one secure place. Operational Data Layers (ODLs) are the recommended pattern for wrangling data to create single views. This post walks the reader through the process of modernizing insurance data models with Relational Migrator, helping insurers migrate off legacy systems to create ODLs. Once the data is organized in our MongoDB collections and ready to be consumed, we can start architecting our solution. Building upon the schema developed in the image search post , we augment our documents by adding a few fields that will allow adjusters to ask more complex questions about the data and solve harder business challenges, such as resolving a claim in a fraction of the time with increased accuracy. Figure 2 shows the resulting document with two highlighted fields, “claimDescription” and its vector representation, “claimDescriptionEmbedding” . We can now create a Vector Search index on this array, a key step to facilitate retrieving the information fed to the LLM. Figure 2: document schema of the claim collection, the highlighted fields are used to retrieve the data that will be passed as context to the LLM Having prepared our data, building the RAG interaction is straightforward; refer to this GitHub repository for the implementation details. Here, we’ll just discuss the high-level architecture and the data flow, as shown in Figure 3 below: The user enters the prompt, a question in natural language. The prompt is vectorized and sent to Atlas Vector Search; similar documents are retrieved. The prompt and the retrieved documents are passed to the LLM as context. The LLM produces an answer to the user (in natural language), considering the context and the prompt. Figure 3: RAG architecture and interaction flow It is important to note how the semantics of the question are preserved throughout the different steps. The reference to “adverse weather” related accidents in the prompt is captured and passed to Atlas Vector Search, which surfaces claim documents whose claim description relates to similar concepts (e.g., rain) without needing to mention them explicitly. Finally, the LLM consumes the relevant documents to produce a context-aware question referencing rain, hail, and fire, as we’d expect based on the user's initial question. So what? To sum it all up, what’s the benefit of combining Atlas Vector Search and LLMs in a Claim Processing RAG application? Speed and accuracy: Having the data centrally organized and ready to be consumed by LLMs, adjusters can find all the necessary information in a fraction of the time. Flexibility: LLMs can answer a wide spectrum of questions, meaning applications require less upfront system design. There is no need to build custom APIs for each piece of information you’re trying to retrieve; just ask the LLM to do it for you. Natural interaction: Applications can be interrogated in plain English without programming skills or system training. Data accessibility: Insurers can finally leverage and explore unstructured data that was previously hard to access. Not just claim processing The same data model and architecture can serve additional personas and use cases within the organization: Customer Service: Operators can quickly pull customer data and answer complex questions without navigating different systems. For example, “Summarize this customer's past interactions,” “What coverages does this customer have?” or “What coverages can I recommend to this customer?” Customer self-service: Simplify your members’ experience by enabling them to ask questions themselves. For example, “My apartment is flooded. Am I covered?” or “How long do windshield repairs take on average?” Underwriting: Underwriters can quickly aggregate and summarize information, providing quotes in a fraction of the time. For example, “Summarize this customer claim history.” “I Am renewing a customer policy. What are the customer's current coverages? Pull everything related to the policy entity/customer. I need to get baseline info. Find relevant underwriting guidelines.” If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: RAG for claim processing GitHub repository From Relational Databases to AI: An Insurance Data Modernization Journey Modernize your insurance data models with MongoDB and Relational Migrator

April 18, 2024