5 Steps to Replacing Elasticsearch and Solr with Atlas Search

Mat Keep
March 28, 2022 | Updated: April 5, 2022
#Atlas Search

What do a global auto manufacturer, multinational media and entertainment company, and a challenger bank have in common? They have all made the switch from Elasticsearch to MongoDB Atlas Search to simplify their technology stack and ship application search faster.

But what problems were they solving and how did they migrate? We have a new 5-step guide that takes you through why they switched, and how they did it.

The need for application search

Type almost anything into a search bar on sites like Google, Amazon, and Netflix and you are instantly presented with relevant results. Whether you make a typo or enter a partial search term, the search engine figures out what you are looking for. Results are returned conveniently sorted by relevance and are easy to navigate with features like highlighting, filters, and counts.

Everyone now expects these same fast and intuitive search experiences in every application they use, whether at home or at work. However, creating these experiences is hard with the burden falling onto developers and ops teams who have to build and run the underlying systems.

The pain of building application search

MongoDB has always focused on accelerating and simplifying how developers build with data for any class of application. From our very earliest MongoDB releases, we saw developers needing to expose the application data stored in their database to search and information discovery.

For simple use cases – where it was enough to just match text in a field – developers were able to use the basic text search operators and index built into the MongoDB database. However these lacked the much more sophisticated speed and relevance tuning features offered by dedicated search engines, typically built on top of Apache Lucene.

As a result many developers ended up bolting on an external search engine such as Elasticsearch or Apache Solr to their database. Elasticsearch and Solr were (and remain) popular and proven. However as Figure 1 shows, they introduced a huge amount of complexity to the application stack, reducing developer velocity while driving up risk, complexity, and cost.

*Figure 1: The pain of bolting on a search engine to your database*

Working with the MongoDB community, our product designers and engineers ideated on ways to make building application search easier for developers – without compromising on the key features they needed. The result is MongoDB Atlas Search.

What is Atlas Search and why switch to it?

Atlas Search embeds a fully-managed Apache Lucene search index directly alongside the database and automatically synchronizes data between them.

By integrating the database, search engine, and sync pipeline into a single, fully-managed platform you get to compress three systems into one and simplify your technology stack. Engineering teams and application owners have reported improved development velocity of 30% to 50% after adopting Atlas Search. This is because they get to:

Eliminate the synchronization tax. Data is automatically and dynamically synced from the Atlas database to the Atlas Search indexes. They avoid having to stand up and manage their own sync mechanism, write custom transformation logic, or remap search indexes as their database schema evolves. They escape the 10% of engineering cycles typically lost to manually recovering sync failures, investing that time to innovate for their users instead. (1)
Ship new features faster. They work with a single, unified API across both database and search operations, simplifying query development. No more context switching between multiple query languages, and with a single driver, build dependencies are streamlined so they release faster. They can test queries and preview results with interactive tools to fine-tune performance and scoring before deploying them directly into application code.
Remove operational heavy-lifting. The fully-managed Atlas platform automates provisioning, replication, patching, upgrades, scaling, security, and disaster recovery while providing deep performance visibility into both database and search. By working with a single system, they avoid an exponential increase in the number of system components they need to design, test, secure, monitor, and maintain.

*Figure 2: Dramatic architectual simplification with integrated database, sync, and search in MongoDB Atlas*

5 steps to make the switch to Atlas Search

The benefits Atlas Search provides has led engineering teams across all industry sectors and geographies to make the switch from bolt-on search engines. Through the experiences gained by working with these teams, we have put together a repeatable 5-step methodology to replacing Elasticsearch and Solr.

The guide steps you through how to:

Qualify target workloads for Atlas Search.
Migrate your indexes to Atlas Search.
Migrate your queries to Atlas Search.
Validate and relevance-tune your Atlas Search queries and indexes.
Size and deploy your Atlas Search infrastructure.

*Figure 3: 5-step methodology to replacing Elasticsearch and Solr with Atlas Search*

The guide wraps up with examples of customers that have made the switch and provides guidance on how to get started with Atlas Search.

What's next?

You can get started today by downloading the 5-step guide to replacing Elasticsearch and Solr with Atlas Search.

The 5-step guide is designed to help you plan and execute your migration project. MongoDB's Professional Services team is also available to you as a trusted delivery partner. We can help you through any of the steps in the methodology or throughout your entire journey to Atlas Search.

If you want to dig deeper into Atlas Search, spin it up at no-cost on the Atlas Free Tier. You can follow along with reference materials and tutorials in the Atlas Search documentation using our sample data sets, or load your own data for experimentation within your own sandbox.

Welcome to a world where application search is, at last, simplified!

1. Based on interviews with engineering teams that have replaced bolt on search engines and the associated sync mechanism.

← Previous

Dashboard Embedding Comes to Charts

Atlas Charts is the native data visualization that lets you create, share, and embed charts from data in MongoDB Atlas. Today, we are happy to announce the release of Dashboard Embedding . What is an embedded dashboard? Embedding provides the ability to bring charts and dashboards into any webpage or application where your users spend time. It’s a great way to share dynamic, live visualizations with a wide audience. We see customers using embedded charts to share real-time information internally within their companies, as well as externally with their customers. Until now, embedding multiple charts where you need them has been time consuming and somewhat complex. With dashboard embedding , rather than individually embedding one chart at a time, you can embed a full dashboard in one go! Embedding charts and dashboards is easy to use and provides the flexibility you need when it comes to visual customizations and data security. This ensures you are able to freely share information and have it look the way you want, while always keeping data secure. How does Dashboard Embedding work in Atlas Charts? Enabling Dashboard Embedding is as simple as picking a dashboard, going into the Embed menu, and enabling either authenticated or unauthenticated embedding as shown below. The Dashboard Embedding Menu Similar to embedding an individual chart, you can publicly embed a URL in an iFrame of your application or website. Unauthenticated embedding allows you to embed your dashboards, and have visualizations immediately ready for public consumption. Alternatively, you can use our embedding SDK to implement authenticated embedding. Authenticated embedding provides more control and security, requiring users to have appropriate permission to access your dashboard. We support Google, Realm and Custom JSON Web Token authentication options . Charts offers a number of configuration options important to ensuring your embedded dashboards look and feel how you want. These options include support for dark mode, background color selection, chart sizing, and refresh rate cadence. Dashboards can even be set to resize responsively based on your screen size, ensuring a consistent user experience. Play around with dashboard embedding and some of these features in our code sandbox . An embedded dashboard in dark mode Get started today! Dashboard Embedding was one of our most frequently requested features in Atlas Charts and we are excited to see how you take advantage of it in your applications! Head over to any of your dashboards in Charts to enable dashboard embedding or take a look at Github to get started with the embedding SDK. New to Atlas Charts? Get started today by logging into or signing up for MongoDB Atlas , deploying or selecting a cluster, and activating Charts for free.

March 24, 2022

Next →

Retrieval Augmented Generation for Claim Processing: Combining MongoDB Atlas Vector Search and Large Language Models

Following up on our previous blog, AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases , we’ll pick up the conversation right where we left it. We discussed extensively how Atlas Vector Search can benefit the claim process in insurance and briefly covered Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More One of the biggest challenges for claim adjusters is pulling and aggregating information from disparate systems and diverse data formats. PDFs of policy guidelines might be stored in a content-sharing platform, customer information locked in a legacy CRM, and claim-related pictures and voice reports in yet another tool. All of this data is not just fragmented across siloed sources and hard to find but also in formats that have been historically nearly impossible to index with traditional methods. Over the years, insurance companies have accumulated terabytes of unstructured data in their data stores but have failed to capitalize on the possibility of accessing and leveraging it to uncover business insights, deliver better customer experiences, and streamline operations. Some of our customers even admit they’re not fully aware of all the data in their archives. There’s a tremendous opportunity to leverage this unstructured data to benefit the insurer and its customers. Our image search post covered part of the solution to these challenges, opening the door to working more easily with unstructured data. RAG takes it a step further, integrating Atlas Vector Search and LLMs, thus allowing insurers to go beyond the limitations of baseline foundational models, making them context-aware by feeding them proprietary data. Figure 1 shows how the interaction works in practice: through a chat prompt, we can ask questions to the system, and the LLM returns answers to the user and shows what references it used to retrieve the information contained in the response. Great! We’ve got a nice UI, but how can we build an RAG application? Let’s open the hood and see what’s in it! Figure 1: UI of the claim adjuster RAG-powered chatbot Architecture and flow Before we start building our application, we need to ensure that our data is easily accessible and in one secure place. Operational Data Layers (ODLs) are the recommended pattern for wrangling data to create single views. This post walks the reader through the process of modernizing insurance data models with Relational Migrator, helping insurers migrate off legacy systems to create ODLs. Once the data is organized in our MongoDB collections and ready to be consumed, we can start architecting our solution. Building upon the schema developed in the image search post , we augment our documents by adding a few fields that will allow adjusters to ask more complex questions about the data and solve harder business challenges, such as resolving a claim in a fraction of the time with increased accuracy. Figure 2 shows the resulting document with two highlighted fields, “claimDescription” and its vector representation, “claimDescriptionEmbedding” . We can now create a Vector Search index on this array, a key step to facilitate retrieving the information fed to the LLM. Figure 2: document schema of the claim collection, the highlighted fields are used to retrieve the data that will be passed as context to the LLM Having prepared our data, building the RAG interaction is straightforward; refer to this GitHub repository for the implementation details. Here, we’ll just discuss the high-level architecture and the data flow, as shown in Figure 3 below: The user enters the prompt, a question in natural language. The prompt is vectorized and sent to Atlas Vector Search; similar documents are retrieved. The prompt and the retrieved documents are passed to the LLM as context. The LLM produces an answer to the user (in natural language), considering the context and the prompt. Figure 3: RAG architecture and interaction flow It is important to note how the semantics of the question are preserved throughout the different steps. The reference to “adverse weather” related accidents in the prompt is captured and passed to Atlas Vector Search, which surfaces claim documents whose claim description relates to similar concepts (e.g., rain) without needing to mention them explicitly. Finally, the LLM consumes the relevant documents to produce a context-aware question referencing rain, hail, and fire, as we’d expect based on the user's initial question. So what? To sum it all up, what’s the benefit of combining Atlas Vector Search and LLMs in a Claim Processing RAG application? Speed and accuracy: Having the data centrally organized and ready to be consumed by LLMs, adjusters can find all the necessary information in a fraction of the time. Flexibility: LLMs can answer a wide spectrum of questions, meaning applications require less upfront system design. There is no need to build custom APIs for each piece of information you’re trying to retrieve; just ask the LLM to do it for you. Natural interaction: Applications can be interrogated in plain English without programming skills or system training. Data accessibility: Insurers can finally leverage and explore unstructured data that was previously hard to access. Not just claim processing The same data model and architecture can serve additional personas and use cases within the organization: Customer Service: Operators can quickly pull customer data and answer complex questions without navigating different systems. For example, “Summarize this customer's past interactions,” “What coverages does this customer have?” or “What coverages can I recommend to this customer?” Customer self-service: Simplify your members’ experience by enabling them to ask questions themselves. For example, “My apartment is flooded. Am I covered?” or “How long do windshield repairs take on average?” Underwriting: Underwriters can quickly aggregate and summarize information, providing quotes in a fraction of the time. For example, “Summarize this customer claim history.” “I Am renewing a customer policy. What are the customer's current coverages? Pull everything related to the policy entity/customer. I need to get baseline info. Find relevant underwriting guidelines.” If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: RAG for claim processing GitHub repository From Relational Databases to AI: An Insurance Data Modernization Journey Modernize your insurance data models with MongoDB and Relational Migrator

April 18, 2024