Choosing the Right Tool for the Job: Understanding the Analytics Spectrum

Mat Keep and Steve Jurczak
November 30, 2022 | Updated: December 7, 2023

Data-driven organizations share a common desire to get more value out of the data they're generating. To maximize that value, many of them are asking the same or similar questions:

  • How long does it take to get analytics and insights from our application data?

  • What would be the business impact if we could make that process faster? What new experiences could we create by having analytics integrated directly within our customer-facing apps?

  • How do our developers access the tools and APIs they need to build sophisticated analytics queries directly into their application code?

  • How do we make sense of voluminous streams of time-series data?

We believe the answer to these questions in today's digital economy is application-driven analytics.

What is Application-Driven Analytics?

Traditionally, there's been a separation at organizations between applications that run the business and analytics that manage the business. They're built by different teams, they serve different audiences, and the data itself is replicated and stored in different systems.

There are benefits to the traditional way of doing things and it's not going away. However, in today's digital economy, where the need to create competitive advantage and reduce costs and risk are paramount, organizations will continue to innovate upon the traditional model.

Today, those needs manifest themselves in the demand for smarter applications that drive better customer experiences and surface insights to initiate intelligent actions automatically. This all happens within the flow of the application on live, operational data in real time. Alongside those applications, the business also wants faster insights so it can see what's happening, when it's happening. This is known as business visibility, and the goal of it is to increase efficiency by enabling faster decisions on fresher data. In-app analytics and real-time visibility are enabled by what we call application-driven analytics.

You can find examples of application-driven analytics in multiple real-world industry use cases including:

  • Hyper-personalization in retail

  • Fraud prevention in financial services

  • Preventative maintenance in manufacturing

  • Single subscriber view in telecommunications

  • Fitness tracking in healthcare

  • A/B testing in gaming

Where Application-Driven Analytics fits in the Analytics Ecosystem

Application-driven analytics complements existing analytics processes where data is moved out of operational systems into centralized data warehouses and data lakes. In no way does it replace them. However, a broader spectrum of capabilities are now required to meet more demanding business requirements.

Contrasting the two approaches, application-driven analytics is designed to continuously query data in your operational systems. The freshest data comes in from the application serving many concurrent users at very low latency. It involves working on much smaller subsets of data compared to centralized analytics systems. Application-driven analytics is typically working with hundreds to possibly a few thousand records at a time. And it's running less complex queries against that data.

At the other end of the spectrum is centralized analytics. These systems are running much more complex queries across massive data sets — hundreds of thousands or maybe millions of records, and maybe at petabyte scale — that have been ingested from many different operational data sources across the organization.

Table 1 below identifies the required capabilities across the spectrum of different classes of analytics. These are designed to help MongoDB’s customers match appropriate technologies and skill sets to each business use case they are building for.

By mapping required capabilities to use cases, you can see how these different classes of analytics serve different purposes. If, for example, we're dealing with recommendations in an e-commerce platform, the centralized data warehouse or data lake will regularly analyze vast troves of first- and third-party customer data. This analysis is then blended with available inventory to create a set of potential customer offers.

These offers are then loaded back into operational systems where application-driven analytics is used to decide which offers are most relevant to the customer based on a set of real-time criteria, such as actual stock availability and which items a shopper might already have in their basket. This real-time decision-making is important because you wouldn't want to serve an offer on a product that can no longer be fulfilled or on an item a customer has already decided to buy. This example demonstrates why it is essential to choose the right tool for the job. Specifically, in order to build a portfolio of potential offers, the centralized data warehouse or data lake is an ideal fit. Such technologies can process hundreds of TBs of customer records and order data in a single query.

The same technologies, however, are completely inappropriate when it comes to serving those offers to customers in real time. Centralized analytics systems are not designed to serve thousands of concurrent user sessions. Nor can they access real-time inventory or basket data in order to make low latency decisions in milliseconds. Instead, for these scenarios, application-driven analytics served from an operational system is the right technology fit.

As we can see, application-driven analytics is complementary to traditional centralized analytics, and in no way competitive to it.

The benefits to organizations of using these complementary classes of analytics include:

  • Maximizing competitive advantage through smarter and more intelligent applications

  • Out-innovating and differentiating in the market

  • Improving customer experience and loyalty

  • Reducing cost by improving business visibility and efficiency

Through its design, MongoDB Atlas unifies the essential data services needed to deliver on application-driven analytics. It gives developers the tools, tech, and skills they need to infuse analytics into their apps. At the same time, Atlas provides business analysts, data scientists, and data engineers direct access to live data using their regular tools without impacting the app.

For more information about how to implement app-driven analytics and how the MongoDB developer data platform gives you the tools needed to succeed, download our white paper, Application-Driven Analytics: Defining the Next Wave of Modern Apps.

← Previous

Built With MongoDB: Inspirit Helps Kids Learn Science Through Immersive Technology

Immersive technologies like AR and VR have tremendous potential to transform learning outcomes for students. By representing complicated, often intangible concepts in an interactive, 3D platform and VR system, they encourage engagement and improve memory recall in STEM (science, technology, engineering, and mathematics) subjects. Palo Alto-based startup, Inspirit , has developed an immersive, interactive VR platform that combines the best of both 3D and VR instruction to allow students in middle school and high school to experience science and learning through virtual reality instead of reading. Inspirit is also part of the MongoDB for Startups program , which helps startups build faster and scale further with free MongoDB Atlas credits, one-on-one technical advice, co-marketing opportunities, and access to a vast partner network. Birth of a Startup Inspirit Co-founders Amrutha Vasan and Aditya Vishwanath were conducting research at Georgia Tech and trying to understand how to bring virtual reality into education when they had a critical insight. "A lot of students tend to lose sight or just lose interest in science by the time they hit high school," Vasan says. "What we realized very quickly was that you can give teachers the tools that they need in order to do things that are way too expensive or just simply impossible to do in the real world." As students get into higher order physics, calculus, biology, and chemistry, they struggle with visualizing difficult concepts. "What we aim to do is provide interactive 3D models and interactive simulations that actually teach them core science concepts so that they build that curiosity and actually engage with the material in order to keep them in STEM programs and moving forward," Vasan says. Company evolution Inspirit has evolved since its early days. The founders initially focused on VR headsets and 3D goggles that students would share in the classroom. Once the global COVID-19 pandemic hit, it was clear that students wouldn't be going back to the classroom anytime soon. "We had to very quickly pivot into building a web platform that students could use at home and teachers could use for hybrid and online teaching," Vasan says. "Through the pandemic, we learned very quickly exactly how the education system would actually be changing. And so through that adaptation, we've now built a cross-platform product." Building a cross-platform solution aligns with one of Inspirit's core values, which is to be accessible. Inspirit is now available on the web in addition to virtual reality headsets so students can access the tools on different platforms. Inspirit isn't just an extracurricular activity. One of the biggest differentiators between Inspirit VR classroom and other platforms is that it addresses core content. It's not something that students need to use independently. "You still need a really great teacher," Vasan says. "There is no way for you to show a student a eukaryotic cell other than by drawing it on a whiteboard or giving them a diagram or a video," she says. "You still need a great teacher in order to use our platform. But they can now use this to help their students visualize difficult science concepts." Inspirit helps students who love science immerse themselves in it more while at the same time helping kids who dislike science experience it in a new, more engaging way. Building with MongoDB Inspirit chose MongoDB over other platforms because of its support and flexibility. As a startup, the founders knew their database structure would be changing constantly. "MongoDB allows for a lot of flexibility," Vasan says, "which is really important because we're probably not going to have the perfect database going into a startup. Allowing us to consistently change it as we're scaling and growing has just been very helpful." Regarding MongoDB support, which is one of the benefits of the MongoDB for Startups program, Vasan says the company got the most support from MongoDB than any of the other databases they were looking at. "That's ultimately why we ended up choosing it for our company," she says. "The things that we love the most about MongoDB are Atlas , Charts , and search indexes . Atlas has been really cool for visualizing all the data and for helping us grow and scale quickly as well." "We've had a really good partnership with the MongoDB for Startups program in particular," Vasan says. Inspirit uses AWS, and Vasan cites how easily Atlas integrates with AWS as another key benefit that she appreciates. "AWS integrates very easily with MongoDB. So to be honest, it's been pretty easy scaling our backend." Compliance was another critical requirement. "We actually need to be COPPA compliant," Vasan says, referring to the Children's Online Privacy Protection Rule. "There are a lot of data regulations for what we're using since we have sensitive student, parent, and teacher data. And so both MongoDB and AWS actually make that a lot easier." If you're interested in learning more about Inspirit's immersive learning tools, visit their website . Are you part of a startup and interested in joining the MongoDB for Startups program? Apply now . For more startup content, check out our previous blog on Qubitro .

November 30, 2022
Next →

Retrieval Augmented Generation for Claim Processing: Combining MongoDB Atlas Vector Search and Large Language Models

Following up on our previous blog, AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases , we’ll pick up the conversation right where we left it. We discussed extensively how Atlas Vector Search can benefit the claim process in insurance and briefly covered Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More One of the biggest challenges for claim adjusters is pulling and aggregating information from disparate systems and diverse data formats. PDFs of policy guidelines might be stored in a content-sharing platform, customer information locked in a legacy CRM, and claim-related pictures and voice reports in yet another tool. All of this data is not just fragmented across siloed sources and hard to find but also in formats that have been historically nearly impossible to index with traditional methods. Over the years, insurance companies have accumulated terabytes of unstructured data in their data stores but have failed to capitalize on the possibility of accessing and leveraging it to uncover business insights, deliver better customer experiences, and streamline operations. Some of our customers even admit they’re not fully aware of all the data in their archives. There’s a tremendous opportunity to leverage this unstructured data to benefit the insurer and its customers. Our image search post covered part of the solution to these challenges, opening the door to working more easily with unstructured data. RAG takes it a step further, integrating Atlas Vector Search and LLMs, thus allowing insurers to go beyond the limitations of baseline foundational models, making them context-aware by feeding them proprietary data. Figure 1 shows how the interaction works in practice: through a chat prompt, we can ask questions to the system, and the LLM returns answers to the user and shows what references it used to retrieve the information contained in the response. Great! We’ve got a nice UI, but how can we build an RAG application? Let’s open the hood and see what’s in it! Figure 1: UI of the claim adjuster RAG-powered chatbot Architecture and flow Before we start building our application, we need to ensure that our data is easily accessible and in one secure place. Operational Data Layers (ODLs) are the recommended pattern for wrangling data to create single views. This post walks the reader through the process of modernizing insurance data models with Relational Migrator, helping insurers migrate off legacy systems to create ODLs. Once the data is organized in our MongoDB collections and ready to be consumed, we can start architecting our solution. Building upon the schema developed in the image search post , we augment our documents by adding a few fields that will allow adjusters to ask more complex questions about the data and solve harder business challenges, such as resolving a claim in a fraction of the time with increased accuracy. Figure 2 shows the resulting document with two highlighted fields, “claimDescription” and its vector representation, “claimDescriptionEmbedding” . We can now create a Vector Search index on this array, a key step to facilitate retrieving the information fed to the LLM. Figure 2: document schema of the claim collection, the highlighted fields are used to retrieve the data that will be passed as context to the LLM Having prepared our data, building the RAG interaction is straightforward; refer to this GitHub repository for the implementation details. Here, we’ll just discuss the high-level architecture and the data flow, as shown in Figure 3 below: The user enters the prompt, a question in natural language. The prompt is vectorized and sent to Atlas Vector Search; similar documents are retrieved. The prompt and the retrieved documents are passed to the LLM as context. The LLM produces an answer to the user (in natural language), considering the context and the prompt. Figure 3: RAG architecture and interaction flow It is important to note how the semantics of the question are preserved throughout the different steps. The reference to “adverse weather” related accidents in the prompt is captured and passed to Atlas Vector Search, which surfaces claim documents whose claim description relates to similar concepts (e.g., rain) without needing to mention them explicitly. Finally, the LLM consumes the relevant documents to produce a context-aware question referencing rain, hail, and fire, as we’d expect based on the user's initial question. So what? To sum it all up, what’s the benefit of combining Atlas Vector Search and LLMs in a Claim Processing RAG application? Speed and accuracy: Having the data centrally organized and ready to be consumed by LLMs, adjusters can find all the necessary information in a fraction of the time. Flexibility: LLMs can answer a wide spectrum of questions, meaning applications require less upfront system design. There is no need to build custom APIs for each piece of information you’re trying to retrieve; just ask the LLM to do it for you. Natural interaction: Applications can be interrogated in plain English without programming skills or system training. Data accessibility: Insurers can finally leverage and explore unstructured data that was previously hard to access. Not just claim processing The same data model and architecture can serve additional personas and use cases within the organization: Customer Service: Operators can quickly pull customer data and answer complex questions without navigating different systems. For example, “Summarize this customer's past interactions,” “What coverages does this customer have?” or “What coverages can I recommend to this customer?” Customer self-service: Simplify your members’ experience by enabling them to ask questions themselves. For example, “My apartment is flooded. Am I covered?” or “How long do windshield repairs take on average?” Underwriting: Underwriters can quickly aggregate and summarize information, providing quotes in a fraction of the time. For example, “Summarize this customer claim history.” “I Am renewing a customer policy. What are the customer's current coverages? Pull everything related to the policy entity/customer. I need to get baseline info. Find relevant underwriting guidelines.” If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: RAG for claim processing GitHub repository From Relational Databases to AI: An Insurance Data Modernization Journey Modernize your insurance data models with MongoDB and Relational Migrator

April 18, 2024