How to Get Mobile Data Sync Right with Mobile Backend as a Service (MBaaS)

Steve Jurczak
January 4, 2023
#Mobile

Twenty years ago, Watts Humphrey, known today as the "Father of Software Quality," declared that every business is a software business. While his insight now seems obvious, digital technology has evolved to where we can add to it: Every business is also a mobile business. According to Gartner, 75% of enterprise data will be generated and processed away from the central data center by 2025. And according to data.ai, 84% of enterprises attribute growth in productivity to mobile apps.

Today, mobile tech transforms every aspect of business. It enables the workforce through point-of-sale, inventory, service, and sales. It streamlines critical business processes like self-checkout and customer communications. And it powers essential work devices from telemetry to IoT to manufacturing.

The data businesses capture on mobile and edge devices can be used to improve operational efficiency, drive process improvements, and deliver richer, real-time app experiences. But all of this requires a solution for synchronizing mobile data with backend systems, where it can be combined with other historical data, analyzed, or fed into predictive intelligence algorithms to surface new insights and trigger other value-add activities. But syncing mobile data with backend systems can be hard for a number of reasons.

Mobile devices are constantly going in and out of coverage. When connections break and then resume, conflicts emerge between edits that were made on devices while offline and other data that's being processed on the backend. So conflict resolution becomes a crucial part of ensuring changes on the mobile device are captured on the backend in a way that ensures data integrity.

Sync and swim

Apps that are not designed with backend sync in mind can take a long time to load, are prone to crashing, and show stale information. When apps don’t deliver positive experiences, people stop trusting them — and stop using them.

On the other hand, an app with robust sync between a device’s local data store and the back end lets workers see live data across users and devices, allowing for real-time collaboration and decision-making. According to Deloitte, 70% of workers don’t sit at a desk every day, so the ability to sync data will increasingly drive business outcomes.

Indian startup, FloBiz, uses MongoDB Atlas Device Sync to handle the difficult job of keeping the mobile, desktop, and web apps in sync. This means even if multiple users were using the same account, going offline and online, there would be no issues, duplications or lost data.

Why data sync is difficult

A lot of organizations choose to build their own sync solutions. DIY solutions can go one of two ways: overly complex or oversimplified, resulting in sync that happens only a few times a day or in only one direction. It can be complicated and time-consuming for developers to write their own conflict-resolution code because building data sync the right way takes potentially thousands of lines of code. Developers frequently underestimate the challenge because it seems straightforward on the surface. They assume sync consists simply of the application making a request from the server, receiving some data, and using that data to update the app’s UI on the device. But when building for mobile devices this is a massive oversimplification.

Representation for how people perceive building sync versus what it is actually like; visualized by an iceberg. Above the surface is what people thing building sync is like, which is as follows; Step 1, applications request. Step 2, received data. Step 3, Update UI. Below the surface of the iceberg is what building sync is actually like, which is a complicated weave of questions and steps intertwined together. There are plenty of chances for the app to fail, which would result in a retry of the system.
Building data sync can be more complicated than people assume.

When developers attempt to build their own sync tool, they typically use RESTful APIs to connect the mobile app with the backend and exchange data between them. Mobile apps are often built more like web apps in the beginning. But once the need to handle offline scenarios arises, and because some functionality requires local persistence, then it becomes necessary to add a mobile database. Syncing with that mobile database then becomes a challenge. The exchange of data between the device and the back end gets complicated. It requires the developer to anticipate numerous offline use cases and write complex conflict-resolution code. It can be done, but it’s a time-consuming process that’s not guaranteed to solve all use cases. When data is requested, applications need to understand whether a network is available, and if not, whether the appropriate data is stored locally, leading to complex query, retry, and error handling logic.

The worst part about all this complexity is that it’s non-differentiating, meaning it doesn’t set the business apart from the competition. Users expect the functionality powered by data sync and won’t tolerate anything less.

An integrated, out-of-the-box solution

MongoDB's Atlas Device Sync combined with Realm is a mobile backend as a service (MBaaS) solution that enables developers to build offline-first applications that automatically refresh when a connection is reestablished. Local and edge data persistence is managed by Realm, a development platform designed for modern, data-driven applications. Developers use Realm to build mobile, web, desktop, and IoT apps. Realm is a fast and scalable alternative to SQLite and Core Data for client-side persistence. The bidirectional data synchronization service between Realm and MongoDB Atlas allows businesses to do more with their data at the edge by tapping into some of MongoDB’s more powerful data processing capabilities in the cloud. Complex synchronization problems such as conflict resolution are handled automatically by MongoDB’s built-in sync.

To learn more about the challenges of building real-time mobile apps that scale, with sample use cases about how thousands of businesses are handling it today, download our white paper, Building Real-time Mobile Apps that Scale.

← Previous

Introducing MongoDB Spark Connector Version 10.1

Today, MongoDB released version 10.1 of the MongoDB Spark Connector. In this post, we highlight key features of this new release. Microbatch streaming support The MongoDB Spark connection version 10 introduced support for Apache Structured Spark Streaming. In this initial release, continuous mode streaming was the only mode supported. In this 10.1 update, microbatch mode is now supported, enabling you to stream writes to destinations that currently do not support continuous mode streams, such as Amazon S3 storage. Increased control of write behavior When the Spark Connector issues a write, the default behavior is for an upsert to occur. This can cause problems in some scenarios in which you may not want an upsert, such as with time series collections. There is a new configuration parameter, upsertDocument , that, when set to false, will only issue insert statements on write. solar.write.format("mongodb").mode("append").option("database", "sensors").option("collection", "panels").option("upsertDocument", "false").save() In the above code snippet we are writing to the "panels" time series collection by setting the upsertDocument to false. Alternatively, you can set operationType to the value, “insert”. Setting this option will ignore any upsertDocument option set. Support for BSON types The data types supported in BSON are not exactly the same as those supported in a Spark dataframe. For example, Spark doesn't support ObjectId as a type specifically. To mitigate these scenarios where you need to leverage different BSON types, you can now set the new configuration values : spark.mongodb.read.outputExtendedJson=<true/false> spark.mongodb.write.convertJson=<true/false> This will enable you to effectively leverage BSON datatypes within your Spark application. Call to action Version 10.1 of the MongoDB Spark Connector continues to enhance the streaming capabilities with support for microbatch processing. This version also adds more granular support for writing to MongoDB supporting use cases like time series collections. For those users wanting to upgrade from the 3.x version but could not because of lack of BSON data type support, the 10.1 version now provides an option for using BSON data types. To learn more about the MongoDB Spark Connector check out the online documentation . You can download the latest version of the MongoDB Spark Connector from the maven repository .

January 4, 2023
Next →

Retrieval Augmented Generation for Claim Processing: Combining MongoDB Atlas Vector Search and Large Language Models

Following up on our previous blog, AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases , we’ll pick up the conversation right where we left it. We discussed extensively how Atlas Vector Search can benefit the claim process in insurance and briefly covered Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More One of the biggest challenges for claim adjusters is pulling and aggregating information from disparate systems and diverse data formats. PDFs of policy guidelines might be stored in a content-sharing platform, customer information locked in a legacy CRM, and claim-related pictures and voice reports in yet another tool. All of this data is not just fragmented across siloed sources and hard to find but also in formats that have been historically nearly impossible to index with traditional methods. Over the years, insurance companies have accumulated terabytes of unstructured data in their data stores but have failed to capitalize on the possibility of accessing and leveraging it to uncover business insights, deliver better customer experiences, and streamline operations. Some of our customers even admit they’re not fully aware of all the data in their archives. There’s a tremendous opportunity to leverage this unstructured data to benefit the insurer and its customers. Our image search post covered part of the solution to these challenges, opening the door to working more easily with unstructured data. RAG takes it a step further, integrating Atlas Vector Search and LLMs, thus allowing insurers to go beyond the limitations of baseline foundational models, making them context-aware by feeding them proprietary data. Figure 1 shows how the interaction works in practice: through a chat prompt, we can ask questions to the system, and the LLM returns answers to the user and shows what references it used to retrieve the information contained in the response. Great! We’ve got a nice UI, but how can we build an RAG application? Let’s open the hood and see what’s in it! Figure 1: UI of the claim adjuster RAG-powered chatbot Architecture and flow Before we start building our application, we need to ensure that our data is easily accessible and in one secure place. Operational Data Layers (ODLs) are the recommended pattern for wrangling data to create single views. This post walks the reader through the process of modernizing insurance data models with Relational Migrator, helping insurers migrate off legacy systems to create ODLs. Once the data is organized in our MongoDB collections and ready to be consumed, we can start architecting our solution. Building upon the schema developed in the image search post , we augment our documents by adding a few fields that will allow adjusters to ask more complex questions about the data and solve harder business challenges, such as resolving a claim in a fraction of the time with increased accuracy. Figure 2 shows the resulting document with two highlighted fields, “claimDescription” and its vector representation, “claimDescriptionEmbedding” . We can now create a Vector Search index on this array, a key step to facilitate retrieving the information fed to the LLM. Figure 2: document schema of the claim collection, the highlighted fields are used to retrieve the data that will be passed as context to the LLM Having prepared our data, building the RAG interaction is straightforward; refer to this GitHub repository for the implementation details. Here, we’ll just discuss the high-level architecture and the data flow, as shown in Figure 3 below: The user enters the prompt, a question in natural language. The prompt is vectorized and sent to Atlas Vector Search; similar documents are retrieved. The prompt and the retrieved documents are passed to the LLM as context. The LLM produces an answer to the user (in natural language), considering the context and the prompt. Figure 3: RAG architecture and interaction flow It is important to note how the semantics of the question are preserved throughout the different steps. The reference to “adverse weather” related accidents in the prompt is captured and passed to Atlas Vector Search, which surfaces claim documents whose claim description relates to similar concepts (e.g., rain) without needing to mention them explicitly. Finally, the LLM consumes the relevant documents to produce a context-aware question referencing rain, hail, and fire, as we’d expect based on the user's initial question. So what? To sum it all up, what’s the benefit of combining Atlas Vector Search and LLMs in a Claim Processing RAG application? Speed and accuracy: Having the data centrally organized and ready to be consumed by LLMs, adjusters can find all the necessary information in a fraction of the time. Flexibility: LLMs can answer a wide spectrum of questions, meaning applications require less upfront system design. There is no need to build custom APIs for each piece of information you’re trying to retrieve; just ask the LLM to do it for you. Natural interaction: Applications can be interrogated in plain English without programming skills or system training. Data accessibility: Insurers can finally leverage and explore unstructured data that was previously hard to access. Not just claim processing The same data model and architecture can serve additional personas and use cases within the organization: Customer Service: Operators can quickly pull customer data and answer complex questions without navigating different systems. For example, “Summarize this customer's past interactions,” “What coverages does this customer have?” or “What coverages can I recommend to this customer?” Customer self-service: Simplify your members’ experience by enabling them to ask questions themselves. For example, “My apartment is flooded. Am I covered?” or “How long do windshield repairs take on average?” Underwriting: Underwriters can quickly aggregate and summarize information, providing quotes in a fraction of the time. For example, “Summarize this customer claim history.” “I Am renewing a customer policy. What are the customer's current coverages? Pull everything related to the policy entity/customer. I need to get baseline info. Find relevant underwriting guidelines.” If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: RAG for claim processing GitHub repository From Relational Databases to AI: An Insurance Data Modernization Journey Modernize your insurance data models with MongoDB and Relational Migrator

April 18, 2024