Migrating Terabytes of IoT Data From a Legacy NoSQL Database to MongoDB Atlas With MongoDB's Custom Migration Tool

Paresh Saraf, Rajesh Vinayagam, and Krishnakumar Sathyanarayanan
July 11, 2022 | Updated: August 17, 2022

In 2020, a large European energy company began an ambitious plan to replace its traditional metering devices — all 7.6 million of them — with smart meters. That would allow the energy company to monitor gas use remotely and allow customers’ bills to more accurately reflect their energy consumption. At the same time, the company began installing smart components along their production network to monitor operations in real-time, manage alarms, use predictive maintenance tools, and find leaks using advanced technologies.

The energy company knew this shift would result in a massive amount of data coming into their systems, and they thought they were ready for it. They understood the complexities of managing and leveraging data from the Internet of Things (IoT), such as the high velocity at which data must be ingested and the need for time-based data aggregations. They rolled out an IoT platform with big data and analytics tools to help them make progress toward their objectives of high-quality, efficient, and safe service.

This article examines how the company migrated its system to MongoDB Atlas in order to handle the massive influx of data.

Managing data

The energy company was managing 3 TB of data on its NoSQL database, with the remainder housed and managed on a relational database. However, it started facing challenges, including a lack of scalability, increasing costs, and poor performance. The costs to maintain the pre-production and production environments were unsustainable, and the situation wasn’t going to get better: By 2023, the energy company planned to increase the number of IoT devices and sensors by a factor of five. They needed a viable solution for the long term.

Migrating to MongoDB Atlas

The energy company decided to migrate to MongoDB Atlas for several reasons. Atlas’s online archive, combined with the ability to create time-series sharded collections, makes Atlas an ideal fit for IoT data, as does the flexibility of the document data model. Additionally, an API that was compatible with the existing database would minimize the impact on application code and make it easier to migrate applications.

The customer chose PeerIslands to be its technical partner and help them with the migration. PeerIslands, a MongoDB partner, is an enterprise-class digital transformation company with an expert, multilingual team with significant experience working across multiple technologies and cloud platforms.

PeerIslands has developed solutions for both homogenous and heterogenous workload migrations. Among these solutions is a MongoDB migration tool that helps perform one-time migrations and change data capture while minimizing downtime. The tool is fully GUI-based, and tasks such as infrastructure provisioning, dump and restore, change stream listeners, and processors have all been automated. For change capture, the tool uses the native MongoDB change stream APIs.

Migration challenges

In working with the energy company to perform the migration, the PeerIslands team faced two particular challenges:

The large volume of data. Initial snap-shotting of the data would take about a day.
The application had significant write loads. On average, it was writing about 12,000 messages per second. However, the load was unevenly distributed, with spikes when devices would “wake up” and report their status.

These two factors quickly generated close to 20 million change events that had to be synced to MongoDB. Meanwhile, new data was constantly being written into the source.

Migration tool

PeerIslands’ migration tool uses mongodump and mongorestore for one-time data migration and MongoDB Kafka Connector for real-time data synchronization. By using Apache Kafka, the migration tool was able to handle the large amount of change stream data and successfully manage the migration.

To address the complexity of the migration, PeerIslands also enhanced the migration tool with additional capabilities:

Parallelize the Kafka change stream processing using partitions. The Kafka partitioning strategy was in sync with the target Atlas sharding strategy.
Use ReplaceOneBusinessKeyStrategy as the write model for Kafka MongoDB sink connector to write into sharded Atlas collections.

By using its in-house tooling, PeerIslands was able to successfully complete the migration with near zero downtime.

Improved performance

With the migration complete, the customer has already begun to realize the benefits of MongoDB Atlas for their massive amounts of IoT data. The user interface has become extremely responsive, even in front of more expensive queries.

Because of the improved performance of the database, the customer is now able to pursue improvements and efficiencies in other areas. With better performance, the company expects consumption of the data to rise and their schema design to evolve. They’re looking to leverage the time-series benefits of MongoDB both to simplify their schema design and deliver richer IoT functionality. They’re also better equipped to rapidly respond to and fulfill business needs, because the database is no longer a limitation. Importantly, costs have decreased for the production environment, and even more dramatic cost reductions have been seen for the pre-production environment.

Learn more about the migration tool and MongoDB’s time series capabilities. Interested in your own data migration? Contact us.

← Previous
How These MongoDB Employees Celebrated Juneteenth
On June 19, 1865, soldiers arrived in Galveston, Texas, announcing that the more than 250,000 enslaved Black people in the state were free by executive decree. This was more than two years after the Emancipation Proclamation was signed. Today, June 19 is celebrated as Juneteenth, a day of hope despite present-day uncertainty. It reminds us that at the end of every struggle there comes a time for a change if we persist and do not give up. Juneteenth is a federal holiday in the United States, and MongoDB recognizes this by providing employees with the day off to celebrate and reflect. Members of MongoDB’s affinity group the Underrepresented People of Color share what they did to celebrate. Supporting Black Businesses Kayla Warner , Internal Communications Manager Some of the delicious food and the hands that prepared it, Chef Will Coleman (@chefwillcoleman). Every year, I have to get soul food on Juneteenth. It makes me feel the most connected to my culture (and it’s always great to support small Black businesses). I spent this Juneteenth at a friend’s restaurant pop-up. Being from the Southern United States, it’s not often that I get to have the comfort foods of home in New York. His pop-up had fried fish po’boys, smoked watermelon feta salad, crab deviled eggs, strawberry shortcake biscuits, and sweet tea (that was actually sweet). These dishes and flavors brought me back to backyard cookouts and fish fries all while in the middle of Bed-Stuy. Showing up for folks in my community and building community with them is deeply important to me. Some of my favorite memories in life are connected to food and fellowship, so I couldn’t pass up the opportunity to create another memory with friends. Juneteenth for me is a time for the Black community to come together in corporal celebration. A day of pure celebration, to honor those who came before us, to reflect on our past and hold one another close as we face the future together. Moreover, I recognize that Juneteenth has recently come into the national conversation as a holiday. The history and importance of this celebration is still being learned by many across the country, and people are still navigating how to participate and acknowledge this day. While it may seem small, it means a lot to me that MongoDB not only held space for employees to share their Junteenth traditions and experiences but also a reminder that my whole self, including my Blackness and my Southern-ness, has a place and is respected and welcomed at MongoDB. Nia Brown , Workplace Coordinator My partner and I are enjoying our meal at Simone’s, Black-woman-owned Caribbean Restaurant in New York. This Juneteenth, while my partner and I were in Toronto, we looked up Black-owned restaurants to support. I was pleased to find there were many options. We support Black-owned businesses year round, but doing it on Juneteenth made it that much more poignant, knowing the Black dollar only lasts six hours in the Black community compared to 28 days in Asian communities, 19 days in Jewish communities, and 17 days in white communities. It’s important we educate ourselves and one another to help build up the Black community, especially on Juneteenth. Knowing the history of this holiday makes me now, more than ever, want to spread knowledge so that we are never left in the dark again. Spending Time With Family Lakuan Smith , Manager of Inclusion This Juneteenth weekend a few of my family members and I rented a house so that we could spend time together and share knowledge on the things we are doing in our lives to improve our physical, mental, and financial wellness. I chose to participate in these activities because one of my takeaways from Juneteenth is the importance of spreading knowledge and information to improve lives. I think about the news that was shared on June 19, 1865, and how important it was for those African Americans to receive the knowledge of freedom. I am also fortunate enough to spread knowledge beyond Juneteenth weekend as a manager of inclusion at MongoDB. My day-to-day consists of expanding perspectives and creating initiatives that improve the professional lives of under-represented communities. At MongoDB, I don't have to do it alone. With the help of company leadership and our affinity groups, things are changing for the better. Members of my family and I gathered together for a weekend get away and graduation party. Bryan Spears , Senior Technical Recruiter Posing with my dad, best friend, and his father after playing a round at Hanover Golf Course in NJ To celebrate Juneteenth, I hit the golf course with some family and friends. At a very young age, my dad got me into golf with my own set of clubs. As he has gotten older, it is becoming less frequent that my pops gets on the course with me, and it had been over five years since his last time swinging a club. To my surprise, but probably not his, my dad was still hitting the ball better than me at the age of 79. He might not swing the club with the same speed, but more often than not, he was hitting clean shots straight down the course! Overall, I really enjoyed being able to spend time with family and friends to celebrate Juneteenth. Thinking about all the things I was able to do with my loved ones really makes me grateful for the sacrifices made by our ancestors so that we could live in a more equal society. My dad was in his late teens and early 20s during the Civil Rights movement; he married my mom in 1969, just two years after the 1967 United States Supreme Court decision Loving v. Virginia, which struck down all anti-miscegenation (racial segregation at the level of marriage and intimate relationships) laws the remaining in 16 U.S. states. Just being around him is like walking with history, and while I appreciate the freedom that we have today, there is still a lot of work to do in order to combat systemic racism and oppression in the U.S. and worldwide. My hope is that while we all enjoy these holidays with loved ones or use the day off to relax and rejuvenate, we also take some time to reflect and educate ourselves so we can continue to take action. Educating and Reflecting Courtney Turner , Campus Recruiter My Juneteenth weekend was spent reflecting on the past, embracing the present, and encouraging others to have a better understanding of the holiday and the injustices that we are still faced with today. I spoke virtually to a group of young African Americans about the struggle and process of getting to what we now call Juneteenth. I also spent time with my friends at an annual Juneteenth festival, enjoying their company and reflecting on what our community has accomplished and the work still ahead of us all. Spending time with friends and speaking to youth gave me the opportunity to appreciate my culture, enjoy fellowship with other African Americans, and most importantly, do my job educating others on black culture. My desire is that, as we educate ourselves about Juneteenth, we realize that being “free” or “equal” goes beyond signing an order or taking the day off. We can’t celebrate Juneteenth but not teach the history of it in our classes; we can’t celebrate but not encourage justice and equality for all. My desire is that we celebrate with a new understanding and purpose for the holiday. MongoDB is committed to building a culture of inclusion where employees of different origins, backgrounds, and experiences feel valued and heard. Learn more about Diversity & Inclusion at MongoDB .
June 28, 2022

Next →
Retrieval Augmented Generation for Claim Processing: Combining MongoDB Atlas Vector Search and Large Language Models
Following up on our previous blog, AI, Vectors, and the Future of Claims Processing: Why Insurance Needs to Understand The Power of Vector Databases , we’ll pick up the conversation right where we left it. We discussed extensively how Atlas Vector Search can benefit the claim process in insurance and briefly covered Retrieval Augmented Generation (RAG) and Large Language Models (LLMs). MongoDB.local NYC Join us in person on May 2, 2024 for our keynote address, announcements, and technical sessions to help you build and deploy mission-critical applications at scale. Use Code Web50 for 50% off your ticket! Learn More One of the biggest challenges for claim adjusters is pulling and aggregating information from disparate systems and diverse data formats. PDFs of policy guidelines might be stored in a content-sharing platform, customer information locked in a legacy CRM, and claim-related pictures and voice reports in yet another tool. All of this data is not just fragmented across siloed sources and hard to find but also in formats that have been historically nearly impossible to index with traditional methods. Over the years, insurance companies have accumulated terabytes of unstructured data in their data stores but have failed to capitalize on the possibility of accessing and leveraging it to uncover business insights, deliver better customer experiences, and streamline operations. Some of our customers even admit they’re not fully aware of all the data in their archives. There’s a tremendous opportunity to leverage this unstructured data to benefit the insurer and its customers. Our image search post covered part of the solution to these challenges, opening the door to working more easily with unstructured data. RAG takes it a step further, integrating Atlas Vector Search and LLMs, thus allowing insurers to go beyond the limitations of baseline foundational models, making them context-aware by feeding them proprietary data. Figure 1 shows how the interaction works in practice: through a chat prompt, we can ask questions to the system, and the LLM returns answers to the user and shows what references it used to retrieve the information contained in the response. Great! We’ve got a nice UI, but how can we build an RAG application? Let’s open the hood and see what’s in it! Figure 1: UI of the claim adjuster RAG-powered chatbot Architecture and flow Before we start building our application, we need to ensure that our data is easily accessible and in one secure place. Operational Data Layers (ODLs) are the recommended pattern for wrangling data to create single views. This post walks the reader through the process of modernizing insurance data models with Relational Migrator, helping insurers migrate off legacy systems to create ODLs. Once the data is organized in our MongoDB collections and ready to be consumed, we can start architecting our solution. Building upon the schema developed in the image search post , we augment our documents by adding a few fields that will allow adjusters to ask more complex questions about the data and solve harder business challenges, such as resolving a claim in a fraction of the time with increased accuracy. Figure 2 shows the resulting document with two highlighted fields, “claimDescription” and its vector representation, “claimDescriptionEmbedding” . We can now create a Vector Search index on this array, a key step to facilitate retrieving the information fed to the LLM. Figure 2: document schema of the claim collection, the highlighted fields are used to retrieve the data that will be passed as context to the LLM Having prepared our data, building the RAG interaction is straightforward; refer to this GitHub repository for the implementation details. Here, we’ll just discuss the high-level architecture and the data flow, as shown in Figure 3 below: The user enters the prompt, a question in natural language. The prompt is vectorized and sent to Atlas Vector Search; similar documents are retrieved. The prompt and the retrieved documents are passed to the LLM as context. The LLM produces an answer to the user (in natural language), considering the context and the prompt. Figure 3: RAG architecture and interaction flow It is important to note how the semantics of the question are preserved throughout the different steps. The reference to “adverse weather” related accidents in the prompt is captured and passed to Atlas Vector Search, which surfaces claim documents whose claim description relates to similar concepts (e.g., rain) without needing to mention them explicitly. Finally, the LLM consumes the relevant documents to produce a context-aware question referencing rain, hail, and fire, as we’d expect based on the user's initial question. So what? To sum it all up, what’s the benefit of combining Atlas Vector Search and LLMs in a Claim Processing RAG application? Speed and accuracy: Having the data centrally organized and ready to be consumed by LLMs, adjusters can find all the necessary information in a fraction of the time. Flexibility: LLMs can answer a wide spectrum of questions, meaning applications require less upfront system design. There is no need to build custom APIs for each piece of information you’re trying to retrieve; just ask the LLM to do it for you. Natural interaction: Applications can be interrogated in plain English without programming skills or system training. Data accessibility: Insurers can finally leverage and explore unstructured data that was previously hard to access. Not just claim processing The same data model and architecture can serve additional personas and use cases within the organization: Customer Service: Operators can quickly pull customer data and answer complex questions without navigating different systems. For example, “Summarize this customer's past interactions,” “What coverages does this customer have?” or “What coverages can I recommend to this customer?” Customer self-service: Simplify your members’ experience by enabling them to ask questions themselves. For example, “My apartment is flooded. Am I covered?” or “How long do windshield repairs take on average?” Underwriting: Underwriters can quickly aggregate and summarize information, providing quotes in a fraction of the time. For example, “Summarize this customer claim history.” “I Am renewing a customer policy. What are the customer's current coverages? Pull everything related to the policy entity/customer. I need to get baseline info. Find relevant underwriting guidelines.” If you would like to discover more about Converged AI and Application Data Stores with MongoDB, take a look at the following resources: RAG for claim processing GitHub repository From Relational Databases to AI: An Insurance Data Modernization Journey Modernize your insurance data models with MongoDB and Relational Migrator
April 18, 2024