Nationwide Stays ‘Focused, Fit and Fast’ in Migrating Data from SQL to MongoDB

It’s that determination to be ‘fit, focused and fast,’ which drove Nationwide’s decision to move away from its previous, entirely on-premises platform. “So we could add guardrails and ensure mutual benefit to customers, the idea was to get rid of all the legacy servers, move everything to the cloud, and have a NoSQL database,” says Yadav, “thereby delivering optimum service to the society and its members.”

The team’s original thinking was that it would leverage some of the cloud capabilities of AWS and use MongoDB as part of an audit data store, only – “that was the initial idea,” says Yadav, who at the time was leading the on-premises team of around 12 people. Nationwide’s overdraft database was on-premises - its services consumed by different areas within the building society, and nearing end of service life. In its journey to move away from the on-premises data store completely, the team decided the overdraft database would be a good place to start.

In April 2023, beginning to analyze the changes it would need, the team consulted MongoDB - with whom it had an established relationship - over how it might help the process. Challenges with mitigating end-of-service life risk meant that a key criterion for Nationwide was moving the data with as little human intervention or development effort as possible. “So, we didn’t have much time to focus on the data migration aspect,” says Yadav. “That’s when MongoDB proposed the Preview version of Relational Migrator.”

It would be the first time the product was used in the UK.

Neha Yadav, Lead Engineer, Nationwide

Using the MongoDB Relational Migrator, Yadav and her team undertook a single migration of the overdraft database from SQL to MongoDB Atlas, during scheduled maintenance. “We wanted to maintain the integrity of the environment and reduce the risk of something being missed or unverified with a continuous migration of data,” she explains. “Hence we opted for a snapshot migration.”

Still, it’s fair to say there was some trepidation about the process among the team given the complex nature of migrations in general and being unsure how the migrated data would be validated or ultimately work. The team was able to run test migrations to help validate the production process; creating a similar test environment that would receive multiple requests - then stopping them to see how it would behave. “That way we were pretty sure it should work,” says Yadav. “And to be safe, we went into the office for the live migration.”

The complete migration project, from on-premise API to the cloud, took around six months. The team deployed microservices to their AWS cluster, so having numerous ETL feeds in the on-premises meant they too needed to be moved, with development time for Lambdas included.

“It wasn’t just the data migration,” says Yadav, “it was the feed migration, the API, and the data.”

She adds, “It involved a lot of proof of concept, and shared resources. If I cut out all the lags it would’ve been three months – it was very fast. With Relational Migrator, I would say we saved 50% of our effort.”

One highly-appreciated feature of Relational Migrator was Query Converter, an AI tool to migrate the SQL queries to MongoDB aggregation pipelines. “That was a very helpful feature to re-validate whatever we had understood, and to know that whatever we had written was okay,” says Yadav.

Neha Yadav, Lead Engineer, Nationwide

“Relational Migrator gave us the confidence that we didn’t need to do any development or validation of data manually to move our data from on-premises to cloud,” sums up Yadav. And so positive has been the experience of migrating the overdraft database from SQL to MongoDB Atlas for the team that they’re now in the process of moving other databases to the cloud. “Relational Migrator is very simple to use,” explains Yadav. “Anyone with basic knowledge can use it. That's the best part of it.”

And it’s been a worthwhile move. “In SQL, it used to take at least an hour to generate one of our feeds,” says Yadav, “having moved to MongoDB, the maximum time is two minutes.” While the team has successfully moved data to MongoDB, optimized queries and improved overall performance, by far the biggest win is that they mitigated the risk of end-of-service life. “And we took a very delayed decision to move the overdraft database completely to cloud, which was our horizon three, but we moved it to horizon two. So, my biggest win is delivering it successfully.”

There is one other point of pride, however; “Before this, Relational Migrator hadn’t been used within Nationwide,” says Yadav. “There was a fair amount of risk involved in that, because if it hadn’t gone live the way I wanted there was a lot at stake – I’m proud that the Nationwide IT team made it work, and about this being a first.”

"The successful modernization of an application backend in only 6 months is particularly impressive in the finance world, where major infrastructure upgrades frequently take multiple years to complete,” she adds. “This new agility will help Nationwide maintain their market dominance in the increasingly technology-centric banking environment".

Neha Yadav, Lead Engineer, Nationwide