DWP's agile response to COVID-19 saw it scale Universal Credit 10x with MongoDB to meet demand.

INTRODUCTION

One critical platform that distributes welfare to UK citizens

The Department for Work and Pensions (DWP) is the largest department in the UK government and is responsible for distributing £191bn in welfare, pensions and child maintenance. It serves over 20 million citizens each year and its critical monthly payments help these people with living costs, childcare needs and employment opportunities - all when they need it most.

This welfare system for working-age people is known as Universal Credit (UC) and it went live in 2014. Six separate claims then were combined into one single digital platform simplifying what had become a complicated and complex set of independent systems. The online platform had made it much easier for citizens to obtain support and manage their options.

Led by the DWP Digital team, the UC platform now manages more than 200m enquiries and seven million claims every year. As for many services across the UK government, evolving with digital and moving online had proved to be successful for the Department.

Yet, on the 23rd March 2020, the Universal Credit programme experienced an enormous increase in demand overnight. As the COVID-19 pandemic hit, the UK was put into a national lockdown and entire industries ground to a halt. It left millions unable to work or unemployed. The system had to stay online to protect the most vulnerable citizens, yet it would need to be refined and re-scaled once again.

THE CHALLENGE

The UC platform had to pivot, scale, and adjust...and fast

The IT infrastructure which underpins Universal Credit has used MongoDB as its data store from day one. It was on the recommendation of the Government Digital Service (GDS) which was promoting that Departments should use the most modern of technologies.

DWP used MongoDB as its underlying core database in its technology stack. It was used to store, manage and analyze troves of data and map that information into its microservices environment. This microservices architecture helped break down the UC application into small autonomous services for better management.

MongoDB’s simple intuitive document model, adaptable schema design, clustering with resilience across availability zones and low cost of entry were all benefits to be had. For DWP Digital, the real crux was that MongoDB offered its development team a new and more productive way to build applications, handle highly diverse data types and manage data efficiently at scale. Today, DWP Digital houses all data types you’d expect (e.g., numbers, strings, binary data, arrays) without requiring the team to predefine a schema. It leaves each microservice to evolve as needed, without breaking the schema and disrupting the service.

However, no-one could have predicted the stress this MongoDB implementation would be placed under by the events of 2020. With COVID-19 spreading rapidly throughout the UK, many citizens were left looking for government support. Commercial markets trembled as everyone was ordered to work from home. Retail, restaurants, hotels, gyms and other services were to close until further notice. Unsurprisingly, this drove a huge spike in traffic to UC’s online services as hundreds of thousands of people tried to assess what assistance was available to them.

“We have solid predictions about normal traffic but we also perform proactive performance testing to resolve any future, potential issues,” explained Tom Padgham, Deputy Director of Engineering, DWP Digital. “March 2020 took us by surprise - and that’s an understatement.”

The concern was that with traffic at more than 10 times the usual volume, the system might fail, leading to thousands of anxious citizens in a time of great distress. Universal Credit as a platform had to pivot, scale and adjust its online services to make sure that people were paid on time and in full.

“For us all, UC failure has real-life consequences and that can mean the difference between paying rent or feeding the family. We just couldn’t let that happen,” added Patrick Downey, Head of Platforms, DWP.

THE SOLUTION

MongoDB steadily scaled to 110TB of capacity across eight different production clusters

As claims flooded in, the Universal Credit platform had to expand to eight MongoDB production clusters to scale the programme. To increase resilience further, each of those clusters had five nodes distributed over three AWS availability zones.

DWP Digital worked with MongoDB to maintain online services during the biggest welfare crisis since the Second World War.

MongoDB's distributed architecture meant the DWP Digital team was able to scale up individual nodes in a rolling fashion, ensuring that while scaling was in progress, the operations of each cluster were never impacted. It meant the team had continued high availability of its data and consistent operational performance. This agility ensured UC applications and their services had the data they needed to function and remain online.

This achievement was made even more impressive by the fact that the team were in a self-managed environment, with limited capabilities around performance monitoring and observability - yet MongoDB still handled the scale without any downtime. During those frantic weeks, the busiest clusters handled more than 15,000 requests per second while the overall system grew to 110TB of uncompressed claim-related data, 5TB of indices, and 6TB of memory.

These numbers and the drastic increase in claims came close to severely impacting operations. The flexibility of MongoDB and its distributed architecture allowed DWP to carry out 76 urgent changes to the entire Universal Credit application stack during the first two months of the pandemic, compared to just six in the previous two months.

“We quickly increased application capacity by adding web and application servers, increased database capacity, moved one MongoDB instance to its own dedicated cluster and boosted compute and memory for two other MongoDB clusters,” Downey added. “Thankfully, that enabled us to handle the traffic that was consistently much higher than average.”

THE RESULTS

Ready to handle 100k claims in a single day whilst maintaining on-time payments

DWP Digital and MongoDB guaranteed that the UC service was always available despite the huge volume of requests. It ensured that vulnerable citizens could access the information and welfare they needed in a timely manner throughout the lockdown months.

“At the peak, we managed to handle 100k new claims and over 2.2 million claim enquiries in a single day and keep our on-time payment rates over 88%, which is pretty phenomenal,” explained Padgham.

“Agile development, multi-disciplinary teams, rapid iteration and weekly releases have all helped us react seamlessly to changes in demand,” Padgham added. “MongoDB has been able to cope with that all the way through and gave us the agility and flexibility we so needed.”

The fact the systems held up under pressure is further evidence that agile delivery and the approach the team has taken is effective. They are now in the process of migrating to a managed database environment with MongoDB Atlas. This migration will help reduce costs and support efforts further, freeing up the team to focus on building applications rather than managing back-end infrastructure.

“I became much more interested in MongoDB Atlas as we scaled in order to take away the stress of worrying about infrastructure, security and resilience. After all, we're here to provide benefits to a nation not to run database clusters,” concludes Downey.

“MongoDB Atlas will give us the agility, scale and speed of delivery, which sometimes you don't get from something you manage yourself. In the future, I'm not going to need to understand the intricacies of configuring and scaling database clusters, because I'll be confident that MongoDB Atlas has my back on that.”

With over five million citizens still claiming welfare through the UC service and as Britain enters its deepest recession since records began, the service the programme provides to the nation will remain critical for months, if not years to come. For the team at DWP Digital, their focus remains on making sure people get paid and have access to the support they need, when they need it.

For more information on DWP Digital and its open opportunities, visit DWP Digital Careers.