MongoDB Microservices Help HM Revenue & Customs Cut Wait Times in Half and Save £8.2m in Operational Costs

Mat Keep
March 30, 2016 | Updated: December 8, 2016
#Business

Taxes and death. The two things in life we can’t escape. So what on god’s green earth could ever compel someone to tweet positive comments about a tax self assessment process? Well, let me explain.

HM Revenue & Customs, the UK’s tax body, has been a leading agency in the government’s digital transformation programme. The aim of which is “building digital services that are simpler, clearer and faster to use”. It’s fair to say that tax services in the UK, were not widely regarded as being simple, clear or fast.

To create the underlying infrastructure that could power that transformation, HMRC, with its partner Equal Experts, built a network of microservices that use MongoDB’s scalability, data model flexibility and ease of development. This project also ushered in a move to agile development and a default preference for open source technology, like Docker containers.

Earlier this month Mike Potter, Digital Transformation Director at HM Revenue & Customs, revealed the full impact of those transformed services can have:

“We received an unprecedented number of Self Assessment returns online this year - 89% or 9.24m customers. This is fantastic and I wanted to share with you some other numbers I’m excited by.”

The online form replaced a paper process that users had accused of being difficult and complex, which led to a high volume of phone calls to the support staff. That’s now changed. Mike continued:

“So how did the 2016 peak compare to 2015? Well, this year we received:

  • 480k more returns online
  • 300k less paper returns - down 21%
  • 41k fewer calls - down 7% - and, in addition, a further 88k customers got the help they needed through webchat instead of having to phone us
  • And we halved phone waiting times, down to just over 5 minutes.”

These are not minor improvements. Waiting times cut in half! That means more time and dramatically less stress in the lives of UK citizens. Mike also highlighted how this had been positively received on social media:

How many times have you seen someone tweet their affection for a tax service, right after filling out a self assessment?

The Digital Transformation director also looked at how the Multi-channel Digital Tax Platform or MDTP had facilitated these dramatic changes. He explained that “[MDTP] hosts the digital tax accounts at the heart of our digital future, plus all of our new online services”.

MDTP has launched 28 services, handled 55 million visits and, get this, saved £8.5 million in operating costs. That’s right, the services are vastly improved and the tax papers seeing savings in excess of, for our American friends, $12 million USD.

These impressive numbers are the result of many people doing incredible work. One part of that is the microservice architecture that supports HMRC’s digital transformation. With the help of partner Equal Experts, HMRC has used modern technologies such as MongoDB and Docker to make services more agile, flexible and scalable. Before the microservices transformation, HMRC were pushing two releases a year. Now it’s 50 releases a week.

Simple, clear and fast. And it saves the taxpayers millions. Digital transformation at HMRC is a demonstration of what’s possible when people take a completely new look at how to solve a problem. As Ben Franklin said: we’ll never escape tax or death. But it’s a relief to know at least one of life’s constants is getting a little less painful.

Find out how microservices can help your business. Read more about enabling microservices.
Enabling Microservices: Containers & Orchestration Explained

About the Author - Mat Keep

Mat is a director within the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.

← Previous

Secure MongoDB with X.509 Authentication

Disclaimer - the following article is intended for a test environment. Overview In this tutorial, I will be describing the detailed process of setting up X.509 based authentication, both for cluster inter-member authentication as well as for client authentication, using a local CA (Certificate Authority). X.509 is one of the multiple authentication mechanisms supported by MongoDB. An X.509 certificate is a digital certificate that uses the widely accepted international X.509 public key infrastructure (PKI) standard to verify that a public key, presented by a client or another member of the cluster, belongs to that said client or member. One of the main benefits compared to conventional password based authentication is it’s more secure in a sense that each machine would need a dedicated key to join the cluster. So stealing an existing key from another machine isn't going to be very helpful for those with an evil agenda. In MongoDB, we need to understand the distinction between member authentication and client authentication. MongoDB is a distributed database and deployments almost always consist of multiple mongod or mongos processes running on multiple machines. Member authentication refers to the fact that these machines need to verify each other to ensure a node attempting to replicate data is indeed part of the current cluster. On the other hand, client authentication refers to those MongoDB clients, including mongo shell, export/import tools and MongoDB drivers. Below is a deployment of standard 3 node replica set and a client. To enable X.509 certificate based authentication, essentially you will need the following set of certificates: 3x server certificates, one for each MongoDB member 1x client certificate for one client 1x certificate for Root CA and 1x certificate for Signing CA All the server certificates and client certificates must be signed by same CA. The bulk part of the work for setting up X.509 authentication would be to create these certificates. The creation of Root CA and Signing CA and the relevant parts of the signing process is a one time effort, while the creation of the server/client certificates are as needed. For simplicity, I am going to use one machine and I will run multiple mongods on different ports for this exercise. I also put the steps into an executable script if you want to jump to the end. You can download and run the script, and you will have a test setup just like the diagram above in no time. This way you can quickly get things working, boost your confidence, and then come back to study the details. Preparation One linux box, I tested on RedHat 7 and Ubuntu 14.04 but it should work with other flavors as well. Download MongoDB Enterprise 3.2 Install MongoDB following instructions documented here Download the demo shell script and save to a clean directory Running the script The script will setup a test replica set on your machine and configure X.509 authentication mechanism for that replica set. Before you run the script, double check: OpenSSL is installed (typically comes with most Linux distributions) Make sure mongod/mongo are in the executable path Make sure no mongod is already running on 27017 port, or change the port numbers in the shell script To run the script, go to the directory that contains the script and execute the following: # chmod +x setup-x509.sh # ./setup-x509.sh If everything goes smoothly, you should have a 3 nodes replica set running with X.509 as member auth. Then in the current directory, you may connect to the primary node with a newly generated client certificate client1.pem: mongo --ssl --sslPEMKeyFile client1.pem --sslCAFile root-ca.pem --sslAllowInvalidHostnames Note above step just allows you to connect to the MongoDB shell. You will not have any permission at this point. To do anything meaningful, you need to authenticate yourself using following command: $mongo ... > db.getSiblingDB("$external").auth( { mechanism: "MONGODB-X509", user: "CN=client1,OU=MyClients,O=MongoDB China,L=Shenzhen,ST=GD,C=CN" } ); > db.test.find() If you are able to execute the last find statement, congratulations, the X.509 authentication is working! Now it's time to perform an anatomy of the script to understand what are the key steps involved in setting up the X.509 authentication mechanism. Main Parts of the Script Initialization Create local CA and signing keys Generate and sign server certificates for member authentication Generate and sign client certificates for client authentication Start MongoDB cluster in non-auth mode Setup replica set and initial users Restart MongoDB replica set in X.509 mode using server certificates 0. Variable initialization First initialize some variables. Feel free to modify the values as appropriate. dn_prefix="/C=CN/ST=GD/L=Shenzhen/O=MongoDB China" ou_member="MyServers" ou_client="MyClients" mongodb_server_hosts=( "server1" "server2" "server3" ) mongodb_client_hosts=( "client1" "client2" ) mongodb_port=27017 Here dn_prefix will be used to construct the full DN name for each of the certificate. ou_member is used to have a different OU than the client certificates. Client certificates uses ou_client in its OU name. mongodb_server_hosts should list the hostname (FQDN) for all the MongoDB servers while mongodb_client_hosts should list the hostnames for all of the client machines. For a clean start, let’s kill the running mongods and clean up the working directory (note: don’t use -9 to kill mongod per the manual ): kill $(ps -ef | grep mongod | grep set509 | awk '{print $2}') mkdir -p db 1. Create local root CA A root CA (Certificate Authority) is at the top of the certificate chain. This is the ultimate source of the trust. Ideally a third party CA should be used. However in the case of an isolated network (very typical in large enterprise environment), or for testing purpose, we need to use local CA to test the functionality. echo "##### STEP 1: Generate root CA " openssl genrsa -out root-ca.key 2048 # !!! In production you will want to password protect the keys # openssl genrsa -aes256 -out root-ca.key 2048 openssl req -new -x509 -days 3650 -key root-ca.key -out root-ca.crt -subj "$dn_prefix/CN=ROOTCA" mkdir -p RootCA/ca.db.certs echo "01" >> RootCA/ca.db.serial touch RootCA/ca.db.index echo $RANDOM >> RootCA/ca.db.rand mv root-ca* RootCA/ Above we first created a key pair root-ca.key with AES256 encryption and 2048 bits strength. Then using openssl req command to generate a self-signed certificate with a validity of 3650 days. One thing to call out here is the argument -x509 which tells openssl to self sign the certificate instead of generating a signing request (as what we will do below). The output is a crt file, a certificate file that contains the public key of the root CA. 2. Create CA config A CA config file is used to provide some default settings during the certificate signing process, such as the directories to store the certificates etc. You may change the defaults in root-ca.cfg file after it is generated or simply change them within the script. echo "##### STEP 2: Create CA config" cat >> root-ca.cfg ${host}.pem done This script is in a for loop to generate multiple certificates. 3 key steps are involved with each certificate: Use openssl genrsa command to create a new key pair Use openssl req command to generate a signing request for the key Use openssl ca command to sign the key and output a certificate, using the Signing CA we created earlier as the signer Notice the variable $ou_member . This signifies the difference between server certificates and client certificates. Server and client certificates must differ in the organization part of the Distinguished Names, or in another word and must differ at least in one of the O, OU, or DC values. 5. Generate and Sign client certificates These certificates are used by clients, such as mongo shell, mongodump, and Java/python/C# drivers to connect to a MongoDB cluster. This step is essentially the same as step 4 except for the use of $ou_client . This will make the combination of the DC/OU/O for these certificates will be different from the server certs above. echo "##### STEP 5: Create client certificates" # Pay attention to the OU part of the subject in "openssl req" command for host in "${mongodb_client_hosts[@]}"; do echo "Generating key for $host" openssl genrsa -out ${host}.key 2048 openssl req -new -days 365 -key ${host}.key -out ${host}.csr -subj "$dn_prefix/OU=$ou_client/CN=${host}" openssl ca -batch -name SigningCA -config root-ca.cfg -out ${host}.crt -infiles ${host}.csr cat ${host}.crt ${host}.key > ${host}.pem done 6. Bring up replicaset in non-auth mode MongoDB does not create a default root/admin user when enabling authentication, and there is no exception with X.509 mode. Instead, the best practice is to create an initial admin user first, then to enable authentication after the admin user has been created. Here we're starting a replica set in non-auth mode. echo "##### STEP 6: Start up replicaset in non-auth mode" mport=$mongodb_port for host in "${mongodb_server_hosts[@]}"; do echo "Starting server $host in non-auth mode" mkdir -p ./db/${host} mongod --replSet set509 --port $mport --dbpath ./db/$host \ --fork --logpath ./db/${host}.log let "mport++" done sleep 3 Now our replica set is up, we need to initialize the replica set and add a user. 7. Initialize replicaset and add initial user When using X.509 client authentication, each client must have a user created in MongoDB and the user must be granted the necessary permissions. The username must be same as the client's DN (Distinguished Name), which can be obtained by running an openssl command: # obtain the subject from the client key: client_subject=`openssl x509 -in client1.pem -inform PEM -subject -nameopt RFC2253 | grep subject | awk '{sub("subject= ",""); print}'` This would return something like: CN=client1,OU=MyClients,O=MongoDB China,L=Shenzhen,ST=GD,C=CN Obviously it's not mandatory to use the openssl command. It's fairly straightforward to deduce the subject string by concatenating the relevant parts of the DN as shown above. Once we have the DN name, let's initialize the replica set and add a user: myhostname=`hostname` cat > setup_auth.js

March 29, 2016
Next →

Five Languages, One Goal: A Developer's Path to Certification Mastery

MongoDB Community Creator Markandey Pathak has become a certified developer in five different programming languages: C#, Java, Node.JS, PHP, and Python. Pursuing multiple certifications equips developers with a diverse skill set, making them invaluable team members. Fluency across different programming languages enables them to foster platform-agnostic solutions and promote adaptability, collaboration, and informed decision-making, which are crucial for success in the global tech landscape. To understand what led Markandey to take on so many certifications while managing a busy and successful career, we spoke with him to gain insights into the challenges and triumphs he faced. What motivated you to pursue certification in multiple programming languages, and how has achieving such a diverse set of skills impacted your career? C was the first programming language I learned, followed by C# and the .NET ecosystem a few years later. Transitioning to a new language like C# after knowing one was straightforward. I then delved into ASP.NET, JAVA, and subsequently PHP. Despite the differing syntax of these languages, I found that fundamental programming concepts remained consistent. This enlightening realization led me to explore JavaScript and, later, Python. Such a diverse skill set made me a go-to resource for many senior leaders seeking insights. This versatility allowed me to transcend categorization based on programming ecosystems in the workplace, evolving my mindset to develop platform-agnostic solutions. I believe in the adage of being a jack of all trades while still mastering one or more. I took on the challenge of discovering MongoDB drivers available for various platforms. I created sample applications to practice basic MongoDB concepts using specific drivers, and soon, everything fell into place effortlessly. What tips or advice would you share with someone who looks up to your achievement and aspires to become a certified developer in multiple languages like C#, Java, Node.JS, PHP, and Python? How can they effectively approach learning and mastering these languages? Before attempting proficiency in MongoDB across multiple languages, it's crucial to prioritize understanding fundamental concepts such as data modeling practices, CRUD operations, and indexes. Mastering MongoDB's shell, MongoSh, is essential to grasp the workings of MongoDB's read and write operations. Following this, individuals should select a programming environment they're most adept in and practice executing MongoDB operations within that ecosystem. Constructing a personal project can aid in practically observing various MongoDB concepts in action. Utilizing resources such as MongoDB Certification Learning Paths , practice tests, and MongoDB Documentation is vital for excelling in certification exams. Additionally, it's advisable to undertake the initial certification in the programming language one feels most comfortable with. Reflection is key; saving or emailing exam scores enables individuals to identify areas needing improvement for future attempts. With proficiency in C#, Java, Node.JS, PHP, and Python, how do you perceive the role of versatility in today's tech industry, especially regarding job opportunities and project flexibility? Programming languages, very much like spoken languages, are merely a medium. The most important thing is knowing what to say. The tech industry depends on problems, and developers seek solutions to them. Once they have a solution, programming languages help make those solutions a reality. It’s not hard to learn different programming languages or even to master them. Knowing the basics of different programming ecosystems can give developers an edge regarding job opportunities. It makes them flexible and enables them to make crucial and informed decisions in choosing the correct tech stack or defining good architecture for solutions. In your experience, how does fluency in multiple languages enhance collaboration and innovation within development teams, particularly in today's globalized tech landscape? Fluency or even practical awareness about programming languages or ecosystems promotes versatility in problem-solving, facilitates cross-functional collaboration, supports agile development, enables integration with legacy systems, fosters global collaboration, reduces dependency, and empowers informed decision-making, all of which are crucial for staying competitive in today's globalized tech landscape. As a MongoDB Community Creator, how do you leverage your expertise in these five languages to contribute to and engage with the broader tech community? What advice would you offer aspiring developers seeking to expand their skill set? I aim to open-source my MongoDB-focused projects across various ecosystems, accompanied by detailed articles outlining their construction. Since these projects were designed with exams in mind, they serve as skill-testing tools for developers and comprehensive guides to the various components comprising certification exams. I advocate for developers to choose a favorite language and compare others to it, as this approach facilitates a quicker and more efficient understanding of concepts. Relating new information to familiar concepts makes learning easier and more effective. The MongoDB Community Advocacy Program is a vibrant global community designed for MongoDB enthusiasts who are passionate about advocating for the platform. Our Community Creators Program welcomes members of all skill levels eager to deepen their involvement in advancing MongoDB's community and technology. We empower our members to expand their expertise, visibility, and leadership by actively engaging with and advocating for MongoDB technologies among users worldwide. Join us and amplify your impact within the MongoDB community! Elevate your career with MongoDB University 's 1,000+ learning assets. Access free courses and hands-on labs, and earn certifications to boost your skills and stand out in tech.

April 24, 2024