Transforms customer experience, fights fraud, and meets the demands of the GDPR with MongoDB Atlas and Apache Kafka running on AWS
In a world where standards and speed in online retail are getting ever higher, one retailer, AO.com has consistently been able to differentiate itself around its core focus: customer experience.
At the beginning of 2017 it was clear there was a big opportunity to use customer data more effectively, to drive:
- Continued improvement to the customer experience
- Faster identification of fraud
- Compliance with the EU’s new GDPR personal privacy regulations
AO responded by starting a new team, giving them the mandate to build a 360-degree single view platform of all its customer data.
To ensure they could move quickly and keep the team focused on business goals, the team chose to use MongoDB Atlas in the cloud. We spoke with Jon Vines, Software Development Team Lead at AO.com, about the experience of building the single customer view application, his development philosophy, and the impact it’s having at AO.
Can you start by telling us about AO.com?
AO.com is one of the UK’s leading online electrical retailers. At AO, what really drives us and makes us special is how obsessed we are with the customer’s experience. We make sure that our employees are empowered to make the call on what’s best for each and every customer, that means no scripts or rules in the contact centre, we just do what we think is right. Our guide has always been to ask whether our mum would be proud of the decisions we make every day and whether we feel we would have treated our own Nan in the same way.
That philosophy extends right through to the core technology decisions we make. For instance, the third of our three core business model pillars is infrastructure. We know it’s incredibly important to have systems that are scalable and extendable while making sure we get all the benefit of operational gearing and pace from our investments and long-term growth.
With quality infrastructure as such a key part of AO’s strategy, what was your team trying to achieve?
We were tasked with pulling together the many sources of customer information within AO, to build a single, holistic view of each of our customers. Our data is spread across many different departments, each using their own technology.
The ultimate goal is to deliver one source of truth for all customer data – to drive a host of new and enhanced applications and business processes. This enables our staff, with the appropriate permissions, to access all the useful data we have at the company, from one single place, and all from one easy to consume operational data layer.
Can you tell us a little bit about what apps will consume the single view?
There are three core apps we are serving today:
- Call center: For us, it is all about the customer and relentlessly striving to make our customers happy. Evaluating our contact centre systems showed us that we could enhance our customer journey by exposing more data points that were not currently available to our agents. This allows us to provide a better level of customer experience.
- Fraud: Anything in the fraud grey area gets passed to our fraud team for the personal touch. As our company grows, we’re constantly looking for ways to be more efficient and effective. This is where the single view comes in. The high-throughput, low-latency capabilities and the aggregation of multiple disparate data sources makes it the perfect vehicle to provide additional decision support and anomaly detection for our fraud team.
- GDPR: We need to be able to tell customers what personal data we store about them. Our marketing teams need to be able to see customer’s preferences in relation to communications with us. The single view makes all of this much easier.
This is just the start. There are many more projects that are under development.
So how did you get started with the single view project?
The team was formed in May 2017, and our first task was to identify source data and define the domain we were operating in. We have a lot of historical data assets that need to be blended with real-time data. This meant working with multiple instances of Microsoft SQL Server, and various data repositories and message queues hosted on Amazon Web Services (AWS), including SQS. We had to figure out a way to extract that data cleanly, without impacting source systems.
We spent several months cataloging our data assets, before turning to prototyping and technology selection. We started development in October 2017 and went into production just three months later in January 2018. The key to this development velocity was the underlying technology we selected to power the single view platform.
What are you using to move data from your source systems?
As a business we were already running in AWS, so we first looked at Kinesis, but decided on using Apache Kafka and Confluent Open Source. We can use Kafka’s Connect API to extract data via the Change Data Capture streams on existing data sources, with Kafka streaming and transforming the source data into our single view data model. This allows us to extract data without creating dependencies on other teams who we would otherwise have to rely upon to publish this data, or give us access to their source systems.
Once the data is in Kafka, it opens up a multitude of potential downstream applications. The single customer view is just the first of these. We can also use the oplog in MongoDB to extract data into Kafka, which allows us to decouple our microservices architecture in a very natural way.
How about the database layer?
It was clear that legacy relational databases would never give the schema flexibility we needed, so we explored more modern database options.
We explored a number of different non-tabular databases before choosing MongoDB. It soon became clear that MongoDB’s document model was the best choice for us. It supports rich customer objects that we can further enrich at any stage without expensive schema migrations. And strongly typed temporal data formats are valuable for our app. In terms of overall functionality, for our use case, MongoDB was the clear choice to help us achieve our goals.
We also found MongoDB’s 10-step methodology guide to building a single view – it was clear many customers had done this before with MongoDB – and they offered a unique blend of expertise, processes, and technology to help us deliver.
Taking all of this onboard, it took less than a day to move our data and code from an early prototype system to MongoDB, and from there, we didn’t look back.
Can you tell us why you decided to run on MongoDB Atlas?
We are a small DevOps team with three developers and a business analyst. We own the single view platform end to end, from prototyping and development through to deployment, security, and ongoing operations. We need to focus on meeting the needs of the business, not on managing the database, and Atlas gives us that. All of the operational best practices are baked in. We don’t have to worry about database operations, monitoring or backups. Atlas automates all of those services, which fits perfectly with our developer-first ethos at AO.com.
How is MongoDB Atlas performing for you?
Atlas has been rock solid since launch. We are running a replica set in AWS Ireland, with the ability to scale up and out on-demand, with no application downtime. This is especially valuable as we are currently managing millions of orders and customer objects in the single view platform, and are now planning to add clickstream data, which will significantly grow the data under management.
We can serve complex queries in around 10ms and writes at 5ms. The Atlas alerting is great – we get instant notifications if anything needs attention, and we can add our own custom alerts as needed. The Performance Advisor has been hugely valuable to the team – any unindexed queries are captured and alerted to us, so we can add a new index in minutes if we need to. Collectively, these features mean we can react in real time to anything that happens in our production environment.
We are also planning to start using the MongoDB Connector for BI to visualize metrics from all customer data that is onboarded to the single view.
Can you tell us more about the development philosophy at AO.com?
Really it’s all around the theme of getting products delivered for the business. Our philosophy subscribes to ‘The Three Ways’:
- Flow from left to right – we are focused on getting ideas into production as quickly as possible. We are reducing dependencies between teams, investing in continuous integration and delivery tools, and making smart technology choices to improve developer productivity. Our goal is to go from code commit to production in less than 30 minutes.
- Amplifying feedback loops – building processes so we can continually work with the business to identify what’s working and what’s not.
- Continuous improvement – validating hypotheses early, using feedback loops and making changes. We are moving towards chaos engineering. MongoDB Atlas has been a great asset on that as it allows us to trigger failover experiments and other conditions to improve code and platform resilience.
If that sounds like the sort of development team you want to work with, then we are hiring!
Finally, how are you measuring the impact of the single view platform on the business?
It’s still the early stages and we see this having a number of major benefits to the business. In the short term we’re focusing on a couple of very concrete metrics:
- Call centre experience – we are able to reduce the number of times a customer needs to call as our agents can access all of the data they need immediately. We’re also working to help reduce the average call handling time by up to 40%.
- Fraud – we’re helping to improve fraud detection lead time and be proactive in monitoring potential fraud trends. We’re also lowering the rejection rate and reducing the number of false positives. All of which frees up resource and, crucially, improves customer experience.
- GDPR compliance – obviously there are many teams involved in making sure we were ready for this, but ensuring we have a platform in place that can easily answer the questions from the legal department has been crucial.
Of course, this is all just the beginning. Next, we’ll be looking at data analysis, exposing the single view data as a service to more applications, and automating more business processes.
Jon, thanks for sharing your story with the MongoDB community
And if you want to find out more and see Jon present at MongoDB Europe, then reserve your tickets here. Use the code AO20 to get 20% off
To get started with your own single view project, download our 10-Step Methodology Guide