FICO is more than just the FICO credit score. Founded in 1956, FICO also offers analytics applications for customer acquisition, service, and security, plus tools for decision management.
One of those applications is the Falcon Assurance Navigator (FAN), a fraud detection system that monitors purchasing and expenses through the full procurement to pay cycle. Consider an expense report: the entities involved include the reporter, the approver, the vendor, the department or business unit, the expense line items, and more. A single report has multiple line items, where each line may be broken into different expense codes, different budget sources, and so on. This translates into a complicated data model that can be nested 6 or 7 layers deep – a great match for MongoDB’s document model, but quite hard to represent in the tabular model of relational databases.
FAN Architecture Overview
The fraud detection engine consists of a series of microservices that operate on transactions in queues that are persisted in MongoDB:
- Each transaction arrives in a receiver service, which places it into a queue.
- An attachment processor service checks for an attachment; if one exists, it sends it to an OCR service and stores the transaction enriched with the OCR data.
- A context creator service analyzes it and associates it with any past transactions that are related to it.
- A decision execution engine runs the rules that have been set up by the client and identifies violations.
- One or more analytics engines review transactions and flag outliers.
- Now decorated with a score, the transaction goes to a case manager service, which decides whether to create a case for human follow-up based on any identified issues.
- At the same time, a notification manager passes updates on the processing of each transaction back to the client’s expense/procurement system.
To learn more, watch FICO’s presentation at MongoDB World 2018.
How Planable Uses MongoDB Atlas to Help Social Media Teams Move Their Creative Processes Forward
In 2015, Nicolae Gudumac was working at a social media agency managing hundreds of campaigns for clients. However, with each campaign requiring multiple rounds of review, he needed a way to streamline the disjointed feedback loop and bring everyone onto the same page. Along with his co-founders, Xenia Muntean and Vlad Calus, he began building a platform that would streamline this time-consuming process for social media managers, agencies, and their clients. The team created Planable , a platform that simplifies planning, visualizing, and approving social media posts. The tool feels like a live mock-up of the social feed, making it easy for teams to collaborate and give real-time feedback in a familiar format. Prioritizing Developer Velocity From the Beginning As the newly minted CTO, Nicolae decided to use MongoDB’s document data model to help his team innovate quickly to stay ahead of the ever-changing social media landscape. “With social media content, requirements are constantly changing as platforms evolve and introduce new formats. Having a flexible data model has been a boon to our productivity.” He also needed to build a foundation that could scale with them as the business grew. Since Planable is a collaboration tool that needs to keep users synced in real-time, Nicolae built the tool on top of Node.js and websockets. The team harnessed MongoDB’s oplog tailing functionality to send real-time updates to all connected users when relevant data changed and recently started to leverage MongoDB change streams to make this process more simple and scalable. To easily scale their app servers at peak times, Nicolae chose to run Planable on AWS container service. Migrating to MongoDB Atlas For managing their MongoDB deployment, the team started off using Compose.com but were having issues with restoring backups and only had access to a very limited set of configuration options. Compose also charged a premium for upgrading their storage engine to WiredTiger. MongoDB Atlas, with its queryable backup snapshots, automated upgrades, and configuration flexibility — especially the ease at which clusters can be scaled horizontally — looked very appealing. We needed to be able to provide our clients with a platform that is as reliable as we are. Atlas’ native cloud first and scale-out architecture aligned well with our increasing performance demands and usage growth. Nicolae Gudumac, CTO, Planable When Planable was accepted into the MongoDB Startup Accelerator , the timing was right to make the move to MongoDB Atlas. The team had customers all over the world and couldn’t afford any downtime with the migration. They used the Atlas Live Migration Service to move over their data from Compose with no downtime. Currently, the team of 6 is split between engineering and business. With a small engineering team, they’re hyper-focused on product improvements and new features that will drive the business forward. Atlas features such as the Real-Time Performance panel and the Performance Advisor, which monitors clusters for slow queries and automatically suggests indexes to improve performance, allow the team to dedicate more of their attention to application improvements. According to Nicolae, “The Performance Advisor has made indexing the database and optimizing queries a no-brainer." Queryable backups have also helped the team quickly address customer questions and as Nicolae recalls, “It’s saved us numerous times when someone accidentally dropped/updated a few documents and they needed to be restored. We’ve managed to quickly inspect and restore data in just a few clicks.” The team is looking forward to the upcoming release of MongoDB Charts which is currently available in beta. “Charts will enable our marketing and business team to gain insights from our database, without resorting to sophisticated and expensive BI tools” says Nicolae. Planable is bringing thousands onto their platform each month and are well on their way to becoming the default tool for social media collaboration. MongoDB Atlas has helped this fast-growing startup focus on helping social media teams move their process forward and allows Nicolae to give valuable time back to his team. “There’s no doubt in my mind that moving to MongoDB Atlas has increased our team’s productivity.” MongoDB Atlas is the easiest and fastest way to get started with MongoDB. Deploy a free cluster in minutes.
Engineering, Done DIRT Cheap: How an Outdated Data Architecture Becomes a Tax on Innovation
In March 2021, I wrote about The Innovation Tax : the idea that clunky processes and outdated technologies make it harder for engineering teams to produce excellent tech that delights customers. In the months since then, my thinking has evolved even further. I couldn’t have guessed how many technology leaders would immediately recognize these problems in their own organizations and share their own deep frustrations with me. This article puts that evolved thought together with the massive feedback that piece received. It will give you actionable ways to decrease your tax burden — and who wouldn’t want that? The innovation tax, like income tax, is real. Of course, it saps morale (with resulting attrition and churn), but it also has other financial and opportunity costs. Taxed organizations see their pace of innovation suffer as people and resources are locked into maintaining rather than innovating. We named this tax DIRT . Why? Well, it’s rooted in data (D), because it so often springs from the difficulty of using legacy databases to support modern applications that require access to real-time data to create rich user experiences. It affects innovation (I), because your teams have little time to innovate if they’re constantly trying to figure out how to support a complex and rickety architecture. It’s recurring (R), because it’s not as if you pay the tax (T) once and get it over with. Quite the opposite. DIRT makes each new project ever more difficult because it introduces so many components, frameworks, and protocols that need to be managed by different teams of people. In retrospect, it’s clear that technology leaders would recognize this tax and immediately grasp the degree to which it’s caused -- or cured -- by their data architecture. Data is sticky, strategic, heavy, intricate -- and the core of the modern digital company. Modern applications have much more sophisticated data requirements than the applications we were building only 10 years ago. Obviously, there is more data, but it’s more complicated than that: Companies are expected to react more quickly and more cleverly to all of the signals in that data. Legacy technologies, including single-model rigid, inefficient, and hard-to-program relational databases, just don’t cut it. In over 300 CxO conversations I've had since joining MongoDB in 2020, fewer than a handful of CTOs disputed this statement. When your tech stack can’t handle the demands of new applications, engineering teams will often bolt on single-purpose niche databases to do the job (think time series, text, graph, etc.). Then they’ll build a series of pipelines to move data back and forth. And everything will get slow and complicated — and even political. Time to polish up that LinkedIn profile. If this were rare, it wouldn’t be such a big deal. But large enterprises can have hundreds or thousands of applications, each with their own sources of data and their own pipelines. Over time, as data stores and pipelines multiply, an organization’s data architecture starts to look like a plate of spaghetti. Soon you’re operating and maintaining an entire middleware layer of ETL, ELT, and streaming. The variety of technologies, each with their own frameworks, protocols, and sometimes languages, makes it harder for developers to collaborate. It makes it extremely difficult to scale, because every architecture is bespoke and brittle. Developers spend their precious “flow” hours doing integration work instead of building new applications and features that the business needs and customers will love. Enterprise architects often end up spending their time on all the wrong things. It’s clear to me that most customers are ready for a new approach to data architecture. One of the best parts of my job is listening to and learning from other CxOs. Since the pandemic made it impossible to do that in person, MongoDB moved these discussions online, inviting technology leaders to hash out some of their biggest problems 1:1 and in groups with me. In one of those sessions, a CTO commented, “Technical debt should be carried on your CFO's balance sheet.” Even on Zoom, the power of that statement was clear. We also started looking at slide decks about data architecture from some of the best-known venture capital firms. Certainly VCs must position each of their portfolio companies as a critical player in the data architecture of the future. But the overall vision was not compelling. One technology leader said, “When I look at 20 net-new technologies I need to learn, it’s terrifying.” Others commented that just looking at these architecture diagrams was a little off-putting, because they knew their own organization’s data architecture was at least that complicated already. They knew they needed to simplify their data architecture, but more than one admitted to postponing this work -- indefinitely -- because it was just too daunting. I recently met with a major health care company whose executives think it’s just barely possible, but they are bravely diving in anyway, knowing that they must do it and that they’ll learn along the way as they tear down their monoliths. In many cases, the innovation tax manifests as the inability to even consider new technology because the underlying architecture is too complex and difficult to maintain, much less understand and transform. This is why a lot of senior people at enterprise companies are sitting with their fingers in the transformation dike, waiting for retirement -- they think they can’t modernize. It won’t surprise you that we also saw how MongoDB, as a general purpose database able to handle all types of data at speed and scale, could help solve this problem. Let me be clear. I’ve been working on or with databases for my entire 35-year career, and I joined MongoDB for a reason: I believe we can build the database and application-building environment that I’ve wanted to create and use for at least 30 of those years. Our vision of MongoDB goes beyond our namesake database to a broader, more versatile application data platform that allows you to accelerate and simplify how you build any type of application. It represents significant progress toward our larger goal, which remains the same as ever: to make data stunningly easy to work with. We want to see data become an enabler of innovation, not a blocker. And we want to finally allow technology teams to start to untangle their sprawl and get rid of their DIRT. Where to start? It’s good to have a better understanding of just how DIRT might be holding your teams back. Do your developers have trouble collaborating because the development environment is so fragmented? Do schema changes take longer to roll out than the application changes they’re designed to support? Do you have trouble building 360-degree views of your customers? And if so, why? These are all good places to start digging in the DIRT. You might also take a hard look at your applications and data sources, as well as what it would take to move your data onto an application data platform. That could mean identifying the objects in your applications and all the applications that interact with them. You could then assign a complexity score to each one based on attributes such as properties, methods, collections, and attributes. Now take a step back and identify each application that connects to each of those objects and rank it based on how mission-critical it is, how many people rely on it, how many tasks it has to perform, and the complexity of those tasks. Once you have a better handle on all this complexity, you’ll be better positioned to create a plan to move off your legacy systems, perhaps starting with the least complex and least integrated data sources. Of course, your metrics and your mileage will vary, but the point is to start. I don’t pretend any of this is easy. Like many of you, I’ve spent most of my career working on problems just like these. But that also means I know progress when I see it, and the beginning of a way for organizations to start to clean up their DIRT. I’ll be continuing to write more about these challenges and hopefully continue to add some perspective. If you’re curious to learn more about DIRT, you can download our white paper . As always, I’m eager to have you tweet your alignment, lack thereof, or other thoughts at @MarkLovesTech . You can also reach out to me on marklovestech.com , where you will find a compilation of my latest musings related to MongoDB and otherwise.