Unlocking Operational Intelligence from the Data Lake: Part 1 - The Rise of the Data Lake

Mat Keep
May 11, 2016 | Updated: May 23, 2016
#Technical

Series Introduction

Hadoop-based data lakes are enabling enterprises and governments to efficiently capture and analyze unprecedented volumes of data generated from their digital transformation initiatives. But without being able to expose that data to operational applications, users are struggling to maximize returns on their Hadoop investments. The longer it takes to surface insight to operational processes, the less valuable that insight is, and the less competitive you are.

In this 3-part blog series, we’re going to cover:

The rise of the data lake, the role of Hadoop, and the challenges in integrating the data lake with operational applications
In part 2 we’ll cover the critical capabilities you need to evaluate in an operational database for your data lake, and a recommended technology design pattern for integrating the database with the data lake
We’ll wrap up with part 3 covering real world examples and best practices from industry leaders

If you want to get a head-start and learn about all of these topics now, just go ahead and download the Operational Data Lake white paper.

The Rise of the Data Lake

The one thing no business lacks today is data – from streams of sensor readings, to social sentiment, to machine logs, mobile apps, and more. Analysts estimate data volumes growing at 40% per annum, with 90% of it unstructured. Uncovering new insights by collecting and analyzing this data carries the promise of competitive advantage and efficiency savings. However, the traditional Enterprise Data Warehouse (EDW) is straining under the load, overwhelmed by the sheer volume and variety of data pouring into the business, and then being able to store it in a cost-efficient way. As a result many organizations have turned to Hadoop as a centralized repository for this new data, creating what many call a data lake.

With its ability to store data of any structure without a predefined schema and scale-out on commodity hardware, Hadoop provides levels of performance, efficiency and low Total Cost of Ownership (TCO) unmatched by the EDW.

The Hadoop Distributed File System (HDFS) is designed for large scale batch processing. Providing a write-once, read-many, append-only storage model for unindexed data stored in files of up 128MB, HDFS is optimized for long running, sequential scans across TBs and PBs of data.

This makes Hadoop incredibly powerful at mining large swaths of multi-structured data to create analytics that companies can use to better inform their business. Example outputs can include:

Customer segmentation models for marketing campaigns and eCommerce recommendations.
Churn analysis for customer service representatives.
Predictive analytics for fleet maintenance and optimization.
Risk modeling for security and fraud detection.

These types of models are typically built from Hadoop queries executed across the data lake with latencies in the range of minutes and hours. However, the data lake, which excels at generating new forms of insight from diverse data sets, is not designed to provide real-time access to operational applications. Users need to make the analytic outputs from Hadoop available to their online, operational apps. These applications have specific access demands that cannot be met by HDFS, including:

Millisecond latency query responsiveness.
Random access to indexed subsets of data.
Supporting expressive ad-hoc queries and aggregations against the data, making online applications smarter and contextual.
Updating fast-changing data in real time as users interact with online applications, without having to rewrite the entire data set.

In our data-driven world, milliseconds matter. In fact, research from IBM observed that 60% of data loses its value within milliseconds of generation. For example, what is the value in identifying a fraudulent transaction minutes after the trade was processed? Furthermore, Gartner analysts predict that 70 percent of Hadoop deployments will not meet cost savings and revenue generation objectives due to skills and integration challenges.

Being able to generate and serve analytics from the data lake to online applications and users in real time can help address these challenges, demanding the integration of a highly scalable, highly flexible operational database layer. Ultimately, the companies that win in the future will not be those that have the largest data lakes. Rather it will be those who are the fastest in acting on the insights and intelligence that data itself creates. Operational databases are essential to executing on the data lake vision.

In part 2 of this blog series, we’ll look at the critical capabilities you need to consider when evaluating and selecting the operational database for your data lake.

Learn more by reading the Operational Data Lake white paper. Unlocking Operational Intelligence from the Data Lake

About the Author - Mat Keep

Mat is director of product and market analysis at MongoDB. He is responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.

← Previous

Digging into D3 Internals to Eliminate Jank over Large Data Sets

Browser JavaScript isn't like most other user-facing application runtimes, where a main thread handles the UI and you can spin work off into background threads. Because it's single-threaded, when you have to do heavy lifting, sometimes you need to get creative to keep your UI responsive. This was the case on one of Cloud Manager’s newest features, the Visual Profiler. It was an ambitious design, and when I was given the initial mocks I was immediately excited by the prospect. As a front-end engineer, I couldn’t wait to start implementing the new chart and table: An early mock of the Visual Profiler We didn’t have anything quite like it in the app, and I found out pretty quickly that our existing charts library wasn’t going to support it. Not missing an opportunity to try something new, I moved forward with a D3 -based solution and got a chart going pretty quickly. Unfortunately, when I connected it to actual data from MongoDB’s built-in Profiler , it ground to a halt. I should have realized it sooner, but the number of data points we were hoping to plot was tremendous. In D3 this is a problem because it’s built on top of the HTML5 SVG element, meaning adding and removing plot points requires a DOM manipulation. Although we do throttle the number of profiler entries we take in from a single customer’s application (no one wants to try to drink from a firehose), the Visual Profiler would still need to handle over 10,000 points. The chart was meant to be responsive, allowing users to select, filter, zoom, and change the statistics being shown, but most people’s browsers just can’t handle inserting that many DOM elements in real time. On my reasonably well-equipped 2014 Macbook Pro, using any of those features froze the browser for a good 5 seconds. Users on older computers might as well force quit their browsers. Well, that’s what we call jank - it's what happens when you ask the browser to do more rendering than it can handle before a single frame is rendered. At 60 frames per second that’s a measly 16 milliseconds. To fix the jank, I was going to have to break up the rendering into smaller chunks that would fit inside that tiny window... and to do that, I was going to have to get up close and personal with the D3 internals. Setting up the problem To simplify the problem, we're going to work on code from this gist . It's about as basic as we can get to demonstrate and solve the problem. If you believe me that the problem exists, we can ignore the full gist and just look at this section of code: function drawCircles(svg, data) { var circles = svg.selectAll('circle').data(data); circles.enter().append('circle') .attr('r', 3); circles.exit().remove(); circles .attr('cx', function(d) { return d.x }) .attr('cy', function(d) { return d.y }); } The function takes a D3 svg selection and an array called data of (x, y) coordinates. With D3's enter() function, we can get the subset of data points that don't have matching DOM elements, and add a circle for each new point. The chart comes out like this: This code will execute all at once when the JavaScript event loop reaches it, without yield, until all the points are rendered. If we call this code with a large enough number of data points on a button press, you'll see your browser hang before it even returns the button to its undepressed state. That's the easiest part to fix, actually, and it will lay the foundation for keeping the whole thing snappy, so let's start there. Putting functions on the message queue To fix the browser hang, we slot our function at the end of JavaScript's message queue . This allows the browser to completely handle the button press event and repaint the button before it executes any of our drawing code. The easiest way to do that is wrap our code in a setTimeout : function drawCircles(svg, data) { setTimeout(function() { var circles = svg.selectAll('circle').data(data); circles.enter().append('circle') .attr('r', 3); circles.exit().remove(); circles .attr('cx', function(d) { return d.x }) .attr('cy', function(d) { return d.y }); }, 0); } In this gist you’ll notice that the button returns to its undepressed state before the circles finish being drawn. “That was easy!” you might say, but now comes the hard part. We've simply delayed the problem a little. While we’re adding and maneuvering all those circles on the page, the browser is still going to hang, and that’s what we’re trying to avoid. So now, we’re going to have to break our D3 Selection circles into batches, and render each batch in a separate message on the queue. Breaking a D3 selection into batches D3 doesn’t provide native functions for breaking up the selection above (circles) into batches for rendering. In other situations, you might be able to use select() or filter() (which give you a subset of a selection) but they don’t preserve the data-binding that provides the convenient joins called update , enter() , and exit() . You might try to break up the data into batches earlier, when you give it to D3, but then D3 would compute inaccurate joins leaving you with mismatched data-binding and fewer circles than you need. Building your own update, enter, and exit selections A D3 selection is a subclass of an array , whose elements represent groups of DOM nodes as more arrays. Who doesn’t like an array of arrays? In our case, there is only one “group of DOM nodes” subarray, so circles[0] gets all the current DOM elements we want. We’ll also need circles.enter() and circles.exit() to have all the possible elements for our data. That’s because for each new element that is entering the data set, D3 puts a null placeholder in circles[0] , and the actual element in the corresponding slot of the enter() selection. That is to say, when we bind the data to our selection, if the nth datum does not have a corresponding element in the DOM, circles[0][n] is a new slot containing null , and circles.enter()[0][n ] contains an element for the new datum. Lastly, to get the first batchSize elements as represented in the update() , enter() and exit() joins, we can slice the arrays of DOM elements and wrap them in a new selection like so: var updateSelection = d3.selectAll(circles[0].slice(0, batchSize)); var enterSelection = d3.selectAll(circles.enter()[0].slice(0, batchSize)); var exitSelection = d3.selectAll(circles.exit()[0].slice(0, batchSize)); Out with the exit, in with the enter! So now we’ve got our three selections, and we want to perform the same operations as before. The way we handle updates and exits doesn’t have to change because these selections refer to actual DOM elements: // equivalent to circles.exit().remove() in the un-batched example exitSelection.remove(); // equivalent to circles.attr in the un-batched example updateSelection .attr('cx', function(d) { return d.x }) .attr('cy', function(d) { return d.y }); But the way we handle enterSelection has to be completely different than how we handle circles.enter() . Calling enter() on a selection generates a sub-type of selection , inside of which are nulls for all existing elements and dummy objects for each new element. This special sub-type of selection knows how to handle those nulls when we call append() , but enterSelection , which was generated via selectAll() , is an ordinary selection, and will not handle nulls correctly. The good news is that the dummy objects that enterSelection contains still have the data that we bound using data() , so we can use each to append the DOM elements we need and give them the right data: enterSelection.each(function(d, i) { var newElement = svg.append('circle')[0][0]; newElement.__data__ = this.__data__; }).attr('r', 3); This will get us an element in the DOM for every element in enterSelection , but the other convenient advantage of D3s joins is that they automatically update one another. Specifically, when you call append() on an enter() join, the DOM elements that are created are added into the enter() join as well as the update() selection you created the enter( ) join from. Our enterSelection is missing this functionality because it isn’t a real enter() join, so to do that we’ll have to modify enterSelection and updateSelection manually as we go along. enterSelection.each(function(d, i) { var newElement = svg.append('circle')[0][0]; newElement.__data__ = this.__data__; enterSelection[0][i] = newElement; updateSelection[0][i] = newElement; }).attr('r', 3); And that’s about it. Yes, we’re reaching into D3’s internals, but we’re walking away with a highly optimized batched rendering process. Putting it all together The last step is to generalize this across multiple batches and split it across multiple timeouts. We do that by calculating a startIndex and stopIndex for a particular batch, and write a new drawBatch function to be called with setTimeout : function drawCircles(svg, data, batchSize) { var circles = svg.selectAll('circle').data(data); function drawBatch(batchNumber) { return function() { var startIndex = batchNumber * batchSize; var stopIndex = Math.min(data.length, startIndex + batchSize); var updateSelection = d3.selectAll(circles[0].slice(startIndex, stopIndex)); var enterSelection = d3.selectAll(circles.enter()[0].slice(startIndex, stopIndex)); var exitSelection = d3.selectAll(circles.exit()[0].slice(startIndex, stopIndex)); enterSelection.each(function(d, i) { var newElement = svg.append('circle')[0][0]; enterSelection[0][i] = newElement; updateSelection[0][i] = newElement; newElement.__data__ = this.__data__; }).attr('r', 3); exitSelection.remove(); updateSelection .attr('cx', function(d) { return d.x }) .attr('cy', function(d) { return d.y }); if (stopIndex < data.length) { setTimeout(drawBatch(batchNumber + 1), 0); } }; } setTimeout(drawBatch(0), 0); } We also had to add a condition to the end of the drawBatch function that calls itself for the next batch if there is another batch to draw. If you run this gist you’ll now notice that it takes a bit longer to render, but the entire page remains responsive while it happens. A responsive visual profiler After all that digging through the internals of D3 and hacking together my own batched rendering, I connected the new chart to real data from our backend. Watching the scatterplot render progressively while the page remained responsive was the ultimate payoff. No jank, just a great user experience with a scatter plot that helps our users find the choke points in their MongoDB deployment. I’m still blown away everytime I look at it! But now I'm blown away because it’s working, not because of all the rendering obstacles standing in the way of getting it done.

May 11, 2016

Next →

Accelerating to T+1 - Have You Got the Speed and Agility Required to Meet the Deadline?

On May 28, 2024, the Securities and Exchange Commission (SEC) will implement a move to a T+1 settlement for standard securities trades , shortening the settlement period from 2 business days after the trade date to one business day. The change aims to address market volatility and reduce credit and settlement risk. The shortened T+1 settlement cycle can potentially decrease market risks, but most firms' current back-office operations cannot handle this change. This is due to several challenges with existing systems, including: Manual processes will be under pressure due to the shortened settlement cycle Batch data processing will not be feasible To prepare for T+1, firms should take urgent action to address these challenges: Automate manual processes to streamline them and improve operational efficiency Event-based real-time processing should replace batch processing for faster settlement In this blog, we will explore how MongoDB can be leveraged to accelerate manual process automation and replace batch processes to enable faster settlement. What is a T+1 and T+2 settlement? T+1 settlement refers to the practice of settling transactions executed before 4:30pm on the following trading day. For example, if a transaction is executed on Monday before 4:30 pm, the settlement will occur on Tuesday. This settlement process involves the transfer of securities and/or funds from the seller's account to the buyer's account. This contrasts with the T+2 settlement, where trades are settled two trading days after the trade date. According to SEC Chair Gary Gensler , “T+1 is designed to benefit investors and reduce the credit, market, and liquidity risks in securities transactions faced by market participants.” Overcoming T+1 transition challenges with MongoDB: Two unique solutions 1. The multi-cloud developer data platform accelerates manual process automation Legacy settlement systems may involve manual intervention for various tasks, including manual matching of trades, manual input of settlement instructions, allocation emails to brokers, reconciliation of trade and settlement details, and manual processing of paper-based documents. These manual processes can be time-consuming and prone to errors. MongoDB (Figure 1 below) can help accelerate developer productivity in several ways: Easy to use: MongoDB is designed to be easy to use, which can reduce the learning curve for developers who are new to the database. Flexible data model: Allows developers to store data in a way that makes sense for their application. This can help accelerate development by reducing the need for complex data transformations or ORM mapping. Scalability: MongoDB is highly scalable , which means it can handle large volumes of trade data and support high levels of concurrency. Rich query language: Allows developers to perform complex queries without writing much code. MongoDB's Apache Lucene-based search can also help screen large volumes of data against sanctions and watch lists in real-time. Figure 1: MongoDB's developer data platform Discover the developer productivity calculator . Developers spend 42% of their work week on maintenance and technical debt. How much does this cost your organization? Calculate how much you can save by working with MongoDB. 2. An operational trade store to replace slow batch processing Back-office technology teams face numerous challenges when consolidating transaction data due to the complexity of legacy batch ETL and integration jobs. Legacy databases have long been the industry standard but are not optimal for post-trade management due to limitations such as rigid schema, difficulty in horizontal scaling, and slow performance. For T+1 settlement, it is crucial to have real-time availability of consolidated positions across assets, geographies, and business lines. It is important to note that the end of the batch cycle will not meet this requirement. As a solution, MongoDB customers use an operational trade data store (ODS) to overcome these challenges for real-time data sharing. By using an ODS, financial firms can improve their operational efficiency by consolidating transaction data in real-time. This allows them to streamline their back-office operations, reduce the complexity of ETL and integration processes, and avoid the limitations of relational databases. As a result, firms can make faster, more informed decisions and gain a competitive edge in the market. Using MongoDB (Figure 2 below), trade desk data is copied into an ODS in real-time through change data capture (CDC), creating a centralized trade store that acts as a live source for downstream trade settlement and compliance systems. This enables faster settlement times, improves data quality and accuracy, and supports full transactionality. As the ODS evolves, it becomes a "system of record/golden source" for many back office and middle office applications, and powers AI/ML-based real-time fraud prevention applications and settlement risk failure systems. Figure 2: Centralized Trade Data Store (ODS) Managing trade settlement risk failure is critical in driving efficiency across the entire securities market ecosystem. Luckily, MongoDB integration capabilities (Figure 3 below) with modern AI and ML platforms enable banks to develop AI/ML models that make managing potential trade settlement fails much more efficient from a cost, time, and quality perspective. Additionally, predictive analytics allow firms to project availability and demand and optimize inventories for lending and borrowing. Figure 3: Event-driven application for real time monitoring Summary Financial institutions face significant challenges in reducing settlement duration from two business days (T+2) to one (T+1), particularly when it comes to addressing the existing back-office issues. However, it's crucial for them to achieve this goal within a year as required by the SEC. This blog highlights how MongoDB's developer data platform can help financial institutions automate manual processes and adopt a best practice approach to replace batch processes with a real-time data store repository (ODS). With the help of MongoDB's developer data platform and best practices, financial institutions can achieve operational excellence and meet the SEC's T+1 settlement deadline on May 28, 2024. In the event of T+0 settlement cycles becoming a reality, institutions with the most flexible data platform will be better equipped to adjust. Top banks in the industry are already adopting MongoDB's developer data platform to modernize their infrastructure, leading to reduced time-to-market, lower total cost of ownership, and improved developer productivity. Looking to learn more about how you can modernize or what MongoDB can do for you? Zero downtime migrations using MongoDB’s flexible schema Accelerate your digital transformation with these 5 Phases of Banking Modernization Reduce time-to-market for your customer lifecycle management applications MongoDB’s financial services hub

May 25, 2023