Unlocking Operational Intelligence from the Data Lake: Part 1 - The Rise of the Data Lake

Mat Keep
May 11, 2016 | Updated: May 23, 2016
#Technical

Series Introduction

Hadoop-based data lakes are enabling enterprises and governments to efficiently capture and analyze unprecedented volumes of data generated from their digital transformation initiatives. But without being able to expose that data to operational applications, users are struggling to maximize returns on their Hadoop investments. The longer it takes to surface insight to operational processes, the less valuable that insight is, and the less competitive you are.

In this 3-part blog series, we’re going to cover:

The rise of the data lake, the role of Hadoop, and the challenges in integrating the data lake with operational applications
In part 2 we’ll cover the critical capabilities you need to evaluate in an operational database for your data lake, and a recommended technology design pattern for integrating the database with the data lake
We’ll wrap up with part 3 covering real world examples and best practices from industry leaders

If you want to get a head-start and learn about all of these topics now, just go ahead and download the Operational Data Lake white paper.

The Rise of the Data Lake

The one thing no business lacks today is data – from streams of sensor readings, to social sentiment, to machine logs, mobile apps, and more. Analysts estimate data volumes growing at 40% per annum, with 90% of it unstructured. Uncovering new insights by collecting and analyzing this data carries the promise of competitive advantage and efficiency savings. However, the traditional Enterprise Data Warehouse (EDW) is straining under the load, overwhelmed by the sheer volume and variety of data pouring into the business, and then being able to store it in a cost-efficient way. As a result many organizations have turned to Hadoop as a centralized repository for this new data, creating what many call a data lake.

With its ability to store data of any structure without a predefined schema and scale-out on commodity hardware, Hadoop provides levels of performance, efficiency and low Total Cost of Ownership (TCO) unmatched by the EDW.

The Hadoop Distributed File System (HDFS) is designed for large scale batch processing. Providing a write-once, read-many, append-only storage model for unindexed data stored in files of up 128MB, HDFS is optimized for long running, sequential scans across TBs and PBs of data.

This makes Hadoop incredibly powerful at mining large swaths of multi-structured data to create analytics that companies can use to better inform their business. Example outputs can include:

Customer segmentation models for marketing campaigns and eCommerce recommendations.
Churn analysis for customer service representatives.
Predictive analytics for fleet maintenance and optimization.
Risk modeling for security and fraud detection.

These types of models are typically built from Hadoop queries executed across the data lake with latencies in the range of minutes and hours. However, the data lake, which excels at generating new forms of insight from diverse data sets, is not designed to provide real-time access to operational applications. Users need to make the analytic outputs from Hadoop available to their online, operational apps. These applications have specific access demands that cannot be met by HDFS, including:

Millisecond latency query responsiveness.
Random access to indexed subsets of data.
Supporting expressive ad-hoc queries and aggregations against the data, making online applications smarter and contextual.
Updating fast-changing data in real time as users interact with online applications, without having to rewrite the entire data set.

In our data-driven world, milliseconds matter. In fact, research from IBM observed that 60% of data loses its value within milliseconds of generation. For example, what is the value in identifying a fraudulent transaction minutes after the trade was processed? Furthermore, Gartner analysts predict that 70 percent of Hadoop deployments will not meet cost savings and revenue generation objectives due to skills and integration challenges.

Being able to generate and serve analytics from the data lake to online applications and users in real time can help address these challenges, demanding the integration of a highly scalable, highly flexible operational database layer. Ultimately, the companies that win in the future will not be those that have the largest data lakes. Rather it will be those who are the fastest in acting on the insights and intelligence that data itself creates. Operational databases are essential to executing on the data lake vision.

In part 2 of this blog series, we’ll look at the critical capabilities you need to consider when evaluating and selecting the operational database for your data lake.

Learn more by reading the Operational Data Lake white paper. Unlocking Operational Intelligence from the Data Lake

About the Author - Mat Keep

Mat is director of product and market analysis at MongoDB. He is responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.

← Previous

Digging into D3 Internals to Eliminate Jank over Large Data Sets

Browser JavaScript isn't like most other user-facing application runtimes, where a main thread handles the UI and you can spin work off into background threads. Because it's single-threaded, when you have to do heavy lifting, sometimes you need to get creative to keep your UI responsive. This was the case on one of Cloud Manager’s newest features, the Visual Profiler. It was an ambitious design, and when I was given the initial mocks I was immediately excited by the prospect. As a front-end engineer, I couldn’t wait to start implementing the new chart and table: An early mock of the Visual Profiler We didn’t have anything quite like it in the app, and I found out pretty quickly that our existing charts library wasn’t going to support it. Not missing an opportunity to try something new, I moved forward with a D3 -based solution and got a chart going pretty quickly. Unfortunately, when I connected it to actual data from MongoDB’s built-in Profiler , it ground to a halt. I should have realized it sooner, but the number of data points we were hoping to plot was tremendous. In D3 this is a problem because it’s built on top of the HTML5 SVG element, meaning adding and removing plot points requires a DOM manipulation. Although we do throttle the number of profiler entries we take in from a single customer’s application (no one wants to try to drink from a firehose), the Visual Profiler would still need to handle over 10,000 points. The chart was meant to be responsive, allowing users to select, filter, zoom, and change the statistics being shown, but most people’s browsers just can’t handle inserting that many DOM elements in real time. On my reasonably well-equipped 2014 Macbook Pro, using any of those features froze the browser for a good 5 seconds. Users on older computers might as well force quit their browsers. Well, that’s what we call jank - it's what happens when you ask the browser to do more rendering than it can handle before a single frame is rendered. At 60 frames per second that’s a measly 16 milliseconds. To fix the jank, I was going to have to break up the rendering into smaller chunks that would fit inside that tiny window... and to do that, I was going to have to get up close and personal with the D3 internals. Setting up the problem To simplify the problem, we're going to work on code from this gist . It's about as basic as we can get to demonstrate and solve the problem. If you believe me that the problem exists, we can ignore the full gist and just look at this section of code: function drawCircles(svg, data) { var circles = svg.selectAll('circle').data(data); circles.enter().append('circle') .attr('r', 3); circles.exit().remove(); circles .attr('cx', function(d) { return d.x }) .attr('cy', function(d) { return d.y }); } The function takes a D3 svg selection and an array called data of (x, y) coordinates. With D3's enter() function, we can get the subset of data points that don't have matching DOM elements, and add a circle for each new point. The chart comes out like this: This code will execute all at once when the JavaScript event loop reaches it, without yield, until all the points are rendered. If we call this code with a large enough number of data points on a button press, you'll see your browser hang before it even returns the button to its undepressed state. That's the easiest part to fix, actually, and it will lay the foundation for keeping the whole thing snappy, so let's start there. Putting functions on the message queue To fix the browser hang, we slot our function at the end of JavaScript's message queue . This allows the browser to completely handle the button press event and repaint the button before it executes any of our drawing code. The easiest way to do that is wrap our code in a setTimeout : function drawCircles(svg, data) { setTimeout(function() { var circles = svg.selectAll('circle').data(data); circles.enter().append('circle') .attr('r', 3); circles.exit().remove(); circles .attr('cx', function(d) { return d.x }) .attr('cy', function(d) { return d.y }); }, 0); } In this gist you’ll notice that the button returns to its undepressed state before the circles finish being drawn. “That was easy!” you might say, but now comes the hard part. We've simply delayed the problem a little. While we’re adding and maneuvering all those circles on the page, the browser is still going to hang, and that’s what we’re trying to avoid. So now, we’re going to have to break our D3 Selection circles into batches, and render each batch in a separate message on the queue. Breaking a D3 selection into batches D3 doesn’t provide native functions for breaking up the selection above (circles) into batches for rendering. In other situations, you might be able to use select() or filter() (which give you a subset of a selection) but they don’t preserve the data-binding that provides the convenient joins called update , enter() , and exit() . You might try to break up the data into batches earlier, when you give it to D3, but then D3 would compute inaccurate joins leaving you with mismatched data-binding and fewer circles than you need. Building your own update, enter, and exit selections A D3 selection is a subclass of an array , whose elements represent groups of DOM nodes as more arrays. Who doesn’t like an array of arrays? In our case, there is only one “group of DOM nodes” subarray, so circles[0] gets all the current DOM elements we want. We’ll also need circles.enter() and circles.exit() to have all the possible elements for our data. That’s because for each new element that is entering the data set, D3 puts a null placeholder in circles[0] , and the actual element in the corresponding slot of the enter() selection. That is to say, when we bind the data to our selection, if the nth datum does not have a corresponding element in the DOM, circles[0][n] is a new slot containing null , and circles.enter()[0][n ] contains an element for the new datum. Lastly, to get the first batchSize elements as represented in the update() , enter() and exit() joins, we can slice the arrays of DOM elements and wrap them in a new selection like so: var updateSelection = d3.selectAll(circles[0].slice(0, batchSize)); var enterSelection = d3.selectAll(circles.enter()[0].slice(0, batchSize)); var exitSelection = d3.selectAll(circles.exit()[0].slice(0, batchSize)); Out with the exit, in with the enter! So now we’ve got our three selections, and we want to perform the same operations as before. The way we handle updates and exits doesn’t have to change because these selections refer to actual DOM elements: // equivalent to circles.exit().remove() in the un-batched example exitSelection.remove(); // equivalent to circles.attr in the un-batched example updateSelection .attr('cx', function(d) { return d.x }) .attr('cy', function(d) { return d.y }); But the way we handle enterSelection has to be completely different than how we handle circles.enter() . Calling enter() on a selection generates a sub-type of selection , inside of which are nulls for all existing elements and dummy objects for each new element. This special sub-type of selection knows how to handle those nulls when we call append() , but enterSelection , which was generated via selectAll() , is an ordinary selection, and will not handle nulls correctly. The good news is that the dummy objects that enterSelection contains still have the data that we bound using data() , so we can use each to append the DOM elements we need and give them the right data: enterSelection.each(function(d, i) { var newElement = svg.append('circle')[0][0]; newElement.__data__ = this.__data__; }).attr('r', 3); This will get us an element in the DOM for every element in enterSelection , but the other convenient advantage of D3s joins is that they automatically update one another. Specifically, when you call append() on an enter() join, the DOM elements that are created are added into the enter() join as well as the update() selection you created the enter( ) join from. Our enterSelection is missing this functionality because it isn’t a real enter() join, so to do that we’ll have to modify enterSelection and updateSelection manually as we go along. enterSelection.each(function(d, i) { var newElement = svg.append('circle')[0][0]; newElement.__data__ = this.__data__; enterSelection[0][i] = newElement; updateSelection[0][i] = newElement; }).attr('r', 3); And that’s about it. Yes, we’re reaching into D3’s internals, but we’re walking away with a highly optimized batched rendering process. Putting it all together The last step is to generalize this across multiple batches and split it across multiple timeouts. We do that by calculating a startIndex and stopIndex for a particular batch, and write a new drawBatch function to be called with setTimeout : function drawCircles(svg, data, batchSize) { var circles = svg.selectAll('circle').data(data); function drawBatch(batchNumber) { return function() { var startIndex = batchNumber * batchSize; var stopIndex = Math.min(data.length, startIndex + batchSize); var updateSelection = d3.selectAll(circles[0].slice(startIndex, stopIndex)); var enterSelection = d3.selectAll(circles.enter()[0].slice(startIndex, stopIndex)); var exitSelection = d3.selectAll(circles.exit()[0].slice(startIndex, stopIndex)); enterSelection.each(function(d, i) { var newElement = svg.append('circle')[0][0]; enterSelection[0][i] = newElement; updateSelection[0][i] = newElement; newElement.__data__ = this.__data__; }).attr('r', 3); exitSelection.remove(); updateSelection .attr('cx', function(d) { return d.x }) .attr('cy', function(d) { return d.y }); if (stopIndex < data.length) { setTimeout(drawBatch(batchNumber + 1), 0); } }; } setTimeout(drawBatch(0), 0); } We also had to add a condition to the end of the drawBatch function that calls itself for the next batch if there is another batch to draw. If you run this gist you’ll now notice that it takes a bit longer to render, but the entire page remains responsive while it happens. A responsive visual profiler After all that digging through the internals of D3 and hacking together my own batched rendering, I connected the new chart to real data from our backend. Watching the scatterplot render progressively while the page remained responsive was the ultimate payoff. No jank, just a great user experience with a scatter plot that helps our users find the choke points in their MongoDB deployment. I’m still blown away everytime I look at it! But now I'm blown away because it’s working, not because of all the rendering obstacles standing in the way of getting it done.

May 11, 2016

Next →

Building a Culture of Growth: SVP Simon Eid on MongoDB's Massive Opportunity in APAC

Simon Eid is Senior Vice President Asia-Pacific (APAC) at MongoDB and leads the sales teams across Australia and New Zealand, India, ASEAN, and Japan. Simon's go-to-market organisation in APAC is growing rapidly and has nearly tripled in size in the past three years. They are hiring in all regions . In this article, Simon discusses MongoDB’s opportunity in APAC and how he builds a culture of growth and accountability. Simon Eid, SVP APAC, MongoDB (left) and Anoop Dhankar, RVP ANZ, MongoDB (right) MongoDB's opportunity in Asia-Pacific Out of the top 13 economies by GDP in the world , five of them are located in APAC: China, Japan, Australia, India, and South Korea. And that's to say nothing of the ASEAN countries which alone have more than 650 million inhabitants. Combine this with the worldwide database market, one of the largest markets in the software industry. IDC estimates that it will grow to $137B in 2027, and MongoDB has just reached $1B in ARR. This gives you a sense of the massive market opportunity we have globally. Regardless of industry, product, or service, almost every company is becoming a technology company, which means that every company is becoming a data company. We believe MongoDB is the Developer Data Platform that is best placed to support and accelerate that trend. We’ve already captured thousands of customers around the globe, but it’s important to keep in mind that our world is still in the early stages of shifting to the cloud and changing how applications are built and run. Compared to other software, what's special about the market we play in is that the database is not a “nice-to-have”; it’s mission-critical for organisations. As our world continues to undergo this digital transformation, we have the opportunity to transform how our customers use software and data to innovate, create, and disrupt industries. For example, look at Cathay Pacific , Hong Kong's home airline carrier operating in more than 60 destinations worldwide. The company's digital team turned to MongoDB on their journey to become one of the first airlines to create a truly paperless flight deck. Flight Folder, their application built on MongoDB, consolidates dozens of different information sources into one place. Since the Flight Folder launch, Cathay Pacific has completed more than 340,000 flights with full digital integration in the flight deck. Their innovation is enabled by MongoDB. Building a team across regions and cultures Our team in APAC is unique because of the different markets and cultures within the region. What this means is that we go to market differently in India than we do in Australia, in Singapore than we do in South Korea, and so on. Each market is completely different, but within all of them, there is a huge opportunity. Different from many of our peers, in APAC we've established business leaders who run regionalized teams in India, ASEAN, and ANZ with all functions reporting to them. These teams essentially operate as their own business and implement local best practices into their strategy. But, it doesn’t mean they’re operating in a silo. At the leadership level, there is an immense amount of collaboration and sharing of experiences to identify what’s working and what isn’t within each region. We also have a fantastic global sales organisation that rolls out extensive training and best practices to help enable our local teams to best help our customers and grow the business. Members of our APAC team at a recent offsite in Phuket Culture The most important thing is culture. We have a very high standard around everything we do and how we interact with each other. We don’t entertain politics. You can teach someone new skills and coach them on how to be successful in a new role, but if they’re not aligned with the culture, they will not be a fit. It’s a non-negotiable for me and why the most important aspect of the hiring process is the cultural aspect. If you get the culture right, everything else starts to fall into place. What I hear at MongoDB and from the teams I've built at other companies is that this is the kind of culture they can really thrive and grow. At MongoDB, our culture is defined and shaped by six core values . One of the values that’s most important to my team is “Embrace the Power of Differences”. Within APAC, there are a variety of cultural identities and nuances that can often be difficult to navigate, whether it is cultural values, beliefs, or go-to-market strategy. It’s important that everyone who joins my team is respectful of each other’s regional culture. What we’ve done within the APAC region, and with teams across the globe, is take everyone on a journey to understand and embrace these cultural differences. Our role as leaders is to develop our teams, from the bottom all the way up, which is part of MongoDB’s BDR to CRO career development initiative. We need to develop the next wave of leaders so that they’re prepared to step up when the time comes. For APAC, this means that regardless of where someone is from, each team member has been coached and developed on the cultural nuances so that they can lead people and go to market in each of the different regions. It’s also important that each team member contributes to a culture of psychological safety. Being part of a high-growth tech company requires taking risks and making mistakes. We have a high standard and we hold each other accountable, but it never comes at the cost of creating an environment where people are afraid to fail. When someone faces setbacks, I encourage them to share those experiences so that we can collectively learn. Through mutual support, we foster a stronger team capable of delivering exceptional results. The future of MongoDB in Asia-Pacific For any organisation to be successful, I believe it’s critically important for the entire ecosystem to act as one. As I mentioned earlier, at MongoDB the whole country ecosystem is aligned around one set of goals, so it's not a case of different teams running off in different directions. The teams are willing to lean in and do what's required to help each other build a great business. I can confidently say that in APAC, we are one team. This means sales, marketing, customer success, solutions consulting, and professional services all working together to focus on three things: making customers successful, building technical champions, and driving new workloads. As we continue to grow our team and MongoDB’s footprint in the region, these are the three things that will drive our success. As I mentioned earlier, there's a huge opportunity for MongoDB in APAC. Despite hiring slowing down or stopping completely at many other organisations, we're continuing to invest heavily in the region. To give you a sense of that - we've nearly tripled the size of our APAC go-to-market team in the past three years, and we've got more open roles across the different functions and regions. If you want to be part of this journey, there are three things I want to reiterate: First, we are extremely passionate about our culture, from the field level up to the leadership level. As a team, this is the brand we bring to the market. Second, the opportunity here is massive based on the total addressable market and our current share. And third, we place critical importance on development. By joining this team, I can promise that you’ll be provided with countless opportunities to develop your career and make an impact. I’m confident in my team and the leadership we have in place who are ready to take MongoDB APAC to the next level. Join us !

September 28, 2023