Unlocking Operational Intelligence from the Data Lake: Part 2 - Operationalizing the Data Lake

Mat Keep
May 23, 2016 | Updated: August 17, 2016
#Technical

As we discussed in part 1, Hadoop-based data lakes excel at generating new forms of insight from diverse data sets, but are not designed to provide real-time access to operational applications.

Users need to make analytic outputs from Hadoop available to their online, operational apps. These applications have specific access demands that cannot be met by HDFS, including:

Millisecond latency query responsiveness.
Random access to indexed subsets of data.
Supporting expressive ad-hoc queries and aggregations against the data, making online applications smarter and contextual.
Updating fast-changing data in real time as users interact with online applications, without having to rewrite the entire data set.

Bringing together operational and analytical processing across high volumes of variably structured data in a single database requires capabilities unique to MongoDB:

Workload isolation. MongoDB replica sets can be provisioned with dedicated analytic nodes. This allows users to simultaneously run real-time analytics and reporting queries against live data, without impacting nodes servicing the operational application, and avoiding lengthy ETL cycles.
Dynamic schema, coupled with data governance. MongoDB's document data model makes it easy for users to store and combine data of any structure, without giving up sophisticated validation rules, data access and rich indexing functionality. If new attributes need to be added – for example enriching user profiles with geo-location data – the schema can be modified without application downtime, and without having to update all existing records.
Expressive queries. The MongoDB query language enables developers to build applications that can query and analyze the data in multiple ways – by single keys, ranges, text search, and geospatial queries through to complex aggregations and MapReduce jobs, returning responses in milliseconds. Complex queries are executed natively in the database without having to use additional analytics frameworks or tools, and avoiding the latency that comes from moving data between operational and analytical engines.
Rich secondary indexes. Providing fast filtering and access to data by any attribute, MongoDB supports compound, unique, array, partial, TTL, geospatial, sparse, and text indexes to optimize for multiple query patterns, data types and application requirements. Indexes are essential when operating across slices of the data, for example updating the churn analysis of a subset of high net worth customers, without having to scan all customer data.
BI & analytics integration. The MongoDB Connector for BI enables industry leading analytical and visualization tools such as Tableau to efficiently access data stored in MongoDB using standard SQL.
Robust security controls. Extensive access controls, auditing for forensic analysis and encryption of data both in-flight and at-rest enables MongoDB to protect valuable information and meet the demands of big data workloads in regulated industries.
Scale-out on commodity hardware. MongoDB can be scaled within and across geographically distributed data centers, providing extreme levels of availability and scalability. As your data lake grows, MongoDB scales easily with no downtime and no application changes.
Advanced management and cloud platform. To reduce data lake TCO and risk of application downtime, MongoDB Ops Manager provides powerful tooling to automate database deployment, scaling, monitoring and alerting, and disaster recovery. Further simplifying operations, MongoDB Atlas delivers MongoDB as a service, providing all of the features of the database, without the operational heavy lifting required for any application. MongoDB Atlas is a great choice if you want the database run for you, or if your data lake and apps are also running on a public cloud platform. MongoDB Atlas is available on-demand through a pay-as-you-go model and billed on an hourly basis.
High skills availability. With availability of Hadoop skills cited by Gartner analysts as a top challenge, it is essential you choose an operational database with a large available talent pool. This enables you to find staff who can rapidly build differentiated big data applications. Across multiple measures, including DB Engines Rankings, The 451 Group NoSQL Skills Index and the Gartner Magic Quadrant for Operational Databases, MongoDB is the leading non-relational database.

In addition, the ability to apply the same distributed processing frameworks such as Apache Spark, MapReduce and Hive to data stored in both HDFS and MongoDB allows developers to converge analytics of both real time, rapidly changing data sets with the models created by batch Hadoop jobs. Through sophisticated connectors, Spark and Hadoop can pass queries as filters and take advantage of MongoDB’s rich secondary indexes to extract and process only the range of data it needs – for example, retrieving all customers located in a specific geography. This is very different from less featured datastores that do not support a rich query language or secondary indexes. In these cases, Spark and Hadoop jobs are limited to extracting all data based on a simple primary key, even if only a subset of that data is required for the query. This means more data movement between the data lake and the database, more processing overhead, more hardware, and longer time-to-insight for the user.

Table 1: How MongoDB stacks up for operational intelligence

As demonstrated in Table 1, operational intelligence requires a fully-featured database serving as a System of Record for online applications. These requirements exceed the capabilities of simple key-value or column-oriented datastores that are typically used for short lived, transient data, or legacy relational databases structured around rigid row and column table formats and scale-up architectures.

Figure 1: Design pattern for operationalizing the data lake

Figure 1 presents a design pattern for integrating MongoDB with a data lake:

Data streams are ingested to a pub/sub message queue, which routes all raw data into HDFS. Processed events that drive real-time actions, such as personalizing an offer to a user browsing a product page, or alarms for vehicle telemetry, are routed to MongoDB for immediate consumption by operational applications.
Distributed processing frameworks such as Spark or MapReduce jobs materialize batch views from the raw data stored in the Hadoop data lake.
MongoDB exposes these models to the operational processes, serving queries and updates against them with real-time responsiveness.
The distributed processing frameworks can re-compute analytics models, against data stored in either HDFS or MongoDB, continuously flowing updates from the operational database to analytics views.

In part 3, we’ll demonstrate how leading companies are using the design pattern discussed above to operationalize their data lakes.

Learn more by reading the Operational Data Lake white paper. Unlocking Operational Intelligence from the Data Lake

About the Author - Mat Keep

Mat is director of product and market analysis at MongoDB. He is responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.

← Previous

Back to Basics: Your First MongoDB Application

We hope you enjoyed part 1 of our Back to Basics series where we introduced you to NoSQL and MongoDB. We all know the pain of immersing yourself in a new technology. I have fought the good fight with X-Windows , the original Macintosh Programmer’s Workshop ( MPW ) and overcome the existing syntax of Javascript :-). But don’t worry, embracing MongoDB is a walk in the park compared to those environments. We expect most programmers to create and query their first database in less time than it takes to download MongoDB. In part II we will actually get to code and build a blogging application. We will start in the Mongo Shell to show you how to interact directly with MongoDB on the command line, then move to an IDE to show you how to build a complete web application with the MongoDB Python driver . Once your application is built we will show you how to add indexes to improve the performance of your queries. Like most databases, indexes can dramatically improve query performance. MongoDB comes with a query analyser and a tool called Explain that can give detailed insight into query performance. At the end of this webinar you will know how to: Run MongoDB Create a basic database and set of collections using the MongoDB Shell Create a basic database and collection using one of our language drivers Add an index to a collection to improve query performance Review the efficiency of your queries using the explain framework Register now for this webinar and all the remaining webinars in the series. Register for the Back to Basics webinar series

May 19, 2016

Next →

Being Latine in Tech: Two MongoDB Employees Share Their Advice on Building Careers in Engineering

Ashley Naranjo and Martin Bajana, members of MongoDB’s employee resource group QueLatine, share their career journeys and offer insight into how other members of the Latine community can build careers in tech. Jackie Denner: How did you make your way into the tech industry? Ashley Naranjo: I am a first-generation Latina with a passion for Information Technology and a knack for problem-solving. After graduating early from high school, I embarked on a career in Nursing. I chose Nursing initially because I wanted to make a difference and help others, but my path took an unexpected turn when COVID-19 reshaped our world. In light of the circumstances, I reevaluated my options and decided to seize an opportunity with a program called Year Up . During the intensive six-month training and deployment phase, I not only completed rigorous coursework but also obtained IT Google Coursera certifications and actively pursued CompTIA certifications. This experience allowed me to secure an internship at Meta (Facebook) as an Enterprise Operation IT Support Tech, where my love for technology blossomed. During my time at Meta, I had the privilege of assisting diverse Meta users worldwide with a wide range of technical issues, including troubleshooting, software and hardware support, internal access permissions, and more. The exposure to a global tech environment further fueled my passion for the field. When my internship concluded, I was offered a 1-year contract role with Meta to continue my work as a support tech for the same team. Throughout that year, I immersed myself in all aspects of technology, maximizing my learning opportunities and applying my networking skills. As time went on, I knew I needed a new challenge. This led me to embark on a search for an exciting role, which eventually brought me to MongoDB. I am passionate about driving technological innovation, and MongoDB is a place where I can make an impact. Martin Bajana: My interest in technology stems from a variety of sources. From a young age, I developed a strong passion for video games and exploring new technologies. Whether it was experimenting with the latest gaming consoles or delving into computer hardware, I relished the opportunity to learn and understand the inner workings of these technologies. In school, I discovered my affinity for mathematics, which further solidified my decision to pursue a career in the tech industry. Choosing to study computer science in college was a natural progression for me, as it allowed me to combine my love for technology with my aptitude for problem-solving. After completing my education, I was recruited by Verizon, where I worked on front-end applications and Android development. Although the transition was initially challenging, I persevered and regained my confidence. It was during this period that I realized a career in technology was my long-term aspiration. Throughout my tenure at Verizon, I embraced opportunities to work across various teams, acquiring valuable experience and honing my skills. Eventually, I made the decision to join MongoDB, which has provided me with an enriching journey and the chance to shape my career in the tech industry. JD: Have there been any challenges you've faced throughout your career? AN: Imposter syndrome has been a significant challenge for me throughout my career, and it's something I still deal with to this day. When surrounded by my talented colleagues, I would often compare myself to them and focus on my perceived weaknesses and flaws, leading to a lack of self-confidence. However, I tackled this issue by addressing my feelings with my manager. Her support and guidance helped me realize my own potential and acknowledge my accomplishments. Maintaining a positive mindset has enabled me to view myself as a competent engineer and recognize the value I bring to my team. I have learned to take ownership of my successes and embrace opportunities for growth. Stepping out of my comfort zone has become a regular practice, as personal and professional development often stems from embracing challenges and discomfort. By giving myself permission to take up space and be confident in my abilities, I have been able to overcome imposter syndrome and continue to thrive in my role. MB: I have been fortunate enough to work for companies and teams that value and respect me for the work I deliver. Being in the tech industry and growing up in a culturally diverse region of the country, I have had exposure to individuals from various backgrounds and identities, which has made me more comfortable as a Latinx individual in the industry. My personal goal is to promote a work environment where everyone is judged based on the contributions they bring to the team, rather than their identity. I believe in supporting and respecting the identities of my peers and coworkers while fostering a culture of inclusivity and equality. JD: How has MongoDB supported your career growth and development? AN: In my time working at MongoDB, I have experienced exceptional support that has greatly contributed to my professional development and growth. As an engineer at MongoDB, I have been provided with numerous opportunities to expand my knowledge and skills through participation in tech talks, hackathons, and continuous learning about emerging technologies. I am grateful for the proactive approach taken by my manager and team leaders in fostering my growth as an engineer. Additionally, MongoDB's commitment to diversity and inclusion is evident through the company's DEI initiatives. Platforms like our employee resource group “QueLatine” have made me feel a stronger sense of connection and belonging, particularly among my Latinx peers. By recognizing the power of our diverse backgrounds and experiences, MongoDB empowers us to have a meaningful impact in the industry. MB: I have experienced full support from my leader since day one. They have proactively sought to understand my career goals and have helped me create a clear career path to achieve those goals. This level of support has enabled me to take on challenging projects and initiatives within the company, allowing me to grow and develop in my career. Furthermore, MongoDB offers a wealth of learning and development resources to its employees, which I have fully utilized to continue learning and growing my skill set. JD: What is your advice for other Latines who want to begin careers in tech? AN: Having made a significant career change myself, I can empathize with the challenges that come with exploring new paths, particularly in the tech industry. As a Latina in tech, I feel a strong desire to encourage and raise awareness within our community about the incredible resources and opportunities that are available to us. My advice to others who may be considering a similar journey is to prioritize the continuous development of your technical skills, actively seek out mentoring opportunities, push yourself beyond your comfort zone by honing your networking abilities, and most importantly, believe in yourself and your ability to achieve great things! MB: Navigating the vast world of technology can certainly be overwhelming, but it's important not to fear feeling lost. Even after 12 years in this career, there are still days where I come across something I've never heard of before. Fortunately, we live in a world abundant with resources for continuous learning. My advice is to take the time to explore and ask questions. Seek out open-source projects that you can contribute to, and connect with other professionals in the tech industry who can share their experiences and provide guidance. Additionally, taking advantage of hackathons and other tech events can expose you to new technologies and ideas. Don't be afraid to make mistakes, and most importantly, don't give up! Join us in transforming the way developers work with data. Build your tech career at MongoDB .

September 20, 2023