MongoDB Connector for Apache Spark: Announcing Early Access Program & New Spark Course

Mat Keep
May 18, 2016 | Updated: August 11, 2016
#Technical

**Update: August 4th 2016** Since this original post, the connector has been declared generally available, for production usage. Click through for a tutorial on using the new MongoDB Connector for Apache Spark.

We live in a world of “big data”. But it isn’t only the data itself that is valuable – it’s the insight it can generate. How quickly an organization can unlock and act on that insight has become a major source of competitive advantage. Collecting data in operational systems and then relying on nightly batch ETL (Extract Transform Load) processes to update the Enterprise Data Warehouse (EDW) is no longer sufficient. Speed-to-insight is critical, and so analytics against live operational data to drive real-time action is fast becoming a necessity, enabled by a new generation of technologies like MongoDB and Apache Spark.

The new native MongoDB Connector for Apache Spark provides higher performance, greater ease of use, and access to more advanced Spark functionality than any connector available today. The new MongoDB University course for Apache Spark provides a fast track introduction for developers and data scientists building new generations of operational applications incorporating sophisticated real-time analytics.

The Rise of Apache Spark

Apache Spark is one of the fastest-growing big data projects in the history of the Apache Software Foundation. With its memory-oriented architecture, flexible processing libraries, and ease-of-use, Spark has emerged as a leading distributed computing framework for real-time analytics.

As a general-purpose framework, Spark is used for many types of data processing – it comes packaged with support for machine learning, interactive queries (SQL), statistical queries with R, graph processing, ETL, and streaming. Spark allows programmers to develop complex, multi-step data pipelines using a directed acyclic graph (DAG) pattern. It supports in-memory data sharing across DAGs, so that different jobs can work with the same data. Additionally, Spark supports a variety of popular programming languages including Scala, Java, and Python.

Sign up for the new Spark course at MongoDB University.

For loading and storing data, Spark integrates with a number of storage and messaging platforms including Amazon S3, Kafka, HDFS, machine logs, relational databases, NoSQL datastores, MongoDB, and more.

MongoDB and Spark Today

While MongoDB natively offers rich real-time analytics capabilities, there are use cases where integrating the Spark engine can extend the processing of operational data managed by MongoDB. This allows users to operationalize results generated from Spark within real-time business processes supported by MongoDB.

Examples of users already using MongoDB and Spark to build modern-data driven applications include:

  • A multinational banking group operating in 31 countries with 51 million clients has implemented a unified real-time monitoring application with Apache Spark and MongoDB. The platform enables the bank to improve customer experience by continuously monitoring client activity across its online channels to check service response times and identify potential issues.
  • A global manufacturing company estimates warranty returns by analyzing material samples from production lines. The collected data enables them to build predictive failure models using Spark machine learning and MongoDB.
  • A video sharing website is using Spark with MongoDB to place relevant advertisements in front of users as they browse, view, and share videos.
  • A global airline has consolidated customer data scattered across more than 100 systems into a single view stored in MongoDB. Spark processes are run against the live operational data in MongoDB to update customer classifications and personalize offers in real time, as the customer is live on the web or speaking with the call center.
  • Artificial intelligence personal assistant company x.ai uses MongoDB and Spark for distributed machine learning problems.

There are a number of ways users integrate MongoDB with Spark. For example, the MongoDB Connector for Hadoop provides a plug-in for Spark. There are also multiple 3rd party connectors available.

Today we are announcing the early access to a new native Spark connector for MongoDB.

Introducing the MongoDB Connector for Apache Spark

The new MongoDB Connector for Apache Spark provides higher performance, greater ease of use and, access to more advanced Spark functionality than the MongoDB Connector for Hadoop. The following table compares the capabilities of both connectors.

 
MongoDB Connector for Spark
MongoDB Connector for Hadoop with Spark Plug-In
Written in Scala, Spark’s native language Yes No, Java
Support for Scala, Java, Python & R APIs Yes Yes
Support for the Spark interactive shell Yes Yes
Support for native Spark RDDs Yes No

Java RDDs. More verbose and complex to work with
Support for Spark DataFrames and Datasets Yes DataFrames Only

Schema must be manually inferred
Automated MongoDB schema inference Yes No
Support for Spark core Yes Yes
Support for Spark SQL Yes Yes
Support for Spark Streaming Yes Yes
Support for Spark Machine Learning Yes Yes
Support for Spark GraphX Yes No
Data locality awareness Yes

The Spark connector is aware which MongoDB partitions are storing data

 No
Support for MongoDB secondary indexes to filter input data Yes Yes
Support for MongoDB aggregation pipeline to filter input data Yes No
Compatibility with MongoDB replica sets and sharded clusters Yes Yes
Support for MongoDB 2.6 and higher Yes Yes
Support for Spark 1.6 and above Yes Yes
Supported for production usage Not Currently

Available for early access evaluation

 Yes

Written in Spark’s native language, the new connector provides a more natural development experience for Spark users as they are quickly able to apply their Scala expertise. The connector provides access to the Spark interactive shell for data exploration and rapid prototyping. The connector exposes all of Spark’s libraries, enabling MongoDB data to be materialized as DataFrames and Datasets for analysis with SQL (benefiting from automatic schema inference), streaming, machine learning, and graph APIs.

The Spark connector can take advantage of MongoDB’s aggregation pipeline and rich secondary indexes to extract, filter, and process only the range of data it needs – for example, analyzing all customers located in a specific geography. This is very different from more simple NoSQL datastores that do not offer either secondary indexes or in-database aggregations. In these cases, Spark would need to extract all data based on a simple primary key, even if only a subset of that data is required for the Spark process. This means more processing overhead, more hardware, and longer time-to-insight for the analyst.

To maximize performance across large, distributed data sets, the Spark connector is aware of data locality in a MongoDB cluster. RDDs are automatically co-located with the associated MongoDB shard to minimize data movement across the cluster. The nearest read preference can be used to route Spark queries to the closest physical node in a MongoDB replica set, thus reducing latency.

Review the MongoDB Connector for Spark documentation to learn how to get started with the connector, and view code snippets for different APIs and libraries.

Fast Track to Apache Spark: New MongoDB University Course

To get the most out of any technology, you need more than documentation and code. Over 350,000 students have registered for developer and operations courses from MongoDB University. Now developers and budding data scientists can get a quick-start introduction to Apache Spark and the MongoDB connector with early access to our new online course.

Getting Started with Spark and MongoDB provides an introduction to Spark and teaches students how to use the new connector to build data analytics applications. In this course, we provide an overview of the Spark Scala and Java APIs with plenty of sample code and demonstrations. Upon completing this course, students will be able to:

  • Outline the roles of major components in the Spark framework
  • Connect Spark to MongoDB
  • Source data from MongoDB for processing in Spark
  • Write data from Spark into MongoDB The course does not assume a prior knowledge of Spark, but does require an intermediate level of expertise with MongoDB.

The course is free. Sign up at MongoDB University.

Next Steps

To wrap up, we are very excited about the possibilities Spark and MongoDB present together, and we hope with the new connector and course, you will be well on your way to building modern, data-driven applications. We would love to hear from you as you explore this new connector and put it through its paces - you can provide feedback and file bugs under the MongoDB Spark Jira project.

Here’s a summary of how to get started:

Sign up for the new Spark course at MongoDB University

← Previous

Leaf in the Wild: MongoDB at CERN – Delivering a Single View of Data from the LHC to Accelerate Scientific Research and Discovery

Leaf in the Wild posts highlight real world MongoDB deployments. Read other stories about how companies are using MongoDB for their mission-critical projects. Multi-Data Center Data Aggregation System Accessed by over 3,000 Physicists from Nearly 200 Research Institutions Across the Globe The European Organisation for Nuclear Research, known as CERN, plays a leading role in the fundamental studies of physics. It has been instrumental in many key global innovations and breakthroughs, and today operates the world's largest particle physics laboratory. The Large Hadron Collider (LHC) nestled under the mountains on the Swiss - Franco border is central to its research into origins of the universe. A key part of CERN’s experiments is the Compact Muon Solenoid (CMS) , a particle detector designed to observe a wide range of particles and phenomena produced in high-energy collisions in the LHC. Scientists use data collected from these collisions to search for new phenomena that will help to answer questions such as: “What is the Universe really made of, and what forces act within it?” “What gives everything substance?” The CMS experiment has used MongoDB for over five years to aid the discovery and analytics of data generated from the LHC. I met up with Valentin Kuznetsov, part of the team responsible for data management in the CMS experiment, to learn more. Can you start by telling us a little bit about what you are doing at CERN? I am a data scientist and research associate at Cornell University where I specialize in the development of data management software for high energy physics experiments. I am also actively involved in data management for the Compact Muon Solenoid (CMS) experiment. Learn more about Apache Spark and MongoDB. CMS is one of the two general purpose particle physics detectors operated at the LHC. The LHC smashes groups of protons together at close to the speed of light: 40 million times per second and with seven times the energy of the most powerful accelerators built up to now. It is designed to explore the fundamental building blocks of the universe, used by more than 3,000 physicists from 183 institutions representing 38 countries. This team drives the design, construction and maintenance of the experiments across the 20PBs of data generated by the LHC each year. How is data managed by the CMS? Experiments of this magnitude require a vast distributed computing and storage infrastructure. The CMS spans more than a hundred data centres, handling raw data from the detector, as well as multiple simulations and associated meta-data. Data is stored in a variety of backend repositories that we call “data-services”, including relational databases, filesystems, message queues, wikis, customized applications and more. At this scale, efficient information discovery within a heterogeneous, distributed environment becomes an important ingredient of successful data analysis. Scientists want to be able to query and combine data from all of the different data-services. The challenge for our users is that this vast and complex collection of data means they don’t necessarily know where to find the right information, or have the domain knowledge necessary to extract the data. What role does MongoDB play in the CMS? MongoDB powers our Data Aggregation System (DAS), providing the ability for researchers to search and aggregate information distributed across all of the CMS backend data-services, and bring that data into a single view. The DAS is implemented as a layer on top of the data-services, allowing researchers to query data via free text-based queries, and then aggregate the results from distributed providers into a single view – while preserving data integrity, security policy and formats. When a user submits a query, it checks if MongoDB has the aggregation the user is asking for and, if it does, returns it. Otherwise the DAS performs the aggregation and saves it to MongoDB. Why was MongoDB selected to power the DAS? MongoDB’s flexible data model was key to our selection. We can’t know the structure of the all the different queries researchers want to run, so a dynamic schema is essential when storing results in a single view. This requirement eliminated relational databases from our evaluation process. We could get similar schema flexibility from other non-relational databases, but what is unique about MongoDB is that it also offers a rich query language and extensive secondary indexes. This gives our users fast and flexible access to data by any query pattern – from simple key-value look-ups, through to complex search, traversals and aggregations across rich data structures, including embedded sub-documents and arrays. We also use CouchDB in other parts of our infrastructure for data replication between different endpoints. Can you describe how MongoDB is deployed in the DAS? MongoDB is used as an intelligent cache on top of distributed data-services. It runs on our storage backends and currently talks to about a dozen of our CMS data-services. The data-services are backed by traditional RDBMS systems based on the Oracle CERN IT cluster. The beauty of this architecture is that it allows us transparently change our data-services without impacting user access into the system. In fact, we have already changed the implementation of a few data-services without affecting our end-users. MongoDB handles the ingestion and expiration of millions of documents managed by the DAS every day. A single query can return up to 10,000 different documents extracted from multiple data-services, which then have to be processed, aggregated and stored in MongoDB, all in real time. The DAS helps scientists easily discover and extract information they need in their research, and it represents one of the many tools which physicists use on a daily basis towards great discoveries. Without the DAS information retrieval would take orders of magnitude longer. MongoDB is deployed on commodity hardware configured with SSDs and running Scientific Linux. Our applications are written in Python, and we are experimenting with Go. Do you have other projects that are using MongoDB? We are in the beta phase of our new Workflow and Data Management (WM) archive system . Agents from systems running data processing pipelines persist log data to MongoDB, which can then be used by our administrators and data scientists to monitor job status, system throughput, error conditions and more. MongoDB provides short-term storage for the archive, providing real-time analytics to staff using MongoDB 3.2’s aggregation pipeline across the past two months of machine data. We collect the throughput of agents at specific sites and aggregate statistics such as total CPU time of the running jobs, their success/failure rates, total size of produced data, and other metrics Data is replicated from MongoDB to Hadoop where it is converted from JSON into an AVRO format, and persisted for long-term storage. Spark jobs are run against the historic archive, with result sets loaded back to MongoDB where they update our real time analytics views, and served to monitoring and visualization apps. Figure 1: Architecture of the CMS WMArchive system Can you share any best practices for getting started with MongoDB? MongoDB’s dynamic schema is great for rapidly prototyping new applications – it gives you the freedom to try out new ideas, and you can be productive in hours. It doesn’t replace the need for proper schema design, but its flexibility means you can quickly iterate to identify the optimum data model for your application. MongoDB also has a vibrant and active community. Never be fearful of reaching out and asking questions, even if you are afraid they may seem pretty basic! MongoDB engineers and community masters are active on the mailing lists, and so you can always get help and guidance. Valentin, thank you for sharing your experiences with the MongoDB community. Learn more about real time analytics and MongoDB by reading our white paper: Apache Spark and MongoDB About the Author - Mat Keep Mat is a director within the MongoDB product marketing team, responsible for building the vision, positioning and content for MongoDB’s products and services, including the analysis of market trends and customer requirements. Prior to MongoDB, Mat was director of product management at Oracle Corp. with responsibility for the MySQL database in web, telecoms, cloud and big data workloads. This followed a series of sales, business development and analyst / programmer positions with both technology vendors and end-user companies.

May 12, 2016
Next →

Tendencias del 2023: Las Medidas de Modernización en el Sector de los Servicios Financieros

Ante la recesión mundial que se avecina, los bancos se enfrentan a unas condiciones económicas difíciles en 2023. Reducir los costes será vital para que muchas organizaciones sigan siendo competitivas en un entorno con un uso intensivo de datos y altamente regulado. Por ello, es importante que cualquier inversión en IT acelere la transformación digital con tecnologías innovadoras que rompan los silos de datos, aumenten la eficiencia operativa y creen experiencias personalizadas para los clientes. Siga leyendo para conocer las áreas en las que los bancos están buscando modernizarse en 2023 para construir mejores experiencias de cliente con un coste menor y a escala. Diseñando un futuro bancario mejor mediante diseños flexibles En un momento en que los bancos están deseando modernizarse e innovar, las entidades deben alejarse de los sistemas heredados que limitan su capacidad de progresar. Situar a los consumidores en el centro de una experiencia bancaria compuesta por servicios interconectados pero independientes ofrece a los bancos, que apuestan por la tecnología, la oportunidad de remodelar sus modelos de negocio y, en consecuencia, aumentar su cuota de mercado y su rentabilidad. Estas oportunidades han hecho posible un diseño de arquitectura componible que permite una innovación más rápida, una mayor eficiencia operativa y la creación de nuevas fuentes de ingresos mediante la ampliación de la cartera de servicios y productos. De este modo, los bancos pueden adoptar el mejor software de su categoría y el que mejor se adapte a sus necesidades, organizando asociaciones estratégicas con las empresas de tecnología financiera y los proveedores de software pertinentes. Esta nueva generación de proveedores puede ofrecer desde servicios de conocimiento del cliente (KYC, del inglés "Know Your Customer") hasta reservas integradas, servicios de carga o funcionalidades básicas de marketing y gestión de portfolios. Este enfoque es más rentable para las entidades que tener que construir y mantener ellas mismas la infraestructura, y es significativamente más rápido en términos de tiempo de comercialización y tiempo de obtención de ingresos. Los bancos que adoptan este enfoque ven a las fintech menos como competidores y más como parte de un ecosistema con el que colaborar para acelerar la innovación y llegar a los clientes. Eficiencia operativa mediante automatización inteligente Las entidades financieras seguirán centrándose en la eficiencia operativa y el control de costes mediante la automatización de los procesos manuales y basados en papel. Los bancos han hecho algunos progresos en la digitalización y automatización de lo que antes eran procesos manuales basados casi exclusivamente en papel. Sin embargo, el principal motor de esta transformación ha sido el cumplimiento de la normativa local, en lugar de una estrategia global para conocer realmente al cliente y lograr su satisfacción. El mercado demanda mejores decisiones automatizadas y basadas en datos, y los sistemas heredados no pueden seguir el ritmo. Crear las experiencias hiperpersonalizadas que demandan los clientes, como chatbots, portales de autoservicio y análisis forense digital, es difícil para las entidades que utilizan tecnología obsoleta. Además, tener una infraestructura de datos en silos impide cualquier experiencia moderna verdaderamente integrada. Mediante una combinación de automatización robótica de procesos (RPA, del inglés "Robotic Process Automation"), aprendizaje automático (ML, del inglés "Machine Learning") e inteligencia artificial (IA), las entidades financieras pueden agilizar los procesos, liberando así a los empleados para que se centren en tareas que tengan un mayor impacto para el cliente y la empresa. Las entidades no deben digitalizar sin tener en cuenta la interacción humana que será sustituida, ya que los clientes prefieren un enfoque híbrido. La capacidad de actuar sobre los datos en tiempo real es el camino a seguir para impulsar el valor y transformar las experiencias de los clientes, lo que debe ir acompañado de la modernización de la arquitectura de datos subyacente. El prerrequisito para alcanzar este objetivo implica la disociación de los datos y las fuentes en un paisaje de datos holístico. Algunos lo llaman data mesh otros fuentes de datos componibles o datos virtualizados. Resolviendo los retos que plantean los datos ESG (del inglés "Environmental, Social, and Governance") Junto con la elevada inflación, la crisis del coste de la vida, las turbulencias energéticas y la subida de los tipos de interés, los aspectos medioambientales, sociales y de gobernanza (ESG) también están en el punto de mira. Los reguladores presionan cada vez más para que se faciliten datos ESG y los inversores para que los portfolios sean sostenibles. El papel de los datos ESG en la realización de análisis de mercado, el apoyo a la asignación de activos y la gestión de riesgos, y proporcionar información sobre la sostenibilidad a largo plazo de las inversiones sigue creciendo. La naturaleza y variabilidad de muchas métricas ESG es un reto importante al que se enfrentan las empresas hoy en día. A diferencia de los datasets financieros, que son en su mayoría numéricos, las métricas ESG pueden incluir datos cuantitativos y cualitativos para ayudar a los inversores y otras partes interesadas a comprender las acciones e intenciones de una empresa. Esta complejidad, unida a la falta de una norma de información ESG de aplicación universal, significa que las instituciones deben considerar diferentes normas con diferentes requisitos de datos. Para dominar la elaboración de informes ESG, incluida la integración de los KPIs pertinentes, se necesitan datos adecuados y de alta calidad que, además, tengan el nivel de granularidad adecuado y cubran los sectores y la región requeridos. Dado el volumen y la complejidad de los datos, las entidades financieras están construyendo plataformas ESG sustentadas en modernas plataformas de datos capaces de consolidar distintos tipos de datos de varios proveedores, crear vistas personalizadas, modelizar datos y realizar operaciones sin barreras. Pagos digitales - Ofreciendo una experiencia enriquecida Impulsado por las nuevas tecnologías y las tendencias mundiales, el mercado de los pagos digitales está floreciendo en todo el mundo. Con una valoración de más de 68.000 millones de dólares en 2021 y expectativas de crecimiento de dos dígitos en la próxima década, los mercados emergentes lideran la expansión relativa. Este crecimiento se ha visto impulsado por los pagos sin efectivo inducidos por la pandemia, el comercio electrónico, el impulso gubernamental y las fintech. Los pagos digitales están transformando la experiencia de pago. Mientras que antes bastaba con que los proveedores de servicios de pago facilitaran información sobre las cuentas y orquestaran transacciones sencillas, ahora los consumidores exigen una experiencia mucho más completa en la que cada transacción ofrezca nuevas perspectivas y servicios de valor añadido. Satisfacer estas expectativas es difícil, especialmente para las empresas que dependen de tecnologías obsoletas que se crearon mucho antes de que las transacciones se realizaran con unos pocos clicks en un dispositivo móvil. Para satisfacer las necesidades de los clientes, las instituciones financieras están modernizando su infraestructura de datos de pagos para crear experiencias de pago personalizadas, seguras y en tiempo real, todo ello protegiendo a los consumidores del fraude. Esta modernización permite a las entidades financieras ingerir cualquier tipo de datos, poner en marcha servicios más rápidamente, a un coste menor y tener la libertad de ejecutarlos en cualquier entorno, desde el local hasta la multi-nube. Seguridad y gestión de riesgos Los datos son fundamentales para todas las instituciones financieras; se reconocen como un activo esencial para impulsar el crecimiento de los clientes y la innovación. Sin embargo, a medida que aumenta la necesidad de aprovechar los datos de forma eficiente, según el 57% de los responsables de la toma de decisiones , la tecnología heredada que aún sustenta a muchas organizaciones es demasiado cara y no cumple los requisitos de las aplicaciones modernas. Esta infraestructura heredada no sólo es compleja, sino que además es incapaz de cumplir los requisitos de seguridad actuales. Dado el enorme volumen de datos confidenciales de clientes y consumidores que el sector de los servicios financieros maneja a diario -y la estricta normativa que los regula-, la seguridad debe ser la máxima prioridad. El valor percibido de estos datos también convierte a las organizaciones de servicios financieros en el principal objetivo de las filtraciones de datos. La protección contra el fraude, la gestión de riesgos y la lucha contra el blanqueo de dinero son prioridades importantes para cualquier nueva plataforma de datos, según el estudio de Forrester What's Driving Next-Generation Data Platform Adoption in Financial Services . Para hacer frente a estos retos, la adopción de plataformas de datos de nueva generación seguirá creciendo a medida que las instituciones financieras se den cuenta de todo su potencial para gestionar costes, maximizar la seguridad y fomentar la innovación. Descargue el estudio completo de Forrester - What's Driving Next-Generation Data Platform Adoption in Financial Services - para obtener más información.

March 23, 2023