What’s New in MongoDB 3.4, Part 1: Multimodel Done Right

Mat Keep
December 21, 2016 | Updated: December 28, 2016
#Technical#Company

MongoDB 3.4 is now Generally Available (GA) and ready for production deployment!

MongoDB 3.4 is the latest release of the industry’s fastest growing database. It offers a major evolution in capabilities and enhancements that enable you to address emerging opportunities and use cases. This 3-part blog series aims to help you navigate everything that is new, and provides the most important resources to get you started:

  • Part 1 covers the extended multimodel capabilities of MongoDB 3.4, including native graph processing, faceted navigation, rich real-time analytics, and powerful connectors for BI and Apache Spark.
  • Part 2 discusses enhanced capabilities to support mission-critical applications, including geo-distributed MongoDB zones, elastic clustering, tunable consistency, and enhanced security controls.
  • Part 3 concludes with modernized DBA and Ops tooling available in the new release.

If you want to get the detail now on everything the new release offers, download the What’s New in MongoDB 3.4 white paper.

Why Multimodel?

Rather than the monolithic codebases of the past, today’s applications are increasingly being decomposed into loosely coupled suites of microservices, each implementing specific functionality within an application. Different services can place very different demands on the database used – from simple key-value lookups to complex analytics, aggregations, and graph traversals, through to rich search queries. Some data may need to be stored only in-memory for predictable low latency, while other data sets may need to be encrypted on disk for regulatory compliance. Data sets may vary from billions of small records, each just several KBs in size, to the management of large, multi-MB objects.

To try and tame the complexity that would come from using a multitude of storage technologies, the industry is moving towards the concept of “multimodel” databases. Such designs are based on the premise of presenting multiple data models within the same platform, thereby serving diverse application requirements. However, many self-described multimodel database are little more than a collection of discrete technologies for data storage, search, and analytics, each with its own domain specific language, API and deployment requirements, and working on its own copy of the data. This approach to multimodel fails to offer much of an improvement over just running multiple independent databases, imposing high complexity, overhead, friction, and cost for developers and operations teams.

MongoDB Takes a Different Approach

How does MongoDB differ?

  • MongoDB’s flexible document data model presents a superset of other database models. It allows data be represented as simple key-value pairs and flat, table-like structures, through to rich documents and objects with deeply nested arrays and sub-documents.
  • With an expressive query language, documents can be queried in many ways – from simple lookups to creating sophisticated processing pipelines for data analytics and transformations, through to faceted search, JOINs and graph traversals.
  • With a flexible storage architecture, application owners can deploy storage engines optimized for different workload and operational requirements.

MongoDB’s approach to delivering multimodel significantly reduces developer and operational complexity, compared to running multiple, separate technologies to satisfy diverse application requirements. Users can leverage the same MongoDB query language, data model, scaling, security, and operational tooling across different applications, all within a single, integrated database platform.

MongoDB 3.4 introduces new native graph processing, faceted navigation, multi-language collations, additional aggregation pipeline operators, a new decimal data type, along with enhanced connectors for BI and Apache Spark integration.

Native Graph Processing

Applications storing data in MongoDB frequently contain data that represents graph or tree type hierarchies. These connections can be as simple as a management reporting chain in a HR application, or as complex as multi-directional, deeply nested relationships maintained by social networks, master data management, recommendation engines, disease taxonomy, fraud detection, and more. While special purpose graph databases are effective at storing and querying graph data, it’s often desirable to store and traverse graph data directly in MongoDB. Here it can be processed, queried, and analyzed alongside all other operational data in real time, without the complexity of duplicating data across two separate databases.

Graph and hierarchical data is commonly queried to uncover indirect or transitive relationships. For example, if company “A” is owned by company “B”, and “B” is owned by company “C”, then “C” indirectly owns company “A”. MongoDB 3.4 offers this functionality via a new aggregation stage called $graphLookup to recursively lookup a set of documents with a specific defined relationship to a starting document. Developers can specify the maximum depth for the recursion, and apply additional filters to only search nodes that meet specific query predicates. $graphLookup can recursively query within a single collection, or across multiple collections.

Review the documentation to learn more about the MongoDB $graphLookup operator for graph processing.

Faceted Navigation

Faceting is a popular analytics and search capability that allows an application to group information into related categories by applying multiple filters to query results. Facets allow users to narrow their search results by selecting a facet value as a filter criteria. Facets also provide an intuitive interface to exploring a data set, and allow convenient navigation to data that is of most interest.

Most databases need to execute multiple GROUP_BY statements to render facets, resulting in long running queries and poor user experience. MongoDB 3.4 introduces new aggregation pipeline stages for the bucketing, grouping and counting of one or more facets in a single round trip to the database. As a result, developers can generate richer and intuitive experiences to help users navigate complex data sets. Review the documentation to learn more about MongoDB faceted navigation.

Multi-Language Collations

Applications addressing global audiences require handling content that spans many languages. Each language has different rules governing the comparison and sorting of data. In order to create intuitive, localized user experiences, applications must handle non-English text with the appropriate rules for that language. For example, French has detailed rules for sorting names with accents on them. German phonebooks order words differently than the German dictionary.

MongoDB 3.4 significantly expands language support capabilities to allow users to build applications that adhere to language-specific comparison rules. Support for collations – the rules governing text comparisons and sorting – has been added throughout the MongoDB Query Language and indexes for over 100 different languages and locales. Each collation can also be further customized to provide precise control over case sensitivity, numeric ordering, whitespace handling, and more.

Developers can specify collation for a collection, an index, a view, or for specific operations that support collation (i.e. find, aggregate, update). You can learn more about collation in MongoDB from the documentation.

Aggregation Pipeline Enhancements

MongoDB developers and data engineers rely on the aggregation pipeline due to its power and flexibility in enabling sophisticated processing and manipulation demanded by real-time analytics and data transformations. MongoDB 3.4 continues to extend the aggregation pipeline by adding new capabilities within the database that simplify application-side code, as well as optimizer enhancements that improve performance.

In addition to the graph and facet features described earlier, many other expressions are added in MongoDB 3.4. These expressions address string manipulation, array handling, type handling, and schema detection and transformation:

  • String handling includes expressions for splitting and manipulating strings either in bytes or code points (a code point can represent a single component of the string, e.g, a character, emoji, or formatting instruction).
  • Array expressions allow more sophisticated manipulation and computations on arrays, including parallel array processing.
  • New expressions allow determining types of fields
  • Case/switch expressions for branching
  • Support for ISO week expressions

MongoDB 3.4 also brings additional performance optimizations to the aggregation pipeline. Where possible, the query optimizer automatically moves the $match stage earlier in the pipeline, and combines it with other stages, to increase cases where indexes can be used to filter results sets. In most cases, no modifications of existing queries need to be made.

You can learn more about the many MongoDB 3.4 aggregation pipeline enhancements from the documentation.

Decimal Data Type

Decimal128 is a 16 byte decimal floating-point number format. It is intended for calculations on decimal numbers where high levels of precision are required, such as financial (i.e. tax calculations, currency conversions) and scientific computations. Decimal128 supports 34 decimal digits of significance and an exponent range of −6143 to +6144.

MongoDB 3.4 adds support for the decimal data type which represents decimal128 values. Unlike the double data type, which only stores approximations of decimal values, the decimal data type stores the exact value. For example, a decimal type ("9.99") has a precise value of 9.99, while 9.99 represented as a double would have an actual value of 9.99000000000000021316282072803, thus creating the potential for rounding errors when it is used in calculations.

Decimal type values are treated like any other numeric type, and compare and sort correctly with other types based on actual numeric value. Operations on decimals are implemented in accordance with the decimal128 standard, so a value of 0.10 will retain its trailing zeros while comparing equal to 0.1, 0.10000 and so on.

Review the documentation to learn more about the new MongoDB decimal data type.

Visualizing MongoDB Data

The MongoDB Connector for BI was introduced in November 2015. For the first time analysts, data scientists, and business users were able to seamlessly visualize semi-structured and unstructured data managed in MongoDB, alongside traditional data from their SQL databases, using the same BI tools deployed within millions of enterprises.

Building on its initial release, the Connector for BI has been reengineered to improve performance, simplify installation and configuration, and support Windows.

Figure 1: Uncover new insights with powerful visualizations generated from MongoDB

Performance and scalability has been improved by moving more query execution down to the MongoDB processes themselves. Queries and complex aggregations are executed natively within the database, thus reducing latency and bandwidth consumption. In addition, installation and authentication has been simplified. Users now authenticate as an existing user already declared within MongoDB, no longer needing to create separate username and password credentials within the connector.

The Connector for BI is part of the Advanced Analytics suite available with MongoDB Enterprise Advanced. Review the MongoDB Connector for BI documentation to learn more.

MongoDB Connector for Apache Spark

Following general availability in June 2016, the MongoDB Connector for Apache Spark has been updated to support the latest Spark 2.0 release. Spark 2.0 support in the connector provides access to the new SparkSession entry point, the unified DataFrame and Dataset API, enhanced SparkSQL and SparkR functionality, and the experimental Structured Streaming feature. The connector exposes all of Spark’s libraries, including Scala, Java, Python, and R. MongoDB data is materialized as DataFrames and Datasets for analysis through machine learning, graph, streaming, and SQL APIs.

Already powering sophisticated analytics at organizations including China Eastern Airlines, Black Swan, and x.ai, the MongoDB Connector for Apache Spark takes advantage of MongoDB’s aggregation framework, rich queries, and secondary indexes to extract, filter, and process only the range of data it needs – for example, all customers located in a specific geography. To maximize performance across large, distributed data sets, the connector can co-locate Resilient Distributed Datasets (RDDs) with the source MongoDB node, thereby minimizing data movement across the cluster and reducing latency.

You can download the MongoDB Connector for Apache Spark from GitHub, and sign up for a free Spark course from MongoDB University.

Next Steps

That wraps up the first part of our 3-part blog series. Remember, you can get the detail now on everything packed into the new release by downloading the What’s New in MongoDB 3.4 white paper.

Alternatively, if you’d had enough of reading about it and want to get started now, then:


← Previous

Meet the MongoDB Europe 2016 Diversity Scholars

In November, a group of rising tech stars attended the first annual MongoDB Europe as part of the MongoDB Diversity Scholarship program. The MongoDB Diversity Scholarship is one of our initiatives to change the ratio in technology . At MongoDB Europe, the Diversity Scholars got complimentary admission to the conference, a lunch sessions with other Scholars, and speed mentoring with MongoDB speakers. Additionally, they received a MongoDB certification voucher and three-month access to on-demand MongoDB University courses. Read on to meet the MongoDB Europe 2016 Diversity Scholars, and to find out how you can apply to the scholarship at MongoDB World in Chicago, June 20-21, 2017. Ousmane Barry, Software Developer, Novencia I’m originally from Senegal and came to France to pursue my higher education. Currently, I’m a Software Engineer at a consulting company, where I help the teams to develop their technical level through Software Craftsmanship. How I started using MongoDB : Like the majority of developers, I come from the world of SQL. I discovered MongoDB through a discussion with a colleague. At the same time, my company wanted to build skills in the field of NoSQL, so I registered on MongoDB University to learn the technology. After that, I passed the DEV certification, followed by the DBA certification. My giant idea : I would like to work on a Big Data project where you can use all the power and flexibility offered by MongoDB. It would be interesting to work on use cases such as IoT or Single View. -- @EnamsuoBarry Quynh Tran Thanh, Quant Analyst, Commerzbank I’m currently a Quant Analyst working at Commerzbank. In my free time, I love assembling IKEA furniture. How I started using MongoDB : I started using MongoDB at a hackathon, where I had to build an app with my team within 48 hours. MongoDB was the best choice for a user friendly database, and perfect for our app as we used JSON data. My giant idea : Recently, I’ve become very interested in cryptocurrencies such as Bitcoin. I’d like to analyze transactions on the blockchain. For that, I’d like to use MongoDB to easily query for transactions. Soyini Taylor, Database Analyst/Data Architect, Reed Business Information I’m a Database Analyst living in London, currently on a 6-month secondment as a Data Architect. How I started using MongoDB : I wanted to learn MongoDB and got started with the free M102 course on MongoDB University . M102 was a good introduction to how the database works. My giant idea : My goal is to complete the next MongoDB University course, M202. I’m looking forward to MongoDB releasing a sandbox area for those working in operations that are charged only with maintenance of the database. I also love the fact that MongoDB provides a Diversity Scholarship to encourage more involvement in technology by underrepresented groups, and hope to be able to contribute back to the community one day. Want to attend MongoDB World on a Diversity Scholarship? The Diversity Scholarship is now open for [MongoDB World](https://www.mongodb.com/world17 "MongoDB World"). If you’re a member of an underrepresented group in technology, you can apply today! Apply Now

December 20, 2016
Next →

Congratulations to the 2023 APAC Innovation Award Winners

I’m thrilled to announce the nine winners of the 2023 MongoDB APAC Innovation Awards . The MongoDB Innovation Awards honor projects and people who dream big. They celebrate the groundbreaking use of data to build compelling applications and the creativity of professionals expanding the limits of technology with MongoDB. This year, we have broken the awards down regionally to celebrate organizations in APAC, from startups to industry-leading enterprises, across a wide variety of industries, who are delivering big results. We are delighted to announce the winners below: 2023 MongoDB APAC Innovation Award Winners: Positive Impact Open Government Products Open Government Products (OGP) is an in-house team of engineers, designers, and product managers, who is a part of the Singapore Government, and is responsible for building technologies for the public good. OGP used MongoDB’s developer data platform, MongoDB Atlas to create its digital form builder, FormSG. Used by the Singapore government and public healthcare institutions, FormSG securely collects data from residents and businesses and helps public officers to create digital government forms in minutes. It eliminates the use of paper forms and the manual process of transcribing physical documents, which had raised concerns around data privacy and protection. During the pandemic, FormSG enabled public officers to collect more than 100,000 daily temperature declarations nationwide. Today, FormSG has served more than 120,000 public officers from 155 agencies and it has created more than 500,000 digital forms to help the government collect data on travel and health declarations by visitors to the country, applications for COVID-19 swab tests, and applications for financial assistance. Organization Transformation Bendigo and Adelaide Bank Bendigo and Adelaide Bank is one of Australia’s largest banks, with around 7,000 employees helping more than 2.2 million customers achieve their financial goals. The bank has been on a multi-year journey of transformation using MongoDB's developer data platform to improve efficiency and deliver a better customer experience as they fulfill their vision to become Australia’s bank of choice. Recently, the cloud team launched Ready-Set-MongoDB (or RSM). This event-driven framework allows developers to streamline the consumption of internal or external APIs, and applies data transformations and storage automatically within a MongoDB collection of their choice. Using MongoDB Atlas Search, the bank also enabled developers to gain insights across its multi-cloud deployments, identifying cost savings, and providing inventory information to account owners and technical stakeholders. Within the first 18 months of launching these programmes, the automation had saved the organization more than 1,100 developers days. It also helped reduce human involvement, removed stale data, and allowed engineers to focus on the things that matter. The development of Ready-Set-MongoDB is ongoing and improving, as new Bendigo multi-cloud challenges arise and new MongoDB products are released. The application is a perfect representation of how Bendigo's Technology Department is using modern technology, rapid development, and innovation-led problem solving to drive organizational transformation. Heroes in Health Redcliffe Lifetech Private Limited Over the last few years, Redcliffe Labs has become India's fastest growing technology-driven diagnostics service provider. Redcliffe Labs is on a mission to serve 500 Million Indians by 2030 with fusion of technology and world- class laboratories. The company already serves thousands of people daily, with more than 73 labs and close to 1500 walk-in centers across 180 cities. Redcliffe Labs has relied on MongoDB Atlas’ flexible document model to power its innovative Smart Health Report, a patient resource that provides a number of indicators and trackers to gauge holistic health. The MongoDB developer data platform's best in class security, compliance, and privacy controls allows Redcliffe's team to confidently handle even the most sensitive patient data. MongoDB Atlas takes care of many of the traditional database management challenges, which means that developers can spend their time building diagnostics for patients, rather than managing databases. Redcliffe Labs is focusing on incorporating next-generation technologies in the diagnostics space with an AI platform that will make Interactive Diagnostics reports, Advanced Health Profiling and more detailed Diagnostics and Health Alerts. Industry Disruptor Cathay Pacific Cathay Pacific , Hong Kong’s home carrier operating in more than 60 destinations worldwide, has been on an impressive journey to become one of the very first airlines to create a truly paperless flight deck. Until recently, a flight from Hong Kong to New York would require a crew to review more than 150 pages of finely printed text and charts before their flight and make ongoing updates throughout the trip. In 2019, Cathay Pacific conducted the first zero paper flight, removing 50kg of manuals, charts, maps, and flight briefing paperwork. They achieved this enormous feat with the help of one seamless and highly customized iPad application: Flight Folder. Built on MongoDB Atlas, Flight Folder is designed to improve the pilot briefing experience. MongoDB helped consolidate dozens of different information sources into one place, and made it possible for flight crews to easily share their experiences with others. It also included a digital refueling feature that helps crews become much more efficient with fueling strategies – saving significant flight time and costs. The use of MongoDB Device Sync enables seamless syncing and no data loss even when the app goes on- and offline mid-flight. Since the Flight Folder launch, Cathay Pacific has completed more than 340,000 flights with full digital integration in the flight deck. In addition to the greatly improved flight crew experience, flight times have been reduced, and digital refueling saves eight minutes of ground time on average. All these efficiencies have helped the company avoid the release of 15,000 tons of carbon. From Batch to Real-Time Adani Digital Labs Adani Digital Labs is the India-based digital innovation arm of the larger Adani group. The lab’s team's mission is to create one single platform – a SuperApp called AdaniOne – to empower a billion stories in India. To address several use cases and the huge scale that will be required by the superapp, the Adani Digital team selected MongoDB Atlas as its the main transactional database that will further enhance the application. A key component of the app is how it can bring together disparate data in order to provide a single view of activity across the application. In the first process, developers had taken out the data in batches and sent it to their database However, this was too slow and unpredictable as far as business requirements are concerned. Also, the consolidated view of customer history, orders, inventory, and supply chain network updates was likely to impact their customer's ability to generate revenue. Therefore, in order to find a better solution, Adani Digital Labs built a more modern architecture in line with MongoDB. Using MongoDB's Change Streams and the data platform's native Kafka connector, they created an event-based architecture that pushes the data out in real-time for analysis. Adani Digital Labs is still in the early phases of the SuperApp's rollout and collaborating with MongoDB as its developer data platform continues to help the firm to grow and deliver insights in real time. Industry 4.0 Dongwha Founded in 1948, the Dongwha Group has evolved from a singular focus on the wood and timber industry into a global leader across a number of sectors including building materials, chemicals and media. As part of its wider digital transformation strategy, Dongwha required smarter factories that would improve and optimize their production efficiency. Dongwha built an innovative Smart Factory Software platform that collects and analyzes data to enhance quality and production management capabilities. Originally, the platform was built with the community version of MongoDB. However, in order to scale and adapt, the team recently migrated to MongoDB Atlas in the cloud. This enabled them to store large volumes in the fastest and most secure way, optimize their solution for time series data, and make it easy to run machine learning across their data. Dongwha completed the migration seamlessly, without any disruption or downtime to their factories, and it has now been launched across five different sites. Over the last year, the application has significantly increased its availability and reliability while performance has improved by as much as 6x . As they look to the future, Dongwha plans to roll out the software to more of its international factories. Digital Native myBillBook India is home to more than 60 million small and medium-sized businesses (SMBs) but only a small portion of those SMBs are taking advantage of digitization and many still operate using pen and paper. In addition, many businesses in India still struggle with fluctuations in internet services, outages, and latency. FloBiz is on a mission to change that with myBillBook , a one-stop solution that helps SMBs create professional invoices, manage stock, collect payments, automate reminders through smart banking, engage with their customers, manage staff attendance and payroll and generate more than 25 business reports for accounting and decision making. The app is also mobile-first, so businesses can access them from their mobile devices and allows users to manage billing and inventory in both online and offline environments. The myBillbook app is powered by MongoDB Atlas, providing the flexible and scalable foundation for the business to do everything from building new features to performing complex analytical queries. In addition, MongoDB Realm, the mobile database within the data platform, supports offline usage and syncing to ensure there is never data loss or functionality for users due to poor internet connection. Because of its success in supporting customers with business critical operations, more than 6.5 million business owners in India are now using myBillbook for their billing, accounting, collection and business growth. Customer Focused KASIKORN Business-Technology Group Established in 1945, Kasikornbank (KBank) is one of the largest and oldest banks in Thailand. Their mission is to strive towards service excellence and empower every customer’s life and business. One of KBank’s subsidiaries, KASIKORN Business-Technology Group (KBTG) , developed a mobile banking application – MAKE by KBank. MongoDB Atlas’ flexibility and ease of development enabled MAKE’s development team to choose the best type of database for its tasks, to automate data tiering with Atlas Online Archive, and to reduce hours spent on operational maintenance. With more time to focus on delivering new innovations to customers, they created unique features like Cloud Pocket which can allocate funds into unlimited customizable pockets for separate usage. They also built Pop Pay, a feature that allows users to easily search for nearby friends and transfer money by clicking their profile picture as well as “Expense Summary" a spending analysis services that helps inform and manage users’ financial habits. As of January 2023, MAKE has acquired more than 1 million users, and increased the number of transactions in MAKE from 900,000 to more than 7.5 million in a span of one year. Massive Scale China Mobile China Mobile provides mobile voice and multimedia services via its nationwide mobile telecommunications network across mainland China and Hong Kong. It is the world's largest mobile network operator by total number of subscribers. The telecommunications leader is using MongoDB to support one of its largest and most critical push services, which sends out billing details to more than 1 billion users every month. Prior to MongoDB, the tech team relied on Oracle, but as the user numbers increased, performance degraded. Despite large investments, it was still taking too long to do basic requests like finalize and deliver bills to users. In 2019, after comprehensive testing, China Mobile migrated to MongoDB. By taking advantage of MongoDB's native sharding, they were able to improve performance by 80% and go from 50 Oracle machines, to just 12 machines for the same workload. The service now handles all current requirements and is set up to scale with future growth. With the support of MongoDB, China Mobile is growing steadily,with more than 168 million monthly users and has one of the highest customer satisfaction scores in the China Mobile group.

March 30, 2023