MongoDB named a leader in The Forrester Wave™: Big Data NoSQL, Q3 2016

Eliot Horowitz
August 17, 2016 | Updated: September 10, 2018
#Business#Company

Today, Forrester released The Forrester Wave™: Big Data NoSQL, Q3 2016, recognizing MongoDB as a Leader based on our performance in the current offering, strategy, and market presence categories. The report said that "MongoDB remains the most popular NoSQL database."

It’s always gratifying to see our efforts acknowledged, but beyond our current position as the most popular non-relational database, it is my view that this Forrester Wave report endorses our long-term strategy as clear and on-target. A little over a year ago, I concluded MongoDB World 2015 with a claim that we had entered a new era in which it was reasonable for MongoDB to be an organization's default database; I believe that this recognition shows that we’re getting there.

The world is ready for a document database to be its default. 61% of the enterprises surveyed by Forrester are using, planning to use, expanding or upgrading to NoSQL over the next 12 months, and we are confident that MongoDB will continue to be the most popular choice. These enterprises have strategic needs that can only be met by a non-relational database, but they must be prudent about where they invest their fiscal and intellectual capital. They don’t want to stitch together a host of new and disparate technologies, each with its own API and narrow band of appropriate use cases, and take on work to re-implement solutions that were working fine in their relational ecosystem.

We developed MongoDB with this in mind, which is why it excels at so many workloads. Our document model is a superset of other data models, including key-value, graph, object, and relational, and we natively support complex manipulations on these data with operators like $lookup and our new graph operators in 3.4. Our replication and sharding architecture, pluggable storage engine framework, and configurable read and write behavior mean that an entire spectrum of data semantics can be achieved through configuration, rather than by mixing and matching from a grab-bag of technologies. And because an unconstrained dynamic schema can sometimes be too flexible, features like document validation and tools like MongoDB Compass provide the integrity checking, schema visualization, query development and performance optimization that DBAs often miss in non-relational solutions.

We are also mindful of the investment that enterprises have made in the business intelligence ecosystem that surrounds their databases. Our BI Connector allows enterprises to leverage tools like Tableau to derive insights from their data. Protecting investments in existing tools, though, doesn’t mean relying on them exclusively. We're also innovating in the next generation of analytics, machine learning, and streaming with our new MongoDB Connector for Apache Spark.

Enterprises also require industrial-grade management solutions for their databases, and MongoDB has met this need with Ops Manager for on-premises management and Cloud Manager for hybrid deployments. Both of these offer monitoring, backup, and management of MongoDB clusters, making it easy to spin up a single instance to experiment with or run a massive cluster with shards spread across the globe.

But these days even enterprises are starting to run their infrastructure entirely in the cloud, and we think this operational model suits a large number of teams. That is why we created our database as a service, MongoDB Atlas: the simplest, most robust, and most cost effective way to run MongoDB in the Cloud. Using Atlas, enterprises can spin up a fully managed, monitored, and backed up cluster in under five minutes. Atlas is available today on AWS, with support for Azure and GCP coming soon. Now, regardless of what type of infrastructure an enterprise wants to run, they have the flexibility to deploy and manage MongoDB with ease.

After all, your data should serve you, not the other way around. We continually build and evolve MongoDB to deliver that vision, which is why MongoDB is already in use by more than half of all Fortune 100 companies.

So thanks to all of our customers, users, and community contributors, for investing in us, for supporting us, for demanding more and more of MongoDB, for pushing it further, into every crazy new use case. We’re right behind you.

About the Author, Eliot Horowitz

Eliot is CTO and Co-Founder of MongoDB. He is one of the core MongoDB kernel committers. Previously, he was Co-Founder and CTO of ShopWiki. Eliot developed the crawling and data extraction algorithm that is the core of its innovative technology. He has quickly become one of Silicon Alley's up and coming entrepreneurs and was selected as one of BusinessWeek's Top 25 Entrepreneurs Under Age 25 nationwide in 2006. Earlier, Eliot was a software developer in the R&D group at DoubleClick (acquired by Google for $3.1 billion). Eliot received a BS in Computer Science from Brown University.

← Previous

MongoDB Atlas Best Practices: Part 4

Operational Management and Securing your Deployment MongoDB Atlas radically simplifies the operation of MongoDB. As with any hosted database as a service there are still decisions you need to take to ensure the best performance and availability for your application. This blog series provides a series of recommendations that will serve as a solid foundation for getting the most out of the MongoDB Atlas service. We’ll cover four main areas over this series of blog posts: In part 1 , we got started by preparing for our deployment, focusing specifically on schema design and application access patterns. In part 2 , we discussed additional considerations as you prepare for your deployment, including indexing, data migration and instance selection. In part 3 , we provided a deep dive into how you scale your MongoDB Atlas deployment, and achieve your required availability SLAs. In this final instalment, we’ll wrap up with best practices for operational management and ensuring data security. If you want to get a head start and learn about all of these topics now, just go ahead and download the MongoDB Atlas Best Practices guide . Managing MongoDB Atlas: Provisioning, Monitoring and Disaster Recovery Created by the engineers who develop the database, MongoDB Atlas is the simplest way to run MongoDB, making it easy to deploy, monitor, backup, and scale MongoDB. MongoDB Atlas incorporates best practices to help keep managed databases healthy and optimized. They ensure operational continuity by converting complex manual tasks into reliable, automated procedures with the click of a button: Deploy. Using your choice of instance size, number of replica set members, and number of shards Scale. Add capacity, without taking the application offline Point-in-time, Scheduled Backups. Restore complete running clusters to any point in time with just a few clicks, because disasters aren't predictable Performance Alerts. Monitor system metrics and get custom alerts Deployments and Upgrades All the user needs to do in order for MongoDB Atlas to automatically deploy the cluster is to select a handful of options: Instance size Storage size (optional) Storage speed (optional) Data volume encryption (optional) Number of replicas in the replica set Number of shards (optional) Automated backups (optional) The database nodes will automatically be kept up date with the latest stable MongoDB and underlying operating system software versions; rolling upgrades ensure that your applications are not impacted during upgrades. Monitoring & Capacity Planning System performance and capacity planning are two important topics that should be addressed as part of any MongoDB deployment. Part of your planning should involve establishing baselines on data volume, system load, performance, and system capacity utilization. These baselines should reflect the workloads you expect the system to perform in production, and they should be revisited periodically as the number of users, application features, performance SLA, or other factors change. Featuring charts and automated alerting, MongoDB Atlas tracks key database and system health metrics including disk free space, operations counters, memory and CPU utilization, replication status, open connections, queues, and node status. Historic performance can be reviewed in order to create operational baselines and to support capacity planning. Integration with existing monitoring tools is also straightforward via the MongoDB Atlas RESTful API, making the deep insights from MongoDB Atlas part of a consolidated view across your operations. *Figure 1: Database monitoring with MongoDB Atlas GUI* MongoDB Atlas allows administrators to set custom alerts when key metrics are out of range. Alerts can be configured for a range of parameters affecting individual hosts and replica sets. Alerts can be sent via email, webhooks, Flowdock, HipChat, and Slack or integrated into existing incident management systems such as PagerDuty. When it's time to scale, just hit the CONFIGURATION button in the MongoDB Atlas GUI and choose the required instance size and number of shards – the automated, on-line scaling will then be performed. Things to Monitor MongoDB Atlas monitors database-specific metrics, including page faults, ops counters, queues, connections and replica set status. Alerts can be configured against each monitored metric to proactively warn administrators of potential issues before users experience a problem. The MongoDB Atlas team are also monitoring the underlying infrastructure, ensuring that it is always in a healthy state. Application Logs And Database Logs Application and database logs should be monitored for errors and other system information. It is important to correlate your application and database logs in order to determine whether activity in the application is ultimately responsible for other issues in the system. For example, a spike in user writes may increase the volume of writes to MongoDB, which in turn may overwhelm the underlying storage system. Without the correlation of application and database logs, it might take more time than necessary to establish that the application is responsible for the increase in writes rather than some process running in MongoDB. Page Faults When a working set ceases to fit in memory, or other operations have moved working set data out of memory, the volume of page faults may spike in your MongoDB system. Disk Beyond memory, disk I/O is also a key performance consideration for a MongoDB system because writes are journaled and regularly flushed to disk. Under heavy write load the underlying disk subsystem may become overwhelmed, or other processes could be contending with MongoDB, or the storage speed chosen may be inadequate for the volume of writes. CPU A variety of issues could trigger high CPU utilization. This may be normal under most circumstances, but if high CPU utilization is observed without other issues such as disk saturation or pagefaults , there may be an unusual issue in the system. For example, a MapReduce job with an infinite loop, or a query that sorts and filters a large number of documents from the working set without good index coverage, might cause a spike in CPU without triggering issues in the disk system or pagefaults . Connections MongoDB drivers implement connection pooling to facilitate efficient use of resources. Each connection consumes up to 1MB of RAM by default, so be careful to monitor the total number of connections so they do not overwhelm RAM and reduce the available memory for the working set. This typically happens when client applications do not properly close their connections, or with Java in particular, that relies on garbage collection to close the connections. Op Counters The utilization baselines for your application will help you determine a normal count of operations. If these counts start to substantially deviate from your baselines it may be an indicator that something has changed in the application, or that a malicious attack is underway. Queues If MongoDB is unable to complete all requests in a timely fashion, requests will begin to queue up. A healthy deployment will exhibit very short queues. If metrics start to deviate from baseline performance, requests from applications will start to queue. The queue is therefore a good first place to look to determine if there are issues that will affect user experience. Shard Balancing One of the goals of sharding is to uniformly distribute data across multiple servers. If the utilization of server resources is not approximately equal across servers there may be an underlying issue that is problematic for the deployment. For example, a poorly selected shard key can result in uneven data distribution. In this case, most if not all of the queries will be directed to the single mongod that is managing the data. Furthermore, MongoDB may be attempting to redistribute the documents to achieve a more ideal balance across the servers. While redistribution will eventually result in a more desirable distribution of documents, there is substantial work associated with rebalancing the data and this activity itself may interfere with achieving the desired performance SLA. If in the course of a deployment it is determined that a new shard key should be used, it will be necessary to reload the data with a new shard key because designation and values of the shard keys are immutable. To support the use of a new shard key, it is possible to write a script that reads each document, updates the shard key, and writes it back to the database. Replication Lag Replication lag is the amount of time it takes a write operation on the primary replica set member to replicate to a secondary member. A small amount of delay is normal, but as replication lag grows, significant issues may arise. If this is observed then replication throughput can be increased by moving to larger MongoDB Atlas instances or adding shards. Disaster Recovery: Backup & Restore A backup and recovery strategy is necessary to protect your mission-critical data against catastrophic failure, such as a software bug or a user accidentally dropping collections. With a backup and recovery strategy in place, administrators can restore business operations without data loss, and the organization can meet regulatory and compliance requirements. Taking regular backups offers other advantages, as well. The backups can be used to seed new environments for development, staging, or QA without impacting production systems. MongoDB Atlas backups are maintained continuously, just a few seconds behind the operational system. If the MongoDB cluster experiences a failure, the most recent backup is only moments behind, minimizing exposure to data loss. mongodump In the vast majority of cases, MongoDB Atlas backups deliver the simplest, safest, and most efficient backup solution. mongodump is useful when data needs to be exported to another system, when a local backup is needed, or when just a subset of the data needs to be backed up. mongodump is a tool bundled with MongoDB that performs a live backup of the data in MongoDB. mongodump may be used to dump an entire database, collection, or result of a query. mongodump can produce a dump of the data that reflects a single moment in time by dumping the oplog entries created during the dump and then replaying it during mongorestore , a tool that imports content from BSON database dumps produced by mongodump . Integrating MongoDB with External Monitoring Solutions The MongoDB Atlas API provides integration with external management frameworks through programmatic access to automation features and monitoring data. APM Integration Many operations teams use Application Performance Monitoring (APM) platforms to gain global oversight of their complete IT infrastructure from a single management UI. Issues that risk affecting customer experience can be quickly identified and isolated to specific components – whether attributable to devices, hardware infrastructure, networks, APIs, application code, databases and, more. The MongoDB drivers include an API that exposes query performance metrics to APM tools. Administrators can monitor time spent on each operation, and identify slow running queries that require further analysis and optimization. In addition, MongoDB Atlas provides packaged integration with the New Relic platform. Key metrics from MongoDB Atlas are accessible to the APM for visualization, enabling MongoDB health to be monitored and correlated with the rest of the application estate. *Figure 2: MongoDB integrated into a single view of application performance* As shown in Figure 1, summary metrics are presented within the APM’s UI. Administrators can also run New Relic Insights for analytics against monitoring data to generate dashboards that provide real-time tracking of Key Performance Indicators (KPIs). Security As with all software, MongoDB administrators must consider security and risk exposure for a MongoDB deployment. There are no magic solutions for risk mitigation, and maintaining a secure MongoDB deployment is an ongoing process. Defense in Depth A Defense in Depth approach is recommended for securing MongoDB deployments, and it addresses a number of different methods for managing risk and reducing risk exposure. MongoDB Atlas features extensive capabilities to defend, detect, and control access to MongoDB, offering among the most complete security controls of any modern database: User Rights Management. Control access to sensitive data using industry standard mechanisms for authentication and authorization at the database level Encryption. Protect data in motion over the network and at rest in persistent storage To ensure a secure system right out of the box, authentication and IP Address whitelisting are automatically enabled. Review the security section of the MongoDB Atlas documentation to learn more about each of the security features discussed below. IP Whitelisting Clients are prevented from accessing the database unless their IP address (or a CIDR covering their IP address) has been added to the IP whitelist for your MongoDB Atlas group. Authorization MongoDB Atlas allows administrators to define permissions for a user or application, and what data it can access when querying MongoDB. MongoDB Atlas provides the ability to provision users with roles specific to a database, making it possible to realize a separation of duties between different entities accessing and managing the data. Additionally, MongoDB's Aggregation Framework Pipeline includes a stage to implement Field-Level Redaction, providing a method to restrict the content of a returned document on a per-field level, based on user permissions. The application must pass the redaction logic to the database on each request. It therefore relies on trusted middleware running in the application to ensure the redaction pipeline stage is appended to any query that requires the redaction logic. Encryption MongoDB Atlas provides encryption of data in flight over the network and at rest on disk. Support for SSL/TLS allows clients to connect to MongoDB over an encrypted channel. Clients are defined as any entity capable of connecting to MongoDB Atlas, including: Users and administrators Applications MongoDB tools (e.g., mongodump, mongorestore) Nodes that make up a MongoDB Atlas cluster, such as replica set members and query routers. Data at rest can optionally be protected using encrypted data volumes. Wrapping Up This brings us to the end of our 4-part blog series. As you’ve seen, MongoDB Atlas automates the operational tasks that usually burdens the user, freeing you up to focus on what you do best – delivering great applications. There remain some tasks that will keep your application running smoothly and quickly; this blog series has described those best practices. Collectively, they will help you get the most out of the database service. Remember, if you want to get a head start and learn about all of our recommendations now, just go ahead and download: Download the MongoDB Atlas Best Practices Guide

August 17, 2016
Next →

Congratulations to the 2023 APAC Innovation Award Winners

I’m thrilled to announce the nine winners of the 2023 MongoDB APAC Innovation Awards . The MongoDB Innovation Awards honor projects and people who dream big. They celebrate the groundbreaking use of data to build compelling applications and the creativity of professionals expanding the limits of technology with MongoDB. This year, we have broken the awards down regionally to celebrate organizations in APAC, from startups to industry-leading enterprises, across a wide variety of industries, who are delivering big results. We are delighted to announce the winners below: 2023 MongoDB APAC Innovation Award Winners: Positive Impact Open Government Products Open Government Products (OGP) is an in-house team of engineers, designers, and product managers, who is a part of the Singapore Government, and is responsible for building technologies for the public good. OGP used MongoDB’s developer data platform, MongoDB Atlas to create its digital form builder, FormSG. Used by the Singapore government and public healthcare institutions, FormSG securely collects data from residents and businesses and helps public officers to create digital government forms in minutes. It eliminates the use of paper forms and the manual process of transcribing physical documents, which had raised concerns around data privacy and protection. During the pandemic, FormSG enabled public officers to collect more than 100,000 daily temperature declarations nationwide. Today, FormSG has served more than 120,000 public officers from 155 agencies and it has created more than 500,000 digital forms to help the government collect data on travel and health declarations by visitors to the country, applications for COVID-19 swab tests, and applications for financial assistance. Organization Transformation Bendigo and Adelaide Bank Bendigo and Adelaide Bank is one of Australia’s largest banks, with around 7,000 employees helping more than 2.2 million customers achieve their financial goals. The bank has been on a multi-year journey of transformation using MongoDB's developer data platform to improve efficiency and deliver a better customer experience as they fulfill their vision to become Australia’s bank of choice. Recently, the cloud team launched Ready-Set-MongoDB (or RSM). This event-driven framework allows developers to streamline the consumption of internal or external APIs, and applies data transformations and storage automatically within a MongoDB collection of their choice. Using MongoDB Atlas Search, the bank also enabled developers to gain insights across its multi-cloud deployments, identifying cost savings, and providing inventory information to account owners and technical stakeholders. Within the first 18 months of launching these programmes, the automation had saved the organization more than 1,100 developers days. It also helped reduce human involvement, removed stale data, and allowed engineers to focus on the things that matter. The development of Ready-Set-MongoDB is ongoing and improving, as new Bendigo multi-cloud challenges arise and new MongoDB products are released. The application is a perfect representation of how Bendigo's Technology Department is using modern technology, rapid development, and innovation-led problem solving to drive organizational transformation. Heroes in Health Redcliffe Lifetech Private Limited Over the last few years, Redcliffe Labs has become India's fastest growing technology-driven diagnostics service provider. Redcliffe Labs is on a mission to serve 500 Million Indians by 2030 with fusion of technology and world- class laboratories. The company already serves thousands of people daily, with more than 73 labs and close to 1500 walk-in centers across 180 cities. Redcliffe Labs has relied on MongoDB Atlas’ flexible document model to power its innovative Smart Health Report, a patient resource that provides a number of indicators and trackers to gauge holistic health. The MongoDB developer data platform's best in class security, compliance, and privacy controls allows Redcliffe's team to confidently handle even the most sensitive patient data. MongoDB Atlas takes care of many of the traditional database management challenges, which means that developers can spend their time building diagnostics for patients, rather than managing databases. Redcliffe Labs is focusing on incorporating next-generation technologies in the diagnostics space with an AI platform that will make Interactive Diagnostics reports, Advanced Health Profiling and more detailed Diagnostics and Health Alerts. Industry Disruptor Cathay Pacific Cathay Pacific , Hong Kong’s home carrier operating in more than 60 destinations worldwide, has been on an impressive journey to become one of the very first airlines to create a truly paperless flight deck. Until recently, a flight from Hong Kong to New York would require a crew to review more than 150 pages of finely printed text and charts before their flight and make ongoing updates throughout the trip. In 2019, Cathay Pacific conducted the first zero paper flight, removing 50kg of manuals, charts, maps, and flight briefing paperwork. They achieved this enormous feat with the help of one seamless and highly customized iPad application: Flight Folder. Built on MongoDB Atlas, Flight Folder is designed to improve the pilot briefing experience. MongoDB helped consolidate dozens of different information sources into one place, and made it possible for flight crews to easily share their experiences with others. It also included a digital refueling feature that helps crews become much more efficient with fueling strategies – saving significant flight time and costs. The use of MongoDB Device Sync enables seamless syncing and no data loss even when the app goes on- and offline mid-flight. Since the Flight Folder launch, Cathay Pacific has completed more than 340,000 flights with full digital integration in the flight deck. In addition to the greatly improved flight crew experience, flight times have been reduced, and digital refueling saves eight minutes of ground time on average. All these efficiencies have helped the company avoid the release of 15,000 tons of carbon. From Batch to Real-Time Adani Digital Labs Adani Digital Labs is the India-based digital innovation arm of the larger Adani group. The lab’s team's mission is to create one single platform – a SuperApp called AdaniOne – to empower a billion stories in India. To address several use cases and the huge scale that will be required by the superapp, the Adani Digital team selected MongoDB Atlas as its the main transactional database that will further enhance the application. A key component of the app is how it can bring together disparate data in order to provide a single view of activity across the application. In the first process, developers had taken out the data in batches and sent it to their database However, this was too slow and unpredictable as far as business requirements are concerned. Also, the consolidated view of customer history, orders, inventory, and supply chain network updates was likely to impact their customer's ability to generate revenue. Therefore, in order to find a better solution, Adani Digital Labs built a more modern architecture in line with MongoDB. Using MongoDB's Change Streams and the data platform's native Kafka connector, they created an event-based architecture that pushes the data out in real-time for analysis. Adani Digital Labs is still in the early phases of the SuperApp's rollout and collaborating with MongoDB as its developer data platform continues to help the firm to grow and deliver insights in real time. Industry 4.0 Dongwha Founded in 1948, the Dongwha Group has evolved from a singular focus on the wood and timber industry into a global leader across a number of sectors including building materials, chemicals and media. As part of its wider digital transformation strategy, Dongwha required smarter factories that would improve and optimize their production efficiency. Dongwha built an innovative Smart Factory Software platform that collects and analyzes data to enhance quality and production management capabilities. Originally, the platform was built with the community version of MongoDB. However, in order to scale and adapt, the team recently migrated to MongoDB Atlas in the cloud. This enabled them to store large volumes in the fastest and most secure way, optimize their solution for time series data, and make it easy to run machine learning across their data. Dongwha completed the migration seamlessly, without any disruption or downtime to their factories, and it has now been launched across five different sites. Over the last year, the application has significantly increased its availability and reliability while performance has improved by as much as 6x . As they look to the future, Dongwha plans to roll out the software to more of its international factories. Digital Native myBillBook India is home to more than 60 million small and medium-sized businesses (SMBs) but only a small portion of those SMBs are taking advantage of digitization and many still operate using pen and paper. In addition, many businesses in India still struggle with fluctuations in internet services, outages, and latency. FloBiz is on a mission to change that with myBillBook , a one-stop solution that helps SMBs create professional invoices, manage stock, collect payments, automate reminders through smart banking, engage with their customers, manage staff attendance and payroll and generate more than 25 business reports for accounting and decision making. The app is also mobile-first, so businesses can access them from their mobile devices and allows users to manage billing and inventory in both online and offline environments. The myBillbook app is powered by MongoDB Atlas, providing the flexible and scalable foundation for the business to do everything from building new features to performing complex analytical queries. In addition, MongoDB Realm, the mobile database within the data platform, supports offline usage and syncing to ensure there is never data loss or functionality for users due to poor internet connection. Because of its success in supporting customers with business critical operations, more than 6.5 million business owners in India are now using myBillbook for their billing, accounting, collection and business growth. Customer Focused KASIKORN Business-Technology Group Established in 1945, Kasikornbank (KBank) is one of the largest and oldest banks in Thailand. Their mission is to strive towards service excellence and empower every customer’s life and business. One of KBank’s subsidiaries, KASIKORN Business-Technology Group (KBTG) , developed a mobile banking application – MAKE by KBank. MongoDB Atlas’ flexibility and ease of development enabled MAKE’s development team to choose the best type of database for its tasks, to automate data tiering with Atlas Online Archive, and to reduce hours spent on operational maintenance. With more time to focus on delivering new innovations to customers, they created unique features like Cloud Pocket which can allocate funds into unlimited customizable pockets for separate usage. They also built Pop Pay, a feature that allows users to easily search for nearby friends and transfer money by clicking their profile picture as well as “Expense Summary" a spending analysis services that helps inform and manage users’ financial habits. As of January 2023, MAKE has acquired more than 1 million users, and increased the number of transactions in MAKE from 900,000 to more than 7.5 million in a span of one year. Massive Scale China Mobile China Mobile provides mobile voice and multimedia services via its nationwide mobile telecommunications network across mainland China and Hong Kong. It is the world's largest mobile network operator by total number of subscribers. The telecommunications leader is using MongoDB to support one of its largest and most critical push services, which sends out billing details to more than 1 billion users every month. Prior to MongoDB, the tech team relied on Oracle, but as the user numbers increased, performance degraded. Despite large investments, it was still taking too long to do basic requests like finalize and deliver bills to users. In 2019, after comprehensive testing, China Mobile migrated to MongoDB. By taking advantage of MongoDB's native sharding, they were able to improve performance by 80% and go from 50 Oracle machines, to just 12 machines for the same workload. The service now handles all current requirements and is set up to scale with future growth. With the support of MongoDB, China Mobile is growing steadily,with more than 168 million monthly users and has one of the highest customer satisfaction scores in the China Mobile group.

March 30, 2023