Performance Best Practices: Query Patterns and Profiling

Mat Keep and Henrik Ingo
February 4, 2020
#data model

Welcome to the second in our series of blog posts covering performance best practices for MongoDB. In this series, we are covering key considerations for achieving performance at scale across a number of important dimensions, including:

Designing the right query patterns and profiling query behavior is essential to the smooth running of your application. Profiling will also help you select the most appropriate indexes. We will cover indexing in the next post in this series.

Start by using the latest drivers

MongoDB’s drivers are engineered by the same team that develops the core database. Drivers are updated more frequently than the database, typically every several months. Always use the most recent version of the drivers when possible, and install native extensions if available for your language. Develop a standard procedure for testing and upgrading drivers so that upgrading is naturally a part of your process.

A list of all the MongoDB drivers, along with documentation and source code is available here. You should also join the MongoDB community mailing list to keep track of updates.

Avoid creating large, unbounded documents

As noted in the data modeling section in the first part of this blog series, the maximum size for documents in MongoDB is 16 MB. In practice, most documents are kilobytes or less.

You should avoid certain application patterns that would allow documents to grow unbounded. For example, in an e-commerce application it would be difficult to estimate how many customer reviews each product might receive. It is also typical that only a subset of reviews is displayed to a customer, such as the most popular or the most recent reviews.

Rather than modeling the product and all its reviews as a single document, it would be better to store the subset of reviews in the product document itself for the fastest access. Other, less relevant reviews can be stored in a separate document with a reference or $lookup to the product document. Our previous blog post in this series provides deeper resources into data modeling best practices by use cases.

Issue updates to only modify fields that have changed

Rather than retrieving the entire document in your application, updating fields, then saving the document back to the database, instead issue an update on specific fields. This has the advantage of less network usage and reduced database overhead.

Update multiple array elements in a single operation

With fully expressive array updates, you can perform complex array manipulations against matching elements of an array – including elements embedded in nested arrays – all in a single update operation. Using the arrayFilters option, the update can specify which elements to modify in the array field.

Use MongoDB Atlas Analytics nodes

If your application performs complex or long-running operations, such as reporting or ETL, you may want to isolate analytics queries from the rest of your operational workload. By isolating different workloads, you can ensure different query types never contend for system resources, and avoid analytics queries flushing the working set from RAM.

You can achieve workload isolation by connecting exclusively to dedicated analytics nodes. Visualization tools like MongoDB Charts can be configured to read from analytics nodes only.

If you are running MongoDB on your own infrastructure, then you can configure replica set tags to achieve the same read isolation as Atlas Analytics nodes.

Profile queries with the explain plan

MongoDB’s explain() method enables you to test queries from your application, showing information about how a query will be, or was, resolved, including:

  • Which indexes were used
  • Whether the query was covered by the index or not
  • Whether an in-memory sort was performed, which indicates an index would be beneficial
  • The number of index entries scanned
  • The number of documents returned, and the number read
  • How long the query took to resolve in milliseconds
  • Which alternative query plans were rejected (when using the allPlansExecution mode)

The explain plan will show 0 milliseconds if the query was resolved in less than 1 ms, which is typical for well-tuned queries.

The MongoDB Compass GUI visualizes explain output, making it even easier for you identify and resolve performance issues.

MongoDB Compass visualizes the queries explain plans to help you optimize performance
Figure 1: MongoDB Compass visualizes your explain plans

You can display explain plans via a tree, or review the full raw JSON output. The documentation has more detail on Compass visual explain plans.

Use the MongoDB Query Profiler

The MongoDB Query Profiler helps expose performance issues by displaying slow-running queries (by default, queries that exceed 100ms) and their key performance statistics directly in the Atlas UI.

A chart provides a high-level view of that information that makes it easy to quickly identify outliers and general trends, while a table offers operation statistics by namespace (database and collection) and operation type. You can choose which metric to filter and list operations. This includes operation execution time, documents scanned to returned ratio, whether an index was used, whether an in-memory sort occurred, and more. You can select a specific time frame for the operations displayed, from the past 15 minutes to the past 24 hours.

MongoDB Atlas Query Profiler lets you analyze all slow queries.
Figure 2: MongoDB Atlas Query Profiler

Once you have identified which operations are potentially problematic, the Query Profiler allows you to dig deeper into operation-level statistics to gain more insight into what’s happening. You can view granular information on a specific operation in the context of similar operations, which can help you identify what general optimizations need to be made to improve performance. The Atlas Query Profiler is available without additional cost or performance overhead.

If you are running MongoDB on-premises, Ops Manager – part of MongoDB Enterprise Advanced – also includes a query profiler.

Other Tools and Utilities

  • The MongoDB Database Profiler collects detailed information about operations and commands executed against a running mongod instance. All data collected by the profiler is written to the system.profile collection, a capped collection in the admin database which you can query yourself for insights, and you configure logging levels based on the granularity of data you want to analyze.

  • mtools is a collection of helper scripts to parse, filter, and visualize MongoDB log files. mloginfo will analyze queries against each collection and group common query patterns to help you identify which queries consumed most resources in aggregate. We will dive into indexes in the next post.

Review the Monitoring for MongoDB documentation for a complete overview of the utilities and 3rd party tools you can also use.

What’s Next

That wraps up this latest installment of the performance best practices series. MongoDB University offers a no-cost, web-based training course on MongoDB performance. This is a great way to learn more about optimizing query patterns.

Next up in this series: indexing.

← Previous

Developing Alexa Skills with MongoDB and Golang

Learn how to develop Amazon Alexa Skills that interact with MongoDB using the Go programming language and AWS Lambda.

January 30, 2020
Next →

Security in Government Solutions: Why Secure By Default is Essential

Data security in government agencies is table stakes at this point. Everyone knows it’s essential, both for compliance and data protection purposes. However, most government agencies are working with solutions that require frequent security patches or built-on tools to protect their data. Today, the federal government is pushing its agencies to move to modernize their solutions and improve their security posture. For example, the DHS and Cybersecurity and Infrastructure Security Agency’s recently issued technical rule for modernization of the Protected Critical Information Infrastructure program – a program that provides legal protections for cyber and physical infrastructure information submitted to DHS. “The PCII Program is essential to CISA’s ability to gather information about risks facing critical infrastructure,” said Dr. David Mussington, Executive Assistant Director for Infrastructure Security. “This technical rule modernizes and clarifies important aspects of the Program, making it easier for our partners to share information with DHS. These revisions further demonstrate our commitment to ensuring that sensitive, proprietary information shared with CISA remains secure and protected.” So how can government agencies modernize their data infrastructure and find solutions that not only protect data but also power innovation? Let’s look into a few different strategies. 1. Why secure by default is key Secure by default means that any piece of software uses default security settings that are configured for the highest possible security out of the box. CISA Director Jen Easterly has addressed how using solutions that are secure by default is critical for any organization. “We have to have [multi-factor authentication] by default. We can't charge extra for security logging and [single sign-on],” Easterly said . “We need to ensure that we're coming together to really protect the technology ecosystem instead of putting the burden on those least able to defend themselves.” “The American people have accepted the fact that they’re constantly going to have to update their software,” she said. “The burden is placed on you as the user and that’s what we have to collectively stop.” Easterly is right. Secure-by-design solutions are vital to the success of data protection. The expectation should alway be that solutions have built-in, not bolt-on security features. One approach that’s gaining traction both in the public and private sectors is zero trust environments. In a zero trust environment, the perimeter is assumed to have been breached. There are no trusted users, and no user or device gains trust simply because of its physical or network location. Every user, device, and connection must be continually verified and audited. As the creator of zero trust, security expert John Kindervag, summed it up: “Never trust, always verify.” For government agencies, that means the underlying database must be secure by default, and it needs to limit users’ opportunities to make it less secure. 2. Security isn't just on-prem anymore; cloud is secure, too Cloud can be a scary word for public sector organizations. Trusting your sensitive data to the cloud might feel risky for those who handle some of the country’s most sensitive data. But, cloud providers are stepping up to meet the security needs of government agencies. There is no need to fear the cloud anymore. Government agencies and other public sector organizations nationwide are navigating cloud modernization through the lens of increased cybersecurity requirements outlined in the 2021 Executive Order on Improving the Nation’s Cybersecurity . “The Federal Government must adopt security best practices; advance toward Zero Trust Architecture; accelerate movement to secure cloud services, including Software as a Service (SaaS), Infrastructure as a Service (IaaS), and Platform as a Service (PaaS); centralize and streamline access to cybersecurity data to drive analytics for identifying and managing cybersecurity risks; and invest in both technology and personnel to match these modernization goals.” Also, the major cloud providers are well established, purpose-built options for government users. AWS GovCloud, for example, is more than a decade old and was “ the first cloud provider to build cloud infrastructure specifically designed to meet U.S. government security and compliance needs.” This push by the federal government toward cloud modernization and increased cybersecurity will be a catalyst in upcoming years for rapid cloud adoption and greater dependence on cloud solutions designed specifically for government users. 3. Security features purpose-built for goverment needs are essential Government agencies are held to a higher standard than those in the private sector. From data used in sometimes life-or-death missions to data for students building their futures in educational institutions (and everything in between), security has real-world consequences. Today, security is non-negotiable and like we explored above, it’s especially crucial that public sector entities have built-in security measures to keep data protected. So, what built-in features should you look for? Network isolation and access It’s critical that your data and underlying systems are fully isolated from other organizations using the same cloud provider. Database resources should be associated with a user group, which is contained in its own Virtual Private Cloud (VPC), and access should be granted by IP access lists, VPC peering, or private endpoints. Encyption in flight, at rest, and in use Encryption should be the standard. For example, when using MongoDB Atlas, all network traffic is encrypted using Transport Layer Security (TLS). Encryption for data at rest is automated using encrypted storage volumes. Customers can use field-level encryption to encrypt sensitive workloads which enables you to encrypt data in your application before you send it over the network to MongoDB clusters. Users can bring their own encryption keys for an additional level of control. Granular database auditing Granular database auditing allows administrators to answer detailed questions about systems activity by tracking all commands against the database. This ensures you always know who has access to what data and how they’re using it. Multi-factor authentication User credentials should always be stored using industry-standard and audited one-way hashing mechanisms, with multi-factor authentication options including SMS, voice call, a multi-factor app, or a multi-factor device, ensuring only approved users have access to your data. MongoDB Atlas for Government: Purpose-built for public sector As we’ve discussed, solutions that are purpose-built with built-in security are ideal for government agencies, and choosing the right one is the best way to keep sensitive data protected. MongoDB Atlas for Government on AWS GovCloud recently secured its FedRAMP Moderate authorization thanks to these security measures built into the solution. FedRAMP is a government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services. To ensure the utmost levels of security, Atlas for Government is an independent, dedicated environment for the U.S. public sector, as well as ISVs looking to build U.S. public sector offerings. Public Sector organizations carry a heavy burden when it comes to keeping data protected. However, with the right data platform underpinning modern applications – a platform with built-in security features – progress doesn’t mean you have to compromise on security. Want to learn more about data protection best practices for public sector organizations? Attend our upcoming webinar on April 12 for deeper insight .

March 28, 2023