Optimizing Your MongoDB Deployment with Performance Advisor
We are happy to announce additional enhancements to MongoDB’s Performance Advisor, now available in MongoDB Atlas , MongoDB Cloud Manager , and MongoDB Ops Manager . MongoDB’s Performance Advisor automatically analyzes logs for slow-running queries and provides index suggestions to improve query performance. In this latest update, we’ve made some key updates, including: A new ranking algorithm and additional performance statistics (e.g., average documents scanned, average documents returned, and average object size) make it easier to understand the relative importance of each index recommendation. Support for additional query types including regexes, negation operators (e.g., $ne, $nin, $not), $count, $distinct, and $match to ensure we cover with optimized index suggestions. Index recommendations are now more deterministic so they are less impacted by time and provide more consistent query performance benefits. Before diving further into MongoDB’s Performance Advisor, let’s look at tools MongoDB provides out of the box to simplify database monitoring. Background Deploying your MongoDB cluster and getting your database running is a critical first step, but another important aspect of managing your database is ensuring that your database is performant and running efficiently. To make this easier for you, MongoDB offers several out-of-the-box monitoring tools , such as the Query Profiler, Performance Advisor, Real-Time Performance Panel, and Metrics Charts, to name a few. Suppose you notice that your database queries are running slower. The first place you might go is to the metrics charts to look at the “Opcounters” metrics to see whether you have more operations running. You might also look at the “Operation Execution Time” to see if your queries are taking longer to run. The “Query Targeting” metric shows the ratio of the number of documents scanned over the number of documents returned. This datapoint is a great measure of the overall efficiency of a query — the higher the ratio, the less efficient the query. These and other metrics can help you identify performance issues with your overall cluster, which you can then use as context to dive a level deeper and perform more targeted diagnostics of individual slow-running queries . MongoDB’s Performance Advisor takes this functionality a step further by automatically scanning your slowest queries and recommending indexes where appropriate to improve query performance. Getting started with Performance Advisor The Performance Advisor is a unique tool that automatically monitors MongoDB logs for slow-running queries and suggests indexes to improve query performance. Performance Advisor also helps improve both your read and write performance by intelligently recommending indexes to create and/or drop (Figure 1). These suggestions are ranked by the determined impact on your cluster. Performance Advisor is available on M10 and above clusters in MongoDB Atlas as well as in Cloud Manager and Ops Manager. Figure 1: Performance Advisor can recommend indexes to create or drop. Performance Advisor will suggest which indexes to create, what queries will be affected by the index, and the expected improvements to query performance. All of these user interactions are available in the user interface directly within Performance Advisor, and indexes can be easily created with just a few clicks. Figure 2 shows additional Performance Advisor statistics about the performance improvements this index would provide. The performance statistics that are highlighted for each index recommendation include: Execution Count: The number of queries per hour that would be covered by the recommended index Avg Execution Time: The average execution time of queries that would be covered by the recommended index Avg Query Targeting: The inefficiency of queries that would be covered by the recommended index, measured by the number of documents or index keys scanned in order to return one document In Memory Sort: The number of in-memory sorts performed per hour for queries that would be covered by the recommended index Avg Docs Scanned: The average number of documents that were scanned by slow queries with this query shape Avg Docs Returned: The average number of documents that were returned by slow queries with this query shape Avg Object Size: The average object size of all objects in the impacted collection If you have multiple index recommendations, they are ranked by their relative impact to query performance so that the most beneficial index suggestion is displayed at the top. This means that the most impactful index is displayed at the top and would be the most beneficial to query performance. Figure 2: Detailed performance statistics. Creating optimal indexes ensures that queries are not scanning more documents than they return. However, creating too many indexes can slow down write performance, as each write operation needs to check each index before writing. Performance Advisor provides suggestions on which indexes to drop based on whether they are unused or redundant (Figure 3). Users also have the option to “hide” indexes as a way to evaluate the impact of dropping an index without actually dropping the index. Figure 3: Performance Advisor shows which indexes are unused or redundant. The Performance Advisor in MongoDB provides a simple and cost-efficient way to ensure you’re getting the best performance out of your MongoDB database. If you’d like to see the Performance Advisor in action, the easiest way to get started is to sign up for MongoDB Atlas , our cloud database service. Performance Advisor is available on MongoDB Atlas on M10 cluster tiers and higher. Learn more from the following resources: Monitor and Improve Slow Queries Monitor Your Database Deployments
Enhancing Atlas Online Archive With Data Expiration and Scheduled Archiving
Atlas's Online Archive feature allows you to set archiving rules to move data that is not frequently accessed from your Atlas cluster to a MongoDB-managed object store. It also allows you to query both your Atlas and Online Archive data in a unified manner without having to worry about the tier in which the data resides. We are enhancing this feature by adding two new capabilities: data expiration and scheduled archiving (in preview). Data expiration: Online Archive makes it easy to tier data out of a live database into an object store, but what if you want to set a second threshold to delete data from the archive entirely? Perhaps you don’t want to store data indefinitely due to costs, or maybe you have a compliance requirement for data expiration that means you need to ensure deletion on a schedule. Previously, there was no option to remove data from the archive except deleting the archive completely, which is not a preferred option for most use cases. With our new data expiration feature, you can specify for how many days data should be stored in the online archive before being deleted. You can set an expiration from the archive as low as seven days and as high as 9,125 days; you can set the archive expiration time through either the Atlas UI or the Admin API. Expiration rules can be edited after creation, if needed. Note that once the data is deleted from the archive there is no way to recover it, so you must define your rules carefully. Archiving with the data expiration feature Scheduled archiving: Previously, the archiving process ran every five minutes. Though in most cases this is acceptable, some customers were concerned that this process could affect cluster performance when, for example, a cluster is running close to capacity during a specific time period. If an archiving window overlaps with this period, it may overload the cluster and lead to stability issues. These customers requested the ability to schedule archiving during an off-peak time, when the clusters have spare capacity. With this requirement in mind, we are thrilled to introduce the scheduled archiving feature. You can configure the scheduled window by setting rules. The window can be scheduled to repeat every day, every week, or every month, depending on your preference. To ensure that the archive process is able to work through any backlog that’s accrued, there is a minimum window requirement of two hours. A scheduled archiving window set to repeat every week You are also able to edit the archive rule and define when you want to archive your data and when you want to delete it. Data expiration and scheduled archiving will provide operational efficiency to Atlas customers. Both are in preview mode and will be generally available soon. See the documentation for additional information about data expiration and scheduled archiving .