> For the complete MongoDB documentation index, see www.mongodb.com/docs/llms.txt # Improve MongoDB Search Index Performance Optimize your MongoDB Search index performance by following resource allocation and scaling recommendations. Use the sections below to understand resource requirements, manage index build times, and plan cluster scaling for your deployment type. ## Resource Requirements ### Index Size and Configuration **Important:** If you create a MongoDB Search index for a collection that has or will soon have more than 2.1 billion index objects, use the [numPartitions](https://www.mongodb.com/docs/search/indexes/index-partition.md#std-label-fts-index-partition) index option to partition your index or [shard your cluster](https://www.mongodb.com/docs/atlas/cluster-additional-settings.md#std-label-create-cluster-sharding). Index partitions are limited to 2.1 billion objects per partition. Some index configuration options can lead to indexes that take up a significant proportion of your disk space. In some cases, your index could be many times larger than the size of your data. Although this is expected behavior, it's important to be aware of the following indexing-intensive features. **Note:** Some limitations apply to MongoDB Search on Free clusters and Flex clusters only. To learn more, see [MongoDB Search Search Free and Flex Tier Limitations.](https://www.mongodb.com/docs/search/deployment/feature-compatibility.md#std-label-atlas-fts-shared-tier-limitations) #### Dynamic vs. Static Mappings When you [create an MongoDB Search index](https://www.mongodb.com/docs/search/indexes/manage-indexes.md#std-label-ref-create-index), field mapping defaults to [dynamic](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings), which means that MongoDB Search dynamically indexes all the [datatypes that can be dynamically indexed](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-bson-data-chart) in your collection. This can lead to large indexes if you have a large number of fields in your documents, or if you have fields with long string values. To reduce the size and performance footprint of your index, we recommend using [static mappings](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings) to specify the fields that you want to index and their types. #### `store` Option When indexing a text field as the MongoDB Search [string](https://www.mongodb.com/docs/search/indexes/field-types/string-type.md#std-label-bson-data-types-string) type, you can set the `store` option to `false` to reduce the size of your index. This prevents MongoDB Search from storing the exact document text as well as the analyzed values in the index. However, setting `store` to `false` prevents you from using the [highlight](https://www.mongodb.com/docs/search/query/highlighting.md#std-label-highlight-ref) option to adds fields to the result set that display search terms in their original context. #### Autocomplete The MongoDB Search [autocomplete](https://www.mongodb.com/docs/search/indexes/field-types/autocomplete-type.md#std-label-bson-data-types-autocomplete) field type creates indexes that you can use with the [autocomplete](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-ref) operator to implement search-as-you-type functionality in your application. However, the `autocomplete` field type can create large indexes and take longer to build than other field types. Use the following recommendations to reduce the size and performance footprint of the `autocomplete` type: - Use the `autocomplete` type only for specific fields in static mappings. Using a custom `typeSet` definition to dynamically index string fields as the `autocomplete` type can apply the `autocomplete` type too broadly and significantly increase index size and resource usage. In addition, the `autocomplete` type has specialized, typeahead-oriented scoring behavior suited to specific field indexing, not broad dynamic application. To learn more about `similarity.type` scoring with the `autocomplete` type, see [text, phrase, queryString, and autocomplete Operators.](https://www.mongodb.com/docs/search/query/score/get-details.md#std-label-fts-similarity-algorithms) - Avoid using `nGram` as the `tokenization` strategy for the `autocomplete` type definition, as it creates more tokens for a given string than `edgeGram` or `rightEdgeGram` tokenization. - Set a smaller range between the `minGrams` and `maxGrams` option values in the `autocomplete` type definition, which reduces the number of tokens created for each string. Generally, we recommend setting `maxGrams` to the character count of the longest word in the field that you want to query. If you are unsure, for English language fields, we recommend starting with `maxGrams` value of `10`. Setting a larger value for `minGrams` can significantly reduce index size for fields in a collection with many documents. For example, setting `minGrams` to `2` instead of `1` creates 100,000 fewer tokens in a collection with 100,000 documents. - Monitor your index size and adjust your `autocomplete` type configuration as needed. You can [view MongoDB Search metrics](https://www.mongodb.com/docs/atlas/review-atlas-search-metrics.md#std-label-review-atlas-search-metrics) for the following index size and tokenization metrics to help you understand the impact of your `autocomplete` type configuration on your index size and resource usage: - `Search Index Size` - `Search Max nGram Fields Indexed` When indexing a `string` field as the [autocomplete](https://www.mongodb.com/docs/search/indexes/field-types/autocomplete-type.md#std-label-bson-data-types-autocomplete) type, we recommend that you index the field as the MongoDB Search [string](https://www.mongodb.com/docs/search/indexes/field-types/string-type.md#std-label-bson-data-types-string) type also for the following advantages: - Boost the score of exact matches when using the [autocomplete](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-ref) operator. - Query the same field in the same query using the [autocomplete](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-ref) operator and another operator that supports string search, such as the [text](https://www.mongodb.com/docs/search/query/operators-collectors/text.md#std-label-text-ref) operator. For an example, see the [Search Across Multiple Fields](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-compound) example. #### Embedded Documents MongoDB Search stops replicating changes for indexes larger than 2,100,000,000 index objects per partition, on a replica set or single shard, where each indexed embedded parent document counts as a single object. Surpassing this limit may result in stale query results. Using the `embeddedDocuments` field type can result in indexing objects over this index size limit, because each indexed embedded document is counted as a single object. If you create a MongoDB Search index that has or will soon have more than 2.1 billion index objects, use the [numPartitions](https://www.mongodb.com/docs/search/indexes/index-partition.md#std-label-fts-index-partition) index option to partition your index (supported only on Search Nodes deployments) or [shard your cluster.](https://www.mongodb.com/docs/atlas/cluster-additional-settings.md#std-label-create-cluster-sharding) The exact number of index objects can vary based on the rate of document changes and deletions. The [Search Max Number of Lucene Docs](https://www.mongodb.com/docs/atlas/review-atlas-search-metrics.md#std-label-review-atlas-search-metrics) metric shows the largest number of Lucene documents in a single partition of any search index within a replica set or shard. For example, suppose the metric displays "1,500,000,000." The index with the highest number of Lucene documents in a single partition has about 1,500,000,000 documents in that partition. If the index uses many partitions, it can contain more documents overall. You can approximate the expected number of index objects in a single index by doing the following steps: 1. Calculate the number of index objects per document. For every level of nesting, each embedded document counts as a separate index object. ```shell number of index objects in document = 1 + number of nested embedded documents ``` 2. Multiply the number of index objects per document by the total number of documents in the collection ```shell total number of index objects = number of index objects in document x total number of documents in collection ``` Note that this approximation is a lower bound. **Example:** Consider the [collection](https://search-playground.mongodb.com/tools/code-sandbox/snapshots/6601831bf3ec7476bd9da8c9) named `schools`, described in this [tutorial](https://www.mongodb.com/docs/search/tutorial/embedded-documents-tutorial.md#std-label-embedded-documents-tutorial-sample-collection), and suppose the collection contains 1000 documents similar to the following: ```json { "_id": 0, "name": "Springfield High", "mascot": "Pumas", "teachers": [ { "first": "Jane", "last": "Smith", "classes": [ { "subject": "art of science", "grade": "12th" }, ... // 2 more embedded documents ] }, ... // 1 more embedded document ], "clubs": { "stem": [ { "club_name": "chess", "description": "provides students opportunity to play the board game of chess informally and competitively in tournaments." }, ... // 1 more embedded document ], ... // 1 more embedded document } } ``` Now consider the index definition for the following fields in the `schools` collection: ### Nested Array The array of documents named `teachers` is indexed as the `embeddedDocuments` type with dynamic mappings enabled. However, the `classes` field *isn't indexed*. Use the following to calculate the index objects: 1. Calculate the number of index objects per document. ```shell Number of ``teachers`` embedded documents = up to 2 Total number of index objects per document = 1 + 2 = 3 ``` 2. Multiply by the total number of documents in the collection. ```shell Number of documents in the collection = 1000 Number of index objects per document = 3 Total number of index objects for collection = 1000 x 3 = 3000 ``` If your collection has large arrays that might generate 2,100,000,000 index objects, you must [shard](https://www.mongodb.com/docs/atlas/cluster-additional-settings.md#std-label-create-cluster-sharding) any clusters that contain indexes with the `embeddedDocuments` type. #### Faceted Search If you want to filter and facet your data using the same field, we recommend that you index the field as following MongoDB Search types: - [token](https://www.mongodb.com/docs/search/indexes/field-types/token-type.md#std-label-bson-data-types-token) - [number](https://www.mongodb.com/docs/search/indexes/field-types/number-type.md#std-label-bson-data-types-number) - [date](https://www.mongodb.com/docs/search/indexes/field-types/date-type.md#std-label-bson-data-types-date) For an example of filtering the data by another field for faceting, see the [How to Use Facets with MongoDB Search](https://www.mongodb.com/docs/search/tutorial/facet-tutorial.md#std-label-facet-tutorial) tutorial. #### `multi` Analyzers Using a `multi` analyzer to analyze the same field multiple different ways can cause large indexes, especially when analyzing fields with very long values. #### Multilingual Search You can use the MongoDB Search [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers) to index many languages. For the list of languages for which MongoDB Search provides built-in analyzers, see the [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers). To index and query languages that are currently not in the list of built-in [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers), you can create a [custom analyzer](https://www.mongodb.com/docs/search/indexes/analyzers/custom.md#std-label-custom-analyzers). For an example, see the [Custom Language Analyzer Example.](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-custom-language-analyzer-eg) Suppose you have one document for each language in your collection. Consider the following: - You can index the fields separately in the same index using the [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers). A single index can support multiple languages in the same query. - Alternatively, you can create an index per language, which is useful in isolating the different language documents. Note that each index is a change stream cursor and so this might be expensive to maintain. - If you have the language documents nested inside a parent document, you can create a single index. However, your index definition payload might be large and your query might be complex. To learn how to do a Multilingual Search, see [Multilingual Search Example](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-fts-multilingual-example) #### Synonym Collections Inserts and updates to a synonym source collection are fast only if the synonym source collection is small. For best performance, we recommend batching inserts and updates to synonym source collections. A [synonym mapping](https://www.mongodb.com/docs/search/indexes/synonyms.md#std-label-synonyms-ref) definition doesn't require additional disk space aside from the disk space utilized by the synonym collection in the database. However, synonym mappings create artifacts in memory and therefore, for synonym collections with many documents, MongoDB Search creates artifacts that occupy more memory. ### Search Memory Management MongoDB Search uses both filesystem cache and JVM heap memory. It stores various objects like query objects, searcher objects, and other temporary data used during indexing and search operations in the JVM heap to process and track the query. It stores memory mapped files such as segment files, dictionary files, and the like in the filesystem cache to enhance its performance, especially when reading index files. In deployments where both the `mongod` and `mongot` processes run on the same node, the memory allocated to the `mongod` varies based on the cluster tier: - For `M40` and higher tier clusters, `mongod` dedicates 50% or more of the physical RAM for the WiredTiger cache and the remaining memory is reserved for in-memory operations, underlying operation systems, and other system services. - For `M30` and lower tier clusters, `mongod` dedicates 25% of the physical RAM for the WiredTiger cache. Therefore, the memory allocated to `mongot` is a subset of the total RAM, which can result in resource contention issues between `mongod` and `mongot` for not only memory, but also for CPU and Disk IO. In deployments where the `mongot` process runs on separate search nodes, Atlas allocates a part of the available RAM to the JVM heap, and uses a small amount of memory processes like monitoring and automation, and the rest of the memory is available for search. If your search indexes are large and available memory is low, you might observe performance degradation during indexing and querying due to insufficient memory. Search indexes might be large if you enable dynamic mappings in your index for documents with arbitrary keys. This could cause [mapping explosions](https://www.mongodb.com/docs/search/performance/index-performance.md#std-label-index-document-mapping-explosions). The excessive memory consumption by `mongot` can also result in `mongot` running OOM (Out Of Memory), which might also crash `mongot`. You can use the following metrics to determine if `mongot` is running OOM (Out Of Memory): - An increase in the number of `Search Page Faults` and `Disk IOPS`. This happens if the OS keeps retrieving the required pages from disk and reads them into the RAM. - Elevated levels of `Normalized Process/System CPU` and `IOWait`, which can result in degraded performance. ### Creating and Updating a MongoDB Search Index Creating a MongoDB Search index is resource-intensive. The performance of your Atlas cluster might be impacted while the index builds. Atlas replicates all writes on the collection. This means that for each collection with MongoDB Search indexes, the writes are amplified to the amount of MongoDB Search indexes defined for that collection. In some instances, your MongoDB Search index must be rebuilt. Rebuilding the MongoDB Search index also consumes resources and might affect database performance. MongoDB Search automatically rebuilds the index in the event of: - [Changes](https://www.mongodb.com/docs/search/indexes/manage-indexes.md#std-label-ref-edit-index) to the index definition - Changes to the number of [index partitions](https://www.mongodb.com/docs/search/indexes/index-partition.md#std-label-fts-index-partition) - Enabling or disabling [customer key management](https://www.mongodb.com/docs/atlas/security-kms-encryption.md#std-label-enable-search-node-encryption), which causes Atlas to perform an initial sync to re-encrypt your cluster data and rebuild your MongoDB Search and MongoDB Vector Search indexes - An [initial sync](https://www.mongodb.com/docs/search/performance/index-performance.md#std-label-troubleshoot-initial-sync) on your cluster, which rebuilds the MongoDB Search indexes on the affected nodes - MongoDB Search version updates that include breaking changes: - Breaking changes are rare. - MongoDB Search provides backwards compatibility for existing indexes in nearly all cases. - When a breaking change does occur, the version update and the resulting index rebuild typically occur during your cluster's maintenance window. - Hardware-related problems such as index corruption **Note:** MongoDB Search supports no-downtime indexing, which means you can continue to run search queries while MongoDB Search rebuilds your index. MongoDB Search keeps your old index up-to-date while the new index is being built. You can view the amount of disk space currently used by your index in the [Search Disk Space Used](https://www.mongodb.com/docs/atlas/review-atlas-search-metrics.md#std-label-review-atlas-search-metrics) metric. If Atlas detects insufficient disk space to rebuild an index in place, it automatically provisions new nodes in the background, rebuilds the index, and swaps to the new index seamlessly. You can also make storage changes manually as described in [Fix Storage Issues](https://www.mongodb.com/docs/atlas/reference/alert-resolutions/disk-space-used.md#std-label-disk-space-used), even for clusters with autoscaling enabled. If you deployed separate Search Nodes, for certain changes such as [Java 21 upgrade](https://www.mongodb.com/docs/atlas/search-changelog.md#std-label-fts20240628), Atlas automatically deploys additional Search Nodes for the duration of the index rebuild and you don't need to allocate any additional free disk space. Once MongoDB Search rebuilds the index, the old index is automatically replaced without any further action from your side. ### Eventual Consistency and Indexing Latency MongoDB Search supports eventual consistency and does not provide any stronger consistency guarantees. This means that data inserted into a MongoDB collection and indexed by MongoDB Search will not be available immediately for `$search` queries. MongoDB Search reads data from MongoDB [change streams](https://www.mongodb.com/docs/manual/changeStreams.md#std-label-changeStreams) and indexes that data in an asynchronous process. This process is typically very fast, but might sometimes be impacted by replication latency, system resource availability, and index definition complexity. A large number of MongoDB Search indexes might also contribute to replication lag and latency for MongoDB Search indexes. Highly complex view transformations can increase indexing and query time. This is because the [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) must read the view definition when it filters and transforms the oplog entries during indexing (initial sync and steady state replication), and when it applies those transformations to the returned documents at query-time. Consider creating a [materialized view](https://www.mongodb.com/docs/manual/core/materialized-views.md#std-label-manual-materialized-views) to avoid extra replication load on Atlas. You can also query the source collection directly to avoid query latency from the view transformation. ### Document Mapping Explosions Mapping explosions occur when MongoDB Search indexes a document with arbitrary keys and you have a [dynamic mapping](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings). The `mongot` process might consume increasing amounts of memory and could crash. If you add too many fields to an index, mapping explosions can occur. To address this issue, you can upgrade your cluster or use a [static mapping](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings) that does not index all fields in your data. When searching over fields using a wildcard path, design your search to use a tuple-like schema. If you perform a wildcard path search that uses a key-value schema, MongoDB Search indexes each key as its own field, which can cause mapping explosions. **Example:** An example of a key-value schema is as follows: ```text ruleBuilder: { ruleName1: , ruleName2: , ..... ruleName1025: } ``` An example of the same data restructured to use a tuple-like schema is as follows: ```text { ruleBuilder: [ {name: ruleName1, data: }, {name: ruleName2, data: }, ... {name: ruleName1025, data: } ] } ``` ### Storing Source Fields You can [configure](https://www.mongodb.com/docs/search/indexes/stored-source-definition.md#std-label-fts-stored-source-definition) fields to store on MongoDB Search and improve performance of subsequent aggregation pipeline stages like [`$sort`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/sort.md#mongodb-pipeline-pipe.-sort), [`$match`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/match.md#mongodb-pipeline-pipe.-match), [`$group`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/group.md#mongodb-pipeline-pipe.-group), and [`$skip`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/skip.md#mongodb-pipeline-pipe.-skip). Use this optimization if your original documents and matched dataset are so large that a full data lookup is inefficient. To learn more about storing specific fields on MongoDB Search and returning those stored fields only, see [Define Stored Source Fields in Your MongoDB Search Index](https://www.mongodb.com/docs/search/indexes/stored-source-definition.md#std-label-fts-stored-source-definition) and [Return Stored Source Fields.](https://www.mongodb.com/docs/search/query/return-stored-source.md#std-label-fts-return-stored-source-option) We recommend storing only the minimum number of fields required for subsequent stages. If necessary, you can use [`$lookup`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/lookup.md#mongodb-pipeline-pipe.-lookup) at the end of the pipeline stage to retrieve entire documents as shown in the [Examples](https://www.mongodb.com/docs/search/query/return-stored-source.md#std-label-fts-return-stored-source-egs). Storing unnecessary fields increases disk utilization and could negatively impact performance during indexing and querying. ## Scaling Considerations ### MongoDB Search Upgrade MongoDB Search is deployed on your Atlas cluster. When a new version of MongoDB Search is deployed, your Atlas cluster might experience brief network failures in returning query results. To mitigate issues during deployment and minimize impact to your application, consider the following: - Implement retry logic in your application. - Configure Atlas [maintenance windows.](https://www.mongodb.com/docs/atlas/tutorial/cluster-maintenance-window.md#std-label-configure-maintenance-window) **Note:** MongoDB Search upgrades start only during the maintenance window and might continue after the maintenance window. To learn more about the changes in each release, see [Atlas Search Changelog.](https://www.mongodb.com/docs/atlas/search-changelog.md#std-label-fts-changelog) ### Scaling Up Indexing Performance You can scale up your initial sync and steady state indexing for a MongoDB Search index by upgrading your cluster to a higher [tier](https://www.mongodb.com/docs/atlas/manage-clusters.md#std-label-create-cluster-instance) with more cores. MongoDB Search uses a percentage of all available cores to run both initial sync and steady state indexing and performance improves as new cores are made available by upgrading your cluster. ### Atlas cluster Configuration Change If you reconfigure your deployment to use the local NVMe (non-volatile memory express) storage type or upscale an NVMe (non-volatile memory express)-based cluster, MongoDB Search performs an initial sync of all configured MongoDB Search indexes after each node completes its underlying configuration or upscale action. If the MongoDB Search index initial syncs take longer than the time it took to complete the cluster configuration change, you can't run [`$search`](https://www.mongodb.com/docs/search/query/aggregation-stages/search.md#mongodb-pipeline-pipe.-search) queries until the initial sync completes on all the nodes in your Atlas cluster. We recommend deploying [dedicated Search Nodes](https://www.mongodb.com/docs/atlas/cluster-config/multi-cloud-distribution.md#std-label-configure-search-nodes) to scale your Atlas cluster and [`$search`](https://www.mongodb.com/docs/search/query/aggregation-stages/search.md#mongodb-pipeline-pipe.-search) workloads independently. Dedicated Search Nodes run only the `mongot` process and therefore improve the availability, performance, and workload balancing of the `mongot` process. Scaling your cluster by adding search nodes or by changing the search tier triggers a rebuild of the full MongoDB Search index. However, if your cluster has dedicated search nodes for which you haven't enabled [Encryption at Rest using Customer Key Management](https://www.mongodb.com/docs/atlas/security-kms-encryption.md#std-label-security-kms-encryption), Atlas provides the following optimizations: - When you scale your search nodes, Atlas uses a recent copy of your index in S3 (Simple Storage Service), Google Cloud Storage, or Azure Blob Storage instead of rebuilding the entire MongoDB Search index on the new node. - For existing nodes, Atlas periodically takes and uploads a new incremental list of index files. Atlas retains index files for up to fourteen (14) days. ## Initial Sync in Progress MongoDB Search starts the initial sync process in the following scenarios: - When you create a new cluster or perform certain upgrades on a cluster, an initial sync process occurs. - If you add shards to a collection with an existing MongoDB Search index, an initial sync occurs on the added shards for that index. - If you shard a collection that already has a MongoDB Search index, an initial sync occurs on shards where the collection begins to exist. The initial sync process includes the following steps: 1. The `mongod` performs an initial sync. 2. The `mongot` performs an initial sync, which rebuilds the search indexes. While a search index rebuilds, you can still perform `$search` queries on the existing indexed fields. However, MongoDB Search might return an error if you run a `$search` query against a new field or a node that you've recently created. **Warning:** Rolling resyncs cause colocated search instances to fall off the oplog, which affects processes dependant on oplog data. We recommend migrating to [dedicated search nodes](https://www.mongodb.com/docs/atlas/cluster-config/multi-cloud-distribution.md#std-label-configure-search-nodes) to avoid this issue. Check the status of the `mongot` initial sync using the following steps: 1. In Atlas, go to the Clusters page for your project. If it's not already displayed, select the organization that contains your desired project from the Organizations menu in the navigation bar. If it's not already displayed, select your desired project from the Projects menu in the navigation bar. In the sidebar, click Clusters under the Database heading. The [Clusters](https://cloud.mongodb.com/go?l=https%3A%2F%2Fcloud.mongodb.com%2Fv2%2F%3Cproject%3E%23%2Fclusters) page displays. 2. In Atlas, go to the Search & Vector Search page for your cluster. You can go the MongoDB Search page from the Search & Vector Search option, or the Data Explorer. ### Search & Vector Search If it's not already displayed, select the organization that contains your project from the Organizations menu in the navigation bar. If it's not already displayed, select your project from the Projects menu in the navigation bar. In the sidebar, click Search & Vector Search under the Database heading. - If you have no clusters, click Create cluster to create one. To learn more, see [Create an Atlas Infinite or Atlas Core Cluster.](https://www.mongodb.com/docs/atlas/tutorial/create-new-cluster.md#std-label-create-new-cluster) - If your project has multiple clusters, select the cluster you want to use from the Select cluster dropdown, then click Go to Search. The [Search & Vector Search](https://cloud.mongodb.com/go?l=https%3A%2F%2Fcloud.mongodb.com%2Fv2%2F%3Cproject%3E%23%2Fclusters%2FatlasSearch%2F%3Ccluster%3E) page displays. 3. View the status details. In the index's `Status` column, click View Status Details. Check the state of the index for the node. During `mongot` initial sync, the status is `INITIAL SYNC`. When `mongot` finishes rebuilding the index, the status is `ACTIVE`. ## Resource Requirements ### Index Size and Configuration **Important:** If you create a MongoDB Search index for a collection that has or will soon have more than 2.1 billion index objects, use the [numPartitions](https://www.mongodb.com/docs/search/indexes/index-partition.md#std-label-fts-index-partition) index option to partition your index or [shard your cluster](https://www.mongodb.com/docs/atlas/cluster-additional-settings.md#std-label-create-cluster-sharding). Index partitions are limited to 2.1 billion objects per partition. Some index configuration options can lead to indexes that take up a significant proportion of your disk space. In some cases, your index could be many times larger than the size of your data. Although this is expected behavior, it's important to be aware of the following indexing-intensive features. **Note:** Some limitations apply to MongoDB Search on Free clusters and Flex clusters only. To learn more, see [MongoDB Search Search Free and Flex Tier Limitations.](https://www.mongodb.com/docs/search/deployment/feature-compatibility.md#std-label-atlas-fts-shared-tier-limitations) #### Dynamic vs. Static Mappings When you [create an MongoDB Search index](https://www.mongodb.com/docs/search/indexes/manage-indexes.md#std-label-ref-create-index), field mapping defaults to [dynamic](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings), which means that MongoDB Search dynamically indexes all the [datatypes that can be dynamically indexed](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-bson-data-chart) in your collection. This can lead to large indexes if you have a large number of fields in your documents, or if you have fields with long string values. To reduce the size and performance footprint of your index, we recommend using [static mappings](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings) to specify the fields that you want to index and their types. #### `store` Option When indexing a text field as the MongoDB Search [string](https://www.mongodb.com/docs/search/indexes/field-types/string-type.md#std-label-bson-data-types-string) type, you can set the `store` option to `false` to reduce the size of your index. This prevents MongoDB Search from storing the exact document text as well as the analyzed values in the index. However, setting `store` to `false` prevents you from using the [highlight](https://www.mongodb.com/docs/search/query/highlighting.md#std-label-highlight-ref) option to adds fields to the result set that display search terms in their original context. #### Autocomplete The MongoDB Search [autocomplete](https://www.mongodb.com/docs/search/indexes/field-types/autocomplete-type.md#std-label-bson-data-types-autocomplete) field type creates indexes that you can use with the [autocomplete](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-ref) operator to implement search-as-you-type functionality in your application. However, the `autocomplete` field type can create large indexes and take longer to build than other field types. Use the following recommendations to reduce the size and performance footprint of the `autocomplete` type: - Use the `autocomplete` type only for specific fields in static mappings. Using a custom `typeSet` definition to dynamically index string fields as the `autocomplete` type can apply the `autocomplete` type too broadly and significantly increase index size and resource usage. In addition, the `autocomplete` type has specialized, typeahead-oriented scoring behavior suited to specific field indexing, not broad dynamic application. To learn more about `similarity.type` scoring with the `autocomplete` type, see [text, phrase, queryString, and autocomplete Operators.](https://www.mongodb.com/docs/search/query/score/get-details.md#std-label-fts-similarity-algorithms) - Avoid using `nGram` as the `tokenization` strategy for the `autocomplete` type definition, as it creates more tokens for a given string than `edgeGram` or `rightEdgeGram` tokenization. - Set a smaller range between the `minGrams` and `maxGrams` option values in the `autocomplete` type definition, which reduces the number of tokens created for each string. Generally, we recommend setting `maxGrams` to the character count of the longest word in the field that you want to query. If you are unsure, for English language fields, we recommend starting with `maxGrams` value of `10`. Setting a larger value for `minGrams` can significantly reduce index size for fields in a collection with many documents. For example, setting `minGrams` to `2` instead of `1` creates 100,000 fewer tokens in a collection with 100,000 documents. - Monitor your index size and adjust your `autocomplete` type configuration as needed. You can [view MongoDB Search metrics](https://www.mongodb.com/docs/atlas/review-atlas-search-metrics.md#std-label-review-atlas-search-metrics) for the following index size and tokenization metrics to help you understand the impact of your `autocomplete` type configuration on your index size and resource usage: - `Search Index Size` - `Search Max nGram Fields Indexed` When indexing a `string` field as the [autocomplete](https://www.mongodb.com/docs/search/indexes/field-types/autocomplete-type.md#std-label-bson-data-types-autocomplete) type, we recommend that you index the field as the MongoDB Search [string](https://www.mongodb.com/docs/search/indexes/field-types/string-type.md#std-label-bson-data-types-string) type also for the following advantages: - Boost the score of exact matches when using the [autocomplete](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-ref) operator. - Query the same field in the same query using the [autocomplete](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-ref) operator and another operator that supports string search, such as the [text](https://www.mongodb.com/docs/search/query/operators-collectors/text.md#std-label-text-ref) operator. For an example, see the [Search Across Multiple Fields](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-compound) example. #### Embedded Documents MongoDB Search stops replicating changes for indexes larger than 2,100,000,000 index objects per partition, on a replica set or single shard, where each indexed embedded parent document counts as a single object. Surpassing this limit may result in stale query results. Using the `embeddedDocuments` field type can result in indexing objects over this index size limit, because each indexed embedded document is counted as a single object. If you create a MongoDB Search index that has or will soon have more than 2.1 billion index objects, use the [numPartitions](https://www.mongodb.com/docs/search/indexes/index-partition.md#std-label-fts-index-partition) index option to partition your index (supported only on Search Nodes deployments) or [shard your cluster.](https://www.mongodb.com/docs/atlas/cluster-additional-settings.md#std-label-create-cluster-sharding) The exact number of index objects can vary based on the rate of document changes and deletions. The [Search Max Number of Lucene Docs](https://www.mongodb.com/docs/atlas/review-atlas-search-metrics.md#std-label-review-atlas-search-metrics) metric shows the largest number of Lucene documents in a single partition of any search index within a replica set or shard. For example, suppose the metric displays "1,500,000,000." The index with the highest number of Lucene documents in a single partition has about 1,500,000,000 documents in that partition. If the index uses many partitions, it can contain more documents overall. You can approximate the expected number of index objects in a single index by doing the following steps: 1. Calculate the number of index objects per document. For every level of nesting, each embedded document counts as a separate index object. ```shell number of index objects in document = 1 + number of nested embedded documents ``` 2. Multiply the number of index objects per document by the total number of documents in the collection ```shell total number of index objects = number of index objects in document x total number of documents in collection ``` Note that this approximation is a lower bound. **Example:** Consider the [collection](https://search-playground.mongodb.com/tools/code-sandbox/snapshots/6601831bf3ec7476bd9da8c9) named `schools`, described in this [tutorial](https://www.mongodb.com/docs/search/tutorial/embedded-documents-tutorial.md#std-label-embedded-documents-tutorial-sample-collection), and suppose the collection contains 1000 documents similar to the following: ```json { "_id": 0, "name": "Springfield High", "mascot": "Pumas", "teachers": [ { "first": "Jane", "last": "Smith", "classes": [ { "subject": "art of science", "grade": "12th" }, ... // 2 more embedded documents ] }, ... // 1 more embedded document ], "clubs": { "stem": [ { "club_name": "chess", "description": "provides students opportunity to play the board game of chess informally and competitively in tournaments." }, ... // 1 more embedded document ], ... // 1 more embedded document } } ``` Now consider the index definition for the following fields in the `schools` collection: ### Nested Array The array of documents named `teachers` is indexed as the `embeddedDocuments` type with dynamic mappings enabled. However, the `classes` field *isn't indexed*. Use the following to calculate the index objects: 1. Calculate the number of index objects per document. ```shell Number of ``teachers`` embedded documents = up to 2 Total number of index objects per document = 1 + 2 = 3 ``` 2. Multiply by the total number of documents in the collection. ```shell Number of documents in the collection = 1000 Number of index objects per document = 3 Total number of index objects for collection = 1000 x 3 = 3000 ``` If your collection has large arrays that might generate 2,100,000,000 index objects, you must [shard](https://www.mongodb.com/docs/atlas/cluster-additional-settings.md#std-label-create-cluster-sharding) any clusters that contain indexes with the `embeddedDocuments` type. #### Faceted Search If you want to filter and facet your data using the same field, we recommend that you index the field as following MongoDB Search types: - [token](https://www.mongodb.com/docs/search/indexes/field-types/token-type.md#std-label-bson-data-types-token) - [number](https://www.mongodb.com/docs/search/indexes/field-types/number-type.md#std-label-bson-data-types-number) - [date](https://www.mongodb.com/docs/search/indexes/field-types/date-type.md#std-label-bson-data-types-date) For an example of filtering the data by another field for faceting, see the [How to Use Facets with MongoDB Search](https://www.mongodb.com/docs/search/tutorial/facet-tutorial.md#std-label-facet-tutorial) tutorial. #### `multi` Analyzers Using a `multi` analyzer to analyze the same field multiple different ways can cause large indexes, especially when analyzing fields with very long values. #### Multilingual Search You can use the MongoDB Search [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers) to index many languages. For the list of languages for which MongoDB Search provides built-in analyzers, see the [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers). To index and query languages that are currently not in the list of built-in [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers), you can create a [custom analyzer](https://www.mongodb.com/docs/search/indexes/analyzers/custom.md#std-label-custom-analyzers). For an example, see the [Custom Language Analyzer Example.](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-custom-language-analyzer-eg) Suppose you have one document for each language in your collection. Consider the following: - You can index the fields separately in the same index using the [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers). A single index can support multiple languages in the same query. - Alternatively, you can create an index per language, which is useful in isolating the different language documents. Note that each index is a change stream cursor and so this might be expensive to maintain. - If you have the language documents nested inside a parent document, you can create a single index. However, your index definition payload might be large and your query might be complex. To learn how to do a Multilingual Search, see [Multilingual Search Example](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-fts-multilingual-example) #### Synonym Collections Inserts and updates to a synonym source collection are fast only if the synonym source collection is small. For best performance, we recommend batching inserts and updates to synonym source collections. A [synonym mapping](https://www.mongodb.com/docs/search/indexes/synonyms.md#std-label-synonyms-ref) definition doesn't require additional disk space aside from the disk space utilized by the synonym collection in the database. However, synonym mappings create artifacts in memory and therefore, for synonym collections with many documents, MongoDB Search creates artifacts that occupy more memory. ### Search Memory Management MongoDB Search uses both filesystem cache and JVM heap memory. It stores various objects like query objects, searcher objects, and other temporary data used during indexing and search operations in the JVM heap to process and track the query. It stores memory mapped files such as segment files, dictionary files, and the like in the filesystem cache to enhance its performance, especially when reading index files. In deployments where both the `mongod` and `mongot` processes run on the same node, the memory allocated to the `mongod` varies based on the cluster tier: - For `M40` and higher tier clusters, `mongod` dedicates 50% or more of the physical RAM for the WiredTiger cache and the remaining memory is reserved for in-memory operations, underlying operation systems, and other system services. - For `M30` and lower tier clusters, `mongod` dedicates 25% of the physical RAM for the WiredTiger cache. Therefore, the memory allocated to `mongot` is a subset of the total RAM, which can result in resource contention issues between `mongod` and `mongot` for not only memory, but also for CPU and Disk IO. In deployments where the `mongot` process runs on separate search nodes, Atlas allocates a part of the available RAM to the JVM heap, and uses a small amount of memory processes like monitoring and automation, and the rest of the memory is available for search. If your search indexes are large and available memory is low, you might observe performance degradation during indexing and querying due to insufficient memory. Search indexes might be large if you enable dynamic mappings in your index for documents with arbitrary keys. This could cause [mapping explosions](https://www.mongodb.com/docs/search/performance/index-performance.md#std-label-index-document-mapping-explosions). The excessive memory consumption by `mongot` can also result in `mongot` running OOM (Out Of Memory), which might also crash `mongot`. You can use the following metrics to determine if `mongot` is running OOM (Out Of Memory): - An increase in the number of `Search Page Faults` and `Disk IOPS`. This happens if the OS keeps retrieving the required pages from disk and reads them into the RAM. - Elevated levels of `Normalized Process/System CPU` and `IOWait`, which can result in degraded performance. ### Creating and Updating a MongoDB Search Index Creating a MongoDB Search index is resource-intensive. The performance of your cluster might be impacted while the index builds. In some instances, your MongoDB Search index must be rebuilt. Rebuilding the MongoDB Search index also consumes resources and might affect database performance. MongoDB Search automatically rebuilds the index in the event of: - [Changes](https://www.mongodb.com/docs/search/indexes/manage-indexes.md#std-label-ref-edit-index) to the index definition - Changes to the number of [index partitions](https://www.mongodb.com/docs/search/indexes/index-partition.md#std-label-fts-index-partition) - An initial sync on your cluster, which rebuilds the MongoDB Search indexes on the affected nodes - MongoDB Search version updates that include breaking changes - Hardware-related problems such as index corruption **Note:** MongoDB Search supports no-downtime indexing, which means you can continue to run search queries while MongoDB Search rebuilds your index. MongoDB Search keeps your old index up-to-date while the new index is being built. If your index rebuild fails due to insufficient disk space, we recommend that you expand your cluster capacity to meet the increased demand. Once MongoDB Search rebuilds the index, the old index is automatically replaced without any further action from your side. ### Eventual Consistency and Indexing Latency MongoDB Search supports eventual consistency and does not provide any stronger consistency guarantees. This means that data inserted into a MongoDB collection and indexed by MongoDB Search will not be available immediately for `$search` queries. MongoDB Search reads data from MongoDB [change streams](https://www.mongodb.com/docs/manual/changeStreams.md#std-label-changeStreams) and indexes that data in an asynchronous process. This process is typically very fast, but might sometimes be impacted by replication latency, system resource availability, and index definition complexity. A large number of MongoDB Search indexes might also contribute to replication lag and latency for MongoDB Search indexes. Highly complex view transformations can increase indexing and query time. This is because the [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) must read the view definition when it filters and transforms the oplog entries during indexing (initial sync and steady state replication), and when it applies those transformations to the returned documents at query-time. Consider creating a [materialized view](https://www.mongodb.com/docs/manual/core/materialized-views.md#std-label-manual-materialized-views) to avoid extra replication load on Atlas. You can also query the source collection directly to avoid query latency from the view transformation. ### Document Mapping Explosions Mapping explosions occur when MongoDB Search indexes a document with arbitrary keys and you have a [dynamic mapping](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings). The `mongot` process might consume increasing amounts of memory and could crash. If you add too many fields to an index, mapping explosions can occur. To address this issue, you can upgrade your cluster or use a [static mapping](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings) that does not index all fields in your data. When searching over fields using a wildcard path, design your search to use a tuple-like schema. If you perform a wildcard path search that uses a key-value schema, MongoDB Search indexes each key as its own field, which can cause mapping explosions. **Example:** An example of a key-value schema is as follows: ```text ruleBuilder: { ruleName1: , ruleName2: , ..... ruleName1025: } ``` An example of the same data restructured to use a tuple-like schema is as follows: ```text { ruleBuilder: [ {name: ruleName1, data: }, {name: ruleName2, data: }, ... {name: ruleName1025, data: } ] } ``` ### Storing Source Fields You can [configure](https://www.mongodb.com/docs/search/indexes/stored-source-definition.md#std-label-fts-stored-source-definition) fields to store on MongoDB Search and improve performance of subsequent aggregation pipeline stages like [`$sort`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/sort.md#mongodb-pipeline-pipe.-sort), [`$match`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/match.md#mongodb-pipeline-pipe.-match), [`$group`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/group.md#mongodb-pipeline-pipe.-group), and [`$skip`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/skip.md#mongodb-pipeline-pipe.-skip). Use this optimization if your original documents and matched dataset are so large that a full data lookup is inefficient. To learn more about storing specific fields on MongoDB Search and returning those stored fields only, see [Define Stored Source Fields in Your MongoDB Search Index](https://www.mongodb.com/docs/search/indexes/stored-source-definition.md#std-label-fts-stored-source-definition) and [Return Stored Source Fields.](https://www.mongodb.com/docs/search/query/return-stored-source.md#std-label-fts-return-stored-source-option) We recommend storing only the minimum number of fields required for subsequent stages. If necessary, you can use [`$lookup`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/lookup.md#mongodb-pipeline-pipe.-lookup) at the end of the pipeline stage to retrieve entire documents as shown in the [Examples](https://www.mongodb.com/docs/search/query/return-stored-source.md#std-label-fts-return-stored-source-egs). Storing unnecessary fields increases disk utilization and could negatively impact performance during indexing and querying. ## Scaling Considerations ### MongoDB Search Upgrade When you deploy a new version of MongoDB Search, your MongoDB cluster might experience brief network failures in returning query results. To mitigate issues during deployment and minimize impact to your application, consider implementing retry logic in your application. For details on sizing, see [Introduction to mongot Deployment Sizing.](https://www.mongodb.com/docs/search/self-managed/current/resource-planning-sizing/introduction.md#std-label-mongot-sizing-intro) To learn more about the changes in each release, see [MongoDB Search Changelog.](https://www.mongodb.com/docs/atlas/search-changelog.md#std-label-fts-changelog) ### Scaling Up Indexing Performance You can scale up your initial sync and steady state indexing for a MongoDB Search index by upgrading your cluster to a higher [tier](https://www.mongodb.com/docs/atlas/manage-clusters.md#std-label-create-cluster-instance) with more cores. MongoDB Search uses a percentage of all available cores to run both initial sync and steady state indexing and performance improves as new cores are made available by upgrading your cluster. ## Resource Requirements ### Index Size and Configuration **Important:** If you create a MongoDB Search index for a collection that has or will soon have more than 2.1 billion index objects, use the [numPartitions](https://www.mongodb.com/docs/search/indexes/index-partition.md#std-label-fts-index-partition) index option to partition your index or [shard your cluster](https://www.mongodb.com/docs/atlas/cluster-additional-settings.md#std-label-create-cluster-sharding). Index partitions are limited to 2.1 billion objects per partition. Some index configuration options can lead to indexes that take up a significant proportion of your disk space. In some cases, your index could be many times larger than the size of your data. Although this is expected behavior, it's important to be aware of the following indexing-intensive features. #### Dynamic vs. Static Mappings When you [create an MongoDB Search index](https://www.mongodb.com/docs/search/indexes/manage-indexes.md#std-label-ref-create-index), field mapping defaults to [dynamic](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings), which means that MongoDB Search dynamically indexes all the [datatypes that can be dynamically indexed](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-bson-data-chart) in your collection. This can lead to large indexes if you have a large number of fields in your documents, or if you have fields with long string values. To reduce the size and performance footprint of your index, we recommend using [static mappings](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings) to specify the fields that you want to index and their types. #### `store` Option When indexing a text field as the MongoDB Search [string](https://www.mongodb.com/docs/search/indexes/field-types/string-type.md#std-label-bson-data-types-string) type, you can set the `store` option to `false` to reduce the size of your index. This prevents MongoDB Search from storing the exact document text as well as the analyzed values in the index. However, setting `store` to `false` prevents you from using the [highlight](https://www.mongodb.com/docs/search/query/highlighting.md#std-label-highlight-ref) option to adds fields to the result set that display search terms in their original context. #### Autocomplete The MongoDB Search [autocomplete](https://www.mongodb.com/docs/search/indexes/field-types/autocomplete-type.md#std-label-bson-data-types-autocomplete) field type creates indexes that you can use with the [autocomplete](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-ref) operator to implement search-as-you-type functionality in your application. However, the `autocomplete` field type can create large indexes and take longer to build than other field types. Use the following recommendations to reduce the size and performance footprint of the `autocomplete` type: - Use the `autocomplete` type only for specific fields in static mappings. Using a custom `typeSet` definition to dynamically index string fields as the `autocomplete` type can apply the `autocomplete` type too broadly and significantly increase index size and resource usage. In addition, the `autocomplete` type has specialized, typeahead-oriented scoring behavior suited to specific field indexing, not broad dynamic application. To learn more about `similarity.type` scoring with the `autocomplete` type, see [text, phrase, queryString, and autocomplete Operators.](https://www.mongodb.com/docs/search/query/score/get-details.md#std-label-fts-similarity-algorithms) - Avoid using `nGram` as the `tokenization` strategy for the `autocomplete` type definition, as it creates more tokens for a given string than `edgeGram` or `rightEdgeGram` tokenization. - Set a smaller range between the `minGrams` and `maxGrams` option values in the `autocomplete` type definition, which reduces the number of tokens created for each string. Generally, we recommend setting `maxGrams` to the character count of the longest word in the field that you want to query. If you are unsure, for English language fields, we recommend starting with `maxGrams` value of `10`. Setting a larger value for `minGrams` can significantly reduce index size for fields in a collection with many documents. For example, setting `minGrams` to `2` instead of `1` creates 100,000 fewer tokens in a collection with 100,000 documents. When indexing a `string` field as the [autocomplete](https://www.mongodb.com/docs/search/indexes/field-types/autocomplete-type.md#std-label-bson-data-types-autocomplete) type, we recommend that you index the field as the MongoDB Search [string](https://www.mongodb.com/docs/search/indexes/field-types/string-type.md#std-label-bson-data-types-string) type also for the following advantages: - Boost the score of exact matches when using the [autocomplete](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-ref) operator. - Query the same field in the same query using the [autocomplete](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-ref) operator and another operator that supports string search, such as the [text](https://www.mongodb.com/docs/search/query/operators-collectors/text.md#std-label-text-ref) operator. For an example, see the [Search Across Multiple Fields](https://www.mongodb.com/docs/search/query/operators-collectors/autocomplete.md#std-label-autocomplete-compound) example. #### Embedded Documents MongoDB Search stops replicating changes for indexes larger than 2,100,000,000 index objects per partition, on a replica set or single shard, where each indexed embedded parent document counts as a single object. Surpassing this limit may result in stale query results. Using the `embeddedDocuments` field type can result in indexing objects over this index size limit, because each indexed embedded document is counted as a single object. If you create a MongoDB Search index that has or will soon have more than 2.1 billion index objects, use the [numPartitions](https://www.mongodb.com/docs/search/indexes/index-partition.md#std-label-fts-index-partition) index option to partition your index (supported only on Search Nodes deployments) or [shard your cluster.](https://www.mongodb.com/docs/atlas/cluster-additional-settings.md#std-label-create-cluster-sharding) The exact number of index objects can vary based on the rate of document changes and deletions. The [Search Max Number of Lucene Docs](https://www.mongodb.com/docs/atlas/review-atlas-search-metrics.md#std-label-review-atlas-search-metrics) metric shows the largest number of Lucene documents in a single partition of any search index within a replica set or shard. For example, suppose the metric displays "1,500,000,000." The index with the highest number of Lucene documents in a single partition has about 1,500,000,000 documents in that partition. If the index uses many partitions, it can contain more documents overall. You can approximate the expected number of index objects in a single index by doing the following steps: 1. Calculate the number of index objects per document. For every level of nesting, each embedded document counts as a separate index object. ```shell number of index objects in document = 1 + number of nested embedded documents ``` 2. Multiply the number of index objects per document by the total number of documents in the collection ```shell total number of index objects = number of index objects in document x total number of documents in collection ``` Note that this approximation is a lower bound. **Example:** Consider the [collection](https://search-playground.mongodb.com/tools/code-sandbox/snapshots/6601831bf3ec7476bd9da8c9) named `schools`, described in this [tutorial](https://www.mongodb.com/docs/search/tutorial/embedded-documents-tutorial.md#std-label-embedded-documents-tutorial-sample-collection), and suppose the collection contains 1000 documents similar to the following: ```json { "_id": 0, "name": "Springfield High", "mascot": "Pumas", "teachers": [ { "first": "Jane", "last": "Smith", "classes": [ { "subject": "art of science", "grade": "12th" }, ... // 2 more embedded documents ] }, ... // 1 more embedded document ], "clubs": { "stem": [ { "club_name": "chess", "description": "provides students opportunity to play the board game of chess informally and competitively in tournaments." }, ... // 1 more embedded document ], ... // 1 more embedded document } } ``` Now consider the index definition for the following fields in the `schools` collection: ### Nested Array The array of documents named `teachers` is indexed as the `embeddedDocuments` type with dynamic mappings enabled. However, the `classes` field *isn't indexed*. Use the following to calculate the index objects: 1. Calculate the number of index objects per document. ```shell Number of ``teachers`` embedded documents = up to 2 Total number of index objects per document = 1 + 2 = 3 ``` 2. Multiply by the total number of documents in the collection. ```shell Number of documents in the collection = 1000 Number of index objects per document = 3 Total number of index objects for collection = 1000 x 3 = 3000 ``` If your collection has large arrays that might generate 2,100,000,000 index objects, you must [shard](https://www.mongodb.com/docs/atlas/cluster-additional-settings.md#std-label-create-cluster-sharding) any clusters that contain indexes with the `embeddedDocuments` type. #### Faceted Search If you want to filter and facet your data using the same field, we recommend that you index the field as following MongoDB Search types: - [token](https://www.mongodb.com/docs/search/indexes/field-types/token-type.md#std-label-bson-data-types-token) - [number](https://www.mongodb.com/docs/search/indexes/field-types/number-type.md#std-label-bson-data-types-number) - [date](https://www.mongodb.com/docs/search/indexes/field-types/date-type.md#std-label-bson-data-types-date) For an example of filtering the data by another field for faceting, see the [How to Use Facets with MongoDB Search](https://www.mongodb.com/docs/search/tutorial/facet-tutorial.md#std-label-facet-tutorial) tutorial. #### `multi` Analyzers Using a `multi` analyzer to analyze the same field multiple different ways can cause large indexes, especially when analyzing fields with very long values. #### Multilingual Search You can use the MongoDB Search [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers) to index many languages. For the list of languages for which MongoDB Search provides built-in analyzers, see the [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers). To index and query languages that are currently not in the list of built-in [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers), you can create a [custom analyzer](https://www.mongodb.com/docs/search/indexes/analyzers/custom.md#std-label-custom-analyzers). For an example, see the [Custom Language Analyzer Example.](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-custom-language-analyzer-eg) Suppose you have one document for each language in your collection. Consider the following: - You can index the fields separately in the same index using the [Language Analyzers](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-ref-language-analyzers). A single index can support multiple languages in the same query. - Alternatively, you can create an index per language, which is useful in isolating the different language documents. Note that each index is a change stream cursor and so this might be expensive to maintain. - If you have the language documents nested inside a parent document, you can create a single index. However, your index definition payload might be large and your query might be complex. To learn how to do a Multilingual Search, see [Multilingual Search Example](https://www.mongodb.com/docs/search/indexes/analyzers/language.md#std-label-fts-multilingual-example) #### Synonym Collections Inserts and updates to a synonym source collection are fast only if the synonym source collection is small. For best performance, we recommend batching inserts and updates to synonym source collections. A [synonym mapping](https://www.mongodb.com/docs/search/indexes/synonyms.md#std-label-synonyms-ref) definition doesn't require additional disk space aside from the disk space utilized by the synonym collection in the database. However, synonym mappings create artifacts in memory and therefore, for synonym collections with many documents, MongoDB Search creates artifacts that occupy more memory. ### Search Memory Management MongoDB Search uses both filesystem cache and JVM heap memory. It stores various objects like query objects, searcher objects, and other temporary data used during indexing and search operations in the JVM heap to process and track the query. It stores memory mapped files such as segment files, dictionary files, and the like in the filesystem cache to enhance its performance, especially when reading index files. In deployments where both the `mongod` and `mongot` processes run on the same node, the memory allocated to the `mongod` varies based on the hardware. The memory allocated to `mongot` is a subset of the total RAM, which can result in resource contention issues between `mongod` and `mongot` for not only memory, but also for CPU and Disk IO. For guidance, see [Hardware Considerations for mongot Deployments.](https://www.mongodb.com/docs/search/self-managed/current/resource-planning-sizing/hardware.md#std-label-mongot-sizing-hardware) In deployments where the `mongot` process runs on separate search nodes, Atlas allocates a part of the available RAM to the JVM heap, and uses a small amount of memory processes like monitoring and automation, and the rest of the memory is available for search. If your search indexes are large and available memory is low, you might observe performance degradation during indexing and querying due to insufficient memory. Search indexes might be large if you enable dynamic mappings in your index for documents with arbitrary keys. This could cause [mapping explosions](https://www.mongodb.com/docs/search/performance/index-performance.md#std-label-index-document-mapping-explosions). The excessive memory consumption by `mongot` can also result in `mongot` running OOM (Out Of Memory), which might also crash `mongot`. ### Creating and Updating a MongoDB Search Index Creating a MongoDB Search index is resource-intensive. The performance of your cluster might be impacted while the index builds. In some instances, your MongoDB Search index must be rebuilt. Rebuilding the MongoDB Search index also consumes resources and might affect database performance. MongoDB Search automatically rebuilds the index in the event of: - [Changes](https://www.mongodb.com/docs/search/indexes/manage-indexes.md#std-label-ref-edit-index) to the index definition - Changes to the number of [index partitions](https://www.mongodb.com/docs/search/indexes/index-partition.md#std-label-fts-index-partition) - An initial sync on your cluster, which rebuilds the MongoDB Search indexes on the affected nodes - MongoDB Search version updates that include breaking changes - Hardware-related problems such as index corruption **Note:** MongoDB Search supports no-downtime indexing, which means you can continue to run search queries while MongoDB Search rebuilds your index. MongoDB Search keeps your old index up-to-date while the new index is being built. If your index rebuild fails due to insufficient disk space, we recommend that you expand your cluster capacity to meet the increased demand. Once MongoDB Search rebuilds the index, the old index is automatically replaced without any further action from your side. ### Eventual Consistency and Indexing Latency MongoDB Search supports eventual consistency and does not provide any stronger consistency guarantees. This means that data inserted into a MongoDB collection and indexed by MongoDB Search will not be available immediately for `$search` queries. MongoDB Search reads data from MongoDB [change streams](https://www.mongodb.com/docs/manual/changeStreams.md#std-label-changeStreams) and indexes that data in an asynchronous process. This process is typically very fast, but might sometimes be impacted by replication latency, system resource availability, and index definition complexity. A large number of MongoDB Search indexes might also contribute to replication lag and latency for MongoDB Search indexes. Highly complex view transformations can increase indexing and query time. This is because the [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) must read the view definition when it filters and transforms the oplog entries during indexing (initial sync and steady state replication), and when it applies those transformations to the returned documents at query-time. Consider creating a [materialized view](https://www.mongodb.com/docs/manual/core/materialized-views.md#std-label-manual-materialized-views) to avoid extra replication load on Atlas. You can also query the source collection directly to avoid query latency from the view transformation. ### Document Mapping Explosions Mapping explosions occur when MongoDB Search indexes a document with arbitrary keys and you have a [dynamic mapping](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings). The `mongot` process might consume increasing amounts of memory and could crash. If you add too many fields to an index, mapping explosions can occur. To address this issue, you can upgrade your cluster or use a [static mapping](https://www.mongodb.com/docs/search/indexes/define-field-mappings.md#std-label-static-dynamic-mappings) that does not index all fields in your data. When searching over fields using a wildcard path, design your search to use a tuple-like schema. If you perform a wildcard path search that uses a key-value schema, MongoDB Search indexes each key as its own field, which can cause mapping explosions. **Example:** An example of a key-value schema is as follows: ```text ruleBuilder: { ruleName1: , ruleName2: , ..... ruleName1025: } ``` An example of the same data restructured to use a tuple-like schema is as follows: ```text { ruleBuilder: [ {name: ruleName1, data: }, {name: ruleName2, data: }, ... {name: ruleName1025, data: } ] } ``` ### Storing Source Fields You can [configure](https://www.mongodb.com/docs/search/indexes/stored-source-definition.md#std-label-fts-stored-source-definition) fields to store on MongoDB Search and improve performance of subsequent aggregation pipeline stages like [`$sort`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/sort.md#mongodb-pipeline-pipe.-sort), [`$match`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/match.md#mongodb-pipeline-pipe.-match), [`$group`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/group.md#mongodb-pipeline-pipe.-group), and [`$skip`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/skip.md#mongodb-pipeline-pipe.-skip). Use this optimization if your original documents and matched dataset are so large that a full data lookup is inefficient. To learn more about storing specific fields on MongoDB Search and returning those stored fields only, see [Define Stored Source Fields in Your MongoDB Search Index](https://www.mongodb.com/docs/search/indexes/stored-source-definition.md#std-label-fts-stored-source-definition) and [Return Stored Source Fields.](https://www.mongodb.com/docs/search/query/return-stored-source.md#std-label-fts-return-stored-source-option) We recommend storing only the minimum number of fields required for subsequent stages. If necessary, you can use [`$lookup`](https://www.mongodb.com/docs/manual/reference/operator/aggregation/lookup.md#mongodb-pipeline-pipe.-lookup) at the end of the pipeline stage to retrieve entire documents as shown in the [Examples](https://www.mongodb.com/docs/search/query/return-stored-source.md#std-label-fts-return-stored-source-egs). Storing unnecessary fields increases disk utilization and could negatively impact performance during indexing and querying. For guidance, see [Disk Sizing Guideline.](https://www.mongodb.com/docs/search/self-managed/current/resource-planning-sizing/resource-allocation.md#std-label-mongot-indexes-data-characteristics) ## Scaling Considerations ### MongoDB Search Upgrade When you deploy a new version of MongoDB Search, your MongoDB cluster might experience brief network failures in returning query results. To mitigate issues during deployment and minimize impact to your application, consider implementing retry logic in your application. For details on sizing, see [Introduction to mongot Deployment Sizing.](https://www.mongodb.com/docs/search/self-managed/current/resource-planning-sizing/introduction.md#std-label-mongot-sizing-intro) To learn more about the changes in each release, see [MongoDB Search Changelog.](https://www.mongodb.com/docs/atlas/search-changelog.md#std-label-fts-changelog) ### Scaling Up Indexing Performance You can scale up your initial sync and steady state indexing for a MongoDB Search index by upgrading your CPU and RAM. MongoDB Search uses a percentage of all available cores to run both initial sync and steady state indexing and performance improves as new cores are made available by upgrading your cluster. ## Initial Sync in Progress MongoDB Search starts the initial sync process in the following scenarios: - When you create a new index, the `mongot` process performs an initial sync to build the index. - If you add shards to a collection with an existing MongoDB Search index, an initial sync occurs on the newly added shards for that index. Each shard's MongoDB Search index contains only the documents from the collection that exist on that shard. - If the `mongod` oplog rolls over before `mongot` catches up, `mongot` falls out of steady state replication and performs an initial sync to rebuild the index. The index isn't queryable while its initial sync runs. To monitor the progress of an initial sync, review the `mongot_replication_mongodb_indexManagerState` metric, which reports the `INITIAL_SYNC` state while the sync runs and the `STEADY_STATE` state after it completes. To learn more, see [Metrics Reference for mongot.](https://www.mongodb.com/docs/search/self-managed/current/monitoring/metrics-reference.md#std-label-mongot-metrics-reference) If an initial sync takes longer than expected or an index repeatedly returns to the `INITIAL_SYNC` state, see the following troubleshooting topics: - [Initial Sync Is Slow or Stuck](https://www.mongodb.com/docs/search/self-managed/current/troubleshooting.md#std-label-mongot-troubleshooting-initial-sync) - [mongot Keeps Re-Syncing](https://www.mongodb.com/docs/search/self-managed/current/troubleshooting.md#std-label-mongot-troubleshooting-resync)