This page explores the results of our MongoDB Vector Search performance benchmark. MongoDB offers a range of solutions that support various implementation needs. Customers can choose the levels of performance and accuracy that are suited to their specific business setup.
Summary of Results
At 15.3M vectors using
voyage-3-largeembeddings at 2048 dimensions, MongoDB Vector Search with quantization configured retains 90-95% accuracy with < 50ms query latency.Binary quantization offers a cheaper solution, at about one fourth the price of serving the index. Since it rescores with full-fidelity vectors, it can be a preferable option for many large scale workloads, though we recommend running tests on the sample data set to ensure the performance fits your needs.
We recommend over 1024 dimensions when running larger workloads with quantization.
Selective filters can improve or worsen performance depending on the value selected for
numCandidates.The additional cost of rescoring for binary quantization shows up in reduced throughput when running highly concurrent workloads.
Sharding slightly improves throughput, but we still recommend scaling out number of Search Nodes or the number of available cores on a Search Node to improve throughput.
Recall and Latency Analysis Across Multidimensional Benchmark
The first set of results shows the tests that we ran against a 5.5M document dataset containing multiple dimensionalities of vectors (256, 512, 1024, 2048), all produced using voyage-3-large, within each document.
To view the full chart, see the Claude artifact.
The scalar quantized results all start at higher levels than the binary quantized results, but stay at their asymptotic level even as numCandidates increases. Conversely, binary quantized queries yield more accurate results as more numCandidates are requested, approaching the asymptote of scalar quantization, and in some cases passing it, at the cost of higher latency, particularly above numCandidates of 1000.
Lower values of limit generally require higher numCandidates to approach 100% accuracy, because the query is more selective about the very top results. This effect is particularly visible in the binary quantization plot. Higher-dimensional vectors (1024d and above) reach the 90-95% target range at lower numCandidates, as the richer representation makes near neighbors easier to distinguish. The tests that limit the results to 100 reach the 90-95% target range with lower numCandidates than the tests that limit the results to 10, because the larger result set gives the search more opportunities to surface relevant neighbors.
Given this information, we determined that when working with a large dataset, we recommend having dimensionality of at least 1024d and applying quantization to scale than having lower dimensionality and not using quantization, with the amount of vectors requested for the use case playing a factor as well.
Larger Benchmark Results
For the larger 15.3M vector dataset, we fixed the dimensionality to 2048d and examined the impact of quantization, filtering and concurrency on performance. We chose to pin on 2048d based on the results from the previous set of tests showing that higher dimensions retained recall in a more favorable manner, though 1024d would have likely served just as well to reach the 90-95% recall target.
Recall and Latency Analysis
We observed that it takes significantly more numCandidates when using binary quantization to achieve the 90-95% recall target compared to baseline. Higher numCandidates generally means higher latency, but this might vary.
To view the full chart, see the Claude artifact.
Filtering
We observed what happens to recall and latency when using a selective filter on the dataset for ~500k items of the 15.3M items are in the Pet Supplies Category (~3% of the corpus):
To view the full chart, see the Claude artifact.
A selective filter is a filter that matches only a small slice of the data (in this case, ~3%). Because the search engine must now find nearest neighbors within this small subset, it has to explore more of the index and evaluate more candidates to reach the same recall. Each query does more work than an unfiltered query.
As anticipated, we can see that the 3% selective filter can cause queries to do more work. For binary quantization at lower limit values, this required roughly 4x as much work to achieve 90-95% recall compared to the unfiltered queries.
Future improvements in Lucene 10, which support Acorn-1 search strategies for Hierarchical Navigable Small Worlds, might improve this process. However, performing ENN when the number of requested candidates exceeds the number of vectors matching the metadata filter within a segment demonstrates that filter selectivity plays a large part in query performance, regardless of the selected quantization regime.
Concurrency
These tests scale concurrent requests between 1, 10, and 100 at the various limit values when using scalar and binary quantization. numCandidates are selected by choosing values that allow 90-95% recall to be achieved:
To view the full chart, see the Claude artifact.
We observe that scalar quantization achieves higher QPS at all values of limit, likely because each query can complete with lower numCandidates and without rescoring. At higher concurrencies, the QPS curves for concurrency 10 and concurrency 100 are close together, suggesting that the system is efficiently using available CPU resources, and that extra concurrency primarily affects latency.
One exceptional data point is limit 10, concurrency 100 for scalar quantization yielding significantly higher QPS. This is likely because no rescoring and lower limit values means fewer comparisons are performed for this query, allowing each request to return more quickly and make the cores available to serve other queries.
Scaling out the number of available vCPUs to serve requests, either by scaling up the Search Node tier or scaling up the number of Search Nodes from the minimum of 2 up to 32 nodes might help resolve concurrency bottlenecks and allow you to scale well into the thousands of QPS.
Sharding
We also observed what would happen if the cluster and collection were sharded (on _id) and unfiltered queries were issued against a binary quantized index.
To view the full chart, see the Claude artifact.
Here, we see that the sharded results have a higher QPS at limit 10 as the lower value of numCandidates can be provided to produce results in the 90-95% recall range. This is because the 15.3M dataset is split across three shards, each of which have their own indexes filled with 5.1M vectors spread across segments containing HNSW graphs. We are functionally doing a less advanced search where it is more likely that each query scatter gathered across 3 shards simultaneously could find the closest n vectors. For this reason, the QPS is slightly higher when sharding since you can reduce numCandidates and have more cores available to serve queries, but the difference is not as significant to justify the increased cost of sharding the cluster. Most often you should shard your cluster for reasons related to your operational workload, not because you need to scale throughput for vector search.
Note
The values are similar for limit 100, numCandidates 200. We might expect this to perform better for filtered queries with an intelligent shard key matching used as a filter.