You can measure the accuracy of your MongoDB Vector Search query by evaluating how closely the results for an ANN search match the results of an ENN search against quantized vectors by using the same query criteria. That is, you can compare the results of ANN search with the results of ENN search and measure how frequently the ANN search results include the nearest neighbors in the results from the ENN search.
Use Cases
You might want to measure the accuracy of the results if you have any of the following:
Quantized vectors
Large numbers of vectors
Low dimensional vectors
Prerequisites
To try the examples on this page, you need the following:
A cluster with, optionally, the sample dataset.
Procedures
To evaluate the accuracy of your $vectorSearch query results, you must do the following:
Create a MongoDB Vector Search index on the vector field and on any other fields that you want to pre-filter the data by.
We recommend using quantized vectors to improve the storage of your vectors and speed of your queries. If you don't have quantized vectors, you can enable automatic quantization when indexing your
vectortype field.Construct and run the ENN query followed by the ANN query.
Compare the results of the ANN query with the results of the ENN query to evaluate the similarities and differences in the results.
This section demonstrates how to perform the preceding 3 steps against data in the sample_mflix.embedded_movies collection. If you don't wish to use the sample dataset, you can perform the procedures against your own data.
Create the MongoDB Vector Search Index
This section demonstrates how to create a MongoDB Vector Search index for running MongoDB Vector Search ANN and ENN queries.
Run the Queries
Compare the Results
The top nine documents in the example ENN and ANN query results are the same and have the same score. This shows a high-level of similarity in the top results for the query. However, the tenth document in the ENN and ANN query results is different, which reflects a slight variation in the exact and approximate nearest neighbor search.
ENN search examines all possible candidates and returns the closest match to the query based on the similarity score. ANN search uses approximations to speed up the search, which might alter the score of the documents. If you increase the numCandidates value in the ANN query, the results will be a closer match to the ENN query results. However, this would consume additional computational resources and might reduce query speed. The tenth document in the results reflects the trade-off between accuracy and speed.
After quantitatively evaluating results against the ENN ground truth, we recommend testing a set of 100 queries in the same manner and computing the "jaccard similarity" between the result sets. Jaccard similarity can be computed by dividing the intersection between two sets, that is, the overlapping items, by the total set size. This gives a sense for recall performance of ANN queries, including those performed against quantized vectors.
If you notice large discrepancies between your ENN and ANN query results, we recommend tuning the numCandidates value to strike an ideal balance between accuracy and speed for your application.
We recommend that you use judgement lists for a structured list of queries with their ideal results for the ANN query or ENN ground truth values. You can use the ENN query results as the baseline judgement list and then evaluate ANN query results against this judgement list to measure recall, overlap, and performance. Judgment lists provide a way to evaluate whether ANN queries meet the desired accuracy or recall thresholds compared to the ENN baseline. Use LLMs to generate the example queries.
Improve Accuracy with Reranking
To improve the accuracy of your $vectorSearch results, you can rerank them by semantic relevance to the query. Reranking reorders your results to prioritize the most pertinent documents. You can apply the $rerank stage after your $vectorSearch stage to reorder the results using a reranker model.
To learn more, see $rerank Aggregation Pipeline Stage.