Same Query, Same Score: Consistent Ranking with StableTfl in Atlas Search

Suppose you run paginated search in your application, or a RAG pipeline where the top ten retrieved documents feed a model. One of your users searches for a recipe collection for "chocolate cake." Page one ends with "Flourless Chocolate Cake." They click next, and the first result on page two is "Flourless Chocolate Cake" again. Between the two requests, nothing about the query changed; only the requested page did. An Atlas Search customer running paginated search brought us this exact problem. The usual workarounds, such as pinning each user to one replica or holding an index snapshot open for every pagination session, require the system to store additional state per query, and they are either too cumbersome or don’t fully close the gap. We fixed it instead by building a new scoring function, StableTfl, that needs no extra state at all. It resolved their issue, and we have since contributed it to Apache Lucene, the open source search engine that powers Atlas Search, so any search system built on top of Lucene can use it.

The two requests were served by two different replicas of the same index, and the replicas ranked the same recipes in a different order. That one reordering produces two concrete failures. The first is the duplicate above: a result from the bottom of page one appears again at the top of page two. The second is a skipped result: the recipe ranked sixth by the first replica should have opened page two, but the second replica ranks it fifth, placing it inside a page one that the user never requested. That recipe never appears on either page.

Neither replica malfunctioned, and neither returned wrong data. The problem sits at the intersection of two things: a scoring function that assumes a single, fixed corpus (the full set of indexed documents), and a distributed system that cannot provide one.

In this blog post, we explore StableTfl, a scoring function we built for Atlas Search to keep search results consistent across replicas. Where BM25 relies on corpus statistics that drift between replicas, StableTfl scores a document from the query and the document alone, so every replica produces the same score for a given query and document, and users see the same results on every page and in every request with nothing extra to build.

Why BM25 scores drift

BM25 is the standard relevance function in lexical search and the default in Lucene. Given a query, BM25 assigns each matching document a score, and results are returned in descending score order. The formula combines three ideas:

• Term frequency. A recipe that mentions "chocolate" ten times is more likely to be about chocolate than one that mentions it once, so more occurrences raise the score, with diminishing returns.
• Length normalization. A long document has more chances to match any given term simply by virtue of its length, so a document's score is discounted by its length relative to the corpus average.
• Inverse document frequency (IDF). Rare terms are more informative. In a query for "flourless chocolate cake", a match on "flourless" indicates relevance more strongly than a match on "chocolate", because far fewer recipes contain it. IDF measures this by counting how many documents contain each term.

Figure 1. The BM25 formula, with corpus-dependent inputs highlighted.

Term frequency is a property of the document alone. The other two are computed from the corpus: the average document length, the total number of documents, and the number of documents containing each term. BM25's score for one document, therefore, depends on every other document in the corpus.

Those corpus statistics describe one node at one moment, and in a live system, they do not stay fixed. Replicas index independently and converge only eventually, so at any instant their statistics disagree by whatever is still in flight. Even a single node's statistics change between two consecutive queries, as new documents commit and segment merges reclaim deletions. When the statistics move, scores move, and when two documents swap ranks across a page boundary, the result is the duplicate and the skip from the introduction.

BM25 itself is not at fault: on a single, static corpus, it is fully deterministic. The failure comes from the environment, and the standard mitigations accordingly try to restore a single fixed corpus. Sticky routing pins a user to one replica, but that replica's own statistics keep changing. Synchronizing statistics keeps replicas consistent with each other, but not with themselves over time. Index snapshots pin a point-in-time view of the index for the lifetime of a pagination session, but the pinned segments cannot be cleaned up, state accumulates per session, and the snapshot covers only one replica. Each mitigation adds a state somewhere. We took a different approach and removed corpus statistics from scoring entirely.

Scoring without the corpus

StableTfl (stable term frequency–length) is the scoring function Evan Darke and I built for MongoDB Atlas Search. We have since contributed it to Apache Lucene as StableTflSimilarity, so it will be available to any system built on Lucene, not only Atlas Search. Its score depends on exactly three inputs: the query term, the term's frequency in the document, and the document's length. Nothing about the rest of the corpus enters the computation, so a document receives the same score on every node at every moment, and two replicas holding the same documents rank them identically.

Figure 2. The StableTfl scoring formula.

The StableTfl formula shares a shape with BM25: it sums over the query terms and, for each term, multiplies a normalized term frequency, which rewards repeated occurrences with diminishing returns, by a rarity weight, which values uncommon terms more than common ones.

The frequency factor is nearly identical. BM25 normalizes the factor by the document’s length relative to the corpus average, so long documents with the same number of occurrences are scored lower. StableTfl normalizes against document length when computing term rarity.

The term rarity weight is where the two scoring functions differ fundamentally. BM25 uses IDF, which looks up how many documents contain the term, and a term found in fewer documents weighs more. StableTfl doesn’t use the corpus statistics. Instead, it estimates term rarity by term length and assigns a higher rarity to longer terms. This generally holds because of a pattern in natural language known as Zipf’s brevity law: frequent words tend to be short, and rare words tend to be long. For the query "chocolate cake," BM25 asks how many recipes contain each word; StableTfl asks only how long each word is, and gives the longer "chocolate" more weight than "cake." Although many individual words break this crude proxy, it holds well enough in aggregate that the cost, measured below, is only about three points of NDCG.

Concretely, StableTfl models p(t, d), the probability that a document of length |d| contains a term of length |t| at least once:

p(t, d) = 1 − (1 − m · 2^(−c · |t|))^|d|

A longer term is exponentially less likely to fill any one position, and a longer document has more positions to fill. A term with low p is rare and receives a high weight through the same log-shaped curve BM25 applies to IDF. Term length is counted in Unicode code points of the analyzed term rather than bytes, so "café" counts as four. The decay constant c = 0.917 was fit by modeling term frequency in English text, and k1 = 1.2 matches Lucene's BM25 default. A later Bayesian hyperparameter search over BEIR returned nearly identical values, so we kept the originals.

StableTfl ships in Atlas Search as a per-field similarity option in the index definition, alongside bm25 and boolean.

Precomputing term rarity

The term rarity function involves a power and a logarithm, and a scoring function can run millions of times for a broad query over a large index. Evaluating transcendental functions for every scored document would be substantially more expensive than BM25's arithmetic.

The solution comes from a detail of Lucene's index format: document lengths are not stored exactly for scoring, but compressed into a single byte. For a given query, term rarity therefore has an input domain of exactly 256 values, and all of them can be computed up front:

Per document, scoring reduces to an array lookup keyed on that byte-encoded length, cache[((byte) encodedNorm) & 0xFF], plus one multiplication and one division, with no transcendental math. Beyond the scoring path, the integration is small: StableTfl is a stateless drop-in Lucene Similarity, and its explain() output mirrors BM25's structure, so existing debugging tools work unchanged.

What it costs, and when to use it

Corpus statistics carry information, and removing them costs relevance. We benchmarked StableTfl against BM25 on 21 datasets from BEIR, a standard benchmark suite for retrieval quality, using identical tokenization for both scorers. BM25 averaged 0.346 NDCG@10 and StableTfl 0.315, a drop of about three points, or 9% relative. NDCG@10 measures the ranking quality of the top ten results; higher is better. Recall@10 shows the same pattern: 0.388 against 0.350.

Figure 3. NDCG@10 per BEIR dataset.

The direction is expected, since StableTfl computes with strictly less information than BM25. StableTfl is the right choice when consistency matters more than the last few points of relevance: multi-replica deployments serving paginated queries, RAG and hybrid pipelines that need reproducible retrieval, and systems where downstream logic depends on stable scores. BM25 remains the default in Atlas Search and the right choice when raw relevance is the top priority.

One further limitation is language. The constants were fit on English text, so other Latin-script languages may need them re-fit, and languages where a word is often a single character, such as Chinese or Japanese, break the term-length proxy entirely.

Future work

Two directions could narrow the quality gap. The constants can be re-fit for other languages and for specialized English domains such as biomedical or legal text. The analyzer also interacts with the model: stemming shortens tokens before the rarity function sees them, so analyzer choice affects the calibration.

The decision itself comes down to the trade described above. BM25 is the right default. StableTfl is for deployments where a duplicated result, or a retrieval set that changes between two identical queries, costs more than three points of NDCG.