Two searches find different documents: how can we combine incompatible scores?

A question the generator cannot solve alone

An assistant searches a technical archive for the right procedure. Keyword search puts a document with the exact component code first; semantic search prefers an explanation using different words. Both look plausible, but their scores use different scales. Adding them can let numerical scale decide instead of relevance. We will explain how Reciprocal Rank Fusion, RRF, combines positions, and especially what information this shortcut discards. The final question is not whether we can produce an order, but whether we have reasons to trust it.

Abstract. We construct two synthetic rankings of five documents and calculate RRF with exact fractions. With identical content, parameter k changes A versus C beyond an analytically derived threshold of about 4.4641. Restricting each list to two results changes the order again; counting the same retriever four times puts an irrelevant document first under the declared labels. We measure average precision on this single case, not a benchmark. RRF avoids direct score-scale comparison, but does not remove decisions about candidates, retriever correlation and evaluation.

What a ranking actually combines

A document is an identifiable object, not merely text similar to another. Our identifiers A, B, C, D and E represent five distinct objects. A ranking is an ordered list with position 1 best. Returning an identifier twice must not give it two votes within one list: our code rejects that input. Identifier deduplication does not remove distinct documents saying the same thing. That second kind of redundancy requires a content decision outside the formula studied here.

L1 is A-B-C-D-E; L2 is B-C-D-E-A. We may picture lexical and semantic retrieval, but no such engines were run: the inputs isolate the mechanism. For evaluation, only B and C are declared relevant. Those labels do not come from the ranks; they are fixed separately, as in a protocol with independent relevance judgments. A real archive would require explaining who defines relevance, for which question, and which documents are missing from the judged sample.

In retrieval-augmented generation, RAG, the generator receives selected documents or chunks before answering. Fusion acts earlier: it decides which candidates to propose and in what priority. It does not check whether a generated sentence is supported or whether a document version is valid. This separation matters: better retrieval may improve available context, but a persuasive ordering does not prove the final answer correct.

A sum of positions, not confidence values

If one engine assigns 12 and another 0.8, adding them has no automatically shared meaning. Rescaling all scores from the first without changing its ordering may change a naive fusion winner. RRF discards score distances and uses positions only. It is therefore invariant to strictly increasing score transformations within each engine, provided candidates and order stay fixed. The price is losing potentially informative distances too: an overwhelming win and a tiny win yield the same first place.

S(d) is document d’s fused score. Index j runs over lists; r_j(d) is its position in list j; k is a nonnegative dimensionless constant. Only lists containing d contribute: under our protocol, absence contributes zero, not an estimated rank. Scores are dimensionless numbers, not relevance probabilities. Ranks start at 1; starting at zero changes denominators and, with k=0, causes division by zero.

The foundational reference is Cormack, Clarke and Büttcher’s SIGIR 2009 paper, pages 758–759. We read its formula, protocol, tables and discussion, not just its abstract. The authors fix k=60 after pilot investigation and compare fusion on TREC collections and LETOR 3. Their own tables contain exceptions to absolute superiority, including CombMNZ results. We do not reproduce those experiments, use their data or transfer their reported performance to our hypothetical archive: new inputs examine the rule’s behavior.

Five documents, every contribution visible

Start with k=0 to make arithmetic transparent. B is second in L1 and first in L2, scoring 1/2+1=1.5. A is first and fifth, scoring 1+1/5=1.2. C is third and second, scoring 1/3+1/2=5/6, about 0.833333. D receives 1/4+1/3=7/12; E receives 1/5+1/4=9/20. Sorting gives B-A-C-D-E. B’s strong support across both lists beats A’s single win, but C remains behind A.

Repeat exactly the same experiment with k=60. B scores 1/62+1/61=0.032522475; C scores 1/63+1/62=0.032002048; A scores 1/61+1/65=0.031778058. D and E follow at about 0.031498016 and 0.031009615. The order is now B-C-A-D-E. No document or relevance label changed. The function turning position into contribution changed. Thus k belongs in the experimental protocol, not among invisible implementation details.

In the table, watch C versus A rather than absolute score size. Moving from roughly 1 to roughly 0.03 does not mean the documents became less relevant. The scales belong to different rules; the value orders candidates under one fixed rule. Adding a list raises scores and may reorder candidates even with fixed k. An operational threshold such as “accept if S exceeds 0.03” cannot transfer across configurations without specific validation.

Deriving where the preference changes

We can explain A versus C without merely trying parameter values. Subtract C’s score from A’s. For nonnegative k every denominator is positive, so the difference’s sign depends only on the common numerator. Combining the four fractions yields a quadratic polynomial. Sensitivity becomes an understandable property of this case, not a plot coincidence. This derives no universally optimal k: it identifies the crossover of two specific rank profiles.

The numerator vanishes at 1 plus or minus the square root of 12. The negative root is outside our domain. For 0≤k<4.4641 it is positive and A precedes C; above that threshold it is negative and C precedes A. Our program uses integers: k=4 retains the first preference, k=5 the second. At the exact threshold the documents tie, requiring an explicit rule. Ordinary ties are broken by identifier, deterministic but unrelated to relevance.

The first-place contribution relative to fifth place is (k+5)/(k+1): 5 at k=0 and about 1.0656 at k=60. Larger k makes nearby rank contributions more alike. For ranks small relative to k, 1/(k+r) is approximately 1/k−r/k². Appearing in m lists then gives a leading contribution near m/k. This is asymptotic intuition, not an identity: it explains why presence counts and list depths matter, while our program always uses complete fractions.

Truncating a list changes the problem

A real system rarely returns every document. Repeat k=60 with only A-B from L1 and B-C from L2. B retains both contributions and its earlier score; A loses its fifth-place presence and retains 1/61; C loses its third-place presence and retains 1/62. The result is B-A-C. We did not merely approximate the tail: we changed preference between present candidates. “Fuse the top two” is a different operational algorithm from fusing both complete lists.

Absence has two distinct meanings: a document may be outside a retriever’s accessible collection, or simply below the requested depth. Our zero does not distinguish them and is not a negative relevance judgment. It avoids inventing an unobserved rank but makes results depend on retrieval budget. Comparisons must record requested depth for each retriever, active filters and whether both saw the same corpus version.

If a relevant document is absent from every list, no fusion of those candidates can recover it. That is a recall limit—the fraction of relevant documents present among candidates—before ordering even starts. Both B and C remain available in our truncated case, so the observed harm concerns position, not relevant-document recall. The distinction guides remedies: broaden candidates when coverage is missing; change fusion or add a reranker when the right documents are present but too low.

Four copies are not four independent pieces of evidence

Keep complete rankings but include four copies of L1 and only one of L2. This deliberately extreme choice represents nearly identical retriever configurations or an integration error counting the same result repeatedly. At k=60, A gets 4/61+1/65=0.080958386 and B gets 4/62+1/61=0.080909572. A wins by a small positive margin; the order becomes A-B-C-D-E. Duplication added no document information; it increased the weight of an existing preference.

RRF does not require statistically independent lists to be computable. The issue is interpreting agreement: five contributions are not five independent checks. Weighted fusion can expose the decision, for example by giving a retriever family one total weight rather than full weight to every variant. We do not prove this improves quality; it is an alternative requiring comparison. Weights and families must not be chosen opportunistically after observing the final test outcome, or evaluation loses its purpose.

Weights w_j are nonnegative dimensionless numbers; this weighted version is discussed, not the system behind our reported results. Giving each of four identical copies weight 1/4 would exactly restore one L1’s total contribution: an algebraic consequence, not a new experiment on real retrievers. Different lists would not satisfy that equivalence. Useful diversity is not the number of services called; it requires examining their errors and additional documents.

Measuring what actually improved

We evaluate the example with average precision for one query. Whenever a relevant document appears, calculate precision up to that position: relevant hits so far divided by positions inspected. Sum these values and divide by the total declared relevant count, two here. Missing relevant documents contribute zero. This rewards early availability of useful documents, not correctness of generated sentences. Declaring the denominator prevents artificially rewarding a list that lost a relevant document.

For B-C-A-D-E, relevant documents occupy positions 1 and 2: AP=(1+1)/2=1. For B-A-C-D-E, they occupy positions 1 and 3: AP=(1+2/3)/2=5/6, about 0.833333. Duplicated L1 puts A first, moving relevant documents to 2 and 3: AP=(1/2+2/3)/2=7/12, about 0.583333. Synthetic labels stay fixed. These are not assistant-reliability percentages, but three ordering measurements for one constructed query.

The essential comparison is with original rankings. L1 has AP=7/12, while L2 already has AP=1. Complete k=60 fusion matches L2; it does not improve it. At k=0 it is worse than L2. Omitting that baseline enables a misleading claim that fusion raises precision from 0.583 to 1, true only after secretly selecting the favorable L1 comparison. Real evaluation needs many queries, documented judgments and across-query variation. One case demonstrates mechanisms and possibilities, not average superiority.

A small implementation with explicit assumptions

Function fuse receives ordered lists and a nonnegative integer k. It accumulates 1/(k+r) per identifier using Python’s standard-library Fraction, preserving exact integer ratios. Only presentation JSON converts results to decimals; it also retains fraction strings. Fraction ordering prevents rounding from deciding a near tie in this small case. This does not require every service to use rational arithmetic: it can be expensive at scale, and each implementation’s numerical behavior needs checking.

With M total positions and U distinct documents, map accumulation takes M updates and O(U) storage. Complete sorting needs O(U log U) comparisons. This counts key operations and comparisons, not latency: rational arithmetic also depends on integer sizes. A partial selection can find only the top K, but we sort everything to expose all scores. No speed benchmark was run.

experiment.py reconstructs all inputs, checks within-list uniqueness, asserts expected orders and computes sensitivity for k=0 through 100. Inputs are deterministic, so no seed applies. The snippet below calls the same attached functions rather than an abbreviated algorithm with different rules. Python suffices for the table; plots require Matplotlib. Versions and outputs are archived. No search engine or generative model was trained or queried to produce these numbers.

Transferring the reasoning to a real archive

A useful evaluation separates three questions. Do candidates contain necessary documents under identical permissions and corpus version? Does fusion make them available within context budget, accounting for chunk length and redundancy? Does the generator use that evidence correctly? Our AP covers only part of the second question, treating documents as comparable units. A very long document may exhaust context before others; document ranking and token selection are not equivalent.

Alternatives include score normalization, retaining distance information while assuming comparability; a reranker, reassessing candidates but adding resources and its own evaluation needs; and retriever-family weighting, explicitly addressing redundancy but requiring validated weights. We assign no performance to these alternatives because they were not implemented and measured on the same query set. Honest comparison first declares what information each uses and which costs or assumptions it adds, before ranking their quality.

The answer: simple fusion, consequential choices

Positions and a rule such as RRF can combine searches without adding incompatible raw scores. But a simple calculation is not a neutral one. Here a parameter changes a preference at a provable threshold, requested depth changes contributions and copying one retriever changes the winner without adding knowledge. The best fusion we executed merely matches an original list: there is no numerical miracle. Trustworthy retrieval requires list provenance, parameters, candidates and independent judgments, followed by separate verification of generated claims.

Cormack, Clarke & Büttcher — Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods, SIGIR 2009, pp.758–759, DOI 10.1145/1571941.1572114.

from experiment import fuse, ap, L1, L2
for name, lists, k in [('full', [L1,L2],60),('top2',[L1[:2],L2[:2]],60),('duplicates',[L1]*4+[L2],60)]:
    result = fuse(lists,k)
    print(name,result['order'],ap(result['order']))

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 16 October 2026.