Two AI models differ by two points: but on which cases?

The dashboard number hides the comparison

A company compares two models that classify component images. On the same set of 1,000 images, A is correct 900 times and B 920. B gains two percentage points: is that enough to call it a reliable improvement? We need information lost from the dashboard: does B correct A’s mistakes without adding others, or solve many cases while worsening almost as many? Final percentages can be identical in these two situations while evidence about the comparison differs substantially.

Abstract. We construct two synthetic tables with the same sample size and identical marginal accuracies but different numbers of discordant cases. We derive exact McNemar testing as a binomial comparison conditional on cases where only one model is correct. The first comparison produces p about 0.00000191, the second about 0.1565, although both show a two-point observed advantage. We explain what this difference means, why it does not measure business value, and which assumptions are required. Calculations and figures are reproducible; no model was trained or evaluated on company data.

Two answers for each image

The essential prerequisite is pairing. Each image has an identifier, a reference label and two outcomes: correct or incorrect for A and for B. Both responses refer to the same case. A difficult image may challenge both models; an easy one may be solved by both. These are not separate experiments. Define four counts: a for both correct, b for only B correct, c for only A correct, and d for both incorrect. These letters are a local convention: check table orientation when comparing software or books.

In comparison I, B retains all 900 of A’s correct answers and adds 20; both fail on the other 80 cases. In comparison II, B fixes 100 of A’s errors but introduces 80 regressions on images A solved. The balance is still +20. These are deliberately constructed outcome populations, not two actual evaluations. The second case, with no shared errors, is intentionally extreme: it exposes information erased by percentages alone. We do not suggest this error structure is typical of real models.

Write what the dashboard calculates. N is the total number of images; Acc is the fraction answered correctly. Difference Δ̂, “estimated delta”, is B’s accuracy minus A’s. Count a cancels in subtraction: shared wins favor neither model. What remains is b minus c, divided by N. The difference is 0.02, or two percentage points, not a 2% relative increase. Relative to 90%, the accuracy increase is about 2.22%; we use points throughout to avoid ambiguity.

What statistical question are we asking?

The sample description is certain: B has 20 additional successes in both tables. Inference concerns a population of cases from which the sample might have come. We ask whether the two fixed models have equal success probability in that population. Null hypothesis H₀ expresses this equality. It does not say they always give the same answer or fail on the same images. Very different models can have equal overall accuracy. The test concerns a difference in rates, not internal architectural similarity.

Assume images are sampled independently from the same target population, reference labels are correct, and models and thresholds were fixed before seeing this test. Required independence is between outcome pairs from different images, not between A and B on one image. Consecutive frames of the same part, duplicates and nearly identical crops can readily violate the first assumption. A mathematically exact test does not repair a flawed collection protocol. These conditions remain explicit because they delimit what the result can support.

From discordant cases to an ideal coin

Let πb be the population probability that only B is correct and πc that only A is correct. The population accuracy difference is πb − πc, by the same algebra as for the counts. Thus H₀ is equivalent to πb = πc. Given a discordant case, we already know one model won: under H₀, which one? The conditional probability that B wins is πb/(πb + πc), equal to 1/2 when the denominator is positive. This is where the ideal coin comes from; it does not assume models literally guess by flipping coins.

m is the observed number of discordant cases: 20 in I and 180 in II. K is a random variable counting how many B wins we would see among m discordant cases if H₀ held. Its observed value is b. Conditioning on m means holding that number fixed during the test. Under the sampling assumptions, win directions follow a binomial distribution. The coin analogy applies to those conditional directions, not to image difficulty or overall error probability.

Building the p-value without a black box

We measure how unusual a win imbalance at least this large would be under H₀. Use a two-sided test because deviations favoring either A or B count. For each possible k, binomial coefficient C(m,k) counts ways to place k B wins in m positions; each sequence of directions has probability 2⁻ᵐ. Sum the tail up to the smaller of b and c, then double it to include the symmetric tail. Cap the result at one because a probability cannot exceed it.

In I, b = 20 and c = 0. The observed extreme is all 20 wins going to B. The two-sided test also counts the opposite extreme, all going to A. These sequences have total probability 2/2²⁰ = 1/524288, about 0.00000190735. This is not the probability that B is worse or that the null is true. It is the probability of outcomes at least this extreme within the null model, conditional on 20 discordant cases. This distinction prevents turning a p-value into product confidence.

In II the test sees 100 B wins and 80 A wins. Under H₀ the distribution centers on 90 wins each; observed B wins are ten above that center. Summing probabilities for K ≤ 80 and K ≥ 100 gives about 0.156531. With significance threshold α = 0.05 fixed beforehand, I rejects H₀ and II does not. Failure to reject does not prove equality: this test, on these data at this threshold, does not provide enough evidence to reject it.

Read the figure from the blue center: these outcomes are most probable if discordant directions are balanced. Then find the black line. It lies at the extreme in the first panel and still near the central region in the second. One bar’s height is not the p-value; both red tails must be summed. Vertical scales differ between panels, so heights should not be directly compared as “how good B is”. The graph visualizes the test model, not measured performance.

Why regressions change uncertainty

We can understand the difference without a hypothesis test. For each image define Dᵢ: +1 if only B wins, −1 if only A wins, and zero if both succeed or both fail. Their average is Δ̂. I contains twenty +1 values and no −1; II contains one hundred +1 and eighty −1. The balance remains twenty, but the second sum results from cancellation of many more contributions. A different sample can alter that balance more easily when many wins offset many losses.

The next formula quantifies the intuition. Dᵢ² is one exactly on discordant cases, so the sum of squares is m. Sample variance of differences is [m − NΔ̂²]/(N−1). Divide by N and take the square root to estimate the standard error of the mean difference. “Standard error” means estimated sampling variability, not the model’s classification error count. It is not the exact p-value, and we do not use this approximation to replace the binomial test just derived.

With the same N = 1,000 and Δ̂ = 0.02, estimated standard error is about 0.00443 in I and 0.01341 in II: 0.443 and 1.341 percentage points. Pretending the accuracies came from independent samples gives about 1.279 points for both with an unpaired formula. This loses exactly the distinction we need. Pairing does not always guarantee a smaller standard error; it depends on the relationship between outcomes. The second comparison shows that ignoring this relationship can also underestimate variability.

Another boundary concerns the origin of the 1,000 images. If they were the entire universe of interest, its difference would simply be the observed one: a test adds no images. Inference makes sense when connecting a sample to future case generation or a larger population. A convenient public benchmark does not automatically represent a production line. Explain the intended generalization, excluded cases and how included cases were selected, even when the arithmetic is impeccable.

Exact does not mean universal

The exact test sums binomial probabilities without a chi-square approximation. “Exact” concerns the reference distribution under assumptions, not perfect labels or a certain conclusion. An asymptotic statistic, (b−c)²/(b+c), can be compared to a chi-square distribution with one degree of freedom; a continuity-corrected variant also exists. With few discordant cases the approximation may be unsatisfactory. We use the same binomial calculation in both examples so that changing test variants does not confound the comparison.

If m = 0 there is no observed discordance; the code returns p = 1, consistent with absent directional evidence. This does not prove models will agree on every future image. Discreteness also matters: not every p-value is possible for fixed m, and the exact test can be conservative. A 5% threshold is not a natural law; a rule should be declared before comparison. Changing it after seeing 0.1565 would adapt the protocol to the result.

From significance to the business decision

A small p-value does not establish that B should be released. The observed advantage remains two points in both cases; assess its size, uncertainty, inference cost and error consequences. In industrial quality, missing a critical defect and rejecting a conforming part can have very different costs. Accuracy counts both as one error. Before testing, decide whether that metric answers the operational question. Comparing weighted losses requires an analysis suited to those losses, not arbitrary substitution of weighted counts into McNemar’s formula.

The test concerns the two fixed models. Retraining A and B with different data, seeds or hyperparameters could produce different models; that variability is absent from these counts. We are not generally comparing two learning algorithms. Likewise, selecting B after comparing dozens of candidates on the same test requires accounting for selection. This procedure does not remove multiple-comparison or repeated-test-use problems. The connection between question, sampling and conclusion must stay visible.

Reproducible code and results we must not invent

The attached package contains both tables, full null distributions and their generating program. Function exact uses integer combinations and Fraction to sum the tail as a rational number, converted to decimal only for output. Function case reconstructs accuracies, discordance, difference and standard error. Assertions check equal accuracies across scenarios and p-values on opposite sides of 0.05. No randomness is used, so a seed is not applicable. The following snippet needs experiment.py in the same extracted directory.

A proposed business evaluation should retain each model’s outcome on each case, not just totals. This enables inspecting regressions, checking labels and linking changes to error categories defined in advance. This article does not perform that evaluation: no images were collected, operators involved or EL-AI products compared. The cover is illustrative. Results demonstrated here are mathematical relations and calculations on synthetic tables; the application protocol remains a proposal to adapt to actual data.

Two points do not tell the whole story

The answer is concrete: when evaluating two models on the same cases, final accuracies are insufficient. We need to know which errors each corrects and which regressions it introduces. Twenty improvements without regressions and one hundred improvements with eighty regressions produce the same balance but different statistical evidence. The exact test makes that difference checkable under stated assumptions. The final decision also requires the operational meaning of errors: a good evaluation report shows both the numerical difference and the conditions making its interpretation credible.

Methodological source and publication limits

We consulted primary NIST Dataplot documentation, “McNemar Test”, updated 11 December 2023: paired comparison definition, assumptions, and exact and asymptotic variants. The substantive hypothesis is equality of discordant probabilities; the word “independence” in the page’s descriptive heading is not interpreted as a test of independence between the models. Counts, numerical explanation, code and figures are this article’s educational examples. They do not reproduce NIST’s demonstration dataset and are not original research or a peer-reviewed article.

NIST Dataplot — McNemar Test (2023-12-11).

from experiment import case
for label, counts in [('I', (900,20,0,80)), ('II', (820,100,80,0))]:
    r = case(*counts)
    print(label, r['A_accuracy'], r['B_accuracy'], r['discordant'])
    print('exact p', r['exact_p'])
    print('paired SE', r['paired_standard_error'])

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 9 October 2026.