A hundred tests, no errors: how much can we trust the AI?

The best-looking result can be the easiest to misread

A document-extraction system passes one hundred checks without an error. Can we say it never makes mistakes? We observed none in those checks, but future risk remains unknown. We want a quantitative interpretation that does not turn a limited success into a universal guarantee. This article uses synthetic cases, not an evaluation of an EL-AI product.

Precisely: after n error-free trials, what error probability remains compatible with a 95% upper confidence bound? We derive it from the probability of seeing no errors, then plan a test. Only basic probability and logarithms are needed. We distinguish the point estimate, zero here, from uncertainty, which remains.

Before calculating: what counts as a trial?

Define error before testing, for example an incorrectly extracted required field. Each trial yields 1 for error, 0 otherwise. Assume n independent requests from the same population and constant error probability p. p is dimensionless; n is a count. These are strong assumptions: a hundred copies of one document are not a hundred independent situations merely because the program runs them separately.

Under these assumptions the error count X is binomial. For zero errors, each trial succeeds with probability 1−p, and independence lets us multiply n times. With true risk 1%, one hundred successes occur with probability 0.99^100, about 36.60%. A perfect observed run is therefore unsurprising even at that nonzero risk.

The bound comes from reversing the question

Find p_U for which zero errors would occur with probability alpha=0.05. At larger risks it is even less likely. Solving (1−p_U)^n=alpha gives the expression below: take logarithms, divide by n, then exponentiate. This is the zero-error case of the exact one-sided binomial bound. We invert a test rather than equating risk with observed frequency.

For n=100 the bound is approximately 0.029513, or 2.9513%; for a thousand it is about 0.2991%. “95% confidence” describes repeated-sampling coverage of the procedure, not a 95% posterior probability for the fixed p after this experiment. Posterior statements require an explicit Bayesian model and prior. The bound also concerns the sampled population and chosen error definition, not every imaginable future document.

Where does the rule of three come from?

For large n, a=log(alpha)/n is small and negative. Expanding exp(a)=1+a+a²/2+… gives p_U≈−log(alpha)/n. Since −log(0.05)≈2.995732, this becomes 3/n. It approximates a one-sided 95% bound, not every confidence level. At twenty trials it gives 0.15 versus exact 0.139108. The normal Wald interval substitutes zero into the variance and collapses to zero, exactly the misleading result to avoid.

Plan the test before seeing its result

To achieve p_U≤p_0 with zero errors, invert again. Dividing by negative log(1−p_0) reverses the inequality: n≥log(alpha)/log(1−p_0), rounded up. Targets of 1% and 0.1% require 299 and 2995 trials. Code verifies n passes while n−1 does not. This plan is conditional on zero errors; observing one requires the binomial bound for that count, not this shortcut.

Three practical limits matter. First, twenty independent documents each repeated five times with identical outcomes may provide twenty independent units, not a hundred; with perfect within-document dependence the bound returns to 13.91%. Second, a central two-sided 95% interval assigns 2.5% to the upper tail, giving the larger 1−0.025^(1/n). Third, selecting the best of five systems after testing does not inherit a simultaneous guarantee from separate bounds. A conservative Bonferroni adjustment uses alpha/5 when the plan requires covering all five.

Repeatedly checking and stopping as soon as the bound crosses a target can also alter coverage without an appropriate sequential plan. Larger n does not repair contaminated data, incomplete labels, or new document types. Recording population, independence, model version, error criterion, and stopping rule identifies the question actually answered. Computing the formula takes constant work; gathering independent evidence is the substantial task.

Repeating the same trial creates no new evidence

The product (1−p)^n depends on independence. If a hundred runs repeat one situation with perfectly shared outcomes, zero errors has probability 1−p, not (1−p)^100. Twenty independent situations each repeated five times with perfect within-group dependence give (1−p)^20. These exactly describe two cases, not a universal effective-sample-size formula. Partial dependence needs an appropriate model, not dividing n by an intuitive factor.

We can also check approximation direction. Set positive x=−log(alpha)/n; the bound is 1−exp(−x). Since exp(−x)≥1−x, the exact bound is at most x, so the logarithmic approximation is conservative here. Code checks all table points. For very large n, subtracting nearly equal floating-point values loses digits. expm1 evaluates exp(a)−1 appropriately near zero; log1p does likewise for log(1+a). These improve calculation without changing statistical assumptions.

No errors is an observation, not a promise

After a hundred independent error-free trials, report zero observed and a one-sided 95% upper bound of about 2.95%, under the stated assumptions. Neither zero risk nor universal reliability follows. The calculation does not diminish a successful test; it tells us the evidence’s strength and the work needed for a stronger target. Attached code computes bounds, not fictitious real evaluations.

Sources and reproducibility

NIST/SEMATECH — binomial probability model.

NIST Dataplot — exact one-sided and two-sided binomial limits.

import math
alpha=.05
for n in [20,100,1000]:
 upper=-math.expm1(math.log(alpha)/n)
 assert abs((1-upper)**n-alpha)<1e-12
 print(n,upper)
print(math.ceil(math.log(alpha)/math.log1p(-.01)))

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 24 October 2026.