Rare defects: how AUC 0.94755 can coexist with 91.74% false alerts

Applied research · AI evaluation · 24 September 2026

Abstract: high AUC does not determine how many alerts are correct

We construct a classifier with three score levels and exact AUROC 0.94755. Keeping class-conditional score distributions identical, one threshold’s precision drops from 90.91% to 8.26% when defect prevalence falls from 10% to 0.1%. Ranking has not deteriorated; population composition has changed. This article derives the result, compares two thresholds, and shows why operational selection requires review capacity and error costs alongside aggregate metrics.

This is a deterministic synthetic experiment calculated with Python: no trained model, production images, or EL-AI client. The setting is industrial quality control, but the distinction between conditional probabilities is broader. Prerequisites are elementary probability, Bayes’ theorem, and confusion matrices. Quantities are known by construction; uncertainty in real estimates is discussed separately.

1. Three denominators, three questions

Define a defective item as positive. TP are flagged defects, FN unflagged defects, FP flagged good items, and TN unflagged good items. Sensitivity TPR divides TP by all defects. False-positive rate FPR divides FP by all good items. Precision PPV divides TP by all flagged items. Confusing these denominators turns good class-conditional performance into an unjustified promise about alerts.

The last formula writes joint defect-and-alert probability as π·TPR, and alert probability as the sum of two disjoint cases: flagged defect and flagged good item. It is Bayes applied to a fixed threshold. With no alerts the denominator is zero and precision is undefined. Calling it one would make a never-alerting system look optimal even when it misses every defect.

2. A complete ranking, not just one ROC point

Computing AUC requires more than sensitivity and FPR at one threshold. Introduce ordered scores H>M>L. Among defects, 60% receive H, 30% M, and 10% L. Among good items, 0.1% receive H, 0.9% M, and 99% L. A strict threshold flags H only; a more permissive one flags H and M. These values define an explicitly synthetic mathematical population, not observed data.

With the standard half-credit convention for ties, AUC is the probability that a randomly selected defect scores above a good item, plus half the equality probability. Strict wins weigh 0.6×0.999+0.3×0.99=0.8964. Ties weigh 0.6×0.001+0.3×0.009+0.1×0.99=0.1023. Thus AUC=0.8964+0.1023/2=0.94755. The script also checks trapezoidal ROC area; both constructions agree.

Segments between points represent the tie convention or randomized decisions within a level, not additional scores possessed by the model. Arbitrarily ordering equal-score examples can produce different staircases. Tie handling must therefore be explicit, especially when discrete or quantized scores create many equal values. Here ties contribute more than 0.05 to AUC; ignoring them is not a negligible detail.

3. Changing prevalence while keeping ROC fixed

Choose N=1,000,000 items as the counting scale. We did not inspect a million objects; exact probabilities are multiplied by a conventional total. At H+M, TPR=0.9 and FPR=0.01. With prevalence 0.1%, there are 1,000 defects: 900 flagged and 100 missed. Among 999,000 good items, 9,990 are falsely flagged. Total alerts are 10,890, only 900 correct.

ROC stays identical because TPR and FPR each condition on a class. Precision changes because it conditions on alerts, mixing classes in proportions depending on π. This invariance requires unchanged class-conditional score distributions; it is a prevalence-shift assumption, not a guarantee when camera, material, lighting, or defect type changes. If scores within classes also change, ROC can change.

4. From precision target to false-alert requirement

Suppose we require at least 80% correct alerts while retaining TPR=0.9. Solving Bayes for FPR gives a quantitative requirement. With π=0.001 and required precision p*=0.8, FPR must be at most about 0.0002252, or 0.02252%. The H+M threshold’s 1% is over forty times larger. A difference that looks small on ROC completely changes review load when negatives comprise nearly the whole population.

Tightening to H lowers FPR to 0.001 but also TPR to 0.6; precision reaches only 37.52%. We cannot retain the old TPR after changing threshold. In this three-level classifier, even randomly selecting a fraction of H cases does not improve expected precision; TP and FP shrink proportionally. Reaching 80% would require new discriminative information, another population, or an effective second check, not a magical threshold.

5. Higher precision can still be the worse choice

At π=0.001, H generates 1,599 alerts and misses 400 defects; H+M generates 10,890 alerts and misses 100. If review capacity is 2,000 items per period, only H satisfies the limit among these choices. But if a missed defect carries 100 penalty units and a false alert one unit, H costs 40,999 versus 19,990 for H+M. These penalties are hypothetical and dimensionless; they expose the trade-off and are not actual prices or damages.

Precision describes alert composition; it does not value missed defects. Capacity specifies a constraint; it does not establish economic preference. A coherent decision must first state the objective and constraints. If a saturated queue also delays true positives, the static cost above is insufficient; a temporal review model is needed. Adding a human reviewer without considering volume does not remove the statistical problem.

6. What a balanced test establishes

A test rich in defects can estimate sensitivity by defect type more efficiently than a random sample containing very few. Problems arise when its precision is transferred directly to production. At artificial prevalence 50%, H+M has PPV=0.9/(0.9+0.01)≈98.90%, far from 8.26% in our rare-event scenario. Positive enrichment helps study the model but does not represent event frequency in the actual stream.

Our rates are exact; real rates are estimated. Zero false positives among 1,000 negatives does not prove FPR=0. With independent trials and fixed rate f, zero events have probability (1−f)^1000. Setting this to 0.05 gives the one-sided 95% binomial upper bound, about 0.00299, much higher than the earlier 0.0002252 requirement. Sufficient representative negatives are needed, not just a percentage rounded to zero.

Independence and representativeness matter: nearly identical frames of one item are not equivalent to independent items, and a clean laboratory line may not represent a worn production line. Choose thresholds on development data and evaluate on a separate test. Repeated adjustment after seeing test errors makes the test part of optimization. Correct Bayes algebra cannot rescue FPR estimated from contaminated data.

7. Sources, reproducibility, and interpretation

Saito and Rehmsmeier (PLOS ONE, 2015) analyze ROC and precision-recall using simulations and applied data; we consulted foundations, methods, and results. Their work helps distinguish ranking from positive-prediction reliability. We use an original discrete population and exact calculations rather than replicating their datasets. The point is not to abolish ROC but to match it to its question, without substituting it for prevalence, threshold, and operational load.

The executed archive uses Python 3.14.0, the standard library, and no randomness; Matplotlib 3.11.2 produces four localized figures. It saves probabilities, counts, AUC checked by two methods, and prevalence-sensitivity curves. Counts are exact for the constructed population apart from floating-point representation. The snippet reconstructs the main effect and binomial bound; it does not train a classifier.

for pi in [.1, .01, .001]:
    precision = pi*.9/(pi*.9+(1-pi)*.01)
    print(pi, precision)
print(1-.05**(1/1000))

For AI applications EL-AI intends to explore, this is a technical evaluation criterion, not a claim of achieved industrial results. The conclusion is verifiable: AUC=0.94755 and precision=8.26% can coexist without mathematical error. Interpretation requires knowing the threshold, incoming population, and consequences of each decision. Competence lies not in showing the highest number but in making its meaning explicit.

Saito T., Rehmsmeier M. (2015), The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets, PLOS ONE 10(3): e0118432.

Code, results, and instructions. JSON results. Source accessed 24 September 2026. AI-assisted analysis without claiming peer review. Illustrative ImageGen cover, not a photograph of an EL-AI or client production line.