When should a classifier abstain? The cost of answers it withholds

Answering less may help, but what are we counting?

A domain classifier recognizes textile defects. A company wants difficult cases deferred rather than always answered. Accepting only high scores seems straightforward, but low error with almost no answers may be unhelpful. Does raising threshold always reduce errors among remaining answers? A complete example separates coverage, risk among answers and total abstention cost.

Abstract. Twenty constructed ordered scores contain six declared errors. Full acceptance gives 30% risk; first seven give zero observed risk,35% coverage and thirteen deferrals. Show nonmonotonic risk-coverage and compute threshold costs with error 1, abstention.1. Observed minimum.065 per case is not independent-test optimization. Derive a one-sided bound for zero errors in a hypothetical independent fixed-threshold sample: seven accepted yield 95% upper risk 34.82%, not zero. No neural network trained.

Define the denominator before the percentage

Classifier f predicts a category; selector g accepts(g=1) or abstains(g=0). For labeled cases loss ℓ is 1 for error,0 otherwise. Selective risk divides accepted errors by accepted count; coverage divides accepted count by all cases. Different denominators matter. Abstention leaves f’s prediction unchanged and decides whether to use or expose it.

At A=0, E/A is undefined: a system never answering has not proved perfect risk. Coverage zero and abstention cost remain defined. These formulas are empirical; future-population risk is distinct. We use observed counts first and later state assumptions for statistical bounds.

Twenty cases, not twenty calibrated probabilities

Scores decrease.99 to.80 by.01. Errors occur at one-based ranks 8,10,13,16,19,20. Full scores and error flags are saved; no real model or customer frequency generated them. Scores rank only:.99 is not 99% correctness probability. Domain models could use margins, softmax or other scores; these methods are not compared here.

Accept first k cases, equivalent to an inclusive score threshold without ties here. All 20 have 6 errors,6/20=.30. First 7 have 0 errors, risk 0/7=0, coverage 7/20=.35. Thirteen still require handling elsewhere, costing time, resources and possibly errors. Reporting correctness without coverage hides part of requested service.

A higher threshold does not guarantee monotonic risk

At k=9 only rank 8 is wrong: risk 1/9≈11.11%. Raising threshold to k=8 removes correct rank 9 but retains the error: risk 1/8=12.5%. Coverage falls 45%→40% while risk rises. The selector rejected a correct answer. At k=7 it also removes the error and risk returns zero. Thus curves can fluctuate; each threshold increase need not improve observed risk.

Score usefulness depends on separating errors from correct cases. Strictly increasing score transforms preserve order and all attainable risk-coverage pairs with matching thresholds, without creating calibrated probabilities. Unlike checking whether 90% confidence means nine successes in ten, selection can be assessed using ranking alone, while explicitly measuring uncovered work.

Rejection costs: count the whole service

Assign normalized cost 1 to accepted error,0 to correct answer,.1 to abstention. These are abstract assumptions, not prices, times or human-review performance. Per-input cost totals accepted-error and deferral costs over all 20. Deferral is modeled by a fixed downstream-inclusive cost, not proof that downstream handling is always correct.

Code evaluates twenty nonempty thresholds plus total abstention, finding observed minimum at k=7:.065 versus.3 always answering and.1 never answering. Selection and reporting use the same constructed errors, no independent test. This is not unbiased future-cost estimation or deployment authorization for a threshold. Different deferral costs can change choice; risk-coverage alone does not define business preferences.

Zero errors do not mean zero future risk

Now a statistical argument separate from constructed data: fix selection before an independent test. Among n accepted independent cases with common error probability r, no errors occurs with probability(1−r)ⁿ. For a one-sided 95% zero-error upper bound solve(1−u)ⁿ=.05; above u all-success is rare. This binomial derivation depends on assumptions and does not certify our artificial twenty cases.

At n=7 upper bound 34.82% is not estimated risk. Under assumptions, repeated sampling gives at least 95% procedure coverage, not a posterior parameter probability.100 accepted without errors give 2.9513%;149 bring below 2%. Accepted sample size matters, not all requests.

The bound does not certify k=7: twenty cases are constructed, not independent population data, and threshold selection used those errors. Pretending it was pre-fixed ignores selection. Independent testing or search-aware corrections are needed. Calling an empirical curve confidence creates no risk control.

Difference from risk control

Geifman/El-Yaniv, Selective Classification for Deep Neural Networks, arXivv 2,1 June 2017, formalize selectors and SGR—Selection with Guaranteed Risk. Definitions, algorithm, binomial bound and experiments were read. Search corrects confidence; fitting/evaluation separate, including 5000/5000 CIFAR-10. Authors’ VGG-16 test coverage.7856, risk.0092 at target.01 are not reproduced here.

Those risks do not transfer to textiles or EL-AI: different data, network, selector. Code uses no SGR, VGG or ImageNet. We separate coverage, observed risk, cost, guarantee. Domain change alters risk/ranking; abstention does not automatically repair it. Rejecting every case in a subgroup can improve totals while hiding absent service.

Reproduce the calculation and answer the question

Package saves 20 scores,20 error flags, all thresholds. Deterministic counts/bounds need no seed. Simple prefix resumming is O(N²); cumulative sums reduce work after general O(Nlog N) sorting. Inspectable denominators, not speed, are the point. Snippet prints points 7,8,9,20 and bounds 7/149, linking code, formulas and figure.

Deployment must evaluate deferral handling, capacity, quality and subgroups, not measured or attributed to EL-AI here. Abstention helps when the full system offers a verified better trade-off. Low accepted error alone can be bought with low coverage, optimistic selection or tiny samples. The answer relates service to risk, not a context-free confidence number.

Yonatan Geifman, Ran El-Yaniv. Selective Classification for Deep Neural Networks, arXiv 1705.08500 v2,1 June 2017. Definitions, SGR, bounds and experiment protocol read 17 October 2026; not reproduced..

from experiment import curve,zero_upper
for row in curve():
    if row["accepted"] in [7,8,9,20]:print(row)
print(zero_upper(7),zero_upper(149))

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 17 October 2026.