The AI model improves—but stops recognising the rarest cases

Which improvement are we rewarding?

A business-document classifier rises from 92.3% to 96.5% correct. Approval seems easy, yet the new version recognises none of the least common category. This is no contradiction: majority improvements can numerically outweigh losing a class. We ask how to read that result and what each metric answers. Data are constructed for analysis, not taken from EL-AI customers or products.

This is single-label classification: each document has exactly one true class A, B, or C and one prediction. Both versions use the same test: 900 A, 90 B, 10 C, one thousand total. Support counts true cases, not predictions. We compare decisions already made, holding aside threshold selection and probability quality to isolate aggregation.

The matrix reveals what the percentage hides

Rows below are true classes; columns are predictions. The diagonal counts correct decisions. Model A’s first row means 850 A cases recognised, 45 called B, and 5 called C. Model B recognises 895 A cases but sends all ten C cases to A. Correct totals rise from 850+65+8=923 to 895+70+0=965. Accuracy divides by one thousand, weighting every document equally without promising to protect every class.

One class at a time: precision, recall and F1

For class c, true positives TP lie on the diagonal, false positives FP elsewhere in its column, and false negatives FN elsewhere in its row. Precision TP/(TP+FP) asks which proposed cases are correct. Recall TP/(TP+FN) asks how many actual cases were found. These differ: making few proposals can preserve precision while missing many true cases.

F1 is the harmonic mean of precision and recall; use 2TP/(2TP+FP+FN) to avoid intermediate denominators. For C in A, TP=8, FP=10, FN=2: F1=16/28≈0.571429, precision 8/18, recall 8/10=0.8. In B, TP=0, FP=0, FN=10, giving F1 and recall zero. Precision is undefined because no C cases are proposed; F1 is defined because its denominator is ten. Software zero-division conventions must not conceal lost coverage.

Three aggregations, three questions

Macro-F1 computes each of K class F1 scores, then their arithmetic mean, weighting classes by 1/K. A gives approximately (0.960452,0.643564,0.571429), averaging 0.725148. B gives (0.980822,0.848485,0), averaging 0.609769. This is not the harmonic mean of averaged precision and recall: F1 is nonlinear, so operation order matters. B improves two classes but loses the third entirely, which macro exposes.

Weighted averaging uses supports 0.90, 0.09, 0.01, giving about 0.928042 for A and 0.959103 for B. It answers a different question, granting rare C only 1% weight. Micro adds TP, FP, FN before taking the ratio. In complete single-label classification, each mistake adds one FP and one FN; with T correct, micro-F1=2T/[2T+2(N−T)]=T/N, accuracy. This equivalence need not hold for multilabel tasks or restricted class subsets.

Even macro cannot make the decision for us

Equal class weights do not mean equal error costs. Macro may overemphasise a marginal category, or still underweight a critical rare event. Ten C examples also support a less stable estimate than nine hundred A examples. The plot is an exact count for this dataset, not statistical proof of future superiority. Independent tests, declared supports, uncertainty, and preselected per-class acceptance criteria remain necessary.

Code reads each matrix, sums rows and columns, and reconstructs every ratio; the archive preserves both versions and assertions checking the paradox. We did not run scikit-learn: its definitions and conventions were consulted, and counts implemented directly in Python. Processing a K×K matrix takes work proportional to K²; N labels can build it in one pass. No execution time is claimed as a service benchmark.

What changes when one error costs more?

Assign a synthetic cost of 100 units to each error on a true C case and one unit to other errors; correct decisions cost zero. These are not measured business costs. A misses 50 A, 25 B and 2 C: 50+25+2·100=275. B misses 5 A, 20 B and 10 C: 5+20+10·100=1025. Over a thousand documents this is 0.275 versus 1.025 units per document, checked by the script. The more accurate version is worse under this cost function.

Mean cost sums each error count times its cost, then divides by N. Macro-F1 does not estimate this cost: both criteria prefer A here for different reasons. If all errors cost one, mean cost would be 1−accuracy and favour B. Selection therefore needs an operational objective, not a supposedly universal best metric. Duplicating every document would also leave the ratios unchanged: the score alone does not reveal independent evidence quantity.

Better for whom, and on which cases?

Version B is better by accuracy and weighted F1, worse by macro, and unable to recognise C in this test. All statements are simultaneously true. Evaluating a specialist model requires population, classes, costs, and required coverage, then reading the matrix, supports, and scores together. The overall total can improve precisely while the service loses a capability we need.

Sources and calculations

scikit-learn 1.9.1 — F1, averaging and undefined cases.

scikit-learn — multiclass aggregation definitions.

m=[[895,5,0],[20,70,0],[10,0,0]]
f=[];N=sum(map(sum,m))
for c in range(3):
 tp=m[c][c];fp=sum(row[c] for row in m)-tp;fn=sum(m[c])-tp
 f.append(2*tp/(2*tp+fp+fn))
print('accuracy',sum(m[c][c] for c in range(3))/N)
print('per-class F1',f,'macro',sum(f)/3)
assert f[2]==0

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 24 October 2026.