Conformal prediction: 90% overall can hide 50% in a rare regime

Abstract: read a guarantee together with its population

A prediction interval with 90% marginal coverage does not promise 90% in every operating regime. We study two synthetic forecast-error regimes: one frequent and concentrated, the other rare and six times more dispersed. The mixture quantile covers every case in the first and only half in the second, while reaching 90% overall. Groupwise calibration corrects the imbalance under precise assumptions but can yield unbounded intervals when a group has little data. We derive both results and check them with 2,000 reproducible calibrations.

The illustrative setting is power forecasting, with errors in kW; these are neither plant measurements nor EL-AI model results. Prerequisites are probability, cumulative distributions, and order statistics. The experiment assumes independent data. Real energy series can have autocorrelation and distribution shifts, so the example does not transfer its guarantee without checking those conditions.

1. Split conformal and the correct rank

Start with a predictor f fixed using separate training data. On n fresh examples compute s_i=|y_i−f(x_i)|. Sort scores and take the kth, k=ceil((n+1)(1−α)). A new interval is [f(x)−q,f(x)+q]. If k=n+1, set q=∞: using the observed maximum instead changes the guarantee. The n+1 correction is not a software detail; it accounts for the new score’s position among calibration scores.

Under exchangeability and no ties, the new score’s rank among n+1 values is uniform. Exactly k ranks are covered, giving probability k/(n+1), at least 1−α. Ties with a ≤ comparison can make coverage conservative. Probability averages over random calibration and the new observation. It does not say every fixed calibration covers exactly 90%, or every input has that conditional probability. Training or choosing scores using calibration labels can invalidate the argument.

With n=99 and α=0.10 the rank is 90, so theoretical marginal coverage with continuous scores is 90/100. With n=4 the same target requires rank 5 and an unbounded interval; the maximum of four scores would cover only 4/5=80% on average. The snippet uses integer arithmetic for 90%, avoiding rounding ambiguity. It does not interpolate generic quantiles that might differ from the required order statistic.

2. A mixture where the rare group bears the errors

In regime A, occurring 80% of the time, absolute residual R is uniform from 0 to 1 kW. In regime B, occurring 20%, R is uniform from 0 to 6 kW. One can construct f=100 kW and add a random-sign error of magnitude R; symmetric coverage only needs R. Group membership is known at prediction time and defined before calibration. Scales are chosen for transparent calculation, not estimated from a plant.

The population 90% quantile is q=3 kW. Both regimes receive total width 6 kW, yet coverage is 100% in A and 50% in B. The marginal guarantee has not failed: the weighted mean is exactly 90%. The operational question—how reliable is the interval in B?—differs from the guaranteed quantity. Using overall coverage to answer that question is an interpretation error.

3. Repeating calibration with analytic test coverage

We draw 2,000 independent calibration sets of 99 residuals, seed 20260924. For each, compute q and population coverage from the known CDFs. We do not simulate millions of test points: coverage conditional on the sampled q is analytically available. The mean across calibrations is a Monte Carlo estimate. Results are 90.0234% overall, 99.9559% in A, and 50.2937% in B. Overall-coverage standard deviation across calibrations is about 3.04 percentage points, not the standard error of the mean.

4. Groupwise calibration: benefit and information cost

Split the same 99 residuals by group and compute quantiles using n_A and n_B, each with the n_g+1 correction. Under within-group exchangeability and predefined groups, the rank proof applies separately. This provides marginal coverage within each group, not conditional coverage at every possible x. At population level q_A=0.9 and q_B=5.4 kW: widths 1.8 and 10.8 kW, weighted mean 3.6 kW. The difficult regime finally receives a wider interval.

Groupwise simulation gives 90.5667% in A and 92.3283% in B. Extra coverage is consistent with coarser rank discretization in the small group. But three calibrations contain fewer than nine B observations, requiring an infinite threshold. Mean width over all replications is therefore infinite. The 3.66475 kW mean applies only to the 1,997 finite replications and must not be reported as the overall mean. The population limit of 3.6 kW does not remove this finite-sample problem.

5. Limits and technical interpretation

Do not silently replace infinity with a finite maximum: that loses the stated guarantee. A known physical response bound can restrict the set, but is an additional documented assumption. Alternatives include more data, less fragmented groups, or a different coverage target. Groups selected after inspecting errors, or a regime classifier that errs at test time, require fresh analysis. Changing regime frequencies also changes coverage of a fixed pooled threshold.

Angelopoulos and Bates’ guide, arXiv v6 dated 7 December 2022, supplies the consulted procedure, diagnostics, group calibration, and Appendix D proof. Our mixture, parameters, and simulation form a separate educational example, not a reproduction of their benchmarks. The operational conclusion is that coverage must specify its averaging population, calibration procedure, and width cost. Marginal coverage is useful when interpreted as the guarantee it actually proves.

Angelopoulos A. N., Bates S., A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification, arXiv:2107.07511v6 (2022), §§1,3,4.1, Appendix D.

from math import inf
def q90(scores):
    n = len(scores)
    k = (9*(n+1)+9)//10  # ceil(0.9*(n+1)), exact integer arithmetic
    return sorted(scores)[k-1] if k <= n else inf
print(q90([1., 2., 3., 4.]))
q = 3.
print(.8*min(q,1.) + .2*min(q/6.,1.))

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 24 September 2026.