Does top-p 0.9 really remove 10%? Inside next-token selection

A familiar control, an often misunderstood promise

We configure a language model with top-p=0.9. It is tempting to interpret this as “keep exactly 90% of possibilities” or “discard the worst 10% of answers”. Neither describes the rule correctly. It concerns probability mass assigned to next-token candidates, not a percentage of sentences or a judgement about truth. This matters when comparing configurations or reproducing generation.

What is actually retained, and how far does truncation change the distribution? Five synthetic tokens A–E have no assigned words or quality labels. Exact fractions let us inspect cutoff boundaries, ties, and temperature. No model is queried: the experiment checks a mathematical mechanism. Research findings discussed later belong to their authors and are not reproduced here.

Probability, tokens, and support

A token is a vocabulary unit: a word, part of one, or another fragment. Given the preceding text, the model assigns each candidate a next-step probability summing to one. Call this distribution P; its support comprises candidates with nonzero probability. A sampler draws from the distribution supplied to it, P or a transformed Q. Reducing support and reallocating probability mass both need description.

Sort candidates by decreasing probability and keep the shortest prefix reaching at least p. This prefix is the nucleus. Let k be its size and q the actual retained mass, the sum of its token probabilities. p is a threshold; q is the outcome. We explicitly choose greater-than-or-equal and deterministic identifier ordering for ties.

V is candidate count, S the nucleus, and Q the post-truncation distribution. Division by q restores total mass one. Probability ratios within S stay unchanged, but each absolute probability rises if q<1. Excluded candidates become impossible at that step. No term labels a token “correct”.

The five-candidate example

Let P=(0.52,0.18,0.15,0.10,0.05), with cumulative masses 0.52,0.70,0.85,0.95,1. At p=0.9 the first three are insufficient; the fourth crosses the threshold. We retain four of five tokens, 80% of candidates, and 95% of probability mass. The discarded mass is 5%, not 10%. Tokens are discrete: standard truncation does not keep only half the fourth candidate.

After dividing by 0.95, Q=(52/95,18/95,3/19,2/19,0). A rises from 0.52 to about 0.54737 and E becomes zero, while the sum stays one. We have not drawn any token: these are exact calculated probabilities, not observed sample frequencies. JSON retains every fraction and code verifies identities without floating-point tolerance.

How far the distribution changes: a complete derivation

To quantify truncation, use total variation distance, TV: half the sum of absolute probability differences for discrete distributions. It is zero for identical distributions and at most one. This measures probabilistic difference, not semantic distance between texts. Within S, Q_i≥P_i; outside S, Q_i=0. The contributions can therefore be separated exactly.

Nucleus mass rises from q to one, an increase of 1−q; the excluded mass falls by the same amount. Division by two avoids double counting. At p=0.9 here, TV=0.05 rather than 0.10. This identity and bound concern one step and this specific Q transformation. They imply neither 90% answer correctness nor at most 10% word differences between long generations.

Generation is autoregressive: the chosen token enters context and changes the next distribution. After different choices, we may no longer compare the same P. Turning a local property into a global guarantee needs a separate analysis of sequence distributions and step dependencies, not an analogy with a tap removing a fixed percentage of text.

Boundaries and ties: details that change results

At p=0.70, the first two sum to exactly 0.70. Our ≥ rule retains two candidates; a strict > rule also retains the third, reaching 0.85. Both are understandable under declared conventions but yield different distributions. Fraction separates this logical distinction from binary-decimal rounding. Reproduction requires checking actual sampler code rather than only the top-p label.

With three equally likely candidates of probability 1/3 and p=1/2, two tokens retain mass 2/3. Choosing which two needs tie-breaking. Our experiment tests two identifier orders, obtaining the same k and q but different supports. This is not labelled a bug; it is configuration information that matters. At p=1 our convention retains all positive-mass candidates needed to reach one.

Temperature and top-k answer different questions

Top-k fixes candidate count; top-p sets a mass threshold and lets count vary. Temperature T>0 changes weights before any truncation: for positive probabilities, P_T(i) is proportional to P(i)^(1/T). Our code separates this floating-point demonstration from earlier exact calculations. At threshold 0.9, T=0.5 retains three tokens, T=1 four, and T=2 all five. Temperature can therefore change the support selected by subsequent top-p.

Positive temperature preserves ranking but not cumulative masses, so selecting the nucleus before or after temperature need not agree. Nor does “four candidates” measure diversity universally: probabilities may be uneven and tokens may express similar ideas. Increased support alone establishes no improvement in truth, safety, creativity, or accuracy.

From origins to recent research: what evidence exists

The foundational reference is The Curious Case of Neural Text Degeneration by Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi: arXiv v2 dated 14 February 2020, published at ICLR 2020. We read sections 3.1–3.3, experimental setup and analysis, including HUSE. Those GPT-2 continuation experiments are not a benchmark of every current model. HUSE combines statistical evaluation and the authors’ human judgements; its truncation protocol also uses interpolation. Those judgements are not our review.

For recent context we read Decoding as Optimisation on the Probability Simplex by Xiaotong Ji, Rasul Tutunov, Matthieu Zimmer, and Haitham Bou-Ammar, arXiv:2602.18292v2 dated 25 February 2026, treated as a preprint. Section 3.4 interprets top-p through constrained support and renormalisation. BoK tables 1–3 use two 7B Qwen models, MATH500, GPQA-diamond, and HumanEval, varied temperatures, maximum length 3072, and EOS stopping. Comparisons are Base, Top-K, and BoK, not experimental proof of top-p superiority. We reproduce no benchmark or universal recommendation.

Reproduce the mechanism without inventing a benchmark

The archive contains experiment.py, plot.py, complete results, and instructions. Main probabilities and thresholds use Fraction; no seed is needed because no draws occur. The temperature demonstration is separately labelled floating point. Assertions check total one, retained mass, TV=1−q, ≥ versus >, and tie rules. Full sorting of V candidates takes order V log V comparisons and cumulative scanning order V; optimised implementations may differ. We measured no LLM-engine latency or memory.

An enterprise application needs another experiment: fixed model and tokenizer, documented prompts and quality criteria, sampler versions, transform order, repeats, and generation budget. That would compare quality and variability rather than nucleus size alone. We attribute no proprietary decoder or undocumented results to EL-AI. This is a technical mechanism analysis, not original peer-reviewed research.

Answer: 0.9 is a threshold, not a quality guarantee

No: top-p=0.9 need not discard 10% of mass and does not choose the best 90% of answers. Our example retains four candidates, mass 0.95, and TV distance 0.05 from P. The rule includes the candidate needed to reach the threshold, then renormalises. Understanding this clarifies configuration and what remains to measure: sequence quality on the actual problem under an appropriate protocol.

References

Holtzman et al. — The Curious Case of Neural Text Degeneration, ICLR 2020, arXiv v2.

Ji et al. — Decoding as Optimisation on the Probability Simplex, 2026, preprint v2.

from fractions import Fraction as F
P=[F(52,100),F(18,100),F(15,100),F(10,100),F(5,100)]
p=F(9,10);mass=F(0);k=0
for value in P:
    mass+=value;k+=1
    if mass>=p:break
Q=[value/mass if j<k else F(0) for j,value in enumerate(P)]
tv=sum(abs(a-b) for a,b in zip(P,Q))/2
assert k==4 and mass==F(19,20) and tv==F(1,20)
print(k,mass,Q,tv)

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 21 October 2026.