Speculative decoding: why correcting a rejection is not resampling from the target

Technical analysis · AI inference · 24 September 2026

Abstract: two guarantees to separate

Speculative decoding proposes tokens with an inexpensive model and verifies a block with the target model. The decisive question has two parts: which correction preserves the target distribution, and when does advance work actually reduce time? We develop a complete categorical example, a counterexample to naive correction, and a cost model. The result is clear: sampling exactness and acceleration are different properties with different assumptions.

This article makes an educational and analytical contribution: exact calculations on three symbols, 200,000 Monte Carlo trials, and an executed Python parameter sweep. We did not measure an LLM, GPU, or EL-AI service. Prerequisites are elementary probability, finite sums, and familiarity with autoregressive generation; knowledge of Transformer training is unnecessary.

1. The probabilistic contract

Fix a confirmed prefix and a finite vocabulary V. p(x) is the target probability of the next token; q(x) is the draft probability. Both are nonnegative and sum to one. They are the distributions actually used for sampling, after temperature, masks, and truncation: using unnormalized logits in the acceptance ratio changes the problem. A token must represent the same event in both vocabularies.

Preserving p means that, under ideal repetitions, every event retains its probability. It does not mean obtaining the same string with the same seed: implementations can consume random numbers in different orders. Nor does it improve factual truth: if the target assigns probability to a false statement, an exact sampler preserves that possibility. The guarantee concerns the statistical law, not semantic correctness.

2. Account for probability mass before coding

The fundamental rule is proposed and proved by Leviathan, Kalman, and Matias (ICML 2023, Sections 2–3 and Appendix A.1): draw x from q, accept with probability min(1,p(x)/q(x)), and on rejection sample from the normalized positive residual of p−q. The following example reconstructs its accounting with numbers chosen for this article.

Call the symbols A, B, and C. A is proposed in 20% of trials and always accepted, contributing 0.20. B is proposed with probability 0.50 but accepted in only 60% of those proposals, contributing 0.30. For C the contribution is 0.30×2/3=0.20. Total accepted mass is 0.70. We need another 0.30 to obtain a complete distribution, and the entire shortfall relative to the target concerns A.

In the formula A denotes the total acceptance rate, while A as a label denotes the first symbol; these roles are distinct. Z is the rejection probability and r is a distribution conditional on rejection. Do not add m+r directly: m is unnormalized mass, while r already sums to one. Weight r by Z. This simple check prevents a common error in illustrative implementations.

Two edge cases clarify the construction. If p=q, Z=0: every proposal is accepted and the residual branch must not be evaluated, avoiding 0/0. If q gives zero probability to an event allowed by p, that event is never proposed but can appear in the residual. No token actually drawn from q requires division by q=0. An incompatible mask applied only after correction can instead remove mass and break the contract.

3. Counterexample: “on rejection, restart from p”

Using the target as a fallback sounds reasonable. But with this particular acceptance rule it is wrong: the accepted branch has already supplied all the mass required for B and C. Drawing from p on rejection adds more probability to them and leaves A underrepresented. The final distribution becomes (0.35, 0.39, 0.26). A plausible sentence cannot reveal this distortion; accounting over three events can.

Total variation distance TV measures the largest probability disagreement over an event set. Here {A} suffices: 0.50 versus 0.35, a difference of 0.15. This is not rounding error. Even with indefinitely many samples, the incorrect version converges to the incorrect distribution. Enlarging the test without checking the limiting law merely makes a biased answer more precise.

4. Reproducible experiment and interpreting error

We ran 200,000 trials with Python 3.14.0, random.Random, and seed 20260923. Each trial shares the proposal and acceptance decision between the variants; on rejection each uses its own recovery distribution. This controls the comparison, but counts across variants are not independent. The table reports frequencies, not language-quality estimates. No corpus is involved.

Correct frequencies differ from the target by less than 0.0016 in this run. For A, the marginal standard error of frequency is approximately √(0.5×0.5/200000)=0.00112; this scale makes fluctuations interpretable without turning one run into a mathematical proof. The script also checks exact mass accounting within numerical tolerance. The proof justifies the method; Monte Carlo detects coding errors.

This snippet computes both laws without simulation. The attached archive contains the full sampler, checks, and figure generation. All experiments are educational and were prepared with AI assistance; we do not claim human scientific review or business validation.

p = [.5, .3, .2]
q = [.2, .5, .3]
m = [min(x,y) for x,y in zip(p,q)]
z = 1-sum(m)
r = [max(x-y,0)/z for x,y in zip(p,q)]
print([a+z*b for a,b in zip(m,r)])
print([a+z*b for a,b in zip(m,p)])

5. From token to block: the prefix is part of the state

A draft of length γ contains autoregressive proposals: each probability depends on preceding tokens. The target evaluates draft positions causally, and only the accepted prefix is retained. After the first rejection, later proposals depend on a token absent from the final text: reusing them without recomputation would query a distribution conditioned on the wrong prefix. The cache must also discard the unconfirmed tail.

Parallelism concerns probability evaluation on an already proposed sequence; autoregressive dependencies do not disappear. If every proposal is accepted, an extra target token is added; otherwise the residual supplies the corrected token. A cycle thus emits one to γ+1 tokens, apart from early stops such as end-of-sequence. Counting the entire draft as useful output artificially inflates throughput.

6. When extending the draft pays off

Now consider a separate cost model. Assume independent acceptances with constant probability α, no early stopping, and a block verification cost equal to one ordinary target step, T. A draft token costs cT. These assumptions isolate the trade-off; they are not measurements of our computer. The probability of retaining at least i consecutive proposals is α to the power i. Summing tail probabilities gives the expected number E of emitted tokens.

The last inequality follows by cross-multiplying the positive denominators of the two speedups. The left side represents the marginal contribution of the new attempt; the right side is its price. With α<1, marginal benefit falls geometrically while the cost of a draft step remains positive. There is therefore no general reason to maximize γ. At α=0 the draft produces no useful tokens, yet its cost remains.

The executed sweep uses γ from 1 to 16, α in {0.3, 0.7, 0.9}, and c in {0.05, 0.20}. At α=0.7 the grid maximum is γ=6 with c=0.05: E=3.058819 and S≈2.353. If the draft costs four times as much, the maximum moves to γ=3 and S≈1.583. The same probabilistic agreement does not imply the same time saving. These are grid and model maxima, not certified optimal server configurations.

7. Where the simplified model stops being enough

If verifying γ tokens costs g(γ)T, replace the denominator with g(γ)+cγ plus normalized overheads. For example, with α=0.7, γ=6, and c=0.05, the numerator remains 3.058819; if g=3, the ratio falls to about 0.927. The method is then slower despite exact sampling. Emitting at least one token per cycle limits the number of serial calls, not the duration of those calls.

Constant α is also a simplification. A proper name, formula, or language switch can change agreement between models. In general E is one plus the sum of joint probabilities of accepting the first i tokens; replacing those probabilities with powers of the average ignores dependencies and prefix selection. Two workloads with the same average acceptance can have different accepted-length distributions. Recording only α hides that distinction.

8. From research to the measurements needed

The 2023 ICML work measured, among other experiments, T5-XXL with smaller drafts on translation and summarization, batch one and TPU-v4. It is a historical reference, not a prediction for current hardware. For a recent direction we consulted Learning to Draft, arXiv v1 dated 2 March 2026, including methods and results: it proposes draft-depth and verification-size policies optimized for time rather than accepted length alone.

That study uses two PPO-trained policies and evaluates multiple models and tasks; comparison against grid search is particularly relevant to separating adaptation from simply choosing better parameters. We do not reproduce its benchmark or extend its conclusions to every system. Its results table also includes cases with greater speed but shorter accepted length: a reason to measure the denominator as well as the numerator.

For a multilingual service, separate Italian, English, Chinese, and Spanish tests while keeping requests and lengths comparable. Measure time to first token, inter-token latency, total time, aggregate throughput, memory, and loaded percentiles. A batch-one improvement can disappear with concurrent requests: the draft occupies resources other users could have used. Reporting model, version, precision, sampling, hardware, and prompt distribution makes the comparison interpretable.

For EL-AI this is a possible inference-evaluation criterion, not an announcement of an implemented feature. The technical conclusion is more useful than a speed promise: first verify that correction preserves the intended law, then find the operating point that reduces cost on the actual workload. The two checks require different tools, and neither substitutes for the other.

Sources and reproducible materials

Leviathan, Kalman, Matias (2023), Fast Inference from Transformers via Speculative Decoding, ICML / PMLR 202, 19274–19286. Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning, arXiv:2603.01639v1 (2026).

Sources accessed 24 September 2026; the second citation specifically identifies the preprint version read. Download code, results, and instructions. JSON results. Original cover generated with ImageGen, illustrative: it does not depict an EL-AI installation.