Imitating an AI model: cover every possibility or focus on one?

The same probabilities, two ways to be wrong

A small model must approximate a richer distribution. Penalising what it misses differs from penalising unsupported proposals when its possibilities are restricted. Kullback–Leibler divergence, KL, is directional. Reversing its arguments can change the chosen approximation. We use three categories and three explicit candidates, not a large-model training experiment.

Who weights each error?

Let p be the reference and q the candidate, nonnegative probabilities summing to one on identical categories. Natural logarithms give nats. D_KL(p||q) weights log(p_i/q_i) by p_i; reversing it weights log(q_i/p_i) by q_i. KL is not a metric: it is asymmetric and need not satisfy the triangle inequality. “The KL loss” omits an essential part of the objective.

With fixed p, the first criterion minimises cross-entropy −Σ p_i log q_i because Σ p_i log p_i is constant across candidates. Categories important to p keep their weight even if q ignores them. In the reverse criterion q determines how often regions contribute. This intuition does not prove that every reverse KL chooses one mode; the candidate-family constraints matter.

A comparison we can fully reproduce

Reference p=(0.495,0.01,0.495) has two frequent outer categories and a rare centre. Wide q_W=(0.34,0.32,0.34) covers both ends but overweights the centre. q_L=(0.98,0.01,0.01) favours the first; q_R reflects it. These three vectors are the entire allowed family, not claimed optima of a Gaussian network or architecture. We compare finite alternatives.

For p||q_L the third term 0.495·log(0.495/0.01) is large: the candidate nearly removes a frequent reference event. For q_W||p the centre contributes 0.32·log(0.32/0.01), penalising overproduction of a rare category. Individual terms can be negative while total KL is nonnegative. Executed sums select W for p||q and tie L, R for q||p.

Zeros change the domain, not just the value

A zero-weight term contributes zero by convention. But p_i>0 with q_i=0 makes p||q infinite. Hard candidate (0.99,0.01,0) therefore has infinite forward KL and finite reverse KL against our strictly positive p. Adding epsilon removes infinity but changes the distribution; renormalise and state the approximated problem.

Without constraints both choose the reference

Remove the three-candidate restriction. If q can equal p, both divergences reach zero and cannot improve. For positive distributions log z≤z−1 gives Σ p_i log(q_i/p_i)≤Σ(q_i−p_i)=0; negate it for KL≥0. Equality requires q=p, with support boundaries handled separately. Our disagreement comes from representation constraints, not incompatible probability truths.

Coordinate derivatives are −p_i/q_i forward and log(q_i/p_i)+1 reverse. A normalised parametrisation must apply its chain rule and sum constraint; these are not ready-made update rules. Actual training requires sampling, gradient estimation and representation analysis, none executed here. We measure no generated-text quality or preference.

Choosing direction means choosing the comparison

Reversing KL is not cosmetic: our constrained example changes the preferred candidate because neglected and overrepresented regions receive different weights. For distillation and generative modelling, specify reference, candidate, support and constraints before choosing loss. Coverage and concentration are problem-specific intuitions, not substitutes for experiments. Sources establish the definition; the finite numerical analysis is our reproducible educational contribution.

SciPy 1.18.0 — Relative entropy.

import math,json,sys
from pathlib import Path
p=[.495,.01,.495];candidates={'wide':[.34,.32,.34],'left':[.98,.01,.01],'right':[.01,.01,.98]}
def kl(a,b):
 assert abs(sum(a)-1)<1e-12 and abs(sum(b)-1)<1e-12
 if any(x>0 and y==0 for x,y in zip(a,b)):return math.inf
 return math.fsum(x*math.log(x/y) for x,y in zip(a,b) if x>0)
rows=[dict(name=k,q=q,forward=kl(p,q),reverse=kl(q,p)) for k,q in candidates.items()]
assert min(rows,key=lambda x:x['forward'])['name']=='wide'
assert rows[1]['reverse']<rows[0]['reverse'] and abs(rows[1]['reverse']-rows[2]['reverse'])<1e-12
assert kl(p,p)==0 and math.isinf(kl(p,[.99,.01,0])) and math.isfinite(kl([.99,.01,0],p))
r=dict(python=sys.version.split()[0],seed=None,scope='Exact finite candidate comparison, not training or inference benchmark',p=p,rows=rows,hardReverse=kl([.99,.01,0],p),hardForward='infinite',units='nats, natural logarithms')
Path(__file__).with_name('results.json').write_text(json.dumps(r,indent=2),encoding='utf8')

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 25 October 2026.