Vertical models: retaining capabilities takes more than a small update

Abstract: weight distance does not measure forgetting

Domain adaptation can improve a new task while degrading earlier capabilities. A small Euclidean update does not guarantee a small regression: direction and old-loss curvature matter. We construct two exactly solvable quadratic tasks. At matched old loss 0.5, a curvature-informed penalty reaches new loss 0.459957 versus 0.810894 for an isotropic penalty, despite moving much farther in parameter space. We then rotate the problem to show how keeping only the diagonal can discard the useful information.

This is an educational optimization analysis, not an experiment on language models, materials, or EL-AI products. Parameters and losses are dimensionless. Prerequisites are linear algebra, derivatives, and quadratic forms. The example isolates one specialization mechanism; it does not prove any continual-learning method preserves every real-model capability. The earlier LoRA study concerns update representation and rank; here we study the function determining its damage to an old task.

1. Two tasks, an explicit geometry

The model has two parameters θ=(θ₁,θ₂). Task A is minimized at the origin and penalizes the first direction one hundred times more than the second. New task B wants both parameters equal to one. Unconstrained adaptation reaches (1,1): B loss is zero, A rises to 50.5. Both functions are known everywhere, so no estimation or optimization error is introduced. This separates objective conflict from other training problems.

At the origin A’s gradient is zero. A gradient-dot-product check therefore detects no first-order conflict. Yet a gradient step on B with rate η gives θ=(η,η), so L_A=50.5η² and L_B=(1−η)². At η=0.1 old loss is already 0.505. Damage is second-order here. Taylor’s ΔL_A≈g_AᵀΔθ+½ΔθᵀH_AΔθ is exact for this quadratic; for a network it is local with a remainder to assess.

2. Uniform penalties versus curvature

An isotropic penalty minimizes L_B+λ||θ−θ_A||²/2. Differentiation gives θ₁=θ₂=1/(1+λ), restraining sensitive and flexible directions equally. An anisotropic penalty uses λθᵀH_Aθ/2, giving θ₁=1/(1+100λ), θ₂=1/(1+λ). Here H_A is exact curvature and the penalty equals λL_A, not an estimated approximation of old capability.

Comparing equal λ would mislead because it multiplies different geometries. Instead impose a common budget L_A≤0.5 and compare B loss. Isotropic λ=√101−1≈9.049876. For the anisotropic case, solve monotone 50/(1+100λ)²+0.5/(1+λ)²=0.5 by bisection, yielding λ≈0.178844891. Code saves the solution and checks the constraint within 10^−12. Losses are comparable because they use the same functions.

The anisotropic point is about (0.052954,0.848288), barely moving the sensitive direction while using the second. Distance from the origin is 0.849939 versus only 0.140720 isotropically, yet both have L_A=0.5. Freezing θ₁ and taking θ₂ to one gives L_B=0.5: simple, but slightly worse than anisotropic optimum 0.459957. Coordinated movement of both parameters can use the budget better than absolute freezing.

3. Why the constructed solution is optimal

The problem min L_B subject to L_A≤ε is convex, and at ε=0.5 the origin is strictly feasible. B’s unconstrained minimum violates the constraint, so the optimum is on its boundary. The Lagrangian L_B+λ(L_A−ε), λ≥0, gives the anisotropic equations. Convexity and optimality conditions establish global optimality for these quadratics. We are not inferring universal superiority from a curve; we know and solve the problem. Nonconvex networks do not inherit that justification automatically.

4. Rotating the problem exposes the diagonal limitation

Represent the same anisotropy in a rotated basis: H=[[50.5,49.5],[49.5,50.5]]. Unit directions u_+=(1,1)/√2 and u_−=(1,−1)/√2 have curvature 100 and 1. A unit displacement causes loss 50 and 0.5 respectively. Dropping off-diagonal entries predicts 25.25 in both directions. The diagonal overestimates one direction’s damage and underestimates the other; it can lose the ordering, not just numerical precision.

For p parameters a full matrix costs O(p²) memory, a diagonal O(p), giving the simplification a practical rationale. Protection quality nevertheless depends on parameterization and ignored correlations. Our script saves these values; it estimates no Fisher from data. Applications must distinguish loss Hessian, expected Fisher, and empirical Fisher: they are not interchangeable without assumptions. An estimated diagonal is not the exact geometry of capabilities.

5. From geometry to evaluating a vertical model

EWC, introduced by Kirkpatrick and colleagues, penalizes changes according to parameter importance; the consulted version uses Fisher-based diagonal precision. We read the method, permuted-MNIST experiments, and sequential Atari results. These are specific protocols, not universal LLM retention proofs. Our two-variable example uses known curvature to illustrate selective regularization, without reproducing network EWC or attributing our numbers to it.

Real specialization needs separate definitions of the new domain and retained capabilities, tests independent of adaptation data, and comparison with the base checkpoint. An average score can hide a rare capability’s regression: report per-task results and acceptance criteria set before selection. Replay, output distillation, freezing, and penalties have different costs and assumptions. A good new-domain training curve is insufficient retention evidence.

“Changing little” must be measured in performance space, not only by parameter counts or norms. Here a displacement about six times larger preserves the same old-loss budget and learns B better. That is a verified property of these quadratics. A real vertical model still requires measuring the tradeoff on relevant tasks, rather than inferring it from the adaptation technique’s name.

Kirkpatrick J. et al. (25 January 2017), Overcoming catastrophic forgetting in neural networks, arXiv:1612.00796v2, method §2 and experiments §§2.1–2.2.

from math import sqrt
def losses(x,y):
    return .5*(100*x*x+y*y), .5*((x-1)**2+(y-1)**2)
lo,hi=0.,100.
for _ in range(100):
    lam=(lo+hi)/2
    if losses(1/(1+100*lam),1/(1+lam))[0]>.5: lo=lam
    else: hi=lam
lam=(lo+hi)/2
print(lam, losses(1/(1+100*lam),1/(1+lam)))
print(losses(1/sqrt(101),1/sqrt(101)))

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 24 September 2026.