A gradient check fails: is the model really wrong?

When the checker also needs checking

When developing an AI model, we often compare a computed derivative with an estimate obtained by slightly changing a parameter. A mismatch can tempt us to blame backpropagation immediately. Yet numerical checks have assumptions and limits too. A very small step can worsen the estimate; a corner may have no derivative to compare. How can we distinguish a useful warning from a false alarm?

We use two scalar functions: sine, smooth with a known derivative, and ReLU, a piecewise function common in neural networks. We train no network and execute no automatic-differentiation library. The experiment exposes the assumptions of a gradient check. Quantities are dimensionless; sine’s argument 0.7 is in radians. All reported numerical results come from the archived code.

A local slope, not merely a difference

A derivative measures output change per small input change in the limit as the change approaches zero. Graphically it is tangent slope. For scalar f at x, a forward ratio uses f(x+h) and f(x), while a central ratio uses points on both sides. The step h must be nonzero in computation. A central ratio is not the definition of a derivative at an irregular point; it is an estimator that agrees under appropriate conditions.

Expand f around x to understand accuracy independently of a chart. With sufficiently smooth derivatives, f(x+h)=f(x)+hf′(x)+h²f″(x)/2+h³f‴(x)/6+… . The forward ratio retains leading error proportional to h. Subtracting the expansion at −h cancels even terms, leaving central leading error h²f‴(x)/6. This explains the central method’s better theoretical order, not guaranteed accuracy for every program.

Why reducing h is not enough

The derivation assumes exact arithmetic. A computer evaluates f with rounding and subtracts nearly equal values. Absolute uncertainty near machine precision in the difference is divided by h. A simplified local error model therefore combines C₁h² with C₂u/h for central differences, where u describes relative precision and constants depend on function and scale. Reducing h shrinks the first term but can enlarge the second.

Balancing orders suggests h proportional to u^(1/3) in this central model after accounting for problem scales. This is not a universal value for every network: noise, rounding, parameter magnitudes, and near-zero terms change the picture. The practical point is to inspect a sequence of steps for a region of agreement rather than equate “smaller” with “more reliable”.

Sine: an executed smooth check

We evaluate f(x)=sin(x) at x=0.7 with steps from 10⁻¹ to 10⁻¹⁶. The reference is math.cos(0.7), approximately 0.7648421872844885: a double-precision evaluation of the analytical derivative, not an arbitrary-precision oracle. The best central error in this sweep occurs at h=10⁻⁵, about 1.70×10⁻¹¹. At h=10⁻¹⁶ the estimate is approximately 1.1102230246251565, differing by about 0.34538. Here x+h and x−h still differ from x: coincident input points are not necessary for a poor check.

Complex step: powerful, with conditions

For a function with an appropriate analytic continuation to complex numbers, use an imaginary step ih. Expansion gives f(x)+ihf′(x)−h²f″(x)/2−ih³f‴(x)/6+… . Taking the imaginary part and dividing by h yields f′(x) with theoretical order-h² error without subtracting nearby real values. Here i²=−1 and Im denotes the imaginary component. Accepting a complex input type alone does not make the technique valid.

For our sine evaluated with cmath.sin at h=10⁻¹⁶, the difference from the reference is about 1.11×10⁻¹⁶. This is a good result for this function, not proof of universally absent rounding. abs supplies a counterexample: at x=0.7 the real derivative of |x| is 1, but abs(x+ih) returns a real magnitude and loses imaginary information. Applying the same recipe returns zero, as the code checks. Branches, comparisons, real casts, and nonanalytic operations therefore need examination first.

ReLU: when 0.5 is not a derivative

ReLU means Rectified Linear Unit: r(x)=max(0, x). For negative x the output and slope are zero; for positive x output equals x and slope is one. At zero the one-sided slopes differ, so the ordinary derivative does not exist. Yet the central ratio is (h−0)/(2h)=1/2 for every positive h. Shrinking the step cannot make it converge to zero or one: it averages two different branches.

Nonsmooth optimisation can use subgradient conventions. For this convex function, the subgradient set at zero is [0,1]. Selecting zero is a possible algorithmic convention, not a statement that an ordinary derivative exists and equals zero. Comparing that convention with central differences gives 0 versus 0.5 without by itself proving a bug. We execute no autograd and attribute no such result to any framework version.

From a scalar check to many parameters

For scalar loss L(theta) with d parameters, coordinate central differences need two evaluations per parameter, about 2d loss evaluations plus the gradient being checked. A directional check compares [L(theta+hv)−L(theta−hv)]/(2h) with g·v, where g is the proposed gradient and v a chosen direction. It is cheaper per direction, but an error orthogonal to v can escape. We do not execute this multidimensional check here; it is a mathematical extension without fabricated network results.

The evaluated function must remain the same: randomness, state updates between calls, or different data can turn a function mismatch into an apparent derivative error. Numerical agreement at a few points is local evidence, not global model correctness. Near a branch change, inspect whether the perturbation crosses it; elsewhere a stable discrepancy across steps remains a serious clue. The conclusion is to interpret checks rather than ignore them.

An answer useful for AI development

A failed gradient check does not identify the culprit by itself. For sine, error grew when the step became too small; for ReLU, we obtained 0.5 at a point without a derivative. These are different reasons not to treat an estimator as absolute truth. Before changing a model, specify function, smoothness, step, precision, and comparison convention. The check becomes a scientific tool with a validity domain rather than an opaque verdict.

Sources and reproducibility

We read SciPy 1.18.0 least_squares documentation, jac parameter, for complex-step conditions; we executed neither SciPy nor its optimisation examples. Our code uses Python 3.14.0 math and cmath without randomness and saves all sixteen steps, one-sided differences, and the abs counterexample. Equations are derived here, not experimental findings from a paper or EL-AI product. Matplotlib generates figures; no hardware performance or human review is claimed.

SciPy 1.18.0 — least_squares, jac.

Python 3.14 — math.

Python 3.14 — cmath.

import math
x=0.7
reference=math.cos(x)
for h in [1e-3,1e-5,1e-16]:
    estimate=(math.sin(x+h)-math.sin(x-h))/(2*h)
    print(h,estimate,abs(estimate-reference))
relu=lambda z:max(0.0,z)
h=1e-6
assert (relu(h)-relu(-h))/(2*h)==0.5

Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 21 October 2026.