Abstract. Calling a network “INT8” does not describe how its numbers are represented. The choice of scale can erase an entire small-weight channel, even while many integer values remain unused for that channel. We compare symmetric per-tensor and per-channel quantization using a controlled linear layer, a derivation of output error and a reproducible experiment. We also construct a counterexample: lower weight error does not guarantee lower error for every input. Finally, we distinguish compression, working memory and performance actually measured on embedded hardware.
Basic linear algebra and probability are assumed. Every experimental number below comes from a NumPy simulation: we did not run inference on a board, measure energy consumption or train a network. Weights use conventional units; error metrics refer to the variables of our educational model.
1. Eight bits do not define a resolution
An integer code takes discrete values; interpreting it as a real number requires a mapping. In the affine representation r̂ = s(q−z), s is a positive scale and z an integer zero-point. Here we choose symmetric weights with z = 0 and codes from −127 to 127. The value −128 is unused. This convention agrees with INT8 weights in the LiteRT specification, which also distinguishes per-tensor from per-axis granularity. Activations in that specification can instead have a nonzero zero-point; these cases should not be confused.
Q = 2^(b−1) − 1 s = a / Q q(w) = clip(round(w/s), −Q, Q) w_hat = s q(w)
a is the largest representable absolute value and b the bit width. We round to the nearest integer, using NumPy's ties-to-even convention. When |w| ≤ a, absolute error cannot exceed s/2. When |w| > a, saturation occurs: the error can be much larger and that bound no longer holds. Choosing a therefore trades resolution in the central region against coverage of the tails.
With b = 8 and a = 10, s is about 0.07874. A weight of 0.01 becomes zero, not because eight bits are inherently insufficient, but because this scale devotes all levels to a much wider interval. With a = 0.01, the same weight is represented exactly by code 127. Scale effects should be examined before generically blaming “low precision” for lost quality.
2. A linear layer with four natural scales
Define a layer y = Wx with eight inputs, four outputs and no bias. Each row is an output channel. Construct a common base vector and multiply it by four amplitudes:
u = [−1, −0.51, −0.11, 0.07, 0.23, 0.49, 0.81, 1] α = [0.01, 0.1, 1, 10] W[c, j] = α[c] u[j] W ∈ R^(4×8), x ∈ R^8, y ∈ R^4
The structure is deliberately controlled: channels share a shape and differ only in amplitude. This is not a model extracted from a product, nor a claim about the distribution of all real networks. It isolates one precise source of error. In a trained network, channel shapes, outliers and sensitivities can differ substantially.
Per-tensor quantization uses one scale s = max|W|/127 = 10/127. Per-channel quantization uses sc = maxj|W[c,j]|/127 = αc/127. We quantize and reconstruct weights while leaving inputs in floating point. This isolates weight effects from activation effects: our experiment is not yet an entirely integer inference pipeline.
3. Deriving output error
Let E = Ŵ−W be the error matrix. Output error for input x is Ex. If x has zero mean and covariance Σ, its mean squared norm can be calculated without simulation:
δy = Ex E[||δy||²] = E[xᵀEᵀEx] = tr(EΣEᵀ) Σ = I ⇒ E[||δy||²] = ||E||F² E[(δy_c)²] = ||E[c,:]||² if Σ = I
The identity uses trace linearity and E[xxᵀ] = Σ for zero-mean inputs. With nonzero mean μ, one must add ||Eμ||². This matters for rectified activations or offset variables: automatically applying the isotropic formula can produce an incorrect prediction.
We generate 100,000 independent standard Gaussian vectors with seed 20260922 and compare empirical mean squared errors with the exact formula. Sampling is a numerical check; the table uses analytical values so Monte Carlo noise is not mistaken for a method difference. The formula predicts neither classification accuracy nor the quality of a nonlinear network: it describes only the declared layer.
4. The result: a channel can disappear
Define relative channel error as ||E[c,:]||₂/||W[c,:]||₂. This is not the percentage of incorrect predictions. A value of one for the first per-tensor channel means, in this case, that all its weights were rounded to zero.
| Amplitude α | INT8 per-tensor relative error | INT8 per-channel relative error | Per-tensor output MSE | Per-channel output MSE |
|---|---|---|---|---|
| 0.01 | 100% | 0.1859% | 3.2262×10⁻⁴ | 1.1148×10⁻⁹ |
| 0.1 | 31.7980% | 0.1859% | 3.2621×10⁻³ | 1.1148×10⁻⁷ |
| 1 | 3.7089% | 0.1859% | 4.4380×10⁻³ | 1.1148×10⁻⁵ |
| 10 | 0.1859% | 0.1859% | 1.1148×10⁻³ | 1.1148×10⁻³ |
The largest channel uses the same scale in both methods and produces the same result. In the others, per-channel scaling preserves relative resolution. Equal relative errors are not a universal law: they follow from every row being a scaled copy of the same vector. This constructed control reveals the mechanism without attributing unjustified generality to it.
We repeated the calculation with b = 4, Q = 7 and symmetric range −7…7. Per-channel relative error rises to 7.6444%; per-tensor quantization erases the two smallest channels. This simulates four-bit codes; it does not verify INT4 runtime support or memory packing. Choosing four versus eight bits requires task evaluation beyond these local errors.
5. A counterexample: lower weight error is not always enough
The isotropic average assigns equal importance to all input directions. An application may mostly visit only some of them. Take the first error rows, et for per-tensor and ec for per-channel, and construct an input orthogonal to et but not ec:
z = ec − et (ecᵀet)/(etᵀet) x = z / ||z|| etᵀx = 0, while ecᵀx can differ from 0
The script constructs this input. First-channel per-tensor error is about −3.25×10⁻¹⁹, numerically zero; per-channel error is about 3.32×10⁻⁵. The method with much lower weight error loses this pointwise comparison. This is not advice to choose the poorer scale: it demonstrates that ranking by a norm does not imply the same ranking for every possible input.
For a known distribution, tr(EΣEᵀ) weights errors according to directions actually visited. A real task also requires measuring the final metric: a small change near a decision threshold may matter, whereas a larger error far from it may leave the decision unchanged. Calibration and testing must therefore reflect operational inputs, without repeatedly using the test set to select scales.
6. Clipping: a finer grid can make results worse
Now use four educational activation values: −0.2, 0.2, 0.8 and 1. Keep a symmetric eight-bit grid and compare three thresholds a. At a = 0.25, the step is about 0.00197, but two values saturate: MSE is 0.21625031. At a = 0.5, the step is about 0.00394 and MSE falls to 0.08500031. At a = 1, the step is coarser, about 0.00787, but no value saturates and MSE is about 0.00000744.
Finer resolution inside the interval does not guarantee lower total error. Conversely, one enormous outlier might impose an excessively coarse grid on the remaining data. The best threshold depends on the distribution and chosen loss: min-max, percentiles and error minimization are not equivalent criteria. Our tiny set illustrates the phenomenon, rather than selecting a universal calibration procedure.
7. From simulated calculation to an integer kernel
The foundational work by Jacob and colleagues, arXiv:1712.05877v1 from 2017, develops affine representation, zero-point handling and integer-arithmetic inference; sections 2–4 cover methods and experiments on particular networks and CPUs. We do not transfer those performance findings to our example or to a contemporary board without measurement. The historical paper also uses conventions that should not automatically be equated with the current INT8 specification.
In the simplified symmetric-weight case, an output accumulates products between weight codes and inputs centred on their zero-point. The real-valued factor is sx·sw,c; a quantized output then requires rescaling toward sy. Bias, rounding, saturation and operation fusion are implementation details that matter. Reconstructing float weights and multiplying them, as we do to isolate error, does not exercise those kernel details.
The accumulator must also be sized appropriately: operand width alone does not determine the width required to sum many products. A bitwise comparison must fix intermediate rounding, saturation and operation order. Before claiming acceleration, verify that the intended backend actually executes quantized operators, without conversions or fallback paths changing execution.
8. Memory: weights are not the whole platform
Our layer has 32 weights. In float32 they would occupy 128 bytes; in INT8, 32 bytes. Float32 scales add 4 bytes per tensor or 16 for four channels. Weights plus scales therefore total 36 or 48 bytes: reductions of about 3.56× and 2.67× from the original 128 bytes, not exactly 4×. These are declared theoretical counts, not LiteRT file sizes; they exclude bias, metadata, alignment and code.
On a microcontroller, weights may reside in flash while activations, temporary buffers and runtime state require RAM. The TFLite Micro memory documentation distinguishes nonpersistent, temporary and persistent arena sections and describes buffer reuse. A small model file does not prove that peak RAM fits the board. Analysis must consider which tensors are alive simultaneously.
Likewise, fewer bytes do not automatically mean lower latency or energy. Sensor acquisition, preprocessing, transfers, available instructions and clock frequency can dominate total time. Power in watts and energy per inference in joules answer different questions; without an on-device protocol we report no performance numbers.
9. Reproduction and a proposed on-device protocol
import numpy as np
u = np.array([-1., -.51, -.11, .07, .23, .49, .81, 1.])
W = np.array([.01, .1, 1., 10.])[:, None] * u
for per_channel in [False, True]:
peak = np.max(np.abs(W), axis=1, keepdims=True) if per_channel else np.max(np.abs(W))
scale = peak / 127
Wq = np.clip(np.rint(W / scale), -127, 127) * scale
E = Wq - W
print(np.linalg.norm(E, axis=1) / np.linalg.norm(W, axis=1))
print(np.sum(E**2, axis=1))Download experiment, figures and instructions; the JSON results file preserves scales, codes, analytical errors, the Monte Carlo comparison and counterexample. The script checks the s/2 bound without clipping and agreement between analytical and empirical MSE within 3%. That tolerance checks the sampled calculation; it is not a general statistical guarantee.
A next step, proposed here but not executed, is to export a real model and compare float, per-tensor INT8 and per-channel INT8 on identical examples separate from calibration. Record board and revision, runtime, compiler, operators, clock, batch, input shape, threads and thermal conditions. Measure task quality, peak RAM, median and tail latency, and energy per inference. Also retain examples whose decisions change: they explain compression costs better than a single average.
10. The design choice and EL-AI's direction
In a hypothetical predictive-maintenance application, a numerically small channel might help distinguish a rare fault. Erasing it because another channel determines the global scale would be a risk to investigate with relevant data, not damage already demonstrated by this experiment. Connecting local calculation to industrial decisions requires evaluation of the complete system.
EL-AI has extended its technical editorial programme to embedded platforms and robotics. This article develops understanding relevant to that direction; it does not describe a proprietary board, an available TinyML product or measurements performed by the company. All four languages preserve the same experiment and limitations.
Conclusion. Scale granularity determines which differences survive integer representation. In this constructed case, per-channel scaling preserves signals erased by a global scale, but guarantees neither superiority for every input nor better performance on arbitrary hardware. Sound evaluation connects representation error, input distribution, task metrics and on-device measurements.
Sources and transparency
Primary sources linked in the text: LiteRT INT8 specification, updated 28 May 2026; Jacob and colleagues, arXiv:1712.05877v1 preprint, 15 December 2017, sections 2–4; TFLite Micro memory-management documentation. Consulted on 22 September 2026. The example, counterexample and figures are reproducible educational analyses, not results from a trained network or peer-reviewed research.
Text and translations prepared with AI assistance; no human review is claimed. The AI-generated illustrative cover is not a photograph of an EL-AI board or installation.