LoRA beyond memory savings: rank, adaptation geometry, and measurable limits

Technical analysis · 21 September 2026 · Linear algebra, model adaptation, reproducible experiments

Abstract

When is a low-rank update sufficient to specialize a model? The answer depends on which directions must change, which inputs make them important, and whether the optimizer can reach them. This article derives LoRA’s parameter count and gradients, examines its representational limit through singular value decomposition, and measures that limit on two controlled problems. For 64 × 64 matrices, the same rank of 8 leaves either 13.53% or 87.50% of the update’s energy unrepresented. A second example demonstrates that minimizing weight reconstruction error can select the wrong direction for the data. This yields a precise principle: rank is a hypothesis about task geometry that must be evaluated alongside the input distribution, optimization, and generalization.

Nature of this contribution. This is a pedagogical analysis with original, reproducible calculations, not new peer-reviewed research. The experiment uses synthetic matrices and an analytical solution: it neither trains an LLM nor measures its accuracy. Familiarity with matrices, derivatives, and supervised learning is assumed.

1. The question that parameter count cannot answer

Specializing a model on technical documents appears to pose an economic question: how much memory does updating it require? Yet another question comes first: which transformations need to change? A matrix containing millions of coefficients might require a correction concentrated in a few directions. Another matrix of the same size might require independent corrections in many directions. The initial model’s coefficient count does not distinguish these cases.

LoRA, introduced by Hu and colleagues in 2021, freezes a pretrained matrix and learns a factorized correction. Its scientific interest lies precisely in the constraint imposed on that correction. The original work documents results on particular models and tasks; it does not establish that every specialization admits a small rank. Here we separate three frequently conflated questions: can the required update be represented? Can training find it? Does the solution retain its quality on new data? [1, Sections 4 and 7]

This distinction prevents a common misreading of experiments: if increasing rank does not improve a score, we have not yet shown that the task is intrinsically simple. Data may be insufficient, the wrong modules may be adapted, or update scaling may be inappropriate. Conversely, a higher rank that reduces training loss does not establish an out-of-sample improvement.

2. From a full matrix to a factorized correction

Consider a linear layer without a bias, to keep the mechanism visible. Input x has k components, output h has d, and the initial matrix W₀ has dimensions d × k. Full fine-tuning can change all dk coefficients. With LoRA we write:

The number r is the update’s maximum rank; s is a nonzero scalar controlling its scale. A compresses the input into r coordinates, and B maps them into the output space. Every column of BA belongs to the space spanned by B’s columns, so rank(BA) ≤ r. The constraint applies to ΔW, not to W₀ or W_eff, both of which can be full rank. Freezing W₀ does not make it small or remove it from memory.

We store rk coefficients in A and dr in B. For a single square matrix with d = k = 4096, this gives the following comparison. These are trainable coefficient counts, excluding biases and other modules:

At rank 8 we train 256 times fewer coefficients for this matrix. A 256-fold reduction in total memory does not follow: frozen weights, activations, buffers and, depending on the implementation, weight copies and optimizer states remain. Gradient propagation through the model is still necessary. Savings depend on numerical precision, sequence length, batch size, and the modules involved.

The factorization also contains redundancy. For any invertible r × r matrix R, the pairs (B, A) and (BR, R⁻¹A) produce the same update. On the subset of matrices with rank exactly r, the matrix manifold has r(d + k − r) degrees of freedom, while the stored parameter count remains r(d + k). This distinction explains why counting numbers in memory is not the same as counting independent functional directions.

For a single vector, the additional branch requires work proportional to r(k + d), on top of multiplication by W₀. After training, sBA can be merged into W₀ to use a single matrix, if the numerical format and infrastructure permit it. The algebraic possibility of merging does not automatically guarantee unchanged latency in every system, especially when adapters are kept separate or weights are quantized.

3. Gradients explain initialization and scaling

Let L denote the loss and G = ∂L/∂W_eff the gradient with respect to the effective matrix. G has dimensions d × k. Starting from dW_eff = s(dB)A + sB(dA) and using the matrix inner product ⟨P,Q⟩ = tr(PᵀQ), we collect the terms in dA and dB separately. This gives:

If B is initialized to zero and A to nonzero random values, the initial update is zero: the model starts with its pretrained function. On the first step A’s gradient is zero, while B’s can be nonzero. Once B changes, A can learn as well. Initializing both to zero would instead block both gradients in this parameterization. This is more than a cosmetic detail: the initial asymmetry preserves the original function without preventing learning.

The equations also reveal why factorization changes optimization. Under simultaneous gradient descent with step size η, neglecting terms of order η², the change in the update is:

Direct fine-tuning of the matrix would instead take a step −ηG. The matrices AᵀA and BBᵀ filter the gradient according to the factors’ current geometry. Even when representational capacity is sufficient, trajectories can differ. Equation (3) describes plain gradient descent; it does not exactly describe Adam, weight decay, or clipping. It isolates a mechanism rather than replacing measurements of actual training.

The conventional choice s = α/r couples scaling and rank. Changing r while holding α fixed therefore changes two things at once: capacity and dynamics. rsLoRA proposes s = α/√r, motivated by an analysis of rank stability under explicit assumptions about initialization and scaling. The practical implication is to compare rank, scale, and learning rate within a coherent protocol, without assuming that one value of α makes every experiment equivalent. [2, Sections 2–4]

4. A measurable limit: how much energy is left out?

Suppose, temporarily, that we know a target correction ΔW*. This artificial assumption is useful for studying capacity: in real specialization, nobody supplies us with this matrix. Write its singular value decomposition, with singular values ordered from largest to smallest:

The vectors u_i and v_i form orthonormal bases in their respective spaces. Truncating the sum retains r elementary transformations. The SVD’s best-approximation property in Frobenius norm states that no matrix of rank at most r can outperform the truncated sum under this criterion. The squared Frobenius norm is the sum of the squares of all matrix entries.

The residual identity is also visible directly: the terms u_iv_iᵀ are mutually orthogonal under the Frobenius inner product, so the energies of discarded terms add up. The optimality of truncation supplies the decisive step: we are describing not just one possible approximation but the best attainable limit for that rank and metric.

LoRA can represent this limit: for positive s, choose B = U_rdiag(√σ_i/√s) and A = diag(√σ_i/√s)V_rᵀ. Then sBA = ΔW*_r. This is an existence construction, not an algorithm for training on text. It does not establish that an optimizer will find these factors, or that reconstructing a matrix is the right objective for an LLM.

5. Executed experiment: the same rank, two different problems

We construct two 64 × 64 matrices with the same orthogonal bases U and V, obtained by QR decomposition of Gaussian matrices. In the first case, σ_i = exp(−(i−1)/8): a few directions hold much of the energy. In the second, every singular value is 1: no direction is privileged. For each rank, we compute the truncated SVD and measure ε_r. There is no noise, no training phase, and no hyperparameter selection on a test set.

The random seed is 20260921. Computation was performed in float64 with Python 3.14.0 and NumPy 2.5.3; the plots were generated from the same results with Matplotlib 3.11.2. To check the functional interpretation, we also evaluate 20,000 independent Gaussian inputs x ∼ N(0,I), comparing mean output-error energy with target-output energy.

For the flat spectrum, the formula simplifies to ε_r = (64−r)/64. Losing 87.5% at rank 8 is unavoidable for this objective, even with a perfect optimizer. For the decaying spectrum, rank 16 instead preserves about 98.17% of the energy. Saying “rank 16 is enough” without specifying the spectrum and metric removes the very information that makes the result useful.

The check using 20,000 inputs yields an empirical residual at rank 8 of 13.4516% in the first case and 87.5061% in the second, close to the expected values. This is a Monte Carlo check, not a confidence interval or a corpus-based result. When singular values are equal, the basis chosen by SVD is not unique: small numerical differences between environments may change the sample residual while leaving the theoretical value unchanged.

Parameter counting reveals another limit: for 64 × 64 matrices, the factorization stores 128r coefficients. At rank 32 it stores 4,096, as many as a full matrix; at rank 64 it stores twice as many. These ranks are included to show the entire capacity curve. They would not be justified by parameter savings. The economic ratios of the 4096 × 4096 example cannot be transferred to this smaller synthetic problem.

6. The counterexample: data can reverse the choice

The Frobenius norm gives equal weight to every coordinate. Real systems do not necessarily receive isotropic inputs. For zero-mean inputs with covariance Σ = E[xxᵀ] and matrix error E = ΔW* − M, the expected squared output error is:

Here E denotes the error matrix and E_x[·] denotes statistical expectation. With Σ = I we recover the previous criterion. Otherwise, what matters is the combination of error and input distribution. Take ΔW* = diag(4,1) and Σ = diag(1,100). The best rank-1 approximation in Frobenius norm is M_F = diag(4,0): it preserves the largest singular value and leaves a squared coefficient error of 1.

But the second input coordinate has variance 100. Discarding it creates an expected output error of 100. Choosing M_Σ = diag(0,1) instead increases coefficient error to 16 but reduces expected output error to 16. With no correction, output error would be 116. A weight-only criterion therefore prefers the worse solution for this distribution.

If Σ is positive definite, solve the weighted problem by truncating the SVD of ΔW*Σ^1/2 and then multiplying on the right by Σ^−1/2. Invertible transformations preserve rank. If Σ is singular, its support must be treated separately; an ordinary inverse cannot be used. Even in a linear problem, the distribution changes which directions deserve the budget.

A Transformer is more complex: layer inputs depend on preceding layers, the final loss is not simply a distance between linear outputs, and adapting multiple modules can alter internal representations. The SVD of an individual update is therefore a diagnostic tool, not an automatic rule for assigning rank across an entire network.

7. From capacity to a protocol for domain-specific models

To specialize a model for a domain, first define what constitutes improvement: extracting fields from documents, using technical terminology correctly, and following constrained instructions are different objectives. An average score can conceal deterioration on rare cases. The boundary between training and testing must also follow the actual information unit: randomly splitting near-identical pages from the same document makes the evaluation too easy.

A proposed protocol, not executed in this article, compares the base model with ranks 4, 8, 16, and 32, initially fixing the adapted modules. Each configuration receives a comparable search budget for learning rate and scale, uses multiple random seeds, and selects hyperparameters on validation data. The final test remains separate until the decision. Then, where possible under an equal parameter budget, adapting a few modules is compared with distributing adaptation across more modules.

Record domain quality, performance on general control tasks, peak memory, elapsed time, token count, and variability between runs. Comparison with full fine-tuning, when feasible, helps separate constraints imposed by the parameterization from limitations of the base model. Additional epochs or tuning attempts must not become an undisclosed advantage for one configuration.

Training and validation curves offer clues, not infallible diagnoses. If both remain poor, capacity, optimization, and data quality remain open hypotheses. If only training improves, higher rank may be fitting sample-specific quirks. If average performance improves but a critical subset deteriorates, the decision should follow the intended use rather than the appeal of an aggregate number.

Another direction is to change the parameterization. DoRA separates weight magnitude from direction and uses a low-rank update for the directional component. The 2024 paper proposes this choice to change adaptation dynamics and evaluates it on particular tasks. It is not simply “LoRA with higher rank,” nor a universal guarantee of superiority. We have reproduced neither DoRA nor rsLoRA here: they are documented methodological comparisons, not results from our experiment. [3, Sections 3–5]

8. Reproduce, check, and delimit

The following core shows how the residual is computed. The downloadable file also includes matrix construction, the Monte Carlo check, the covariance counterexample, and finite-difference verification of the gradients. The latter produced a maximum absolute error of approximately 2.71 × 10⁻⁹.

u, singular, vh = np.linalg.svd(target, full_matrices=False)
approx = (u[:, :r] * singular[:r]) @ vh[:r, :]
residual = target - approx
epsilon = np.sum(singular[r:]**2) / np.sum(singular**2)
direct = np.sum(residual**2) / np.sum(target**2)
assert abs(epsilon - direct) < 1e-12

To rerun everything, install NumPy and run python experiment.py --out results.json. The JSON records versions, seed, dimensions, and results. The script requires no models, credentials, or external datasets. Numerical reproducibility means agreement within the tolerances specified in the code, not identical digits across every BLAS library.

Download code, results, plotting script, and instructions · Download the JSON results.

The main limitation is deliberate: we use an oracle that knows ΔW*, rather than a process that learns it. We do not measure convergence, label noise, linguistic generalization, GPU memory, or latency. The isotropic case and anisotropic counterexample establish mathematical properties of the constructed problems; they do not estimate the update distribution of a particular commercial model. The cited papers also have their own dates and protocols: this is not an exhaustive review of the state of the art in 2026.

9. The technical conclusion

LoRA makes a constrained family of updates economical. Its suitability depends on the energy and functional relevance of the directions that family can express. Our calculation exposes two reasoning failures: equal rank does not imply equal relative reconstruction capacity, and better weight reconstruction does not imply lower error on data. The gradient equations add a third level: a representable solution may be difficult to learn with the chosen parameterization.

For the domain-specific model work that EL-AI intends to explore, this establishes a concrete design question: which adaptation directions does the task require, and what measurements can demonstrate that they have been learned? The attached experiment is educational material for this publication, not evidence of an already validated product or client project. The scientifically useful next step would be to test these hypotheses with separated domain data, a declared budget, and a repeatable comparison.

Primary references

1. Hu, E. J. and colleagues. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685, version 2, 2021. Methods and rank analysis: Sections 4 and 7.
2. Kalajdzievski, D. A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. arXiv:2312.03732, version 1, 2023. Scaling assumptions and analysis: Sections 2–4.
3. Liu, S.-Y. and colleagues. DoRA: Weight-Decomposed Low-Rank Adaptation. arXiv:2402.09353, version 3, 2024. Decomposition, method, and evaluation: Sections 3–5.
4. James, D., Solomon, J. Singular Value Decomposition. Stanford CS 205A, 2016. Slides 25–26: Eckart–Young theorem and matrix norms. [4]

Illustrative AI-generated cover: it does not depict actual EL-AI premises or installations. The scientific figure was generated from the attached calculation data.