Two specialisations do not combine just because the files are small
A company adapts the same model to two tasks and obtains two small LoRA (Low-Rank Adaptation) adapters. Averaging their parameters seems a natural way to retain something from each. But which parameters are being averaged: the model’s actual update or the two factors representing it? This is not a merely formal distinction. We show two adapters with exactly the same update whose naive factor average produces a zero update.
The central question is how to combine factorised updates without introducing unwanted terms, and what is lost if the original rank must be retained. We use constructed 2 by 2 matrices, executed calculations, and a checked compression. We have not trained a model, measured accuracy, or reproduced a benchmark. ICML 2026 research provides current context, separate from our educational results; this monograph is not original research.
The product first, its compact representation second
For one linear layer, let W₀ be the starting model matrix with d rows and k columns. LoRA represents a change using B with d rows and r columns and A with r rows and k columns. Their product BA has the same shape as W₀ but rank at most r. Rank limits the number of independent directions the update can express; it does not directly measure answer quality.
s is the adapter scaling factor. Our examples set it to 1; a real system must include it in the effective update before defining a merge. Adapters must refer to the same base model and compatible layers. Matching matrix shapes alone does not justify merging models trained from different bases. This condition belongs to the problem definition and cannot be replaced by inspecting factor files alone.
The factorisation is not unique
Think of B as building output directions and A as assigning coordinates along them. Scaling a B direction and inversely scaling the corresponding A coordinates leaves the product unchanged. More generally, for an invertible r by r matrix Q, BQ and Q inverse A still represent BA. The model sees the product, not the arbitrary choice of internal coordinates.
Neither the task nor the layer effect has changed: we have described the same matrix in two ways. A method claiming to average updates should respect this equivalence. If its result changes after an adapter is refactorised without changing its product, it is also averaging a representation choice. Our first experiment isolates exactly this problem without introducing conflict between tasks.
Two identical updates that become zero
Choose a rank 1 update acting only on the first coordinate. Adapter one uses B₁=(1,0) as a column and A₁=(1,0) as a row. For adapter two, reverse both signs. Its product also has 1 in the upper-left corner and zeros elsewhere because the negative signs cancel. The intended product average is therefore that same matrix, not zero.
But averaging B₁ with B₂ and A₁ with A₂ separately makes both averaged factors zero. Their product is zero. An equivalent representation of the second adapter has erased the entire update even though there were no different effects to reconcile. This is a counterexample to naive factor averaging, not a proof that every form of LoRA merging is impossible.
A sign change is not the only case. Our code multiplies B₂ by a nonzero c and divides A₂ by c, preserving the product. With c=4, factor averaging gives 1.5625 in the first entry instead of 1; with c=−2 it gives−0.125. Thus it can amplify or even reverse an update under a change that leaves both adapters’ linear effects unchanged.
Expanding the product reveals cross terms
To identify the cause, take a weight w between 0 and 1 and compare two operations. Update averaging combines wB₁ A₁ with (1−w) B₂ A₂. Factor averaging first combines the B factors and A factors, then multiplies the results. Expansion introduces B₁ A₂ and B₂ A₁, pairing factors from different adapters, while each original product receives a squared weight. This is not the stated update average.
The program verifies the last identity for five w values. It also explains special cases: identical B factors or identical A factors make the difference vanish, as do w=0 or w=1. These are sufficient conditions, not permission to assume shared factors in independently trained adapters. Matrix shapes do not establish that internal coordinates are already aligned.
An exact sum exists, but rank can grow
To compute D exactly, concatenate weighted B factors horizontally and stack A factors vertically. Block multiplication produces only corresponding products, without cross terms. For input ranks r₁ and r₂, the result has rank at most r₁+r₂, also bounded by d and k. Each adapter scale can be absorbed into its weighted B factor, so scales need not be identical.
The cost is visible in the shapes: if R is the sum of ranks, the two factors contain R(d+k) coefficients, not r(d+k) at the original rank r. This counts parameters, not total RAM or latency. In our tiny 2 by 2 example, rank 2 factorisation does not even save storage compared with the dense matrix; its purpose is transparent algebra. For large layers, the benefit depends on R relative to d and k.
When adapters add independent directions
In the second experiment, we genuinely combine different effects: the first update is diag(2,0), the second diag(0,1). Equal weighting gives D=diag(1,0.5), of rank 2. Each input adapter has rank 1, but their average contains two independent directions. This is a structural change, not numerical error. A single rank 1 adapter cannot represent that matrix exactly.
Naive factor averaging produces the last matrix: it changes diagonal values and introduces off-diagonal coefficients absent from both original updates. Concatenation returns D exactly. We compare errors using the Frobenius norm, the square root of the sum of squared coefficient differences. This is a matrix distance, not a task-accuracy percentage.
Returning to the original rank means choosing what to lose
Singular value decomposition, SVD, separates a matrix into input and output directions with nonnegative strengths. D has singular values 1 and 0.5. Its best rank 1 approximation in Frobenius distance retains 1 and discards 0.5, giving diag(1,0) with error 0.5. It is optimal for that distance, not necessarily for every task the model may serve.
| Method | Result rank | Frobenius error |
|---|---|---|
| Factor average | 1 | 0.7905694150 |
| Rank 1 SVD | 1 | 0.5000000000 |
| Exact concatenation | 2 | 0.0000000000 |
To make the limitation concrete, use input e₂=(0,1). The intended update returns (0,0.5); the truncated one returns zero. We do not claim the model lost 100% accuracy; there is no benchmark here. The precise verified statement is that the linear response along this direction was completely removed. A compression criterion must be interpreted alongside the inputs and objectives that matter.
Compressing without immediately building the dense matrix
Exact summation and representation cost are separate questions. If concatenated rank R is small relative to d and k, we can work with thin factors. QR decomposition writes each factor as an orthonormal-column basis times a compact matrix. Orthonormal directions are perpendicular and unit length; expressing the represented subspace in these coordinates preserves its Euclidean distances.
When R does not exceed d or k, the core Rb times Ra transpose is R by R. We can compute its SVD, retain r components, and map back through Qb and Qa. Our program verifies agreement with the truncated SVD of D in the constructed example. This is not an implementation of the paper’s CtM method; it checks a QR-core route for exact summation followed by compression.
For dense thin factors, indicative arithmetic costs scale as(d+k) R²for the QR steps and R³for core SVD under those shape assumptions. We do not convert these orders into milliseconds or measured memory savings; no large model was run. As R grows, the core becomes less advantageous too. Factorisation avoids some costs but does not make merging arbitrarily many adapters free.
The2026context: choosing subspaces before merging
Compress then Merge by Zhengbao He and colleagues appears in ICML 2026 proceedings, PMLR 306, pages 41212–41243. We read arXiv 2606.03723 v 1 dated 2 June₂026 for its method and experimental protocol, and separately verified proceedings publication. The work separates shared-subspace selection from the subsequent merge operator, enforcing output rank by construction rather than only after merging.
These results cannot automatically transfer to a company use case. The inspected protocol uses rank 16 adapters with CLIP ViT-B/32 on eight vision tasks and LLaMA3-8B on six natural language inference (NLI) tasks, which assess relations between sentences, selecting hyperparameters on validation and normalising test results against single adapters. We did not reproduce the experiments. A matrix-rank guarantee is not a universal capability guarantee; subspace choices still need task-relevant evaluation.
What we proved and what still needs measurement
Our results separate three often-confused issues: algebraic correctness of averaging, representability of the result at a chosen rank, and model quality on data. The sign counterexample concerns the first; the rank 2 diagonal matrix concerns the second. The third needs datasets, baselines, adaptation configuration, and independent evaluation. Neither constructed example alone establishes improved responses from a real model.
No example contains EL-AI product weights or customer data. Code uses Python and NumPy with fixed matrices and no randomness, saving results and tolerance 10⁻¹². In the snippet, three assertions compare the intended effect, the wrong result, and correct concatenation. Checking factor shapes alone would miss the issue; products and declared properties provide the test reference.
The answer is clear: averaging factors generally differs from averaging updates because cross terms appear and a nonunique representation matters. Concatenation gives an exact construction but may increase rank. Compressing it requires choosing acceptable loss, which Frobenius error alone does not translate into business utility. Before combining capabilities, establish which mathematical operation the software actually performs.
Sources and related reading
LoRA: rank, adaptation geometry, and measurable limits
import numpy as np
B1=np.array([[1.],[0.]])
A1=np.array([[1.,0.]])
B2=-B1; A2=-A1
expected=(B1@A1+B2@A2)/2
factor_average=((B1+B2)/2)@((A1+A2)/2)
assert np.allclose(expected,[[1,0],[0,0]])
assert np.allclose(factor_average,0)
Bcat=np.concatenate([B1/2,B2/2],axis=1)
Acat=np.concatenate([A1,A2],axis=0)
assert np.allclose(Bcat@Acat,expected)
print(expected, factor_average)
Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 21 October 2026.