A clause at the beginning, a question at the end
A company wants to query a long manual: a rule appears near the beginning, while the question comes after hundreds of sections. Increasing the input limit seems natural. Yet accepting a document, representing distances and retrieving the rule are different capabilities. We study the second to explain why it does not prove the third. Starting with planar rotation, we derive position scaling and compare mathematical evidence with evidence for actual context extension.
Abstract. Explain Rotary Position Embedding, RoPE, using two-dimensional vectors and dot products. Derive relative-distance dependence and verify translation invariance. In an untrained example, dividing positions by 4 maps distance 80 to the phase of 20 while reducing neighbor encoding distance from.099958 to.024999. Two frequencies show why one channel’s periodicity does not describe the whole representation. RoFormer and LongRoPE 2 separate encoding properties, authors’ results and business-task understanding, which we do not measure.
Before positions: what attention compares
A token is a sequence unit, not necessarily a whole word. A model builds numerical vectors: a query represents what one position searches for, a key what another can match. These are operational interpretations, not human intentions. Dot products sum coordinate products to form scores, then normalized to combine value vectors. We isolate pre-normalization score, not probability of answering correctly.
To isolate order, keep content vectors fixed while changing positions. A comparison without positional information cannot by itself distinguish locations. Real vectors depend on text and preceding layers. Fixing them makes encoding effects readable but prevents predicting full natural-language behavior.
Rotating without changing length
A planar rotation changes direction while preserving length. Use column-vector pairs q and k. At integer m rotate q by mθ radians; at n rotate k by nθ. Here θ=.1 radians per position step. Arrows explain numbers, not neurons or physical motion inside a processor.
Rows combine coordinates with sine and cosine. Transpose T swaps rows and columns and equals opposite rotation. Rotation times inverse is identity I, preserving Euclidean length. Code checks vector(3,4), length 5, retains squared norm 25 at angles.1,2,8 radians. This checks an identity, not trained-LLM stability.
Why relative distance remains
What remains after rotations? Substitute q_m,k_n and group matrices. Rotation by nθ then−mθ equals(n−m)θ. Explicit dependence becomes positional difference, RoPE’s core in RoFormer. The derivation uses elementary identities; our example does not reproduce authors’ training.
Adding integer c to both positions preserves n−m and fixed-vector score. Code compares(0,20),(7,27),(40,3) with shifts 100, tolerance 10⁻¹⁴ in Python floating point. Prepending 100 tokens need not preserve answers: content, masks, normalization and vectors change. This is an isolated local property.
A complete example: one frequency
Choose q=k=(1,0): S(Δ)=cos(.1Δ), Δ=n−m in token steps. Distance 20 gives phase 2 radians, score−.416147;80 gives phase 8 and−.145500. Negative pre-normalization scores are valid, not negative probabilities. Reference 0–20 compares phases; no 20-token model was trained.
One frequency repeats after continuous distance 2π/θ≈62.831853. Integer 63 nearly coincides,.999859, not exactly 0. This limits one channel, not proof an LLM confuses 0 and 63. Coordinates, contents and layers all affect output.
Compressing positions: benefit and cost
To preserve maximum phase at 80 divide positions by s=4:80 ×.1/4=2, previously 20. But distance 20 now has phase.5 and score.877583 instead of−.416147. Geometry changes within the old range. Fractional rotation coordinates do not split text tokens.
Adjacent unit-circle encodings have chord twice the sine of half-angle. With θ=.1, separation.0999583 becomes.0249993 at s=4. Geometric compression is not lost-accuracy percentage. Models may adapt weights, frequencies or layers; success requires unperformed training and evaluation. Benefit and cost are representation properties.
| Distance (tokens) | Unscaled | Scale4 |
|---|---|---|
| 1 | 0.995004 | 0.999688 |
| 20 | −0.416147 | 0.877583 |
| 63 | 0.999859 | −0.004204 |
| 80 | −0.145500 | −0.416147 |
Two frequencies are not two trained models
Construct K(Δ)=[cos(.1Δ)+cos(.01Δ)]/2 with two equal-weight pairs, not a specific model’s complete kernel. At 63 the first cosine is near 1, second.808028, average.903943 not 1. At 80 mean.275603 becomes.281960 scaled, matching old 20. Integer distances 0–80 are saved for plotting. Frequencies simplify arithmetic, not replicate a commercial model or linguistic observations.
Combining frequencies resolves some ambiguities but certifies no safe length, common global period, reliable maximum or monotonic score-understanding link. A rising curve does not prove distant relevance with fixed content and weights. This is controlled geometry, not LLM attention. Key intuition: enlarging scale changes the ruler distinguishing nearby positions.
Guided appendix: content changes the curve
The q=k example hides a detail: relative distance is signed, and keys before and after a query need not score equally. Let J=[[0,−1],[1,0]], a 90-degree rotation. Since R(φ)=I cosφ+J sinφ, the product expands into two content-dependent coefficients: the ordinary dot product and q compared with quarter-turned k. Position therefore does not act identically on all content pairs.
For q=(1,2),k=(3,−1), coefficients are 1 × 3+2 ×(−1)=1 and 1 × 1+2 × 3=7. Reversing Δ preserves cosine but flips sine. Code evaluates direct and expanded products at−20,−1,0,1,20 and asserts agreement within 10⁻¹⁴. Main figure remains explicitly symmetric; extra results are in JSON. Coefficient 7 is not learned or an EL-AI parameter: it follows from chosen coordinates.
This also clarifies causal masking. A generating model may allow a query to use only earlier positions, separately from rotation, excluding a key regardless of score. Our calculations have no mask or softmax and include positive distances to show geometry. They are not causal attention execution. Rotation invariance also does not imply unchanged output after new keys are added: normalization depends on available alternatives.
The manual problem now separates useful representations, accessible evidence and task use of evidence. Connected conditions can be tested separately. Derivation does not select the highest-weight real clause; it explains why testing only position generation checks an insufficient part. Keeping elementary calculation and its assumptions makes the boundary with document understanding explicit.
What the papers establish
RoFormer by Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen and Yunfeng Liu was submitted 20 April 2021; we read arXiv v 5,8 November 2023, sections 3–4. It introduces paired rotations and linguistic evaluations. WMT 2014 English–German uses about 4.5 million pairs, reporting BLEU 27.3 versus 27.5 under authors’ settings. This proves no unlimited understanding; we downloaded no dataset or trained model.
LongRoPE 2 by Ning Shang and colleagues appears in ICML 2025, PMLR 267:54203–54218. We also read arXiv v 1,27 February 2025: scaling search uses perplexity on distant retrieval tokens, with short/long-window weight adaptation. That version reports LLaMA 3-8 B RULER mean 82.03 at 128 k across 13 synthetic tasks, Phi 3-mini 58.81, using 10 billion tokens and 64 A 100 s. These are authors’ results. Proceedings status is verified separately from the technical version read.
Different scores and synthetic tasks matter: an average is not reliability on every manual question. Methods involve search, adaptation and short-window evaluation, not merely changing a constant. Our rotation does not reproduce LongRoPE 2 or select factors. These are verified references, not an exhaustive leaderboard or denial of later 2026 developments.
Testing the objective beyond the input limit
For the manual, test retrieval and application amid exceptions and versions. A proposed protocol varies length, evidence position and distractors with verifiable answers, separating literal retrieval from multi-rule reasoning. Compare the same model version and record RoPE configuration, precision, prompt and sample. No such test or success rate is reported here. Geometry motivates but does not replace it.
Cost differs from quality. Pairwise rotation is linear in dimension per token without a dense d×d matrix. It removes neither dense-attention interactions nor determines implementation memory. No GPU, latency or energy measurements. Code uses Python math without trained weights, saving results and assertions. Figures connect integer-distance calculations for readability, not physical measurements.
More pages establish input capacity, not understanding relationships. RoPE represents distances explicitly; scaling changes structure and needs model/task evaluation. The example computes geometric benefit and local cost. Companies can request relevant evidence without treating nominal window size as outcome guarantee.
from experiment import score
for distance in [1,20,63,80]:
print(distance, score(0,distance), score(0,distance,4))Code, data, and instructions · JSON. Educational calculations executed with Python 3.14.0; figures with Matplotlib 3.11.2. AI-assisted analysis, without claiming peer review or human review. Original illustrative ImageGen cover: it does not document EL-AI people, premises, or installations. Sources accessed 17 October 2026.