Lecture 3: Rank, Determinant, LoRA Intuition

Rank, Determinant, LoRA Intuition

How much information does a matrix actually carry?

Vishnu Boddeti

Example 1

LoRA adapts a model through a low-rank update

Frozen weight matrix W

Small factors B and A

Adapted weight W + BA

Concrete exampleInstead of updating a billion-parameter layer, training learns two thin matrices whose product changes only a few directions.

Low rank reduces trainable parameters and memory while retaining useful adaptation capacity.

LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)

Why can a foundation model be adapted with a small LoRA update?

Fine-tuning updates a huge weight matrix, but LoRA replaces that update with the product of two tiny matrices — why does that work without losing much?

Rank counts independent output directions

The rank of a matrix is the dimension of its column space: the number of independent output directions.
For $\Delta W=ab^\top$, column $j$ is $b_j a$, so every column lies in $\operatorname{span}(a)$.
The rank is 1 when both vectors are nonzero, and 0 when either vector is zero.

Build a rank-one LoRA update

Change the first component $a_1$ Change the second component $a_2$ Change the first component $b_1$ Change the second component $b_2$

Every column of $ab^\top$ is a multiple of $a$, so the update can add at most one independent direction.

Example 2

Exact-likelihood flows track volume change

Base density

Invertible transformation

Data density plus Jacobian correction

Concrete exampleA continuous normalizing flow deforms a Gaussian and integrates divergence to recover the final log likelihood.

Exact likelihood requires accounting for local expansion and contraction; flow matching can train the velocity field without that full computation.

FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (Grathwohl et al., 2018)

What does an exact-likelihood flow compute that flow matching can avoid?

Exact-likelihood normalizing flows must track how every transformation stretches probability mass.
Current flow-matching systems can train a velocity field without evaluating this factor.
But the geometry is unchanged: what quantity measures how much a small patch is stretched or squashed?

The determinant measures signed area change

For $y=Wx$, $\det(W)$ is the signed area-scale factor.
Its magnitude gives the change in area: $|\det(W)|=2$ doubles the unit square's area.
Its sign records orientation; $\det(W)=0$ means at least one direction collapses.

Map a unit square and read off $|\det(W)|$

W₁₁ (top-left) W₁₂ (top-right) W₂₁ (bottom-left) W₂₂ (bottom-right)

The mapped square has area $|\det(W)|$. A value near zero means that $W$ nearly collapses the plane.

Example 3

RoPE encodes position with rotations

Query and key feature pairs

Rotate by token position

Relative phase shapes attention

Concrete exampleA long-context transformer rotates each two-coordinate query and key pair before taking their dot product.

The rotation has determinant one and inverse equal to its transpose, so it preserves norms and area while relative position changes attention.

RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021)

Why can RoPE add position without changing vector length?

For each frequency, RoPE applies a $2\times2$ rotation $R_m$ to a query or key coordinate pair.
$R_m^\top R_m=I$ and $\det(R_m)=1$, so lengths, angles, and oriented area are preserved.
$q_m^\top k_n=q^\top R_{n-m}k$, so the attention score depends on relative position.

Increase relative position and compare geometry

Move the control to compare the two normalized responses.

Relative token offset

The same-position rotations cancel in the dot product; the relative phase remains.

Example 4

Rank deficiency can hide agent failures

Agent trajectory features

Evaluator projection S

Score and pass or fail

Concrete exampleTwo tool-use traces can differ in policy compliance and recovery behavior yet receive the same score when their difference lies in the kernel of S.

A rank-deficient evaluator cannot observe every independent failure direction. Collision-pair tests expose blind spots hidden by aggregate success rates.

τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024)

Can an evaluator distinguish every agent failure mode?

An evaluator maps trajectory features $x$ to rubric scores $s=Sx$.
If $v\in\ker(S)$, then $S(x+v)=Sx$: a real behavioral difference is invisible to the score.
Pair matched traces that differ in policy compliance, recovery, or tool use to test evaluator coverage.

Raise evaluator rank and compare coverage with cost

Move the control to compare the two normalized responses.

Independent rubric signals

Add score dimensions only when failure-pair probes show that they detect a previously hidden behavior.

Practice problems

Each scenario hides a mathematical structure from this lecture. Identify the structure, justify it, and work through the resulting problem.

Problem 1 Gradient propagation

Backpropagate through a frozen projection

During adapter training, a frozen projection turns a large activation into a smaller feature vector. The backward pass supplies a gradient in the smaller space, and now you must send it back to the original activation without mixing up the direction of the map.

Model evidence

$z=Wx$, with $W=\begin{bmatrix}1&2&0\\-1&0&3\end{bmatrix}$, and the downstream gradient is $g_z=(4,-2)^\top$.

Which linear map carries $g_z$ back to the input coordinates?
What input gradient does it produce for the given numbers?
Why would multiplying by $W$ itself be both dimensionally and mathematically wrong?

Problem 2 Model interoperability

Load an embedding checkpoint in a new basis

Two labs trained decoders on the same latent plane but named its coordinates differently. A checkpoint from Lab A therefore looks wrong when loaded by Lab B, even though both labs represent the same underlying vector.

Model evidence

In ambient coordinates, $P=\begin{bmatrix}1&1\\0&1\end{bmatrix}$ contains Team A's basis vectors and $Q=\begin{bmatrix}1&0\\1&1\end{bmatrix}$ contains Team B's. The checkpoint coordinate is $c_A=(2,-1)^\top$.

What ambient latent vector does Lab A's checkpoint represent?
What coordinates should Lab B use for that same vector?
What matrix converts arbitrary coordinates from basis A to basis B?

Problem 3 Feature fusion

When does a feature mixer destroy information?

A feature-mixing layer has a knob $\alpha$. Most settings merely remix information, but one unlucky setting makes an entire latent direction disappear. Where is that failure hiding?

Model evidence

$A_\alpha=\begin{bmatrix}1&0&1\\0&1&1\\1&1&\alpha\end{bmatrix}$.

For which values of $\alpha$ does the layer erase a nonzero feature vector?
At the failing value, what is the rank, and which direction is erased?
What do the determinant and rank tell you about information flowing through the layer?