Lecture 3: Rank, Determinant, LoRA Intuition

Rank, Determinant, LoRA Intuition

How much information does a matrix actually carry?

Vishnu Boddeti

Example 1

LoRA adapts a model through a low-rank update

Frozen weight matrix W

Small factors B and A

Adapted weight W + BA

Concrete exampleInstead of updating a billion-parameter layer, training learns two thin matrices whose product changes only a few directions.

Low rank reduces trainable parameters and memory while retaining useful adaptation capacity.

LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)

Why can a foundation model be adapted with a small LoRA update?

Fine-tuning updates a huge weight matrix, but LoRA replaces that update with the product of two tiny matrices — why does that work without losing much?

Rank counts independent output directions

The rank of a matrix is the dimension of its column space: the number of independent output directions.
For $\Delta W=ab^\top$, column $j$ is $b_j a$, so every column lies in $\operatorname{span}(a)$.
The rank is 1 when both vectors are nonzero, and 0 when either vector is zero.

Build a rank-one LoRA update

Change the first component $a_1$ Change the second component $a_2$ Change the first component $b_1$ Change the second component $b_2$

Every column of $ab^\top$ is a multiple of $a$, so the update can add at most one independent direction.

Example 2

Exact-likelihood flows track volume change

Base density

Invertible transformation

Data density plus Jacobian correction

Concrete exampleA continuous normalizing flow deforms a Gaussian and integrates divergence to recover the final log likelihood.

Exact likelihood requires accounting for local expansion and contraction; flow matching can train the velocity field without that full computation.

FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (Grathwohl et al., 2018)

What does an exact-likelihood flow compute that flow matching can avoid?

Exact-likelihood normalizing flows must track how every transformation stretches probability mass.
Current flow-matching systems can train a velocity field without evaluating this factor.
But the geometry is unchanged: what quantity measures how much a small patch is stretched or squashed?

The determinant measures signed area change

For $y=Wx$, $\det(W)$ is the signed area-scale factor.
Its magnitude gives the change in area: $|\det(W)|=2$ doubles the unit square's area.
Its sign records orientation; $\det(W)=0$ means at least one direction collapses.

Map a unit square and read off $|\det(W)|$

W₁₁ (top-left) W₁₂ (top-right) W₂₁ (bottom-left) W₂₂ (bottom-right)

The mapped square has area $|\det(W)|$. A value near zero means that $W$ nearly collapses the plane.

Example 3

RoPE encodes position with rotations

Query and key feature pairs

Rotate by token position

Relative phase shapes attention

Concrete exampleA long-context transformer rotates each two-coordinate query and key pair before taking their dot product.

The rotation has determinant one and inverse equal to its transpose, so it preserves norms and area while relative position changes attention.

RoFormer: Enhanced Transformer with Rotary Position Embedding (Su et al., 2021)

Why can RoPE add position without changing vector length?

For each frequency, RoPE applies a $2\times2$ rotation $R_m$ to a query or key coordinate pair.
$R_m^\top R_m=I$ and $\det(R_m)=1$, so lengths, angles, and oriented area are preserved.
$q_m^\top k_n=q^\top R_{n-m}k$, so the attention score depends on relative position.

Increase relative position and compare geometry

Move the control to compare the two normalized responses.

Relative token offset

The same-position rotations cancel in the dot product; the relative phase remains.

Example 4

Rank deficiency can hide agent failures

Agent trajectory features

Evaluator projection S

Score and pass or fail

Concrete exampleTwo tool-use traces can differ in policy compliance and recovery behavior yet receive the same score when their difference lies in the kernel of S.

A rank-deficient evaluator cannot observe every independent failure direction. Collision-pair tests expose blind spots hidden by aggregate success rates.

τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., 2024)

Can an evaluator distinguish every agent failure mode?

An evaluator maps trajectory features $x$ to rubric scores $s=Sx$.
If $v\in\ker(S)$, then $S(x+v)=Sx$: a real behavioral difference is invisible to the score.
Pair matched traces that differ in policy compliance, recovery, or tool use to test evaluator coverage.

Raise evaluator rank and compare coverage with cost

Move the control to compare the two normalized responses.

Independent rubric signals

Add score dimensions only when failure-pair probes show that they detect a previously hidden behavior.

Practice problems

Each scenario hides a mathematical structure from this lecture. Identify the structure, justify your choice, and then solve the resulting problem.

1 Choose 2 Solve

Problem 1 Gradient propagation

Backpropagate through a frozen projection

A frozen projection sends an activation vector to a smaller feature vector. During adapter training, the optimizer receives the gradient with respect to projected features and needs the gradient with respect to the original activation.

Model evidence

$z=Wx$, with $W=\begin{bmatrix}1&2&0\\-1&0&3\end{bmatrix}$, and the downstream gradient is $g_z=(4,-2)^\top$.

Derive the linear map that sends $g_z$ back to the input coordinates.
Compute the input gradient for this example.
Explain why using $W$ itself would be dimensionally and mathematically wrong.

Choose
Solve

Problem 2 Model interoperability

Load an embedding checkpoint in a new basis

Two teams use different bases for the same two-dimensional latent subspace. A checkpoint stores coordinates in Team A's basis, while Team B's decoder expects its own coordinates.

Model evidence

In ambient coordinates, $P=\begin{bmatrix}1&1\\0&1\end{bmatrix}$ contains Team A's basis vectors and $Q=\begin{bmatrix}1&0\\1&1\end{bmatrix}$ contains Team B's. The checkpoint coordinate is $c_A=(2,-1)^\top$.

Recover the ambient latent vector represented by the checkpoint.
Compute its coordinates $c_B$ for Team B without changing the latent vector.
Derive the general coordinate-conversion matrix from A to B.

Choose
Solve

Problem 3 Feature fusion

When does a feature mixer destroy information?

A fusion layer mixes three learned features. A hyperparameter controls how strongly the third output combines the first two, and the layer must not collapse a latent direction.

Model evidence

$A_\alpha=\begin{bmatrix}1&0&1\\0&1&1\\1&1&\alpha\end{bmatrix}$.

Find every value of $\alpha$ for which some nonzero feature vector is erased.
At the failing value, determine the rank and exhibit an erased direction.
Interpret the determinant and rank results as a statement about information flow.

Choose
Solve