Lecture 5: p-norms, Sparsity, Whitening

p-norms, Sparsity, Whitening

How do sparsity penalties and representation covariance shape modern models?

Vishnu Boddeti

Example 1

Structured pruning removes coherent model components

Groups of related weights

Group sparsity penalty

Whole group becomes zero

Concrete exampleAll weights feeding one attention head shrink together, allowing the complete head to be removed.

Hardware benefits when sparsity removes structured blocks rather than isolated weights that remain expensive to index.

The Group Lasso for Logistic Regression (Meier, van de Geer, and Bühlmann, 2008)

How can structured sparsity remove whole heads, channels, or experts?

Current compression and conditional-compute systems often want an entire attention head, channel, or expert block to become inactive.
Group-sparsity penalties apply an L2 norm within each block and an L1-style sum across blocks, so the geometry can drive whole groups—not isolated scalars—to exactly zero.

p-norm and quasi-norm balls

For $p \ge 1$, $\|x\|_p = \left(\sum_i |x_i|^p\right)^{1/p}$ is a p-norm.
For $0 < p < 1$, the same formula defines a quasi-norm, because it does not satisfy the triangle inequality.
The $p=2$ norm ball is a smooth circle, while the $p=1$ norm ball is a diamond with corners on the coordinate axes.
Smaller values such as $p=0.5$ produce a nonconvex quasi-norm ball with even sharper axis points.

Change $p$ and compare the boundary near each axis

Change the exponent $p$: 2.0

Drag the orange point and compare its L1 and L2 norms

Interpret each coordinate as a group norm: axis-aligned corners explain why a structured penalty can deactivate a whole head, channel, or expert.

Example 2

Feature covariance exposes redundant representations

Two augmented views

Encoder feature batches

Cross-correlation target

Concrete exampleTwo crops of the same image should share corresponding features without every feature collapsing to the same value.

Diagonal terms encourage invariance while off-diagonal penalties discourage redundant coordinates.

Barlow Twins: Self-Supervised Learning via Redundancy Reduction (Zbontar et al., 2021)

Why do representation-learning systems monitor and regularize feature covariance?

Embedding dimensions can become redundant or collapse onto a few directions.
Current representation-learning objectives therefore monitor off-diagonal covariance
and some pipelines decorrelate or whiten embeddings before retrieval or a downstream head.
What transformation separates correlation from scale?

Whitening rotates, then rescales

A positive definite covariance matrix $\Sigma$ diagonalizes as $\Sigma = Q\Lambda Q^\top$.
Multiplying centered data by $Q^\top$ rotates it into the eigenbasis, so the coordinates are decorrelated but retain variances $\lambda_i$.
Full whitening also rescales each coordinate by $1/\sqrt{\lambda_i}$.
The complete transform is $z = \Lambda^{-1/2}Q^\top x$, and the covariance of $z$ is the identity matrix.

Step through rotation and rescaling of the data

Change the entries of covariance matrix Σ a₁₁ a₁₂ = a₂₁ a₂₂

Covariance regularization discourages redundant directions. Full whitening goes further: rotation removes correlation, then Λ⁻¹ᐟ² gives every direction unit variance.

Example 3

RMSNorm stabilizes activation scale

Residual-stream vector

Divide by root mean square

Rescaled vector

Concrete exampleA transformer normalizes a token vector by its magnitude but does not subtract the coordinate mean.

Controlling scale improves optimization while preserving mean information and avoiding the centering operation.

Root Mean Square Layer Normalization (Zhang and Sennrich, 2019)

How does RMSNorm control activation scale without centering features?

activation vector x→divide by RMS(x)→scale-stable direction

RMSNorm rescales a token representation using its root-mean-square magnitude.
Unlike whitening, it does not rotate features or subtract their mean; it stabilizes scale while preserving the direction of the vector.

Increase input magnitude and compare raw with normalized activation scale

Move the control to compare the two normalized responses.

Input-scale multiplier

Scaling every coordinate changes raw magnitude, while normalization keeps the layer input on a controlled scale.

Example 4

Agents should activate only useful tools

Large tool catalog

Cost-aware sparse selection

Small executable tool set

Concrete exampleFor a weather question, the agent keeps search and weather APIs but excludes payments, email, and code execution.

Sparse selection reduces latency, prompt size, attack surface, and opportunities for an irrelevant action.

Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., 2023)

How should an agent choose a sparse tool set under a cost budget?

large tool catalog→group-sparse router→small active tool set

A broad tool catalog increases capability but also routing ambiguity, latency, and permissions exposure.
A structured sparsity penalty can deactivate whole tool groups while preserving a compact set that covers the task distribution.

Activate more tools and compare task coverage with execution cost

Move the control to compare the two normalized responses.

Active tool-catalog fraction

The useful optimum is sparse by groups: enough tools to cover tasks, few enough to route and authorize reliably.

Practice problems

Each scenario hides a mathematical structure from this lecture. Identify the structure, justify it, and work through the resulting problem.

Problem 1 Deep linear networks

Compute a deep linear block without multiplying 100 matrices

A tied-weight block reuses the same matrix one hundred times. Multiplying one hundred matrices is tedious and hides the simple story: in the right coordinates, each component only gets rescaled.

Model evidence

$A=\begin{bmatrix}3&1\\0&2\end{bmatrix}$ and the block output is $A^{100}x$.

Can you choose coordinates in which every application of $A$ acts component by component?
Build those coordinates and write a closed expression for $A^{100}x$.
Which component overwhelms the other numerically, and why?

Problem 2 Representation learning

Whiten correlated embedding features before retrieval

A retrieval embedding has two coordinates that mostly move together, so distance is dominated by one common fluctuation. You want to rotate into meaningful directions and rescale them so that each has unit variance before retrieval.

Model evidence

The centered embedding covariance is $\Sigma=\begin{bmatrix}5&4\\4&5\end{bmatrix}$, and one centered embedding is $x=(3,1)^\top$.

What are the eigenvalues and an orthonormal eigenbasis of $\Sigma$?
Build $W=\Lambda^{-1/2}Q^\top$ and check that $W\Sigma W^\top=I$.
What is $z=Wx$? Which coordinate records common motion, and which records contrast?

Problem 3 Autoregressive models

Debug a causal mixer from its diagonal

An autoregressive feature mixer is triangular because later internal features may use earlier ones, but not vice versa. From that structure alone, you should be able to spot an unstable mode and solve the layer backward efficiently.

Model evidence

$T=\begin{bmatrix}0.9&2&-1\\0&0.5&3\\0&0&1.1\end{bmatrix}$.

Which spectral values can you read directly from the matrix?
Will every hidden state decay if the mixer is applied repeatedly?
How does triangular structure turn $Tx=y$ into a sequence of simple solves?