Lecture 4: Eigenvalues, Trace, and PCA on Embeddings

Eigenvalues, Trace, and PCA on Embeddings

Which directions does a matrix refuse to rotate?

Vishnu Boddeti

Example 1

PCA reveals dominant directions in embedding spaces

Centered embedding cloud

Covariance eigensystem

Principal coordinates

Concrete exampleTop components can capture shared frequency, style, or formatting structure across language-model embeddings.

PCA supports diagnosis and compression, but high variance is not the same as downstream importance; some post-processing methods even remove dominant common directions.

All-but-the-Top: Simple and Effective Postprocessing for Word Representations (Mu, Bhat, and Viswanath, 2017)

What does PCA see in an embedding cloud?

Embedding clouds are rarely isotropic: frequency, style, or formatting effects can dominate a few directions.
PCA finds the orthogonal axes with the most centered variance before we decide what to retain, visualize, or remove.

Covariance eigenvectors are PCA directions

$C=\frac{1}{n}\sum_i(x_i-\bar{x})(x_i-\bar{x})^\top$ is symmetric and positive semidefinite.
$Cv_i=\lambda_i v_i$ means covariance stretches principal direction $v_i$ without rotating it.
$\lambda_i$ is the variance along $v_i$; the leading eigenvectors give the principal coordinates.

Change the covariance and identify the first PCA direction

Horizontal standard deviation σ₁ Vertical standard deviation σ₂ Correlation ρ

When the two eigenvalues are equal, every direction is principal and the first component is not uniquely defined.

Example 2

Hessian spectra summarize local training curvature

Current parameters

Hessian-vector products

Trace and leading eigenvalues

Concrete exampleAutomatic differentiation can probe sharp and flat parameter directions without materializing the full Hessian.

Leading eigenvalues reveal directional sharpness, while a stochastic trace estimate summarizes total signed curvature across all directions.

Eigenvalues of the Hessian in Deep Learning (Sagun et al., 2016)

How can training estimate curvature without forming the Hessian?

A foundation-model Hessian is far too large to form or store.
Automatic differentiation can compute a Hessian-vector product $Hz$ without materializing $H$.
Averaging $z^\top Hz$ over random probes estimates total curvature.

Trace is the sum of curvature eigenvalues

$\operatorname{tr}(H)=\sum_i H_{ii}=\sum_i\lambda_i$ for a Hessian $H$.
If $\mathbb{E}[zz^\top]=I$, then $\mathbb{E}[z^\top Hz]=\operatorname{tr}(H)$: the Hutchinson identity.
Trace summarizes signed total curvature; leading eigenvalues separately expose sharp directions.

Change a symmetric Hessian and verify its trace two ways

H₁₁ H₁₂ = H₂₁ H₂₂

The trace fixes the sum of eigenvalues, not how curvature is distributed between directions.

Example 3

Correlated agent failures reveal a common mode

Failure indicators by task

Failure covariance matrix

Principal failure direction

Concrete exampleRetrieval errors, unsupported claims, and failed recovery can rise together when one latent weakness drives all three.

The top eigenvector exposes a shared pattern worth investigating, while residual eigenmodes identify failures that need separate interventions.

Holistic Evaluation of Language Models (Liang et al., 2022)

Which failure pattern dominates an agent evaluation?

Standardize the three failure indicators for each agent trace and form $\Sigma=\operatorname{Cov}(f)$.
The top eigenvector is the failure combination with the most shared variation.
A common mode is diagnostic evidence, not a causal explanation; inspect tasks and traces before intervening.

Increase cross-failure correlation and watch the common mode

Move the control to compare shared and residual variation.

Correlation between failure indicators

As correlation rises, the shared $(1,1,1)$ direction dominates; residual modes shrink, but the eigenvector still does not identify a cause.

Example 4

Generalized eigenvectors trade utility against invariance

Reward target information

Penalize sensitive information

Solve the matrix-pair eigensystem

Concrete exampleAn invariant representation should retain information useful for predicting the target while suppressing information about a sensitive or nuisance attribute.

The paper turns this competing-objective problem into a generalized eigendecomposition, giving globally optimal directions within the chosen kernel feature space.

On Characterizing Trade-off in Invariant Representation Learning (Sadeghi, Dehdashtian, Boddeti, 2022)

Why does invariant representation learning need two matrices?

$A_\lambda=(1-\lambda)C_Y-\lambda C_S$, with $C_Y=\Sigma_{YX}^*\Sigma_{YX}$ and $C_S=\Sigma_{SX}^*\Sigma_{SX}$, rewards target dependence and penalizes sensitive dependence.
The positive-definite matrix $B_\gamma=C_X+\gamma I$ defines scale and the constraint $w^\top B_\gamma w=1$.
The leading generalized eigenvectors maximize the trade-off Rayleigh quotient $\frac{w^\top A_\lambda w}{w^\top B_\gamma w}$.

Move the trade-off and watch the representation rotate

Invariance weight $\lambda$ Geometry regularizer $\gamma$

Increasing $\lambda$ rotates $w$ away from the sensitive direction, but can also reduce target alignment: invariance is not free when $Y$ and $S$ are statistically coupled.

Practice problems

Each scenario hides a mathematical structure from this lecture. Identify the structure, justify your choice, and then solve the resulting problem.

1 Choose 2 Solve

Problem 1 Recurrent and state-space models

Will a recurrent representation explode?

A linearized recurrent or state-space block repeatedly applies the same transition. The team wants to predict long-horizon stability without simulating thousands of token steps.

Model evidence

$h_{t+1}=Wh_t$ with $W=\begin{bmatrix}1.1&0\\0&0.7\end{bmatrix}$ and $h_0=(a,b)^\top$.

Describe $h_t$ in closed form and identify which initial states remain bounded.
Determine the asymptotic direction for a generic nonzero $a$.
State the spectral condition that would make every trajectory decay to zero.

Choose
Solve

Problem 2 Graph neural networks

Diagnose oversmoothing from two summary statistics

A two-channel message-passing operator has unknown eigenvalues. Logging records only its trace and determinant. Repeated propagation oversmooths when one mode persists while the other rapidly vanishes.

Model evidence

The operator is $2\times2$, with $\operatorname{tr}(A)=1.2$ and $\det(A)=0.2$.

Recover the two eigenvalues from the logged statistics.
Predict which spectral mode dominates after many layers.
Explain what additional information is required to identify the actual dominant direction in feature coordinates.

Choose
Solve

Problem 3 Self-supervised learning

Which feature survives a consistency update?

A linearized self-supervised consistency update is applied repeatedly to an embedding. Engineers expect one semantic feature to remain invariant while nuisance disagreement decays.

Model evidence

$A=\begin{bmatrix}0.8&0.2\\0.1&0.9\end{bmatrix}$ is the local update on two-dimensional embeddings.

Write “preserved exactly” as an eigenvector equation and determine the eigenvalue it requires.
Recover a preserved direction and one direction whose magnitude contracts.
Predict the limiting representation of $A^tx$ for a generic $x$.

Choose
Solve