Linear Maps, Matrices, Invertibility


What is a neural net layer actually doing to your data?

Vishnu Boddeti

Example 1

Transformer projections are linear maps

What does a transformer projection do to a token state?

residual state h WQ query feature q q = WQh — linear before attention
  • An attention head first computes $q=W_Qh$. This projection preserves linear combinations; normalization and softmax attention happen separately and are nonlinear.

Linear maps & matrices

input space x₁ x₂ W y = Wx output space Wx₁ Wx₂ W = w₁₁ w₁₂ w₂₁ w₂₂
  • A layer computes $y = Wx$, a linear map: it sends lines through the origin to lines through the origin
  • and preserves addition and scaling, $W(\alpha x_1 + \beta x_2) = \alpha Wx_1 + \beta Wx_2$.
  • Every such map on $\mathbb{R}^d$ is represented by a matrix $W$ once you fix a basis.

Change $W$ and compare the input with its image

Every point moves by the same linear rule. Watch how $W$ changes direction, scale, and area.

Example 2

Invertible channel mixing preserves information

Why must Glow keep its channel-mixing matrix invertible?

channels x mixed y 1 × 1 convolution W inverse W⁻¹ the same channel map is applied at every pixel
  • Glow mixes the channel vector at every spatial location using the same square matrix $W$.
  • The reverse pass requires $x=W^{-1}y$, so $W$ must be nonsingular.
  • The determinant measures volume scaling and exposes when channel information collapses.

Invertibility

det(W) ≠ 0 full 2-D area preserved det(W) = 0 collapsed to a line — info lost only invertible when area survives the map
  • A linear map $y = Wx$ is invertible iff $\det(W) \neq 0$.
  • Only then does $W^{-1}$ exist, so $x$ can be recovered exactly from $y$ via $x = W^{-1}y$.
  • If $\det(W) = 0$, $W$ collapses space onto a lower-dimensional subspace and information is lost forever.

Change $W$, then test whether the points can be recovered

Recovery works only when $det(W) \neq 0$, because no input direction has been collapsed.

Example 3

Transformer residual streams preserve an identity path

Why does a transformer block retain its previous residual state?

  • The shortcut applies the identity map $I(h)=h$.
  • The attention or MLP branch contributes a learned update $F(h)$.
  • The whole block $h+F(h)$ is generally nonlinear even though its shortcut is linear.

Scale the learned branch while the identity path remains available

Move the control to compare the two normalized responses.
The identity signal remains available while the nonlinear branch learns a correction around it.

Example 4

A linear tool router can collapse distinct requests

When do different requests receive identical tool scores?

request h (1.2, 0.4, 0.2) request h + v v ∈ ker(W) W W scores Wh (1.4, 0.6) scores W(h + v) (1.4, 0.6) Wv = 0, so the router cannot distinguish the pair
  • A router maps a request representation to tool logits using $r=Wh$.
  • If $v\in\ker(W)$, then $W(h+v)=Wh$ exactly.
  • Collision-pair and low-margin evaluations expose distinctions hidden by average routing accuracy.

Move a request along a router-blind direction

request A request B = h + tv tool scores for both requests search 1.4 calendar 0.6
The requests separate in embedding space while their tool scores remain identical. This is a router collision, not a rollback problem.

Practice problems

Each scenario hides a mathematical structure from this lecture. Identify the structure, justify your choice, and then solve the resulting problem.

1 Choose 2 Solve
Problem 1 Representation learning

Can a linear encoder avoid feature collisions?

A linear encoder compresses three sensor readings into two latent coordinates. The product team claims that the original sensor vector can still be recovered uniquely from every embedding.

Model evidence

$T(x)=Ax$ with $A=\begin{bmatrix}1&0&1\\0&1&1\end{bmatrix}$.

  1. Test the uniqueness claim by finding all sensor changes that leave the embedding unchanged.
  2. Construct two distinct sensor vectors that collide, if possible.
  3. Explain whether any decoder can recover every input exactly from this embedding.
  1. Choose
  2. Solve
Problem 2 Model calibration

Reverse a learned calibration layer

A deployment pipeline applies a learned linear calibration to three logits. A debugging tool sees only calibrated logits and must reconstruct the original vector.

Model evidence

$y=Ax$ where $A=\begin{bmatrix}1&1&0\\0&1&1\\1&0&1\end{bmatrix}$ and the observed output is $y=(3,2,4)^\top$.

  1. Decide whether reconstruction is well-defined for every possible output produced by the layer.
  2. Recover the original logits for the observed output.
  3. Give a matrix expression for the debugging tool and identify the property that makes it valid.
  1. Choose
  2. Solve
Problem 3 Input pipelines

Which preprocessing rule can be absorbed into a layer?

Engineers want to fuse preprocessing into the first matrix multiplication. This is valid only when preprocessing respects mixtures and rescaling of examples.

Model evidence

Compare $P_1(x)=2x$, $P_2(x)=x-\mu$ for fixed nonzero $\mu$, and coordinate clipping $P_3(x)_i=\max(-1,\min(1,x_i))$.

  1. Determine which rules can be represented by a matrix with no added bias.
  2. For each rejected rule, produce a concrete violation using either addition or scalar multiplication.
  3. Explain which rejected rule becomes affine and what architectural change would implement it.
  1. Choose
  2. Solve