Lecture 7: Inner Products and Orthogonal Matrices

Inner Products and Orthogonal Matrices

How does attention decide what to "attend" to?

Vishnu Boddeti

Example 1

Attention performs content-based information routing

Query from current token

Compare with keys

Weighted sum of values

Concrete exampleIn The animal did not cross the street because it was tired, attention can connect it to animal.

Dot products become normalized weights that decide which token information reaches the next representation.

Attention Is All You Need (Vaswani et al., 2017)

How does attention decide what to "attend" to?

A transformer computes a score between a query vector and a key vector for every token pair, then uses those scores to decide how much each token should "attend" to every other token.
What operation turns two vectors into a single number that captures how well they align?

Attention score as an inner product

The attention score between a query $q$ and a key $k$ is $\propto \langle q, k \rangle = q^\top k = \sum_i q_i k_i$.
Vectors pointing in the same direction (parallel, small angle) score high; vectors at right angles (orthogonal) score zero; vectors pointing opposite ways score negative.
The dot product is exactly $\|q\|\|k\|\cos\theta$, so it measures alignment scaled by magnitude.

Drag $q$ and $k$ to compare alignment with attention score

query $q$ key $k$

Drag either arrow tip and watch the angle and score change.

Signed attention score $\langle q,k\rangle$

Parallel vectors have a positive score, orthogonal vectors score zero, and opposite vectors have a negative score.

Example 2

LatentCompass isolates attributes in orthogonal spaces

Collect concept exemplars

Construct orthogonal attribute directions

Steer without attribute leakage

Concrete exampleA text-to-image model can change style or safety-related attributes while suppressing unintended movement along other learned concept directions.

Orthogonal coordinates turn inner-product geometry into a practical control mechanism: one steering direction should have zero component along protected directions.

LatentCompass: T2I Diffusion Steering via Orthogonal Attribute Spaces (Sepehr Dehdashtian, Mashrur M. Morshed, Milad Afshari, Jacob H. Seidman, and Vishnu Boddeti, 2026)

How can steering one generative attribute avoid changing another?

LatentCompass constructs low-dimensional concept coordinates from exemplars.
Orthogonal directions separate intended steering from protected attributes.
A generic matrix can stretch or mix those directions and reintroduce leakage.

Orthogonal matrices preserve norms and angles

A matrix $W$ is orthogonal iff $W^\top W = I$, equivalently its columns are unit vectors that are mutually perpendicular.
Orthogonal maps preserve inner products, $\langle Wx, Wy\rangle = \langle x,y\rangle$, so they preserve norms ($\|Wx\| = \|x\|$) and angles
no vector gets stretched, shrunk, or sheared.
Geometrically, the unit circle maps to itself exactly when $W$ is orthogonal; otherwise it maps to an ellipse.

Change $W$ and test whether concept geometry is preserved

W₁₁ (top-left) W₁₂ (top-right) W₂₁ (bottom-left) W₂₂ (bottom-right)

An orthogonal steering map preserves lengths and angles between concept directions. A generic matrix can shear the space and couple attributes that were separated.

Example 3

Dense retrieval turns meaning into nearest-neighbor search

Encode query and documents

Compare vector angles

Return nearest items

Concrete exampleWhere was Marie Curie born retrieves a passage about Warsaw even without exact word overlap.

Cosine similarity ignores vector magnitude and compares semantic direction in the learned space.

Dense Passage Retrieval for Open-Domain Question Answering (Karpukhin et al., 2020)

Why does cosine similarity retrieve semantically related embeddings?

query embedding q→normalized inner product→ranked item embedding

Retrieval systems commonly normalize query and item embeddings, turning the inner product into cosine similarity.
Direction then carries semantic agreement while vector magnitude no longer dominates ranking.

Rotate two unit embeddings and watch similarity fall with angle

Move the control to compare the two normalized responses.

Angle between normalized embeddings

After normalization, alignment—not raw magnitude—determines which item receives the larger similarity score.

Example 4

Agent memory retrieval needs authorization as well as relevance

Query plus user identity

Similarity and ACL filters

Permitted memory context

Concrete exampleA payroll question may be similar to many records, but the agent can retrieve only documents the current employee may access.

Semantic closeness is not permission; retrieval must enforce access control before memories enter the model context.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)

How should an agent retrieve relevant memory without exposing unrelated records?

query embedding + permissions→filtered similarity search→authorized memories

A memory retriever ranks normalized embeddings by inner product, then applies metadata and permission filters.
Raising the similarity threshold improves precision but can hide useful context; isolation filters must remain hard constraints rather than soft similarity preferences.

Raise the retrieval threshold and compare precision with memory recall

Move the control to compare the two normalized responses.

Memory-similarity threshold

Semantic similarity decides relevance only after authorization determines which records are eligible.

Practice problems

Each scenario hides a mathematical structure from this lecture. Identify the structure, justify it, and work through the resulting problem.

Problem 1 Representation debiasing

Remove a nuisance direction from an embedding

A probe finds a unit direction $b$ associated with an unwanted attribute in an embedding. You want to remove that attribute before retrieval while disturbing the original vector as little as possible.

Model evidence

$x=(2,1,3)^\top$ and $b=(1,1,0)^\top/\sqrt2$.

What part of the embedding points along $b$, and what remains after removing it?
How can you verify that the cleaned embedding has no component along $b$?
Why is this the smallest possible Euclidean change satisfying the constraint?

Problem 2 Linear probing

Fit a linear head using only a subspace

A frozen representation lets a linear head predict only combinations of two feature templates. The desired target sits outside that plane, so exact fitting is impossible. What is the closest prediction the head can make?

Model evidence

$u_1=(1,0,1)^\top$, $u_2=(0,1,1)^\top$, and target $y=(2,1,0)^\top$.

Which coefficients give the closest allowable prediction?
Is the remaining error orthogonal to both allowable directions?
Why does that orthogonality certify a least-squares optimum?

Problem 3 Embedding alignment

Does a latent rotation preserve retrieval scores?

A privacy layer rotates every query and database embedding before retrieval. The results should look scrambled in coordinates but produce exactly the same neighbors. What must be true of the rotation matrix?

Model evidence

The transformed embedding is $x'=Qx$ for a square real matrix $Q$.

Which matrix identity guarantees that all pairwise dot products are preserved?
Why does the same identity preserve every Euclidean distance?
Does $Q=\frac1{\sqrt2}\begin{bmatrix}1&-1\\1&1\end{bmatrix}$ pass the test?