Lecture 1: Vector Spaces, Basis, Direct Sum

Vector Spaces, Basis, Direct Sum

Where do embeddings actually live?

Vishnu Boddeti

Example 1

From tokens to contextual vectors

Token: bank

Transformer reads surrounding words

Context-specific hidden state

Concrete exampleThe bank approved the loan and the canoe reached the bank begin with the same token but end with different vectors.

Self-attention makes a token representation depend on the whole sequence, so geometry can encode word sense and use.

BERT: Pre-training of Deep Bidirectional Transformers (Devlin et al., 2018)

How do transformers encode the same token differently in different contexts?

Same token, new state. Self-attention updates “bank” using its surrounding words.
Finance and river contexts yield different hidden-state vectors.
The displacement between vectors describes how the representation changed.

Vector spaces & basis

Embeddings live in $\mathbb{R}^d$, closed under addition and scalar multiplication.
A basis $\{e_1,\dots,e_d\}$ gives each vector a unique expansion $v=\sum_i c_i e_i$.
The model's output values $c_i$ are coordinates in its learned basis.

Change the coordinates and track the embedding

Move $v$ along $e_1$ with coordinate $c_1$ Move $v$ along $e_2$ with coordinate $c_2$

Each coordinate changes one basis component. Together, the components determine one vector.

A basis gives each vector unique coordinates

Every $v\in\mathbb{R}^d$ has one coordinate tuple $(c_1,\ldots,c_d)$ in a fixed basis.
Changing one coordinate moves the vector along one basis direction.
Dimension $d$ equals both the basis size and the embedding's degrees of freedom.

Example 2

How vision and language enter one transformer

Image patches and text tokens

Learned modality projectors

Shared-width token sequence

Concrete exampleA photo of a red bus and the caption red bus are projected to tokens with the same model width before attention mixes them.

The projector aligns dimensions; token type and position retain provenance while cross-attention learns useful correspondences.

Learning Transferable Visual Models From Natural Language Supervision (Radford et al., 2021)

How do multimodal transformers place image patches and text tokens into one residual space?

Text and image encoders initially produce different feature types.
Learned projectors map both streams to the transformer's residual width $d$.
Attention mixes the tokens while position and modality tags preserve provenance.

Typed direct sum before learned fusion

Before learned fusion, keep text features in $U$ and image features in $W$.
The direct sum $U\oplus W$ gives every joint state one unique pair $(u,w)$.
Projectors map each component to width $d$; raw features need not be directly comparable.

Change each component and reconstruct the joint vector

Change the text component $u_1$ on the horizontal axis Change the image component $w_2$ on the vertical axis

The direct sum formalizes provenance before fusion; learned projectors and attention determine how the modalities interact afterward.

Example 3

Steering behavior at inference time

Prompt produces hidden state

Add a learned behavior direction

Generation shifts

Concrete exampleA harmlessness direction is added to activations, increasing refusal behavior without changing model weights.

Activation steering treats behavior as a manipulable direction, but the intervention can also change unrelated capabilities.

Representation Engineering: A Top-Down Approach to AI Transparency (Zou et al., 2023)

Why can activation steering add a concept direction to a hidden state?

hidden state h→+ α concept direction v→steered state h′

A steering vector is a learned direction in activation space.
The coefficient $\alpha$ controls how strongly it changes the hidden state.
Large interventions can strengthen the target behavior while distorting unrelated capabilities.

Sweep the steering coefficient and watch benefit trade against distortion

Move the control to compare the two normalized responses.

Steering coefficient α

A useful steering vector has a region where the target feature rises faster than collateral distortion.

Example 4

What an agent carries between steps

Plan, memory, constraints

Tool call and observation

Updated working state

Concrete exampleA travel agent keeps the user budget separate from a flight-search result before deciding whether to book.

Typed state supports attribution, permission checks, and recovery when a tool result is stale or incorrect.

ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022)

How can an agent combine plan, memory, and tool observations without losing provenance?

typed plan + memory + tools→provenance-preserving state→learned context fusion

Keep the plan, memory, tool results, and user constraints as typed components.
New observations enter with explicit source and timestamp metadata.
Fusion supports coordination; component identity keeps provenance recoverable.

Increase cross-component fusion and compare coordination with provenance

Move the control to compare the two normalized responses.

Agent-state fusion strength

Typed state makes provenance recoverable; fusion helps coordination but should not erase which tool or memory supplied a claim.

Practice problems

Each scenario hides a mathematical structure from this lecture. Identify the structure, justify your choice, and then solve the resulting problem.

1 Choose 2 Solve

Problem 1 Multimodal embeddings

Can two modalities share one latent address?

A vision-language model emits a 2-coordinate text representation and a 3-coordinate image representation. The fusion layer stores both without mixing their coordinates so that either modality can be recovered exactly from the fused vector.

Model evidence

Text features live in $U=\mathbb{R}^2$ and image features in $W=\mathbb{R}^3$. A fused representation is written $(u_1,u_2,w_1,w_2,w_3)$.

Describe the mathematical space containing every fused representation and construct a natural basis for it.
Determine its dimension and prove that the text and image components of a fused vector are unique.
Now suppose encoders $P_U:U\to V$ and $P_W:W\to V$ project both modalities into a common latent space $V$, and fusion adds their projected vectors. Explain what architectural failure occurs if $\operatorname{im}(P_U)\cap\operatorname{im}(P_W)$ contains a nonzero vector.

Choose
Solve

Problem 2 Parameter-efficient tuning

How many LoRA directions are actually new?

A small adapter proposes three update directions for a frozen layer. Before allocating three trainable coefficients, the team wants to know whether one direction is redundant and whether a requested update can be represented.

Model evidence

$a_1=(1,0,1)$, $a_2=(0,1,1)$, $a_3=(1,1,2)$, and the desired update is $\Delta=(2,1,3)$.

Find the number of independent trainable directions and select a basis for the adapter subspace.
Express $\Delta$ in that basis or prove that it cannot be represented.
State what the answer implies about the adapter rank the implementation should use.

Choose
Solve

Problem 3 Representation monitoring

Is a new embedding genuinely out of distribution?

A monitor flags an embedding as novel whenever it lies outside the subspace generated by embeddings from the training population. The alert should depend on geometry, not on whether the vector appeared verbatim in the data.

Model evidence

Training directions are $v_1=(1,0,1,0)$, $v_2=(0,1,0,1)$, and $v_3=(1,1,1,1)$. The new embedding is $q=(2,-1,2,-1)$.

Reduce the training directions to a basis for their span.
Decide whether the monitor should flag $q$, showing the representation coefficients when they exist.
Distinguish “not seen before” from “outside the learned subspace” in this example.

Choose
Solve