Lecture 1: Vector Spaces, Basis, Direct Sum

Vector Spaces, Basis, Direct Sum

Where do embeddings actually live?

Vishnu Boddeti

Example 1

From tokens to contextual vectors

Token: bank

Transformer reads surrounding words

Context-specific hidden state

Concrete exampleThe bank approved the loan and the canoe reached the bank begin with the same token but end with different vectors.

Self-attention makes a token representation depend on the whole sequence, so geometry can encode word sense and use.

BERT: Pre-training of Deep Bidirectional Transformers (Devlin et al., 2018)

How do transformers encode the same token differently in different contexts?

Same token, new state. Self-attention updates “bank” using its surrounding words.
Finance and river contexts yield different hidden-state vectors.
The displacement between vectors describes how the representation changed.

Vector spaces & basis

Embeddings live in $\mathbb{R}^d$, closed under addition and scalar multiplication.
A basis $\{e_1,\dots,e_d\}$ gives each vector a unique expansion $v=\sum_i c_i e_i$.
The model's output values $c_i$ are coordinates in its learned basis.

Change the coordinates and track the embedding

Move $v$ along $e_1$ with coordinate $c_1$ Move $v$ along $e_2$ with coordinate $c_2$

Each coordinate changes one basis component. Together, the components determine one vector.

A basis gives each vector unique coordinates

Every $v\in\mathbb{R}^d$ has one coordinate tuple $(c_1,\ldots,c_d)$ in a fixed basis.
Changing one coordinate moves the vector along one basis direction.
Dimension $d$ equals both the basis size and the embedding's degrees of freedom.

Example 2

How vision and language enter one transformer

Image patches and text tokens

Learned modality projectors

Shared-width token sequence

Concrete exampleA photo of a red bus and the caption red bus are projected to tokens with the same model width before attention mixes them.

The projector aligns dimensions; token type and position retain provenance while cross-attention learns useful correspondences.

Learning Transferable Visual Models From Natural Language Supervision (Radford et al., 2021)

How do multimodal transformers place image patches and text tokens into one residual space?

Text and image encoders initially produce different feature types.
Learned projectors map both streams to the transformer's residual width $d$.
Attention mixes the tokens while position and modality tags preserve provenance.

Typed direct sum before learned fusion

Before learned fusion, keep text features in $U$ and image features in $W$.
The direct sum $U\oplus W$ gives every joint state one unique pair $(u,w)$.
Projectors map each component to width $d$; raw features need not be directly comparable.

Change each component and reconstruct the joint vector

Change the text component $u_1$ on the horizontal axis Change the image component $w_2$ on the vertical axis

The direct sum formalizes provenance before fusion; learned projectors and attention determine how the modalities interact afterward.

Example 3

Steering behavior at inference time

Prompt produces hidden state

Add a learned behavior direction

Generation shifts

Concrete exampleA harmlessness direction is added to activations, increasing refusal behavior without changing model weights.

Activation steering treats behavior as a manipulable direction, but the intervention can also change unrelated capabilities.

Representation Engineering: A Top-Down Approach to AI Transparency (Zou et al., 2023)

Why can activation steering add a concept direction to a hidden state?

hidden state h→+ α concept direction v→steered state h′

A steering vector is a learned direction in activation space.
The coefficient $\alpha$ controls how strongly it changes the hidden state.
Large interventions can strengthen the target behavior while distorting unrelated capabilities.

Sweep the steering coefficient and watch benefit trade against distortion

Move the control to compare the two normalized responses.

Steering coefficient α

A useful steering vector has a region where the target feature rises faster than collateral distortion.

Example 4

What an agent carries between steps

Plan, memory, constraints

Tool call and observation

Updated working state

Concrete exampleA travel agent keeps the user budget separate from a flight-search result before deciding whether to book.

Typed state supports attribution, permission checks, and recovery when a tool result is stale or incorrect.

ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022)

How can an agent combine plan, memory, and tool observations without losing provenance?

typed plan + memory + tools→provenance-preserving state→learned context fusion

Keep the plan, memory, tool results, and user constraints as typed components.
New observations enter with explicit source and timestamp metadata.
Fusion supports coordination; component identity keeps provenance recoverable.

Increase cross-component fusion and compare coordination with provenance

Move the control to compare the two normalized responses.

Agent-state fusion strength

Typed state makes provenance recoverable; fusion helps coordination but should not erase which tool or memory supplied a claim.

Practice problems

Each scenario hides a mathematical structure from this lecture. Identify the structure, justify it, and work through the resulting problem.

Problem 1 Multimodal embeddings

Can two modalities share one latent address?

A vision-language model stores a two-number caption vector beside a three-number image vector. Since the coordinates are tagged rather than mixed, can both pieces always be recovered without ambiguity?

Model evidence

Text features live in $U=\mathbb{R}^2$ and image features in $W=\mathbb{R}^3$. A fused representation is written $(u_1,u_2,w_1,w_2,w_3)$.

What vector space naturally describes the fused representation, and what is an obvious basis for it?
What is its dimension, and why does every fused vector have exactly one text-image decomposition?
If both modalities are mapped into $V$ and then added, what ambiguity appears when $\operatorname{im}(P_U)\cap\operatorname{im}(P_W)$ contains a nonzero vector?

Problem 2 Parameter-efficient tuning

How many LoRA directions are actually new?

You inherit a LoRA adapter advertised as having three trainable directions. A quick look suggests that one direction may be assembled from the other two. Before spending parameters, you also want to know whether the requested update is reachable at all.

Model evidence

$a_1=(1,0,1)$, $a_2=(0,1,1)$, $a_3=(1,1,2)$, and the desired update is $\Delta=(2,1,3)$.

How many genuinely independent update directions are there, and which vectors would you keep as a basis?
Can the desired update $\Delta$ be built from that basis? If so, with which coefficients?
What rank should the adapter actually use?

Problem 3 Representation monitoring

Is a new embedding genuinely out of distribution?

A monitoring dashboard labels any unfamiliar embedding “out of distribution.” But an embedding can be new without pointing in a new geometric direction. Does this particular alert reveal a genuinely new direction or merely a new combination of familiar ones?

Model evidence

Training directions are $v_1=(1,0,1,0)$, $v_2=(0,1,0,1)$, and $v_3=(1,1,1,1)$. The new embedding is $q=(2,-1,2,-1)$.

Which training directions form a basis for the learned subspace?
Does $q$ belong to that subspace? If it does, find its coordinates in your basis.
In this example, what is the difference between “never seen before” and “outside the learned subspace”?