Differentiation on R^n: Gradient Fields and Directional Derivatives

Gradient Fields and Directional Derivatives

From computation graphs to LLM, VLM, and agent gradients

Vishnu Boddeti

Example 1

Reverse-mode autodiff turns a program into reusable local derivatives

Run the program forward

Send adjoints backward

Accumulate every parameter gradient

Concrete exampleA single loss depends on millions of weights, but each operation only needs its saved inputs and an incoming adjoint.

Reverse mode reuses the graph, so one backward sweep produces all parameter gradients at roughly the cost of a few forward evaluations.

Automatic Differentiation in Machine Learning: a Survey (Baydin et al., 2018)

A program becomes a computation graph

A computer program represented as a directed acyclic computation graph, with a forward evaluation and a backward gradient sweep

Forward: compute and save intermediate values. Backward: revisit the same operations in reverse and apply the chain rule.

Baydin et al., “Automatic Differentiation in Machine Learning: a Survey,” JMLR, 2018. Adapted from the CSE 802 autodiff lecture.

Reverse mode is local message passing

A computation graph node receiving gradient messages from its children and sending messages to its parents

Each child sends one contribution to the adjoint of $z$.
The node sums all paths that depend on $z$.
Local vector–Jacobian products carry the result to the parents.

\[ \bar z=\sum_{s\in \mathrm{Child}(z)} \left(\frac{\partial s}{\partial z}\right)^{\!\top}\bar s \]

Baydin et al., 2018. Figure adapted from the CSE 802 reverse-mode autodiff slides.

The gradient answers every local direction question

● Gradient $\nabla f$ ● Direction $u$ ● Projection $D_u f$

$\nabla f(x)$ collects the partial derivatives.
$D_u f(x)=\nabla f(x)^\top u$ is the slope along unit direction $u$.
The projection is largest when $u$ aligns with the gradient.

\[ |D_u f(x)|\le \|\nabla f(x)\|_2 \]

Move the point, then align the direction with the gradient

point $(x,y)$ — drag it $\nabla f$ $u(\theta)$

For $f(x,y)=x^2+2y^2$, the gradient is $(2x,4y)$.

Rotate $u(\theta)$: °

The directional derivative is largest uphill, zero across the contour, and negative downhill.

Example 2

LLM pretraining clips rare gradient spikes

Watch the global gradient norm

Compare it with threshold c

Rescale only an oversized step

Concrete exampleDuring HLAT pretraining, gradient norms were monitored across 1.8 trillion tokens and global-norm clipping used a threshold of 1.0.

A brief spike can create an enormous optimizer step. Clipping keeps the direction but caps the step length.

HLAT: High-quality Large Language Model Pre-trained on AWS (Chen et al., 2024)

LLM pretraining sees brief, extreme gradient spikes

HLAT gradient norm over roughly 1.8 trillion training tokens, showing many sharp spikes

The baseline is small; occasional norms jump by orders of magnitude.

Histogram of the number of steps required for gradient spikes to stabilize

Most sustained spikes disappear after one step.

Clipping is a guardrail for the rare spike—not a replacement for the ordinary gradient.

Chen et al., “HLAT: High-quality Large Language Model Pre-trained on AWS,” 2024, Figs. 1(b) and 2.

Global-norm clipping caps length without changing direction

\[ g_{\mathrm{clip}}=g\min\!\left(1,\frac{c}{\|g\|_2}\right) \]

Increase the raw gradient and watch the update saturate

Move the control to compare the two update rules.

Raw gradient norm

Ordinary gradients pass through. Only the oversized step is rescaled.

Example 3

Input gradients can steer VLM outputs

Choose a target score

Differentiate with respect to pixels

Follow the most sensitive direction

Concrete exampleIn Chain of Attack, learned pixel perturbations steer a bird image caption toward a baseball target.

The same derivative can explain local sensitivity or construct a first-order adversarial attack.

Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks (Xie et al., 2025)

A real gradient attack changes a VLM caption step by step

Chain of Attack paper example: learned pixel perturbations progressively steer a bird image caption toward a baseball target caption

Each noisy square is a learned pixel perturbation. The bird still looks unchanged, while its caption moves toward the baseball target.

Xie et al., “Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks,” CVPR 2025, Fig. 3.

Increase the pixel step and test the local prediction

Move the control away from the linearization point.

Pixel perturbation size $\varepsilon$

Near zero, the gradient is predictive. Farther away, curvature and discrete preprocessing matter.

Example 4

Agent training must assign credit across tool calls

Record the full trajectory

Score the final outcome

Assign credit to earlier decisions

Concrete exampleAn agent searches, writes SQL, runs code, and only then receives a success score. Which action deserves the update?

Tool calls are discrete and the reward is delayed, so ordinary backpropagation cannot simply flow through the whole trace.

Agent Lightning: Train ANY AI Agents with Reinforcement Learning (Luo et al., 2025)

One reward must travel back through an agent trajectory

Gradients train the model inside a step. Policy gradients or another credit-assignment method connect discrete tool calls across steps.

Luo et al., “Agent Lightning: Train ANY AI Agents with Reinforcement Learning,” 2025.

Lengthen the trajectory and watch credit become ambiguous

Move the control to add more tool calls before the reward.

Agent trajectory length

A final score says whether the run worked. Credit assignment estimates which earlier actions should change.

Practice problems

Each scenario hides a mathematical structure from this lecture. Identify the structure, justify it, and work through the resulting problem.

Problem 1 Adversarial examples

Which perturbation increases loss fastest?

A red team cannot perturb an input arbitrarily; the hardware allows movement only along one physical sensor direction. Before launching an attack, can you predict which way along that line makes the loss rise fastest?

Model evidence

$L(x_1,x_2)=x_1^2+2x_1x_2+3x_2$, the current input is $x=(1,-1)$, and the feasible unit direction is $v=(1,1)/\sqrt2$.

What first-order quantity measures the loss change along the feasible direction?
Should the attacker move along $v$ or $-v$ initially?
How does this constrained attack compare with unconstrained steepest ascent?

Problem 2 Backpropagation

Differentiate through an encoder and prediction head

A model first builds nonlinear features and then scores them. Instead of differentiating the whole expression from scratch, you have the local derivative of each stage. How do the two pieces snap together?

Model evidence

$h(x_1,x_2)=(x_1+x_2,x_1x_2)$, $g(z_1,z_2)=z_1^2+z_2$, and $x=(1,2)$.

What are the encoder's local linear map and the head's local sensitivity?
What derivative do they give for the full model $g\circ h$?
Does direct differentiation of one output coordinate agree?

Problem 3 Interpretability

Separate two feature effects in a response surface

A response surface predicts model error from data imbalance and label noise. You want to nudge both experimental settings and estimate the effect before running another expensive training job.

Model evidence

$E(a,b)=a^2e^b+ab$, evaluated at $(a,b)=(1,0)$.

What is the isolated local effect of each variable?
What linear approximation describes a simultaneous change $(\Delta a,\Delta b)$?
What error change does it predict for $(0.02,-0.01)$?