Automatic Differentiation

CSE 891: Deep Learning

Vishnu Boddeti

Monday September 21, 2020

Automatic Differentiation Modes

• Consider:
$$\mathcal{L} = F(G(H(x)))$$
• Forward-Mode Differentiation:
$$ \begin{eqnarray} \frac{\partial \mathcal{L}}{\partial x} =\left(\frac{\partial \mathcal{L}}{\partial F}\left(\frac{\partial F}{\partial G}\left(\frac{\partial G}{\partial H}\frac{\partial H}{\partial x}\right)\right)\right) \nonumber \end{eqnarray} $$
• Reverse-Mode Differentiation:
$$ \begin{eqnarray} \frac{\partial \mathcal{L}}{\partial x} = \left(\left(\left(\frac{\partial \mathcal{L}}{\partial F}\frac{\partial F}{\partial G}\right)\frac{\partial G}{\partial H}\right)\frac{\partial H}{\partial x}\right) \nonumber \end{eqnarray} $$