Power Series, Taylor Series


How do polynomial models, encrypted inference, and local curvature use Taylor ideas?

Vishnu Boddeti

Example 1

AutoFHE co-designs polynomial CNNs for encrypted inference

AutoFHE searches for a polynomial CNN

AutoFHE converts a non-arithmetic neural network into polynomial networks and returns accuracy-latency trade-offs for encrypted inference
The private input stays encrypted. AutoFHE changes the model—not the privacy promise.

FHE keeps the CNN—but replaces ReLU

CNN under homomorphic encryption with polynomial activations in place of ReLU
Convolution, addition, and rotation are compatible with CKKS. Comparison-based ReLU is not, so each ReLU becomes a polynomial.

Polynomial depth is a cryptographic budget

A polynomial CNN consumes ciphertext levels and uses bootstrapping to restore the remaining level budget
Multiplications consume ciphertext levels. A bootstrap restores the budget, but adds latency and memory.

Composite polynomials approximate the sign function

A composite polynomial closely approximates the sign function outside a narrow region around zero
  • FHE evaluates the smooth polynomial using only additions and multiplications.
  • The difficult region is near zero, where the sign function jumps.
  • ReLU follows from $\mathrm{ReLU}(x)=\tfrac{x+x\,\mathrm{sign}(x)}{2}$.

What problem does AutoFHE solve?

Private image never revealed Encrypt ciphertext Cloud: polynomial CNN conv p₃(x) conv p₅(x) additions and multiplications only mixed degrees save depth; bootstrap only where the budget runs low

AutoFHE overview: private data remain encrypted while the cloud evaluates a polynomial CNN.

From an encrypted-inference constraint to a polynomial approximation

exact call tanh / erf / exp slow polynomial c₀ + c₁x + c₂x² fast curves agree where inputs actually live
  • Encrypted inference reduces a neural network to additions and multiplications, so unsupported nonlinear functions must be represented by polynomials.
  • Taylor’s theorem gives the cleanest mathematical model for a smooth function on a bounded neighborhood: match derivatives near one point, then track how the remainder grows.
  • Production FHE systems also use range-aware minimax or composite polynomials. ReLU has no Taylor series at its kink, so Taylor is motivation—not the complete AutoFHE algorithm.

Taylor series

a f(x) Taylor poly (near a) at a: same value, slope, and curvature
  • If $f$ is smooth at $a$, its Taylor series around $a$ is $f(x) \approx \sum_{k=0}^{\infty} \tfrac{f^{(k)}(a)}{k!}(x-a)^k$.
  • Truncating after finitely many terms gives a polynomial that matches $f$ and its first several derivatives exactly at $a$, and therefore tracks $f$ closely in a neighborhood of $a$
  • the approximation degrades as $x$ moves farther from $a$, or as fewer terms are kept.

Approximate a bounded $\tanh(x)$ kernel and expose the out-of-range error

$\tanh(x)$ Taylor approximation
Extra terms improve the fit near $a=0$, but accuracy can still degrade farther from the expansion point.

Example 2

Shampoo uses the matrix shape of a neural layer

Shampoo keeps one preconditioner per tensor mode

The Shampoo paper illustrates one preconditioner matrix for each dimension of a three-dimensional gradient tensor
  • $G$ is the gradient tensor.
  • $L$, $R$, and $U$ summarize correlations along its three modes.
  • Each matrix is far smaller than a full preconditioner over every tensor entry.

For a matrix, Shampoo learns row and column geometry

Gradient$G_t\in\mathbb{R}^{m\times n}$

One matrix-shaped update
Row statistic$L_t=\epsilon I+\sum_{s\le t}G_sG_s^\top$

$m\times m$
Column statistic$R_t=\epsilon I+\sum_{s\le t}G_s^\top G_s$

$n\times n$
$\widetilde G_t=L_t^{-1/4}\,G_t\,R_t^{-1/4}$

The left factor rescales row directions. The right factor rescales column directions.

Correlations change the update direction

raw gradient path structured preconditioned step
  • A scalar learning rate can only shorten or lengthen the raw gradient.
  • A structured preconditioner can also rotate the update toward the valley.

Newton's step minimizes a local quadratic model

loss curve local quadratic model
current xₙ vertex xₙ₊₁
$x_{n+1}=x_n-\dfrac{f'(x_n)}{f''(x_n)}$

What problem does Shampoo solve?

matrix gradient Gₜ row geometry Lₜ Σ GₛGₛᵀ column geometry Rₜ Σ GₛᵀGₛ rescale both sides Lₜ⁻¹⁄⁴ Gₜ Rₜ⁻¹⁄⁴ structured update two small geometry matrices replace one enormous full preconditioner

Shampoo preserves the row-column structure of a layer instead of flattening all parameters into one enormous vector.

Why does Shampoo need more geometry than one scale per parameter?

start gradient descent: several steps curvature-scaled: one step local minimum
  • Shampoo uses row and column second-moment matrices to capture correlations that a diagonal optimizer discards.
  • It does not form a Hessian. Newton's method is the clean one-dimensional prototype that shows why direction-dependent scaling can improve a badly conditioned update.

Newton's method from the 2nd-order Taylor term

loss f(x) local quadratic qₙ(x) current xₙ Newton step vertex = xₙ₊₁
  • Expand $f$ to second order around $x_n$: $f(x) \approx f(x_n) + f'(x_n)(x-x_n) + \tfrac{1}{2}f''(x_n)(x-x_n)^2$.
  • Minimizing this quadratic approximation in closed form gives the Newton update $x_{n+1} = x_n - \dfrac{f'(x_n)}{f''(x_n)}$.
  • Unlike gradient descent, which takes a small step along the slope, Newton's step lands exactly at the vertex of the local parabola
  • which is why it converges in far fewer steps near a well-behaved minimum, provided $f''(x_n) \neq 0$.

Compare a raw gradient step with an ideal curvature-scaled step

$f(x) = x^4 - 3x^2 + x$ drag to set start gradient path Newton path
Newton is an idealized curvature-scaled update. Practical large-model optimizers approximate or precondition this geometry because an exact Hessian inverse is too expensive.

Example 3

ATLAS searches layer-wise approximations for encrypted transformers

A transformer mixes FHE-friendly and unsupported operations

Attention
$QK^\top$ and $AV$ ✓
softmax needs a replacement
Normalization
sums and products ✓
inverse square root needs a replacement
MLP
matrix multiply ✓
GELU or SiLU needs a replacement
ATLAS replaces each unsupported function with an FHE-compatible approximation.

Each layer gets its own approximation

Layer 1 · GELUdegree 5
range $[-3,3]$
Layer 2 · RMSNorm2 inverse-root iterations
narrow error tolerance
Layer 3 · softmaxdegree 7
range $[-6,6]$
generate candidates → reject unstable choices → keep the accuracy–latency Pareto set
  • A layer's activation range controls where its polynomial must be accurate.
  • Using the largest degree everywhere wastes multiplicative depth.

What problem does ATLAS solve?

Transformer Layer 1 · GELU Layer 2 · softmax Layer 3 · norm ATLAS search degree 3 · 5 · 7 range narrow · wide iterations 1 · 2 · 3 discard unstable Pareto set latency → quality ↑ a different approximation can be best for each layer

ATLAS turns encrypted-transformer approximation into a multi-objective configuration search.

Why does encrypted inference need a different polynomial in each layer?

  • FHE evaluates additions and multiplications efficiently, but not ordinary nonlinear functions.
  • A polynomial is accurate only over its designed interval; deeper layers can have different activation ranges.
  • ATLAS searches per-layer approximation choices instead of assigning one costly degree everywhere.

Increase approximation budget and compare accuracy with encrypted latency

Move the control to expose the approximation-quality and encrypted-cost trade-off.
Higher degree can reduce local approximation error, but multiplicative depth makes encrypted inference rapidly more expensive. The useful solution is layer-specific, not uniformly maximal.

Example 4

Π-Nets learn explicit high-order polynomial functions

A Π-Net raises degree with multiplication nodes

z linearU₁ᵀz × degree 2features × degree 3features G(z) new projectionU₂ᵀz new projectionU₃ᵀz linear feature → product feature → higher-order polynomial

Each product raises the polynomial degree

Degree 1
$x_1=U_1^\top z$

learned linear features
Degree 2
$x_2=(U_2^\top z)\odot x_1+x_1$

pairwise interactions appear
Degree 3
$x_3=(U_3^\top z)\odot x_2+x_2$

third-order interactions appear
$z_1$ and $z_2$ are linear features; multiplying them creates the nonlinear term $z_1z_2$.
  • Depth matters because the blocks multiply—not because affine layers are stacked.
  • The factorized recursion avoids storing every coefficient of a dense multivariate polynomial.

What problem do polynomial neural networks solve?

Multiplicative skip blocks raise polynomial degree while reusing factorized linear projections.

How do multiplicative blocks build a high-order polynomial?

  • A linear projection creates first-order features. Multiplying it by earlier features creates products such as $z_i z_j$ and raises the polynomial degree.
  • Repeating the block produces high-order interactions, while residual additions preserve lower-order terms and help optimization.
  • The learned factorization represents the polynomial without explicitly listing every coefficient $c_\alpha$.

Increase interaction order and compare it with naive coefficient growth

Move the normalized control from low to high degree and see why higher-order models need a factorized architecture.
Higher integer degree adds richer cross-feature interactions, but a naive monomial basis grows combinatorially. Π-Nets factor the polynomial instead.

Practice problems

Each scenario hides a mathematical structure from this lecture. Identify the structure, justify it, and work through the resulting problem.

Problem 1 Efficient attention

Approximate an exponential inside attention

An edge accelerator replaces the exponential in attention with a cheap quadratic whenever logits stay between $-0.2$ and $0.2$. Before trusting it, you need both the polynomial and a guarantee on its worst mistake.

Model evidence

Use a Taylor expansion about $x=0$ through degree two.

  1. Which quadratic Taylor polynomial should the accelerator use?
  2. How large can the remainder be anywhere in the allowed interval?
  3. Does that guarantee keep the error below $0.002$?
Problem 2 Invertible networks

Invert a residual block without a matrix inverse

A residual block must be inverted on hardware that can apply $A$ repeatedly but cannot form a dense inverse. Someone proposes adding $I+A+A^2+\cdots$ and stopping after a manageable number of terms. When does that trick work?

Model evidence

Assume $\|A\|<1$ and consider $S_k=I+A+\cdots+A^k$.

  1. What does $(I-A)S_k$ simplify to?
  2. What operator does $S_k$ approach, and under what condition is the limit valid?
  3. What norm bound controls the error after $k$ terms?
Problem 3 Optimization

Build a cheap local surrogate for logistic loss

A second-order optimizer wants a quick local model of logistic loss near zero margin. A quadratic would reveal not only the current slope but also the local curvature without evaluating another exponential.

Model evidence

$\ell(z)=\log(1+e^{-z})$ and the expansion point is $z=0$.

  1. What is the degree-two Taylor approximation at zero?
  2. What loss does it predict at $z=0.1$?
  3. How do its linear and quadratic terms encode the local gradient and curvature?