Recap: Simple Neural Network

Recap: Multilayer Neural Network

Empirical Risk Minimization

• Framework to design learning algorithms
$$\underset{\theta}{\operatorname{arg min}} \mathbb{E}_{P(x,y)}\left[l\left(f(\mathbf{x}^t;\mathbf{\theta}),\mathbf{y}^t\right)\right] + \lambda\Omega(\mathbf{\theta})$$ $$\underset{\theta}{\operatorname{arg min}} \frac{1}{T} \sum_t l\left(f(\mathbf{x}^t;\mathbf{\theta}),\mathbf{y}^t\right) + \lambda\Omega(\mathbf{\theta})$$
• Learning is cast as an optimization problem
• Loss Function: $l(f(\mathbf{x}^t;\mathbf{\theta}),\mathbf{y}^t)$
• $\Omega(\mathbf{\theta})$ is a regularizer

Log-Likelihood

• NN estimates $f(\mathbf{x})_c=p(y=c|\mathbf{x})$
• we could maximize the probabilities of $y_t$ given $\mathbf{x}_t$ in the training set
• To frame as minimization, we minimize the negative log-likelihood
$$l(f(\mathbf{x}),y) = -\log \sum_c 1_{y=c}f(\mathbf{x})_c = -\log f(\mathbf{x})_y$$
• we take the log to simplify numerical stability and math simplicity
• sometimes referred to as cross-entropy

Loss Functions: Classification

Loss Functions: Regression

Loss Functions: Embeddings

• Cosine Distance: $\frac{\mathbf{x}^T\mathbf{y}}{\|x\|\|y\|}$
• Triplet Loss: $\left(1+d(\mathbf{x}_i,\mathbf{x}_j)-d(\mathbf{x}_i,\mathbf{x}_k)\right)_{+}$
• Mahalanobis Distance: $(\mathbf{x}-\mathbf{y})^T\mathbf{M}(\mathbf{x}-\mathbf{y})$

Cross-Entropy Loss Example

Regularization

• What is regularization?
• the process of constraining the parameter space
• alternatively, penalize certain values of $\mathbf{\theta}$
• Why do we need regularization?
• In Machine Learning regularization $\Leftrightarrow$ generalization
• In Deep Learning regularization $\neq$ generalization

$L_2$ Regularization

• Gradient: $\nabla_{\mathbf{W}^k}\Omega(\mathbf{\theta}) = 2\mathbf{W}^k$
• Only applied on weights, not on biases (weight decay)
• Can be interpreted as having a Gaussian prior over the weights

$L_1$ Regularization

• Gradient: $\nabla_{\mathbf{W}^k}\Omega(\mathbf{\theta}) = sign(\mathbf{W}^k)$
$$sign(W^k)_{i,j} = \begin{cases} 1 & \quad \text{for } W^k_{ij} > 0\\ -1 & \quad \text{for } W^k_{ij} < 0 \end{cases}$$
• Only applied on weights, but not to biases
• Unlike $L_2$, $L_1$ will push certain weights to be exactly zero
• Can be interpreted as having a Laplacian prior over the weights

Regularization Geometry

Looking Through the Bayesian Lens

• Bayes Rule:
$$\begin{eqnarray} p(\mathbf{\theta}|\mathbf{x},\mathbf{y}) &=& \frac{p(\mathbf{y}|\mathbf{x},\mathbf{\theta})p(\mathbf{\theta})}{p(\mathbf{y}|\mathbf{x})} \\ &=& \frac{p(\mathbf{y}|\mathbf{x},\mathbf{\theta})p(\mathbf{\theta})}{\int_{\mathbf{\theta}}p(\mathbf{y}|\mathbf{x},\mathbf{\theta})p(\mathbf{\theta})} \end{eqnarray}$$
• Likelihood:
$$p(\mathbf{y}|\mathbf{x},\mathbf{\theta}) = \frac{1}{Z}e^{-E(\mathbf{\theta},\mathbf{x},\mathbf{y})}$$
• Prior:
$$p(\mathbf{\theta}) = \frac{1}{Z}e^{-E(\mathbf{\theta})}$$

Maximum-Aposteriori-Learning (MAP)

• Example: