Anki Deck Changes

To ensure continuous differentiability at \(r = \pm\delta\):

• The slopes must match: \(L_{\text{square}}' = r\), so the abs part gets slope \(\pm\delta\).
• The y-values must match: square gives \(\frac{1}{2}\delta^2\) but \(\delta|\pm\delta| = \delta^2\), so we subtract \(\frac{1}{2}\delta^2\).

e.g. mean squared error or mean Huber loss

We want \(\hat{\boldsymbol{w}}\) minimizing \(\|\boldsymbol{y} - \boldsymbol{X}\boldsymbol{w}\|_2^2\).

Set gradient to zero:
\(\nabla_{\boldsymbol{w}} \|\boldsymbol{y} - \boldsymbol{X}\hat{\boldsymbol{w}}\|_2^2 = 2\boldsymbol{X}^\top(\boldsymbol{X}\hat{\boldsymbol{w}} - \boldsymbol{y}) = 0\)
\(\iff \boldsymbol{X}^\top \boldsymbol{X} \hat{\boldsymbol{w}} = \boldsymbol{X}^\top \boldsymbol{y}\)

Objective: \(\|\boldsymbol{y} - \boldsymbol{X}\boldsymbol{w}\|_2^2\)

Optimal weights: \(\hat{\boldsymbol{w}} = \arg\min_{\boldsymbol{w} \in \mathbb{R}^d} \|\boldsymbol{y} - \boldsymbol{X}\boldsymbol{w}\|_2^2\)

A function \(\ell(r)\) that characterizes how "bad" a prediction is, where \(r = y - \hat{y}\) is the residual.

At a minimum the gradient must be zero. The factor 2 cancels out, leaving the normal equation.

Very sensitive to outliers since the residual is squared.

\[\ell_{\text{huber}}(r) = \begin{cases} \frac{1}{2} r^2 & |r| \le \delta, \\ \delta |r| - \frac{1}{2} \delta^2 & |r| > \delta. \end{cases}\]
Uses square loss for \([-\delta, \delta]\) and absolute loss outside.

Like absolute loss, but with two different slopes on each side of the y-axis.