Huber loss

Remarks

The two pieces agree at $\abs{a} = \delta$, where both equal $\delta^2/2$, and so do their derivatives. So $L_\delta$ is convex and continuously differentiable, with derivative $\psi_\delta(a) = \max(-\delta, \min(\delta, a))$, the residual clipped to $[-\delta, \delta]$. Huber's M-estimator of location minimises $\sum_{i=1}^{n} L_\delta(x_i - \mu)$ over $\mu \in \R$, or equivalently solves $\sum_i \psi_\delta(x_i - \mu) = 0$. As $\delta \to \infty$ it tends to the mean, and as $\delta \to 0$ it tends to the median.

Example: with $\delta = 1$, $L_1(0.5) = 0.125$, as for squared loss, and $L_1(3) = 3 - 0.5 = 2.5$, much less than $3^2/2 = 4.5$.

Non-example: the function $a \mapsto \abs{a}$ grows linearly but is not differentiable at $0$. It is not quadratic on any interval around $0$, so it is not a Huber loss for any $\delta$.