Tikhonov regularization

Remarks

The operator $K^* K + \alpha I$ satisfies $\ip{(K^* K + \alpha I) x}{x} \ge \alpha \norm{x}^2$, so it is invertible with inverse of norm at most $1/\alpha$; the minimizer is characterized by the normal equation $(K^* K + \alpha I) x = K^* y$. So $x_\alpha$ exists, is unique, and depends continuously on $y$.

If $y$ lies in the range of $K$, then $x_\alpha \to K^\dagger y$ as $\alpha \to 0$, where $K^\dagger y$ is the solution of smallest norm, given by the Moore–Penrose inverse. For noisy data $y^\delta$ with $\norm{y^\delta - y} \le \delta$, the choice $\alpha(\delta) \to 0$ with $\delta^2 / \alpha(\delta) \to 0$ gives $x_{\alpha(\delta)}^\delta \to K^\dagger y$ as $\delta \to 0$. For compact $K$ with singular system $(\sigma_n, v_n, u_n)$, $x_\alpha = \sum_n \frac{\sigma_n}{\sigma_n^2 + \alpha} \ip{y}{u_n} v_n$, so the unbounded factors $1/\sigma_n$ of Picard's criterion are replaced by factors bounded by $1/(2\sqrt{\alpha})$.

Example: $X = Y = \R$ and $K x = k x$ with $k \ne 0$. Then $J_\alpha(x) = (k x - y)^2 + \alpha x^2$ has derivative $2k(k x - y) + 2\alpha x$, which vanishes at $x_\alpha = k y / (k^2 + \alpha)$, in agreement with the formula.

Non-example: the case $\alpha = 0$ is excluded. For $K = 0$ on $X = Y = \R$, every $x$ minimizes $\norm{K x - y}^2$, so there is no unique minimizer.

In finite dimensions, with $K$ a data matrix, $x_\alpha$ is the ridge regression estimator.