Kernel approximation in reproducing kernel Hilbert spaces

Remarks

The interpolant $s$ is the unique element of minimal norm in $H_K$ among those $g$ with $g(x_i) = y_i$ for all $i$. If $f \in H_K$ and $y_i = f(x_i)$, then $s$ is the orthogonal projection of $f$ onto the span of $K(\cdot, x_1), \dots, K(\cdot, x_n)$. The regularized approximant $s_\lambda$ minimizes $\sum_{i} (g(x_i) - y_i)^2 + \lambda \norm{g}_{H_K}^2$ over $g \in H_K$. For $f \in H_K$, $\abs{f(x) - s(x)} \le P(x) \norm{f}_{H_K}$, where the power function is $P(x)^2 = K(x, x) - k(x)^{T} \mathbf{K}^{-1} k(x)$ with $k(x) = (K(x, x_j))_j$. Example: $K(x, y) = e^{-(x - y)^2}$ on $X = \R$, points $x_1 = 0$, $x_2 = 1$, data $y = (1, 1)$. Here $\mathbf{K} = \begin{pmatrix} 1 & e^{-1} \\ e^{-1} & 1 \end{pmatrix}$ is invertible, and $c_1 = c_2 = 1/(1 + e^{-1})$ solves $\mathbf{K} c = y$. Non-example: the kernel $K(x, y) = xy$ on $\R$ is only positive semidefinite; for the points $1$ and $2$, $\mathbf{K} = \begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}$ is singular, and the data $y = (1, 0)$ cannot be interpolated by any $c\,x$.