Least squares approximation

Remarks

The least squares approximation exists, is unique, and is the orthogonal projection of $f$ onto $V$. It is characterised by $\ip{f - v^*}{v} = 0$ for all $v \in V$. If $\varphi_0, \dots, \varphi_n$ is an orthogonal basis of $V$, then $v^* = \sum_{k=0}^{n} \frac{\ip{f}{\varphi_k}}{\ip{\varphi_k}{\varphi_k}} \varphi_k$. For a general basis, the coefficients solve the normal equations, whose matrix is the Gram matrix $(\ip{\varphi_j}{\varphi_k})$.

Example: $f(x) = x^2$ on $[-1, 1]$ with $w = 1$ and $V$ the polynomials of degree at most $1$. With the orthogonal basis $1, x$, one gets $\ip{f}{1} / \ip{1}{1} = (2/3)/2 = 1/3$ and $\ip{f}{x} = 0$, so $v^* = 1/3$. Check: $\int_{-1}^{1} (x^2 - 1/3) \, dx = 0$ and $\int_{-1}^{1} (x^2 - 1/3) x \, dx = 0$.

Non-example: the uniform best line for $x^2$ on $[-1, 1]$ is $1/2$, not $1/3$. It minimises the largest error, not the integral of the squared error, so it fails the defining condition.

A discrete version replaces the integral by a weighted sum $\sum_i w_i \abs{f(x_i) - v(x_i)}^2$ over data points; this is curve fitting.