Local regression

Remarks

The minimisation is a weighted least squares problem with weights $w_i = K((x_i - x)/h)$. It has a unique solution when at least $p + 1$ distinct points $x_i$ have $w_i > 0$. In LOESS the bandwidth depends on $x$: $h(x)$ is the distance from $x$ to its $\lceil \alpha n \rceil$-th nearest point $x_i$, for a fixed fraction $\alpha \in (0, 1]$ called the span. LOWESS adds rounds of reweighting that lower the weights of points with large residuals.

Example: take $p = 1$ and data that lie on a line, $y_i = a + b x_i$. Then $\beta = (a + bx, b)$ makes every residual $y_i - \beta_0 - \beta_1 (x_i - x)$ zero, so it is a minimiser and $\hat m(x) = a + bx$; local linear regression reproduces lines exactly.

Non-example: ordinary least squares with a global polynomial uses the same weight for every point at every $x$. Its weights do not depend on the distance $\abs{x_i - x}$, so it is not a local fit.

For $p = 0$ the estimate is the weighted average $\sum_i w_i y_i / \sum_i w_i$, the Nadaraya–Watson estimate of kernel regression.