Universal approximation theorem

Remarks

The idea of the proof: if $\sigma$ is a polynomial of degree $d$, every $g \in \mathcal{N}_\sigma$ is a polynomial of degree at most $d$. These polynomials do not form a dense set. Conversely, by smoothing one may assume $\sigma$ is smooth; then difference quotients in the weight show that $t \mapsto t^k \sigma^{(k)}(w t + b)$ is a limit of elements of $\mathcal{N}_\sigma$ in one variable. Since $\sigma$ is not a polynomial, for each $k$ there is a $b$ with $\sigma^{(k)}(b) \ne 0$, so all monomials $t^k$ are limits, and the Weierstrass theorem applies in one variable. The case of $n$ variables reduces to one variable because the ridge functions $h(w \cdot x)$ span a dense set in $C(K)$.

Example: the logistic sigmoid $\sigma(t) = 1 / (1 + e^{-t})$ and the rectifier $\sigma(t) = \max(0, t)$ are not polynomials. So networks with one hidden layer and either activation give uniform approximations of every continuous function on every compact set.

The theorem says nothing about how large $N$ must be for a given accuracy, and it does not say how to find the weights. The sigmoid case was proved by George Cybenko in 1989, and the characterization for continuous $\sigma$ by Moshe Leshno, Vladimir Lin, Allan Pinkus and Shimon Schocken in 1993.

Going deeper

For a student

This is a density theorem in the same family as the Weierstrass and Stone–Weierstrass theorems, with ridge functions $\sigma(w \cdot x + b)$ as building blocks. It justifies neural networks as a flexible class of approximants but gives no rates. Rates depend on the smoothness of the target and on the dimension, as in classical approximation theory. Related results treat deep networks of bounded width and approximation in $L^p$ norms.