Neural network approximation

Summary

Neural network approximation studies how well functions can be approximated by the functions that neural networks compute. A shallow network computes a sum $\sum_{k=1}^n c_k \sigma(w_k \cdot x + b_k)$ of ridge functions built from one activation function $\sigma$; a deep network composes several such layers. The universal approximation theorem says that shallow networks with a continuous non-polynomial activation are dense in the continuous functions on every compact set of $\R^d$. Rates of approximation in terms of the number of neurons or parameters are a main topic of nonlinear approximation.