Neural network approximation

Remarks

Other norms, such as $L^p$ norms on $K$, are also used. A deep network with $L$ layers is a composition $T_L \circ \sigma \circ T_{L-1} \circ \cdots \circ \sigma \circ T_1$ of affine maps $T_j \colon \R^{d_{j-1}} \to \R^{d_j}$, with $d_0 = d$ and $d_L = 1$, where $\sigma$ acts on each coordinate; the shallow case is $L = 2$. The set $\Sigma_n(\sigma)$ is in general not a linear space, since the sum of two networks with $n$ neurons may need $2n$ neurons. For continuous $\sigma$, $E_n(f) \to 0$ as $n \to \infty$ for all $K$ and $f$ if and only if $\sigma$ is not a polynomial. Example: for the ReLU $\sigma(t) = \max(t, 0)$, $d = 1$, $K = [-1, 1]$ and $f(x) = \abs{x}$, one has $f = \sigma(x) + \sigma(-x) \in \Sigma_2(\sigma)$, so $E_2(f) = 0$. Non-example: for $\sigma(t) = t$, every element of $\Sigma_n(\sigma)$ is an affine function, so on $K = [-1, 1]$ the function $x^2$ has $E_n(x^2) = 1/2$ for every $n$ and is not approximated.