{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Definition\n\nLet $\\sigma \\colon \\R \\to \\R$ be a function, called the *activation function*, and let $d, n \\ge 1$ be integers.\nA *shallow neural network* with $n$ neurons is a function $\\R^d \\to \\R$ of the form\n$$x \\mapsto \\sum_{k=1}^{n} c_k \\, \\sigma(w_k \\cdot x + b_k), \\qquad c_k, b_k \\in \\R, \\ w_k \\in \\R^d .$$\nLet $\\Sigma_n(\\sigma)$ be the set of these functions.\nLet $K \\subseteq \\R^d$ be a [[compact set|0TVJZYB6SH]] and let $f \\colon K \\to \\R$ be [[continuous|0X3WGX95FE]].\n*Neural network approximation* of $f$ is its approximation by elements of $\\Sigma_n(\\sigma)$, measured by the error\n$$E_n(f) = \\inf\\set{\\sup_{x \\in K} \\abs{f(x) - g(x)} : g \\in \\Sigma_n(\\sigma)} .$$\n\n## Remarks\n\nOther [[norms|03J5RAKB30]], such as $L^p$ norms on $K$, are also used.\nA *deep* network with $L$ layers is a composition $T_L \\circ \\sigma \\circ T_{L-1} \\circ \\cdots \\circ \\sigma \\circ T_1$ of affine maps $T_j \\colon \\R^{d_{j-1}} \\to \\R^{d_j}$, with $d_0 = d$ and $d_L = 1$, where $\\sigma$ acts on each coordinate; the shallow case is $L = 2$.\nThe set $\\Sigma_n(\\sigma)$ is in general not a linear space, since the sum of two networks with $n$ neurons may need $2n$ neurons.\nFor continuous $\\sigma$, $E_n(f) \\to 0$ as $n \\to \\infty$ for all $K$ and $f$ if and only if $\\sigma$ is not a polynomial.\nExample: for the ReLU $\\sigma(t) = \\max(t, 0)$, $d = 1$, $K = [-1, 1]$ and $f(x) = \\abs{x}$, one has $f = \\sigma(x) + \\sigma(-x) \\in \\Sigma_2(\\sigma)$, so $E_2(f) = 0$.\nNon-example: for $\\sigma(t) = t$, every element of $\\Sigma_n(\\sigma)$ is an affine function, so on $K = [-1, 1]$ the function $x^2$ has $E_n(x^2) = 1/2$ for every $n$ and is not approximated.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "defines": "W7CK84MZFV",
    "kind": "definition",
    "msc": [
      "41A46",
      "68T07",
      "41A30"
    ],
    "name": "Neural network approximation",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Universal_approximation_theorem"
      }
    ],
    "tag": "42AGEAAQ8H",
    "uses": [
      "0X3WGX95FE",
      "0TVJZYB6SH",
      "03J5RAKB30"
    ]
  },
  "kind": "definition",
  "link": "https://tamnd.github.io/math2/node/42AGEAAQ8H/",
  "links": [
    {
      "field": "defines",
      "target": "W7CK84MZFV"
    },
    {
      "field": "uses",
      "target": "0X3WGX95FE"
    },
    {
      "field": "uses",
      "target": "0TVJZYB6SH"
    },
    {
      "field": "uses",
      "target": "03J5RAKB30"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/42AGEAAQ8H.md",
  "msc": [
    "41A46",
    "68T07",
    "41A30"
  ],
  "name": "Neural network approximation",
  "prov": null,
  "role": "",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/42A/42AGEAAQ8H.md",
  "status": null,
  "tag": "42AGEAAQ8H",
  "uri": "m2://42AGEAAQ8H",
  "url": "https://tamnd.github.io/math2/wiki/neural-network-approximation-definition/",
  "writing": {
    "tag": "42AGEAAQ8H",
    "kind": "definition",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}