{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Statement\n\nLet $\\sigma \\colon \\R \\to \\R$ be [[continuous|0X3WGX95FE]], and let $n \\ge 1$.\nLet $\\mathcal{N}_\\sigma$ be the set of all functions $g \\colon \\R^n \\to \\R$ of the form\n$$g(x) = \\sum_{i=1}^{N} c_i \\, \\sigma(w_i \\cdot x + b_i).$$\nHere $N \\ge 1$, $c_i, b_i \\in \\R$ and $w_i \\in \\R^n$.\nThese are the functions computed by a neural network with one hidden layer and activation $\\sigma$.\nThen the following are equivalent.\n\n1. For every [[compact set|0R6EWFKRCF]] $K \\subseteq \\R^n$, the restrictions of the functions in $\\mathcal{N}_\\sigma$ to $K$ are [[dense|0BSJSX7ME2]] in $C(K)$ with the supremum norm.\n2. $\\sigma$ is not a polynomial.\n\n## Remarks\n\nThe idea of the proof: if $\\sigma$ is a polynomial of degree $d$, every $g \\in \\mathcal{N}_\\sigma$ is a polynomial of degree at most $d$.\nThese polynomials do not form a dense set.\nConversely, by smoothing one may assume $\\sigma$ is smooth; then difference quotients in the weight show that $t \\mapsto t^k \\sigma^{(k)}(w t + b)$ is a limit of elements of $\\mathcal{N}_\\sigma$ in one variable.\nSince $\\sigma$ is not a polynomial, for each $k$ there is a $b$ with $\\sigma^{(k)}(b) \\ne 0$, so all monomials $t^k$ are limits, and the Weierstrass theorem applies in one variable.\nThe case of $n$ variables reduces to one variable because the ridge functions $h(w \\cdot x)$ span a dense set in $C(K)$.\n\nExample: the logistic sigmoid $\\sigma(t) = 1 / (1 + e^{-t})$ and the rectifier $\\sigma(t) = \\max(0, t)$ are not polynomials.\nSo networks with one hidden layer and either activation give [[uniform approximations|XAARVNKXF1]] of every continuous function on every compact set.\n\nThe theorem says nothing about how large $N$ must be for a given accuracy, and it does not say how to find the weights.\nThe sigmoid case was proved by George Cybenko in 1989, and the characterization for continuous $\\sigma$ by Moshe Leshno, Vladimir Lin, Allan Pinkus and Shimon Schocken in 1993.\n\n## Explanations\n\n### L1 human\n\nA neural network with one hidden layer adds up many copies of one fixed function $\\sigma$, each stretched, shifted and pointed in some direction.\nThe theorem says that such sums can imitate any continuous function on a bounded closed region as closely as you want.\nThe only condition on $\\sigma$ is that it is not a polynomial.\n\n### L2 human\n\nThis is a density theorem in the same family as the Weierstrass and Stone–Weierstrass theorems, with ridge functions $\\sigma(w \\cdot x + b)$ as building blocks.\nIt justifies neural networks as a flexible class of approximants but gives no rates.\nRates depend on the smoothness of the target and on the dimension, as in classical approximation theory.\nRelated results treat deep networks of bounded width and approximation in $L^p$ norms.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "about": [
      "XAARVNKXF1"
    ],
    "kind": "statement",
    "msc": [
      "41A30",
      "68T07",
      "41A63"
    ],
    "name": "Universal approximation theorem",
    "role": "theorem",
    "same_as": {
      "wikidata": "Q7894110",
      "wikipedia": "Universal approximation theorem"
    },
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Universal_approximation_theorem"
      }
    ],
    "tag": "HSRNND3TG5",
    "uses": [
      "XAARVNKXF1",
      "0BSJSX7ME2",
      "0R6EWFKRCF",
      "0X3WGX95FE"
    ]
  },
  "kind": "statement",
  "link": "https://tamnd.github.io/math2/node/HSRNND3TG5/",
  "links": [
    {
      "field": "about",
      "target": "XAARVNKXF1"
    },
    {
      "field": "uses",
      "target": "XAARVNKXF1"
    },
    {
      "field": "uses",
      "target": "0BSJSX7ME2"
    },
    {
      "field": "uses",
      "target": "0R6EWFKRCF"
    },
    {
      "field": "uses",
      "target": "0X3WGX95FE"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/HSRNND3TG5.md",
  "msc": [
    "41A30",
    "68T07",
    "41A63"
  ],
  "name": "Universal approximation theorem",
  "prov": null,
  "role": "theorem",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/HSR/HSRNND3TG5.md",
  "status": null,
  "tag": "HSRNND3TG5",
  "uri": "m2://HSRNND3TG5",
  "url": "https://tamnd.github.io/math2/wiki/universal-approximation-theorem/",
  "writing": {
    "tag": "HSRNND3TG5",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}