{
  "aliases": null,
  "backlinks": [
    {
      "field": "defines",
      "target": "42AGEAAQ8H"
    }
  ],
  "body": "\n## Summary\n\nNeural network approximation studies how well functions can be approximated by the functions that neural networks compute.\nA shallow network computes a sum $\\sum_{k=1}^n c_k \\sigma(w_k \\cdot x + b_k)$ of ridge functions built from one activation function $\\sigma$; a deep network composes several such layers.\nThe universal approximation theorem says that shallow networks with a continuous non-polynomial activation are dense in the [[continuous functions|0X3WGX95FE]] on every [[compact set|0TVJZYB6SH]] of $\\R^d$.\nRates of approximation in terms of the number of neurons or parameters are a main topic of [[nonlinear approximation|YS1GRQRVY7]].\n\n## Explanations\n\n### L1 human\n\nA neural network adds up many copies of one simple function, each one shifted, stretched and pointed in its own direction.\nWith enough copies, such sums can follow any continuous shape on a bounded region as closely as one likes.\nWith the ReLU activation $\\sigma(t) = \\max(t, 0)$, each copy is a hinge, and sums of hinges in one variable are exactly the continuous piecewise linear functions with finitely many pieces.\nFor example, $\\abs{x} = \\sigma(x) + \\sigma(-x)$ is a network with two neurons.\nApproximation theory asks how many neurons or layers are needed to reach a given accuracy, and for which functions.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "kind": "concept",
    "msc": [
      "41A46",
      "68T07",
      "41A30"
    ],
    "name": "Neural network approximation",
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Universal_approximation_theorem"
      }
    ],
    "specializes": [
      "YS1GRQRVY7"
    ],
    "tag": "W7CK84MZFV"
  },
  "kind": "concept",
  "link": "https://tamnd.github.io/math2/node/W7CK84MZFV/",
  "links": [
    {
      "field": "specializes",
      "target": "YS1GRQRVY7"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/W7CK84MZFV.md",
  "msc": [
    "41A46",
    "68T07",
    "41A30"
  ],
  "name": "Neural network approximation",
  "prov": null,
  "role": "",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/W7C/W7CK84MZFV.md",
  "status": null,
  "tag": "W7CK84MZFV",
  "uri": "m2://W7CK84MZFV",
  "url": "https://tamnd.github.io/math2/wiki/neural-network-approximation/",
  "writing": {
    "tag": "W7CK84MZFV",
    "kind": "concept",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}