{
  "aliases": null,
  "backlinks": null,
  "body": "\n## Statement\n\nLet $S = S_1 \\times \\cdots \\times S_d$ be a product of countable sets, and let $\\pi$ be a probability distribution on $S$.\nFor $x \\in S$ and $1 \\le i \\le d$, write $x_{-i}$ for $x$ with its $i$-th coordinate removed, and let $Z_i(x) = \\sum_{z \\in S, \\, z_{-i} = x_{-i}} \\pi(z)$.\nLet $K_i$ be the [[transition matrix|0F462493EA]] on $S$ with\n$$K_i(x,y) = \\frac{\\pi(y)}{Z_i(x)} \\quad \\text{if } y_{-i} = x_{-i} \\text{ and } Z_i(x) \u003e 0,$$\n$K_i(x,y) = 0$ for the other $y$ when $Z_i(x) \u003e 0$, and $K_i(x, \\cdot) = \\delta_x$ when $Z_i(x) = 0$.\nSo $K_i$ redraws the $i$-th coordinate from its [[conditional distribution|0ZN9EZTEM6]] under $\\pi$ given the others.\nThen:\n\n1. each $K_i$ satisfies [[detailed balance|CTGN6CK1ZS]] with respect to $\\pi$, that is, $\\pi(x) K_i(x,y) = \\pi(y) K_i(y,x)$ for all $x, y \\in S$;\n2. $\\pi$ is a [[stationary distribution|0KRH5157WP]] of the systematic-scan Gibbs sampler $K_1 K_2 \\cdots K_d$ and of the random-scan Gibbs sampler $\\frac{1}{d} \\sum_{i=1}^{d} K_i$.\n\n## Remarks\n\nFor clause 1, if $y_{-i} = x_{-i}$ and $Z_i(x) \u003e 0$, then $Z_i(y) = Z_i(x)$ and both sides equal $\\pi(x) \\pi(y) / Z_i(x)$.\nIf $Z_i(x) = 0$ then $\\pi(x) = \\pi(y) = 0$ for all such $y$, and in all other cases both sides are $0$.\nSumming detailed balance over $x$ gives $\\pi K_i = \\pi$, and clause 2 follows because a product or an average of kernels that fix $\\pi$ also fixes $\\pi$.\nThe same argument works for densities on $\\R^d$, with sums replaced by integrals.\nStationarity alone does not give convergence of the chain to $\\pi$.\nIf $S$ is finite and $\\pi(x) \u003e 0$ for every $x$, every entry of $K_1 \\cdots K_d$ is positive, so the chain is irreducible and aperiodic and converges to $\\pi$ by the [[convergence theorem|B1E1ES1BYD]].\nWithout positivity this can fail: for $\\pi$ uniform on $\\set{(0,0), (1,1)} \\subset \\set{0,1}^2$, each $K_i$ fixes every state of positive mass, so the chain never moves.\n\n## Explanations\n\n### L1 human\n\nGibbs sampling draws random samples from a joint distribution of many variables when sampling them all at once is hard.\nIt updates one variable at a time, drawing it from its distribution given the current values of all the others.\nRepeating these simple updates builds a random walk through the space.\nThe result says that this walk leaves the target distribution unchanged.\nSo, when the walk mixes well, the visited states are samples from the target.\n\n### L2 human\n\nGibbs sampling is the [[Markov chain Monte Carlo|ZGNBR2SPZ0]] method of choice when the full conditional distributions are easy to sample.\nThis is the case in many Bayesian hierarchical models with conjugate priors and in lattice models such as the Ising model.\nEach update is a Metropolis–Hastings step whose proposal is the full conditional, so its acceptance probability is always $1$.\nThe systematic-scan sampler is in general not reversible, though each $K_i$ is; the random-scan sampler is reversible.\nStrong dependence between coordinates slows mixing, since moves along single coordinates then make little progress; blocking several coordinates together is the usual remedy.\n",
  "commit": "ff8a40749dc16545e3d2fc637f6b0196e81d0bfa",
  "front": {
    "about": [
      "ZGNBR2SPZ0"
    ],
    "kind": "statement",
    "msc": [
      "65C05",
      "65C40",
      "60J22"
    ],
    "name": "Gibbs sampling",
    "role": "proposition",
    "same_as": {
      "wikidata": "Q1191905",
      "wikipedia": "Gibbs sampling"
    },
    "sources": [
      {
        "url": "https://en.wikipedia.org/wiki/Gibbs_sampling"
      }
    ],
    "tag": "KS91STZCGM",
    "uses": [
      "ZGNBR2SPZ0",
      "0ZN9EZTEM6",
      "0F462493EA",
      "0KRH5157WP",
      "CTGN6CK1ZS",
      "B1E1ES1BYD"
    ]
  },
  "kind": "statement",
  "link": "https://tamnd.github.io/math2/node/KS91STZCGM/",
  "links": [
    {
      "field": "about",
      "target": "ZGNBR2SPZ0"
    },
    {
      "field": "uses",
      "target": "ZGNBR2SPZ0"
    },
    {
      "field": "uses",
      "target": "0ZN9EZTEM6"
    },
    {
      "field": "uses",
      "target": "0F462493EA"
    },
    {
      "field": "uses",
      "target": "0KRH5157WP"
    },
    {
      "field": "uses",
      "target": "CTGN6CK1ZS"
    },
    {
      "field": "uses",
      "target": "B1E1ES1BYD"
    }
  ],
  "markdown": "https://tamnd.github.io/math2/node/KS91STZCGM.md",
  "msc": [
    "65C05",
    "65C40",
    "60J22"
  ],
  "name": "Gibbs sampling",
  "prov": null,
  "role": "proposition",
  "source": "https://github.com/tamnd/math2/blob/main/nodes/KS9/KS91STZCGM.md",
  "status": null,
  "tag": "KS91STZCGM",
  "uri": "m2://KS91STZCGM",
  "url": "https://tamnd.github.io/math2/wiki/gibbs-sampling/",
  "writing": {
    "tag": "KS91STZCGM",
    "kind": "statement",
    "level": "W0",
    "next": [
      {
        "what": "an inventory row",
        "note": "the row of the inventory that the node accounts for",
        "have": 0,
        "want": 1
      },
      {
        "what": "a source with a locator",
        "have": 0,
        "want": 1
      }
    ]
  }
}