Bootstrap

Remarks

In practice one draws $B$ independent bootstrap samples, computes $\hat\theta^*_1, \dots, \hat\theta^*_B$, and estimates the standard error of $\hat\theta$ by $$\widehat{\mathrm{se}}_B = \Big( \frac{1}{B - 1} \sum_{b=1}^{B} \big(\hat\theta^*_b - \bar\theta^*\big)^2 \Big)^{1/2}, \qquad \bar\theta^* = \frac{1}{B} \sum_{b=1}^{B} \hat\theta^*_b.$$ The percentile interval takes the $\alpha/2$ and $1 - \alpha/2$ empirical quantiles of the $\hat\theta^*_b$ as a confidence interval of level about $1 - \alpha$.

Example: data $(1, 3)$ and $\hat\theta = \bar X$. The four equally likely bootstrap samples $(1,1), (1,3), (3,1), (3,3)$ have means $1, 2, 2, 3$. Their variance is $(1 + 0 + 0 + 1)/4 = 1/2$, which equals $\hat\sigma^2 / n$ with the plug-in variance $\hat\sigma^2 = 1$.

Non-example: for $X_i$ uniform on $[0, \theta]$ and $\hat\theta = \max_i X_i$, the bootstrap fails. A bootstrap sample repeats the observed maximum with probability $1 - (1 - 1/n)^n \to 1 - e^{-1}$, so the bootstrap law of $n(\hat\theta - \hat\theta^*)$ has an atom at $0$, while the true law of $n(\theta - \hat\theta)$ is continuous.

The jackknife is an older resampling method that leaves out one observation at a time.