Jackknife

Remarks

If the bias of $\hat\theta$ has the form $a/n + O(1/n^2)$, the bias of $\hat\theta_{\mathrm{jack}}$ is $O(1/n^2)$.

Example: for the sample mean $\hat\theta = \bar x$, one has $\hat\theta_{(i)} = (n \bar x - x_i)/(n - 1)$, so $\hat\theta_{(i)} - \hat\theta_{(\cdot)} = (\bar x - x_i)/(n-1)$. Then $\widehat{\operatorname{Var}}_{\mathrm{jack}} = \frac{1}{n(n-1)} \sum_i (x_i - \bar x)^2 = s^2/n$, the usual estimate of the variance of $\bar x$, with $s^2$ the sample variance. For the plug-in variance $\hat\theta = \frac1n \sum_i (x_i - \bar x)^2$, the bias-corrected estimate is exactly $s^2 = \frac{1}{n-1}\sum_i (x_i - \bar x)^2$.

Non-example: for the sample median the variance estimate in clause 3 is not consistent. The median is not a smooth function of the data, and its leave-one-out values take only two or three distinct values.