Permutation test

Summary

A permutation test is a hypothesis test that gets its null distribution by rearranging the data. It recomputes the test statistic on all rearrangements that the null hypothesis allows. In the two-sample case the null hypothesis says that the group labels do not matter, so every reassignment of the labels is equally likely. The p-value is the share of rearrangements whose statistic is at least as extreme as the observed one. Under the null hypothesis the chance of a false rejection is at most the chosen level, with no assumption on the shape of the distribution.