Empirical risk minimization

Summary

Empirical risk minimization is the principle of choosing, from a class of prediction rules, one that has the smallest average loss on the training data. The average loss on the data, the empirical risk, stands in for the true risk, which depends on an unknown distribution. Ordinary least squares and maximum likelihood estimation are instances. Whether it works is a question of statistical learning theory. It succeeds when the empirical risk is uniformly close to the true risk over the class. In binary classification this holds for classes of finite VC dimension.