---
tag: 56NVR0SS61
kind: concept
name: Empirical risk minimization
msc: [68Q32, 62H30, 68T05]
sources:
  - url: https://en.wikipedia.org/wiki/Empirical_risk_minimization
same_as:
  wikidata: Q5374254
  wikipedia: Empirical risk minimization
---

## Summary

Empirical risk minimization is the principle of choosing, from a class of prediction rules, one that has the smallest average [[loss|3BTG6VVCGZ]] on the training data.
The average loss on the data, the empirical risk, stands in for the true risk, which depends on an unknown distribution.
[[Ordinary least squares|YM708KHV2W]] and [[maximum likelihood estimation|0FREPFKNS7]] are instances.
Whether it works is a question of [[statistical learning theory|5MR9ZSC3CX]].
It succeeds when the empirical risk is uniformly close to the true risk over the class.
In binary classification this holds for classes of finite [[VC dimension|KCFXR9K3GD]].

## Explanations

### L1 human

You cannot test a rule on all future data, so you test it on the data you have and pick the rule that does best.
For example, to fit a line through points, choose the line with the smallest average squared vertical distance to the points.
This is a good idea when the family of rules is not too rich.
If the family can fit anything, the winner may just memorise the data and predict badly on new points.
Adding a penalty for complexity, as in [[ridge regression|3KZW2930V2]], is the usual fix.