Automatic differentiation

Summary

Automatic differentiation computes exact values of derivatives of a function given as a computer program. It applies the chain rule to each elementary operation of the program. So its only errors are rounding errors, unlike numerical differentiation. In forward mode it propagates directional derivatives along with the values; in reverse mode it propagates sensitivities backwards from the outputs. Reverse mode computes the gradient of a scalar function at a small constant multiple of the cost of the function. The backpropagation algorithm of machine learning is a special case of reverse mode.