---
tag: A16HP3P271
kind: definition
name: Nonlinear regression
msc: [62J02]
defines: C7E69V030F
uses: [41FVC4P5JB, 0BYTQ4X1HC, 07EXVNF9VP]
sources:
  - url: https://en.wikipedia.org/wiki/Nonlinear_regression
---

## Definition

Let $\Theta \subseteq \R^p$ be a parameter set, let $x_1, \dots, x_n$ be known regressor values in a set $\mathcal{X}$, and let $f \colon \mathcal{X} \times \Theta \to \R$ be a known function.
Assume that $\theta \mapsto f(x, \theta)$ is not affine for some $x$.
A *nonlinear regression model* states that
$$y_i = f(x_i, \theta) + \varepsilon_i, \quad i = 1, \dots, n,$$
for an unknown $\theta \in \Theta$ and random errors $\varepsilon_i$ with mean $0$.
The *nonlinear least squares estimate* is a minimiser over $\theta \in \Theta$ of
$$S(\theta) = \sum_{i=1}^n \big( y_i - f(x_i, \theta) \big)^2.$$

## Remarks

The estimate is computed iteratively.
The Gauss–Newton method replaces $f(x_i, \theta)$ near the current value $\theta^{(k)}$ by its first-order Taylor expansion, solves the resulting linear [[least squares|0BYTQ4X1HC]] problem, and repeats.
It is an approximation of [[Newton's method|07EXVNF9VP]] for minimising $S$.
The minimiser may not be unique, and the iteration may converge to a local minimum.

Example: the exponential decay model $f(x, \theta) = \theta_1 e^{-\theta_2 x}$ with $\theta_1, \theta_2 > 0$.
The map $\theta \mapsto f(x, \theta)$ is not affine, since $\partial f / \partial \theta_1 = e^{-\theta_2 x}$ depends on $\theta_2$ for $x \ne 0$.

Non-example: $f(x, \theta) = \theta_0 + \theta_1 x + \theta_2 x^2$ is affine in $\theta$ for each $x$, so this model is [[linear regression|41FVC4P5JB]], although the curve is a parabola.

If the errors are independent $N(0, \sigma^2)$, the least squares estimate is the maximum likelihood estimate.