Math 2.0

Statistics Definition

Nonlinear regression

Builds on Linear regressionLeast squaresNewton's method

Definition

Let Θ⊆ℝp\Theta \subseteq \R^p be a parameter set, let x1,…,xnx_1, \dots, x_n be known regressor values in a set 𝒳\mathcal{X}, and let f ⁣:𝒳×Θ→ℝf \colon \mathcal{X} \times \Theta \to \R be a known function. Assume that θ↦f(x,θ)\theta \mapsto f(x, \theta) is not affine for some xx. A nonlinear regression model states that yi=f(xi,θ)+εi,i=1,…,n,y_i = f(x_i, \theta) + \varepsilon_i, \quad i = 1, \dots, n, for an unknown θ∈Θ\theta \in \Theta and random errors εi\varepsilon_i with mean 00. The nonlinear least squares estimate is a minimiser over θ∈Θ\theta \in \Theta of S(θ)=∑i=1n(yi−f(xi,θ))2.S(\theta) = \sum_{i=1}^n \big( y_i - f(x_i, \theta) \big)^2.

Remarks

The estimate is computed iteratively. The Gauss–Newton method replaces f(xi,θ)f(x_i, \theta) near the current value θ(k)\theta^{(k)} by its first-order Taylor expansion, solves the resulting linear least squares problem, and repeats. It is an approximation of Newton's method for minimising SS. The minimiser may not be unique, and the iteration may converge to a local minimum.

Example: the exponential decay model f(x,θ)=θ1e−θ2xf(x, \theta) = \theta_1 e^{-\theta_2 x} with θ1,θ2>0\theta_1, \theta_2 > 0. The map θ↦f(x,θ)\theta \mapsto f(x, \theta) is not affine, since ∂f/∂θ1=e−θ2x\partial f / \partial \theta_1 = e^{-\theta_2 x} depends on θ2\theta_2 for x≠0x \ne 0.

Non-example: f(x,θ)=θ0+θ1x+θ2x2f(x, \theta) = \theta_0 + \theta_1 x + \theta_2 x^2 is affine in θ\theta for each xx, so this model is linear regression, although the curve is a parabola.

If the errors are independent N(0,σ2)N(0, \sigma^2), the least squares estimate is the maximum likelihood estimate.

Reading path

To understand this page from the ground up, read these 26 pages in order. The foundations come first.

Show the path as a graph
Limit point of a set, definition, no evidence levelLimit point of a setdefinitionLinear regression, concept, no evidence levelLinear regressionconceptMatrix, definition, no evidence levelMatrixdefinitionVector space, definition, no evidence levelVector spacedefinitionLimit of a function, definition, no evidence levelLimit of a functiondefinitionTranspose, definition, no evidence levelTransposedefinitionInner product, definition, no evidence levelInner productdefinitionLinear map, definition, no evidence levelLinear mapdefinitionLinear subspace, definition, no evidence levelLinear subspacedefinitionNorm, definition, no evidence levelNormdefinitionDerivative, definition, no evidence levelDerivativedefinitionInner product space, definition, no evidence levelInner product spacedefinitionOrthogonal vectors, definition, no evidence levelOrthogonal vectorsdefinitionEndomorphism, definition, no evidence levelEndomorphismdefinitionSum of subspaces, definition, no evidence levelSum of subspacesdefinitionTangent line, concept, no evidence levelTangent lineconceptNorm induced by an inner product, definition, no evidence levelNorm induced by an in…definitionOrthogonal complement, definition, no evidence levelOrthogonal complementdefinitionDirect sum of subspaces, definition, no evidence levelDirect sum of subspac…definitionNewton's method, concept, no evidence levelNewton's methodconceptA space is the direct sum of a subspace and its orthogonal complement, statement, no evidence levelA space is the direct…statementProjection, definition, no evidence levelProjectiondefinitionOrthogonal projection, definition, no evidence levelOrthogonal projectiondefinitionOrthogonal projection is the closest point, statement, no evidence levelOrthogonal projection…statementLeast squares, concept, no evidence levelLeast squaresconceptNonlinear regression, definition, no evidence levelNonlinear regressiondefinition
Each page uses the pages above it. The colour of the bar on a box shows how strong the evidence is, and the box also gives it as text. The list has the same pages.
  • no evidence level
  1. Limit point of a set definition
  2. Linear regression concept
  3. Matrix definition
  4. Vector space definition
  5. Limit of a function definition
  6. Transpose definition
  7. Inner product definition
  8. Linear map definition
  9. Linear subspace definition
  10. Norm definition
  11. Derivative definition
  12. Inner product space definition
  13. Orthogonal vectors definition
  14. Endomorphism definition
  15. Sum of subspaces definition
  16. Tangent line concept
  17. Norm induced by an inner product definition
  18. Orthogonal complement definition
  19. Direct sum of subspaces definition
  20. Newton's method concept
  21. A space is the direct sum of a subspace and its orthogonal complement statement
  22. Projection definition
  23. Orthogonal projection definition
  24. Orthogonal projection is the closest point statement
  25. Least squares concept
  26. Nonlinear regression definition

Further reading

  1. Web https://en.wikipedia.org/wiki/Nonlinear_regression
About this page
TagA16HP3P271
URIm2://A16HP3P271
Kinddefinition
WritingW0 stub
MSC62J02
DefinesNonlinear regression
UsesLinear regression, Least squares, Newton's method

Markdown · JSON · Edit on GitHub

What this page needs for W1

  • an inventory row: the row of the inventory that the node accounts for (0 of 1)
  • a source with a locator (0 of 1)

The writing levels say how complete the text of a page is. They do not measure its truth.

What links here

No page links to this page yet.