Math 2.0

Statistics Definition

Observed information

Builds on Likelihood functionHessian matrixMaximum likelihood estimationFisher informationPoisson distribution

Definition

Let {f(⋅ ;θ):θ∈Θ}\set{f(\cdot\,; \theta) : \theta \in \Theta} be a family of densities with Θ⊆ℝd\Theta \subseteq \R^d open. Let x1,…,xnx_1, \dots, x_n be observed data, and let ℓ(θ)=∑i=1nlog⁡f(xi;θ)\ell(\theta) = \sum_{i=1}^{n} \log f(x_i; \theta) be the log-likelihood, assumed twice differentiable in θ\theta. The observed information at θ\theta is the d×dd \times d matrix J(θ)=−∇2ℓ(θ),Jjk(θ)=−∂2ℓ(θ)∂θj ∂θk,J(\theta) = -\nabla^2 \ell(\theta), \qquad J_{jk}(\theta) = -\frac{\partial^2 \ell(\theta)}{\partial \theta_j \, \partial \theta_k}, that is, minus the Hessian matrix of the log-likelihood. For d=1d = 1 it is the number J(θ)=−ℓ′′(θ)J(\theta) = -\ell''(\theta).

Remarks

It is usually evaluated at the maximum likelihood estimate θ^\hat\theta, and J(θ^)J(\hat\theta) is then positive semidefinite when θ^\hat\theta is an interior maximum. If the data are independent with density f(⋅ ;θ)f(\cdot\,; \theta), then under regularity conditions Eθ[J(θ)]=nI(θ)E_\theta[J(\theta)] = n I(\theta), where II is the Fisher information. The inverse J(θ^)−1J(\hat\theta)^{-1} is a standard estimate of the covariance matrix of θ^\hat\theta.

Example: Poisson data with mean λ>0\lambda > 0 and t=x1+⋯+xn>0t = x_1 + \cdots + x_n > 0. Then ℓ(λ)=tlog⁡λ−nλ−∑ilog⁡(xi!)\ell(\lambda) = t \log \lambda - n\lambda - \sum_i \log(x_i!), so J(λ)=t/λ2J(\lambda) = t/\lambda^2. At λ^=t/n\hat\lambda = t/n this gives J(λ^)=n2/tJ(\hat\lambda) = n^2/t.

Non-example: minus the second derivative of the likelihood L=eℓL = e^{\ell} itself is not the observed information. In the Poisson example with n=1n = 1 and t=1t = 1, −L′′(λ)-L''(\lambda) at λ^=1\hat\lambda = 1 equals e−1e^{-1}, while J(1)=1J(1) = 1.

Reading path

To understand this page from the ground up, read these 83 pages in order. The foundations come first.

  1. Construction of the reals from Cauchy sequences concept
  2. Element definition
  3. Euclidean space object
  4. Fisher information concept
  5. Function concept
  6. Integers object
  7. Likelihood function concept
  8. Ordered field definition
  9. Poisson distribution concept
  10. Proposition concept
  11. Random experiment concept
  12. Rational numbers object
  13. Real numbers object
  14. Set definition
  15. Function of several variables concept
  16. Ring of integers statement
  17. Disjunction definition
  18. Implication definition
  19. Predicate definition
  20. Ordered pair definition
  21. Partial derivative concept
  22. Union definition
  23. Existential quantifier definition
  24. Sample space definition
  25. Universal quantifier definition
  26. The rationals form a field statement
  27. Cartesian product definition
  28. Higher partial derivatives concept
  29. Subset definition
  30. Uniqueness quantifier definition
  31. The rationals form an ordered field statement
  32. Binary relation definition
  33. Hessian matrix concept
  34. Universal set definition
  35. Order on the reals definition
  36. Power set definition
  37. Function definition
  38. Complement definition
  39. Interval definition
  40. Injective function definition
  41. Preimage of a set definition
  42. Sigma-algebra definition
  43. Countable set definition
  44. Event definition
  45. Borel sigma-algebra definition
  46. Probability axioms concept
  47. Probability measure definition
  48. Probability distribution definition
  49. Probability space definition
  50. Conditional probability definition
  51. Random variable definition
  52. Cumulative distribution function definition
  53. Discrete random variable definition
  54. Law of a random variable definition
  55. Independence of two events definition
  56. Convergence in probability definition
  57. Random variable with a density definition
  58. Joint probability distribution definition
  59. Probability mass function definition
  60. Mutual independence of events definition
  61. Expected value definition
  62. Identically distributed random variables definition
  63. Independence of two random variables definition
  64. Probability density function definition
  65. Marginal distribution definition
  66. Independent sigma-algebras definition
  67. Variance definition
  68. Linearity of expectation statement
  69. Independence of random variables definition
  70. Independence means the joint law is the product of the marginals statement
  71. Covariance definition
  72. Independent and identically distributed random variables definition
  73. Variance of a sum statement
  74. Expectation of a product of independent random variables statement
  75. Statistic definition
  76. Likelihood function definition
  77. Independent random variables are uncorrelated statement
  78. Estimator definition
  79. Variance of a sum of independent random variables statement
  80. Sample mean definition
  81. Consistent estimator definition
  82. Maximum likelihood estimation concept
  83. Observed information definition

Further reading

  1. Web https://en.wikipedia.org/wiki/Observed_information
About this page
TagZPPN50CEEX
URIm2://ZPPN50CEEX
Kinddefinition
WritingW0 stub
MSC62F10, 62B10
DefinesObserved information
UsesLikelihood function, Hessian matrix, Maximum likelihood estimation, Fisher information, Poisson distribution

Markdown · JSON · Edit on GitHub

What this page needs for W1

  • an inventory row: the row of the inventory that the node accounts for (0 of 1)
  • a source with a locator (0 of 1)

The writing levels say how complete the text of a page is. They do not measure its truth.

What links here

No page links to this page yet.