Math 2.0

Probability theory and stochastic processes Definition

Markov chain

Builds on Stochastic processConditional probabilityMarkov propertyCountable set

Definition

Let SS be a nonempty countable set. Let (Xn)n≥0(X_n)_{n \ge 0} be a stochastic process with values in SS. It is a Markov chain with state space SS if P(Xn+1=j∣X0=i0,…,Xn−1=in−1,Xn=i)=P(Xn+1=j∣Xn=i)P(X_{n+1} = j \mid X_0 = i_0, \dots, X_{n-1} = i_{n-1}, X_n = i) = P(X_{n+1} = j \mid X_n = i) for each n≥0n \ge 0 and all states i0,…,in−1,i,j∈Si_0, \dots, i_{n-1}, i, j \in S with P(X0=i0,…,Xn=i)>0P(X_0 = i_0, \dots, X_n = i) > 0. The chain is time-homogeneous if pij=P(Xn+1=j∣Xn=i)p_{ij} = P(X_{n+1} = j \mid X_n = i) does not depend on nn.

Remarks

The condition is the Markov property in discrete time. For a time-homogeneous chain, the numbers pijp_{ij} form the transition matrix P=(pij)i,j∈SP = (p_{ij})_{i, j \in S}. The initial distribution λi=P(X0=i)\lambda_i = P(X_0 = i) and PP determine the law of the chain: P(X0=i0,…,Xn=in)=λi0pi0i1⋯pin−1inP(X_0 = i_0, \dots, X_n = i_n) = \lambda_{i_0} p_{i_0 i_1} \cdots p_{i_{n-1} i_n}. Markov chains also exist in continuous time and on general state spaces.

Reading path

To understand this page from the ground up, read these 50 pages in order. The foundations come first.

  1. Construction of the reals from Cauchy sequences concept
  2. Element definition
  3. Integers object
  4. Ordered field definition
  5. Proposition concept
  6. Random experiment concept
  7. Rational numbers object
  8. Real numbers object
  9. Set definition
  10. Ring of integers statement
  11. Disjunction definition
  12. Implication definition
  13. Predicate definition
  14. Ordered pair definition
  15. Union definition
  16. Sample space definition
  17. The rationals form a field statement
  18. Existential quantifier definition
  19. Universal quantifier definition
  20. Cartesian product definition
  21. Subset definition
  22. The rationals form an ordered field statement
  23. Uniqueness quantifier definition
  24. Binary relation definition
  25. Universal set definition
  26. Order on the reals definition
  27. Power set definition
  28. Complement definition
  29. Function definition
  30. Interval definition
  31. Injective function definition
  32. Preimage of a set definition
  33. Sigma-algebra definition
  34. Countable set definition
  35. Event definition
  36. Borel sigma-algebra definition
  37. Probability axioms concept
  38. Probability measure definition
  39. Probability space definition
  40. Conditional probability definition
  41. Random variable definition
  42. Independence of two events definition
  43. Discrete random variable definition
  44. Stochastic process concept
  45. Random variable with a density definition
  46. Conditional independence definition
  47. Expected value definition
  48. Conditional expectation definition
  49. Markov property definition
  50. Markov chain definition

Where this leads

16 pages build on this one.

Further reading

  1. Web https://en.wikipedia.org/wiki/Markov_chain, Discrete-time Markov chain
About this page
Tag033TJBZDG0
URIm2://033TJBZDG0
Kinddefinition
WritingW1 stated
MSC60J10
DefinesMarkov chain
UsesStochastic process, Conditional probability, Markov property, Countable set

Markdown · JSON · Edit on GitHub

What this page needs for W2

  • edge not_instance_of in: one non-example that fails one clause (0 of 1)

The writing levels say how complete the text of a page is. They do not measure its truth.

What links here

Used by