(X_{n})_{n} denotes a homogeneous Markov chain of finite or countable states with its transition matrix Q: X_{n} models the state of a system at time n.

The initial law of the chain (X_{n}) having been fixed, only the linked states intervene in the study of the evolution of the chain: X_{a} = {x ∈ X, there exists n ≥ 0, P(X_{n} = x) > 0}.

For a state e ∈ X_{a}, we denote by T^{(n)}_{e} the time it takes the chain to reach the state e strictly after the instant n:

- T
^{(n)}_{e}denotes the smallest integer k> 0 such that X_{n+k}= e if the chain goes through e after the instant n - T
^{(n)}_{e}= +∞ if the chain does not go through state e after moment n.

Let’s i, e ∈ X_{a}.

- For all n ∈ N and k ∈ N
^{∗}, the probability of reaching the state e at the instant n + k for the first time after the instant n knowing that X_{n}= i does not depend on n, we note f_{i,e}(k) = P(T^{(n)}_{e}= k|X_{n}= i). It satisfies the following equation: f_{i,e}(1) = Q(i, e) and f_{i,e}(k) = ∑_{j∈X\{e}}Q(i, j)f_{j,e}(k − 1) for all k ≥ 2.

The equation for f_{i,e}(k) simply translates the fact that to arrive for the first time in the state e in k steps, starting from the state i, it is necessary to go from the state i to a state j ≠ e, then starting from j, arrive for the first time in e in k – 1 step.

- The probability of reaching state e after time n, knowing that X
_{n}= i does not depend on n, we denote it f_{i,e}: f_{i,e}= P(T^{(n)}_{e}< +∞|X_{n}= i). It verifies: f_{i,e}= Q(i e) + ∑_{j∈X\{e}}Q(i,j)f_{j,e}.

The equation for f_{i,e} reflects the fact that the chain reached from a state i either directly or goes from state i to a state j ≠ e then reached e from state j.

Note that f_{i,e}=1 if and only if f_{j,e}=1 for all states j ≠ e such that Q(i, j)> 0.

When the chain is ergodic, it is possible to calculate the return time in a state by calculating the inverse of the stationary probability. For this it is necessary that all the states are a positive recurrence, that is to say that its expected value is positive and not infinite:

## Example

Suppose that a player has 1 euro and plays a game of chance where the bet is 1 euro until he has reached the sum of 3 euros or until it is ruined. Remind that p is the probability that he wins a game and therefore wins 1 euro and 1-p is the probability that he loses the game and therefore loses 1 euro.

We are looking for the probability that he will thus succeed in having 3 euros, that is to say f_{1,3}. The equation with i = 1 and e = 3 is written f_{1,3} = pf_{2,3} + (1 − p)f_{0,3}. We have f_{0,3} = 0 since the player can not play if he does not have money initially (0 is an absorbing state). It remains to write the equation for f_{2,3} which is the probability that a player gets to get 3 euros if he initially has 2 euros. We obtain: f_{2,3} = 1 − p + (1 − p)f_{1,3}. We therefore have to solve a system of 2 equations with 2 unknowns: {f_{1,3} = pf_{2,3}; f_{2,3} = p + (1 − p)f_{1,3}}.

The unique solution is: f_{2,3} =p/(1−p(1−p) and f_{1,3} =p²/1−p(1−p).

In particular, if the game is fair, ie if p = 1/2, we have f_{1,3} = 1/3 which means that if the player initially has 1 euro, he has one chance out of three d ‘get to get 3 euros. The probability that the game stops because the player is ruined is calculated in a similar way by writing the equations satisfied by f_{1,0}, f_{2,0} and f_{3,0} and solving the system obtained. This calculation makes it possible to see that f_{1,0} + f_{1,3} = 1, which means that the player necessarily stops after a finite number of games either because he managed to obtain 3 euros, or because he is ruined.

## Time to reach a class / state

This time is the smallest positive solution of the system:

where the absorbent class designates the states in which the system begins. Indeed, since we start from these states, the average time to reach them is 0 movement.

## Example

A flea jumps on a triangle, with a probability 2/3 clockwise, and 1/3 counterclockwise.

- Let’s call the vertices: 1,2,3.
- Let’s look at reaching times. We start from the vertex 1. We have i = 1 : x
_{1}=0 - For i = 2, 3, we have to solve the following system:
- x
_{2}=1+ 1/3 x_{1}+ 0 x_{2}+ 2/3 x_{3} - x
_{3}=1+ 2/3x_{1}+ 1/3 x_{2}+ 0 x_{3}

- x

The solution is (0, 15/7, 12/7)