Continuous-time Markov chains

Continuous-time Markov chains

In the case of discrete time, we observe the states over instantaneous and immutable moments. In the context of continuous-time Markov chains, the observations are continuous, ie without temporal interruption.

Continuous time

Let M + 1 be mutually exclusive states. The analysis starts at time 0 and time runs continuously, we call X (t) the state of the system at time t. The state change points t_i are random points in time (they are not necessarily integers). It is impossible to have two state changes at the same time.

Consider three consecutive points in time where there has been a change of states r in the past, s at the present time and s + t in the future. X (s) = i and X (r) = l. A continuous-time stochastic process with the Markov property if:

The transition probabilities are stationary since they are independent of s. We note p_ij(t) = P (X (t) = j, X (0) = i) the continuous-time transition probability function.

Denote by T_i a random variable denoting the time spent in state i before moving to another state, i∈{0, …,M}. Suppose the process enters state i at time t' = s. Then for a duration t > 0, T_i > t ⇔ X(t' = i ), ∀t'∈[s, s + t].

The stationarity property of the transition probabilities results in: P (T_i > s + t, T_i > s) = P (T_i > t). The distribution of the time remaining until the next output of i by the process is the same regardless of the time already spent in state i. The variable T_i is memoryless. The only continuous random variable distribution having this property is the exponential distribution.

The exponential distribution T_i has only one parameter q_i and its mean (mathematical expectation) is R [T_i ] = 1 / q_i. This result allows us to describe a markov chain in continuous time in an equivalent way as follows:

• The random variable T_i has an exponential distribution of parameter λ
• when the process leaves state i, it goes to state j with probability p_ij such as (similar to a discrete-time Markov chain): 
• the next state visited after i is independent of the time spent in state i.
• The continuous-time Markov chain has the same Class and Irreducibility properties as the discrete-time chains.

Here are some properties of the exponential law:

Duration model

Thus, if we consider μ_i the exponential random variable parameter associated with state i, we can represent the continuous-time Markov chain as follows:

We clearly see the discrete-time Markov chain included, hence the possibility of carrying out a study of the discrete model. It should be noted that there is no notion of periodicity in this context.

If we consider that we move from state i to state j after a time T_ij and that we consider this time as an exponential random variable of rate μ_ij, then it is possible to write the Markov chain in continuous time under a duration model:

Note that there is a big change between the stochastic side of the movement from one state to another and the continuous side over time. It is important to understand that the transition matrix of a continuous-time Markov chain is always a time model.

The transition matrix of a duration model has the following properties:

This matrix is called the infinitesimal generator.

Thus, from the graph following discrete Markov equation (the exponential law is the same in the three states):

It is possible to obtain the following time model: