Chapter 4

Discrete time Markov Chain

Learning objectives :

Introduce discrete time Markov Chain

Model manufacturing systems using Markov Chain
Able to evaluate the steady-state performances

Textbook :
C. Cassandras and S. Lafortune, Introduction to Discrete
Event Systems, Springer, 2007
1

Plan

Basic definitions of discrete time Markov Chains

Classification of Discrete Time Markov Chains
Analysis of Discrete Time Markov Chains

Basic definitions
of discrete time Markov chains

Discrete Time Markov Chain (DTMC)

Definition : a stochastic process with discrete state space and
discrete time {Xn, n > 0} is a discrete time Markov Chain
(DTMC) iff
P[Xn+1 = j Xn = in, ..., X0 = i0] = P[Xn+1 = j Xn = in] = pij(n)
In a DTMC, the past history impacts on the future evolution of
the system via the current state of the system
pij(n) is called transition probability from state i to state j at
time n.
4

Discrete Time Markov Chain (DTMC)

Stochastic
process
Continuous
event

Discrete
events

Discrete
time

Memoryless

Continuous
time

A DTMC is a discrete
time and memoriless
discrete event
stochastic process.

Example: a mouse in a maze ( )

5
exit

start

Which stochastic process can be used to represent the position of

the mouse at time t?
Under which assumptions, the system can be represented by a
discrete time Markov chain?
6

Example: a mouse in a maze

1/2

1/2

start

1/2
1/4
1

1/4

1/4

exit

1/2

1/4

Let {Xn}n=0, 1, 2, ... the position of the mouse after n rooms visited
Assume that the mouse does not have any memory of rooms
visited previously and that she chooses any corridor equiprobably.

Homogenuous DTMC

A DTMC is said homogenuous iff its transitions

probabilities do not depend on the time n, i.e.
P[Xn+1 = j Xn = i] = P[X1 = j X0 = i] = pij
A homogenuous DTMC is then defined by its
transition matrix P =[pij]i,jE

What is the transition matrix of the process?

1/2

1/2

start

1/2
1/4
1

1/4

1/4

exit

1/2

1/4

Let {Xn}n=0, 1, 2, ... the position of the mouse after n rooms visited
Assume that the mouse does not have any memory of rooms
visited previously and that she chooses any corridor equiprobably.

Stochastic Matrix

A square matrix is said stochastic iff

all entries are non negative
each line sums to 1
Properties:
A transition matrix is a stochastic matrix
If P is stochastic, then Pn is stochastic
The eigenvalues of P are all smaller than 1, i.e. || 1
10

Assumptions

In the remaining of the chapter, we limit ourselves to

Markov chain
of discrete time
defined on a finite state space E
homogeneous in time.
Note that most results extend to countable state
space.
11

Graphic representation of a DTMC

12

Classification of Discrete Time

Markov Chains

13

Classification of states
Let fjj be the probability of returning to state j after leaving j.
A state j is said transient if fjj < 1
A state j is said recurrent if fjj = 1
A state j is said absorbing if pjj = 1.
Let Tjj be the average reccurn time, i.e. time of returning to j
A recurrent state j is positive recurrent if E[Tjj] is finite.
A recurrent state j is null recurrent if E[Tjj] = .
14

Classify the states of the example

1/2

1/2

start

1/2
1/4
1

1/4

1/4

exit

1/2

1/4

15

Irreducible Markov chain

A DTMC is said irreducible iff a state j can be

reached in a finite number of steps from any other
state i.
An irreducible DTMC is a strongly connected graph.

16

Irreducble Markov chain

17

Periodic Markov chain

A state j is said periodic if it is visited only in a

number of steps which is multiple of an integer d > 1,
called period.
A state j is said aperiodic otherwise
A state with a self-loop transition
(i.e. pii > 0) is always aperiodic.
All states of an irreducible Markov
chain have the same period.
18

Partitionning a DTMC into irreducible sub-chains

A DTMC can be partitionned into strongly connected

components, each corresponding to an irreducible
sub-chain.

19

Classification of irreducible sub-chains

A sub-chain is said absorbing if there is no arc going

out of it.
Otherwise, the sub-chain is transient.
transcient
sub-chain
absorbing
sub-chain

absorbing
sub-chain
20

Canonic form of transition matrix

Q : transitions of transient sub-chains

Pi : transititions between states of aborbing sub-chain i
Ri: Transitions toward absorbing sub-chain i
21

Formal definitions
A state j is said reachable from a state i if there is a path from
i to j in the state transition diagram.
A subset S of states is said closed if there is no transition
leaving S.
A closed set S is said irreducible if all states in S are mutually
reachable.
A Markov chain is said irreducible if its state space is
irreducible.

22

Theorems
Th1. If a Markov chain has a finite state space, then at least one state
is recurrent.
Th2. If i is a recurrent state and j is reachable from i, then state j is
recurrent.
Th3. If S is a finite closed irreducible set of states, then every state in
S is recurrent.
Th4. If i is a positive recurrent state and j is reachable from i, then
state j is positive recurrent.
Th5. If S is a closed irreducible set of states, then every state in S is
positive recurrent or every state in S is null recurrent or every
state in S is transient.
Th6. If S is a finite closed irreducible set of states, then every state in
S is positive recurrent.
23

Analysis of DTMC

24

Sojourn time in a state

Let Ti be the temps spent in state i before jumping to

other states.
P Ti n piin 1 1 pii

Ti is a random variable of geometric distribution.

25

Properties of geometric distribution

Let X be a random variable of geometric distribution with
parameter p, i.e. P{X = n} = (1-p)n-1p.
E[X] = 1/p
Var(X) = 1/p2
X = 1/p
Coefficient of variation = X / E[X] = 1
Memoryless (only discrete distribution of this property):
P X n m
P X n m X n
P X n
n m 1
p
1 p
m 1

p
p P X m

n 1
1 p

26

m-step transition probabilities

The probability of going from i to j in m steps is
pij(m) = P{Xn+m = j|Xn=i} = P{Xm = j|X0=i}.
Let P(m) = [pij(m)] be the m-step transition matrix
Properties (to prove):
P(m) = Pm
Chapman-Kolmogorov equation:
P(l+m) = P(l)P(m)
l m
l
m
or
p
p p
ij

k E

ik

kj

27

Example
1/2

1/2

start

1/2
1/4
1

1/4

1/4

exit

1/2

1/4

What is the probability that the mouse is still in room 2 at time

4? (p22(4))
28

Probability of going from i to j in exactly n steps

fij(n) : probability of going from i to j in exactly n steps (without
passing j before)
fij: probability of going from i to j in a finite number of steps

fij f ij n
n 1

fij pij pik f kj

k j

Similar approach can be used to determine the

average time Tij it takes for going from i to j
29

Probability distribution of states

i(n) : probability of being in state i at time n
i(n) = P{Xn = i}
(n) = (1(n), 2(n), ...) : vector of probability distribution
over the state space at time n
The probability distribution (n) depends on
the transition matrix P
the initial distribution (0)
Remark: if the system is at state i for certainty, then i(0) =
1 and j(n) = 0, for j i
What is the relation between (n), (0), and P?

30

Transient state equations

By conditioning on the state at time n,

Property:
Let P be the transition matrix of a markov chain and (0) the
initial distribution, then over the state space at time n
(n+1) = (n)P
(n)= (0)Pn
31

Steady-state distribution
Key questions :
Is the distribution (n) converges when n goes to infinity?
If the distribution converges, does its limit = (1, 2, ...)
depend on the initial distribution (0)?
If a state is recurrent, what is the percentage of time spent in
this state and what is the number of transitions between two
successive visits to the state?
If a state is absorbing, what is the probability of ending at this
state? What is the average time to this state?

32

Steady state distribution

Theorem : For a irreducible and aperiodic DTMC with positive
recurrent states, the distribution (n) converges to a limit
vector which is independent of (0) and is the unique
solution of the system:
P

iE

j i pij , j E

iE

iE

Normalization equation

balance equation
equilibrium equation

i are also called stationary probabilities (also called steady

state or equilibrium distribution).
For an irreducible and periodic DTMC, i are the percentage 33
of time spent in state i

Flow balance equation

i ij
Equation j
can be interpretated as balance
i E
equation of probability flow.

A probability flow ipij is associated to each transition (i, j).

i psum
ij
is
the
of probability flow into node j
iE
or j p ji
isj the
sum of flow out of node j
iE

The flow balance equation : Outgoing flow = Incoming flow

34

A manufaturing system

Consider a machine which can be either UP or DOWN.

The state of the machine is checked every day.
The average time to failure of an UP machine is 10 days.
The average time for repair of a DOWN machine is 1.5 days.

Determine the conditions for the state of the machine {X n} at the begining
of each day to be a Markov chain.
Draw the Markov chain model.
Find the transient distribution by starting from state UP and DOWN.
Check whether the Markov chain is recurrent and aperiodic.
Determine the steady state distribution.
Determine the availability of the machine.

35

A telephone call process

Discrete time model with time slots indexed by k = 0, 1, 2, ...

At most one telephone call can occur in a single time slot, and there is a
probability that a call occurs in any slot

If the phone is busy, the call is lost; otherwise, the call is processed.

There is a probability that a call in process completes in any time slot

If both a call arrival and a call completion occur in the same time slot, the
new call will be processed.

Issues to solve:
Markov chain model
Loss probability
36