College of Management, NCTU

Operation Research II

Spring, 2009

Chap16 Markov Chains

We often are faced with making decision based upon phenomena that have
uncertainty associated with them. Their variation can be described by a
probability model.
9 We present probability models for processes that evolve over time in a
probabilistic manner. Such processes are called stochastic processes.
A stochastic process is defined to be an indexed collection of random variables
{Xt}, where the index t runs through a given set T.
9 Xt represents a measurable characteristic of interest at time t.
9 Example: {X0, X1, X2, } represent the collection of weekly inventory of a
product.
A general structure of a stochastic process
9 At particular points of time t labeled 0, 1, , the system is found in exactly one
of a finite number of mutually exclusive categories or states labeled 0, 1, , M.
There are M+1 states totally.
9 Thus, the stochastic process {Xt} = {X0, X1, X2, } provides a mathematical
representation of how the status of the physical system evolves over time.
9 This kind of process is referred to as being a discrete time stochastic process
with a finite state space.
A Weather Example
9 The probability of being dry tomorrow is 0.8 if it is dry today, but is only 0.6 if it
rains today.
9 These probabilities do not change if information about the weather before today
is also taken into account.
9 The evolution of the weather from day to day is a stochastic process.
9 Starting on some initial day (labeled as day 0), the weather is observed on each
day t, for t = 0, 1, 2, . The state of the system on day t can be either State 0
(day t is dry) or State 1 (day t has rain).
9 Thus, the random variable Xt = 0, if day t is dry; and Xt = 1, if day t has rain.
An Inventory Example
9 A store stocks a particular model camera that can be ordered weekly.

Jin Y. Wang

Chap16-1

College of Management, NCTU

Operation Research II

Spring, 2009

9 Random variables D1, D2,, Dt represent the demand for this camera during the
tth week (number of cameras that would be sold in week t if the inventory is not
depleted).
Assume that the Dt are independent and identically distributed random
variable having a Poisson distribution with a mean of 1.
9 Let X0 represent the number of cameras on hand at the outset, Xt (t = 1, 2, ) the
number of cameras on hand at the end of week t.
9 {Xt } = {X0, X1, X2, } is a stochastic process where the random variable Xt
represents the state of the system at time t.
9 Assume X0 is 3, so that week 1 begins with three cameras on hand.
9 Order is placed at the end of each week t with the following policy:
If Xt = 0, order 3 cameras.
If Xt > 0, do not order any camera.
9 Thus, Xt+1 = max{3 Dt+1, 0}, if Xt = 0;
Xt+1 = max{Xt Dt+1, 0}, if Xt 1.
9 The possible states are 0, 1, 2, and 3.
Markov Chain
9 A stochastic process {Xt} is said to have the Markovian property if
P{Xt+1 = j | X0 = k0, X1 = k1, , Xt-1 = kt-1, Xt = i} = P{Xt+1 = j | Xt = i},
for t = 0, 1, and every sequence i, j, k0, k1, , kt-1.
9 That is, the conditional probability of any future event depends only upon the
present state.
9 A stochastic process {Xt } (t = 0, 1, ) is a Markov chain if it has the
Markovian property.
9 The conditional probabilities P{Xt+1 = j | Xt = i} are called (one-step) transition
probabilities.
9 If, for each i and j, P{Xt+1 = j | Xt = i} = P{X1 = j | X0 = i} for all t = 0, 1., then
the (one-step) transition probabilities are said to be stationary and are denoted
by pij.
The stationary transition probability implies that the transition
probabilities do not change over time.
9 The stationary transition (one-step) probabilities implies that P{Xt+n = j | Xt = i}
= P{Xn = j | X0 = i} for each i, j, and n (n = 0, 1, 2, ). Usually, denote as n-step
transition probabilities pij(n).
Jin Y. Wang

Chap16-2

College of Management, NCTU

Operation Research II

Spring, 2009

When n = 1, pij(1) = pij.

pij(n) 0.
M

p
j =0

(n)
ij

= 1 for all i; n = 0, 1, 2, .

A convenient notation for representing the n-step transition probabilities is

P(n) =

(n)
p 00
(n)
p10
...
(n)
p M 0

P01( n )
p11( n )
...

...
...
...

p M( n1) ...

p 0( nM)

p1(Mn )
...
(n)
p MM

We drop the superscript n when n = 1 and refer to this as the transition

matrix.

P=

p00

p10
...

p M 0

P01

...

p11 ...
... ...
p M 1 ...

p0 M

p1M
...

p MM

Formulating the Weather Example as a Markov Chain

9 Recall that P{Xt+1 = 0 | Xt = 0} = 0.8 and P{Xt+1 = 0 | Xt = 1} = 0.6.
9 These probabilities do not change if information about the weather before today
(day t) is also taken into account.
P{Xt+1 = 0 | X0 = k0, X1 = k1, , Xt-1 = kt-1, Xt = 0} = P{Xt+1 = 0 | Xt = 0}
P{Xt+1 = 0 | X0 = k0, X1 = k1, , Xt-1 = kt-1, Xt = 1} = P{Xt+1 = 0 | Xt = 1}
(Also hold if Xt+1 = 0 is replaced by Xt+1 = 1.)
9 This stochastic process has the Markovian property; so the process is a Markov
chain.
9 Thus, the transition matrix is
P=
Formulating the Inventory Example as a Markov Chain
9 Xt is the number of cameras in stock at the end of week t is a Markov chain.

Jin Y. Wang

Chap16-3

College of Management, NCTU

Operation Research II

9 Need to obtain the (one-step) transition probabilities P =

Spring, 2009

p 00
p
10
p 20

p30

p 01
p11

p 02
p12

p 21

p 22

p31

p32

p 03
p13
p 23

p33

9 Given that Dt+1 has a Poisson distribution with mean = 1. Thus

(1) n e 1
, for n = 0, 1,
P{Dt+1 = n} =
n!

9 Compute each element pij in P.

Jin Y. Wang

Chap16-4

College of Management, NCTU

9 P=

0.080
0.632

0.264

0.080

Operation Research II

Spring, 2009

0.184 0.368 0.368

0.368
0
0
.
0.368 0.368
0

0.184 0.368 0.368

A Stock Example
9 If the stock has gone up, the probability that it will go up tomorrow is 0.7.
9 If the stock has gone down, the probability that it will go up tomorrow is 0.5.
9 State 0: The stock increased on this day.
State 1: The stock decreased on this day.

9 The transition matrix is P =

A Second Stock Example

9 Suppose now that the stock market model is changed so that the stocks going
up tomorrow depends upon whether it increased today and yesterday.
If the stock has increased for the past two days, it will increase tomorrow
with probability 0.9.
If the stock increased today but decreased yesterday, it will increase
tomorrow with probability 0.6.
If the stock decreased today but increased yesterday, then it will increase
tomorrow with probability 0.5.
If the stock decreased for the past two days, it will increase tomorrow with
probability 0.3.
9 Does this process have the Markovian property?

9 Define the following states to make it a Markov Process

Jin Y. Wang

State 0: The stock increased both today and yesterday.

State 1: The stock increased today and decreased yesterday.
State 2: The stock decreased today and increased yesterday.
State 3: The stock decreased both today and yesterday.

Chap16-5

College of Management, NCTU

Operation Research II

Spring, 2009

0 .9 0 0 .1 0
0 .6 0 0 .4 0

.
0 0 .5 0 0 .5

0 0 .3 0 0 .7

Thus, P =

A Gambling Example
9 A player has $1 and with each play wins $1 with probability p > 0 or loses $1
with probability 1 p.
9 The game ends when the player either accumulates $3 or goes broke.

0
1
1 p
0

0
1 p

0
0

P=

0
p
0
0

0
0
.
p

Chapman-Kolmogorov Equations
9 Provide a method for computing the n-step transition probabilities.
M

pij( n ) = pik( m ) p kj( n m ) , for all i, j, and any m = 1, 2, , n1, n = m+1, m+2, ..
k =0

9 These equations point out that in going from state i to state j in n steps, the
process will be in some state k after exactly m steps.
9 When m = 1 and m = n1, we have p

(n)
ij

= pik p
k =0

( n 1)
kj

and p

(n)
ij

= pik( n 1) p kj for
k =0

all states i and j.

M

9 For n = 2, pij( 2 ) = pik p kj , for all states i and j.

k =0

Jin Y. Wang

Chap16-6

College of Management, NCTU

Operation Research II

Spring, 2009

( 2)
9 Note that pij is the elements of matrix P(2). Also note that these elements are
obtained by multiplying the matrix of one-step transition probabilities by itself;
i.e., P(2) = P P = P2.

9 More generally, it follows that the matrix of n-step transition probabilities can
be obtained from P(n) = P P(n-1) = P(n-1)P =
PPn-1 = Pn-1P = Pn.

n-step Transition Matrices for the Weather Example

0.8 0.2 0.8 0.2

0.76 0.24

9 P(2) = P P =

0.6 0.4 0.6 0.4 0.72 0.28

0.8 0.2 0.76 0.24

0.752 0.248

9 P(3) = P3 = P P2 =

0.6 0.4 0.72 0.28 0.744 0.256

0.8 0.2 0.752 0.248 0.75 0.25

0.6 0.4 0.744 0.256 0.749 0.251

9 P(4) = P4 = P P3 =

0.8 0.2 0.75 0.25 0.75 0.25

0.6 0.4 0.749 0.251 0.75 0.25

9 P(5) = P5 = P P4 =

9 Note that the two rows have identical entries (in P(5)). This reflects the fact that
the probability of the weather being in a particular state is essentially
independent of the state of the weather five days before.
9 Thus, the probabilities in either row of this five-step transition matrix are
referred to as the steady-state probabilities of this Markov chain.
n-step Transition Matrices for the Inventory Example
0.249
0.283
9 P(2) = P2 =
0.351

0.249

Jin Y. Wang

0.286 0.300 0.165

0.252 0.233 0.233
0.319 0.233 0.097

0.286 0.300 0.165

Chap16-7

College of Management, NCTU

Operation Research II

0.289
0.282
(4)
4
(2 )
(2 )
9 P =P =P P =
0.284

0.289

0.286 0.261 0.164

0.285 0.268 0.166
0.283 0.263 0.171

0.286 0.261 0.164

0.286
0.286
(4)
P =
0.286

0.286

0.285 0.264 0.166

0.285 0.264 0.166
0.285 0.264 0.166

0.285 0.264 0.166

9 P(8) = P8 = P(4)

Spring, 2009

Unconditional State Probabilities

9 If the unconditional probability P{Xn = j} is desired, it is necessary to specify
the probability distribution of the initial state, namely, P{X0 = i}.
9 P{Xn = j} = P{X0 = 0}p0j(n) + P{X0 = 1} p1j(n) + + P{X0 = M} pMj(n).
9 In the inventory example, it was assumed that initially there were 3 units in
stock, that is, X0 = 3.
9 Thus, P{X0 = 0} = P{X0 = 1} = P{X0 = 2} = 0 and P{X0 = 3} = 1.
9 P{X2 = 3} =

Classification of States of a Markov Chain

9 State j is said to be accessible from state i if pij(n) > 0 for some n 0.
In the inventory problem, pij(2) > 0 for all i, j, so every state is accessible
from every other state.
0.249

2 0.283
P =
0.351

0.249

0.286 0.300 0.165

0.252 0.233 0.233
0.319 0.233 0.097

0.286 0.300 0.165

In the gambling example, state 2 is not accessible from state 3, but state 3 is
accessible from state 2.

9 A sufficient condition for all states to be accessible is that there exist a value of n
for which pij(n) > 0 for all i and j.

9 If state j is accessible from state i and state i is accessible from state j, then states
i and j are said to communicate.
Any state communicates with itself (because pij(0) > 0 ).

Jin Y. Wang

Chap16-8

College of Management, NCTU

Operation Research II

Spring, 2009

If state i communicates with state j and state j communicates with state k,

then state i communicates with state k.
z

In both the weather and inventory examples, all states communicate.

In the gambling example, state 2 and 3 do not communicate.

9 The states may be partitioned into one or more separate classes such that those
states that communicate with each other are in the same class.
9 If there is only one class (i.e., all states communicate), the Markov chain is said
to be irreducible.
There is only one class in the weather and inventory examples.
There are three classes in the gambling example.

Recurrent States and Transient States

9 A state is said to be a transient state if, upon entering this state, the process may
never return to this state again.
State i is transient if and only if there exists a state j ( j i ) that is
accessible from state i but not vice versa, that is, state i is not accessible
from state j.

9 A state is said to be a recurrent state if, upon entering this state, the process
definitely will return to this state again.
A state is recurrent if and only if it is not transient.
Since a recurrent state definitely will be revisited, it will be visited
infinitely often if the process continues forever.
9 A state is said to be an absorbing state if, upon entering this state, the process
never will leave this state again.
State i is an absorbing state if and only if pii = 1.
9 Recurrent is a class property
All states in a class are either recurrent or transient.
9 In a finite-state Markov Chain, not all sates can be transient. Therefore, all
states in an irreducible finite-state Markov chain are recurrent.

Jin Y. Wang

Chap16-9

College of Management, NCTU

Operation Research II

Spring, 2009

An irreducible finite-state Markov chain (all states are recurrent) can be

identified by showing that all states communicate.
9 All states in the inventory example are recurrent, since pij(2) > 0 for all i and j.
9 Both the weather example and the fist stock example contain only recurrent
states, since pij is positive for all i and j.
9 By calculating pij(2) for all i and j in the second stock example, it follows that all
states are recurrent since pij(2) > 0.
0 .9 0 0 .1 0 0 .9 0 0 .1 0
0 .6 0 0 .4 0 0 .6 0 0 .4 0
2
=

P =
0 0 .5 0 0 .5 0 0 .5 0 0 .5

0 0 .3 0 0 .7 0 0 .3 0 0 .7

9 An Example P =

0
1 / 4 3 / 4 0
1 / 2 1 / 2 0
0

0
0
1
0

0 1/ 2 2 / 3
0
1
0
0
0

0
0
0 . (state starts from 0)

0
0

State 2 is an absorbing (recurrent) state.

States 3 and 4 are transient states.

States 0 and 1 are recurrent states.

Periodicity Properties
9 The period of state i is defined to be the integer t (t > 1) such that pii(n) = 0 for all
values of n other than t, 2t, 3t, and t is the largest integer with this property.

In the gambling example, starting in state 1, we can enter state 1 only at

times 2, 4,State 1 has period 2.
Jin Y. Wang

Chap16-10

College of Management, NCTU

Operation Research II

Spring, 2009

9 If there are two consecutive numbers s and s+1 such that the process can be in
state i at times s and s+1, the state is said to have period 1 and is called an
aperiodic state.
9 Periodicity is also a class property.
In the gambling example, state 2 has period 2 because it is in the same class
as state 1 and we noted that state 1 has period 2.
9 In a finite-state Markov chain, recurrent states that are aperiodic are called
ergodic states.
9 A Markov chain is said to be ergodic if all its states are ergodic states.
Steady-State Probabilities
9 While calculating the n-step transition probabilities for both the weather and
inventory examples, if n is large enough, all the rows of the matrix have
identical entries.
The probability that the system is in each state j no longer depends on the
initial state.
The inventory problem has
0.286
0.286
(8)
8
4 4
P =P =P P =
0.286

0.286

0.285
0.285
0.285
0.285

0.264
0.264
0.264
0.264

0.166
0.166
.
0.166

0.166

9 Does there exists steady-state probabilities? If it does, how can we compute

them?

pij( n ) exists and is independent of

9 For any irreducible ergodic Markov chain, lim
n
i.

pij( n ) = j > 0, where j uniquely satisfy the following

9 Furthermore, lim
n
steady-state equations
M

j = i pij , for j = 0, 1, , M
i =0

j =0

=1.

9 The j are called steady-state probabilities of the Markov chain.

9 Note that the steady-state equations consist of M+2 equations in M+ 1
unknowns.

Jin Y. Wang

Chap16-11

College of Management, NCTU

Operation Research II

Spring, 2009

9 Because it has a unique solution, at least one equation must be redundant and
M

can, therefore, be deleted. This redundant equation cannot be j = 1 , because

i =0

j = 0 for all j will satisfy the other M+1 equations.

Application to the Weather Example
0.8 0.2
.
0.6 0.4

9 Recall that P =

9 The steady-state equations are

0 = 0 p00 + 1 p10 0 = 0.8 0 + 0.6 1

1 = 0 p 01 + 1 p11 1 = 0.2 0 + 0.4 1
0 + 1 = 1
9 Solve the above equations, we have

Application to the Inventory Example

0.080
0.632
9 Recall that P =
0.264

0.080

0.184 0.368 0.368

0.368
0
0
.
0.368 0.368
0

0.184 0.368 0.368

9 The steady-state equations are

0 = 0 p00 + 1 p10 + 2 p 20 + 3 p30

1 = 0 p01 + 1 p11 + 2 p 21 + 3 p31

2 = 0 p02 + 1 p12 + 2 p 22 + 3 p32

3 = 0 p03 + 1 p13 + 2 p 23 + 3 p33

0 + 1 + 2 + 3 = 1

Jin Y. Wang

Chap16-12

College of Management, NCTU

Operation Research II

Spring, 2009

9 Then, we have

0 = 0.080 0 + 0.632 1 + 0.264 2 + 0.080 3

1 = 0.184 0 + 0.368 1 + 0.368 2 + 0.184 3

2 = 0.368 0 + 0.368 2 + 0.368 3

3 = 0.368 0 + 0.368 3
0 + 1 + 2 + 3 = 1
More about Steady-State Probabilities
9 If i and j are recurrent states belonging to different classes, then pij(n) = 0, for all
n.

pij( n ) = 0, for all i.

9 If j is a transient state, then lim
n

Expected Average Cost per Unit Time

9 The (long-run) expected average cost per unit time is

j =0

C ( j ) , where C(j) is

the cost of being in state j.

9 Recall the inventory example. Suppose C(xt = 0) = 0; C(xt = 1) = 2; C(xt = 2) =
8; C(xt = 3) = 18.
9 The long-run expected average storage cost per week is

= 0.286(0) + 0.285(2) + 0.263(8) + 0.166(18) = 5.662.

Expected Average Cost per Unit Time for Complex Cost Functions
9 Still recall the inventory example.
9 If z > 0 cameras are ordered, the cost incurred is (10 + 25z) dollars.
9 For each unit of unsatisfied demand, there is a penalty of $50.
9 C(Xt1, Dt) represents the cost when the demand at week t is Dt and the inventory
at the end of week t-1 is Xt1.
Thus, C(0, Dt) = 10 + (25)(3) + 50 max {Dt 3, 0}.

Jin Y. Wang

Chap16-13

College of Management, NCTU

Operation Research II

Spring, 2009

C(Xt1, Dt) = 50 max {Dt Xt1, 0} , if Xt1 1.

9 k(0) = E[C(0, Dt)] = 85 + 50[PD(4) + 2 PD(5) + 3 PD(6) + ]

9 We already know PD(4) = 0.015, PD(5) = 0.003, PD(6) = 0.001. Ignore PD(i) for
i larger than 6 since the value is too small. We obtain k(0) = 86.2.
9 Similar calculations yield k(1) = 50[PD(2) + 2PD(3) + 3PD(4) + ] = 18.4
k(2) =
k(3) =
9 Thus, the (long-run) expected average cost per week is
3

k ( j )
j =0

= 86.2(0.286) + 18.4(0.285) + 5.2(0.263) + 1.2(0.166) = $31.46.

First Passage Times

9 It is often desirable to make probability statements about the number of
transitions made by the process in going from state i to state j for the first time.
9 This length of time is called the first passage time in going from state i to state
j.
9 When j = i, the first passage time is called the recurrence time for state i.
9 Recall the camera example. Suppose X0 = 3, X1 = 2, X2 = 1, X3 = 0, X4 = 3, X5 =
1. Then, the first passage time in going from state 3 to state 1 is 2 weeks, and the
recurrent time for state 3 is 4 weeks.
9 The first passage times are random variables, and hence have probability
distributions associated with them.
(n )

9 Let f ij denote the probability that the first passage time from state i to j is
equal to n.

f ij(1) = pij(1) = pij

f ij( 2) = pik f kj(1)
k j

f ij( n ) = p ik f kj( n 1)
k j

Jin Y. Wang

Chap16-14

College of Management, NCTU

Operation Research II

Spring, 2009

9 In the inventory problem,

9 For fixed i and j, the fij(n) are nonnegative numbers such that

f
n =1

(n)
ij

1.

When the sum is less than 1, it implies that a process initially in state i may
never reach state j.
When the sum does equal 1, fij(n) can be considered as a probability
distribution for the random variable, the first passage time.
9 Denote the expected first passage time from state i to state j by ij ,

if
ij =
nf ij( n )
if
n =1

9 When

f
n =1

(n)
ij

f
n =1

(n)
ij

<1

f
n =1

(n)
ij

=1

= 1 , ij satisfies uniquely the equation ij = 1 + p ik kj .

k j

9 Revisit the inventory problem.

Jin Y. Wang

Chap16-15

College of Management, NCTU

Operation Research II

Spring, 2009

9 For the case of ij , where i = j, ii is called the expected recurrence time for
state i.
9 After obtaining the steady-state probabilities ( 0 , 1 , 2 , , M ), these expected
recurrence times can be calculated by ii = 1 / i .
9 For the inventory example, where 0 = 0.286, 1 = 0.285, 2 = 0.263, 3 = 0.166,
the corresponding expected recurrence times are:

Absorbing States
9 If k is an absorbing state, and the process starts in state i, the probability of ever
going to state k is called the probability of absorption into state k, given that
the system started in state i. This probability is denoted by fik.
9 If the state k is an absorbing state, fik satisfies the system of equations
M

fik =

p
j =0

ij

f jk , for i = 0, 1, , M.

subject to the conditions

fkk = 1,
fik = 0, if state i is recurrent and i k .
9 A random walk is a Markov chain with the property that if the system is in a
state i, then in a single transition the system either remains at i or moves to one
of the two states immediately adjacent to i.
A random walk often is used as a model for situations involving gambling.
9 A Gambling Example Two players (A and B), each having $2, agree to keep
playing the game and betting $1 at a time until one player is broke. The
probability of A winning a single bet is 1/3.
9 This is a Markov chain with transition matrix

P=

Jin Y. Wang

0
0
0
0
1
2 / 3 0 1 / 3 0
0

0 2 / 3 0 1/ 3 0 .

0 2 / 3 0 1 / 3
0
0
0
0
0
1
Chap16-16

College of Management, NCTU

Operation Research II

Spring, 2009

9 Starting from state 2, the probability of absorption into state 0 can be obtained
by solving for f20.
f00 = 1 (since state 0 is an absorbing state).
f10 = (2/3) f00 + (1/3)f20
f20 = (2/3) f10 + (1/3)f30
f30 = (2/3) f20 + (1/3)f40
f10 = (2/3) f00 + (1/3)f20
f40 = 0 (since sate 4 is an absorbing state)

9 The probability of A finishing with $4 when starting with $2 is obtained by

solving for f24.

9 A Credit Evaluation Example There are four categories of customers, fully

paid (state 0), 1 to 30 days in arrears (state 1), 31 to 60 days in arrears (state 2),
and bad debt (state 3), with the following transition probability matrix.
State 0

State 1

State 2

State 3

State 0

State 1

0.7

0.2

0.1

State 2

0.5

0.1

0.2

0.2

State 3

Jin Y. Wang

Chap16-17

College of Management, NCTU

Operation Research II

Spring, 2009

9 How to obtain f13 and f23?

Jin Y. Wang

Chap16-18