Discrete-Time Markov

Chains
Professor Izhak Rubin
Electrical Engineering Department
UCLA
2014-2015 by Izhak Rubin

Discrete-Time Markov Chain:

Definition

X = {Xk, k = 0,1,2,} is a discrete time stochastic process; states assume values in a

countable state space S, such as S = {0,1,2.}, or S = {0,1,2, N), or S = {a1, a2, a3,.}.
It is said to be a Markov Chain if it satisfies the Markov Property:

Markov Property (Given present and past states, distribution of future states
is independent of the past):
P(X k+1 = j| X 0 = i 0 ,..., X k-1 = i k-1 , X k = i) = P(X k+1 = j| X k = i ) = Pk (i,j),
for each time k 1, and states (i, j , i0 ,...ik 1 ) S .
Assume a time homogeneous process: its statistical behavior is charaterized
by the (stationary) transition probability function (TPF)
Pk (i, j ) P(i, j ) P ( X k 1 j | X k i ) P ( X 1 j | X 0 i ), i,j S,k 1.
X2
Transition Probability Matrix:
Xk
X4
PT {P(i, j ), i, j S}.
X3

0
Prof. Izhak Rubin

k
2

Transition Probability Function

(TPF): Properties

Properties of PT

1. P(i, j ) 0, each i, j S ;
2.

jS

P(i, j ) 1, each i S .

Initial Distribution:

P0 (i) P( X 0 i), i S

Calculation of joint state distribution:

k

P( X 0 i0 , X1 i1 ,..., X k ik ) P0 (i0 ) P(i j 1 , i j ), for k 1,(i0 ,..., ik ) S

j 1

Prof. Izhak Rubin

Example 1: Two State Markov Chain

X= DTMC with binary RVs on state space S={0,1}

Transition probability function is given by:

1
P

0 1; 0 1

Prof. Izhak Rubin

Example 2 Binomial Counting Process;

Geometric Point Process
M k No. of arrivals

during

the

k th slot

N {N k, k 0 ) discrete-time counting process .

where
k

N k M i No arrivals in first k slots, k 1, 2,3...

i 1

N0 0 :
Assume :{M k , k 1} - i.i.d. RVs, with
1 p, j 0
P( M k j )
p, j 1
Then N is a DT Markov Chain with
1-p if j=i
P(i,j) =
p if j=i+1
Prof. Izhak Rubin

Example 2 (Cont.) Binomial Counting

Process and Geometric Point Process
Markov Property holds:
P N k 1 j | N1 n1 , N 2 n2 ,..., N k i
k
k 1

P M i j | M 1 n1 , M 2 M 2 n2 ,..., M i j
i 1
i 1

P i M k 1 j

1 p, j i

P (i, j ) N is a DT MC
p, j i 1
Associated discrete time point process A { An , n 0}, A0 0
An time (slot) of n-th occurence.
Tn An An 1 , n 1
P Tn i (1 p )i 1 p; Hence, A = DT renewal point process with intervals
that are Geometrically distributed = Geometric Point Process
Distribution of the counting variable is Binomial:
k
k n
P N k n p n 1 p , n 0,1,..., k
n
Prof. Izhak Rubin

Transient State Analysis

State distribution at time k:
Pk j P X k j
Define the m-step TPF:
P m i, j P X m j | X 0 i
Compute P m i, j recursively:
P m 1 i, j P X m 1 j , X m l | X 0 i
l S

P X m 1 j | X 0 i, X m l P X m l | X 0 i P m i, l P l , j .
l S

l S

P m 1 i, j P m i, l P l , j .

used to recursively compute the m-step TPF.

l S

We can compute the state distribution at time k by using the k-step TPF:
Pk j P0 i P k i, j .
i S

Alternatively, starting with a given initial distribution P0 i , we can proceed recursively:

Pk 1 j Pk i P i, j , k=0,1,2,....; j S.
i S

Note:

P m n i, j P m i, l P n l , j .

each m 1, n 1. (Chapman-Kolmogorov Equation)

l S

Prof. Izhak Rubin

Transient State Analysis:

Two State Markov Chain
Example: Two State discretere-time Markov Chain, X, with state space S={0,1}.
Use: Pk 1 j Pk i P i, j , k 0,
i S

to obtain
Pk 1 0 Pk 0 1 Pk 1
Pk 1 1 Pk 0 Pk 11 .
Normalization condition:
Pk 1 0 Pk 1 1 1.
Hence: Pk 1 0 Pk 0 1 .
By iteration, we obtain:

k
Pk 0 P0 0
1

, Pk 1 1 Pk 0

Note: As k : Pk 0

, Pk 1

Prof. Izhak Rubin

Steady State Distribution

Under certain conditions, the DT MC will have the limiting (steady state)
distribution:
P j lim P n i, j , j S
n

for any i S ;such that

P j 1; P j 0.
jS

We can write
lim Pk 1 j lim Pk i P i, j lim Pk i P i, j
k
k
k

iS
iS
leading to the following set of linear equations:
P j P i P i, j , j S

(1.1)

P j 1

(1.2)

iS

jS

If above set of linear equations has a unique solution j P j , j S ,

it is said to be the stationary distribution of the Markov Chain.
If the above limits exist, yielding the chain's steady state distribution,
the later is equal to the stationary distribution: P j j , j S .
Prof. Izhak Rubin

Example
Consider a DTMC X over the state space S = {0,1,2} with TPF:
0.2 0.3 0.5
P = 0.4 0.2 0.4
0.6 0.3 0.1

The stationary distribution j P j , j S is obtained by solving

P j P j P i, j , j S

(1.1)

P j 1

(1.2)

iS

jS

also written in matrix form

P , | | 1

(2)

where {P(i ), i S} is a row vector and | | P j .

jS

Prof. Izhak Rubin

10

Example (Cont.)
For this example we write:
P(0)=0.2P(0)+0.4P(1)+0.6P(2) (1)
P(1)=0.3P(0)+0.2P(1)+0.3P(2) (2)
P(2)=0.5P(0)+0.4P(1)+0.1P(2) (3)
1=P(0)+P(1)+P(2)
(4)
One of Eqs. (1) - (3) is redundant (these equations are linearly
dependent) and is not used.
We obtain the solution:
P(0)=30/77; P(1)=3/11; P(2)=26/77.
Prof. Izhak Rubin

11

Example: Discrete-Time
Birth & Death Markov Chain
A discrete-time Markov chain X={X k ,k 0}
over the state space S={0,1,2,...,}
is said to be a Discrete Time Birth-and-Death
(DTBD) process if its TPF is given by
for j i 1, i 0
i ,
,
Xk
for j i 1, i 1
i
P i, j
1 i i , for j i, i 0
0,
otherwise
where 0 0; 0 0; i 1: i 0; i 0; and i i 1 for i 0.

i = (admitted) arrival intensity at state i

i = departure intensity from state i.

Prof. Izhak Rubin

i+1 i+1
i

12

DTBD: Stationary Distribution

The set of equations for the stationary distribution becomes
P 0 P 0 1 0 P 1 1 ;

P j P j 1 j 1 P j 1 j j P j 1 j 1 , j 1.
Rearranging, we obtain the balance equations
P 1 1 P 0 0 0;
P j 1 j 1 P j j P j j P j 1 j 1 , j 1.
Hence,
P j 1 j 1 P j j 0, j 0.
Prof. Izhak Rubin

13

DT Birth & Death MC: Stationary

Distribution (Cont.)
Define: a0

1; a j

01 j 1
, j 1.
12 j

We conclude: P(j) = P(0)a j , j 0. To compute P(0), we use the normalization

condition:

j 0

j 0

1 P( j ) P(0) a j .

If

a
j 0

, We can compute P(0), so that the process is ergodic (positive recurrent),

and a unique stationary distribution P = {P(j), j S} exists; it is given by:

P j

aj

a
i 0

, j 0.

Prof. Izhak Rubin

14

Limiting Probabilities

In turn, if

a
j 0

, no stationary distribution exists;

the process is non-ergodic.

We conclude then that (when the limit exists) and the process
is non-ergodic, we have:
lim P X k j 0, j 0.

For a DTBD process, when i i 1 for some state i,

we observe the process to be aperiodic.
Then, if the process is also ergodic and thus has a stationary
distribution, given above, it also has a steady state distribution,
so that
lim P X k j P j j 0.

Prof. Izhak Rubin

15

Finite State DT Birth and Death

Markov Chain
A discrete-time Markov chain X={X k ,k 0} over the state space S={0,1,2,...N}
is said to be a finite state Discrete Time Birth-and-Death (DTBD) process if its TPF is given by
for j i 1, N i 0
i ,
,
for j i 1, i 1
i
P i, j
1 i i , for j i, i 0
0,
otherwise
where 0 0; N 0; otherwise : i 0; i 0; i i 1.
The set of equations for the stationary distribution are
written as done for the infinite state case,
yielding the same recursive formula,
yet limited for states in S:

Xk
N

P j 1 j 1 P j j 0, N-1 j 0.

Prof. Izhak Rubin

16

Finite State DTBD: Stationary

Distribution
Define:

a0

1; a j

01 j 1
, j 0.
12 j
N

Since now we always have that

a
j 0

, the process is always ergodic (positive recurrent),

and a unique stationary distribution P = {P(j), j S} always exists; it is given by:

P j

aj
N

a
i 0

, N j 0.

For a DTBD process, when i i 1 for some state i, we observe the process to
be aperiodic. Then, it also has a steady state distribution, so that
lim P X k j P j , N j 0.
k

Prof. Izhak Rubin

17