Markov Chains with Countably Infinite State Spaces Classification

p p p • recurrent: state i is recurrent if Fii (∞) = 1

q 0 1 2 ...
• transient: state i is transient if Fii (∞) < 1
q q q
• Note that above definition are consistent with the ones for
p = 1−q finite state M.C. (what about the other way around?)

p p p
• State 0: recurrent or transient? q 0 1 2 ...
q q q
• p<q

• p=q
p = 1−q
• p>q

Renewal theory: Interarrival time must be a r.v.

1 3

First Passage-Time Fij (∞)

• fij (n): “first passage time probability” • Fij (n) = Pij +

P
Pik Fkj (n − 1); n > 1, Fij (1) = Pij
  k6=j
– fij (n) = P Xn = j, Xn−1 6= j, ..., X1 6= j|X0 = i • Fij (n): non-decreasing in n and upper-bounded by 1 -
– Pij (Xn = j|X0 = i) fij (n) therefore limn→∞ Fij (n) exists.
P
• fij (n) = k6=j Pik fkj (n − 1), n > 1 and fij (1) = Pij
P
• Fij (∞) = Pij + k6=j Pik Fkj (∞)
We can compute fij (n) recursively
– does a solution exist?
Pn
• Fij (n) = m=1 fij (m) – is there a unique solution?
• Tij : first passage time • Lemma: Let state i be accessible from j and let j be recurrent.
– fij (n): probability mass function Then Fij (∞) = 1
– Fij (n): probability distribution functionq
• Fij (∞): If Fij (∞) = 1 then Tij is a R.V. - otherwise Tij is a
defective R.V.

2 4
 Renewal Theory Results

• Given X0 = i, let {Nij (t); t ≥ 0} be the counting process where • Lemma: Let {Njj (t); t ≥ 0} be the counting process for
Nij (t) is the number of transitions into state j by time t occurrences of state j up to time t in a Markov chain starting at
(including self-transitions). state j. The state j is recurrent if and only if

• Similarly, {Njj (t); t ≥ 0} lim Njj (t) = ∞, w.p.1

t→∞
• If j is recurrent, then {Njj (t); t ≥ 0} is a renewal process.
Also, j is recurrent if and only if
• We know (whether or not E[Tjj ] is finite) n
X
k
– limt→∞ Njj (t) = ∞, w.p. 1 lim Pjj = ∞.
n→∞
k=1
– limt→∞ E[Njj (t)] = ∞
• Lemma: If state i is recurrent and state i and j are in the same
• If j is transient, then {Njj (t); t ≥ 0} is not a renewal process.
class, the state j is recurrent.
Fjj (∞)
– Expected number of returns: 1−Fjj (∞) <∞

5 7

Renewal Theory Results

h i
n
• Pjj = E Njj (n) − Njj (n − 1) • Lemma: Let {Nij (t); t ≥ 0} be the counting process for
h i P occurrences of state j up to time t in a Markov chain starting at
• E Njj (n) = nk=1 Pjjk
state i. Then if i and j are in the same class and i and j are
recurrent, then {Nij (t); t ≥ 0} is a delayed renewal process.

6 8