Markov processes (Markov chains)

Construct a Bayes net from these variables: parents?

Markov assumption: Xt depends on bounded subset of X0:t−1
Temporal probability models First-order Markov process: P(Xt|X0:t−1) = P(Xt|Xt−1)
Second-order Markov process: P(Xt|X0:t−1) = P(Xt|Xt−2, Xt−1)
First−order X t −2 X t −1 Xt X t +1 X t +2
Chapter 15, Sections 1–5
Second−order X t −2 X t −1 Xt X t +1 X t +2
Sensor Markov assumption: P(Et|X0:t, E0:t−1) = P(Et|Xt)
Stationary process: transition model P(Xt|Xt−1) and
sensor model P(Et|Xt) fixed for all t
Chapter 15, Sections 1–5 1 Chapter 15, Sections 1–5 4
Outline Example
♦ Time and uncertainty R t −1 P(R t )
t 0.7
f
♦ Inference: filtering, prediction, smoothing 0.3
Raint −1 Raint Raint +1
♦ Hidden Markov models Rt P(U t )
t 0.9
♦ Kalman filters (a brief mention) f 0.2
Umbrella t −1 Umbrella t Umbrella t +1
♦ Dynamic Bayesian networks
♦ Particle filtering First-order Markov assumption not exactly true in real world!
Possible fixes:
1. Increase order of Markov process
2. Augment state, e.g., add T empt, P ressuret
Example: robot motion.
Augment position and velocity with Batteryt
Chapter 15, Sections 1–5 2 Chapter 15, Sections 1–5 5
Time and uncertainty Inference tasks
The world changes; we need to track and predict it Filtering: P(Xt|e1:t)
belief state—input to the decision process of a rational agent
Diabetes management vs vehicle diagnosis
Prediction: P(Xt+k |e1:t) for k > 0
Basic idea: copy state and evidence variables for each time step evaluation of possible action sequences;
Xt = set of unobservable state variables at time t like filtering without the evidence
e.g., BloodSugart, StomachContentst, etc. Smoothing: P(Xk |e1:t) for 0 ≤ k < t
Et = set of observable evidence variables at time t better estimate of past states, essential for learning
e.g., M easuredBloodSugart, P ulseRatet , F oodEatent Most likely explanation: arg maxx1:t P (x1:t|e1:t)
This assumes discrete time; step size depends on problem speech recognition, decoding with a noisy channel
Notation: Xa:b = Xa, Xa+1, . . . , Xb−1, Xb
Chapter 15, Sections 1–5 3 Chapter 15, Sections 1–5 6
 Filtering Smoothing example
0.500 0.627
Aim: devise a recursive state estimation algorithm: 0.500 0.373
True 0.500 0.818 0.883
P(Xt+1|e1:t+1) = f (et+1, P(Xt|e1:t)) False 0.500 0.182 0.117
forward
0.883 0.883
smoothed
0.117 0.117
P(Xt+1|e1:t+1) = P(Xt+1|e1:t, et+1) 0.690 1.000
backward
0.410 1.000
= αP(et+1|Xt+1, e1:t)P(Xt+1|e1:t)
= αP(et+1|Xt+1)P(Xt+1|e1:t) Rain 0 Rain 1 Rain 2
I.e., prediction + estimation. Prediction by summing out Xt:
P(Xt+1|e1:t+1) = αP(et+1|Xt+1)Σxt P(Xt+1|xt, e1:t)P (xt|e1:t) Umbrella 1 Umbrella 2
= αP(et+1|Xt+1)Σxt P(Xt+1|xt)P (xt|e1:t)
f1:t+1 = Forward(f1:t, et+1) where f1:t = P(Xt|e1:t) Forward–backward algorithm: cache forward messages along the way
Time and space constant (independent of t) Time linear in t (polytree inference), space O(t|f|)
Chapter 15, Sections 1–5 7 Chapter 15, Sections 1–5 10
Filtering example Most likely explanation
0.500 0.627 Most likely sequence 6= sequence of most likely states!!!!
0.500 0.373
Most likely path to each xt+1
True 0.500 0.818 0.883 = most likely path to some xt plus one more step
False 0.500 0.182 0.117
max P(x1, . . . , xt, Xt+1|e1:t+1)
x1...xt
Rain 0 Rain 1 Rain 2  
= P(et+1|Xt+1) max
x
P(X
t+1 |xt ) x max
...x
P (x1, . . . , xt−1, xt|e1:t)
t 1 t−1
Identical to filtering, except f1:t replaced by
Umbrella 1 Umbrella 2
m1:t = x max
...x
P(x1, . . . , xt−1, Xt|e1:t),
1 t−1
I.e., m1:t(i) gives the probability of the most likely path to state i.
Update has sum replaced by max, giving the Viterbi algorithm:
m1:t+1 = P(et+1|Xt+1) max
x
(P(Xt+1|xt)m1:t)
t
Chapter 15, Sections 1–5 8 Chapter 15, Sections 1–5 11
Smoothing Viterbi example
X0 X1 Xk Xt
Rain 1 Rain 2 Rain 3 Rain 4 Rain 5
E1 Ek Et state true true true true true
space
paths
Divide evidence e1:t into e1:k , ek+1:t: false false false false false
P(Xk |e1:t) = P(Xk |e1:k , ek+1:t) umbrella true true false true true
= αP(Xk |e1:k )P(ek+1:t|Xk , e1:k ) .8182 .5155 .0361 .0334 .0210
most
= αP(Xk |e1:k )P(ek+1:t|Xk ) likely
paths
= αf1:k bk+1:t .1818 .0491 .1237 .0173 .0024
m 1:1 m 1:2 m 1:3 m 1:4 m 1:5
Backward message computed by a backwards recursion:
P(ek+1:t|Xk ) = Σxk+1 P(ek+1:t|Xk , xk+1)P(xk+1|Xk )
= Σxk+1 P (ek+1:t|xk+1)P(xk+1|Xk )
= Σxk+1 P (ek+1|xk+1)P (ek+2:t|xk+1)P(xk+1|Xk )
Chapter 15, Sections 1–5 9 Chapter 15, Sections 1–5 12
 Hidden Markov models Country dance algorithm
Xt is a single, discrete variable (usually Et is too) Can avoid storing all forward messages in smoothing by running
Domain of Xt is {1, . . . , S} forward algorithm backwards:
f1:t+1 = αOt+1T>f1:t
 
0.7 0.3 
Transition matrix Tij = P (Xt = j|Xt−1 = i), e.g.,


0.3 0.7 −1 >
 
Ot+1 f 1:t+1 = αT f1:t
Sensor matrix Ot for each time step, diagonal elements P (et|Xt = i) α0(T>)−1Ot+1
−1
f1:t+1 = f1:t

0.9 0 
e.g., with U1 = true, O1 = 

Algorithm: forward pass computes ft, backward pass does fi, bi
0 0.2

Forward and backward messages as column vectors:
f1:t+1 = αOt+1T>f1:t
bk+1:t = TOk+1bk+2:t
Forward-backward algorithm needs time O(S 2t) and space O(St)
Chapter 15, Sections 1–5 13 Chapter 15, Sections 1–5 16
Country dance algorithm Country dance algorithm
Can avoid storing all forward messages in smoothing by running Can avoid storing all forward messages in smoothing by running
forward algorithm backwards: forward algorithm backwards:
f1:t+1 = αOt+1T>f1:t f1:t+1 = αOt+1T>f1:t
−1 > −1 >
Ot+1 f 1:t+1 = αT f1:t Ot+1 f 1:t+1 = αT f1:t
α0(T>)−1Ot+1
−1
f1:t+1 = f1:t α0(T>)−1Ot+1
−1
f1:t+1 = f1:t
Algorithm: forward pass computes ft, backward pass does fi, bi Algorithm: forward pass computes ft, backward pass does fi, bi
Chapter 15, Sections 1–5 14 Chapter 15, Sections 1–5 17
Country dance algorithm Country dance algorithm
Can avoid storing all forward messages in smoothing by running Can avoid storing all forward messages in smoothing by running
forward algorithm backwards: forward algorithm backwards:
f1:t+1 = αOt+1T>f1:t f1:t+1 = αOt+1T>f1:t
−1 > −1 >
Ot+1 f 1:t+1 = αT f1:t Ot+1 f 1:t+1 = αT f1:t
α0(T>)−1Ot+1
−1
f1:t+1 = f1:t α0(T>)−1Ot+1
−1
f1:t+1 = f1:t
Algorithm: forward pass computes ft, backward pass does fi, bi Algorithm: forward pass computes ft, backward pass does fi, bi
Chapter 15, Sections 1–5 15 Chapter 15, Sections 1–5 18
 Country dance algorithm Country dance algorithm
Can avoid storing all forward messages in smoothing by running Can avoid storing all forward messages in smoothing by running
forward algorithm backwards: forward algorithm backwards:
f1:t+1 = αOt+1T>f1:t f1:t+1 = αOt+1T>f1:t
−1 > −1 >
Ot+1 f 1:t+1 = αT f1:t Ot+1 f 1:t+1 = αT f1:t
α0(T>)−1Ot+1
−1
f1:t+1 = f1:t α0(T>)−1Ot+1
−1
f1:t+1 = f1:t
Algorithm: forward pass computes ft, backward pass does fi, bi Algorithm: forward pass computes ft, backward pass does fi, bi
Chapter 15, Sections 1–5 19 Chapter 15, Sections 1–5 22
Country dance algorithm Country dance algorithm
Can avoid storing all forward messages in smoothing by running Can avoid storing all forward messages in smoothing by running
forward algorithm backwards: forward algorithm backwards:
f1:t+1 = αOt+1T>f1:t f1:t+1 = αOt+1T>f1:t
−1 > −1 >
Ot+1 f 1:t+1 = αT f1:t Ot+1 f 1:t+1 = αT f1:t
α0(T>)−1Ot+1
−1
f1:t+1 = f1:t α0(T>)−1Ot+1
−1
f1:t+1 = f1:t
Algorithm: forward pass computes ft, backward pass does fi, bi Algorithm: forward pass computes ft, backward pass does fi, bi
Chapter 15, Sections 1–5 20 Chapter 15, Sections 1–5 23
Country dance algorithm Kalman filters
Can avoid storing all forward messages in smoothing by running Modelling systems described by a set of continuous variables,
forward algorithm backwards: e.g., tracking a bird flying—Xt = X, Y, Z, Ẋ, Ẏ , Ż.
Airplanes, robots, ecosystems, economies, chemical plants, planets, . . .
f1:t+1 = αOt+1T>f1:t
−1 >
Ot+1 f 1:t+1 = αT f1:t Xt X t+1
α0(T>)−1Ot+1
−1
f1:t+1 = f1:t
Algorithm: forward pass computes ft, backward pass does fi, bi Xt X t+1
Zt Zt+1
Gaussian prior, linear Gaussian transition model and sensor model
Chapter 15, Sections 1–5 21 Chapter 15, Sections 1–5 24
 Updating Gaussian distributions 2-D tracking example: filtering
Prediction step: if P(Xt|e1:t) is Gaussian, then prediction 2D filtering
Z 12
P(Xt+1|e1:t) = xt P(Xt+1|xt)P (xt|e1:t) dxt true
observed
is Gaussian. If P(Xt+1|e1:t) is Gaussian, then the updated distribution 11 filtered
P(Xt+1|e1:t+1) = αP(et+1|Xt+1)P(Xt+1|e1:t)
10
is Gaussian
Hence P(Xt|e1:t) is multivariate Gaussian N (µt, Σt) for all t 9
Y

General (nonlinear, non-Gaussian) process: description of posterior grows

8
unboundedly as t → ∞
7
6
8 10 12 14 16 18 20 22 24 26
X
Chapter 15, Sections 1–5 25 Chapter 15, Sections 1–5 28
Simple 1-D example 2-D tracking example: smoothing
Gaussian random walk on X–axis, s.d. σx, sensor s.d. σz 2D smoothing
12
(σt2 + σx2)zt+1 + σz2µt 2 (σt2 + σx2)σz2
µt+1 = σt+1 = true
σt2 + σx2 + σz2 σt2 + σx2 + σz2 observed
smoothed
11
0.45
0.4 10
0.35
P(x1 | z1=2.5)
0.3
9
Y

0.25 P(x0)
P(X)

0.2
8
0.15
0.1 P(x1)
7
0.05
0 *
z1
-8 -6 -4 -2 0 2 4 6 8
6
8 10 12 14 16 18 20 22 24 26
X position X
Chapter 15, Sections 1–5 26 Chapter 15, Sections 1–5 29
General Kalman update Where it breaks
Transition and sensor models: Cannot be applied if the transition model is nonlinear
P (xt+1|xt) = N (Fxt, Σx)(xt+1) Extended Kalman Filter models transition as locally linear around xt = µt
P (zt|xt) = N (Hxt, Σz )(zt) Fails if systems is locally unsmooth
F is the matrix for the transition; Σx the transition noise covariance
H is the matrix for the sensors; Σz the sensor noise covariance
Filter computes the following update:
µt+1 = Fµt + Kt+1(zt+1 − HFµt)
Σt+1 = (I − Kt+1)(FΣtF> + Σx)
where Kt+1 = (FΣtF> + Σx)H>(H(FΣtF> + Σx)H> + Σz )−1
is the Kalman gain matrix
Σt and Kt are independent of observation sequence, so compute offline
Chapter 15, Sections 1–5 27 Chapter 15, Sections 1–5 30
 Dynamic Bayesian networks Exact inference in DBNs
Xt, Et contain arbitrarily many variables in a replicated Bayes net Naive method: unroll the network and run any exact algorithm
BMeter 1 P(R 0)
0.7

R0
t
f

P(R 1 )
0.7
0.3

P(R 0)
0.7

R0
t
f

P(R 1 )
0.7
0.3

R0
t
f

P(R 1 )
0.7
0.3

R0
t
f

P(R 1 )
0.7
0.3

R0
t
f

P(R 1 )
0.7
0.3

R0
t
f

P(R 1 )
0.7
0.3

R0
t
f

P(R 1 )
0.7
0.3

R0
t
f

P(R 1 )
0.7
0.3
Rain 0 Rain 1 Rain 0 Rain 1 Rain 2 Rain 3 Rain 4 Rain 5 Rain 6 Rain 7
P(R 0) R0 P(R 1 ) R1 P(U1 ) R1 P(U1 ) R1 P(U1 ) R1 P(U1 ) R1 P(U1 ) R1 P(U1 ) R1 P(U1 ) R1 P(U1 )
t 0.9 t 0.9 t 0.9 t 0.9 t 0.9 t 0.9 t 0.9 t 0.9
f 0.2 f 0.2 f 0.2 f 0.2 f 0.2 f 0.2 f 0.2 f 0.2
t 0.7
0.7 f 0.3 Battery 0 Battery 1 Umbrella 1 Umbrella 1 Umbrella 2 Umbrella 3 Umbrella 4 Umbrella 5 Umbrella 6 Umbrella 7
Rain 0 Rain 1
R1 P(U1 ) X0 X1 Problem: inference cost for each update grows with t
t 0.9
f 0.2
Umbrella 1 XX
t0 X1 Rollup filtering: add slice t + 1, “sum out” slice t using variable elimination
Largest factor is O(dn+1), update cost O(dn+2)
Z1
(cf. HMM update cost O(d2n))
Chapter 15, Sections 1–5 31 Chapter 15, Sections 1–5 34
DBNs vs. HMMs Likelihood weighting for DBNs
Every HMM is a single-variable DBN; every discrete DBN is an HMM Set of weighted samples approximates the belief state
Rain 0 Rain 1 Rain 2 Rain 3 Rain 4 Rain 5
Xt X t+1
Umbrella 1 Umbrella 2 Umbrella 3 Umbrella 4 Umbrella 5
Yt Y t+1 LW samples pay no attention to the evidence!
⇒ fraction “agreeing” falls exponentially with t
⇒ number of samples required grows exponentially with t
Z t Z t+1 1
LW(10)
LW(100)
0.8 LW(1000)
LW(10000)
Sparse dependencies ⇒ exponentially fewer parameters;

RMS error
0.6
e.g., 20 state variables, three parents each 0.4
DBN has 20 × 23 = 160 parameters, HMM has 220 × 220 ≈ 1012 0.2
0
0 5 10 15 20 25 30 35 40 45 50
Time step
Chapter 15, Sections 1–5 32 Chapter 15, Sections 1–5 35
DBNs vs Kalman filters Particle filtering
Every Kalman filter model is a DBN, but few DBNs are KFs; Basic idea: ensure that the population of samples (“particles”)
real world requires non-Gaussian posteriors tracks the high-likelihood regions of the state-space
E.g., where are bin Laden and my keys? What’s the battery charge? Replicate particles proportional to likelihood for et
BMBroken 0 BMBroken 1 Rain t Rain t +1 Rain t +1 Rain t +1
true
BMeter 1
false
E(Battery|...5555005555...)
5
Battery 0 Battery 1
4 E(Battery|...5555000000...) (a) Propagate (b) Weight (c) Resample

E(Battery)
3
X0 X1
2
Widely used for tracking nonlinear systems, esp. in vision
P(BMBroken|...5555000000...)
1
XX
t0 X1 Also used for simultaneous localization and mapping in mobile robots
0
P(BMBroken|...5555005555...) 105-dimensional state space
-1
Z1 15 20 25 30
Time step
Chapter 15, Sections 1–5 33 Chapter 15, Sections 1–5 36
 Particle filtering contd.
Assume consistent at time t: N (xt|e1:t)/N = P (xt|e1:t)
Propagate forward: populations of xt+1 are
N (xt+1|e1:t) = Σxt P (xt+1|xt)N (xt|e1:t)
Weight samples by their likelihood for et+1:
W (xt+1|e1:t+1) = P (et+1|xt+1)N (xt+1|e1:t)
Resample to obtain populations proportional to W :
N (xt+1|e1:t+1)/N = αW (xt+1|e1:t+1) = αP (et+1|xt+1)N (xt+1|e1:t)
= αP (et+1|xt+1)Σxt P (xt+1|xt)N (xt|e1:t)
= α0P (et+1|xt+1)Σxt P (xt+1|xt)P (xt|e1:t)
= P (xt+1|e1:t+1)
Chapter 15, Sections 1–5 37
Particle filtering performance
Approximation error of particle filtering remains bounded over time,
at least empirically—theoretical analysis is difficult
1
LW(25)
LW(100)
LW(1000)
0.8 LW(10000)
ER/SOF(25)
Avg absolute error

0.6
0.4
0.2
0
0 5 10 15 20 25 30 35 40 45 50
Time step
Chapter 15, Sections 1–5 38
Summary
Temporal models use state and sensor variables replicated over time
Markov assumptions and stationarity assumption, so we need
– transition modelP(Xt|Xt−1)
– sensor model P(Et|Xt)
Tasks are filtering, prediction, smoothing, most likely sequence;
all done recursively with constant cost per time step
Hidden Markov models have a single discrete state variable; used
for speech recognition
Kalman filters allow n state variables, linear Gaussian, O(n3) update
Dynamic Bayes nets subsume HMMs, Kalman filters; exact update intractable
Particle filtering is a good approximate filtering algorithm for DBNs
Chapter 15, Sections 1–5 39