Stochastic Processes and Markov Chains

Exercise 2: Stochastic Processes and Markov Chains
Roman Dunaytsev
Department of Communications Engineering Tampere University of Technology dunaytse@cs.tut.fi
November 05, 2009
Outline
Random variables Stochastic processes Markov chains Transition matrices and probability vectors Regular matrices Probability to get from one state to another in a given number of steps Probability distribution after several steps Long-term behavior
Roman Dunaytsev (TUT)
TLT-2716, Exercise 2
November 05, 2009
2 / 40
Outline
November 05, 2009
3 / 40
Random Variables
Suppose that to each point of a sample space we assign a real number We then have a real-valued function dened on the sample space This function is called a random variable It is usually denoted by a capital letter such as X Example: Suppose that a coin is tossed twice so that the sample space is = {TT , TH, HT , HH} Let X represent the number of heads that can come up Thus, we have: X (TT ) = 0, X (TH) = 1, X (HT ) = 1, X (HH) = 2
November 05, 2009
4 / 40
Random Variables (contd)

Example: In an experiment involving 2 rolls of a die, the following are examples of random variables:
The sum of the 2 rolls The number of 6s in the 2 rolls The second roll raised to the 5th power
Example: In an experiment involving the transmission of a message, the following are examples of random variables:
The number of symbols received in error The number of retransmissions required to get an error-free copy The time needed to transmit the message
Example: You ask people whether they approve of the present government. The sample space could be: {disapprove, indierent, approve} To analyze your results, you could use X = {1, 0, 1} or X = {1, 2, 3}
Roman Dunaytsev (TUT) TLT-2716, Exercise 2 November 05, 2009 5 / 40
Random Variables (contd)

A discrete random variable maps events to values of a countable set (e.g., the integers) A continuous random variable maps events to values of an uncountable set (e.g., the real numbers) A function of a random variable denes another random variable
November 05, 2009
6 / 40
Outline
November 05, 2009
7 / 40
Stochastic Processes
A stochastic process is a collection of random variables {S(t), t T }, where t is a parameter that runs over an index set T
In general, we call t the time-parameter
Each S(t) takes values in some set E called the state space Then S(t) is the state of the process at time (or step) t E.g., S(t) may be:
The number of incoming emails at time t The balance of a bank account on day t The number of heads shown by t ips of a coin
Since any stochastic process is simply a collection of random variables, the name random process is also used for them
November 05, 2009
8 / 40
Stochastic Processes (contd)

Stochastic processes are classied in a number of ways, such as by the index set and by the state space When the index set is countable, the stochastic process is said to be a discrete-time stochastic process
I.e., T = {0, 1, 2, . . . } or T = {0, 1, 2, . . . }
When the index set is an interval of the real line, the stochastic process is said to be a continuous-time stochastic process
I.e., T = {t : t 0} or T = {t : < t < }
When the state space is countable, the stochastic process is said to be a discrete-space stochastic process When the state space is an interval of the real line, the stochastic process is said to be a continuous-space stochastic process
November 05, 2009
9 / 40
Stochastic Processes (contd)

S DT & DS S CT & DS
t
Income of a self-employed person at day t Income of an employee at time t in the course of year
DT & CS
CT & CS
t
Air temperature at noon over t days
Roman Dunaytsev (TUT) TLT-2716, Exercise 2
t
Air temperature at time t
November 05, 2009
10 / 40
Outline
November 05, 2009
11 / 40
Markov Chains and Markov Processes

A Markov chain , named after the Russian mathematician Andrey Markov, is a stochastic process for which the future behavior only depends on the present and not on the past A stochastic process has the Markov property (aka memorylessness or one-step memory) if the likelihood of a given future state, at any given moment, depends only on its present state, and not on any past states A process with this property is called Markovian or a Markov process The classic example of a Markov chain is a frog sitting in a pond full of lily pads
Each pad represents a state The frog starts on one of the pads and then jumps from lily pad to lily pad with the appropriate transition probabilities
Markov Chains
A Markov chain as a frog jumping on a set of lily pads
November 05, 2009
13 / 40
Markov Chains (contd)

A nite Markov chain is a stochastic process with a nite number of states in which the probability of being in a particular state at step n + 1 depends only on the state occupied at step n Let S = {S1 , S2 , . . . , Sm } be the possible states Let (n) = (p1 , p2 , . . . , pm ) be the vector of probabilities of each p state at step n
Here pi , i = 1, 2, . . . , m, is the probability that the process is in state Si at step n
For such a probability vector, p1 + p2 + + pm = 1 A Markov process at time n is fully dened by pij = Pr {S(n + 1) = j|S(n) = i} Where pij is the conditional probability of being in state j at step n + 1 given that the process was in state i at step n

Let P be the transition matrix (aka the stochastic matrix) Then P contains all the conditional probabilities of the Markov chain p11 . . . p1m . . .. . P= . . . . pm1 . . . pmm
A vector = (p1 , p2 , . . . , pm ) is called a probability vector if the p components are nonnegative and their sum is 1 The entries in a probability vector can represent the probabilities of nding a system in each of the states A square matrix P is called a transition matrix if each of its rows is a probability vector

Example: Which of the following vectors are probability vectors? (0.75, 0, 0.25, 0.5), (0.75, 0.5, 0, 0.25), (0.25, 0.25, 0.5)
Example: Which of the following matrices are matrices of transition probabilities? 1/3 0 2/3 0 1 0 1/4 3/4 3/4 1/2 1/4 1/2 1/6 1/3 1/3 1/3 1/3 1/3 1/3 1/3 2/3 0
November 05, 2009
16 / 40

Example: A meteorologist studying the weather in a region decides to classify each day as {sunny} and {cloudy} After analyzing several years of weather records, he nds:
The day after a sunny day is sunny 80% of the time, and cloudy 20% of the time The day after a cloudy day is sunny 60% of the time, and cloudy 40% of the time
Thus, the transition matrix and the transition diagram are as follows: P= pss pcs psc pcc
0.2 0.8
sunny cloudy
0.8 0.2 0.6 0.4
0.4
0.6

Example from Ryan ODonnells lecture: Every day I wake up and try to get a lot of work done When I am working, I am easily distracted though After each minute of work, I only keep working with probability 0.4, and with probability 0.6, I begin surng the Web After each minute of surng the Web, I only return to working with probability 0.1, and with probability 0.6 I continue surng the Web With probability 0.3 I feel kind of guilty, so I check my email, which is sort of like working After each minute of email-checking, I have probability 0.5 of coming back to work With probability 0.5 I continue checking my email
November 05, 2009
18 / 40

Missing arrows indicate zero probability
0.6 0.4
work surf
0.6 0.1 0.5 0.3 0.5
email
pw w psw P= pew
pw s pss pes
pw e 0.4 0.6 0 pse = 0.1 0.6 0.3 pee 0.5 0 0.5
November 05, 2009
19 / 40

Example: 3 boys, A, B and C, are throwing a ball to each other A always throws the ball to B and B always throws the ball to C C is just as likely to throw the ball to B as to A Find the transition matrix of the Markov chain This is a Markov chain since the person throwing the ball is not inuenced by those who previously had the ball paa pab pac 0 1 0 0 1 P = pba pbb pbc = 0 pca pcb pcc 0.5 0.5 0 The rst row of the matrix corresponds to the fact that A always throws the ball to B The second row corresponds to the fact that B always throws it to C The last row corresponds to the fact that C throws the ball to A or B with equal probability, and does not throw it to himself

Let us consider a particle which moves in a straight line in unit steps Each step is one unit to the right with probability p or one unit to the left with probability q It moves until it reaches one of two extreme points which are called boundary points We consider the case of 5 states: S1 , S2 , S3 , S4 , S5
States S1 and S5 are the boundary states Then states S2 , S3 , and S4 are interior states
S1 S1 p11 S2 . . . S3 . . . S4 . . . S5 p51
S2 ... ... ... ... ...
S3 ... ... ... ... ...
S4 ... ... ... ... ...
S5 p15 ... ... ... p55

November 05, 2009 21 / 40

Example: Let us assume that once particle remains there 1 0 q 0 P = 0 q 0 0 0 0 state S1 or S5 is reached, the 0 p 0 q 0 0 0 p 0 0 0 0 0 p 1
A state Si of a Markov chain is called absorbing if it is impossible to leave it (i.e., pii = 1) A Markov chain is absorbing if it has at least one absorbing state, and if from every state it is possible to go to an absorbing state (not necessarily in one step) In an absorbing Markov chain, a state which is not absorbing is called transient

Example: Assume that the particle is reected when it reaches a boundary point and returns to the point from which it came Thus, if it reaches S1 , it goes on the next step back to S2 If it hits S5 , it goes on the next step back to S4 Find the matrix of transition probabilities 0 q P = 0 0 0 Since q + p = 1, then q = 1 p 1 0 q 0 0 0 p 0 q 0 0 0 p 0 1 0 0 0 p 0
November 05, 2009
23 / 40

Example: Assume that whenever the particle hits one of the boundary states, it goes directly to the center state S3 We may think of this as the process started at state S3 and repeated each time the boundary is reached Find the matrix of transition probabilities 0 q P = 0 0 0 q =1p 0 0 q 0 0 1 p 0 q 1 0 0 p 0 0 0 0 0 p 0
November 05, 2009
24 / 40

Example: Assume that once a boundary state is reached, the particle stays at this state with probability 0.5 and moves to the other boundary state with probability 0.5 Find the matrix of transition probabilities 0.5 q P= 0 0 0.5 q =1p 0 0 q 0 0 0 p 0 q 0 0 0.5 0 0 p 0 0 p 0 0.5
November 05, 2009
25 / 40

A matrix P is called a stochastic matrix, if it does not contain any negative entries and the sum of each row of the matrix is equal to 1 The product of 2 stochastic matrices is again a stochastic matrix Therefore, all powers of stochastic matrices P n are stochastic matrices A stochastic matrix P is said to be regular if all elements of at least one particular power of P are positive and dierent from 0
November 05, 2009
26 / 40
Matrix Multiplication
In order for matrix multiplication to be dened, the dimensions of the matrices must satisfy (n m)(m p) = (n p) The product C of 2 matrices A and B is dened as
a1,1 a1,2 a2,1 a2,2 a3,1 a3,2 A x c1,1 c1,2 c1,3 = c2,1 c2,2 c2,3 c3,1 c3,2 c3,3 C
b1,1 b1,2 b1,3 b2,1 b2,2 b2,3 B
That is c1,2 = a1,1 b1,2 + a1,2 b2,2 c3,3 = a3,1 b1,3 + a3,2 b2,3
November 05, 2009
27 / 40

Example: Let A be a stochastic matrix: A= Let us calculate A2 : A2 = Thus, we get: A2 = a1,1 a1,1 + a1,2 a2,1 a1,1 a1,2 + a1,2 a2,2 a2,1 a1,1 + a2,2 a2,1 a2,1 a1,2 + a2,2 a2,2 = 0.5 0.5 0.25 0.75 a1,1 a1,2 a2,1 a2,2 a1,1 a1,2 a2,1 a2,2 a1,1 a1,2 a2,1 a2,2 = 0 1 0.5 0.5
Hence, A is regular
November 05, 2009
28 / 40

The entry pij in the transition matrix P of a Markov chain is the probability that the system changes from state Si to state Sj in 1 step p11 . . . p1m . . .. . P= . . . . pm1 . . . pmm
What is the probability that the system changes from state Si to state Sj in exactly n steps?
As a rule, this probability is denoted as pij
(n)
The probability of going from any state Si to another state Sj in a nite Markov chain with the transition matrix P in n steps is given by the element (i, j) of the matrix P n

Example: The Land of Oz is blessed by many things, but not by good weather:
They never have 2 nice days in a row If they have a nice day, they are just as likely to have snow as rain the next day If they have snow or rain, they have an even chance of having the same the next day If there is change from snow or rain, only half of the time is this a change to a nice day
With this information let us form a Markov chain Let us denote as states the kinds of weather: {rain}, {nice}, and {snow} From the above information we determine the transition matrix: pr r pr n pr s 1/2 1/4 1/4 P = pnr pnn pns = 1/2 0 1/2 psr psn pss 1/4 1/4 1/2

We consider the question of determining the probability that, given the chain is in state Si today, it will be in state Sj n days from now (n) (pij ) Example: Let us determine the probability that if it is rainy today (2) then the event that it is snowy 2 days from now (pr s ) We have the transition matrix: 1/2 1/4 1/4 p1,1 p1,2 p1,3 pr r pr n pr s pnr pnn pns = p2,1 p2,2 p2,3 = 1/2 0 1/2 p3,1 p3,2 p3,3 1/4 1/4 1/2 psr psn pss Then 3 (2) (2) (2) (2) P 2 = p2,1 p2,2 p2,3 p1,3 = p1,1 p1,3 + p1,2 p2,3 + p1,3 p3,3 = 8 (2) (2) (2) p3,1 p3,2 p3,3
p1,1 p1,2 p1,3
(2)
(2)
(2)

Consider a Markov chain process with the transition matrix P (0) (0) (0) Let (0) = (p , p , . . . , p ) be the initial probability distribution p
1 2 m (n) (n) (n) And let (n) = (p1 , p2 , . . . , pm ) be the probability distribution at p step n
Then the probability distribution at step n can be found as (n) = (0) P n p p That is (1) = (0) P p p (2) = (1) P = (0) P 2 p p p ... (n) = (n1) P = (0) P n p p p

Example: In the Land of Oz pr r pr n P = pnr pnn psr psn example, we have: pr s 0.5 0.25 0.25 pns = 0.5 0 0.5 pss 0.25 0.25 0.5
Let the initial probability vector be (0) = (1/3, 1/3, 1/3) p Hence, the probability distribution of the states after 3 days is (3) = (0) P 3 : p p 0.406 0.203 0.391 (1/3, 1/3, 1/3) 0.406 0.188 0.406 = (0.401, 0.198, 0.401) 0.391 0.203 0.406

Example: Every day, a man either catches a bus or drives his car to work Suppose he never goes by bus 2 days in a row But if he drives to work, then the next day he is just as likely to drive again as he is to travel by bus The state space of the system is {bus} and {car} This stochastic process is a Markov chain since the outcome on any day depends only on what happened the preceding day pbb pbc pcb pcc 0 1 1/2 1/2
P=
November 05, 2009
34 / 40

Example: Suppose now that on the rst day the man tosses a fair die and drives to work if and only if a 6 appeared Then the initial probability distribution is given by (0) = (5/6, 1/6), p P= pbb pbc pcb pcc = 0 1 1/2 1/2
Find the probability distribution after 4 days We have that (4) = (0) P 4 p p Then (5/6, 1/6) 3/8 5/8 5/16 11/16 = (35/96, 61/96)
Thus, the probability of traveling to work by bus is 35/96 and driving to work is 61/96

Let P be the transition matrix for a regular chain Then, as n , the powers P n approach a limiting matrix W , where all rows approach the same vector w is a steady-state vector The vector w A regular Markov chain has only one steady-state vector The steady-state vector gives w the long-term probability distribution of the states of the Markov chain = P, w w wi = 1
i
Thus, if a Markov system is regular, then its long-term transition matrix is given by the square matrix whose rows are all the same and equal to the steady-state vector

Example from Ryan ODonnells lecture: After 1 minute: pw w pw s pw e 0.4 0.6 0 P = psw pss pse = 0.1 0.6 0.3 pew pes pee 0.5 0 0.5 After 10 minutes we get: 0.2940 0.4413 0.2648 P 10 0.2942 0.4411 0.2648 0.2942 0.4413 0.2648 And after 1 hour (i.e., P 60 ) the result is almost the same 0.294117647058823 0.441176470588235 0.264705882352941 0.294117647068823 0.441176470588235 0.264705882352941 0.294117647068823 0.441176470588235 0.264705882352941

Example: After 1 day, the weather in the Land of Oz pr r pr n pr s 0.5 0.25 pnr pnn pns = 0.5 0 P= psr psn pss 0.25 0.25 After 1 week: is given by 0.25 0.5 0.5
0.4 0.2 0.4 P 7 = 0.4 0.2 0.4 0.4 0.2 0.4 Let the initial probability vector (0) be either (1/3, 1/3, 1/3) or p (1/10, 8/10, 1/10) Even then, after 1 week the result is the same (7) = (0) P 7 = (0.4, 0.2, 0.4) p p Hence, in the long run, the starting state does not really matter

If the higher and higher powers of P approach a xed matrix P , we refer to P as the steady-state or long-term transition matrix It is often possible to approximate P with great accuracy by simply computing a very large power of P
Q: How large? A: You know it is large enough when the rows are all the same with the accuracy you desire
Matrix Algebra Tool: http://people.hofstra.edu/stefan waner/RealWorld/matrixalgebra/ fancymatrixalg2.html
November 05, 2009
39 / 40

Example: For the Land of Oz, the transition matrix is pr r pr n pr s 0.5 0.25 0.25 0 0.5 P = pnr pnn pns = 0.5 psr psn pss 0.25 0.25 0.5 The steady-state vector is given by = (0.4, 0.2, 0.4), so w 0.5 0.25 0.25 P = (0.4, 0.2, 0.4) 0.5 0 0.5 = (0.4, 0.2, 0.4) = w w 0.25 0.25 0.5 Therefore, the long-term transition matrix is 0.4 0.2 0.4 W = P = 0.4 0.2 0.4 0.4 0.2 0.4

Stochastic Processes and Markov Chains

Caricato da

Informazioni sul documento

Descrizione originale:

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Stochastic Processes and Markov Chains

Caricato da

Copyright:

Formati disponibili

Exercise 2: Stochastic Processes and Markov Chains

November 05, 2009

Roman Dunaytsev (TUT)

November 05, 2009

Roman Dunaytsev (TUT)

November 05, 2009

Roman Dunaytsev (TUT)

November 05, 2009

Random Variables (contd)

Random Variables (contd)

Roman Dunaytsev (TUT)

November 05, 2009

Roman Dunaytsev (TUT)

November 05, 2009

Roman Dunaytsev (TUT)

November 05, 2009

Stochastic Processes (contd)

Roman Dunaytsev (TUT)

November 05, 2009

Stochastic Processes (contd)

November 05, 2009

Roman Dunaytsev (TUT)

November 05, 2009

Markov Chains and Markov Processes

Roman Dunaytsev (TUT)

November 05, 2009

Markov Chains (contd)

Markov Chains (contd)

Markov Chains (contd)

Roman Dunaytsev (TUT)

November 05, 2009

Markov Chains (contd)

0.8 0.2 0.6 0.4

Markov Chains (contd)

Roman Dunaytsev (TUT)

November 05, 2009

Markov Chains (contd)

0.6 0.1 0.5 0.3 0.5

pw e 0.4 0.6 0 pse = 0.1 0.6 0.3 pee 0.5 0 0.5

Roman Dunaytsev (TUT)

November 05, 2009

Markov Chains (contd)

Markov Chains (contd)

S2 ... ... ... ... ...

S3 ... ... ... ... ...

S4 ... ... ... ... ...

S5 p15 ... ... ... p55

Markov Chains (contd)

Markov Chains (contd)

Roman Dunaytsev (TUT)

November 05, 2009

Markov Chains (contd)

Roman Dunaytsev (TUT)

November 05, 2009

Markov Chains (contd)

Roman Dunaytsev (TUT)

November 05, 2009

Markov Chains (contd)

Roman Dunaytsev (TUT)

November 05, 2009

b1,1 b1,2 b1,3 b2,1 b2,2 b2,3 B

Roman Dunaytsev (TUT)

November 05, 2009