PHASE - Solve problems by applying the algorithms of Unit 3

Presented by

Presented a

Course

THEORY OF DECISIONS

Group:

212066

UNIVERSIDAD NACIONAL ABIERTA Y A DISTANCIA

UNAD
May-2019
 Introduction
A Markov decision process (MDP) is a discrete time stochastic control process.
It provides a mathematical framework for modeling decision making in situations
where outcomes are partly random and partly under the control of a decision
maker. MDPs are useful for studying optimization problems solved via dynamic
programming and reinforcement learning. MDPs were known at least as early
as the 1950; a core body of research on Markov decision processes resulted
from Howard's 1960 book, Dynamic Programming and Markov Processes. They
are used in many disciplines, including robotics, automatic control, economics
and manufacturing. The name of MDPs comes from the Russian mathematician
Andrey Markov.
For more understanding of the unit we will solve some exercises.
 Problem 1. Markov chains (steady state):
XYZ insurance company charges its customers according to their accident
history. If you have not had accidents the last two years will be charged for the
new policy $ 715,000 (state 0); if you have had an accident in each of the last
two years you will be charged $ 835,000 (State 1); If you had accidents the first
of the last two years you will be charged $ 789.000 (state 2) and if you had an
accident the second of the last two years will be charged $ 813.000 (State 3).
The historical behavior of each state is given by the following cases of accident,
taken in four different events.

PART 13. Markov chains (steady state):

According to Table 1 by applying the Markovian processes, i.e. finding the
transition matrix and solving the respective equations of p * q, where p is the
transition matrix and q the vector [W X Y Z]. Answer:
a. What is the transition matrix resulting from proportionality according to
the accident history?
b. What is the average premium paid by a customer in Payoff, according to
historical accident rate?

Solving the four states

E0 If youhave not had accidents the last two years will be charged for the new policy 715.000 $
E1 if you have had an accident in each of the last two years you will be charged 835.000 $
E2 If you had accidents the first of the last two years you will be charged 789.000 $
E3 f you had an accident the second of the last two years will be charged 813.000 $

ACCIDENTS IN THE YEAR

STATE E0 E1 E2 E3 TOTAL
E0 920 1380 1840 460 4600
E1 1740 0 1160 2900 5800
E2 900 900 1800 900 4500
E3 1140 1520 0 1140 3800

STATE E0 E1 E2 E3
E0 0,2 0,3 0,4 0,1
E1 0,3 0 0,2 0,5
E2 0,2 0,2 0,4 0,2
E3 0,3 0,4 0 0,3

Transition probability matrix

∑ q*p=
W 0,2 0,3 0,4 0,1 = 1 EC1 0,2W+0,3X+0,2Y+0,3Z = W
X q= 0,3 0 0,2 0,5 = 1 p= [ W X Y Z] EC2 0,3W+0X+0,2Y+0,4Z = X
Y 0,2 0,2 0,4 0,2 = 1 EC3 0,4W+0,2X+0,4Y+0Z = Y
Z 0,3 0,4 0 0,3 = 1 EC4 0,1W+0,5X+0,2Y+0,3Z = Z
EC5 W+X+Y+Z = 1
 EC1 0,2W-W+0,3X+0,2Y+0,3Z = 0
EC2 0,3W+0X-X+0,2Y+0,4Z = 0
EC3 0,4W+0,2X+0,4Y-Y+0Z = 0
EC4 0,1W+0,5X+0,2Y+0,3Z-Z = 0
EC5 W+X+Y+Z -1 = 0

EC1 -0,8W+0,3X+0,2Y+0,3Z = 0
EC2 0,3W-1X+0,2Y+0,4Z = 0
EC3 0,4W+0,2X-0,6Y+0Z = 0
EC4 0,1W+0,5X+0,2Y-0,7Z = 0
EC5 W+X+Y+Z -1 = 0

E0 E1 E2 E3
W X Y Z
0,25049 0,23288 0,244618 0,27202

COEFFICIENTS
W X Y Z VALUE IND Equal to
-0,8 0,3 0,2 0,3 0 0,00000
0,3 -1 0,2 0,4 0 0,00000
0,4 0,2 -0,6 0 0 0,00000
0,1 0,5 0,2 -0,7 0 0,00000
1 1 1 1 -1 0,00000

After the average premium paid in the insurance company WXYZ is: 0.25049 *
($ 715,000) +0.23288 * ($ 835,000) +0.24462 * ($ 789,000) +0.27202 * ($
813,000)
787.704,50 $

Problem 2. Markov chains (Initial state multiplication):

In Colombia there are 5 main mobile operators such as Tigo, Comcel, Movistar,
ETB and Uff, which we will call states. The following chart summarizes the odds
that each client must stay in their current operator or make a change of
company.
The current percentages of each operator in the current market are for Tigo 0.3
for Comcel 0.2, for Movistar 0.3, for ETB 0.1 and 0.1 for Uff (initial state).

STATE TIGO COMCEL MOVISTAR ETB UFF TOTAL

TIGO 0,1 0,2 0,4 0,1 0,2 1
COMCEL 0,3 0,2 0,1 0,2 0,2 1
MOVISTAR 0,1 0,3 0,2 0,2 0,2 1
 ETB 0,1 0,3 0,2 0,1 0,3 1
UFF 0,1 0,2 0,3 0,3 0,1 1

INITIAL STATE
TIGO COMCEL MOVISTAR ETB UFF
0,3 0,2 0,3 0,1 0,1

PART 14. Markov Chains (Initial State Multiplication):

Find the probability that each user stays with the mobile company for the next
period.

TIGO COMCEL MOVISTAR ETB UFF

Probabilities 0,14 0,24 0,25 0,17 0,2

Problem 3. Markov chains (Initial state multiplication):

In Colombia there are 6 main mobile operators such as Avantel, Tigo, Comcel,
Movistar, ETB and Uff, which we will call states. The following chart
summarizes the odds that each client has to stay in their current operator or
make a change of company.
The current percentages of each operator in the current market are for Avantel
0.1, Tigo 0.2 for Comcel 0.2, for Movistar 0.3, for ETB 0.1 and 0.2 for Uff (initial
state).

STATE TIGO COMCEL MOVISTAR ETB AVANTEL UFF total

TIGO 0,1 0,2 0,4 0,1 0,1 0,1 1
COMCEL 0,1 0,2 0,1 0,2 0,3 0,1 1
MOVISTAR 0,1 0,3 0,2 0,2 0,2 0 1
ETB 0,1 0,3 0,2 0,1 0,1 0,2 1
AVANTEL 0,3 0 0,2 0,2 0,2 0,1 1
UFF 0,1 0,2 0,2 0,3 0 0,2 1

INITIAL STATE
TIGO COMCEL MOVISTAR ETB AVANTEL UFF
0,2 0,2 0,3 0,1 0,1 0,2

PART 14. Markov Chains (Initial State Multiplication):

Find the probability that each user stays with the mobile company for the next
period.

TIGO COMCEL MOVISTAR ETB AVANTEL UFF

Probabilities 0,13 0,24 0,24 0,21 0,17 0,11
Problem 4. Markov chains (Initial state multiplication):
Suppose that 4 types of soft drinks are obtained in the market: Colombian,
Pepsi Cola, Fanta and Coca Cola when a person has bought Colombian there
is a probability that they will continue to consume 40%, 20% of which will buy
Pepsi Cola, 10% that Fanta buys and 30% that Coca Cola consumes; when the
buyer currently consumes Pepsi Cola there is a probability that he will continue
to buy 30%, 20% buy Colombiana, 20% that Fanta consumes and 30% Coca
Cola; if Fanta is currently consumed, the likelihood of it continuing to be
consumed is 20%, 40% buy Colombian, 20% consume Pepsi Cola and 20% go
to Coca Cola. If you currently consume Coca Cola the probability that it will
continue to consume is 50%, 20% buy Colombian, 20% that consumes Pepsi
Cola and 10% that is passed to Fanta.
At present, each Colombian brand, Pepsi Cola, Fanta and Coca Cola have the
following percentages in market share respectively (30%, 25%, 15% and 30%)
during week 3.

PART 15. Markov chains (Initial state multiplication):

a. Find the transition matrix.
b. Find the probability that each user stays with the mark or change to another
for period

COCA
STATE COLOMBIANA PEPSI FANTA COLA
COLOMBIANA 40% 20% 10% 30%
PEPSI 20% 30% 20% 30%
FANTA 40% 20% 20% 20%
COCA COLA 20% 20% 10% 50%

INITIAL STATE
COLOMBIANA PEPSI FANTA COCA COLA
30% 25% 15% 30%

COCA
COLOMBIANA PEPSI FANTA COLA
1 WEEK 0,29 0,225 0,14 0,345
2 WEEKS 0,286 0,223 0,1365 0,355
3 WEEKS 0,2845 0,222 0,1359 0,35735
 Problem 5. Markov chains (Initial state multiplication):
Suppose you get 6 types of Jeans brands in the Colombian market: Brand 1,
Brand 2, Brand 3, Brand 4, Brand 5 and Brand 6. The following table shows the
odds that you continue to use the same brand or change it.
At present, brand, have the following percentages in market share respectively
(20%, 15%, 17%, 15%, 13% y 20%) during week 4.

PART 15. Markov chains (Initial state multiplication):

a. Find the transition matrix.
b. Find the probability that each user stays with the mark or change to another
for period
STATE BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6
BRAND 1 0,2 0,16 0,15 0,21 0,18 0,1
BRAND 2 0,14 0,18 0,2 0,19 0,15 0,14
BRAND 3 0,13 0,16 0,15 0,21 0,2 0,15
BRAND 4 0,21 0,2 0,15 0,2 0,18 0,06
BRAND 5 0,15 0,15 0,15 0,19 0,15 0,21
BRAND 6 0,17 0,16 0,17 0,18 0,19 0,13

INITIAL STATE
BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6
20% 15% 17% 15% 13% 20%

BRAND 1 BRAND 2 BRAND 3 BRAND 4 BRAND 5 BRAND 6

1 WEEK 0,1681 0,1677 0,1615 0,1969 0,177 0,1288
2 WEEKS 0,167888 0,16946 0,160961 0,197273 0,174177 0,130241
3 WEEKS 0,16792178 0,16953835 0,16107782 0,1972473 0,17421252 0,13000223
4 WEEKS 0,16791403 0,16953853 0,16107696 0,19725244 0,17420905 0,13000898

PROPOSED STRATEGY
PART 13. Markov chains (steady state):
According to Table 1 by applying the Markovian processes, ie finding the
transition matrix and solving the respective equations of p * q, where p is the
transition matrix and q the vector [W X Y Z]. Answer:
a. What is the transition matrix resulting from proportionality according to
the accident history?
b. What is the average premium paid by a customer in Payoff, according to
historical accident rate?

PART 16. PRACTICAL WORK ENVIRONMENT.

Step 20. Enter the Practical Environment, in this space videos are presented for
the use of the WinQSB or the Excel Solver Plug-in and practical tutorials to
develop the proposed activities, remember to attach screenshots to your final
collaborative work, the income and results table for the problems raised. In this
same space you can carefully review the Guide for the use of educational
resources.
 Conclusions
It is a tool which allows us:
 Teach the reflexes of the thought system of the people being trained.
 Analyze and compare assumptions and mental models about how things
work.
 Obtain a qualitative view about the functioning of a system or the
consequences of a decision.
 Recognize archetypes of dysfunctional systems in daily practice.
 Bibliography

Ibe, O. (2013). Markov Processes for Stochastic Modeling: Massachusetts,

USA: University of Massachusetts Editorial. Retrieved from
http://bibliotecavirtual.unad.edu.co:2051/login.aspx?direct=true&db=nlebk&AN=
516132&lang=es&site=eds-live
Dynkin, E. (1982). Markov Processes and Related Problems of Analysis:
Oxford, UK: Mathematical Institute Editorial. Retrieved from
http://bibliotecavirtual.unad.edu.co:2048/login?url=http://search.ebscohost.com/l
ogin.aspx?direct=true&db=e000xww&AN=552478&lang=es&site=ehost-live
Pineda, R. (2017). Virtual learning object Unit 3. Markov decision processes.
[Video File]. Retrieved from http://hdl.handle.net/10596/13271
Piunovskiy, A. (2012). Examples In Markov Decision Processes: Singapore:
Imperial College Press Optimization Series. Retrieved from
http://bibliotecavirtual.unad.edu.co:2051/login.aspx?direct=true&db=nlebk&AN=
545467&lang=es&site=eds-live