Assessment of

Probability Distributions
Analytica User Group Webinar

Lonnie Chrisman
Lumina Decision Systems
6 Mar 2008

While you are waiting….

Download “Probability assessment.ana” from talk abstract on User Group wiki page.

http://lumina.com/wiki/images/6/6b/Probability_assessment.ana

Do not start the assessment yet.

 Explicit Uncertainty
Gas
• Models bottom-out in parameters mileage
Fuel
that must be estimated. Annual consumed
mileage
– Point estimates Annual fuel
cost
– Probability distributions fuel price
volatility Fuel spot
price
Current fuel
price

• Benefits of encoding distributions:

– Awareness of forecast precision
– Better decisions (decisions based on uncertainty)
– Makes risks explicit
– Estimate can be “correct”, even if imprecise.
 Subjective Probability
Knowledge

Available Data Elicitation Probability

Process Distribution
Available Information
Literature
• Assessed distribution reflects
available knowledge (not an
inherent property of the world).
• There is no “correct distribution”.
• Quality (e.g., “calibration”) is
determined by the elicitation
process (not the final result).
 Measured outcome frequency
Assessment Quality
• Calibration • Discrimination
1
0.9
0.8 Ability to adjust assessment
0.7
0.6 for different conditions.
0.5
0.4
0.3
0.2
More expertise  less entropy
0.1
0
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Subjective probability

These are measured across many assessments using a fixed elicitation

process. Hence, they grade the elicitation process, not an individual
probability or distribution assessment.
 Scoring Rules
(binary outcome)
• Assess one number: p = Pr(X=1)
• Observe actual outcome, x
• Scoring rule:
– S(p,x) = quality of prediction (a number)
– Rewards both calibration and discrimination.
• “Proper scoring rule”
– E[S(p,x)|x~q] is maximal when p=q
– Cannot be “gamed”
– Examples:
• Log score: ln p(x)
• Scaled log score: 1 + log2 p(x)
• Quadratic score p(x)2
– Improper rule: linear score: p(x)
 Probability Assessment Game
Download “Probability assessment.ana” from talk abstract on User Group page.
• http://lumina.com/wiki/images/6/6b/Probability_assessment.ana

• These are not intended to be trick questions.

• Do not look up the answer before you’ve answered.
 Cognitive Biases
Common biases include:
• Over-confidence
• Anchoring / insufficient adjustment
• Motivational bias
– Management bias
– Expert bias
• Denial of uncertainty
 De-biasing…
Probability that Hillary Clinton will be the
Democratic Presidential Nominee this
year?

Discuss:
• Relevant facts. (Clinton: has 1462 needs 563 more. Obama: has 1567 needs 458 more.
Voters: 611 remaining. Undecided superdelegates: 346 [SJ Merc. News 6 Mar 2008]).
• Describe possible scenarios that could prevent this from happening.
• What events could transpire to cause it to happen?
• Have you separated what you want from your estimate? Are you
clear that your estimate does not influence the outcome?
 Quartile assessment
(real-valued quantities)

(oops – its quartile, not quantile)

Scoring:
• Scaled log score: 1+log2 α p(x / α), α=|x| adjusts for scale
• Interval: 50% of your response should be inside quartiles,
50% should be outside your quartiles
Combating Cognitive Biases
 Identify meaningful variables
• People are less susceptible to biases in
their own areas of expertise, and when the
question is meaningful to them.
• Make sure the measurement scale is
meaningful to the person. (Units and
linear/log scale).
• Most of the time, the selection of variables
and influence structure is far more
important than numeric uncertainty!
 Defining the Quantity
• “Clairvoyant test”: Would an oracle know the
value without needing further clarification?
• Probability is not a measure of fuzziness.
• Don’t worry if you don’t know, or can never know
the value, as long as it is unambiguous and
meaningful.

• In principle, you can assess any quantity.

• As a modeler, you can chose to decompose a
quantity into simpler quantities for assessment.
 Decompose to meaningful
variables

• If you feel the variable is conditional on

some other variable, incorporate the
conditionality into the model.
• If you can decompose into variables that
are easier to assess, make that explicit in
the model.
 Pre-assessment
Before making any estimate of a particular
variable, do the following:
• List all relevant knowledge that you can.
• Describe scenarios of how the extremes may
occur – especially the least likely extreme.
• Assess any background frequencies before
taking the situational specifics into account.

These help you to avoid anchoring and can help

reduce over-confidence.
 Design the assessment
• What “shape” is the distribution?
– Usually consists of identifying the distribution
type, e.g.: Normal, uniform, LogNormal, etc.
• What parameters will you estimate?
– Quartiles: 25%, 50%, 75%, etc.
– Min, max
– Mean, Std.dev.
– Probability (or cum prob) of selected values
– Indirect: Betting scenarios, comparisons, etc.
 Assess
• Consider different approaches to
assessing the same quantity. Do you get
consistent estimates? If not, don’t just
split the difference – dig deeper and
explain why.
• Take into account take you are likely to be
over-confident, and that the less you
know, the more over-confident you are
likely you are likely to be.
 Sensitivity Analysis
• Use “Make Importance” and other
sensitivity analyses to determine the
sensitivity of your results to your
assessments.
• Revisit critical assessments.
 Practice
• Practice on almanac problems (as we did
today).
 Fast Assessment
• Your (numeric) subjective assessment can
depend on how long you think about it.
• Less-thought  higher entropy,
less discrimination
• Calibration should not (ideally) suffer.

• Fast distribution assessments can take less time

than point assessments.

(Variable definitions, model structure, should never be done hastily.)

 Selecting a Distribution
(Discrete numeric)

Binary (yes/no) events:

• Bernoulli(p)
Tallies/counts (number of events):
• Poisson(p)
• Binomial(n,p)
• Geometric(p), HyperGeometric(…)
Integer values:
• Uniform(min,max,integer:true)
 Selecting a Distribution
(categorical)

• ChanceDist
• ProbTable

In both, you assess the probability of each

possible outcome.

Note also the task of identifying the

outcomes. Neglecting possible outcomes
can also be a source of bias.
 Selecting a Distribution
(Bounded Continuous)
Distribution over a probability (0,1)-interval:
• Beta(x,y), Beta_m_sd(m,sd)
– Assess mean = x/(x+y)
– Spread (σ) best judged graphically, plot and re-plot.
Bounded on both ends.
• Uniform(min,max)
• Triangular(min,mode,max)
Bounded from below (usually by 0):
• Exponential(mean)
• LogNormal(median:m,stdDev:sd)
• Gamma(mean)
• Weibull(shape,scale) – failure times
 Selecting a Distribution
(unbounded continuous)

General:
• Normal(m,sd)
• ProbDist, CumDist – custom distributions
Describing deviation from mean:
• StudentT(dof)
• ChiSquared(dof)
Describing growth:
• Logistic(mean,scale)
 Final words
• Most critical step: Identifying the variables.
Don't let yourself be deterred by the fact that you do not know, nor
can you know, some of the quantities in your model. Build the
model as if you could know everything you would like.
• Subjective uncertainty describes your
state of knowledge – not a property of the
world.
• Humans exhibit cognitive biases when
assessing probabilities, most notably over-
confidence.