How To Measure Anything:

Finding the Value of Intangibles in Business

Copyright HDR 2008 dwhubbard@hubbardresearch.com

How to Measure Anything

In the past 12 years, I have conducted over 60 major risk/return analysis projects that included a variety of impossible measurements I found such a high need for measuring difficult things that I decided I had to write a book The objective today is to explain an approach to making the immeasurable measurable and the surprising things we find when we accomplish this

Copyright HDR 2008 dwhubbard@hubbardresearch.com

What is AIE?
Applied AppliedInformation InformationEconomics Economics(AIE) (AIE)is isthe thepractical practicalapplication applicationof ofscientific scientificand andmathematical mathematical methods to quantify the value of management choices regardless of how difficult methods to quantify the value of management choices - regardless of how difficultthe the measurement challenge appears to be. measurement challenge appears to be. Economics Operations Research Modern Portfolio Theory Options Theory Decision/Game Theory Statistics Information Theory

Applied Information Economics

Quantifying the risk and comparing its risk/return with other investments sets AIE apart from other methodologies. It can substantially assist in financially justifying a project -- especially projects that promise significant intangible benefits. The Gartner Group AIE represents a rigorous, quantitative approach to improving IT investment decision making..this investment will return multiples by enabling much better decision making. Giga recommends that IT executives learn more about AIE and begin to adopt its tools and methodologies, especially for large IT projects. Giga Information Group

Copyright HDR 2008 dwhubbard@hubbardresearch.com

A Few Measurement Examples

Risk of IT The Risk of obsolescence The value of a human life The value of saving an endangered species The value of better information The value of public health Forecasting fuel demand for the battlefield The value of better security The effects of an initiative when many other variables affect performance The future demand for space tourism The risk of a .com venture capital startup The risks of a major construction project
4

Copyright HDR 2008 dwhubbard@hubbardresearch.com

Three Illusions of Intangibles

(The howtomeasureanything.com approach)

The perceived impossibility of measurement is an illusion caused by not understanding: the Concept of measurement the Object of measurement the Methods of measurement See my Everything is Measurable article in CIO Magazine (go to articles link on www.hubbardresearch.com
Copyright HDR 2008 dwhubbard@hubbardresearch.com 5

Uncertainty, Risk & Measurement

Measuring Uncertainty, Risk and the Value of Information are closely related concepts, important measurements themselves, and precursors to most other measurements
The Measurement Theory definition of measurement: A measurement is an observation that results in information (reduction of uncertainty) about a quantity. An Actuary's approach to Risk Measurement: To quantify probability and loss of an undesirable possibility The value of a Measurement: The monetized reduction in risk from making decisions under less uncertainty We model uncertainty statistically with Monte Carlo simulations

Copyright HDR 2008 dwhubbard@hubbardresearch.com

Ideal vs. Real-world Measurements

Ideal Values: Point Real-world Meas.
Normal Distribution Uniform Distribution Lognormal Distribution Hybrid Threshold confidence 15% 85% No Assumptions: Most things we DO know are better represented by ranges and probabilities we dont have to assume anything we dont really know. Assumptions: Most values in business cases are represented as exact values even though exact values are almost never known

Copyright HDR 2007 dwhubbard@hubbardresearch.com

An Approach That Works

1. Define the relevant decision and clarify the inputs 2. Model what you know now 3. Compute the value of additional information 4. Measure where the information value is high 5. Update the model and optimize the decision

Copyright HDR 2008 dwhubbard@hubbardresearch.com

Defining the Decision

The EPA needed to compute the ROI of the Safe Drinking Water Information System (SDWIS) As with any AIE project, we built a spreadsheet model that connected the expected effects of the system to relevant impacts in this case public health and its economic value

Copyright HDR 2008 dwhubbard@hubbardresearch.com

Model What You Know

Decades of studies show that most managers are statistically overconfident when assessing their own uncertainty
Studies showed that bookies were great at assessing odds subjectively, while doctors were terrible

Studies also show that measuring your own uncertainty about a quantity is a general skill that can be taught with a measurable improvement Training can calibrate people so that of all the times they say they are 90% confident, they will be right 90% of the time

Copyright HDR 2008 dwhubbard@hubbardresearch.com

10

Calibration Test Examples

How many grams does the average Jelly Belly Jelly Bean weight?
_____ to ____ (90% Confidence)

What is the height in meters of the Sears Tower?

_____ to ____ (90% Confidence)

Idaho has a larger area than Iraq.

True/False ____% Confidence

In the English Language, the word strategy is used more often than the word celebrate.
True/False ____% Confidence

Copyright HDR 2008 dwhubbard@hubbardresearch.com

11

Calibrated Estimates: Ranges

Calibrated probability assessment results from various studies
90% Confidence Interval

Copyright HDR 2008 dwhubbard@hubbardresearch.com

12

Calibrated Estimates of Events

1997: An experiment Hubbard conducted with Giga Information Group proves people can be trained to assess probabilities of uncertain forecasts Hundreds have been calibrated since then Calibrated probabilities are the basis for modeling the current state of uncertainty
100%
17

90%

Percent Correct

21 45 65 68 152 75 71 65 58 21

Ideal Confidence Statistical Error Giga Clients Giga Analysts

99 # of Responses

80% 70% 60% 50% 40%

25

30% 50%

60%

70%

80%

90% 100%

Assessed Chance Of Being Correct

Copyright HDR 2008 dwhubbard@hubbardresearch.com 13

The Value of Information

z z z EVI = p( ri ) max V1, j p( j | ri ), V2 , j p( j | ri ),... Vl , j p( j | ri ), EV * i =1 j =1 j =1 j =1 k

The formula for the value of information has been around for almost 60 years. It is widely used in many parts of industry and government as part of the decision analysis methods but still mostly unheard of in the parts of business where it might do the most good. What it means: 1.Information reduces uncertainty 2.Reduced uncertainty improves decisions 3.Improved decisions have observable consequences with measurable value
Copyright HDR 2008 dwhubbard@hubbardresearch.com 14

The EOL Method

The simplest approach computes the change in Expected Opportunity Loss Opportunity Loss is the loss (compared to the alternative) if it turns out you made the wrong decision Expected Opportunity Loss (EOL) is the cost of being wrong times the chance of being wrong The reduction in EOL from more information is the value of the information. In the case of perfect information (if that were possible) the value of information is equal to the EOL. Simple Binary Example: You are about to make a $20 million investment to upgrade the equipment in a factory to make a new product. If the new product does well, you save $50 million in manufacturing. If not, you lose (net) $10 million. There is a 20% chance of the new product failing. Whats it worth to have perfect certainty about this investment if that were possible? Answer: 20% x $10 million = $2 million

Copyright HDR 2008 dwhubbard@hubbardresearch.com

15

Information Value w/Ranges

The value of information is computed a little differently with a distribution, but the same basic concepts apply For each variable, there is a Threshold where the investment just breaks even If the threshold is within the range of possible values, then there is a chance that you would make a different decision with better measurements
90% Confidence Interval Threshold 5% tail Threshold Probability Mean 5% tail

Copyright HDR 2008 dwhubbard@hubbardresearch.com

16

Normal Distribution VIA

The curve on the other side of the threshold is divided up into hundreds of slices Each slice has an assigned quantity (such as a potential productivity improvement) and a probability of occurrence For each assigned quantity, there is an Opportunity Loss Each slices Opportunity Loss is multiplied by probability to compute its Expected Opportunity Loss The total EOL for all slices is the EVPI of the uncertain variable

Total of all EOLs = $58,989 # effected by virus: 15%

# effected by virus: 15% Opportunity OpportunityLoss: Loss:$1,855,000 $1,855,000 Probability: Probability:0.0053% 0.0053% EOL: $98.31 EOL: $98.31

5%

10%

15%

20%

25%

Productivity Improvement in Process X Copyright HDR 2008 dwhubbard@hubbardresearch.com 17

Increasing Value & Cost of Info.

The value of information levels off while the cost of information accelerates Information value grows fastest at the beginning of information collection Use iterative measurements that err on the side of small bites at the steep part of the slope $$$
Dollar Value/Cost Aim for this range

EVPI ENBI

EVI

Maximum ENBI

EVPI Expected Value of Perfect Information ECI Expected Cost of Information EVI Expected Value of Information ENBI Expected Net Benefit of Information

ECI
$0 Low accuracy High accuracy
Copyright HDR 2008 dwhubbard@hubbardresearch.com 18

The Measurement Inversion

After the information values for over 3,500 Measurement Attention variables was computed, a pattern emerged. vs. Relevance The highest value measurements were almost never measured while most measurement effort was spent on less relevant factors
Costs were measured more than the more uncertain benefits Small hard benefits would be measured more than large soft benefits

Also, we found that, if anything, fewer measurements were required after the information values were known.

See my article The IT Measurement Inversion in CIO Magazine (its also on my website at www.hubbardresearch.com under the articles link)
Copyright HDR 2008 dwhubbard@hubbardresearch.com 19

Economic Relevance

Typical Attention

Next Step: Observations

Once weve determined what to measure, we can think of observations that would reduce uncertainty The value of the information limits what methods we should use, but we have a variety of methods available Take the Nike Method: Just Do It dont let imagined difficulties get in the way of starting observations

Copyright HDR 2008 dwhubbard@hubbardresearch.com

20

Practical Assumptions
Its been measured before You have more data than you think You need less data than you think Its more economical than you think Your subjective estimate of possible measurement errors is exaggerated Its amazing what you can see when you look Yogi Berra

Copyright HDR 2008 dwhubbard@hubbardresearch.com

21

The Math-less Statistics Table

Measurement is based on observation and most observations are just samples Reducing your uncertainty with random samples is not made intuitive in most statistics texts This table makes computing a 90% confidence interval easy
Copyright HDR 2008 dwhubbard@hubbardresearch.com 22

Measuring to the Threshold

Measurements have value usually because there is some point where the quantity makes a difference Its often much harder to ask How much is X than Is X enough
Number Sampled Chance the Median is Below the Threshold
2 50% 40% 30% 20% 10%
5% 2%

8 10 12 14 16 18 20

1%
0.5% 0.2%

0.1%

10

Samples Below Threshold

Copyright HDR 2008 dwhubbard@hubbardresearch.com

23

Statistics Goes to War

Several clever sampling methods exist that can measure more with less data than you might think Examples: estimating the population of fish in the ocean, estimating the number of tanks created by the Germans in WWII, extremely small samples, etc.
Copyright HDR 2008 dwhubbard@hubbardresearch.com 24

Reducing Inconsistency
The Lens Model is another method used to improve on expert intuition The chart shows the reduction in error from this method on intuitive estimates In every case, this method equaled or bettered the judgment of experts IT Portfolio Priorities Battlefield Fuel Forecasts Student ratings of teaching effectiveness Cancer patient life-expectancy Psychology course grades Graduate students grades Changes in stock prices IQ scores using Rorschach tests Mental illness using personality tests Business failures using financial ratios Life-insurance salesrep performance
Source: Hubbard Decision Research

My My Studies Studies

0%

10%

20%

30%

40%

Reduction in Errors
25

Copyright HDR 2008 dwhubbard@hubbardresearch.com

The Simplest Method

Bayesian methods in statistics use new information to update prior knowledge Bayesian methods can be even more elaborate that other statistical methods BUT It turns out that calibrated people are already mostly instinctively Bayesian The instinctive Bayesian approach:
Assess your initial subjective uncertainty with a calibrated probability Gather and study new information about the topic (it could be qualitative or even tangentially related) Give another subjective calibrated probability assessment with this new information

In studies where people were asked to do this, thier results were usually not irrational compared to what would be computed with Bayesian statistics calibrated people do even better
Copyright HDR 2008 dwhubbard@hubbardresearch.com 26

Comparison of Methods

Typical Un-calibrated Expert Overconfident (Stated uncertainty is lower than rational) Gullible Stubborn

NonBayesian Bayesian Statistics Calibrated Expert Under-confident (Stated uncertainty is higher than rational)

Vacillating, Indecisive

Overly Cautious

Ignores Prior Knowledge; Emphasizes new data

Ignores New data; Emphasizes Prior Knowledge

Traditional non-Bayesian statistics (what you probably learned in the first semester of stats) assumes you knew nothing prior to the samples you took - this is almost never true in reality Most un-calibrated experts are overconfident and slightly overemphasize new information Calibrated experts are not overconfident, but slightly ignore prior knowledge Bayesian analysis is the perfect balance; neither under- nor over- confident, uses both new and old information
27

Copyright HDR 2008 dwhubbard@hubbardresearch.com

Risk/ROI w/ Monte Carlo

A Monte Carlo simulation generates thousands of random scenarios using the defined probabilities and ranges The result is a range ROI not a point ROI Administrative Cost Reduction Customer Retention Increase Total Project Cost
$2 million $4 million $6 million

5% 10%

10% 20%

15% 30%

ROI
-50% 0% 50% 100%
28

Copyright HDR 2008 dwhubbard@hubbardresearch.com

Quantifying Risk Aversion

The simplest element of Harry Markowitzs Nobel Prize-winning method Modern Portfolio Theory is documenting how much risk an investor accepts for a given return. The Investment Boundary states how much risk an investor is willing to accept for a given return. For our purposes, we modified Markowitzs approach a bit. Acceptable Risk/Return Boundary Investment Region Investment

Copyright HDR 2008 dwhubbard@hubbardresearch.com

29

Full RRA in 7 Steps

Define Decision Model Populate Model with Calibrated Estimates & Measurements Calibrate Estimators

Conduct Value of Information Analysis (VIA)

Measure according to VIA results and update model

Analyze Remaining Risk

Optimize Decision

Copyright HDR 2008 dwhubbard@hubbardresearch.com

30

Forecasting Fuel for Battle

The US Marine Corps with the Office of Naval Research needed a better method for forecasting fuel for wartime operations The VIA showed that the big uncertainty was really supply route conditions, not whether they are engaging the enemy Consequently, we performed a series of experiments with supply trucks rigged with GPS and fuel-flow meters
Copyright HDR 2008 dwhubbard@hubbardresearch.com 31

Reactions: Fuel for the Marines

The biggest surprise was that we can save so much fuel. We freed up vehicles because we didnt have to move as much fuel. For a logistics person that's critical. Now vehicles that moved fuel can move ammunition. Luis Torres, Fuel Study Manager, Office of Naval Research What surprised me was that [the model] showed most fuel was burned on logistics routes. The study even uncovered that tank operators would not turn tanks off if they didnt think they could get replacement starters. Thats something that a logistician in a 100 years probably wouldnt have thought of. Chief Warrant Officer Terry Kunneman, Bulk Fuel Planning, HQ Marine Corps

Copyright HDR 2008 dwhubbard@hubbardresearch.com

32

Final Tips
Learn how to think about uncertainty, risk and information value in a quantitative way Assume its been measured before You have more data than you think and you need less data than you think Methods that reduce your uncertainty are more economical than many managers assume Dont let exception anxiety cause you to avoid any observations at all Just do it
Copyright HDR 2008 dwhubbard@hubbardresearch.com 33

Questions?
Doug Hubbard Hubbard Decision Research dwhubbard@hubbardresearch.com www.hubbardresearch.com 630 858 2788
If you want electronic copies of this presentation and copies of supporting articles I mention, please leave me a business card with Presentation written on the back

Copyright HDR 2008 dwhubbard@hubbardresearch.com

34