Sei sulla pagina 1di 34
How To Measure Anything : Finding the Value of Intangibles in Business Copyright Copyright HDR

How To Measure Anything:

Finding the Value of Intangibles in Business

Anything : Finding the Value of Intangibles in Business Copyright Copyright HDR HDR 2008 2008

How to Measure Anything

How to Measure Anything • In the past 12 years, I have conducted over 60 major
How to Measure Anything • In the past 12 years, I have conducted over 60 major

• In the past 12 years, I have conducted over 60 major risk/return analysis projects that included a variety of “impossible” measurements

• I found such a high need for measuring difficult things that I decided I had to write a book

• The objective today is to explain an approach to making the “immeasurable” measurable and the surprising things we find when we accomplish this

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

22

What is AIE?

What is AIE? EconomicsEconomics Decision/GameDecision/Game OperationsOperations ResearchResearch AppliedApplied
EconomicsEconomics Decision/GameDecision/Game OperationsOperations ResearchResearch AppliedApplied TheoryTheory
EconomicsEconomics
Decision/GameDecision/Game
OperationsOperations ResearchResearch
AppliedApplied
TheoryTheory
ModernModern PortfolioPortfolio
TheoryTheory
OptionsOptions
TheoryTheory
InformationInformation
EconomicsEconomics
StatisticsStatistics
InformationInformation TheoryTheory

• “Quantifying the risk and comparing its risk/return with other investments sets AIE apart from other methodologies. It can substantially assist in financially justifying a project -- especially projects that promise significant intangible benefits.” The Gartner Group

• “AIE represents a rigorous, quantitative approach to improving IT investment decision

making…

recommends that IT executives learn more about AIE and begin to adopt its tools and

methodologies, especially for large IT projects.” Giga Information Group

this

investment will return multiples by enabling much better decision making. Giga

multiples by enab ling much better dec ision making. Giga Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

33

A Few Measurement Examples

• Risk of IT

• The Risk of obsolescence

• The value of a human life

• The value of saving an endangered species

• The value of better information

• The value of public health

• Forecasting fuel demand for the battlefield

• The value of better security

• The effects of an initiative when many other variables affect performance

• The future demand for space tourism

• The risk of a .com venture capital startup

• The risks of a major construction project

startup • The risks of a major construction project Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

44

Three Illusions of Intangibles

(The “howtomeasureanything.com” approach)

• The perceived impossibility of measurement is an illusion caused by not understanding:

– the Concept of measurement

– the Object of measurement

– the Methods of measurement

the Object of measurement – the Methods of measurement • See my “Everything is Measurable” article

• See my “Everything is Measurable” article in CIO Magazine (go to “articles” link on www.hubbardresearch.com

(go to “articles” link on www.hubbardresearch.com Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

55

Uncertainty, Risk & Measurement

Measuring Uncertainty, Risk and the Value of Information are closely related concepts, important measurements themselves, and precursors to most other measurements

• The “Measurement Theory” definition of measurement: “A measurement is an observation that results in information (reduction of uncertainty) about a quantity.”

• An Actuary's approach to Risk Measurement: “To quantify probability and loss of an undesirable possibility”

• The value of a Measurement: “The monetized reduction in risk from making decisions under less uncertainty”

• We model uncertainty statistically – with Monte Carlo simulations

uncertainty statistically – with Monte Carlo simulations Copyright Copyright HDR HDR 2008 2008
uncertainty statistically – with Monte Carlo simulations Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

66

Ideal vs. Real-world Measurements

Assumptions: Most values in business Assumptions: cases are represented as exact values – even though exact values are almost never known cases are represented as exact values – even though exact values are almost never known

values – even though exact values are almost never known Ideal Values: Point Real-world Meas. Normal

Ideal Values: Point

Real-world Meas.

Normal Distribution

Ideal Values: Point Real-world Meas. Normal Distribution Uniform Distribution Lognormal Distribution Hybrid Threshold

Uniform Distribution

Lognormal Distribution

Hybrid

Threshold confidence

Lognormal Distribution Hybrid Threshold confidence No Assumptions: Most things we DO know are better

No Assumptions: Most things we DO know are better represented by ranges and probabilities – we don’t Most things we DO know are better represented by ranges and probabilities – we don’t have to assume anything we don’t really know.

we don’t have to assume anything we don’t really know. 15% 85% Copyright Copyright HDR HDR
we don’t have to assume anything we don’t really know. 15% 85% Copyright Copyright HDR HDR
15% 85%
15%
85%
have to assume anything we don’t really know. 15% 85% Copyright Copyright HDR HDR 2007 2007

CopyrightCopyright HDRHDR 20072007 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

77

An Approach That Works

1. Define the relevant decision and clarify the inputs

2. Model what you know now

3. Compute the value of additional information

4. Measure where the information value is high

5. Update the model and optimize the decision

value is high 5. Update the model and optimize the decision Copyright Copyright HDR HDR 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

88

Defining the Decision

• The EPA needed to compute the ROI of the Safe Drinking Water Information System (SDWIS)

• As with any AIE project, we built a spreadsheet model that connected the expected effects of the system to relevant impacts – in this case public health and its economic value

– in this case public health and its economic value Copyright Copyright HDR HDR 2008 2008
– in this case public health and its economic value Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

99

Model What You Know

• Decades of studies show that most managers are statistically “overconfident” when assessing their own uncertainty

– Studies showed that bookies were great at assessing odds subjectively, while doctors were terrible

• Studies also show that measuring your own uncertainty about a quantity is a general skill that can be taught with a measurable improvement

• Training can “calibrate” people so that of all the times they say they are 90% confident, they will be right 90% of the time

they are 90% confident, they will be right 90% of the time Copyright Copyright HDR HDR

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

1010

Calibration Test Examples

• How many grams does the average Jelly Belly Jelly Bean weight?

– to

(90% Confidence)

• What is the height in meters of the Sears Tower?

– to

(90% Confidence)

• Idaho has a larger area than Iraq.

– True/False

% Confidence

• In the English Language, the word “strategy” is used more often than the word “celebrate”.

• True/False

more often than the word “celebrate”. • True/False % Confidence Copyright Copyright HDR HDR 2008 2008

% Confidence

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

1111

Calibrated Estimates: Ranges

Calibrated probability assessment results from various studies

90% Confidence Interval

results from various studies 90% Confidence Interval Copyright Copyright HDR HDR 2008 2008
results from various studies 90% Confidence Interval Copyright Copyright HDR HDR 2008 2008
results from various studies 90% Confidence Interval Copyright Copyright HDR HDR 2008 2008
results from various studies 90% Confidence Interval Copyright Copyright HDR HDR 2008 2008
results from various studies 90% Confidence Interval Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

1212

Calibrated Estimates of Events

• 1997: An experiment Hubbard conducted with Giga Information Group proves people can be trained to assess probabilities of uncertain forecasts

• Hundreds have been calibrated since then

• Calibrated probabilities are the basis for modeling the current state of uncertainty

100% 17 90% “Ideal” Confidence 21 45 Statistical Error 80% 65 70% Giga Clients 68
100%
17
90%
“Ideal” Confidence
21
45
Statistical Error
80%
65
70%
Giga Clients
68
152
21
60%
75
71
65
Giga Analysts
58
50%
99 # of Responses
40%
25
30%
50%
60%
70%
80%
90%
100%
Percent Correct
40% 25 30% 50% 60% 70% 80% 90% 100% Percent Correct Assessed Chance Of Being Correct

Assessed Chance Of Being Correct

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

1313

The Value of Information

k z z z ⎤ EVI = ∑ p r ( ) max ⎢ ⎡
k
z
z
z
EVI
=
p r
(
) max
⎢ ⎡ ∑
V
p
(
ΘΘ
|
r
),
V
p
(
|
r
),
V
p
(
Θ
|
r
),
EV
*
i
1,, ∑
j
ji
2
j
j
i
lj ,
j
i
i =
1
j
=
1
j
=
1
j
=
1

The formula for the value of information has been around for almost 60 years. It is widely used in many parts of industry and government as part of the “decision analysis” methods – but still mostly unheard of in the parts of business where it might do the most good.

What it means:

1.Information reduces uncertainty 2.Reduced uncertainty improves decisions 3.Improved decisions have observable consequences with measurable value

have observable consequences with measurable value Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

1414

The EOL Method

• The simplest approach computes the change in “Expected Opportunity Loss”

• “Opportunity Loss” is the loss (compared to the alternative) if it turns out you made the wrong decision

• Expected Opportunity Loss (EOL) is the cost of being wrong times the chance of being wrong

• The reduction in EOL from more information is the value of the information.

• In the case of perfect information (if that were possible) the value of information is equal to the EOL.

• Simple Binary Example: You are about to make a $20 million investment to upgrade the equipment in a factory to make a new product. If the new product does well, you save $50 million in manufacturing. If not, you lose (net) $10 million. There is a 20% chance of the new product failing. What’s it worth to have perfect certainty about this investment if that were possible?

• Answer: 20% x $10 million = $2 million

were possible? • Answer: 20% x $10 million = $2 million Copyright Copyright HDR HDR 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

1515

Information Value w/Ranges

• The value of information is computed a little differently with a distribution, but the same basic concepts apply

• For each variable, there is a “Threshold” where the investment just breaks even

• If the threshold is within the range of possible values, then there is a chance that you would make a different decision with better measurements

90%

Confidence

Interval

decision with better measurements 90% Confidence Interval Threshold Mean 5% tail Threshold 5% tail Probability
decision with better measurements 90% Confidence Interval Threshold Mean 5% tail Threshold 5% tail Probability

Threshold

Mean

Threshold Mean
5% tail Threshold
5% tail
Threshold

5% tail

Probability

Threshold Mean 5% tail Threshold 5% tail Probability Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

1616

Normal Distribution VIA

• The curve on the other side of the threshold is divided up into hundreds of “slices”

• Each slice has an assigned quantity (such as a potential productivity improvement) and a probability of occurrence

• For each assigned quantity, there is an Opportunity Loss

• Each slice’s Opportunity Loss is multiplied by probability to compute its Expected Opportunity Loss

• The total EOL for all slices is the EVPI of the uncertain variable

Total of all EOL’s = $58,989 # # effected effected by by virus: virus: 15%
Total of all EOL’s =
$58,989
# # effected effected by by virus: virus: 15% 15%
Opportunity Opportunity Loss: Loss: $1,855,000 $1,855,000
Probability: Probability: 0.0053% 0.0053%
EOL: EOL: $98.31 $98.31
5%
10%
15%
20%
25%
0.0053% EOL: EOL: $98.31 $98.31 5% 10% 15% 20% 25% Productivity Improvement in Process X Copyright

Productivity Improvement in Process X

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

1717

Dollar Value/Cost

Increasing Value & Cost of Info.

The value of information levels off while the cost of information accelerates

Information value grows fastest at the beginning of information collection

Use iterative measurements that err on the side of “small bites” at the steep part of the slope

Aim for this range

$$$

EVPI

EVI Maximum ENBI ECI
EVI
Maximum
ENBI
ECI

EVPI – Expected Value of Perfect Information

ECI – Expected Cost of Information

EVI – Expected Value of Information

ENBI – Expected Net Benefit of Information

ENBI

$0

L o w a c c u r a c y
L o w a c c u r a c y

Low accuracy

L o w a c c u r a c y
L o w a c c u r a c y

High accuracy

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

1818

The Measurement Inversion

• After the information values for over 3,500 variables was computed, a pattern emerged.

• The highest value measurements were almost never measured while most measurement effort was spent on less relevant factors

Measurement Attention vs. Relevance

less relevant factors Measurement Attention vs. Relevance Typical Attention Economic Relevance – Costs were
Typical Attention Economic Relevance
Typical Attention
Economic Relevance

– Costs were measured more than the more uncertain benefits

– Small “hard” benefits would be measured more than large “soft” benefits

• Also, we found that, if anything, fewer measurements were required after the information values were known.

See my article “The IT Measurement Inversion” in CIO Magazine (its also on my website at www.hubbardresearch.com under the “articles” link)

at www.hubbardresearch.com under the “articles” link) Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

1919

Next Step: Observations

• Once we’ve determined what to measure, we can think of observations that would reduce uncertainty

• The value of the information limits what methods we should use, but we have a variety of methods available

• Take the “Nike Method”: Just Do It – don’t let imagined difficulties get in the way of starting observations

difficulties get in the way of starting observations Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

2020

Practical Assumptions

Its been measured before

You have more data than you think

You need less data than you think

Its more economical than you think

Your subjective estimate of possible measurement errors is exaggerated

“It’s amazing what you can see when you look” Yogi Berra

amazing what you can see when you look” Yogi Berra Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

2121

The “Math-less” Statistics Table

The “Math-less” Statistics Table • Measurement is based on observation and most observations are just samples

• Measurement is based on observation and most observations are just samples

• Reducing your uncertainty with random samples is not made intuitive in most statistics texts

• This table makes computing a 90% confidence interval easy

This table makes computing a 90% confidence interval easy Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

2222

Measuring to the Threshold

• Measurements have value usually because there is some point where the quantity makes a difference

• Its often much harder to ask “How much is X” than “Is X enough”

Number Sampled

2 4 6 8 10 12 14 16 18 20 50% 40% 30% 20% 10%
2
4
6
8
10
12
14
16
18
20
50%
40%
30%
20%
10%
5%
2%
1%
0.5%
0.2%
0.1%
0 123 4
5
6
7
8
9
10
Chance the Median is Below the Threshold

Samples Below Threshold

the Median is Below the Threshold Samples Below Threshold Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

2323

Statistics Goes to War

Statistics Goes to War • Several clever sampling methods exist that can measure more with less

• Several clever sampling methods exist that can measure more with less data than you might think

• Examples: estimating the population of fish in the ocean, estimating the number of tanks created by the Germans in WWII, extremely small samples, etc.

by the Germans in WWII, extremely small samples, etc. Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

2424

Reducing Inconsistency

• The “Lens Model” is another method used to improve on expert intuition

• The chart shows the reduction in error from this method on intuitive estimates

• In every case, this method equaled or bettered the judgment of experts

IT Portfolio Priorities Battlefield Fuel Forecasts

Student ratings of teaching effectiveness Cancer patient life-expectancy Psychology course grades Graduate students grades Changes in stock prices IQ scores using Rorschach tests Mental illness using personality tests Business failures using financial ratios Life-insurance salesrep performance

My My Studies Studies
My My Studies Studies

0%

Source: Hubbard Decision Research

10%

20%

30%

Reduction in Errors

Hubbard Decision Research 10% 20% 30% Reduction in Errors Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

40%

2525

The Simplest Method

• “Bayesian” methods in statistics use new information to update prior knowledge

• Bayesian methods can be even more elaborate that other statistical methods BUT…

• It turns out that calibrated people are already mostly “instinctively Bayesian”

• The instinctive Bayesian approach:

– Assess your initial subjective uncertainty with a calibrated probability

– Gather and study new information about the topic (it could be qualitative or even tangentially related)

– Give another subjective calibrated probability assessment with this new information

• In studies where people were asked to do this, thier results were usually not irrational compared to what would be computed with Bayesian statistics – calibrated people do even better

Bayesian statistics – calibrated people do even better Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

2626

Comparison of Methods

• Traditional non-Bayesian statistics (what you probably learned in the first semester of stats) assumes you knew nothing prior to the samples you took - this is almost never true in reality

• Most un-calibrated experts are overconfident and slightly overemphasize new information

• Calibrated experts are not overconfident, but slightly ignore prior knowledge

• Bayesian analysis is the perfect balance; neither under- nor over- confident, uses both new and old information

Typical Un-calibrated Gullible Stubborn Expert Non- Bayesian Bayesian Statistics Calibrated Expert
Typical
Un-calibrated
Gullible
Stubborn
Expert
Non-
Bayesian
Bayesian
Statistics
Calibrated
Expert
Vacillating,
Overly
Indecisive
Cautious

Ignores Prior Knowledge; Emphasizes new data

Ignores New data; Emphasizes Prior Knowledge

Overconfident

(Stated

uncertainty is

lower than

rational)

Under-confident (Stated uncertainty is higher than rational)

(Stated uncertainty is higher than rational) Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

2727

Risk/ROI w/ “Monte Carlo”

• A Monte Carlo simulation generates thousands of random scenarios using the defined probabilities and
• A Monte Carlo simulation generates
thousands of random scenarios using
the defined probabilities and ranges
• The result is a range ROI not a point
ROI
Administrative Cost
Reduction
5%
10%
15%
Customer Retention
Increase
10%
20%
30%
Total Project Cost
$2 million
$4 million
$6 million
ROI
-50%
0%
50%
100%
$2 million $4 million $6 million ROI -50% 0% 50% 100% Copyright Copyright HDR HDR 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

2828

Quantifying Risk Aversion

• The simplest element of Harry Markowitz’s Nobel Prize-winning method “Modern Portfolio Theory” is documenting how much risk an investor accepts for a given return.

• The “Investment Boundary” states how much risk an investor is willing to accept for a given return.

• For our purposes, we modified Markowitz’s approach a bit.

For our purposes, we modified Markowitz’s approach a bit. Acceptable Risk/Return Boundary Investment Region
For our purposes, we modified Markowitz’s approach a bit. Acceptable Risk/Return Boundary Investment Region

Acceptable Risk/Return Boundary

Investment Region

Investment

Risk/Return Boundary Investment Region Investment Copyright Copyright HDR HDR 2008 2008
Risk/Return Boundary Investment Region Investment Copyright Copyright HDR HDR 2008 2008
Risk/Return Boundary Investment Region Investment Copyright Copyright HDR HDR 2008 2008
Risk/Return Boundary Investment Region Investment Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

2929

“Full” RRA in 7 Steps

Define Define Decision Decision Calibrate Calibrate Model Model Estimators Estimators Populate Populate Model Model
Define Define Decision Decision
Calibrate Calibrate
Model Model
Estimators Estimators
Populate Populate Model Model
with with Calibrated Calibrated
Estimates Estimates & &
Measurements Measurements
Conduct Conduct Value Value
of of Information Information
Analysis Analysis (VIA) (VIA)
Analyze
Analyze Risk
Optimize Optimize
Remaining
Remaining Risk
Decision Decision
Optimize Remaining Remaining Risk Decision Decision Copyright Copyright HDR HDR 2008 2008
Optimize Remaining Remaining Risk Decision Decision Copyright Copyright HDR HDR 2008 2008
Optimize Remaining Remaining Risk Decision Decision Copyright Copyright HDR HDR 2008 2008
Optimize Remaining Remaining Risk Decision Decision Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

3030

Forecasting Fuel for Battle

Forecasting Fuel for Battle • The US Marine Corps with the Office of Naval Research needed

• The US Marine Corps with the Office of Naval Research needed a better method for forecasting fuel for wartime operations

• The VIA showed that the big uncertainty was really supply route conditions, not whether they are engaging the enemy

• Consequently, we performed a series of experiments with supply trucks rigged with GPS and fuel-flow meters

with supply trucks rigged with GPS and fuel-flow meters Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

3131

Reactions: Fuel for the Marines

• “The biggest surprise was that we can save so much fuel. We freed up vehicles because we didn’t have to move as much fuel. For a logistics person that's critical. Now vehicles that moved fuel can move ammunition.“ Luis Torres, Fuel Study Manager, Office of Naval Research

• “What surprised me was that [the model] showed most fuel was burned on logistics routes. The study even uncovered that tank operators would not turn tanks off if they didn’t think they could get replacement starters. That’s something that a logistician in a 100 years probably wouldn’t have thought of.” Chief Warrant Officer Terry Kunneman, Bulk Fuel Planning, HQ Marine Corps

Officer Terry Kunneman, Bulk Fuel Planning, HQ Marine Corps Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

3232

Final Tips

• Learn how to think about uncertainty, risk and information value in a quantitative way

• Assume its been measured before

• You have more data than you think and you need less data than you think

• Methods that reduce your uncertainty are more economical than many managers assume

• Don’t let “exception anxiety” cause you to avoid any observations at all

• Just do it

cause you to avoid any observations at all • Just do it Copyright Copyright HDR HDR

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

3333

Questions?

Doug Hubbard Hubbard Decision Research dwhubbard@hubbardresearch.com www.hubbardresearch.com 630 858 2788

If you want electronic copies of this presentation and

copies of supporting articles I mention, please leave me

a business card with “Presentation” written on the back

a business card with “Presentation” written on the back Copyright Copyright HDR HDR 2008 2008

CopyrightCopyright HDRHDR 20082008 dwhubbard@hubbardresearch.comdwhubbard@hubbardresearch.com

3434