METHODS AND SETTINGS FOR SENSITIVITY ANALYSIS

As we already know that for our model all cross-product terms

with i /= j
are zero, this can be reduced to the convenient
r

A
S

= 1,

(1.47)

in which uncertainty is divided among themes, each theme

comprising an indicator and its weight. It is easy to imagine
similar applications. For example, one could divide uncertainty
among observational data, estima- tion, model assumptions,
model resolution and so on.

1.2.16

Further Methods

So far we have discussed the following tools for sensitivity

analysis:

derivatives and sigma-normalized derivatives;

regression coefficients (standardized);
variance-based measures;
scatterplots.

We have shown the equivalence of sigma-normalized

coefficients
Sa
i = aZi aY/aY aXi, regression coefficients f3i and variance-based
first-order
sensitivity indices Si for linear models, as well as how Si is a
model-free
extension of the variance decomposition scheme to models of
unknown linearity. We have discussed how nonadditive models
can be treated in the variance-based sensitivity framework. We
have also indicated that scatter- plots are a powerful tool for
sensitivity analysis and shown how Si can be interpreted in
relation to the existence of shape in an Xi versus Y
scatterplot.
At a greater level of detail (Ratto et al., 2007) one can use
modern regres- sion tools (such as state-space filtering
methods) to interpolate points in the scatterplots, producing
very reliable E (Y I Xi = x) curves. The curves can then be used
for sensitivity analysis. Their shape is more evident than that of
dense scatterplots (compare Figure 1.7 with Figure 1.8).
Furthermore, one can derive the first-order sensitivity indices

2
directly from those curves, so that an efficient way to estimate
Si is to use state-space regression on the scatterplots and then
take the variances of these.
i
In general, for a model of unknown linearity,
monotonicity
and additivity, variance-based measures constitute a good
means of tackling settings such as factor fixing and factor
prioritization. We shall discuss one further setting before the
end of this chapter, but let us first consider whether there are
alternatives to the use of variance-based methods for the
settings so far described.
METHODS AND SETTINGS FOR SENSITIVITY ANALYSIS

Why might we need an alternative? The main problem with

variance- based measures is computational cost. Estimating the
sensitivity coefficients takes many model runs (see Chapter 4).
Accelerating the computation of sensitivity indices of all orders
or even simply of the Si, STi couple is the most intensely
researched topic in sensitivity analysis (see the filtering
approach just mentioned). It can reasonably be expected that
the estimation of these measures will become more efficient
over time.
At the same time, and if only for screening purposes, it would
be useful to have methods to find approximate sensitivity
information at lower sample sizes. One such method is the
Elementary Effect Test.

1.2.17

Elementary Effect Test

The Elementary Effect Test is simply an average of derivatives

over the space of factors. The method is very simple. Consider
a model with k independent input factors X i ,i = 1, 2 , . . . , k,
which varies across p levels. The input space is the discretized
p-level grid n. For a given value of X, the elementary effect of
the ith input factor is defined as
EEi
=

[Y (X1, X2, . . . , Xi1, Xi + Ll, . . . Xk) Y (X1, X2, . . . , Xk)]

(1.48)

Ll

where p is the number of levels, Ll is a value in {1/(p 1 ) ,. .. , 1

1/(p 1)}, X = (X1, X2, . . . Xk) is any selected value in n such
that the transformed point (X + eiLl) is still in n for each index i
= 1 , . . . , k, and ei is a vector of zeros but with a unit as its ith
component. Then the absolute values of the
EEi, computed at r different grid points for each factor, are
averaged

J. i

1
)
r

j=1

IEEi I
(1.49)

and the factors ranked according to the obtained imean J..

In order to compute efficiently, a well-chosen strategy is
needed for moving from one effect to the next, so that the

input space is explored with a minimum of points (see Chapter

3).
Leaving aside computational issues for the moment, J. is a useful
measure
for the following reasons:

1. It is semi-quantitative the factors are ranked on an interval

scale;

2. It is numerically efficient;
3. It is very good for factor fixing it is indeed a good proxy for
STi;

4. It can be applied to sets of factors.

METHODS AND SETTINGS FOR SENSITIVITY ANALYSIS

Due to its semi-quantitative nature the J. can be considered

a screening method, especially useful for investigating models
with many (from a few dozen to 100) uncertain factors. It can
also be used before applying a variance-based measure to
prune the number of factors to be considered. As far as point
(3) above is concerned, J. is rather resilient against type II
errors, i.e. if a factor is deemed noninfluential by J. it is unlikely
to be identified as influential by another measure.

1.2.18

Monte Carlo Filtering

While J. is a method of tackling factor fixing at lower sample

size, the next method we present is linked to an altogether
different setting for sensitivity analysis. We call this factor
mapping and it relates to situations in which we are especially
concerned with a particular portion of the distribution of output
Y . For example, we are often interested in Y being above or
below a given threshold. If Y were a dose of contaminant, we
might be interested in how much (how often) a threshold level
for this contaminant is being exceeded. Or Y could be a loss
(e.g. financial) and we might be interested in how often a
maximum admissible loss is being exceeded. In these settings
we tend to divide the realization of Y into good and bad. This
leads to Monte Carlo filtering (MCF, see Saltelli et al., 2004,
pp. 151191 for a review). In MCF one runs a Monte Carlo
experiment producing realizations of the output of interest
corresponding to different sampled points in the input factor
space, as for variance-based or regression analysis. Having
done this, one filters the realizations, e.g. elements of the Y
-vector. This may entail comparing them with some sort of
evidence or for plausibility (e.g. one may have good reason to
reject all negative values of Y ). Or one might simply compare Y
against thresholds, as just mentioned. This will divide the
vector Y into two subsets: that of the well-behaved realizations
and that of the misbehaving ones. The same will apply to the
(marginal) distributions of each of the input factors. Note that in
this context one is not interested in the variance of Y as much
as in that part of the distribution of Y that matters for
example, the lower-end tail of the distribution may be irrelevant
compared to the upper-end tail or vice versa, depending on the
problem. Thus the analysis is not concerned with which factor
drives the variance of Y as much as with which factor produces
realizations of Y in the forbidden zone. Clearly, if a factor has
been judged noninfluential by either J. or STi, it will be unlikely
to show up in an MCF. Steps for MCF are as follows:

A simulation is classified as either B, for behavioural, or B, for

nonbe- havioural (Figure 1.11).

Thus a set of binary elements is defined, allowing for the

identification of
two subsets for each Xi: one containing a number n of

INTRODUCTION TO SENSITIVITY ANALYSIS

elements denoted

(X |B )

(X |B )
B

Figure 1.11 Mapping behavioural and nonbehavioural realizations with

Monte Carlo filtering

(X I B) and a complementary X I B\ containing the remaining

n=
set (
N n simulations (Figure 1.11).
A statistical test can be performed for each factor
independently,
analysing theofmaximum
the cumulative
distributions
the (X I B)distance
and (X I between
B\ sets (Figure
1.12).

If the two sets are visually and statistically18 different, then

Xi is an influential factor in the factor mapping setting.

1
Prior
Fm(Xi|B )

0.8

Fn(Xi|B )

0.6
F(
Xi)
dn,n

0.4

0.2

0.2

0.4

0.6

0.8

Figure 1.12 Distinguishing between the two sets using a test

statistic

18

Smirnov two-sample test (two-sided version) is used in Figure 1.12 (see Saltelli
et al., 2004, pp. 3839).

POSSIBLE PITFALLS FOR A SENSITIVITY ANALYSIS

41

1.3 NONINDEPENDENT INPUT FACTORS

Throughout this introductory chapter we have systematically
assumed that input factors are independent of one another.
The main reason for this assumption is of a very practical
nature: dependent input samples are more laborious to
generate (although methods are available for this; see Saltelli
et al., 2000) and, even worse, the sample size needed to
compute sensi- tivity measures for nonindependent samples is
much higher than in the case of uncorrelated samples.19 For this
reason we advise the analyst to work on uncorrelated samples
as much as possible, e.g. by treating dependen- cies as explicit
relationships with a noise term.20 Note that when working with
the MCF just described a dependency structure is generated by
the filtering itself. The filtered factors will probably correlate
with one another even if they were independent in the original
unfiltered sample. This could be a useful strategy to circumvent
the use of correlated samples in sensi- tivity analysis. Still there
might be very particular instances where the use of correlated
factors is unavoidable. A case could occur with the parametric bootstrap approach described in Figure 1.3. After the
estimation step the factors will in general be correlated with
one another, and if a sample is to be drawn from these, it will
have to respect the correlation structure.
Another special instance when one has to take factors
dependence into consideration is when analyst A tries to
demonstrate the falsity of an uncer- tainty analysis produced by
analyst B. In such an adversarial context, A needs to show that
Bs analysis is wrong (e.g. nonconservative) even when taking
due consideration of the covariance of the input factors as
explicitly or implicitly framed by B.

1.4

POSSIBLE PITFALLS FOR A SENSITIVITY

ANALYSIS

As mentioned when discussing the need for settings, a

sensitivity analysis can fail if its underlying purpose is left
undefined; diverse statistical tests and measures may be
thrown at a problem, producing a range of different factor
rankings but leaving the researcher none the wiser as to
which

19

Dependence and correlation are not synonymous. Correlation implies

dependence, while the opposite is not true. Dependencies are nevertheless
described via correlations for practical numerical computations.
20

Instead of entering X1 and X2 as correlated factors one can enter X1 and X3 ,

with X3 being a factor describing noise, and model X2 as a function of X1 and X3
.

1
INTRODUCTION TO SENSITIVITY ANALYSIS
1
ranking to believe or privilege. Another potential danger is to
present sensi- tivity measures for too many output variables Y .
Although exploring the sensitivity of several model outputs is
sound practice for testing the quality of the model, it is better,
when presenting the results of the sensitivity anal- ysis, to
focus on the key inference suggested by the model, rather than
to confuse the reader with arrays of sensitivity indices relating
to intermediate output variables. Piecewise sensitivity analysis,
such as when investigating one model compartment at a time,
can lead to type II errors if interactions among factors of
different compartments are neglected. It is also worth noting
that, once a model-based analysis has been produced, most
modellers will not willingly submit it to a revision via sensitivity
analysis by a third party.
This anticipation of criticism by sensitivity analysis is also one
of the 10 commandments of applied econometrics according to
Peter Kennedy:
Thou
shall criticism
confess in
the
presence
of sensitivity.
Corollary:
Thou shall
anticipate
[
] When
reporting
a sensitivity
analysis,
researchers
should explain
fully their specification search so that the readers can judge for
themselves how the results may have been affected. This is basically
an honesty is the best policy approach, advocated by Leamer,
(1978, p. vi) (Kennedy, 2007).

To avoid this pitfall, an analyst should implement uncertainty

and sensi- tivity analyses routinely, both in the process of
modelling and in the oper- ational use of the model to produce
useful inferences.
Finally the danger of type III error should be kept in mind.
Framing error can occur commonly. If a sensitivity analysis is
jointly implemented by the owner of the problem (which may
coincide with the modeller) and a practitioner (who could again
be a modeller or a statistician or a practitioner of sensitivity
analysis), it is important to avoid the former asking for just
some technical help from the latter upon a predefined
framing of the problem. Most often than not the practitioner
will challenge the framing before anything else.

1.5 CONCLUDING REMARKS

1. We have just shown different settings for sensitivity analysis,

1
2 such as:

INTRODUCTION TO SENSITIVITY ANALYSIS

factor prioritization, linked to Si;

factor fixing, linked to STi or u;
factor mapping, linked to MCF;
metamodelling (hints).

CONCLUDING REMARKS

43

The authors have found these settings useful in a number of

applications. This does not mean that other settings cannot
be defined and usefully applied.

2. We have discussed fitness for purpose as a key element of

3.

4.

5.

model quality. If the purpose is well defined, the output of

interest will also be well identified. In the context of a
controversy, this is where attention will be focused and where
sensitivity analysis should be concentrated.
As discussed, a few factors often account for most of the
variation. Advantage should be taken of this feature to
simplify the results of a sensitivity analysis. Group sensitivities
are also useful for presenting results in a concise fashion.
Assuming models to be true is always dangerous. An uncertainty/sensitivity analysis is always more convincing when
uncertainty has been propagated through more than just one
model. Using a parsi- monious data-driven and a less
parsimonious law-driven model for the same application can be
especially effective and compelling.
When communicating scientific results transparency is an asset.
As the assumptions of a parsimonious model are more easily
assessed, sensitivity analysis should be followed by a model
simplification.

The reader will find in this and the following chapters didactic
examples for the purpose of familiarization with sensitivity
measures. Most of the exercises will be based on models whose
output (and possibly the associated sensitivity measures) can be
computed analytically. In most practical instances the model under
analysis or development will be a computational one, without a
closed analytic formula.
Typically, models will involve differential equations or
optimization algo- rithms involving numerical solutions. For this
reason the best available prac- tices for numerical computations
will be presented in the following chapters. For the Elementary
Effects Test, we shall offer numerical procedures devel- oped by
Campolongo et al. (1999b, 2000, 2007). For the variance-based
measures we shall present the Monte Carlo based design
developed by Saltelli (2002) as well as the Random Balance
Designs based on Fourier Amplitude Sensitivity Test (FAST-RBD,
Tarantola et al., 2006, see Chapter 4). All these methods are based
on true points in the space of the input factors, i.e. on actual
computations of the model at these points. An important and
powerful class of methods will be presented in Chapter 5; such
techniques are based on meta- modelling, e.g. on estimates of the
model at untried points. Metamodelling allows for a great reduction

in the cost of the analysis and becomes in fact the only option
when the model is expensive to run, e.g. when a single simulation
of the model takes tens of minutes or hours or more. The
drawback is that metamodelling tools such as those developed
by Ratto et al. (2007) are less straightforward to encode than
plain Monte Carlo. Where possible, pointers will be given to
available software.

1
5

ANSWERS

1.6 EXERCISES
V(Y ) = E(Y 2) E2(Y ).

1. Prove
that

2. Prove that for an additive model of two independent variables

3.
4.

X1 and
X2, fixing one variable can only decrease the variance of the
model.
Why in f.L are absolute differences used rather than simple
differences?
If the variance of Y as results from an uncertainty analysis is
too large, and the objective is to reduce it, sensitivity analysis
can be used to suggest how many and which factors should be
better determined. Is this a new setting? Would you be
inclined to fix factors with a larger first-order term or rather
those with a larger total effect term?

5. Suppose X1 and X2 are uniform variates on the interval [0, 1].

6.
7.
8.

What is the mean? What is the variance? What is the mean of

X1 + X2? What is the variance of X1 + X2?
Compute Si analytically for model (1.3, 1.4) with the following
values:
r = 2,a = {1, 2} and O = {2, 1}.
Write a model (an analytic function and the factor distribution
functions)
in which fixing an uncertain factor increases the variance.
What would have been the result of using zero-centred
distributions for the Ds in Equation (1.27)?

1.7 ANSWERS
1. Given a function Y = f (X) where X = (X1, X2, Xk) and X p
(X)
(
where p (X) is the joint distribution of X with p (X) dX = 1,
then the
function mean can be defined as
r
E(Y ) = f (X) p (X) dX,
and its variance as
Var(Y) = r (f(X) E(Y ))2p(X)dX

1
6

INTRODUCTION TO SENSITIVITY ANALYSIS

= f (X)p(X)dX + E (Y )
2

E(Y )f(X)p(X)dX

= E(Y 2) + E2(Y ) 2E 2 (Y)

= E(Y 2) E2(Y ).
Using this formula it can easily be proven that Var(Y) = Var(Y
+ c), with c an arbitrary constant. This result is used in Monte
Carlo-based

variance (and conditional variance) computation by rescaling

all values of Y subtracting E(Y ). This is done because the
numerical error in the variance estimate increases with the
value of Y .

2. We can write the additive model of two variables X1

and X2 as Y = f1 (X1) + f2 (X2), where f1 is only a function of X1

and f2 is only a function of X2.
Recalling
that the variance of (Y ) can be written as V(Y ) = E
)
Y2
E2 (Y ), where E stands for the expectation value, and applying
it to Y
we obtain
)
V(Y ) = E f 2 + f 2 + 2f f + f ).
E2 (f
1

1 2

Given that E (f1f2) = E (f1) E (f2) for independent variables, then

the above can be reduced to
)
)
V(Y ) = E f 2 + E f 2 E2 (f ) E2 (f ),
1

which can be rewritten as

V(Y ) = V (f1) + V (f2) ,
which proves that fixing either X1 or X2 can only reduce the
variance of Y .
3. Modulus incremental ratios are used in order to avoid
positive and
negative values cancelling each other out when calculating the
average.
4. It is a new setting. In Saltelli et al. (2004) we called it the
variance cutting setting, when the objective of sensitivity
analysis is the reduction of the output variance to a lower
level by fixing the smallest number of input factors. This
setting can be considered as relevant in, for example, risk
assessment studies. Fixing the factors with the highest total
effect term increases our chances of fixing, besides the firstorder terms, some interaction terms possibly enclosed in the
totals, thus maximizing our chances of reducing the variance
(see Saltelli et al., 2004).
5. Both X1 and X2 are uniformly distributed in [0, 1], i.e.
p(X1) = p(X2) = U(0, 1).
This means that p(Xi) is 1 for Xi [0, 1] and zero otherwise.
Thus

r
E(X1) = E(X2) =

p(x)xdx
=

x=0

r
r

1
x=
0

p(x
)

Further:
Var(X1) = Var(X2)

x2

1
2

dx
=

1
x=
0

x2 x +1 dx
4

x3

x2

=[

+ x]0 = + =
3
2
4
3
2
4
12
E(X1 + X2) = E(X1) + E(X2) = 1,
as the variables are separable in the integral.
Given that X1 + X2 is an additive model (see Exercise 1) it is
also true that
1
Var(X1 + X2) = Var(X1) + Var(X2) =

.
6
The same result is obtained integrating explicitly
Var(X1 + X2)
=

x1

x2

=0

=0

dx1dx2.

p(x) (x + x
2

1)

6. Note that the model (1.3, 1.4) is linear and additive. Further,

its prob- ability density function can be written as the

product of the factors marginal distributions (independent
factors). Writing the model for r = 2 we have
Y(Z1, Z2) = n1Z1 + n2Z2
wit
h

(Z1)2
1
Z1 N (0, 1) or equivalently
p(Z1) =

aZ

2
e 2aZ1
21r

and a similar equation for p(Z2). Note that by definition the

distributions are normalized, i.e. the integral of each p(Zi)
over its own variable Z1 is 1, so that the mean of Y can be
reduced to
E(Y ) =

n1

+oo
oo

Z1p(Z1)dZ1 +
n2

r
+oo

Z2p(Z2)dZ2 .

oo

xex2 dx, whose

ex2 /
2
primitive

vanishes at the extremes of integration, so that E(Y ) = 0.

Given that the model is additive, the variance will be
These integrals are of the type

V(Y ) = VZ1 + VZ2 = V (n1Z1) + V (n 2Z2).

For either Z1 or Z2 it will be

V = V (n Z ) = n2V (Z ).
Zi

i
i

We
write
2

V(Zi) = E(Zi ) E (Zi) = E (Zi )

an
d

1
E(Z2) =
i
2 aZi

oo

2 zi /2aZ

zi e

oo

dzi .

The tabled form is

r

+oo

2 at
d
t e
t
= 4
which gives with an easy transformation
0

E(Z2) = a 2
i

Zi

so that
2

VZ = Di aZ
i

an
d
V(Y ) = D2 a2
1

an
d

+ D2a2

Z1

Z2

2
D2

SZ =
i

aZ
i

V(Y ) .

Inserting the values a = {1 2} and n = {2 1} we obtain V(Y )

= 8 and
1
SZ1 = SZ2 = 2
. The result above can be obtained by explicitly applying the
formula
for Si to our model:
SZ = V(E(Y I Z ))
i
i
V(Y )
which entails computing first E(Y I Zi = z). Applying this to our
i
model
Y = D1Z1 + D2Z2 we obtain, for example, for factor Z1:
E(Y I Z1 = z )
=
r
+oo

p(z )p(z ) (D z + D z ) dz
dz

= D z .

oo

1
1

1
2

Hence VZ1 the variance over z1 of D1z1 is, as before, equal

to D1aZ1
and
D2 a 2i
SZ =
.
i
i
D2 2 Z 2 2
1aZ1 + D2aZ2

7. We consider the model Y = X1 X2, with the factors identically

distributed as

X1, X2 N (0, ).
Based on the previous exercise it is easy to see that
E(Y ) = E (X1X2) = E (X1)E(X2) = 0,
so
that

V(Y ) = E X 2 X 2 = E X 2 E X 2 =

If X2 is fixed to a generic value

2 x , then

E (X1x ) = x E (X ) = 0
2

as in a previous exercise, and

2 = (x 2
(X2 (x ) )
E2 X 2 = (x
V(Y I X2 = x ) = V (X x )
)
=E

2
2

It is easy to see that V (X21x ) becomes bigger than V(Y )

whenever the
modulus of x is bigger than .
2
Further, from the relation
V(Y I X2 = x ) = (x )2
2

one gets

Given that

it must be
that

E(V(Y I X2)) =

= V(Y ).

E(V(Y I X2)) + V(E(Y I X2)) =

V(Y ) V(E(Y I X2)) = 0,

i.e. the first-order sensitivity index is null for both X1 and X2.
These results are illustrated in the two figures which follow.
2
Figure 1.13 shows a plot of
(Y I X2 = x ),
X
V

i.e. VX
(Y I X2 = x2 ) at 1
2

different values of2 x for = 1. The horizontal line is the

unconditional
variance of Y . The ordinate is zero2 0, and becomes higher
than
for x =

V(Y ) for x2 1.
Figure 1.14 shows a scatterplot of Y versus
x (the same
1

shape would
appear
for
x
).
It
is
clear
from
the plot that
2
whatever the value of