Ecological Modelling 141 (2001) 171 183

www.elsevier.com/locate/ecolmodel

PlackettBurman technique for sensitivity analysis of

many-parametered models
Diane L. Beres a,*, Douglas M. Hawkins b
b

a
Conser6ation Biology Program, Uni6ersity of Minnesota, 1987 Upper Buford Circle, St. Paul, MN 55108, USA
Department of Applied Statistics, 357 Ford Hall, Uni6ersity of Minnesota, 224 Church St. SE, Minneapolis, MN 55455, USA

Received 7 June 1999; received in revised form 15 February 2001; accepted 23 February 2001

Abstract
A method for sensitivity analysis of a simulation model is presented, together with an illustrative example. The
PlackettBurman technique allows the concurrent consideration of numerous parameters. The required number of
model scenarios necessary for completion of the analysis is approximately twice the number of parameters.
Advantages to the technique include simultaneously investigating all parameters, as well as acquiring information
about two-way parameter interactions, by means of a relatively small number of scenarios. A drawback is aliasing
among interactions, which may confound interpretation. 2001 Elsevier Science B.V. All rights reserved.
Keywords: Sensitivity analysis; Parameter interactions; Plackett Burman; Simulation model

1. Introduction
Sensitivity analysis illuminates the relative importance of a models parameters in bringing
about outcomes. In a sensitivity analysis, one
systematically and comprehensively tests to see
how changes in the parameters of the model affect
the models output (Starfield and Bleloch, 1991,
p. 58). Which of the parameters exert a significant
influence on the output variables? Which are inconsequential? Do increments in any parameters
* Corresponding author. Present address: Department of
Biology, Ripon College, PO Box 248, 300 Seward Street,
Ripon, WI 54971-0248, USA; Tel.: +1-920-7485874; fax:
+1-920-7487243.
E-mail address: beresd@mail.ripon.edu (D.L. Beres).

produce unexpectedly large alterations in results?

Without the answers to these questions, understanding of a model is incomplete. Recommendations based upon a model without explicit
sensitivity analysis lack foundation. Thus, thorough sensitivity analysis is a significant aspect of
every modelers job. Unless the hierarchy of
parameter strength is revealed, along with any
information or insights gained from the model,
the modeling task is unfinished (Beres et al.,
2001). As the term sensitivity analysis is used in
this paper, it applies to a wide range of models,
including both simulation and analytic models.
Despite the acknowledged importance of sensitivity analysis, there is a dearth of information
(Henderson-Sellers and Henderson-Sellers, 1996,
p. 291) on how it should be performed. There is

0304-3800/01/$ - see front matter 2001 Elsevier Science B.V. All rights reserved.
PII: S 0 3 0 4 - 3 8 0 0 ( 0 1 ) 0 0 2 7 1 - X

172

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

no single, universally accepted procedure for the

sensitivity analysis of stochastic models (McCarthy et al., 1995, p. 93). A common approach
to sensitivity analysis is to explore the effects of
changing parameters, one at a time, on a target
output variable (Henderson-Sellers and Henderson-Sellers, 1996; Swartzman and Kaluzny, 1987).
This procedure holds all other things are equal
while each parameter is altered in turn, to determine its effect in isolation from the possible effects of others. Indeed, the definition for the
sensitivity, S, of a parameter, P given by S=
[(x/x]/[(P/P], where x is the state variable under
consideration (Jorgensen, 1994, p. 23) implies
that the analysis will proceed one parameter at a
time. Often an increment (up or down) of 10% of
the nominal or usual value is tested (McCarthy
et al., 1995; Burgman et al., 1993). The sensitivity
investigation may consider the state variables of
greatest interest in the model (Jorgensen, 1994,
p. 57) and omit other parameters presumed
inconsequential.
Even when the sensitivity of all individual
parameters is investigated, the one-at-a-time
(OAAT) method does not uncover potentially
important interactions between two parameters
(Daniel, 1973). Possibly significant effects could
be produced by a pair of parameters acting in
concert (Burgman et al., 1993). Such interaction
effects would be o6er and abo6e the sum of the
individual effects of the two parameters in question. Furthermore, these interactions would not
necessarily be caused by pairs of the most important individual parameters. Thus, an experimental
design for sensitivity analysis, which will give
insight into these otherwise unknown interactions,
is needed (Henderson-Sellers and Henderson-Sellers, 1996; Daniel, 1973). The modeler is interested
in both the main effects of individual parameters and also the 2-way interactions of pairs of
parameters (Swartzman and Kaluzny, 1987). Potentially, there could also be notable higher order
interactions (3-way, 4-way, etc.). The sparsity of
effects principle (Montgomery, 1997) notes that
these are much less common than 2-way interactions, so higher order interactions will be disregarded in the ensuing discussion.

One way to the obtain the interactions of interest would be to implement the sensitivity analysis
by means of a complete factorial design. The
complete factorial would consist of all possible
combinations (assemblies) of selected high and
low values for the parameters. Using an upper
and lower value for each parameter, such a design
would reveal interactions of all orders in addition
to main effects. The number of distinct scenarios
required to execute the complete factorial is 2c
where c is the number of parameters and 2 is a
consequence of using two values for each. For
even as few as 10 parameters, implementing a
complete design would be inconvenient (210 =
1024), to say the least. Henderson-Sellers and
Henderson-Sellers (1996) recommended using
fractional factorial experimentation to reduce the
number of scenarios needed for the sensitivity
analysis. However, for 10 parameters, even the 1/4
fractional design would require a fairly large number (28 = 256) of distinct scenarios.
Another approach might be to devise a design
combining OAAT technique together with consideration of all pairings of parameters. Such an
approach is feasible when there are but a few
parameters (56 scenarios for the case of 10
parameters). However, if the model contains a
large number of parameters, as may be the case
with ecological models, even this combined design
might not be practicable. A better option is explained below.
A Plackett Burman (PB) design (Plackett and
Burman, 1946) with foldover (Box et al., 1978;
Montgomery, 1997) provides an alternative which
allows simultaneous examination of the entire
suite of parameters and is both convenient and
informative. The PB design specifies a selected
subset of the scenarios prescribed by a complete
factorial. Thus, in contrast to the sampling
schemes for stochastic model sensitivity analysis
described by Swartzman and Kaluzny (1987), PB
provides a simple blueprint for accomplishing the
investigation. Furthermore, PlackettBurman
sensitivity analysis (PBSA) is context-free; it is
equally useful for any sort of model having many
parameters (and a numerical response), but especially for computer-coded simulation models.
PBSA addresses the subject of the sensitivity of
stochastic models in a prescriptive way.

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

Implementing the folded-over PB design

(foldover is explained below) enables the estimation of effects with the same accuracy as if
attention had been concentrated on varying a
single component throughout (Plackett and Burman, 1946, p. 305) and requires approximately
twice as many scenarios as parameters (components). The precise number of scenarios for the
folded-over design is 2 times that multiple of 4
which is next greater than the number of parameters. For example, with 10 parameters, 24 scenarios would be needed: 12 is the next larger multiple
of four, and 212 is 24. Thus, with 10 parameters PBSA provides a considerable savings over
the 1024 scenarios which would be required for a
complete factorial design. And PBSA is superior
to a half- (or quarter-) factorial design because, in
the case of 10 parameters, that design would still
need 512 (or 256) scenarios. Hence PBSA has the
advantage of efficiency while still being able to
detect interactions. Also, effects of a parameter
are not measured with all other things being
equal, but are averaged over variations made in
all other parameters.
Plackett and Burman devised their designs in
the context of assemblages of components in a
manufacturing process. Ergo, the terms component (manufacturing), factor (experimental design),
and
parameter
(modeling)
are
interchangeable in this discussion. Plackett and
Burman described the problem of determining

Table 1
Concise outline of method for PBSA of simulation model
(1) Determine parameters to be tested; select upper and
lower values.
(2) Find appropriate PB pattern.
(3) Create PBSA matrix.
(4) Run prescribed scenarios.
(5) Calculate (main) effect of each parameter on target
response variable.
(6) Sort effects; decide which are important.
(7) Calculate effects of two-way interactions of paired
parameters.
(8) Sort interaction effects; decide which are important.
(9) Interpret results.

173

suitable tolerances for the components of a certain assembly; more generally of ascertaining the
effect of quantitative or qualitative alterations
in the various components upon some measured
characteristic of the complete assembly (Plackett
and Burman, 1946, p. 305) This description
is a reasonable definition of sensitivity analysis if
the measured characteristic is understood
to be an output variable from the model under
scrutiny.
To summarize, the rationale for PBSA includes
these points:
PBSA is not a OAAT method
PBSA finds 2-way interactions
PBSA is not restricted to any particular type of
model
PBSA is prescriptive, using pre-determined
designs
PBSA is efficient in terms of number of scenarios needed
PBSA designs for up to 100 parameters are
readily available
PBSA rankings are easy to compute
PBSA works with categorical as well as numerical parameters
PBSA does not require parameters to be considered over identical intervals
PBSA is statistically sound
Despite these virtues, PBSA does not appear to be
well known by ecologists. None of the references
located in a science citations search on the Plackett and Burman (1946) paper were from ecological
or biological literature.
Below is a guide for performing PBSA, accompanied by a detailed example. This tutorial is
intended for an audience conversant in the vocabulary of mathematics and modeling. No particular
statistical background is required. An outline of
the procedure (Table 1) provides an overview. The
guide consists of explanation for each of the steps
in the Table 1 outline. A sample analysis, crafted
to illustrate PBSA technique, follows the guide. A
conventional OAAT sensitivity analysis for the
Example Model follows the PBSA. The drawbacks of the OAAT are delineated. Further information on PBSA may be found in many standard
texts on design and analysis of experiments.

174

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

2. How to implement Plackett Burman sensitivity

analysis

2.1. Determine the parameters to be tested

Enumerate a list of the models parameters,
ordered in any way which makes sense. Any
ordering may be used, but it must remain fixed
throughout the procedure. There is no particular
advantage to paring down the number of parameters n; all should be included in order to gain the
maximum benefit of the PBSA. For each parameter, select an upper and lower value. These values
should be the endpoints of a plausible range for
the parameter. The typical 910% rationale
should be followed only if it makes sense in the
models context. The ranges do not need to be
identically sized, but their selection should be
reasoned. The analysis will indicate the effects of
varying each parameter by the amount of the
chosen range.

2.2. Find the appropriate Plackett Burman

pattern
A paper in Biometrika (Plackett and Burman,
1946) is the original source for PB patterns. These
patterns are lists, or strings, of (ordered) pluses
and minuses. The designs come in sizes which are
multiples of four. The article contains patterns for
all designs up to 100, except for 92 (for which see
Baumert et al., 1962). Texts on design and analysis of experiments (e.g., Montgomery, 1997) may
also have PB patterns which would accommodate
up to a couple dozen parameters. The actual
length of the string, i.e., the number of signs in
the list, is one less than the design size d. Select
any d for which the number of model parameters
n Bd; the smallest such d requires running the
fewest sensitivity analysis scenarios of the model.
It is not required that the smallest possible d be
used, only that d should be larger than n.

pattern form the next d 2 rows. Then append a

row of all minuses. The matrix at this point has d
rows. The foldo6er consists of another set of d
rows, having the exact opposite signs of the original rows. This completes the PBSA matrix; it has
2d rows, the last consisting of all pluses.
Before the foldover, the matrix contains a Resolution III design, a statistical term which means
that the main effects of individual factors are
separate from each another, but may be impossible to distinguish from some 2-way interaction
effects, i.e., they are aliased. Adding the foldover
raises the design to Resolution IV (Box and
Draper, 1987). Increasing the resolution means
that the main effects are not aliased with one
another nor with 2-factor interactions, although
some of the 2-factor interactions may be aliased
with one another. It is not difficult to resolve
apparent aliasing, with additional runs, but the
explanation of how to do so is beyond the scope
of this paper.

2.4. Run the prescribed scenarios

Each row of the PBSA matrix specifies a distinct scenario of the model. Each column is associated with a particular parameter/factor/
component. Column 1 is assigned to the first item
in the established parameter list, Column 2 the
second, etc. If the number of parameters is less
than the number of columns (i.e., nBd 1), ignore the surplus columns. The signs in the rows of
the PBSA matrix indicate whether the upper or
lower of the extremes chosen for each parameter
should be used in a particular scenario. Where
pattern has plus, use upper parameter value;
where pattern has minus, use lower parameter
value. Run the 2d scenarios specified and record
the results. If the model is stochastic, determination of the appropriate number of replicates is the
province of the modeler.

2.5. Calculate the (main) effect of each parameter

2.3. Create a Plackett Burman sensiti6ity
analysis matrix
The matrix is an array of signs. The top row is
the PB pattern. All the cyclic permutations of this

Each scenario (an assemblage of upper and

lower values for the parameters) will produce
output (target variables or quantities). The number of output variables will depend on the model

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

being analyzed. Analysis for each output variable

follows the procedure explained here. To determine the (main) effect of particular component/
factor/parameter, sum the (signed) outputs as
follows. Take the signs from the PBSA matrix
column associated with that parameter, attribute
the signs to the output values, row by row, sum
these numbers and then divide the sum by the
design size (d). Thus, for each parameter an effect
on the target variable has been given a PB score.
Each column of the PBSA matrix contains d
pluses and d minuses. An equivalent way to calculate the score is to average of all of the plus
outputs and subtract from this the average of all
of the minus outputs.

2.6. Sort the effects decide which are important

The major influences are caused by the parameters having the largest scores from the preceding
procedure. The standard interpretation of the
(main) effects is a change of this amount in
component/factor f produces a corresponding
change of indicated size in response variable r.
Alternately, it can be said that the ratio of the
size of the change in factor to the size of effect has
the observed magnitude. The sign of the score
indicates the direction of the change for the target
variable. Thus, positive scores indicate an increase
in the response when the parameter is raised and
negative scores mean a decrease in the response
accompanies an increase in the parameter.

2.7. Calculate the effects of two-way interactions

of the parameters
To determine the effects of 2-way interactions
of pairs of parameters, find the appropriate sign
for the pair by multiplying the signs for the
individual parameters, row by row: if the signs are
the same, the result is plus, if they are different,
minus. The matrix of sign products thereby created will have C(n,2) columns, each column representing one of the possible pairs. Associate these
derived signs with the output for each scenario
run (row by row, as for main effects); sum the
(signed) outputs, divide the sum by d, and sort the
results. Identical scores may occur for more than

175

one pair, a consequence of aliasing (see next

paragraph).
For each target variable, the influence of pairs
can be determined, subject to aliasing. That is,
some of the effects are actually the consequence of
influence from more than one pair. Those pairs
having the same PBSA score are members of an
alias group. For tractability, assume that the pairs
within an alias group whose constituent elements
were determined individually to be most significant are the ones responsible for the main portion
of the groups effect and the other pairs may be
disregarded sparsity of effects principle (Montgomery, 1997). But, in truth, the observed effects
are the result of a linear combination of the pairs
having the same outcomes from the calculation
procedure. Thus, the assumption is that the coefficients for some of the terms in the linear combination are very small, making those terms negligible.

2.8. Sort the interaction effects decide which

are important
Use the same process as that applied to the
individual parameter effects, explained above.

2.9. Interpret the results

The standard interpretation of the 2-way interactions is changes of these amounts in components/factors/parameters f and g together produce
a corresponding change of indicated size in response variable r. This effect is over and above
the individual influences of factors f and g. At a
more sophisticated level of interpretation, a normal probability plot of the effects may be used to
draw a line between those large enough to be
real and those that could be due to randomness
in the model runs (Daniel, 1976).
Meaning and consequences of the analyses lie
within the context of the model and remain the
judgment of the investigator.

3. Example: application of PlackettBurman

sensitivity analysis to a simple model
The example here has been drastically sim-

176

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

Table 2
Parameter values selected for PBSA of Example Model
Parameter

Lower value

Upper value

S0 (chick survival)
S1 (juvenile survival)
S2 (subadult survival)
S3 (adult survival)
DC (double-clutching)
RF (release fraction)

0.675
0.675
0.765
0.85
No
0.7

0.825
0.825
0.935
0.99
Yes
0.9

plified from a real population model. It is designed to focus on the PBSA procedure and allow
uncomplicated explanations. The example is
derived from a detailed model of an endangered
North American bird which has its population
augmented through a captive breeding program.
However, the point of this paper is to focus on
the PBSA features and technique, not the particular model or species. The example has been
severely pared down to present a useful illustration; it is not intended to represent reality. A full
analysis of the actual model utilized 128 combinations of its 57 parameters and reported on three
response variables (Beres, 2000). The details of the
model structure are unimportant for the understanding of PBSA methodology and are not discussed here. The Example Model predicts the
number of individuals after a designated period of
time in a small population of long-lived birds. The
population has two components, captive and wild.
Annual releases from a captive-breeding program
augment the wild population. Outputs other than
population are available from the actual model,
but one response variable suffices for purposes of
demonstrating the execution of PBSA. The size of
the wild (sub)population at the end of a 30-yr
period is the response variable used here. It
should be observed, however, that the PBSA procedure is equally applicable to all sorts of manyparametered ecological models, not only to
population models.

3.1. Determine the parameters to be tested

The example analysis examines six parameters:
annual survival probability for wild birds in each

of four age classes (S0, S1, S2, S3), whether or not

management should include double-clutching
(double-clutching, commonly used to increase a
captive population faster than normal, refers to
removal of the first clutch of eggs from a nest in
order to promote the laying of a second clutch) in
the captive population (DC), and the portion of
the years captively-bred juveniles to release into
the wild (RF). Extreme values have been selected
for each parameter (Table 2). The ordering, S0,
S1, S2, S3, DC, RF remains fixed throughout the
procedure. Observe that the ranges selected for
the survival parameters are not uniform in size.
Note that two of the parameters (DC, RF) are
not attributes of the population itself, but are
management decisions about the population. Furthermore, one of these (DC) is essentially a
switch, i.e., either DC is in effect (on) or not
(off); there is no partial DC.

3.2. Find the appropriate PlackettBurman

pattern
Consulting Plackett and Burman (1946) provides the string of seven signs [+ + + +
] which corresponds to a PB design of size eight.
If there were eight parameters, design size 12
(having 11 signs) would be needed. In that case,
three columns of the constructed matrix would be
ignored. It is convenient to disregard the final
columns, but any three could be inactive. For
the example, six of the seven design columns are
needed; the 7th is unused.

3.3. Create a PlackettBurman sensiti6ity

analysis matrix
The pattern itself forms row 1, its cyclic permutations form rows 27. In this case, the permutations were cycled to the right. It makes no
difference to the results of the analysis which
direction is used. Row 8 is all minuses, completing
the top half of the desired matrix (Table 3). Next,
each of the signs in the top half is switched to give
an additional eight rows, and these are appended
to the previous matrix, yielding the PBSA matrix
(Table 4). Note that the final row is all pluses;
there are 16 rows and seven columns.

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

3.4. Run the prescribed scenarios

The PBSA matrix has 16 rows hence 16 scenarios are necessary for the analysis. Each of the first
six columns is associated with a parameter and
each row specifies the parameter settings for a
scenario. The 7th column is disregarded. Hence,
combining Table 2 (transposed) and Table 4 specifies the actual parameter settings for the required scenarios of the model (Table 5).
Replication of each scenario must occur at an
appropriate level for the model being analyzed.
Each scenario of the Example Model was run
with 1000 replicates using the same settings. The

Table 3
Top half of PBSA matrixa
+

+
+

+
+

+
+
+

+
+
+

+
+
+

+
+
+

+
+
+

Row 1 is the PB pattern from design of size 8.

Table 4
Complete PBSA matrixa
+

+
+

+
+

+
+

+
+

+
+
+

+
+

+
+

+
+
+

+
+

+
+
+

+
+
+

+
+
+

+
+

+
+
+

+
+

a
Top half is identical to Table 3. Lower half is formed by
foldover: signs in the lower eight rows are opposite to those in
upper eight rows. Columns l6 associate with the parameters:
S0, S1, S2, S3, DC, RF. Final column is unused.

177

mean of the output from these replicates is the

target response variable P, wild population after
30 years.

3.5. Calculate the (main) effect of each parameter

To determine the effect of S3 on P, take the
vector of signs from Column 4 of the PBSA
matrix (Table 4) and associate row by row with
the output numbers from the population column
of Table 5. The sum of the resulting signed means
[53.4+128.9+126.8 + 62.2 17.8 25.4+ 39.0
+ 235.6 + 33.1 12.1 11.1 25.2 + 86.9 + 61.5
40.85.9] is 582.1. Divide the sum by eight
which is the design size d. The main effect of S3
on P is thus determined to be (582.1)/8 = 72.8.
Explanation is given in the next section. The
actual output numbers and computations contained three decimal places; only one is reported
here.
In the same way, calculate the (main) effects on
population for the other parameters (Table 6). If
there are multiple output variables of interest,
their values as derived from the same 16 scenarios
may be used to calculate effects on them. Additional scenarios are not required. A spreadsheet,
such as EXCEL, makes the computations quite
simple, even for a large number of parameters and
output variables.

3.6. Sort the effects decide which are important

Sorting the scores from the previous step gives
a ranking for the main effects (Table 7).
Examination of Table 7 indicates that all of the
effects are positive. Intuitively, the effects of improved survival should be positive on the population, so this seems reasonable. The analysis
provides evidence that the two management
strategies have the potential to increase the population over the time span modeled. However, the
most significant effects are shown to be the survival of birds in the two highest age classes, S2
and S3.
The magnitude of the effects for S2 or S3 is
seen to be approximately twice that of the next
highest parameter DC. Survival for the youngest

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

178

Table 5
PBSA parameter values for Example Model, with population (response variable) outputa

Scenario
Scenario
Scenario
Scenario
Scenario
Scenario
Scenario
Scenario
Scenario
Scenario
Scenario
Scenario
Scenario
Scenario
Scenario
Scenario
a

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

S0

S1

S2

S3

DC

RF

Population

0.825
0.675
0.675
0.825
0.675
0.825
0.825
0.675
0.675
0.825
0.825
0.675
0.825
0.675
0.675
0.825

0.825
0.825
0.675
0.675
0.825
0.675
0.825
0.675
0.675
0.675
0.825
0.825
0.675
0.825
0.675
0.825

0.935
0.935
0.935
0.765
0.765
0.935
0.765
0.765
0.765
0.765
0.765
0.935
0.935
0.765
0.935
0.935

0.85
0.99
0.99
0.99
0.85
0.85
0.99
0.85
0.99
0.85
0.85
0.85
0.99
0.99
0.85
0.99

Yes
No
Yes
Yes
Yes
No
No
No
No
Yes
No
No
No
Yes
Yes
Yes

0.7
0.9
0.7
0.9
0.9
0.9
0.7
0.7
0.9
0.7
0.9
0.7
0.7
0.7
0.9
0.9

53.4
128.9
126.8
62.2
17.8
25.4
39.0
5.9
33.1
12.1
11.1
25.2
86.9
61.5
40.8
235.6

( 914.4)
( 926.4)
( 925.1)
( 914.9)
( 96.6)
( 98.9)
( 911.5)
( 93.4)
( 99.6)
( 95.5)
( 95.5)
( 98.7)
( 920.6)
( 915.1)
( 911.9)
( 946.3)

Population numbers (birds) are means ( 9 S.D.) over 1000 replicates having the settings specified by the particular scenario.

Table 6
Effects of parameters on response variable P (population)

Main effects on population

S0

S1

S2

S3

DC

RF

10.7

22.4

60.0

72.8

31.8

18.0

Table 7
Effects (of parameters on response variable) in descending order

Main effects on population, sorted

S3

S2

DC

S1

RF

S0

72.8

60.0

31.8

22.4

18.0

10.7

birds (S0) is the least influential parameter. The

release fraction RF has a relatively small
influence.
Specifically, for the adult age class S3, increasing the survival by 0.07 is expected to provide the
wild population with approximately 35 more birds
(not necessarily adults) over the modeled time. To
draw this conclusion, find the point halfway between the extreme values chosen for S3 (Table 2),
(0.99 0.85)/2 =0.07. This represents the amount
of change that was under consideration in the
PBSA procedure. Hence, this amount of change
(up or down) is found to have an effect on the
wild population of (72.8)/2 =36.4, or about 35.

The direction of the change in S3 and the anticipated increment in Population is the same; if S3 is
decreased by 0.07, then the result would be 35
fewer birds. For survival in the youngest age class
S0, the effect of a increment of 0.075, either up or
down, is a change of approximately five birds, in
the same direction, at the end of the time period
modeled. The size of the change for S0 was calculated by (0.8250.675)/2 = 0.075 and the score
10.7 was divided by two to get the size of the
related effect. Although the increment in S0 is
larger than the one is S3, it has a smaller effect.
Approximately 30 more birds in the wild population (over the modeled time) is the anticipated

Scenario (row)

S3S0

S3S1

S3S2

S3DC

S3RF

S2S0

S2S1

S2DC

S2RF

S1S0

S1DC

S1RF

S0DC

S0RF

DCRF

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

+
+

+
+

+
+

+
+

+
+
+

+
+
+

+
+

+
+

+
+

+
+

+
+

+
+
+

+
+

+
+

+
+

+
+

+
+
+

+
+
+

+
+

+
+
+

+
+
+

+
+
+

+
+

+
+

+
+

+
+

+
+

+
+

+
+

+
+
+

+
+

+
+

+
+

+
+

+
+

+
+

a
Signs derived by multiplying the appropriate columns of Table 4. Pairs, obtained from all combinations of two parameters, may be placed in any sequence, and order of the 2 elements
within any pair does not matter.

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

Table 8
Signs used to calculate effects of parameter pairsa

179

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

180

effect of including double-clutching in the husbandry for the captives. Thus, eliminating doubleclutching would reduce the size of the population
by 30 compared to what would otherwise be
predicted. Because DC is a discrete parameter,
there is no amount of change to calculate, as
can be done for the survival parameters.

3.7. Calculate the effects of two-way interactions

of the parameters
Two-way interactions are the consequence of
parameter pairs acting together. There are C(6,2)
or 15 such pairs for the Example Model. Find the
appropriate signs for the pair effects and then
associate the signs with the output similarly to
above. The appropriate sign for the pair is found
by multiplying the signs for the individual
Table 9
Interaction scores for all parameter pairs in Example Model
S3S0
7.6

S3S1
16.6

S3S2
35.6

S3DC
17.7

S3RF
18.4

S2S0
9.2

S2S1
18.4

S2DC
15.7

S2RF
16.6

S1S0
15.7

S1DC
9.2

S1RF
35.6

S0DC
18.4

S0RF
17.7

DCRF
7.6

Table 10
Interactions by decreasing impacta
Pair

Interaction effect

S1RF
S3S2
S0DC
S2S1
S3RF
S0RF
S3DC
S2RF
S3S1
S1S0
S2DC
S1DC
S2S0
DCRF
S3S0

35.6
35.6
18.4
18.4
18.4
17.7
17.7
16.6
16.6
15.7
15.7
9.2
9.2
7.6
7.6

Identical values are the result of aliasing.

parameters: if the signs have the same parity, plus,

if opposite, minus. Thus, for the pair S3 S0 the
(Table 4) first row product is plus times minus
which gives minus. The 5th row product is minus
times minus which gives plus, as does the 7th row
plus times plus. The second column in Table 8
contains the string of signs which arise from
pairwise multiplication of the sign columns for S3
and S0: [ + + + + + +
+ + ]. Find the remaining signs in the same way.
In the same manner as for the main effects,
associate the column of signs with the output
population numbers row by row. The PB scores
for the interaction effects are then calculated as
before. Sum each of the 15 columns and divide
each sum by eight, the PB design size. Thus, for
S3 S0, add [ 53.4 128.9 126.8+ 62.2+
17.8 25.4 + 39.0 +235.6 33.1 12.1 11.1
+25.2+86.961.5+40.8+5.9] to arrive at
61.0. The division by eight results in 7.6 (Table 9).
As before, actual output numbers and computations contained three decimal places; only one is
reported here. Note that the same outputs, in this
case means over 1000 replicates from 16 distinct
scenarios, are used for the computations in both
the main effects and the 2-way interaction effects.

3.8. Sort the interaction effects decide which

are important
When all 15 interaction effects have been calculated (Table 9), they may be sorted (Table 10).
The situation at this point is not as straightforward as was the interpretation of the main effects.
There are some pairs and even a trio of interactions having the same PB score, the result of
aliasing. The S1 RF pair together with the S3
S2 pair actually produce the effect with magnitude
35.6. However, since the S3 and S2 main effects
were large but the S1 and RF main effects were
modest, the sparsity of effects assumption mentioned earlier may be invoked. The simplifying
assumption is that the part played by the S1 RF
pair may be disregarded and the response attributed to S3S2. Thus, the pair S3 and S2
acting in concert produce an effect in addition to
the individual effects produced by S3 and S2.
Hence, when both S3 and S2 are augmented by

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

Table 11
Selected interactions in decreasing ordera
Pair

Interaction effect

S3S2
S2S1 or S3RFb
S3DC
S3S1 or S2RFb
S2DC
S2S0
S3S0

35.6
18.4
17.7
16.6
15.7
9.2
7.6

a
One pair from each alias group was chosen, by importance
of individual effects. Only pairs involving S3, or S2 were
included.
b
If it were deemed necessary, the question over which of the
pairs is chiefly responsible for the effect could be resolved.

half the distance between their plus and minus PBSA values, the population should be increased by 84.2, the sum of 36.4, 30.0 and 17.8, or
half the sum of 72.8, 60.0, and 35.6. Observe that
the interaction effect S3S2 has similar magnitude to the individual effect of DC (35.631.8).
If the interactions are pared down to those
concerning only the two parameters whose main
effects were the largest, a simpler table results
(Table 11). A reduction of this sort (based upon
the sparsity of effects principle) may be useful for
situations involving a large number of parameters.
Further consideration of the relative sizes of the
remaining pairs (Table 11) leads to the observation that only the S3S2 seems large enough to
matter much. However, if it were considered desirable to resolve the question of attributing the
major portion of the effect from a particular alias
group, this could be accomplished with some additional runs. Discussion of this refinement is
beyond the scope of the present paper.

3.9. Interpret the results

As stated above, it is the responsibility of the
modeler to make sense of the outcomes of the
analysis within the context of the model. As each
effect is examined, the assumption is that the
effect is linear over the interval being considered.
In carrying out an actual PBSA, it might be
more productive to integrate the two ranked listings of main effects and interaction effects prior

181

to deciding which are the important ones. They

were segregated here to facilitate explanations,
but in practice they should be considered
together.

4. Discussion
In the case of the example, recommendations
for the management are possible, as well as advice
regarding data collection. If the most significant
of the parameters does not have values based on
good data, then the suggestion would be to obtain
more/better data, if possible, or to be cautious in
the decision-making if actions must be taken without more data. If there is confidence in the quality
of the data, then the counsel would be to do all
possible to maximize survival for the two upper
age classes. Furthermore, double-clutching in the
captive population appears to be a useful strategy
because the effect of DC, though smaller than S3
and S2, is still appreciable. If management questions involve making choices, the PBSA may assist in ranking them. For example, it would be
more advisable to choose actions which ensure a
high value for S2 than to expend resources on
improving S1. Debate about small changes in RF
would not be productive; RF makes little difference in population. Because they are derived from
the analysis of the Example Model, all of these
conclusions are hypothetical. They are merely intended to illustrate the scope of inference.
While PB with foldover provides a clear estimate of the main effects and has a very good
chance of finding the important interactions (Box
et al., 1978), it does not guarantee uncovering the
latter. The tradeoff between efficiency and complete information must be acknowledged. Also,
apparent interaction effects may be confounded
(aliased) and thus, care must be used in interpretation of the output from the design (Montgomery, 1997).

5. Comparison
To compare with the PBSA, a conventional
OAAT sensitivity analysis was performed on the

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

182

Example Model, using the same six parameters.

For each of these except DC, a decrease of approximately 10% from its default (usual or best
estimate) value was explored. DC is categorical,
so 10% is meaningless in its case. The default and
tested (alternate) values are displayed in Table 12.
The ranking of the parameters from this analysis (Table 13) is the same as the main effects
ranking derived from the PBSA (Table 7). The
positions in the list were determined by finding
the ratios of the amount of change to the default
result. The nature of DC makes it impossible to
calculate (F(x +Dx) F(x))/Dx and, thus, percent changes for all parameters were calculated
instead. The relative importance of the effects for
DC and S1 is different. Most importantly, OAAT
fails to provide any information about
interactions.
Table 12
Parameter values used for OAAT sensitivity analysis of Example Model.
Parameter

Default

Alternate value

S0 (chick survival)
S1 (juvenile survival)
S2 (subadult survival)
S3 (adult survival)
DC (double-clutching)
RF (release fraction)

0.75
0.75
0.85
0.95
Yes
0.8

0.675
0.675
0.765
0.85
No
0.7

Table 13
OAAT sensitivity analysis outputs (mean 9 S.D.) from Example Model, ranked by percent change
Scenarioa

Population

All defaults
S3
DC
S2
S1
RF
S0

72.0
27.1
41.5
41.7
61.3
65.0
67.4

( 9 17.3)
( 9 8.8)
( 912.3)
( 911.9)
( 9 15.4)
( 915.4)
( 916.3)

%
Changeb
na
62.4
42.4
42.1
14.9
9.7
6.4

Scenario indicates which parameter was at its lower value.

Because there is no delta for DC, percent changes were
calculated for each parameter: the change in population was
divided by the default population.
b

6. Conclusion
PBSA makes possible the consideration of a
multitude of parameters simultaneously. It requires a modest number of scenarios (approximately twice the number of parameters) in
comparison to those that would be needed for a
complete factorial design. There is no need to
restrict the analysis to the presumed most powerful parameters. All parameters may be easily investigated. Most importantly, PBSA enables the
modeler to uncover 2-way interactions which
might not otherwise be suspected to be influential.
In setting up PBSA, the parameters do not need
to be tested over identical intervals, but rather the
upper and lower values chosen for the analysis
should be selected as the reasonable extremes for
the particular parameter. Categorical parameters,
such as yesno decisions, may be included in the
analysis.
The main effects are clearly delineated in the
analysis, and are separate from the 2-way
interactions, as well. The interaction effects
may be aliased, but further analysis (not
presented here) makes it possible to tease these
apart, if desired. Within an alias group the effects
are caused by a linear combination of the pairs
involved. The sparsity of effects principle
allows the assumption that the greatest contribution is made by the pair whose individual
components were of greatest importance. However, these individual parameters may not be the
most influential of all single factors. Thus, the
PBSA may expose otherwise undetected
effects. PBSA is simple, convenient, and informative. For ecological modelers, PBSA is an essential tool.

Acknowledgements
During preparation of the manuscript,
valuable suggestions were made by Anthony M.
Starfield, Norbert J. Kuenzi, Robert G. Haight
and Karl A. Beres. The authors also thank two
anonymous reviewers for their useful recommendations.

D.L. Beres, D.M. Hawkins / Ecological Modelling 141 (2001) 171183

References
Baumert, L., Golomb, S.W., Hall, M., 1962. Discovery of an
Hadamard matrix of order 92. American Mathematical
Society Bulletin 68, 237 238.
Beres, D.L., 2000. A methodological study of modeling for
California condors. PhD dissertation, University of Minnesota.
Beres, D.L., C. Clark, G. Swartzman and A.M. Starfield,
2001. Truth in modeling. Natural Resource Modeling 14
(3) in press.
Box, G.E.P., Draper, N.R., 1987. Empirical Model-building
and Response Surfaces. Wiley, New York, p. 669.
Box, G.E.P., Hunter, W.G., Hunter, J.S., 1978. Statistics
for Experimenters: an Introduction to Design, Data
Analysis, and Model Building. Wiley, New York,
p. 653.
Burgman, M.A., Ferson, S., Akcakaya, H.R., 1993. Risk
Assessment in Conservation Biology. Chapman & Hall,
London, p. 314.

183

Daniel, C., 1973. One-at-a-time plans. Journal of the American Statistical Association 68, 353 360.
Daniel, C., 1976. Applications of Statistics to Industrial Experimentation. Wiley, New York, p. 294.
Henderson-Sellers, B., Henderson-Sellers, A., 1996. Sensitivity
evaluation of environmental models using fractional factorial experimentation. Ecological Modelling 86, 291 295.
Jorgensen, S.E., 1994. Fundamentals of Ecological Modelling,
Second ed. Elsevier, Amsterdam, p. 628.
McCarthy, M.A., Burgman, M.A., Ferson, S., 1995. Sensitivity analysis for models of population viability. Biological
Conservation 73, 93 100.
Montgomery, D.C., 1997. Design and Analysis of Experiments. Wiley, New York, p. 704.
Plackett, R.L., Burman, J.P., 1946. The design of optimum
multifactorial experiments. Biometrika 33, 305 325.
Starfield, A.M., Bleloch, A.L., 1991. Building Models for
Conservation and Wildlife Management. Burgess International Group Inc., Edina, MN, p. 253.
Swartzman, G.L., Kaluzny, S.P., 1987. Ecological Simulation
Primer. Macmillan, New York, p. 370.