DESIGNING EXPERIMENTS –
An Overview1
Gregory F. Gruska
Maureen S. Heaphy
ABSTRACT
Statistical experiments are commonly used to identify
factors affecting quality and productivity. The analysis of
data collected during these experiments is most frequent-
ly recognized as the analysis of variance (ANOVA).
Unfortunately there is little attention shown, in the litera-
ture or in practice by the engineer, to the designing of
experiments. That is, one needs to plan an experiment
prior to collecting any data. This paper addresses certain
characteristics of an experiment that are prerequisites to
conducting a meaningful experiment. Different types of
designs are discussed as well as the advantages and dis-
advantages of each.
An experiment can be defined, in a general sense, as
“a trial made to confirm or disprove something, or to
demonstrate some known truth . . .” In other words, it is
the conducting of tests to answer specific questions.
However, in line with statistical thinking, the first step
should be the designing or planning of the experiment
within a PDSA cycle2.
A designed experiment has advantages over the classic
change one variable at a time (AB-BA) approach. First of
all, the designed experiment can detect if there are any
interactions among the variables, but the classic approach
cannot. Next, the design approach uses all of the data
simultaneously in analyses of the data whereas the classic
uses only a subset of the data. The reduced sample size
present in the subset reduces the power of the analysis in
the classic method. Finally, the results obtained from a
designed experiment are valid over a wide range of con-
ditions (environment rich) as compared to the results
from the classic approach which are valid only for the
actual test conditions (environment free).
TERMINOLOGY
Certain terms are defined so that their usage in later
discussions will be understood.
INDEPENDENT FACTOR - What we start out with: the
conditions that are being deliberately varied in a con-
trolled manner are called the independent factors or inde-
pendent variables. These factors may be quantitative fac-
tors such as time or temperature, which can be varied
along a continuous scale, or they may be qualitative fac-
tors such as different machines or a switch turned on/off.
8 ASQ STATISTICS DIVISION NEWSLETTER, VOL. 19, NO. 2
DEPENDENT VARIABLE or RESPONSE - What we end
up with: those variables, which are evaluated for changes
caused by the experiment. The output of interest is called
the dependent or response variable. For example the
response could be the diameter, surface finish, RPM,
torque and so on.
FACTOR LEVEL - The factor level is the number of val-
ues of the factor (independent variable) will be set within
the experiment. For quantitative factors, each tested value
becomes a level; e.g., if the experiment is to be conduct-
ed at four different speeds, then the factor speed has four
levels. Switch on or off denotes two levels for the switch
factor.
CELL – That portion of the test environment which is
likely to be more homogeneous; i.e. those samples which
have the same settings for each of the factors.
REPLICATION – Samples, which have the same set-
tings for each of the factors within the test sequence.
REPEATS – The number of times the entire test
sequence is implemented.
BACKGROUND FACTORS – In addition to the factors
that are varied in a controlled fashion, the experimenter
may be aware of certain background variables that might
affect the outcome of the tests. These background vari-
ables must be considered in the planning of the experi-
ment, so that:
• the possible effects due to background variables do
not affect information obtained about the factors of
primary interest; or
• some information about the effects of the back-
ground variables can be obtained.
RANDOMIZATION – There may be variables of which
the experimenter is unaware that have an effect on the
outcome of the experiment. To increase the likelihood
that the effect of these variables is balanced out, an
experiment should always be randomized. This involves
assigning the occurrence of the controlled conditions in a
purely chance fashion. To eliminate bias from the experi-
ment, variables which are not specifically controlled as
factors or "blocked out" by block designs (see below)
should be randomized. Randomization also assures valid
estimates of experimental error and makes possible the
application of statistical tests of significance and the con-
struction of confidence intervals.
Continued on page 9
DESIGNING EXPERIMENTS
Continued from page 8
PREREQUISITES TO CONDUCTING
EXPERIMENT
There are certain characteristics of an experiment that
are prerequisites to conducting a meaningful experiment.
Some of these are:
• The experiment should have well defined
objectives. These should include identifying the
factors and their ranges; choosing experimental
procedure and equipment; and stating the applica-
bility of the results.
• As much as possible, effects of the independent
factors should not be obscured by other variables.
This is accomplished by designing the experiment
such that the effects of uncontrolled variables are
minimized.
• As much as possible the experiment should be free
from bias. This involves the use of randomization
and replications.
• The experiment should provide a measure of preci-
sion (experimental error), unless it is known from
previous experimentation. Replications provide the
measure of precision while randomization assures
the validity of the measure of precision.
• The expected precision of the experiment should be
sufficient to meet the defined objectives. There gen-
erally is a trade-off between the expense of addi-
tional experimentation and the precision of the
results. These trade-offs should be examined prior
to the collection of data. Also, greater precision may
be obtained by use of blocked designs when appro-
priate.
SIGNAL AND NOISE
Consider the variables of interest (factors) as generat-
ing a signal reflected in the response variable. Consider
the effect of all other (background) variables in the test
environment to be noise. To increase the efficiency and
effectiveness of measuring the effect of the independent
variables, either the signal can be increased or the noise
can be controlled.
CONTROL OF NOISE
The noise or nuisance variables are those factors that
might interfere with the measurement of the effect of the
independent variable. There are basically four ways to
control the effects of noise variables. Three methods are
experimental control and the fourth is statistical control.
The three methods of experimental control are:
ASQ STATISTICS DIVISION NEWSLETTER, VOL. 19, NO. 2
• hold constant - although the noise variable might
have an influence on the dependent variable, it
should have the same effect throughout the experi-
ment if we hold it at a fixed level. Consequently the
effect of the independent variables on the depen-
dent variable can be quantified. Note assumptions:
1) we can hold the noise variable constant and 2)
the noise and signal variables are independent.
• assign randomly - again the noise variable might
influence the dependent variable but by assigning it
randomly and by replicating the experiment, the
effects of the independent variable can be mea-
sured. Note assumptions: 1) we can randomize with
respect to the noise variables, and 2) the noise and
signal variables are independent.
• include as independent variables - if the noise
variables are identifiable and can be controlled then
they could be considered to be independent
variables.
Statistical control of a noise variable involves
measuring the level of that noise variable for each experi-
mental test and then using a statistical technique called
Analysis of Covariance. This will enable us to "back out"
the effects of the noise variable and quantify the effect of
the independent variables on the response variable.
TYPES OF DESIGNS
Once the prerequisites to conducting an experiment
are met, the type of experimental design can be chosen.
These designs have certain relationships to the purposes,
needs, and physical limitations of experiments. They also
have certain advantages in economy of experimentation
and yield straightforward and unbiased estimates of
experimental effects and valid estimates of precision.
There are a number of ways by which experiment
designs might be classified:
• By the number of experimental factors to be investi-
gated (e.g., single-factor vs. multifactor designs)
• By the structure of the experiment design (e.g.,
blocked designs vs. randomized designs); or
• By the kind of information the experiment is pri-
marily intended to provide (e.g., estimates of effects
or estimates of variability).
The types of designs to be discussed are Randomized,
Block, Latin Square, Factorial, and Nested.
Continued on page 10
9
DESIGNING EXPERIMENTS
Continued from page 9
COMPLETELY RANDOMIZED DESIGN
A completely randomized design is appropriate when
the effects of only one signal variable are being investi-
gated. The effects of any and all noise variables will be
controlled by either holding constant or randomization.
Suppose a total of N experimental units are available
for the experiment and there are k factor levels to be
investigated. Then the total N units are assigned randomly
to the k levels. The sample size in each cell does not
need to be equal.
This simple one-factor design is called “completely
randomized” to distinguish it from other experiment
designs where the principle of “blocking” or planned
grouping has been made part of the structure.
APPLICABILITY - The plan is simple and may be the
best choice when the experimental material is homo-
geneous and when background conditions can be well
controlled during the experiment. This type of design is
used to estimate and compare the effects of one
independent variable, the signal. In some cases the
precision of that estimate can be quantified; i.e., how
much variation is there in the estimate of the effect of the
independent variable.
ADVANTAGES - There are three main advantages to a
completely randomized design:
• With only one independent variable, the model and
the calculations are simple and straightforward.
• Equal sample sizes are not required for each cell.
• It allows for the maximum degrees of freedom in
estimating the error.
DISADVANTAGES - There are four main disadvantages
to a completely randomized design:
• Appropriate only when experimental material is
homogeneous.
• Restricted to evaluating only one signal factor.
• Appropriate only when the noise conditions can be
controlled.
• Large sample size may be required to obtain suffi-
cient precision.
BLOCK DESIGNS
An important class of experimental designs is charac-
terized by planned grouping. This class is called block
designs. The use of blocking arose in comparative experi-
ments in agricultural research, in recognition of the fact
that plots that were close together in a field were usually
more alike than plots that were far apart. In industrial and
engineering research, the tool of planned grouping can
be used to take advantage of naturally homogeneous
groupings in materials, machines, time, etc., and also to
take account of ‘background variables’ which are not
directly ‘factors’ in the experiment.
10
The block may consist of observations taken at nearly
the same time or place. If a machine can test four items at
one time, then each run may be regarded as a block of
four units, each item being a unit.
A variety of especially advantageous configurations of
block designs have been developed. They are named and
classified by their structure into randomized blocks,
incomplete blocks, Latin squares Youden squares, etc.
Sometimes the factor and the block are of almost equal
interest. In this case a ‘block design’ is almost a ‘two-
factor experiment,’ but the experimenter must be sure
that the two factors do not interact before using a block
design. If interaction between factors exists or is sus-
pected, the design and analysis for a factorial experiment
with two factors must be used. In other words a block
design (with one-way blocking) can be considered as a
one-factor design or a two-factor-no interaction design.
The simplest design with one-way block is the
‘Randomized Block Design.’ This and others will be
discussed.
RANDOMIZED BLOCK DESIGN
In comparing a number of factor levels, it is clearly
desirable that all other conditions be kept as nearly con-
stant as possible. Often the required number of tests is
too large to be carried out under similar conditions. In
such cases, we may be able to divide the experiment into
blocks, or planned homogeneous groups. When each
such group in the experiment contains exactly one obser-
vation on every treatment, the experimental plan is called
a randomized block design.
APPLICABILITY - This design recognizes one signal
and one noise variable. For example, a testing scheme
may take several days to complete. If we expect some
measurable differences between days, then a day would
represent a block. In another situation, several persons
may be conducting the tests or making the observations,
and differences between operators are expected. The
tests or observations made by a given operator can be
considered to represent a block. Also, the size of a block
may be restricted by physical considerations.
This type of design is used to estimate and compare
factor level effects after removing block effects. Although
the primary interest is to estimate factor effects, it is also
possible to estimate the block effects.
ADVANTAGES - There are three main advantages of a
randomized block design:
• The analysis of the data is relatively simple and
straightforward.
• The experimental material can be divided into
blocks or homogeneous groups.
Continued on page 11
ASQ STATISTICS DIVISION NEWSLETTER, VOL. 19, NO. 2
DESIGNING EXPERIMENTS
Continued from page 10

• The effects of individual differences are minimized.

DISADVANTAGES - There are three main disadvan-
CAR I II III IV
tages of a randomized block design:
• It may be difficult to minimize the ‘within’ block FR A A D A
variability for a large number of factor levels. Position FL A A C D
• For a Fixed Effect Model (i.e., models where factors RR D B B B
levels tested represent all levels of interest), bias is RL D C B C
present in the test for factor level effects.
• Each level of the factor must be run in each block. Brands Randomly Assigned to a Car
This drawback can be overcome by using an incom-
plete randomized block design.
With this random assignment, tire brand A happens to
LATIN SQUARE DESIGN be assigned to the front positions only and is not
A Latin square design is appropriate when there are assigned to vehicle 111. Again from the test results, the
two variables that the experimenter wishes to block out. effect of tire brand will not be isolated since there is dri-
These two variables are sources of noise and could affect ver/vehicle influence and tire position. Even if each tire
the test results. These two noise variables also thought of brand is restricted to being assigned only once on each
as non-homogeneous conditions, are secondary factors in car, test results still could be influenced by tire position.
the test whereas a third variable is the primary factor or The solution for this problem is the Latin square. The
signal of interest. The use of the Latin square is to associ- primary interest is the four tire brands but the two noise
ate the primary factor with the two noise variables in a variables, driver/vehicle and tire position, need to be
prescribed fashion. blocked out. An acceptable assignment is shown below.
APPLICABILITY - There are a number of situations
where two noise variables need to be blocked out. The
noise variables might be machines, positions, operators,
days. A classic example is one where testing is to be CAR I II III IV
done to measure tread loss on four brands of tires, using FR C D A B
four similar vehicles. Let the four tire brands be identified Position FL B C D A
as A,B,C,D and the four vehicles as 1, 11, 111, IV. There RR A B C D
are four tires of each brand to be tested. In setting up the RL D A B C
test one possible assignment of tire brand and vehicle
could be to have one brand assigned to one car as shown
Each Brand Assigned to Each Vehicle and to Each
below.
Position

CAR I II III IV
ADVANTAGES - The primary advantage of a Latin
FR A B C D square design is the ability to block out the effects of two
Position FL A B C D noise variables thereby isolating the influence of the
RR A B C D primary factor. With this type of design it is possible to
RL A B C D estimate and compare the effects of the two blocked
variables. Also, the analysis is considered to be relatively
One Brand Assigned to a Car simple.
DISADVANTAGES - One disadvantage is that the
design assumes no interactions exist among the three
With this assignment, the test results cannot be variables. A second item, that initially seems to be a
attributed to tire brand differences only since there might restriction, is there must be the same number of levels for
be driver/vehicle influence. To eliminate the driver/ all three variables. If this were not the case, then an
vehicle influence, one might decide to randomly assign incomplete Latin square, called a Youden square, would
the sixteen tires (4 tires of each of 4 brands) to the be appropriate.
four vehicles. The results-of such and assignment could
be as shown as follows. Continued on page 12

ASQ STATISTICS DIVISION NEWSLETTER, VOL. 19, NO. 2 11

DESIGNING EXPERIMENTS
Continued from page 11
FACTORIAL DESIGN
In a factorial experiment several independent factors
are controlled and their effects are investigated at each of
two or more levels. The experimental design consists of
taking an observation at each one of all possible combi-
nations that can be formed for the different levels of the
factors.
APPLICABILITY - This type of design recognizes two
or more signal variables. When two or more variables
being studied have an interaction effect, the factorial
design may be appropriate to use. With this type of
design, the results are used to estimate and compare the
effects of several factors and estimate possible interaction
effects. If there were two factors, A and B. then the
interaction, if it existed, would be between A and B
represented as AB. If there were three factors A, B,
and C, then the possible interactions are AB, BC, AC and
ABC.
ADVANTAGES - The main benefit of using this type of
design is that it facilitates the evaluation of interactive
effects. A second consideration is that all of the results are
used to evaluate the effects of the factors. Also, the results
are applicable over a wide range of experimental condi-
tions.
DISADVANTAGES - The required sample size may be
large, but this problem can be resolved by running a sub-
set of the full factorial. This is referred to as a fractional
factorial design. Another disadvantage is the explanation
and interpretation of some interactions may be complex.
This type of design is generally less efficient in determin-
ing optimum levels of factors than sequential experi-
ments.
NESTED DESIGN
With some experiments the independent factors may
be hierarchical sources of variation and the main purpose
of the experiment is to obtain the relative variability of
the sources. Designs intended to provide the relative vari-
ation in various strata are called nested designs.
APPLICABILITY - This type of design is appropriate
when distinct strata or nestings are present. For example,
a test run on a two spindle machine results in two strata,
spindles 1 and 2. This type of situation is present in many
industrial experiments but is often not recognized.
ADVANTAGE- Relative variability is correctly identified
instead of main effect variation. A nested design is gener-
ally the only appropriate design for stratified data.
DISADVANTAGE - Nesting is a specialized design
appropriate only when some hierarchical structure is
present.
12
SUMMARY
A statistical experiment should be part of a PDSA
cycle. It should be planned (designed) before any data is
collected. The plan should consider what data are
required to meet the stated objective. The type of design
chosen should provide the maximum efficiency in
addressing the hypothesis. Then an adequate amount of
data needs to be collected to provide adequate precision.
All too often the data is collected first without designing
the experiment, resulting in misleading and incorrect
results.
REFERENCES
1. Afifi, A.A. and Azen, S.P., Statistical Analysis a Computer
Oriented Approach, Academic Press Inc., 1972.
2. Dixon, W.J. and Massey, F.J. Introduction to Statistical
Analysis, McGraw-Hill, 1969.
3. Draper, N.R. and Smith, H., Applied Regression Analysis,
Wiley, 1966.
4. Hicks, C. R., Fundamental Concepts in the Design of
Experiments, Holt, Rinehart and Winston, 1973.
5. Johnson, N. L. and Leone, F.C., Statistics and Experimental
Design, Vol. ll, Wiley, 1964.
6. Juran, J.M., Quality Control Handbook, McGraw-Hill, 1962.
7. Lipson, C. and Sheth, N.J., Statistical Design and Analysis
of Engineering Experiments, McGraw-Hill, 1973.
8. Mendenhall, W., Introduction to Linear Models and the
Design and Analysis of Experiments, Wadsworth, 1968.
9. Miller, 1. and Freund, J.E., Probability and Statistics for
Engineers, Prentice Hall, 1977.
10. Natrella, M.G., Experimental Statistics, National Bureau of
Standards Handbook 91, U.S. Government Printing Office, 1966.
11. Romano, A., Applied Statistics for Science and Industry,
Allyn and Bacon, 1977.
12. Scheffe, H., The Analysis of Variance, Wiley, 1959.
1This paper was adapted from a paper presented at the 1982
ASQC Conference in Toronto, Canada.
2PDSA - Plan-Do-Study-Act Cycle also known as the Deming
Cycle or the Shewhart Cycle.
ASQ STATISTICS DIVISION NEWSLETTER, VOL. 19, NO. 2
 DESIGNING EXPERIMENTS – SUPPLEMENTAL
Since the initial development of this paper (1982), the
Quality field has begun to evolve from a tolerance
philosophy to a target philosophy (i.e. continual improve-
ment). One impact of this is the move to look at a
designed experiment as a sequence of smaller experi-
ments rather than a large ‘grand’ experiment (i.e. the
experiment to end all experiments). The general rule is
that the first step should use no more than twenty-five
percent of the total number of samples allocated.
In the past it was assumed that all sources of variation
were known and could be handled in the manner dis-
cussed. Special causes were only considered if they were
frequent enough to become part of the abnormal
process”.
The primary purpose of most industrial experiments
was and is to determine the components of variation so
that we can predict the optimal settings in the future. This
tacitly assumes that the process is in-statistical-control, i.e.
no special causes. If this simple assumption is not true,
analysis of the experimental data can lead to wrong con-
clusions and decisions. To be valid, a designed experi-
ment must be conducted on a process that is known to
be in-statistical-control. (Note: the study could be planned
to include a determination if this assumption is true.)
Study Types
The basic study types are given in the main paper.
Every designed experiment is a combination or adapta-
tion of these basic types. Some of these combinations
have become known by unique names. Among the more
popular are:
Fractional Factorial - a factorial design where the
number of samples is reduced by ‘confounding’ or
mathematically mixing a higher order interaction results
with a lower order result. That is, if a two level inter-
action was intertwined mathematically with a seven level
interaction and the analysis showed that this setup was
significant, we would not know if the variation was
caused by the two level or the seven level interaction.
Since higher order interactions are generally not major
contributors to the total variation, this technique can save
time and expense.
Orthogonal Main Effect Designs - the highest fraction-
ated factorial design possible since it confounds higher
order effects with the main effects. It gets its name
because it uses orthogonal arrays in the development of
the experimental design in order to make the analysis
calculations easier. The U.S. Government has published
several manuals listing various standard designs of this
type. Its primary use is to filter out non-signal variables in
situations where there are many potential signal variables
and little product and process knowledge. As a first
(filtering) experiment, this tool is very useful.
ASQ STATISTICS DIVISION NEWSLETTER, VOL. 19, NO. 2
Taguchi Designs - basically Orthogonal Main Effects
Designs as promoted by Dr. Genichi Taguchi. Dr. Taguchi
has advanced the use of this tool by using graphs to
assist the analyst in selecting the proper design and
integrating it in “off -line quality control” activities. Dr.
Taguchi has also developed specific signal-to-noise (SN)
ratios for use with these designs. This analysis is valid
only when the underlying variation model is known; i.e.
the purpose of the study is to determine the parameters
of the model for optimal response not to determine the
model itself.
Response Surface Methodology (RSM) - a sequential
experimental design which has the goal of developing a
mathematical model (i.e. response surface) of the process
in order to optimize and control it more efficiently. RMS
assumes a constant experimental error throughout the
experimental region, and, that only effects of order less
than three are important. Because of it sequential nature,
it is a highly effective tool.
Evolutionary Operations (EVOP) - an application of
RSM to an operating process. This design starts with a
process that is in statistical control and changes the
process parameters in such a manner that normal
production is not adversely affected and that the optimal
setting will be determined after a minimal number of
tests.
REFERENCES
Box G.E.P. and Draper N.R., Empirical Model-Building and
Response Surfaces, John Wiley & Sons, New York, 1987.
Box G.E.P., Hunter W.G., and Hunter J. S., Statistics for
Experimenters, John Wiley & Sons, New York, 1978.
Moen R.D., Nolan T. W., and Provost L. P. Improving Quality
Through Planned Experimentation, McGraw-Hill, New York,
1991.
Taguchi G., System of Experimental Design, Unipub, White
Plains NY, 1987.
13