IT’S ALL IN THE TECHNIQUE

You can’t understand your data unless you control

To err is human, to purr feline.
Robert Byrne
extraneous variance attributable to the way you
select samples, the way you measure variable
values, and any influences of the environment in
which you are working. Using the concepts of
reference, replication and randomization, you can
control, minimize, or at least be able to assess the
effects of extraneous variability in at least five
ways:

Procedural Controls — used primarily to prevent measurement variability by

reference.
Quality Samples and Measurements — used primarily to identify sources of
measurement and environmental variability by reference.
Sampling Controls — used primarily to correct sampling variability by reference and
randomization.
Experimental Controls — used primarily to correct measurement and environmental
variability by reference and randomization.
Statistical Controls — used primarily to correct environmental variability by
reference.

Procedural Controls
Procedural controls include the field, office, and laboratory procedures that specify how
activities related to data generation should be carried out. Forms of procedural control include
Standard Operating Procedures (SOPs), chain-of-custody (CoC) procedures, survey scripts and
instructions, checklists, calibration procedures, training courses and so on. Procedural controls
are the main way to minimize extraneous variation before it occurs.

Quality Samples and Measurements

When a physical sample is collected, especially for laboratory analysis, there are a variety of
samples that can be collected to verify that the analyses produce valid results. The aim of quality
tests is to identify quality problems and sources of undesirable variability. Some quality samples
and tests allow corrections to data points to correct bias. Quality samples used to assess
variability in the collection and measurement of the experimental samples include:
Replicate samples — multiple samples collected of the same subject to assess
variability. Replicate samples may be collected by splitting a large sample into two or
more subsamples (called split samples). Alternatively, discrete samples may be collected
sequentially (called co-located samples). Most replicates are co-located samples because
they are less time-consuming to collect. Co-located samples of mixable media (e.g.,
fluids like water and air) will usually be very similar compared to co-located samples of
 non-mixable media (e.g., solids like soil). Duplicate is a generic term for a replicate
sample that may be either a split sample or a co-located sample.
Blind samples — split samples submitted with unique sample identifiers so they appear
to be from different sources. Blind samples are an independent check of laboratory
methods.
Trip blanks — distilled water placed in sample containers and sealed in the lab. Trip
blanks are carried by the field sampling team and returned to the lab with the test
samples. If any aspect of the sample handling and transport process might have
compromised the quality of the samples within their containers, the effect should be
detectable in the trip blank. It is very rare for trip blanks to contain contaminants.
Rinse blanks — distilled water that has been poured over reusable sampling equipment
that has been decontaminated. Rinse blanks are sometimes called field blanks, rinsate
blanks, equipment blanks, or decontamination blanks because they are a test of the
cleanliness of the sampling equipment.
Atmosphere blanks — distilled water placed in sample containers that are opened
during sample collection. Atmosphere blanks are a test of whether any air pollutants or
windblown particulates might have contaminated test samples during the process of
collecting the samples.
Water blanks — potable water used for site activities, particularly rock drilling. Water
blanks are also collected from wells when domestic plumbing is being assessed for lead
contamination.
Preservative blanks — preserved samples of distilled water. Preservative blanks are
used to assess possible contamination of the acids, bases, and other reagents used to
preserve test samples.
Analytical results from these samples are usually checked during data validation and exploratory
data analysis. Except for replicates and blind samples (which are usually averaged), they are not
included in datasets to be used for statistical analysis. Other QA/QC samples used to check
laboratory procedures include:
Laboratory blanks — distilled water or purified solid prepared in the lab to assess
contamination from reagents, glassware, and analytical hardware. These samples are
sometimes called method blanks because they are a test of the laboratory method.
Performance evaluation samples — samples containing known concentrations of
specific compounds. PE samples are usually used to certify laboratories for a particular
type of analysis rather than check quality on a specific data collection task.
Calibration samples — samples containing known concentrations of an analyte similar
in chemical behavior to the analytes of interest. These samples are used to calibrate
instruments and assess method bias.
Matrix spikes — samples of the media being analyzed that have been spiked with known
concentrations of a representative analyte. These samples are used to check for
interferences between the analytes being tested and the sample matrix. Matrix spike
duplicates are routinely analyzed by laboratories to assess measurement variability.
Analytical results from these samples are usually checked during data scrubbing. They are not
included in datasets to be used for statistical analysis. Most of these QA/QC samples are used
only for chemistry laboratories. Quality samples for other types of laboratory analyses (e.g.,
geotechnical, radiological, biological) are usually limited to replicates.
These are commonly used quality samples but there are many others possible. You create the
samples to fit the experimental situation. Don’t feel limited to laboratory samples. You can use
the same approach to create tests or other methods of variance assessment. If you understand the
possible sources of variation in your data, you can create relevant quality tests for survey
questions, industrial processes, or whatever your study will involve.

Sampling Control
Some types of samples have inherent properties that may introduce extraneous variability into
data being generated. For instance, differences in sex, age, and social class, may introduce
extraneous variability in sociological surveys. Environmentally related examples might include:
soil type, geologic strata, species, location and depth, and season (or other time unit). Applying
sampling controls involves grouping the data by the control factor and calculating statistical
analyses for each homogeneous group. Stratified sampling designs are one form of sampling
control.
Sampling controls can be used to prevent, identify, or correct all kinds of extraneous variation.
As a consequence, they are used to some extent in most statistical studies.

Experimental Control
Statistical studies can be categorized into two types — observational and experimental. In an
observational study, the phenomenon under investigation is a characteristic of the objects being
sampled. The concentrations of arsenic in the soils of a waste management facility would be an
example. Variability and bias in observational studies can be assessed through the use of control
samples. Control samples (like quality samples) are groups of samples that don’t have the
condition being tested but are otherwise identical to the experimental group. Adding an offsite
(or background) area to the study of the waste management facility would be an example of a
control group.
In an experimental study, the phenomenon under investigation is assigned to the objects being
sampled. Testing a pollution cleanup technique on several plots of contaminated soil would be an
example of an experimental study. In such a study, the cleanup techniques would be randomly
assigned to the plots in a manner that would help control some of the variation in the plots.
So, the difference between experimental and observational studies is that in an:
 Experimental study, samples are randomly assigned to controlled conditions.
 Observational study, samples are randomly selected from preexisting conditions.
Another way to control variability and bias is through the use of placebos. A placebo is usually
thought of as a faux drug because of its association with statistical tests of pharmaceuticals, but it
can be any item or action that gives the appearance of being a valid treatment. For example, give
two patients blue pills, one of which has some active ingredient and the other does not. To test a
new pesticide, spray two agricultural test plots, one of which contains the pesticide and the other
contains just water. The key is that the subject or data generator can’t tell the difference. In
Season 5, episode 14 of the television series House, the nurse can tell which patients are being
given the placebo during Dr. Foreman’s drug trial because the real pharmaceutical has a strong
odor. This would not be a good example of an effective placebo.
Placebos are a type of blinding. In any experiment, there are subjects or samples, experimenters,
data collectors or generators, and data analysts. Sometimes, one individual will fill several roles,
such as the experimenter who designs the experiment
and then generates and analyzes the data. But whenever
there are humans involved, there exists the possibility of
intentional or unintentional bias. Blinding is simply the
act of denying one or more of the study participants
with information that might induce them to behave
differently.
To test the effects of a pharmaceutical, for example, a
single-blind study might involve not telling subjects
I’m not telling who got the whether they are receiving the active drug or a placebo.
placebo. (Cat stays in bag.) A double-blind study might involve not telling either
the subjects or the data generators (the nurses who collect physical measurements on the subjects
or the lab technicians who analyze blood samples) who received the drug and who received the
placebo. A triple-blind study might involve not telling the subjects, the data generators, or the
data analysts who received the drug and who received the placebo. It’s ironic that the less
informed the participants are, the better the statistics. Too bad government doesn’t work that
way.

Statistical Control
Special statistics and statistical procedures can be used to partition variability shared by
measurements so that extraneous variability can be assessed. For example, partial correlations
quantify the relationship between two variables while holding the effects of other variables
constant. A covariate is a continuous variable that is incorporated into an analysis of variance
design to eliminate extraneous variability so that tests of the grouping factors will be more
sensitive. Statistical controls are typically used when variation cannot be controlled adequately
through other means.
 Techniques for Controlling Extraneous Variability.
Variance Control Technique
Quality Samples and
Measurements

Experimental
Procedural

Replicates

Statistical
Sampling
Blanks
Tests
Prevent       
    
Use

Identify

Correct      
Reference     
Variance

Strategy
Control

Replication      
Randomization      
Sampling       
Controlled
Variance

Measurement     

Environmental      
 Primary Importance  Secondary Importance  Not Very Important

Join the Stats with Cats group on Facebook.

http://statswithcats.wordpress.com/2010/09/19/it%E2%80%99s-all-in-the-technique/