31A_Coverpg_Cumitech_557049 6/24/10 9:18 AM Page 1

31A
Verification and
Validation of
Procedures in the
Clinical Microbiology
Laboratory
RICHARD B. CLARK, MICHAEL A. LEWINSKI,
MICHAEL J. LOEFFELHOLZ, AND
ROBERT J. TIBBETTS

COORDINATING EDITOR
SUSAN E. SHARP

Cumitech
CUMULATIVE TECHNIQUES AND PROCEDURES IN CLINICAL MICROBIOLOGY
31A_Coverpg_Cumitech_557049 6/24/10 9:18 AM Page 2

Cumitech 1C Blood Cultures IV (2005) Cumitech 31A Verification and Validation of Procedures
Cumitech 2C Laboratory Diagnosis of Urinary Tract in the Clinical Microbiology Laboratory
Infections (2009) (2009)
Cumitech 3B Quality Systems in the Clinical Cumitech 32 Laboratory Diagnosis of Zoonotic
Microbiology Laboratory (2005) Infections: Viral, Rickettsial, and Parasitic
Agents Obtained from Food Animals and
Cumitech 7B Lower Respiratory Tract Infections (2004)
Wildlife (1999)
Cumitech 10A Laboratory Diagnosis of Upper Respiratory
Tract Infections (2006) Cumitech 33 Laboratory Safety, Management, and
Diagnosis of Biological Agents Associated
Cumitech 12A Laboratory Diagnosis of Bacterial Diarrhea with Bioterrorism (2000)
(1992)
Cumitech 34 Laboratory Diagnosis of Mycoplasmal
Cumitech 13A Laboratory Diagnosis of Ocular Infections
Infections (2001)
(1994)
Cumitech 16A Laboratory Diagnosis of the Cumitech 35 Postmortem Microbiology (2001)
Mycobacterioses (1994) Cumitech 36 Biosafety Considerations for Large-Scale
Cumitech 18A Laboratory Diagnosis of Hepatitis Viruses Production of Microorganisms (2002)
(1998) Cumitech 37 Laboratory Diagnosis of Bacterial and
Cumitech 21 Laboratory Diagnosis of Viral Respiratory Fungal Infections Common to Humans,
Disease (1986) Livestock, and Wildlife (2003)
Cumitech 23 Infections of the Skin and Subcutaneous Cumitech 38 Human Cytomegalovirus (2003)
Tissues (1988)
Cumitech 39 Competency Assessment in the Clinical
Cumitech 24 Rapid Detection of Viruses by Microbiology Laboratory (2003)
Immunofluorescence (1988)
Cumitech 40 Packing and Shipping of Diagnostic
Cumitech 26 Laboratory Diagnosis of Viral Infections Specimens and Infectious Substances (2004)
Producing Enteritis (1989)
Cumitech 41 Detection and Prevention of Clinical
Cumitech 27 Laboratory Diagnosis of Zoonotic
Microbiology Laboratory-Associated Errors
Infections: Bacterial Infections
(2004)
Obtained from Companion and Laboratory
Animals (1996) Cumitech 42 Infections in Hemopoietic Stem Cell
Cumitech 28 Laboratory Diagnosis of Zoonotic Transplant Recipients (2005)
Infections: Chlamydial, Fungal, Viral, and Cumitech 43 Cystic Fibrosis Microbiology (2006)
Parasitic Infections Obtained
from Companion and Laboratory Animals Cumitech 44 Nucleic Acid Amplification Tests for
(1996) Detection of Chlamydia trachomatis and
Neisseria gonorrhoeae (2006)
Cumitech 29 Laboratory Safety in Clinical Microbiology
(1996) Cumitech 45 Infections in Solid-Organ Transplant
Recipients (2008)
Cumitech 30A Selection and Use of Laboratory Procedures
for Diagnosis of Parasitic Infections of the Cumitech 46 Laboratory Procedures for Diagnosis of
Gastrointestinal Tract (2003) Blood-Borne Parasitic Diseases (2008)

Cumitechs should be cited as follows, e.g.: Clark, R. B., M. A. Lewinski, M. J. Loeffelholz, and R. J. Tibbetts, 2009. Cumitech 31A, Verification and Vali-
dation of Procedures in the Clinical Microbiology Laboratory. Coordinating ed., S. E. Sharp. ASM Press, Washington, DC.
Editorial Board for ASM Cumitechs: Alice S. Weissfeld, Chair; Maria D. Appleman, Vickie Baselski, Mitchell l. Burken, Roberta Carey, Lynne Garcia, Larry
Gray, Amy L. Leber, Andrea J. Linscott, Yvette S. McCarter, Susan E. Sharp, James W. Snyder, Allan L. Truant, Punam Verma.
Effective as of January 2000, the purpose of the Cumitech series is to provide consensus recommendations regarding the judicious use of clinical micro-
biology and immunology laboratories and their role in patient care. Each Cumitech is written by a team of clinicians, laboratorians, and other interested
stakeholders to provide a broad overview of various aspects of infectious disease testing. These aspects include a discussion of relevant clinical consid-
erations; collection, transport, processing, and interpretive guidelines; the clinical utility of culture-based and non-culture-based methods and emerging
technologies; and issues surrounding coding, medical necessity, frequency limits, and reimbursement. The recommendations in Cumitechs do not repre-
sent the official views or policies of any third-party payer.
Copyright © 2009 ASM Press Address editorial correspondence to ASM Press,
American Society for Microbiology 1752 N St. NW, Washington, DC 20036-2904, USA
1752 N St. NW E-mail: books@asmusa.org
Washington, DC 20036-2904, USA Send orders to ASM Press, P.O. Box 605, Herndon, VA 20172, USA
ISBN 978-1-55581-530-1 Phone: (800) 546-2416 or (703) 661-1593 • Fax: (703) 661-1501
Online: estore.asm.org
All Rights Reserved
10 9 8 7 6 5 4 3 2 1
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 1

Verification and Validation

of Procedures in the
Clinical Microbiology
Laboratory

Richard B. Clark
Quest Diagnostics Nichols Institute, Chantilly, VA 20151
Michael A. Lewinski
University of California, Los Angeles, CA 90095
Michael J. Loeffelholz
University of Texas Medical Branch, Galveston, TX 77555
Robert J. Tibbetts
Henry Ford Health System, Detroit, MI 48202
COORDINATING EDITOR: Susan E. Sharp
Kaiser Permanente—NW, Portland, OR 97230

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Federal Regulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Role of the FDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
Role of CLIA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Selection of a Laboratory Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Verification of Common Microbiology Tests . . . . . . . . . . . . . . . . . . . . . . . . . 10
Verification Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
Verification Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
AST Systems (Unmodified, FDA Cleared) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
AST System Verification Studies Not Arbitrated with a Reference Method
(Test 30 or More Isolates per Panel/Card) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
AST System Verification Studies Arbitrated with a Reference Method
(Test up to 100 or More Isolates per Panel/Card) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Diagnostic Microbiology and Microbial Identification Tests . . . . . . . . . . . . . . . . . . . . . . . . 14
Verification of Unmodified, FDA-Cleared Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Verification of LDTs, Modified FDA-Cleared Tests, and Commercially
Available Kits That Are Not FDA Cleared . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17
Validation of Diagnostic Tests Used in Clinical Microbiology . . . . . . . . . . . . 18
Components of the Validation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Frequency of Test Validation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Summation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Appendix A: Method Selection and Verification Example . . . . . . . . . . . . . . 21
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Appendix B: Process for Selection of a Test Method . . . . . . . . . . . . . . . . . . 23

INTRODUCTION tors, as defined below, that are readily measurable

and have become the hallmarks for analyzing tests as
well as determining laboratory and technologist pro-

T
he most important attributes of a laboratory
test are its ability to produce accurate, precise,
and reproducible results with rapid turnaround
time and clinical utility. There are a number of fac-
ficiency. When a new laboratory test is introduced,
these factors may be weighed with other issues, such
as clinical relevance, cost, instrumentation, and ease

1
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 2
2 Clark et al.
of performance. Laboratories are required to estab-
lish policies and procedures to maintain or improve
the reliability, efficiency, and clinical utility of labo-
ratory tests. Two of the most important analyses per-
formed in the laboratory are verification and valida-
tion. While the definitions of these processes are
often used interchangeably they are in fact two sepa-
rate processes. Verification is the one-time process
performed to determine or to confirm a test’s ex-
pected performance prior to implementation in the
clinical laboratory; simply put, it asks “does the test
work?” Validation is an ongoing process of monitor-
ing a test, procedure, or method to ensure that it con-
tinuously performs as expected; simply put, it asks
“does the test still work?” Validation is achieved
through normal quality control, proficiency testing,
staff competency testing, and instrument calibration.
The purpose of this Cumitech is to provide guid-
ance on the necessary criteria that may be required as
new tests are considered for clinical use and as old
tests are reevaluated for their clinical relevance. Once
verified, additional analyses can be performed to de-
termine the test’s sensitivity and specificity compared
to an existing test or to the gold standard. Addition-
ally, positive and negative predictive values (PPV and
NPV, respectively) can be determined based on the
relative prevalence of a particular disease. The guide-
lines within this Cumitech apply equally to simple
single-reagent tests and the most complex of instru-
ments generating a variety of analytic results and in-
terpretations. However, these are simply guidelines
and should be used in addition to other criteria or
policies which regulatory, accrediting, licensing, or
standard-setting agencies use in assessing either the
compliance of a laboratory or the accuracy of indi-
vidual tests or instruments.
Lastly, the information in this Cumitech is not all-
inclusive. Much information can be found in the lit-
erature regarding the verification and validation of
test methods; however, this information often applies
to quantitative analyses more commonly found in the
chemistry laboratory. When these processes are ap-
plied to microbiology, where qualitative results are
more common, where subjective interpretations are
required, or where the results include the identifica-
tion of microorganisms with biological variation,
more flexibility is required. Thus, the focus of this
Cumitech is on the common qualitative and semi-
quantitative test procedures performed in the clinical
microbiology laboratory.
DEFINITIONS
The primary processes addressed in this Cumitech
are the verification of a new test or method prior to
introduction into the laboratory and the ongoing val-
CUMITECH 31A
idation of the performance of existing test methodol-
ogy. However, before these processes can be dis-
cussed in detail, relevant definitions must be estab-
lished.
Accuracy (22, 23): Technical accuracy is the nearness
of an individual measurement to the true value, as
determined by a reference method, or the agreement
between two tests. Clinical accuracy is the overall
ability of a test to both rule in and rule out an ana-
lyte or a specific disease. Accuracy is synonymous
with test efficiency and can be expressed mathemati-
cally as a percentage:
No. of correct results
 100
Total no. of results
Analyte (22): The component of a specimen or organ-
ism which is to be measured or demonstrated. An ana-
lyte may be a particular antigen, antibody, nucleic
acid, organism, enzyme, species, or metabolic product.
ASR: The Food and Drug Administration (FDA) de-
fines an analyte-specific reagent (ASR) as “antibodies,
both polyclonal and monoclonal, specific receptor
proteins, ligands, nucleic acid sequences, and similar
agents which, through specific binding or chemical
reaction with substances in a specimen, are intended
for use in a diagnostic application for identification
and quantification of an individual chemical sub-
stance or ligand in biological specimens” (http://
www.fda.gov/downloads/MedicalDevices/Device
RegulationandGuidance/GuidanceDocuments/UCM
071269.pdf; accessed 6 August 2009). In simpler
terms, from a clinical standpoint, ASRs are the sepa-
rate building blocks of a laboratory-developed test
(LDT) used to identify a particular analyte. Examples
are the primers and probes used to detect and iden-
tify Bordetella pertussis in a nasopharyngeal swab
specimen. Guidance on the use of ASRs can be found
in the “Guidance for Industry and FDA Staff Com-
mercially Distributed Analyte Specific Reagents
(ASRs): Frequently Asked Questions,” which can be
found at the website above.
Gold standard: A commonly used term generally in-
dicating a test method currently accepted as reason-
ably, but not necessarily 100%, accurate (15, 22). It
is used as the reference method for assessing the per-
formance characteristics of another test method. See
also “Reference method.” A caveat regarding the
gold standard method of analysis: when the true dis-
ease status of a patient is unknown and the disease
state is being determined by using a test compared
with an imperfect gold standard, the results will be
skewed and errors of the gold standard or reference
method will be magnified. As is often the case with
new and improved technologies, the new test may be
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 3
CUMITECH 31A Verification and Validation of Procedures in the Clinical Microbiology Laboratory
more accurate than the accepted gold standard. Dis-
agreement between the two tests will manifest them-
selves as false positives or false negatives. As such, it
has been suggested that the concepts of sensitivity
and specificity should not be used when the true dis-
ease state of the patient is unknown (15). In this sit-
uation, it may be more appropriate to display the
agreement and disagreement between the gold stan-
dard and new test in graphic or tabular form such as
a 2-by-2 table. Areas of disagreement may then be
further investigated by other tests or by monitoring
the patient’s condition to determine if disease devel-
ops. In those instances in which it cannot be deter-
mined whether the new test is better than the gold
standard, a decision to use the new test alone or in
combination with the gold standard may be made by
using a cost-benefit analysis (15). Problems associ-
ated with use of an imperfect gold standard are more
fully discussed in the CLSI publication I/LA 18-A2
(22) and in references 1, 14, and 19.
Home-brew test: See “LDT.”
Incidence (8): The number of new cases over a period
of time. It is most often given as the incidence rate or
the number of new cases of infection in a specified
population at risk during a defined period relative to
the overall number of people in the population at risk.
In vitro diagnostic (IVD) product: Any medical de-
vice which is a reagent, reagent product, calibrator,
control material, kit, instrument, apparatus, equip-
ment, or system, whether used alone or in combina-
tion, intended by the manufacturer to be used in vitro
for the examination of specimens, including blood
and tissue donations, derived from the human body,
solely or principally for the purpose of providing in-
formation concerning a physiological or pathological
state or a congenital abnormality, determining safety
and compatibility with potential recipients, or moni-
toring therapeutic measures.
LDT (13): Procedures developed in-house that use
reagents that are either commercially available or
produced in-house. Since an LDT is not reviewed
and/or approved by the FDA and since the reagents
used therein cannot be reviewed for their intended
use, these tests require more extensive performance
verification (4). LDTs routinely use reagents pro-
vided by companies for either investigative use only
(IUO) or research use only (RUO); however, this does
not deem the LDT an IUO or RUO test. Conversely,
IUO or RUO kits typically carry with them manu-
facturer’s limitations and, therefore, must be used in
accordance with these guidelines described by the
company and/or the FDA and must not be used for
IVD testing. Please refer to the document located
at http://www.devicelink.com/ivdt/archive/08/03/004
3
.html (accessed 8 July 2009) for further information
regarding IUO and RUO test kit use in the clinical
microbiology lab and the role of such tests in clinical
diagnostics.
Medical device: An instrument, apparatus, implement,
machine, contrivance, implant, in vitro reagent, or
other similar or related article, including a compo-
nent part, or accessory which is recognized in the of-
ficial National Formulary, the United States Pharma-
copoeia, or any supplement to them intended for use
in the diagnosis of disease or other conditions or in
the cure, mitigation, treatment, or prevention of dis-
ease in humans or other animals intended to affect
the structure or any function of the body of humans
or other animals, which does not achieve any of its
primary intended purposes through chemical action
within or on the body of humans or other animals
and which is not dependent upon being metabolized
for the achievement of any of its primary intended
purposes.
New test: A new test includes any test not previously
offered by a laboratory, a procedure or methodology
change, or a test performed in-house that was previ-
ously performed at a reference laboratory. Such tests
include detection or identification of a totally new
analyte, the use of totally new methodology, a new
approach to detecting an analyte, a change from a
manual method to an automated one, a new applica-
tion of existing technology, or the test of a new ma-
trix (old analyte in a different specimen).
Old test: An old test is any procedure for detection of
a disease, analyte, or characteristic (e.g., antimicro-
bial susceptibility) that had been in use before 24 April
2003, the effective date of the final Clinical Labora-
tory Improvement Amendments (CLIA). Laborato-
ries are not required to verify or establish perfor-
mance specifications for any test system used by the
laboratory before 24 April 2003.
Precision (22): A measure of the extent to which
replicate analyses (using identical procedures) of a
homogeneous analyte agree with each other. Preci-
sion implies freedom from inconsistency and random
error but does not guarantee accuracy. Precision is
synonymous with reproducibility; however, the use
of the term precision is generally applied to quantita-
tive assays, while reproducibility is used with quali-
tative analyses. Mathematically, precision can be ex-
pressed as a percentage:
No. of repeated results in agreement
 100
Total no. of results
Predictive value (15, 22): The predictive value of a
test is the probability that a positive result (PPV)
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 4
4 Clark et al.
accurately indicates the presence of an analyte or a
specific disease or that a negative result (NPV) accu-
rately indicates the absence of an analyte or a specific
disease. PPV is expressed as a percentage:
No. of true-positive results
 100
No. of true-positive plus
false-positive results
NPV is expressed as a percentage:
No. of true-negative results
 100
No. of true-negative plus
false-negative results
Predictive values can vary significantly with the
prevalence of the disease or analyte unless the test is
100% sensitive (for NPV) or specific (for PPV). The
highest predictive values are desired when inappro-
priate treatment due to false-positive or -negative re-
sults has serious clinical, emotional, epidemiological,
public health, or economic consequences. Predictive
values are most meaningful in evaluating a test’s per-
formance in specific risk population groups.
Prevalence (8): The frequency of a disease in the pop-
ulation of interest at a given point in time (point
prevalence) or during a defined period of time (pe-
riod prevalence).
Quality assurance (10, 17): A system for continuously
improving and monitoring the reliability, efficiency,
and clinical utilization of laboratory tests. Quality
control, quality improvement, and method validation
are integral components of quality assurance.
Quality control (10, 17): The process of ongoing per-
formance checks, including personnel performance,
using known organisms or analytes to ensure on a
regular and frequent basis that a method which has
gone through the verification process (see below) and
is now part of the laboratory’s routine test battery is
performing as expected. Quality control systemati-
cally detects deficiencies in testing by setting limits of
acceptable performance (accuracy and precision). It
thus allows detection and corrective action, where
appropriate, of major problems or errors with test
systems and their performance. Quality control im-
plies that there exist standard analytes that have
known reactions or reactivity. Quality control is an
integral part of the test validation process.
Quality improvement (10, 17): The prevention of test
deficiencies and enhancement of a test’s clinical util-
ity by establishing a thorough understanding of the
test’s capabilities and limitations, as gathered from
CUMITECH 31A
experience and observation, and the subsequent use
of this knowledge to make and verify procedural
changes for improved test performance.
Reference method: A thoroughly investigated method
in which exact and clear descriptions of the necessary
conditions and procedures are given for the accurate
determination of one or more values; the docu-
mented accuracy and precision of the method are
commensurate with the method’s use for assessing
the accuracy of other methods for measuring the same
property values or for assigning reference method val-
ues to reference materials. A currently used method
is unacceptable as a reference method unless there is
on-site or peer-reviewed-journal documentation of
an acceptable level of accuracy and precision of the
method. See also “Gold standard.”
Reproducibility: See “Precision.”
Sensitivity (2, 15, 22, 23): Clinical sensitivity is a
measure of a test’s ability to accurately detect pa-
tients with a specific disease. Analytic sensitivity
measures the smallest quantity of an analyte that can
be reproducibly distinguished from background levels,
or a zero calibrator, in a given assay system; it is usu-
ally defined at the 0.95 confidence level (2 standard
deviations) and is more appropriately called the limit
of detection. In microbiology, for example, the detec-
tion limit can be correlated to the number of colonies
in culture or the lowest quantity of antigen, antibody,
or nucleic acid a test can detect. The highest sensitiv-
ity is desired when a disease is serious and treatable
and when false-positive results will not lead to seri-
ous clinical or economic problems. Mathematically,
sensitivity is expressed as a percentage:
No. of true-positive results
 100
No. of true-positive plus
false-negative results
Specificity (2, 15, 22, 23): Clinical specificity is a
measure of a test’s ability to accurately identify all
noninfected patients. Analytic specificity is the ability
of an analytical method to detect only the analyte
that it was designed to measure. The highest speci-
ficity is desired when the disease is serious but not
treatable, when disease absence has either psycholog-
ical or public health value, or when false-positive re-
sults might cause serious clinical or economic prob-
lems. Mathematically, specificity is expressed as a
percentage:
No. of true-negative results
 100
No. of true-negative plus
false-positive results
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 5
CUMITECH 31A Verification and Validation of Procedures in the Clinical Microbiology Laboratory
User-modified test: Any procedure that incorporates
modifications of the manufacturer’s FDA-cleared or
-approved package insert instructions.
Validation: The documentation that a test which has
already been verified is repeatedly giving the ex-
pected results as the test is performed over a period
of time. Validation confirms that the test continues to
perform satisfactorily according to the laboratory’s
requirements or the manufacturer’s claims or, for
LDTs, according to its intended use. The require-
ments for test validation may include personnel com-
petency assessment, quality control, internal and ex-
ternal proficiency testing, and correlation with
clinical findings. Validation thus becomes an integral
part of the laboratory’s quality assurance program.
Verification (4): The documentation of either com-
mercial or LDT performance. For FDA-cleared or
-approved tests, it is the process of examination or
evaluation of a test system to determine whether the
claims stipulated by the manufacturer in the package
insert as they relate to the product, the process, the
results, or the interpretation can be achieved. Verifi-
cation requires determination or confirmation of the
test performance characteristics, including sensitivity,
specificity, and where appropriate, the predictive val-
ues, precision, and accuracy of the test. Verification
is a one-time process, completed before the test or
system is used for patient testing (see Appendix A).
It is important to note that the definitions for val-
idation and verification as written above are accepted
by CLSI, the CLIA, and the International Organiza-
tion for Standardization. However, the definitions ac-
cepted by The Joint Commission and the College of
American Pathologists (CAP) are the inverse.
FEDERAL REGULATION
Role of the FDA
The FDA and the Centers for Medicare and Medi-
caid Services (CMS) are two principal agencies in the
U.S. Department of Health and Human Services re-
sponsible for regulating laboratory tests. This section
attempts to eliminate some of the confusion that of-
ten exists between the regulatory requirements for
FDA clearance or approval of an IVD product while
introducing the different requirements for laborato-
ries to implement tests already approved or cleared
by the FDA relative to those that have not been
cleared for IVD use.
Laboratory tests are considered medical devices
and, as such, are under the regulatory control of the
FDA. The Medical Device Amendments to the Food,
Drug, and Cosmetic (FD&C) Act in 1976 expanded
5
the role of the FDA to regulate medical devices under
the appropriate control levels necessary to ensure
safety and effectiveness. The Safe Medical Devices
Act of 1990, a major revision of the 1976 amend-
ments, added new provisions to better ensure that de-
vices entering the market were safe and effective. It
also provided means for the FDA to discover serious
problems more quickly and remove defective devices
from the market. More recently, the ASR Rule, en-
acted 23 November 1998 and updated on 15 Septem-
ber 2008 (21 CFR §809), defines the FDA’s approach
to regulate manufacturers of reagents that are used as
components in LDTs. The FDA considers clinical lab-
oratories that develop in-house tests to be manufac-
turers of medical devices and they are subject to FDA
jurisdiction under the FD&C Act. However, the FDA
has generally exercised enforcement discretion over
LDTs and decided to ensure the quality of the reagents
used in LDTs through the ASR Rule. Before a test can
be marketed (commercially distributed) in the United
States, it must first be reviewed and cleared by the
FDA for IVD use. However, laboratories are permit-
ted to verify and validate user-developed procedures
and offer testing services (as opposed to the commer-
cial distribution of their test) providing they meet the
regulatory requirements defined by both CMS in
CLIA (discussed previously) and the FDA. The FD&C
Act, the Safe Medical Devices Act, and the ASR Rule
are administered by the FDA’s Center for Devices and
Radiological Health through the Office of Device
Evaluation/Division of Clinical Laboratory Devices.
Analogous to the regulatory requirements for IVD
clearance in the United States, the In Vitro Diagnos-
tic Directive (98/79/EC) (IVDD) was introduced in the
European Union as one in a series of medical device
directives intended to ensure that only safe and effec-
tive products are sold in the European market. The
IVDD includes strict regulations regarding manufac-
turing, import, and marketing. The “CE” mark on the
manufacturer’s product indicates compliance with
the IVDD, just as the marking of products with “For
In Vitro Diagnostic Use” indicates compliance with
the FDA in the United States. However, a CE-marked
product is not automatically granted FDA clearance,
and premarket review is still required and vice versa.
FDA Clearance or Approval of an IVD
Medical Device Classification
The FDA recognizes three classes of medical de-
vices based on the level of control necessary to ensure
the safety and effectiveness of the device. Unless they
are deemed exempt, the FDA will classify IVD prod-
ucts as class I, class II, or class III devices. This clas-
sification determines the process the manufacturer
must complete to obtain FDA clearance or approval
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 6
6 Clark et al.
to market its product; namely, either premarket noti-
fication [510(k)] or premarket approval (PMA).
Most devices are cleared for commercial distribution
in the United States by the premarket notification
[510(k)] process.
Class I devices present minimal potential harm to
the user, are typically simple in design and manufac-
ture, and have a history of safe use. Examples of class
I devices include tongue depressors, arm slings, and
handheld surgical instruments. Most class I devices
are exempt from the 510(k) requirement by regulation
but are not exempt from other general controls, such
as product registration and device listing. However, a
class I device may require a premarket notification
[510(k)] based on its stated or purported use. Such
devices are referred to as reserved class I devices.
A class II device is any device for which reasonable
assurance of safety and effectiveness can be obtained
by applying both general controls, as for class I devices,
and special controls. Special controls may include spe-
cial labeling requirements, mandatory performance
standards, development of patient registries, and a
requirement for postmarket surveillance. Examples
of class II devices include physiologic monitors, X-ray
systems, gas analyzers, pumps, and surgical drapes.
Most class II devices require premarket notification
by submission and FDA review of a 510(k) for clear-
ance to market the IVD. A few class II devices are ex-
empt from the premarket notification. Information
on class II exempt devices is located within the device
regulation (21 CFR 862–892).
Class III devices are in the most stringent regula-
tory category and are usually devices that “support
or sustain human life, are of substantial importance
in preventing impairment of human health, or which
present a potential, unreasonable risk of illness or in-
jury.” Due to the higher level of risk associated with
class III devices, the FDA has determined that general
and special controls alone are insufficient to ensure
the safety and effectiveness of class III devices. There-
fore, these devices require a PMA application to ob-
tain marketing clearance in the United States. Some
class III medical devices, such as a preamendment de-
vice (a device in use before 1976), require only a
510(k) clearance by the FDA to be marketed. Exam-
ples of class III devices that require a PMA include re-
placement heart valves, silicone gel-filled breast im-
plants, and implanted cerebellar stimulators. For
more information on test complexity and to research
specific products, go to http://www.accessdata.fda
.gov/scripts/cdrh/cfdocs/cfCLIA/search.cfm (accessed
8 July 2009).
Premarket Notification or 510(k) Clearance Process
The 510(k) clearance process is required for re-
served class I and class II medical devices and select
CUMITECH 31A
class III medical devices (e.g., a preamendment de-
vice). The 510(k) submission identifies characteristics
of the new or modified medical device compared to a
medical device with similar intended use currently
legally marketed in the United States. The currently le-
gally marketed device is referred to as the “predi-
cate” device. If the device is a high-risk device and
has been found to be not substantially equivalent to
a class I or II device or class III device requiring
510(k), then the device must have an approved PMA
before being marketed in the United States. Some de-
vices that are found to be not substantially equivalent
to a cleared class I, II, or III (not requiring PMA) de-
vice may be eligible for down classification to a class
I or class II device. If the FDA assessment of the
510(k) submission indicates that it is substantially
equivalent to a legally marketed device, the device is
cleared and the manufacturer is free to market it in
the United States. The FDA has granted exemptions
from the requirement for 510(k) notification for a
variety of generic-type devices, including such micro-
biology products as anaerobic chambers, incubators,
gas-generating devices, and most media.
PMA Process
PMA is the most stringent type of device market-
ing application required by the FDA. Unlike premar-
ket notification [510(k)], PMA approval is based on
the FDA’s determination that the PMA submission
contains sufficient valid scientific evidence that pro-
vides reasonable assurance that the device is safe and
effective for its intended use(s). A PMA is required to
receive clearance to market or continue to market a
class III medical device in the United States. How-
ever, as previously mentioned, there are some class III
devices currently being marketed without a PMA,
primarily due to commercial distribution prior to the
FD&C Act, but these devices must have a cleared
premarket notification 510(k) prior to marketing.
Before approval, a PMA receives an in-depth scien-
tific review, and the manufacturer’s facilities must
undergo a comprehensive inspection compliance with
current good manufacturing practices. Finally, the
PMA is reviewed by an FDA advisory panel of outside
experts who provide recommendations to the FDA
for approval with or without conditions or for disap-
proval of the application. Examples of microbiology
devices requiring PMA applications are those in-
tended for the detection or typing of human papillo-
mavirus; all hepatitis and human immunodeficiency
virus diagnostic, detection, and monitoring devices;
and devices using nucleic acid amplification tech-
niques for direct detection of Mycobacterium tuber-
culosis from clinical material. Refer to the FDA web-
site above for regulatory requirements of class III
devices prior to commercial distribution in the
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 7
CUMITECH 31A Verification and Validation of Procedures in the Clinical Microbiology Laboratory
United States and refer to section 515(c)(1) of the
FD&C Act for the required contents of a PMA.
FDA Clearance and Product Labeling
After all analytical and clinical data have been
critically reviewed, the final step of the FDA review
is to clear or approve the product labeling. That is,
the FDA review process confirms that sufficient evi-
dence was presented which supports the claims of the
manufacturer and does not establish claims or con-
firm test performance. The FDA pays particular at-
tention to the following components of the package
insert, all of which are required by law (21 CFR
809.10): intended use; specimen collection, trans-
port, and storage recommendations; warnings and
limitations; expected values; validation of cutoff; re-
sults and their interpretation; quality control recom-
mendations; and specific performance characteristics.
All microbiology products which undergo a scientific
evaluation of data to substantiate product perfor-
mance claims as stated in the product insert (i.e.,
moderate- and high-risk devices) are expected by the
FDA to maintain that performance throughout the
life of the product. Failure to maintain that expected
performance could result in compliance or regulatory
action. Promotional and advertising materials also
fall under the labeling regulation. Such material can
only be reflective of the information contained in the
package insert for the device.
RUO and IUO Products
Devices which are in the laboratory research phase
of development may not be represented as effective
diagnostic products, and the statement “For Research
Use Only (RUO). Not for use in diagnostic proce-
dures” must be prominently placed in product label-
ing. A product being shipped or delivered for prod-
uct testing prior to full commercial marketing must
prominently bear the statement “For Investigational
Use Only (IUO). The performance characteristics of
this product have not been established.” Only in
vitro devices which have been 510(k) cleared or have
received PMA from the FDA may legally include the
statement “For In Vitro Diagnostic Use” as part of
their product labeling and package insert. In the clin-
ical laboratory, results from the following must not
be reported without verification: (i) tests or proce-
dures that have been developed in-house and use
class I reagents that do not have an indicated use for
that test or (ii) reagents or tests provided by compa-
nies for IUO or RUO that are not used within the
guidelines described by the company or by the FDA.
Reports of such results should indicate that the test is
not FDA cleared or approved but has been developed
and verified in-house. Since LDTs are not reviewed
by the FDA and since the reagents used therein can-
not be reviewed for their use in those tests, these tests
7
require more extensive performance verification. The
laboratory’s responsibility for proficiency testing ex-
tends to LDTs, and this proficiency testing should be
performed at least two times a year.
ASRs
ASRs are the active ingredients of tests that are
used to identify one specific disease or condition. ASRs
include “antibodies, both polyclonal and mono-
clonal, specific receptor proteins, ligands, nucleic
acid sequences, and similar agents which, through
specific binding or chemical reaction with substances
in a specimen, are intended for use in a diagnostic ap-
plication for identification and quantification of an
individual chemical substance or ligand in biological
specimens.” ASRs are purchased by manufacturers
who use them as components of tests that are cleared
or approved by the FDA and also by clinical labora-
tories that use the ASRs to develop LDTs used exclu-
sively by that laboratory. As mentioned previously, the
FDA classifies medical devices, including diagnostic
devices such as ASRs, into class I, II, or III according
to the level of regulatory control that is necessary to
provide a reasonable assurance of safety and effec-
tiveness. These classifications include consideration
of the level of risk associated with the device. The
classification of an ASR determines the appropriate
premarket process. ASR class I devices are subject to
general controls but exempt from premarket notifi-
cation. The manufacturer must label class I, exempt
ASRs with the statement “Analyte Specific Reagent.
Analytical and performance characteristics are not
established.” An ASR class II device is a reagent used
as a component in a blood banking test of a type that
has been classified as a class II device (e.g., certain cy-
tomegalovirus serological and Treponema pallidum
nontreponemal test reagents). An ASR class III
reagent is one intended as a component in tests in-
tended either (i) to diagnose a contagious condition
that is highly likely to result in a fatal outcome for
which prompt, accurate diagnosis offers the oppor-
tunity to mitigate the public health impact of the con-
dition (e.g., human immunodeficiency virus or M. tu-
berculosis) or (ii) for use in donor screening for
conditions for which the FDA has recommended or
required testing to safeguard the blood supply or es-
tablish the safe use of blood and blood products (e.g.,
tests for hepatitis or for identifying blood groups).
All ASRs require manufacturing under current good
manufacturing practices, and both class II and class
III reagents require special controls.
LDTs and Special Considerations
LDTs are procedures developed in-house that have not
been reviewed and/or approved by the FDA. This in-
cludes the use of a laboratory-modified, FDA-cleared
test (e.g., off-label use of an FDA-cleared test) or a
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 8
8 Clark et al.
laboratory-verified procedure that incorporates either
commercially available reagents not cleared by the
FDA for IVD use (e.g., RUO reagents or ASRs) or
reagents produced in-house. Since LDTs are not re-
viewed and/or approved by the FDA and since the
reagents used as components in the test cannot be re-
viewed for their intended use, these tests require more
extensive performance verification (4). Moreover,
federal regulations require that LDTs that incorpo-
rate an ASR are required to include a statement on the
report. For example, an LDT incorporating an ASR
class I reagent would include “This test was developed
and its performance characteristics determined by
[Laboratory Name]. It has not been cleared or ap-
proved by the U. S. Food and Drug Administration.”
Although the use of an RUO reagent as a compo-
nent in an LDT remains controversial and is subject
to the terms defined in the written agreement be-
tween the manufacturer and the laboratory, the CAP
established rules for its accredited labs regarding the
use of RUO reagents when no comparable FDA-
cleared product is available. According to the CAP’s
anatomic pathology checklist, RUOs “purchased
from commercial sources may be used in laboratory-
developed tests only if the laboratory has made a rea-
sonable effort to search for IVD- or ASR-class
reagents and that these efforts should be documented
by the laboratory director” (http://www.devicelink
.com/ivdt/archive/08/03/004.html; accessed 8 July
2009). Although there are no federal regulations to
include a statement on an LDT incorporating an RUO
reagent, it is strongly recommended that the labora-
tory use language analogous to that described above
for a class I ASR. It is important to note that the use
of unmodified kits provided by companies for RUO
or IUO are not considered LDTs, are not to be used
for diagnostic purposes, and are subject to federal reg-
ulations and the language specified in the certifica-
tion program for each product. Please refer to http://
www.devicelink.com/ivdt/archive/08/03/004.html
(accessed 8 July 2009) for further information re-
garding IUO/RUO test kit use in the clinical micro-
biology lab and the role of such kits in clinical
diagnostics.
Since LDTs have not been reviewed or cleared by
the FDA, they require more extensive verification be-
fore patient testing. Moreover, FDA clearance or ap-
proval of a laboratory test does not predict perfor-
mance in a particular laboratory’s patient population
and therefore does not substitute for verification of
test performance before use. Thus, laboratories are
required to verify the performance of tests, FDA
cleared or otherwise, prior to patient testing and
must maintain an active quality assurance program
that validates the test performance over time. In ad-
dition, test verification and validation by clinical lab-
CUMITECH 31A
oratories now also play a critical role in ensuring that
devices and medical products previously cleared or
approved by the FDA are performing as expected.
Laboratories have the responsibility of reporting de-
vice problems to the FDA and manufacturers. Con-
cerns about the quality, performance, or safety of any
medical device should be reported. There are both
voluntary and mandatory reporting systems through
the FDA’s adverse event reporting program, Med-
Watch (http://www.fda.gov/medwatch; accessed 8
July 2009).
Role of CLIA
The CMS regulates all laboratory testing (except re-
search) performed on humans in the United States
through the CLIA regulations. The final CLIA qual-
ity system regulations became effective on 24 April
2003 (4). The objective of the CLIA program is to en-
sure quality laboratory testing. Since 13 November
2003, the FDA has assumed primary responsibility
for performing the CLIA complexity categorization
functions that include the process of assigning com-
mercially marketed IVD test systems to one of three
CLIA regulatory categories based on their potential
risk to public health: waived, moderate complexity,
or high complexity.
Sections 493.1200 through 493.1299 of subpart K
of the regulation set forth the quality control re-
quirements for tests of moderate or high complexity
(waived tests are not subject to quality control). Ef-
fective 24 April 2003, quality control (including ver-
ification) of moderate- and high-complexity tests was
merged into one standard. Section 493.1253 of the
CLIA regulations describes the requirements for the
establishment and verification of method perfor-
mance specifications. For unmodified, FDA-cleared
or -approved tests (either moderate or high complex-
ity), the laboratory is required to verify the manufac-
turer’s performance specifications provided in the
package insert before reporting patient test results.
These specifications are:
• Accuracy
• Precision
• Reportable range
• Reference range
This requirement applies when the laboratory replaces
a test system or instrument (with the same model or a
different model), adds a new test, or changes the man-
ufacturer of a test system.
For LDTs (not FDA cleared or approved) or FDA-
cleared or -approved tests modified by the labora-
tory, the laboratory must establish applicable perfor-
mance characteristics. The requirement only applies
to instruments, kits, or test systems introduced on or
after the effective date of the quality control regula-
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 9
CUMITECH 31A Verification and Validation of Procedures in the Clinical Microbiology Laboratory
tions, 24 April 2003. Prior to reporting patient test
results for methods that fall within these categories,
the laboratory must establish the following perfor-
mance characteristics, as applicable (see Appendix A
for more details):
• Accuracy
• Precision
• Analytical sensitivity
• Analytical specificity, to include interfering sub-
stances
• Reportable range of patient test results
• Reference range(s)
• Any other characteristic required for test perfor-
mance and interpretation of results
In addition, control procedures for patient testing
must be established on the basis of the verified perfor-
mance specifications. The laboratory is required to
have documentation of the verification of the manu-
facturer’s specifications or establishment of perfor-
mance specifications for LDTs or modified FDA-
cleared or -approved tests. All laboratory accrediting
programs which have been granted deemed status by
CMS (including CAP, The Joint Commission, and the
Commission on Laboratory Accreditation) must in-
clude these requirements as part of their inspection cri-
teria. For example, the CAP Laboratory General
checklist includes verification and establishment of
performance specifications (http://www.cap.org/apps
/docs/laboratory_accreditation/checklists/laboratory
_general_sep07.pdf; accessed 25 March 2009).
SELECTION OF A LABORATORY METHOD
Once a laboratory has reached the decision to offer a
new test, the next step in the process is generally the
selection of the method by which the test will be per-
formed. Few laboratories have the time or resources
to perform in-house evaluations of the large number
of available test systems, kits, or methods which may
be available to the microbiology laboratory for de-
tection of an organism, antigen, or other analyte of
interest. Thus, it becomes crucial to approach the se-
lection of a new method in an organized fashion,
making use of all available information to narrow
down the selection without performing expensive in-
house studies. The following steps are designed to
serve as a guide for the initial selection of a labora-
tory method. Although all steps may not be necessary
for every test method under consideration by a labo-
ratory, the basic process can be followed for the ma-
jority of tests utilized by the microbiology laboratory.
9
1. Define the purpose for which the method is to be
used. Common purposes for tests include the fol-
lowing (22):
Screening: Screening is used for testing large pop-
ulations of patients for the presence of a disease
state or analyte (such as an infectious agent). In
general, screening tests should have high (i.e.,
greater than 95%) clinical sensitivity and NPVs. In
most cases, the recommended specificity and PPV
can be lower than those of diagnostic and confir-
matory tests. Thus, a negative screening test result
should indicate that the person has a high proba-
bility of being free of the characteristic, whereas a
positive test result might reflect only the need for
confirmatory testing.
Diagnosis: Diagnosis is used for the evaluation of
persons suspected of having a given disease state
or characteristic (e.g., a particular type of infec-
tion). If the characteristic is important, either for
treatment or for prognostic considerations, sensi-
tivity should be as high as possible. When diag-
nostic test results are not confirmed by additional
laboratory or clinical data, specificity may also
need to be very high. However, if an accurate con-
firmatory test is readily available, a high degree of
specificity might not be necessary. The majority of
clinical tests for infectious diseases are for diag-
nostic use.
Supplemental: Supplemental (may also be referred
to as confirmatory) testing is used after obtaining
a positive screening or diagnostic test result to en-
sure the accuracy of that initial result. Specificity
and PPV, rather than sensitivity and NPV, are usu-
ally the primary considerations for supplemental
tests; specificity should exceed 98%. Supplemen-
tal tests may not be necessary when the screening
or diagnostic test has high specificity and PPVs.
The specificity of supplemental tests is, by defini-
tion, established when the test is used in conjunc-
tion with, and subsequent to, a screening or diag-
nostic test. Supplemental tests should generally
not be used in lieu of screening or diagnostic tests,
unless their performance has been thoroughly ver-
ified for this purpose in well-defined patient pop-
ulations (e.g., treponemal tests for diagnosis of la-
tent or late syphilis).
2. Decide what analyte (e.g., organism, antigen, or
nucleic acid, etc.) is to be detected and what the
reference method or gold standard will be for com-
parison. Note, if the new test is likely to be more
sensitive than the gold standard, then scientifically
valid ways to arbitrate discrepant results (e.g.,
clinical data or other assays, etc.) should be de-
fined prior to beginning verification studies (1, 14,
19, 22).
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 10
10 Clark et al.
3. In collaboration with the end user of the test (e.g.,
the physician) combined with the information from
steps 1 and 2, determine the medical usefulness of
the test (e.g., whether the test will lead to im-
proved patient care and/or shortened hospital stay)
and preliminary clinical and/or microbiological
requirements for test sensitivity, specificity, and
predictive values (as appropriate).
4. Survey the technical and medical literature for per-
formance claims of various methods that may in-
dicate that one or more methods will meet the ini-
tial requirements for sensitivity and specificity, etc.
When reviewing the literature, confirm that the
method described is actually the test (unmodified)
that is to be evaluated in the laboratory.
5. Determine the characteristics of the method(s) of
interest. The choice of method may also be based
on the following practical parameters (3, 16, 26).
The laboratory should prioritize these parameters
based on their patient population and institution’s
mission.
• Cost of the method
What are the comparative costs for material
and labor relative to alternatives to the test?
If applicable, what are costs of service contracts?
For instrumentation, what are the comparative
costs of purchase versus reagent rental versus
lease?
What is the extent of reimbursement?
• Practicality in the laboratory setting
Can the test be performed on all necessary shifts?
Does the test require special equipment?
What is the turnaround time for the test?
What are the personnel and training require-
ments?
Are quality control and proficiency test mate-
rials available?
What is the extent of quality control that will
need to be performed?
What is the extent of preventive maintenance
that will need to be performed?
Is there adequate space in the laboratory to per-
form the test?
Can the test be automated to reduce labor?
Does the system have an indication for all of the
uses and/or organisms that are of interest?
• Specimen requirements
Volume and type of specimen needed
Collection requirements
Transport requirements
Storage requirements
Quality of specimen
CUMITECH 31A
• Quantities of reagents and controls needed for
test; storage requirements
• Shelf life of reagents and controls before and
after opening
• Availability of supplies, service, and/or techni-
cal support
• Possible safety hazards related to performing test
• Whether a reference range is appropriate for
the test and how it will be determined for the
institution
6. Make a preliminary selection of a test method,
perform the in-house verification (see Appendix
A), and write a report to include, but not neces-
sarily limited to, data and conclusions. The labo-
ratory director or designee must sign the report. It
is recommended that a verification study protocol
be written prior to conducting the study. The pro-
tocol should include the study design, methods for
data analysis, and performance expectations.
A brief outline of the laboratory method selection
process is provided in Appendix B.
VERIFICATION OF COMMON
MICROBIOLOGY TESTS
CLIA regulations require that laboratories verify the
manufacturer’s performance specifications for new
unmodified tests prior to reporting patient test re-
sults. These performance specifications include accu-
racy (agreement), precision (or reproducibility), re-
portable range, and reference range (normal values)
(4) (http://www.cms.hhs.gov/clia/downloads/6064bk
.pdf; accessed 5 March 2009).
The accuracy portion of test verification is usually
accomplished by performing the new or revised test
method in parallel with a reference method that has
an established and satisfactory level of accuracy. How-
ever, in the absence of a suitable reference method,
analytic performance characteristics can be estab-
lished using previously characterized clinical samples,
characterized clinical isolates, or simulated samples
prepared by spiking the appropriate test matrix with
various concentrations of characterized clinical iso-
lates or the analyte of interest. That is, the accuracy
of a new test can be established by either method
comparison or recovery studies and is a measure of
concordance between observed and expected results.
Precision (or reproducibility) should be verified
within runs, between runs, and between operators, if
the test system is considered to be highly operator de-
pendent. Fully automated systems are generally not
considered to be operator dependent. Precision can
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 11
CUMITECH 31A Verification and Validation of Procedures in the Clinical Microbiology Laboratory
be verified by testing several negative and positive
specimens (or for quantitative tests, various levels of
analyte) at least in duplicate in the same run and in
different runs. For operator-dependent tests, this
process is performed by at least two different opera-
tors. The comparability of repeated results (either
qualitative or quantitative, as appropriate) is assessed
relative to the manufacturer’s specifications.
The reportable range is verified by testing charac-
terized samples with known values near the lower
and upper ends of the manufacturer’s reportable range
as well as samples in the normal range (usually neg-
ative for analyte).
The reference range (normal values) provided by
the test manufacturer or as published in textbooks or
scientific articles can be used by the laboratory, pro-
vided the laboratory’s patient population is similar. If
the laboratory chooses to evaluate specimens from
known healthy patients, the results should be within
the test’s reference range.
With careful planning, all of these test characteris-
tics may be verified using the same set of samples. For
example, the laboratory can test samples with known
values at the lower and upper ends of the manufac-
turer’s reportable range along with samples in the
normal range for the institution’s patient population,
in different runs, on different days, and by different
personnel.
The results of test verification should indicate one
of three possibilities:
• The test is acceptable for routine use.
• Further verification studies are required.
• Immediate corrective action is required by the
manufacturer (if commercially obtained), the user,
or both. The test is unsuitable for routine use un-
til its performance parameters can be verified.
Certain commonly used microbiology items are
not considered instruments, kits, or test systems under
the CLIA of 1988 and may not require a complete
verification process prior to use. Items such as media
or individual reagents used as components of the
identification process (e.g., oxidase or catalase) may
instead be monitored through the quality control
protocols of the laboratory. However, each labora-
tory must assess the nature and purpose of each of
these components and may choose to perform more
elaborate in-house verification. For example, a deci-
sion may reasonably be made not to perform verifi-
cation on common media with a history of limited
failures, such as most of the media listed in Table 3
of CLSI document M22-A3 (21), but it should be
strongly considered for some of the more highly com-
plex media which are multifunctional (e.g., a
medium which is selective for an organism and con-
11
tains biochemicals that are then used to differentiate
or presumptively identify the organism). This would
include Campylobacter agar, media for the selective
isolation of the pathogenic Neisseria spp., and any
other media not listed in Table 3 of reference 21.
In the following sections, suggested methods are
included for verification of many of the commonly
used tests and test systems found in the microbiology
laboratory. These suggestions are not meant to be all-
inclusive, and alternative approaches may be utilized
in individual laboratories. In addition, it is recog-
nized that the verification process can be timely and
expensive, often complicated by a paucity of specimens
containing (or lacking in some circumstances) the de-
sired analyte. In some cases, laboratories will need to
make difficult choices about the extent of verification
that is possible, taking into consideration how widely
the test has been used and accepted by the micro-
biology community, the extent and results of pub-
lished evaluations, and the impact of an incorrect test
result on the patient. In some cases, repeat testing of
selected control material near the test cutoff value(s)
may give a level of satisfaction. In other cases, labo-
ratories may decide that they are unable to perform
a reasonable verification and may choose to refer the
test to another laboratory. Whenever possible, the
purchase of a new system or test methodology should
be made contingent upon the results of the verifica-
tion studies. Records of the actual test verification re-
sults must be maintained for at least 2 years after re-
tirement of the test system.
Verification Study Design
The verification study design, the statistical analysis
of study results, and the criteria establishing accep-
table performance of a new test will depend on the
type of assay and its intended use. The verification
study of an assay should demonstrate that it can de-
tect or accurately characterize the analyte of interest,
and the verification plan and acceptance criteria
should be established prior to beginning the verifica-
tion studies, if possible. The studies and the accep-
tance criteria should be consistent with the industry
standard for the assay at the time and/or as estab-
lished by peer-reviewed publications. One approach
for defining acceptance criteria is the concept of total
allowable error for a test, based on medical or clinical
requirements (28). While generally applied to quan-
titative testing, this approach can be used to assess the
performance of a new antimicrobial susceptibility test
(AST) system where one considers concordance be-
tween methods to mean a result within 1 doubling
dilution and there is no difference in the interpreta-
tion of resistance. The implication for laboratory
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 12
12 Clark et al.
performance is that the laboratory should verify that
analytical variation is low enough so as not to cause
a clinically significant change in results. This means
that estimates of inaccuracy (bias or lack of trueness)
and imprecision (repeatability and reproducibility)
should show that they are sufficiently small and
within the defined allowable total error. For quanti-
tative methods, a stringent criterion has been recom-
mended where the test value (3 standard devia-
tions) is within the established allowable total error
(28). Other examples of acceptance criteria are dis-
cussed in the following sections.
Verification Samples
Samples used for method verification should be well-
characterized clinical specimens or culture isolates
from either retrospective or prospective clinical spec-
imens or studies whenever possible. All samples should
be collected in compliance with local, regional, and
federal statutes. Proficiency and/or commercially pre-
pared reference panels characterized by one or more
methods are suitable for method verification studies.
However, they may not reflect the sample matrix
or reference range for the user institution’s patient
population.
The FDA permits the use of remnant samples in
certain circumstances (including method verification
studies) provided the specimens are collected for rou-
tine clinical care, they would otherwise have been
discarded, and they are not individually identifiable.
In the case of a rare event or analyte, the use of
archived and/or retrospective samples in addition to
prospective patient samples may be required to gen-
erate data from a large enough sample set for statis-
tical analysis. Alternatively, simulated specimens pre-
pared using sample matrix spiked with characterized
clinical isolates or analyte may be used (often re-
ferred to as spiking, recovery, or seed-and-recovery
studies). In these studies, simulated specimens are
prepared using sample matrix collected from patients
presumed to be without disease, and positive samples
are prepared by seeding (spiking) negative sample
matrix with various concentrations of analyte. A
panel of simulated positive and negative specimens is
randomized and tested blindly. Caution must be
taken when extrapolating assay performance from
simulated specimens. Laboratories using simulated
samples as part of assay verification should docu-
ment the potential limitations of their findings.
AST Systems (Unmodified, FDA Cleared)
The generation of AST results is one of the most im-
portant functions of the microbiology laboratory,
as the results may directly affect the therapy chosen
for the treatment of a patient (9). Thus, it is critical
CUMITECH 31A
that the microbiologist be confident that the system
chosen is able to provide accurate and reliable results
in the user’s own laboratory. Within the past 15 years,
a number of recommendations for verification of sus-
ceptibility test systems have been discussed in the lit-
erature (18, 20, 26). Although it may be difficult for
laboratories to perform a rigorous study of a new
system, use of selected controls (i.e., those obtained
from the American Type Culture Collection [ATCC]
or other reputable sources) and clinical isolates can
aid in the effort to verify the claims made in the lit-
erature and by the manufacturer regarding the accu-
racy and reproducibility of a system (18). Isolates
saved from proficiency surveys may also be used for
verification studies. The evaluation design (two are
suggested below for comparing AST systems) should
allow for the detection of categorical errors (18, 25).
Though not recommended, it is possible to verify an
AST system against disk diffusion testing, even
though only categorical agreement and not essential
agreement will be verified.
Evaluation of susceptibility test methods should
be done using a distribution of organisms similar to
those commonly isolated and representing resistant
phenotypes observed in the institution. For example,
verification of gram-positive antibiotic panels should
include methicillin-resistant Staphylococcus aureus
(MRSA), D-test-positive S. aureus, coagulase-negative
staphylococci, and vancomycin-resistant enterococci;
verification of gram-negative panels should include
extended-spectrum beta-lactamase-, carbapenemase-,
and AmpC-producing Enterobacteriaceae, along with
multidrug-resistant Pseudomonas aeruginosa and
Acinetobacter to the extent that these are encoun-
tered in the institution. Other rare antibiotic-resistant
isolates (such as vancomycin-resistant S. aureus and
beta-lactamase-positive enterococci) may also be in-
cluded in the evaluation. In total, at least 30 isolates
should be tested with each antibiotic panel/card.
While institutions should make every effort to con-
duct studies using their own isolates, those with lim-
ited resources may consider assistance from the ven-
dor in the form of reagents, characterized bacterial
strains, and technical support. However, only labora-
tory personnel, not the vendor technical experts,
should perform the verification studies.
Precision (reproducibility) studies may be per-
formed by testing five isolates in triplicate for 3 to 5
days. Isolates with known antibiotic resistance phe-
notypes such as quality control strains from ATCC
can be used. It would be preferable that at least two
of the precision test isolates be antibiotic resistant
(for example, MRSA and carbapenemase-producing
Enterobacteriaceae). Acceptable precision results
should be 95%, at a minimum (including both es-
sential and categorical agreement; see below).
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 13
CUMITECH 31A Verification and Validation of Procedures in the Clinical Microbiology Laboratory
Precision (reproducibility) essential agreement (PEA):
Agreement within 1 twofold dilution of the preci-
sion test isolate antibiotic MIC.
No. of comparisons within 1
well of known reference MIC
PEA   100
Total no. of results
The total tested can be calculated for all organisms
and drugs combined or for each drug.
Precision (reproducibility) categorical agreement
(PCA): Agreement with the interpretative results
(susceptible/intermediate/resistant [SIR]) of the preci-
sion test isolate using FDA/CLSI interpretive criteria.
No. of categorical result matches
PCA   100
Total no. of results
The total tested can be calculated for all organisms
and drugs combined or for each drug.
AST System Verification Studies Not Arbitrated
with a Reference Method (Test 30 or More
Isolates per Panel/Card)
Many evaluations (due to limited resources) do not
compare the test AST system under evaluation against
a reference method but only compare it against the
current AST system of the laboratory. For this rea-
son, the test AST system may not be incorrect when
results are discrepant between both AST systems.
One cannot automatically call the existing system
correct when the new AST system result differs. For
this type of evaluation when a reference method is
not used, the following types of discrepancies can be
substituted for the traditional error rates (see below).
A minor discrepancy is defined when one AST system
result is intermediate and the other AST system result
is susceptible or resistant. A major discrepancy is
having one result susceptible and the other resistant.
For this type of evaluation, there are no very major
errors (VME). Overall, there should be less than 5%
major errors (ME) and an overall essential agreement
(EA) and category agreement (CA) of 90%. Both
ME and minor errors (MINE) should be a combined
10%. In addition, both the within-1-dilution agree-
ment (EA) and the CA (agreement of interpretative
results [SIR] between both AST system results)
should be 90%.
EA: Agreement within 1 twofold dilution of the
new AST system under evaluation with the current
AST system.
13
No. of comparisons within 1 well
EA   100
Total tested
The total tested can be calculated for all organisms
and drugs combined or for each drug.
CA: Agreement of interpretative results (SIR) between
the new AST system under evaluation and the current
AST system using FDA/CLSI interpretive criteria.
No. of categorical result matches
CA   100
Total tested
The total tested can be calculated for all organisms
and drugs combined or for each drug.
ME: One AST system under evaluation indicates a
susceptible or resistant response and the other AST
system indicates the opposite result as a resistant or
susceptible response.
No. of ME discrepancies
ME   100
Total no. of resistant organisms
tested by both methods
MinE: One AST system indicates an intermediate re-
sponse, and the other AST system indicates either a
susceptible or resistant response.
No. of MinE discrepancies
MinE   100
Total no. of organisms tested
The total tested can be calculated for all organisms
and drugs combined or for each drug.
AST System Verification Studies Arbitrated
with a Reference Method (Test up to 100 or
More Isolates per Panel/Card)
In larger microbiology laboratories with a higher
throughput and potentially higher normal back-
ground of antibiotic-resistant isolates, it may be prefer-
able to perform larger studies of up to 100 or more
isolates per panel/card type. Ideally, at least 50% of
the comparison isolates should exhibit some form of
antibiotic resistance. Discrepancies between AST sys-
tems need to be arbitrated with a reference method
(i.e., microbroth dilution, macrobroth dilution,
and/or agar dilution). It is acceptable to verify a new
AST system against a reference MIC method alone.
The traditional error rates defined below may be
utilized for these larger arbitrated studies between
AST systems or an AST system versus a reference
MIC method. Acceptable performance rates for EA
and CA should be 90%, whereas acceptable
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 14
14 Clark et al.
performance for the VME rate should be 3% (min-
imum of 35 resistant isolates); not more than 1 in 33
of these antibiotic-resistant isolates should repeatedly
test falsely susceptible (18). The ME rate should be
3%. For ME and MinE combined (a minimum of
100 strains), the error rate should be a combined
7%. Additional detailed FDA recommendations
for industry AST system evaluations can be found at
http://www.fda.gov/MedicalDevices/DeviceRegulation
andGuidance/GuidanceDocuments/ucm080564.htm
(accessed 8 July 2009).
EA: Agreement within 1 twofold dilution of the
new AST system under evaluation with the current
test AST system as verified by the reference method.
No. of comparisons within  1 well
EA   100
Total no. of organisms tested
The total tested can be calculated for all organisms
and drugs combined or for each drug.
CA: Agreement of interpretative results (SIR) be-
tween the new AST system under evaluation and the
current AST system as verified by the reference
method using FDA/CLSI interpretive criteria.
No. of categorical result matches
CA   100
Total no. of organisms tested
The total tested can be calculated for all organisms
and drugs combined or for each drug.
VME: The new AST system under evaluation indi-
cates a susceptible response while the current AST
system, as verified by the reference method, indicates
a resistant response. Clinically, VME are the most se-
rious type of error and can only be detected by test-
ing organisms resistant to each antimicrobial agent.
No. of VME discrepancies
VME   100
Total no. of resistant organisms
by current AST system as verified
by the reference method
ME: The new AST system under evaluation indicates
a resistant response, while the current AST system
verified by the reference method indicates a suscepti-
ble response. ME can only be detected by testing or-
ganisms susceptible to each antimicrobial agent.
No. of ME discrepancies
ME   100
Total no. of susceptible organisms
by current AST system as verified
by the reference method
CUMITECH 31A
MinE: Either the new AST system under evaluation
or the current AST system as verified by the reference
method indicates an intermediate response. The
other method indicates either a susceptible or resis-
tant response.
No. of MinE discrepancies
MinE   100
Total no. of organisms tested
The total tested can be calculated for all organisms
and drugs combined or for each drug.
Furthermore, users should investigate all categori-
cal errors for all of the above types of AST system ver-
ifications. If the specified limits are exceeded for any
antibiotic, the test must be considered unverified and
withdrawn from consideration or corrective action
must be taken in conjunction with the manufacturer
to attempt to resolve the discrepancies. Following cor-
rective action, the new or revised test should be run
again in parallel with the reference method on a min-
imum of 20 appropriate isolates (isolates which will
demonstrate that the problem[s] has been corrected).
Error rates may not be significant with certain anti-
microbial agent-organism combinations (i.e., these
types of errors may be considered for elimination
from the study). In addition, if a significant number of
organisms have MICs near the breakpoint, the cate-
gorical agreement may be 90% due to the inherent
plus-minus dilution variabilities of AST systems.
Diagnostic Microbiology and Microbial
Identification Tests
Processes for verifying diagnostic microbiology tests
and microbial identification tests are similar.
Examples of analytes detected by diagnostic micro-
biology tests include the following:
• Microorganisms
• Microbial antigens
• Nucleic acids
• Antibodies
Examples of microbial identification test formats
include the following:
• Antisera
• Antigens
• Chemicals
• Stains
• Instruments
• Reagents
• Kits
The process for verifying a microbiology test may
be relatively straightforward for analytes which are
common in the population or for organisms fre-
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 15
CUMITECH 31A Verification and Validation of Procedures in the Clinical Microbiology Laboratory
quently encountered. When the new test will be de-
tecting a relatively uncommon analyte or organism
(for example, a direct test for the detection of Cryp-
tosporidium), assessment of the sensitivity or accu-
racy of the test in the user’s own patient population
becomes difficult. In these circumstances, the manu-
facturer, other laboratories, or commercial sources
may be able to provide specimens of known content
or isolates to be used in the evaluation.
It is important to remember that a test may be
suitable for one population of patients and not for
another. Laboratories must evaluate this individually
and be prepared to provide physicians with the spec-
ifications of the test for individual populations with
different disease prevalences if pertinent.
Verification of Unmodified, FDA-Cleared Tests
Diagnostic Tests
The new test should be performed in parallel with the
existing test or a reference method on at least 20
specimens (http://www.cms.hhs.gov/clia/downloads/
6064bk.pdf; accessed 5 March 2009), generally di-
vided equally among those positive for the analyte
and those negative for the analyte. Flexibility in the
distribution of positives and negatives is important
to account for such factors as rare analytes and the
institution’s patient population. Both weakly and
strongly positive specimens should be included to
verify the reportable range of the assay. If weakly
positive specimens are unavailable, dilution of
strongly positive specimens with the appropriate
specimen matrix can achieve the same effect. In some
circumstances, it may be necessary to test more spec-
imens to document that the test meets the required
level of sensitivity, specificity, or PPV and NPV. An
example of this decision-making process can be
found in Appendix A.
After the results of the new test are compared with
those of the reference method (gold standard), the
number of true-positive, true-negative, false-positive,
and false-negative results obtained with the test to
be verified should be documented. Total accuracy
should be at least 90% relative to the reference
method (3). When the reference method is known to
be an imperfect standard, an attempt to resolve dis-
crepancies should be made. The calculated predictive
values may not be indicative of the performance of
the test in an actual patient population if the preva-
lence of the target analyte is different from that used
in the verification study. A test with a specificity of
95% when the prevalence of a target disease is only
1% will have a predictive value of a positive result of
only 16.7%. Individual predictive values may be cal-
culated from the sensitivity and specificity data for
known different prevalences, as seen in the example
in Appendix A. Note that while it is common to as-
15
sess sensitivity, specificity, and predictive values, it is
only required to assess overall accuracy (agreement)
of unmodified FDA-cleared tests. If the accuracy study
includes samples with known values at the lower and
upper ends of the manufacturer’s reportable range
(weakly and strongly positive), the laboratory is also
able to verify the reportable range. Together with a
precision (reproducibility) study and package insert
data on the reference range, the verification require-
ments are met.
The laboratory should also establish a procedure
and acceptance criteria for the precision/repro-
ducibility study. Precision should be verified within
runs, between runs, and between operators, if the test
system is considered to be highly operator depen-
dent. To assess within-run precision, several mem-
bers of the 20-specimen panel (such as two negative
specimens and two positive specimens) should be run
at least in duplicate. This process is then repeated on
a different run and, if necessary, by a different opera-
tor. For qualitative diagnostic assays, the same or
comparable results should be obtained. Any devia-
tions should be investigated. For quantitative diag-
nostic assays (a value is reported), the laboratory
should calculate the percent coefficient of variation
(within runs, between runs, and total) and compare
the results to the manufacturer’s package insert data.
The test may be considered verified if it meets the
requirements initially established for performance by
the manufacturer of the test (22) or as defined by the
user in the verification protocol.
Microbial Identification Tests
Verification of automated, multi-analyte test systems
for identification of microorganisms to the species
level should be conducted with a minimum of 20 iso-
lates representing a wide-range of clinically relevant
organisms for the institution (http://www.cms.hhs
.gov/clia/downloads/6064bk.pdf; accessed 5 March
2009). This may include gram-positive organisms such
as S. aureus, coagulase-negative staphylococci, ente-
rococci, and Streptococcus pneumoniae and gram-
negative organisms such as members of the Entero-
bacteriaceae and non-lactose fermenters. Note that
large institutions with complex patient populations
and a wide variety of clinically relevant organisms
would likely need to evaluate more than 20 isolates.
In addition, the appropriate quality control organ-
isms should also be tested and included during the
verification process. For identification methods that
detect only one analyte (e.g., immunofluorescent
reagents), the new test should be run in parallel with
the existing test or a reference method on a minimum
of 20 microbial isolates (http://www.cms.hhs.gov/
clia/downloads/6064bk.pdf; accessed 5 March 2009)
generally divided equally among those positive for
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 16
16 Clark et al.
the analyte and those negative for the analyte. Flexi-
bility in the distribution of isolates containing and
lacking the analyte is important to account for such
factors as rare analytes and the institution’s patient
population.
For microbial identification tests, there should be
at least 90% agreement with the existing system or
reference method before the new method is consid-
ered verified (3). Due to a variety of factors, the level
of misidentification deemed acceptable should be de-
termined by each laboratory, and should take into ac-
count the manufacturer’s performance specifications
provided in the package insert. Certain groups of or-
ganisms may be more challenging for new systems to
identify (e.g., nonfermenters, corynebacteria, coagu-
lase-negative staphylococci), and greater flexibility
may be necessary in assessing the accuracy of the new
method (i.e., identification to the genus level only
may be acceptable performance). In addition, the types
of disagreements encountered with the new system
should be scrutinized. The new system may misiden-
tify an organism, may require further tests before
identifying an organism, or may give no identifica-
tion at all. Misidentification is the most serious error
for an identification system; however, a laboratory
may choose to accept a certain number of isolates
with no or appropriate partial identification if other
factors (e.g., cost or speed) outweigh the inconven-
ience of further testing and would have minimal im-
pact on patient care.
If the accuracy (agreement) of the new or revised
test does not satisfy the verification requirements, the
test must be considered unverified and withdrawn
from consideration or corrective action must be
taken by the manufacturer, the user, or both. Follow-
ing corrective action, the new or revised test should
be run again in parallel with the reference method
and interpreted as described above.
Blood Culture Systems
Meaningful verification of a new blood culture sys-
tem is one of the most difficult tasks facing the clini-
cal microbiologist. Parallel testing requires collection
of additional blood from each patient and may not
be possible in some patients or institutions. The low
level of positivity for clinical pathogens (usually in
the range of 8 to 14%) (3) means that most of the
specimens collected will be of little value in the com-
parison. In addition, the incidence of contamination
(usually 1 to 3%) and predominance of a limited
number of pathogens may result in an evaluation
skewed toward only a few of the potentially clinically
significant organisms. Thus, verification studies
should be designed to answer the following funda-
mental questions:
CUMITECH 31A
• Will the media used by the system support the
growth of organisms (including yeasts, anaerobes,
and fastidious organisms, where appropriate)
commonly seen in the user’s patient population?
• Will the instrument (for automated systems) de-
tect, in a timely fashion, the majority of patho-
genic organisms from blood cultures which con-
tain these microorganisms?
Two approaches for verification of blood culture
systems are discussed below. Laboratories may also
choose to combine these approaches to take advan-
tage of the strong points of each (e.g., perform paral-
lel studies to assess the ability of the system to detect
commonly isolated organisms and perform seeded
blood cultures to assess less common pathogens).
There may be circumstances in which the labora-
tory is implementing a newer version of a test system.
Blood culture systems may be a good example of this.
Depending on the changes to the new system version, a
verification study may not be necessary. If the differ-
ences between the current and new systems are lim-
ited to the blood culture instrument (e.g., hardware
and software) and the blood culture bottles are not
changed, then an instrument function check by a ven-
dor technical representative is sufficient to verify ad-
equate performance of the complete blood culture
system in the user’s laboratory. The functional check
would verify that the incubation and optical systems
and the software are operating per the manufac-
turer’s specifications.
Seeded Blood Culture Studies
Select a minimum of 20 isolates representative of
blood culture isolates normally seen in the institu-
tion. Include both gram-positive and gram-negative
bacteria and yeasts, as appropriate. To the greatest
extent possible, use actual patient isolates rather than
stock strains. Prepare seeded blood cultures with iso-
lates of each of the above species. To challenge the
system, the minimum amount of sterile, antibiotic-
free human blood recommended by the manufacturer
should be placed in each bottle. In addition, the
numbers of organisms placed in each bottle should
approximate those found in cases of septicemia
(which can be less than 0.1 CFU per ml of blood) (3).
This is accomplished by making serial dilutions of
the organisms prior to inoculation to achieve ap-
proximately 5 to 30 CFU per bottle.
The method is considered verified if all isolates are
detected within time frames specified by the user.
Times to positivity must be consistent with the liter-
ature for given organisms. As few as 3 days may be
sufficient to recover at least 95% of clinically rele-
vant bacteria and yeasts (7). Any problems with de-
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 17
CUMITECH 31A Verification and Validation of Procedures in the Clinical Microbiology Laboratory
tection should be investigated by repeating the tests
with the same patient strains. If detection is still not
obtained, corrective action must be taken by the user
and/or the manufacturer prior to instituting use of
the system in the laboratory.
Parallel Blood Culture Studies
Performance of parallel blood cultures allows the
laboratory to evaluate all aspects of the new system
under actual patient and laboratory conditions.
When a laboratory chooses to perform parallel stud-
ies of commercially available systems, duplicate sets
of blood cultures inoculated with equivalent blood
volumes should be obtained until a minimum of 20
positive blood cultures (not to include contaminants
[7]) representative of the blood culture isolates nor-
mally seen in the institution are evaluated.
The new method is considered verified if the sen-
sitivity is at least 95%, relative to the reference
method, and times to detection are not significantly
different. If the new method does not meet perfor-
mance requirements, it should be withdrawn from
consideration or corrective action should be taken by
the user (and the manufacturer where appropriate)
and the verification study should be repeated.
Verification of LDTs, Modified FDA-Cleared
Tests, and Commercially Available Kits
That Are Not FDA Cleared
Diagnostic Tests
The accuracy of these tests should be verified as de-
scribed above, except that a minimum of 50 speci-
mens that contain the target analyte and a minimum
of 100 specimens that lack the target analyte should
be studied (11). Both positive accuracy (sensitivity)
and negative accuracy (specificity) should be at least
95% compared to the reference method (11), or as
defined by the user in the verification protocol.
Note that this section only covers accuracy. CLIA
regulations also require that the user establish per-
formance specifications for precision/reproducibility,
reportable range, reference range, analytical sensitiv-
ity, analytical specificity (including interfering sub-
stances), as well as additional characteristics for
quantitative tests (4). This process is described briefly
in Appendix A.
In some instances, test verification will be required
because of very small modifications (e.g., a minimal
change in incubation time) of an existing protocol
which has been previously verified. In this situation,
verification must be performed to the extent neces-
sary to demonstrate that the change has not affected
the performance of the test, but it may not require
the extensive testing performed initially. It may be
useful to maintain a panel of a limited number of
17
well-characterized specimens and to do a complete
verification only if these specimens do not give satis-
factory results.
Microbial Identification Tests
Microbial identification tests that identify isolates to
the species level should be tested in parallel with the
existing test or another reference method on a mini-
mum of 200 isolates (11). Whenever possible, these
isolates should include all species identifiable by the
new or revised test. The same criteria for method
agreement described for unmodified, FDA-cleared
methods (at least 90% overall agreement) should be
met by these tests to consider them verified (11). If
the new or modified test identifies a particular ana-
lyte, it should be tested in parallel with the existing
test or a reference method on a minimum of 50 mi-
crobial isolates that contain the target analyte and a
minimum of 100 isolates that lack the target analyte
(11). Both positive accuracy (sensitivity) and negative
accuracy (specificity) should be at least 95% com-
pared to the reference method (11), or as defined by
the user in the verification protocol.
If the new or modified test does not satisfy the ver-
ification requirements, the test must be considered
unverified and withdrawn from consideration or cor-
rective action must be taken. Following corrective ac-
tion, the new or revised test should be compared
again with the reference method as described above.
Some state or local agencies may regulate LDTs
and provide specific requirements for verification
studies. For example, the New York State Department
of Health/Wadsworth Center has established submis-
sion guidelines for laboratory-developed, nucleic acid
amplification tests for infectious agents (http://
www.wadsworth.org/labcert/TestApproval/forms/
NAATSubmissionGuidelines.pdf; accessed 25 March
2009). These guidelines specify that a minimum of
40 specimens be evaluated, including at least 30 pos-
itive for an analyte and at least 10 that are negative.
This Cumitech recommends a minimum of 50 posi-
tive and 100 negative specimens for verification of
laboratory-developed diagnostic microbiology tests
(as stated in reference 11).
VALIDATION OF DIAGNOSTIC TESTS USED
IN CLINICAL MICROBIOLOGY
While verification of a new or revised test serves to
establish that test performance parameters are satis-
factory, it does not provide ongoing assurance that
the expected test performance is satisfactory under
routine use over extended periods of time. Test vali-
dation is the ongoing process used by the laboratory
providing tests to provide this assurance.
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 18
18 Clark et al.
The components of the validation process are ad-
dressed by the CLIA regulations (http://wwwn.cdc
.gov/clia/regs/toc.aspx; accessed 25 March 2009).
Assays covered by CLIA comprise the following:
FDA-cleared tests, FDA-cleared modified tests, LDTs,
IUO tests, RUO tests, and tests incorporating ASRs.
The components of this key process for laboratories
providing nonwaived testing include a total testing
process (i.e., preanalytic, analytic, and postanalytic)
along with general laboratory systems. Each labora-
tory must maintain written policies and procedures
that implement and monitor these quality preana-
lytic, analytic, and postanalytic phases. These vari-
ous components (see below) must meet the testing
specialty and subspecialty assay needs.
The end result of validation will indicate one of
three possibilities: (i) the test continues to be accept-
able for routine use, (ii) immediate corrective action
must be undertaken by the manufacturer (if commer-
cially obtained), the user, or both, or (iii) the test
must be considered unsuitable for continued routine
use until it can be validated.
Components of the Validation Process
The standard components of a validation process
have expanded since the CLIA of 1988 and now in-
clude the parameters listed below (4). In addition, the
laboratory must monitor and evaluate the overall
quality of the general laboratory systems on an on-
going basis and must identify and correct issues. The
quality review must include the general effectiveness
of the corrective action used to fix issues.
Personnel Competency Assessment and Training
The laboratory must have a written documented sys-
tem to establish the competency of each laboratory
employee. This is an extremely important component
of laboratory quality control, since it ensures person-
nel are trained appropriately to perform prompt and
accurate testing. All employees must be assessed for
competency semiannually during their first year in a
new laboratory department and then annually there-
after (12, 24, 27). All newly implemented assays must
have proper, documented personnel training before
patient test results are reported to clients (training
verification).
The CLIA regulatory requirement lists six areas
that must be part of a competency assessment pro-
gram:
• Direct observations of performance of routine pa-
tient tests
• Monitoring of the recording and reporting of test
results
CUMITECH 31A
• Review of intermediate test results or worksheets,
quality control records, proficiency testing results,
and preventive maintenance records
• Direct observation of performance of instrument
maintenance and function checks
• Assessment of test performance through testing
previously analyzed specimens, internal blind test-
ing samples, or external proficiency testing sam-
ples
• Assessment of problem-solving skills
Not every type of assessment needs to be per-
formed for each area being reviewed, and the type of
assessment tool used (from the above list) should be
selected based on whether it will provide an accurate
reflection of the performance and competence of the
worker. Direct observation may be more important
when first assessing a worker’s ability to perform a
test to which they have been recently trained, whereas
assessment of ongoing competence for a fully trained
and experienced worker may be performed using
some direct observation of work performed com-
bined with worksheet review, maintenance record re-
view, written examination, and/or review of profi-
ciency testing samples in the specified area.
Training may be divided into three parts:
• Initial training occurs when a new or current em-
ployee is trained to perform a new test or when a
new assay is introduced.
• Retraining may occur when periodic competency
assessments are unacceptable.
• Training updates are required whenever changes
are implemented to a procedure or when additions
are made to the training document, such as new
learning objectives.
It is helpful to prepare a checklist for each of the
above procedures to ensure competency for each crit-
ical component. Direct observation includes all as-
pects of testing, including specimen handling, speci-
men processing, testing, and instrument maintenance
and function checks, if applicable. Record review in-
cludes assessing the trainee’s ability to record inter-
mediate test results, such as on worksheets; the
recording of quality control, proficiency testing, and
preventive maintenance records; and the final record-
ing and reporting of test results. Test performance in-
cludes the use of previously analyzed specimens, in-
ternal blind samples, or even proficiency samples to
assess competency. Problem-solving skill assessments
may be tested through written quizzes or by the doc-
umentation of a problem resolution in which the
worker participated. Any unacceptable perfor-
mance(s) noted among employees with the above six
procedures must be addressed immediately with re-
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 19

CUMITECH 31A Verification and Validation of Procedures in the Clinical Microbiology Laboratory 19

medial training. The appropriate supervisor (with di- samples with the specific analyte (i.e., microorganism,
rector oversight) must sign off on the appropriate antigen, or amplification target sequence), or photo-
paperwork indicating that the trainee has been graphic images obtained from a reference laboratory.
trained properly for the specific assay(s). The reader
Quality Control Organisms
is referred to three comprehensive reviews of this
For FDA-cleared or -approved tests, laboratories
topic for further details (12, 24, 27).
must minimally perform the number of tests and type
of quality control testing as described in the package
Proficiency Testing
insert. Additional controls may be considered. Each
CLIA-certified laboratories must enroll in a CMS-
laboratory should establish its own collection of
approved proficiency program for each specialty and
quality control isolates. These isolates should be ob-
subspecialty for which they seek certification. For a
tained from a reputable source (i.e., ATCC or an-
current list of proficiency testing providers, see http://
other recognized vendor). Isolates should be main-
www.cms.hhs.gov/CLIA/14_Proficiency_Testing_
tained frozen or lyophilized. For some assays (such as
Providers.asp (accessed 2 June 2009). Proficiency
antibiotic susceptibility testing), bacterial isolates
testing must be performed at least twice per year.
should be subcultured from the frozen or lyophilized
Also, the laboratory must test the proficiency sample
state on a monthly basis and then stored on a slant
in the same manner as any other sample in the labo-
for weekly testing. Other isolates used for quality
ratory. For example, if a standard patient sample
control of commercial microbial identification sys-
would not normally require a repeat analysis by an
tems may require less-frequent recovery from the
assay, then the proficiency sample should not be re-
freezer (yearly may suffice). If there is some question
peated by that same assay. If the standard operating
regarding the validity of an isolate during quality
procedure requires or accepts duplicate testing of pa-
control testing, then a new subculture should be im-
tient samples for an assay, then it is acceptable to run
mediately obtained from the freezer stock or other
the proficiency sample in duplicate by this same assay.
reputable source. If nonreference strains are used, the
In addition, laboratories must not participate in
laboratory should have a complete record of the his-
any interlaboratory communication related to the
tory of the organism including characterization, stor-
proficiency sample(s) test results until after the date
age, and recovery from storage.
on which the laboratory reports proficiency test re-
Each user needs to determine which quality con-
sults to the proficiency testing program. Failure to
trol strain(s) is to be used with each assay and how
score at least an 80% on a certain testing event in a
often the quality control testing should be per-
specialty (i.e., microbiology, parasitology, mycobac-
formed. In many instances (see package inserts), the
teriology, mycology, or virology) indicates an unsat-
vendor will recommend the specific isolates to test
isfactory performance. The laboratory must then
and how often to test. When there is no specific ven-
undertake retraining with appropriate documenta-
dor recommendation, the laboratory can research the
tion. Failure to achieve an overall satisfactory per-
appropriate CLSI document and/or obtain informa-
formance for two consecutive testing events or two
tion from authoritative sources such as the Manual
out of three consecutive testing events is unsuccessful
of Clinical Microbiology to determine the best course
performance. Depending upon the circumstances,
of action. The frequency of testing and actions to be
CMS may then impose sanctions which may include
taken after quality control failure should follow the
the suspension of testing in that specialty area.
vendor’s recommendations and those of regulatory
For tests that have no external proficiency testing
agencies (5, 6, 21).
program available, the laboratory must establish an
internal proficiency program to monitor the accuracy Quality Control Analytes
and reliability of these tests. These tests must be as- Quality control analytes are a metabolic product, nu-
sessed at least semiannually. Microbiology tests for cleic acid, enzyme, or antigen usually provided by the
which proficiency testing is not normally available manufacturer of a test or system for the routine qual-
include the serum bactericidal test, minimal bacteri- ity control of a specific procedure or instrument. The
cidal concentration test, and culture for Mycoplasma analyte should be identified with a lot number or
hominis and/or Ureaplasma. Testing material may be should have other traceable identification; further
difficult to collect for some of these assays. Alterna- description should indicate concentration, titer
tive performance assessment procedures may include (where appropriate), use, date of preparation, and
split-sample analysis with reference or other labora- storage conditions. For an analyte introduced by the
tories, split-samples with an established in-house user laboratory, a defined record of its development
method, blind testing of quality control products and assay should be available. For FDA-cleared or
obtained from another laboratory, seeded clinical -approved tests, laboratories must minimally
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 20
20 Clark et al.
perform the number of tests and type of quality con-
trol testing as described in the package insert. Addi-
tional controls may be considered. The frequency of
testing should follow the vendors’ recommendations
or those stipulated by one or more of the various reg-
ulatory or advisory agencies (5, 6, 21). For LDTs,
positive, negative, and other relevant controls should
be selected from recommendations for the test found
in appropriate CLSI guidelines or the Clinical Micro-
biology Procedures Handbook (17).
Comparison of Test Results or Multiple
Instruments and Testing Sites
If the laboratory performs the same test method us-
ing multiple units of the same instrument or performs
the same test method at multiple testing sites, there
should be a program in place to demonstrate equiva-
lency of test results. It is not necessary to duplicate
the entire assay verification process on each instru-
ment or at each testing site if the sites operate on the
same CLIA certificate. However, as part of the ongo-
ing validation process, laboratories should demon-
strate that multiple instruments produce equivalent
test results, both prior to performing a new test and
periodically thereafter (e.g., at least twice a year). This
can be accomplished by testing well-characterized
patient specimens (or isolates), archived proficiency
samples (or isolates), or controls. When regulations
require testing of quality control material at a certain
frequency, alternating the quality control material
among multiple instruments can also serve to verify
equivalent performance. For example, if a laboratory
has two identical instruments for bacterial identifica-
tion and antibiotic susceptibility testing, one-half of
the quality control organisms could be tested on in-
strument A and the other half on instrument B one
week and then reversed each week.
Implementation of New Assays
Manufacturers routinely introduce a number of new
assays each year. Some of these assays are indeed an
improvement over current laboratory technology.
However, some newer tests are significantly more ex-
pensive (e.g., MRSA PCR) than standard methods
(routine MRSA culture screen). The hospital labora-
tory needs to work with its administration to decide
whether or not to implement these new expensive as-
says. If the expense of the new testing can be justified
by cost reductions in other parts of the hospital (i.e.,
lowered MRSA infection rates), then it may be justi-
fiable to implement the new test for routine testing in
the laboratory (see Appendix B).
Instrument Calibration
Certain instruments require that specific components
or internal systems be checked on a regular basis. It
is imperative that the manufacturer’s directions for
CUMITECH 31A
the calibration be carried out at the specified time
intervals.
Use of Historical Laboratory Data
Laboratories are encouraged to utilize historical data
concerning recovery and detection of pathogens in
their own populations as an aid in confirming that
the system is operating as expected. If significant
changes are seen in the distribution and/or frequency
of recovery of isolates obtained from patients over a
period of time (for example, a substantial reduction
in the number of isolates of S. pneumoniae from that
obtained with the old system during an equal time
period), a more intensive investigation into the abil-
ity of the new system to support and detect these
species is warranted. Monthly assessments of ampli-
fication positivity rates (i.e., MRSA and Chlamydia
trachomatis, etc.) are currently a CAP requirement in
the microbiology laboratory.
Confidentiality of Patient Information
There must be a system available to maintain patient
confidentiality throughout all phases of laboratory
testing.
Specimen Identification and Integrity
There must be a written policy in place that protects
the identification and integrity of each sample during
the testing process that is under the laboratory’s con-
trol. This policy must include the maintenance of test
records.
Complaint Investigations
There must be a system in place to handle laboratory
complaint investigations. This system must include a
policy for taking corrective action whenever test re-
sults, equipment or methodologies, or patient test
results fall outside the reportable range of test results
for the assay system in question. Corrective action
would also be required whenever normal values or
reference intervals for a test procedure are inappro-
priate for the laboratory’s patient population or
whenever control material fails.
Communications
There must be a system in place to handle issues of
communication breakdown with an authorized indi-
vidual who orders tests. Excellent communication
must exist between the laboratory and the individu-
als who order tests to eliminate potential errors.
When the above 12 components of the validation
process are in place, the user has assurance that the
test or test system meets the validation requirements.
Frequency of Test Validation
Test validation is essentially a continuous process. In-
dividual laboratories are responsible for ensuring
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 21
CUMITECH 31A Verification and Validation of Procedures in the Clinical Microbiology Laboratory
that the specific components of test validation, listed
above, occur frequently enough to ensure the contin-
ued performance of a laboratory test. In most cases,
following the manufacturer’s guidelines and/or the
requirements of the regulatory or accrediting agen-
cies will provide this assurance.
SUMMATION
This updated document provides guidance in per-
forming test verification and validation in the micro-
biology laboratory. As stated at the outset, these are
guidelines and should not be considered regulatory
standards. For those responsible for establishing and
maintaining standards in clinical laboratories, there
are numerous excellent documents available, many
of which are cited throughout this text. Our goal
here is to make this information more microbiology
friendly. Ensuring good laboratory practice, which
includes complying with regulations from various
agencies, can be a challenge. The availability of clear
and useful guidelines for performing verification and
validation which specifically address clinical micro-
biology should make the accomplishment of this as-
pect of good laboratory practice easier to achieve.
APPENDIX A
METHOD SELECTION AND
VERIFICATION EXAMPLE
Laboratory A has decided to switch enzyme immunoassay
(EIA) kits for the detection of Clostridium difficile toxins A
and B from stool samples. Brand X (new non-FDA-cleared
kit) is compared to the current laboratory A method (Acme
assay). Laboratory A utilizes the Acme assay as the refer-
ence method (gold standard) against the brand X test kit.
The department management decided to test a total of 150
stool samples (50 positive and 100 negative for C. difficile
toxins A and B by the Acme assay) in both systems. The
following was obtained (Table A1): accuracy (agreement) 
(148/150)  100  99%. Precision (reproducibility) was
evaluated by testing five replicates of positive and negative
kit controls as well as a positive external control (a stool
sample positive for C. difficile toxin) on three different
days. The reproducibility was 100% (complete concor-
dance, qualitatively, both inter-run and intra-run). The lab-
oratory management then contacted the vendor who pro-
vided them with the analytical sensitivity (lower detection
limit) for toxin detection. Specific analytical interferences
TABLE A1. Non-FDA-cleared assay (brand X)
No. of specimens tested by
Test result (brand X) gold standard (Acme assay) with result
Positive Negative
Positive 49 1 a
Negative 1 99
21
and reference range (negative for C. difficile toxins A and
B) were also obtained from the vendor (Table A2). The ref-
erence range was verified by the negative results (n  99)
obtained from stool specimens from patients without C.
difficile disease (n  100). The reportable range was veri-
fied by testing positive reference specimens with high and
low values (provided by the vendor). All specimens yielded
positive results by the brand X test kit.
A new C. difficile assay (brand Y) became FDA cleared,
and the department management decided to test the new
assay against their Acme assay (gold standard). A precision
(reproducibility) study was performed as described for the
brand X kit and showed 100% reproducibility (see Table
4). As the brand Y kit is FDA cleared, the department man-
agement decided to test a total of 20 stool samples (10 pos-
itive and 10 negative for C. difficile toxins A and B by the
Acme assay) in both systems. The following was obtained
(Table A3): accuracy (agreement)  (18/20)  100  90%.
Precision (reproducibility) was evaluated by testing five
replicates of positive and negative kit controls as well as a
positive external control (stool sample positive for C. diffi-
cile toxin) on three different days. The reproducibility was
100% (complete concordance, qualitatively, both inter-run
and intra-run). The laboratory management then contacted
the vendor who provided the reference range (negative for
C. difficile toxins A and B) (Table A4). As the brand Y kit
is FDA cleared, it was not necessary to document analyti-
cal sensitivity and specificity (interferences). The reportable
range was verified by testing positive reference specimens
with high and low values (provided by the vendor). All
TABLE A2. Summary of verification studies for C. difficile
toxin A/toxin B EIA
Result for non-FDA-cleared
Parameter brand X assay (C. difficile
toxin A/toxin B EIA)
Specimen type. . . . . . . . . . . . . Stool
Accuracy (method
comparison) . . . . . . . . . . . . . 99%
Intra-run precision . . . . . . . . . . 100%, qualitative
Inter-run precision . . . . . . . . . . 100%, qualitative
Analytical sensitivity (lower
detection limit)a . . . . . . . . . .Toxin A, 0.4 ng/ml; toxin B,
0.5 ng/ml (obtain from
vendor)
Analytical specificity
(interference)a . . . . . . . . . . . . Not affected by blood, barium,
or treatment with antibiotics
(obtained from vendor)
Reference range . . . . . . . . . . . Negative for C. difficile toxins A
and B, as determined by
testing specimens from unin-
fected patients (accuracy
study) representative of insti-
tution’s patient population
Reportable range . . . . . . . . . . . 100% detection of reference
specimens (provided by ven-
dor) with high and low values
Additional performance specifications that must be determined for non-
FDA-cleared assays.
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 22

22 Clark et al. CUMITECH 31A

TABLE A3. FDA-cleared assay (brand Y) 2. Carpenter, A. B. 2007. Immunoassays for the diagno-
sis of infectious diseases, p. 257–270. In P. R. Murray,
No. of specimens tested by gold
standard (Acme assay) with result E. J. Baron, J. H. Jorgensen, M. L. Landry, and M. A.
Test result (brand Y) Pfaller (ed.), Manual of Clinical Microbiology, 9th ed.
Positive Negative American Society for Microbiology, Washington, DC.
Positive 9 1 3. Carroll, K. C., and M. P. Weinstein. 2007. Manual
Negative 1 9 and automated systems for detection and identifica-
tion of microorganisms, p. 192–217. In P. R. Murray,
E. J. Baron, J. H. Jorgensen, M. L. Landry, and M. A.
TABLE A4. Summary of verification studies for C. difficile Pfaller (ed.), Manual of Clinical Microbiology, 9th ed.
toxin A/toxin B EIA American Society for Microbiology, Washington, DC.

Result for FDA-cleared 4. Centers for Medicare and Medicaid Services. 2003.
Parameter brand Y assay (C. difficile Medicare, Medicaid and CLIA programs; laboratory
toxin A/toxin B EIA) requirements relating to quality systems and certain
personnel qualifications; final rule. Fed. Regist. 68:
Specimen type . . . . . . . . . . . . .
Stool. 3640–3714.
Accuracy (method comparison) .
90% .
Intra-run precision . . . . . . . . . . . .
100%, qualitative 5. Clinical and Laboratory Standards Institute. 2006.
Inter-run precision . . . . . . . . . . . .
100%, qualitative Methods for Dilution Antimicrobial Susceptibility
Reference range . . . . . . . . . . . . .
Negative for C. difficile toxins Tests for Bacteria That Grow Aerobically; Approved
A and B (obtained from Standard, 7th ed. CLSI document M7-A7. Clinical
vendor) and Laboratory Standards Institute, Wayne, PA.
Reportable range . . . . . . . . . . . . 100% detection of reference
specimens (provided by 6. Clinical and Laboratory Standards Institute. 2006.
vendor) with high and Performance Standards for Antimicrobial Disk Sus-
low values ceptibility Tests; Approved Standard, 9th ed. CLSI
document M2-A9. Clinical and Laboratory Standards
Institute, Wayne, PA.
7. Clinical and Laboratory Standards Institute. 2007.
specimens yielded positive results by the brand Y test kit Principles and Procedures for Blood Cultures; Ap-
(Table A4). proved Guideline. CLSI document M47-A. Clinical
After completion of both verification studies, the depart- and Laboratory Standards Institute, Wayne, PA.
ment management decided to implement the FDA-cleared 8. Diekema, D. J., and M. A. Pfaller. 2007. Infection con-
test (brand Y) due to billing reasons (non-FDA-cleared tests trol epidemiology and clinical microbiology, p. 118–
are not billable to the federal government). All of the above 128. In P. R. Murray, E. J. Baron, J. H. Jorgensen,
required parameters need to be included in a formal report. M. L. Landry, and M. A. Pfaller (ed.), Manual of Clin-
The laboratory director or designee is required to approve ical Microbiology, 9th ed. American Society for Micro-
all verification studies. Laboratory inspectors may require biology, Washington, DC.
review of verification studies during on-site audits. The
9. Doern, G. V., R. Vautour, M. Gauder, and B. Levy.
verification report should be maintained for the life of
1994. Clinical impact of rapid in vitro susceptibility
the assay but may be discarded 2 years after the assay is
testing and bacterial identification. J. Clin. Microbiol.
discontinued. In addition, for laboratories that are licensed
32:1757–1762.
to accept samples from New York State patients, the New
York State Health Department may require review of veri- 10. Dunne, W. M., and M. T. LaRocco. 2007. Laboratory
fication data for modified, FDA-cleared tests and LDTs be- management, p. 4–19. In P. R. Murray, E. J. Baron, J. H.
fore New York patient testing is initiated. Jorgensen, M. L. Landry, and M. A. Pfaller (ed.), Man-
ual of Clinical Microbiology, 9th ed. American Society
for Microbiology, Washington, DC.
ACKNOWLEDGMENTS
11. Elder, B. L., S. A. Hansen, J. A. Kellogg, F. J. Marsik,
We recognize the valuable contributions of the authors and R. J. Zabransky. 1997. Cumitech 31, Verification
of the first edition of this Cumitech: B. Laurel Elder, and Validation of Procedures in the Clinical Micro-
Sharon A. Hansen, James A. Kellogg, Frederic J. Marsik, biology Laboratory. Coordinating ed., B. W. McCurdy.
Ronald J. Zabransky, and Brenda W. McCurdy (coordinat- American Society for Microbiology, Washington, DC.
ing editor).
12. Elder, B. L., and S. E. Sharp. 2003. Cumitech 39,
Competency Assessment in the Clinical Microbiology
REFERENCES Laboratory. Coordinating ed., S. E. Sharp. American
1. Alonzo, T. A., and M. S. Pepe. 1999. Using a combi- Society for Microbiology, Washington, DC.
nation of reference tests to assess the accuracy of a 13. Gutman, S. 1999. The role of Food and Drug Admin-
new diagnostic test. Stat. Med. 18:2987–3003. istration regulation of in vitro diagnostic devices—
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 23

CUMITECH 31A Verification and Validation of Procedures in the Clinical Microbiology Laboratory 23

APPENDIX B
PROCESS FOR SELECTION OF A TEST METHOD

applications to genetic testing. Clin. Chem. 45:746– p. 14.0.1–14.4.10. In L. S. Garcia and H. D. Isenberg
749. (ed.), Clinical Microbiology Procedures Handbook,
14. Hadgu, A. 1996. The discrepancy in discrepant analy- 2nd ed., vol. 3. American Society for Microbiology,
sis. Lancet 348:592–593. Washington, DC.

15. Ilstrup, D. 1990. Statistical methods in microbiology.
Clin. Microbiol. Rev. 3:219–226.
16. Jenkins, S. G. 1995. Evaluation of new technology in
the clinical microbiology laboratory. Diagn. Micro-
biol. Infect. Dis. 23:53–60.
17. Jenkins, S. G. (section ed.). 2007. Quality assurance,
quality control, laboratory records, and water quality,
18. Jorgensen, J. H. 1993. Selection criteria for an anti-
microbial susceptibility testing system. J. Clin. Micro-
biol. 31:2841–2844.
19. McAdam, A. J. 2000. Discrepant analysis: how can
we test a test? J. Clin. Microbiol. 38:2027–2029.
20. Murray, P. R., A. C. Niles, and R. L. Heeren. 1987.
Comparison of a highly automated 5-h susceptibility
31A_Cumitech_557040-rpt 6/24/10 9:17 AM Page 24
24 Clark et al.
testing system, the Cobas-Bact, with two reference
methods: Kirby- Bauer disk diffusion and broth micro-
dilution. J. Clin. Microbiol. 25:2372–2377.
21. NCCLS. 2004. Quality Control for Commercially
Prepared Microbiological Culture Media; Approved
Standard, 3rd ed. NCCLS/CLSI document M22-A3.
NCCLS, Wayne, PA.
22. NCCLS. 2001. Specifications for Immunological Test-
ing for Infectious Diseases; Approved Guideline, 2nd
ed. NCCLS/CLSI document I/LA 18-A2. NCCLS,
Wayne, PA.
23. NCCLS. 1995. Assessment of the Clinical Accuracy of
Laboratory Tests Using Receiver Operating Charac-
teristic (ROC) Plots; Approved Guideline. NCCLS/
CLSI document GP10-A. NCCLS, Wayne, PA.
24. NCCLS. 2004. Training and Competence Assessment;
Approved Guideline, 2nd ed. NCCLS/CLSI document
GP21-A2. NCCLS, Wayne, PA.
CUMITECH 31A
25. Radetsky, M., and J. K. Todd. 1984. Criteria for the
evaluation of new diagnostic tests. Pediatr. Infect. Dis.
3:461–466.
26. Richter, S. S., and M. J. Ferraro. 2007. Susceptibility
testing instrumentation and computerized expert sys-
tems for data analysis and interpretation, p. 245–256.
In P. R. Murray, E. J. Baron, J. H. Jorgensen, M. L.
Landry, and M. A. Pfaller (ed.), Manual of Clinical
Microbiology, 9th ed. American Society for Micro-
biology, Washington, DC.
27. Sharp, S. E., and B. L. Elder. 2004. Competency as-
sessment in the clinical microbiology laboratory. Clin.
Microbiol. Rev. 17:681–694.
28. Westgard, J. O. 2008. Basic Method Validation, 3rd
ed. Westgard QC, Inc., Madison, WI.