Consensus Standards for

Method Validation

FDA/ASA Industry Workshop

September 29, 2006

Daniel W. Tholen, M.S.

Dan Tholen Statistical Consulting

Traverse City, Michigan

©2006: Dan Tholen 1

Statistical Consulting
Sources

 CLSI: Clinical and Laboratory

Standards Institute, Wayne, PA
– Area Committee for Evaluation Protocols
 ISO: International Organization for
Standardization, Geneva
– Technical Committee 69: Application of
Statistical Methods

2
Purpose of presentation

 Promote broader use of TC 69 and

CLSI EP consensus documents
 More active involvement by FDA and
IVD Industry Statisticians
– TC 69 US TAG
– EP AC Members, Observers, project
participants

3
My Background

 CLSI EP Area Committee

– Observer, Adviser, Member, since 1984
– Chairman, EP5, EP6, EP17
– Co-author EP7, EP9, EP10, EP15
 ISO TC 69
– Member, US TAG
– ILAC Liaison (International Laboratory
Accreditation Cooperation), since 2002
– “Expert” for ISO 13528, 11843-5
4
My Background
 Medical Laboratories
– Statistician, CAP, 1981-1995
– Expert Consultant, CDC DLS (2000-current)
– Adviser, CLSI AC on General Lab Practice (1998)
 International Standards
– ISO CASCO (Convener, WG 28 (PT))
– European Commission (EEE-PT, IMEP)
– Asia-Pacific (Member, APLAC PT Committee)
 Other fields of testing
– EPA (NELAC), FDA (CFSAN, CBER), AOAC
– A2LA Assessor for ISO 17025, Guides 43 & 34 5
Method Capability Statistics:
Consensus vs. Traditional
 Traditional: Publications, convention
– One person (or team)
– Specific field of application
– Peer review
– Expedited publication
 Consensus standards
– Many people, countries, areas of interest
– Exhaustive process
– Very general language, basic approach
– No “pride of authorship” 6
Consensus organizations

 Accredited SDO (in US, by ANSI)

– Impartiality
– All stakeholders represented
– All comments addressed
– Records
 Organizations
– ISO (100+ TCs, CASCO, REMCO)
– CLSI
– ASTM
7
Consensus process

 ISO
– Anyone participates, through TAG or
“mirror committee”
– One vote per country
– CD, DIS, FDIS, Approved stages
 CLSI
– Anyone participates
– Balance Industry, Profession, Government
– SC Draft, Proposed, Approved stages 8
TC 69 Scope

• Standardization in the application of

statistical methods, including
generation, collection (planning and
design), analysis, presentation and
interpretation of data.
• Advisor to all ISO technical
committees in matters concerning the
application of statistical methods in
standardization. 9
TC 69 Subcommittees

 TC: Basic Statistical Techniques

 SC 1: Terminology
 SC 4: Process Management
 SC 5: Acceptance Sampling
 SC 6: Measurement Methods and
Results
 SC 7 (New): Procedures for Six Sigma
10
TC 69 Basic Techniques

 Estimation & Tests for Mean & Variances

 Tests for departure from Normality
 Confidence & Tolerance Intervals
 Estimation of median

11
SC 1: Terminology

 ISO 3534 Series -- Vocabulary and

symbols
– Part 1: Probability and general statistical
terms
– Part 2: Statistical quality control
– Part 3: Design of experiments
 Mostly consistent with VIM, more frequently
updated
12
SC 4: Process
Management
 Control Charts
– Acceptance
– Shewhart
– CUSUM
 Process Capability
 Machine Performance
 Statistical Process Control

13
SC 5: Acceptance
Sampling
 Inspection by variables
 Inspection by attributes
 Assessment of declared quality levels
 Double sampling
 Sequential sampling
 Accept-zero sampling plans

14
SC 6: Measurement
Methods and Results
 ISO 5725: Accuracy (trueness and precision)
of measurement methods and results
– Part 1: General principles
– Part 2: Repeatability and reproducibility
– Part 3: Intermediate precision
– Part 4: Trueness
– Part 5: Alternative methods for precision
– Part 6: Use of accuracy values

15
ISO 5725 Procedures

 Part 2 (precision)
– Selection of laboratories
– Selection of materials (levels)
– Statistical analysis
 Outliers (labs, samples, replicates)
 Graphics (h and k statistics)

 1-way ANOVA for each level

 Linear model for r and R vs. level

16
ISO 5725 Procedures

 Part 3 (intermediate precision)

– Similar to Part 2
– Consideration of sources
 Time
 Operator

 Equipment

 Conditions

– Possible use of nested ANOVA

17
ISO 5725 Procedures

 Part 4 (Trueness)
– Comparison to reference method
 By level
 Across levels

18
SC 6: Limit of Detection

 ISO 11843: Capability of Detection

– Part 1: Terms and definitions
– 2: Methodology in linear calibration case
– 3: When no calibration data exist
– 4: Comparing minimum detectable value
with a given value
– 5: Linear and nonlinear calibration (under
development)
19
SC 6: 11843 Procedures

 “Currie” model
– Considers Type 1 and Type 2 error
(consistent with new VIM)
– Assumes normal distribution
– Assumes repeatability conditions
– No “less than” values allowed

20
SC 6: Other Documents

 ISO 10576: Conformity with Specified

Requirements
 ISO 11095: Linear calibration using
reference materials
 Issues with Measurement Uncertainty
– Formal Liaison to JCGM WG1

21
CLSI Evaluation Protocols

 Protocols for evaluating clinical

method capability
 Consistent with ISO
– Language for clinical lab
– User friendly
 Guidance: NOT mandatory
 Strong preference by FDA CDRH
22
EP Accuracy Documents

 EP5: Precision of Quantitative Methods

 EP9: Method Comparison and Bias
Estimation
 EP12: Evaluation of Qualitative Test
Performance
 EP21: Estimation of Total Analytical Error
 GP10 (EP24): Accuracy Estimation with
Receiver Operating Characteristic Plots

23
EP 5 Procedures

 Nested 2 factor design

– Days (at least 20)
– Runs (usually 2/day)
– (revision: add device, calibration)
 Two or more levels (revision: 5 or
more)

24
EP 9 Procedures

 Comparison with reference method

 Use certified materials of patient
samples
 Test entire measuring interval

25
EP 21 Procedures

 Combines precision and bias

 Requires setting goals

26
EP Process Capability

 EP6: Linearity of Quantitative Methods

 EP17: Limits of Detection and
Quantification

27
EP 6 Procedures

 Straight line and polynomial regression

 Statistical significance of nonlinear
coefficients
 Difference between straight line and
polynomial, compared to goal, at
every point (not same as CAP LN)

28
EP 17 Procedures

 “Currie” method, Type I and Type II

errors
 Truncated at zero
 Nonparametric
 LOQ based on goal for error

29
EP Method Verification
 EP7: Interference Testing
 EP10: Preliminary Evaluation of Quantitative
Methods
 EP14: Evaluation of Matrix Effects
 EP15: User Verification of Precision and
Trueness
 EP18: Quality Management for Unit-Use
Testing

30
EP Risk Management

 EP22: Establishment of Manufacturer's

Recommendations for User Quality
Control (under development)
 EP23: User Verification of
Manufacturer's Recommendations for
Quality Control (about to start)
 Started as a result of CMS initiative on
“Equivalent Quality Control” (EQC)
31
Other CLSI Activities

 Other Area Committees

– Field specific (Chemistry, Hematology,
Microbiology, Molecular Methods, General
Lab Practice, etc.)
 Secretariat for ISO TC212: IVD
Devices
– Also for US TAG to TC 212

32
Contact Information

 General TC 69: www.iso.org

 US TAG: standards@asq.org
 CLSI: www.clsi.org
 CLSI EP: lschmidt@clsi.org
 Me: tholen.dan@gmail.com
1.231.929.1721

33
Thank you!
(Questions?)

34