Measurement

Systems Analysis

Design for Six Sigma

Week 2

© 2003-2007 Cummins Inc. All Rights Reserved.

Certain content © 2000-2002 Sigma Breakthrough Technologies, Inc. Used with permission.
 Objectives

Introduce Measurement Systems Analysis

Define basic measurement terms

Outline procedure for performing a Gage Study (Measurement
Systems Analysis)

Perform an exercise to practice methodology

2
 DFSS Work Flow
 RACI Diagram  Discussion Guide
Determine Voice of the Customer  Market Perceived Value Profile  Image KJ
 Customer Selection Matrix  Weakness KJ
C
CONCEPT
Develop requirements and  Translation Worksheet  QFD HOQ
measurable top level CFR(s)  Requirements KJ  CPM Scorecard
 Importance Questionnaire

Generate and select  Concept Generation  DFA Analysis Worksheet

the solution concept  Pugh Screening Matrix  DFMEA

Flow down top level CFR(s) to  Critical Parameter Tree  Correlation

D
DESIGN
lower level critical parameters  Cause & Effect Matrix  Process Mapping

 Regression  Basic Stats / Graphs

Determine strategic transfer functions  Design of Experiments  Test For Means
 Response Surface Design  Test For Variances
 ANOVA
Determine CFR(s) capability using  Monte Carlo Simulation  Control Charting
transfer functions and X distributions  Empirical Tolerancing x

Measurement Systems Analysis
Dimensional Variation Analysis
 Response Optimization  Regression Applied to Life Data
O
OPTIMIZE
Optimize for robust performance
 Robust Design 

Accelerated Life Testing (ALT)
Confidence Intervals
Is CFR(s) capable?   P-Diagram
Y/N Capability Analysis
 Life Data Analysis
 Sample Size Planning
Determine actual CFR(s) capability.  2 Proportions Comparison
 Capability Analysis
C
CAPABILITY
Compare to predicted
capability where possible. Y/N  Control Plan
 CPM Scorecard

3
 Measurement Process

The ideal measurement system produces “true” measurements

every time

Quality of the measurement system is characterized by
statistically evaluated sources of measurement variation

The measurement process should include:
– Design and certification
– Capability assessment over Time
– Control
– Repair and re-certification

Properties of a Good Measurement System
– Must be in Statistical Control over time
– Variability must be small compared to product specifications
– Variability must be small compared to process variation
– Resolution should be about one-tenth of product specification or
process variation
4
 Possible Sources of Observed
Process Variation
Observed Process Variation

Actual Process Variation Measurement Variation

Long-term Short-term Variation Variation due Variation due

Process Variation Process Variation w/i sample to gage to operators

Repeatability Calibration Stability Linearity

To address actual process variability, the variation due to the

measurement system must first be identified and separated from that
of the process

We will look at “repeatability” and “reproducibility” as primary

contributors to measurement variation

5
 Measurement Process Model

Process Measure Value Analyze Decision

σact σMS σobs

σ2obs = σ2act + σ2MS

σobs Observed process variance

σact Actual process variance
σMS Measurement system variance
6
Effect of Measurement System
Variation – Single Measurement

“True” value

σMS

Range of observed measured results

7
 Effect of Measurement System
Variation – Poor Capability

Actual Product or
Process Variability

Measurement
Variability

Total Variability
(Observed
variability)

LSL USL

8
 Information to be obtained

How big is the measurement variation?

What are the sources of measurement variation?

Has the gage been calibrated?

Is the gage stable over time?

Is the gage capable for this study?

How do we improve the measurement system?

9
 Sources of Measurement
Variation

Measurement variation can come from many sources.

A gage R&R study, if conducted correctly, can capture many of
the sources, if the studies are designed to look at both short and
long term sources of variation.

There are sources of measurement variation that can not be
captured through repeated measurements and must be
evaluated separately.

10
 Categories of Measurement
Variation

Reference Environment
Standard

Measurement Measurement Measurement

Set-Up Equipment Subject

Total
Measurement
Variation
Definition of Procedures Software
Measurand Calculations

Physical Personnel
Constants

Note: Examples of these categories of variation can be found in the Appendix

11
Now a few Definitions

12
 Measurement Accuracy vs.
Precision

Accuracy
– Closeness of the agreement between the result of a measurement
(Observed) and the true value of the measurand (reference value).

Precision
– The net effect of discrimination, sensitivity, repeatability over the
range of interest of the measurement system.
– In some organizations precision is used interchangeably with
repeatability.

13
 A B

Accurate & Precise Precise but not

Accurate

C D

Accurate but not Not Accurate

Precise and not Precise
14
 Resolution

The smallest difference between indications that can be

meaningfully distinguished.

Must be suitable for needs.

Poor Resolution Improved Resolution

X X
15
 Repeatability

Closeness of agreement between the results of successive

measurements of the same measurand carried out under the
same conditions of measurement.

16
 Reproducibility

Closeness of agreement between the results of measurements

of the same measurand carried out under changing conditions of
measurement.

A valid statement of reproducibility requires specification of the
conditions changed.

17
 Bias

An observed difference between the expected value and the

measured value, estimated over an appropriate number of
repeated measurements of calibration standards or calibrated
parts.

Bias

Measurement Expected
Result Value
18
 Stability

The ability of a measuring instrument to maintain a constant bias

over time.

Bias t3

Bias t2
Time

Bias t1

Expected Measurement Results

Value
19
 Linearity

Characteristic of a measurement scale in which the spacing

between each scale increment is proportional to the measurand.
Measurand

Non-Linear

Measurement Results
20
 Sources of Variation that can be
Evaluated by a Gage R&R

Repeatability

Reproducibility

Linearity

Stability

Bias

21
 Sources of Variation that
“Typically” Evaluated by R&R

Repeatability

Reproducibility

Linearity

Stability

Bias

22
 A Measurement Capability Index
- P/T

Precision to Tolerance Ratio

6.00 *σ MS Usually expressed

P /T = as percent
Tolerance

Addresses what percent of the tolerance is taken up by
measurement variation

Includes both repeatability and reproducibility
– Operator x Unit x Trial experiment

Best case: 10% Acceptable: 30%

The P/T ratio will determine whether the measurement system
can sort parts acceptably against the product specifications.

Note: 6.0 standard deviations accounts for 99.73% of MS variation.

23
 Precision to Tolerance Ratio

LSL USL
Product Tolerance

P/T = 20% measurement

system variation

P/T = 100%

P/T = 200%

24
 Uses of P/T Ratio (% Tolerance)

The P/T ratio (% Tolerance in MINITAB) is the most common

estimate of Measurement System precision
– This estimate may be appropriate for evaluating how well the
Measurement System can perform with respect to specifications
– Specifications, however, may be too tight or too loose
– Generally, the P/T ratio is a good estimate when the measurement
system is only used to classify production samples
• Even then, if process capability (Cpk) is not adequate, the P/T ratio may
give you a false sense of security

25
 Another Measurement Capability
Index -- %R&R or Gage R&R

σ MS
% R& R = × 100 Usually expressed
σ Total as percent

Addresses what percent of the Total Variation is taken up by

measurement variation

Includes both repeatability and reproducibility
– Operator x Unit x Trial experiment

As a target, look for %R&R < 30%

26
 % Repeatability &
Reproducibility

Observed Process Variation

%R&R = 20% measurement

system variation

%R&R = 75%

%R&R = 100%

27
 Uses of %R&R (% Study
Variation)

The %R&R is the best measure for the Design Specialist
– This estimates how well the Measurement System performs with
respect to the overall process variation
– %R&R is the best estimate when performing Design studies

Care must be taken to use samples representing full, but

typical, process variation

28
 LSL USL
Product Tolerance

Observed Process Variation

P/T = 20% %R&R = 20%

P/T = 50% %R&R = 50%

P/T = 100% %R&R = 100%

measurement system variation

29
 LSL USL
Product Tolerance

Observed Process Variation

P/T = 50% %R&R = 25%

P/T = 100% %R&R = 50%

P/T = 200% %R&R = 100%

measurement system variation

30
 LSL USL
Product Tolerance

Observed
Process Variation

P/T = 10% %R&R = 20%

P/T = 20% %R&R = 40%

P/T = 50% %R&R = 100%

measurement system variation

31
 Effect of Measurement Variation
on Capability Indices

The higher the Measurement Variation the more dramatic the

impact on your ability to understand the true design/process
capability!

32
 Gage R&R Study Set-up

Generally two to three operators

Generally 10 units to measure

Each unit is measured 2-3 times by each operator
– This will quantify variation due to repeatability and reproducibility

33
 Designing Your MSA Study

When evaluating the stability or linearity of the measurement

system, assure the calibration standard is capable e.g.
– calibration standard’s stability < 25% measurement system’s stability

Where practical get at least 30 measurement values to assure a
statistically valid sample.

Measurement should be spread across trials that represent the
major sources of variation.

34
 Potential Sources of Variation in
Your MSA Study

Category Variation Measurements Conditions

Environment Temp changes 3 Am, noon, pm

dramatically during the
day

Equipment Suspect calibration issue 1 5 measurements of

calibrated part and set-
up

Set-Up Part mounting on fixture 10 30 set-ups with same

may not be repeatable, operator
but is not influenced by
operator
Parts Surface finish may be 10 10 parts
issue

35
 Gage R&R Study Sample
Selection

For an accurate Gage R&R calculation, samples should be pulled from
the process that span the normal variation of the process
– Example: If you produce a material with a mean thickness of 0.010” and a
sigma of 0.001”, get samples that have thicknesses from 0.007 - 0.013”
(99% of the range)
– If the part-to-part variation is too small the GR&R value would be artificially
high

Do not compare different part numbers
– If you produce the same base material with different assays with the same
process, sub-group them and perform the R&R study
– Example:
• A process produces a material with thicknesses of 0.025, 0.050, and 0.080 all with
a +/- 0.005 tolerance, all measured with the same system
• Perform 3 studies - one for each thickness
– If you lumped the above samples together, the GR&R value would be
artificially low

In either case, the σMS will still be accurate

36
 What if the Test is Destructive?
Sample Selection Guidelines

Sample selection is an issue with destructive and on-line
measurement systems

True repeatability is not possible

Sample selection can be done to minimize between part
variability in the following manner
– Collect the number of ‘Master’ samples necessary for your study
– Sub-divide them into enough smaller samples to cover the Operator
x Sample x Trial value

Assumption
– The variability between the smaller samples is negligible and can be
ignored

37
 Destructive Testing
Sample Selection Example

Assume we have a 2 Operator, 2 Trial, 10 Sample study we wish

to conduct
We would need 10 Master Samples and 2 x 2 x 10 = 40 smaller
samples 1 2 3 4 5

6 7 8 9 10

38
 Destructive Testing Methods

Procedure
– For each Operator / Trial / Sample combination, randomly pick a
smaller sample from the respective group
– Each operator performs the measurement independently

DO NOT MIX ALL SMALLER SAMPLES TOGETHER!

39
 General Procedure for
Performing an R&R Study

Calibrate the gage, or assure that it has been calibrated

Have the operator 1 measure all samples once in random order

Have the operator 2 measure all samples once in random order

Continue until all operators have measured samples once (this is
Trial 1)

Repeat steps 2-4 for the required number of trials

Use the form provided to determine the statistics of the R&R
study
– Repeatability
– Reproducibility
– Standard deviations of each of the above
– %R&R
– P/T Ratio

Analyze results and determine follow-up action, if any 40
 Gage R&R Example

Let’s use MINITAB to analyze some data

Open worksheet Thickness GR&R in file MSA.MPJ

We will be using MINITAB’s Gage R&R Study functions

41
 Gage R&R Example: Laminate
Thickness Measurement

This is a gage study for an over-arm micrometer to measure the
thickness of a material

The values represented are multiplied by 1000 for ease of data
entry (i.e. 8.9 = 0.0089”)

The specification value is needed to determine %P/T. For this
example the specification is +/-0.7, or a tolerance of 1.4

Let’s start analyzing the data
– Stat>Quality Tools>Gage Study>Gage R&R Study (Crossed)

42
MINITAB GR&R Study

43
Options / Info Screens

Enter the specific gage

information here for records
Important when doing a
number of iterations

Enter the Process Tolerance

in the Options screen
REMEMBER-tolerance is the
entire range of the
specifications!
You can add a descriptive
Title as well

44
 MINITAB Output

MINITAB produces both analytical and graphical analysis

information
– Analytical Results
• ANOVA table(s)
• Components of Variation
• Percent contribution table
– Graphical Results
• X-Bar / R chart
• Components of Variation
• Operator*Part Interaction Plot
• By Operator and By Part plots

Let’s look at the analytical first and then return to the graphical

45
 Gage R&R Report

Study Var %Study Var %Tolerance

Source StdDev (SD) (6 * SD) (%SV) (SV/Toler)
Total Gage R&R 0.0496522 0.297913 54.67 21.28
Repeatability 0.0476514 0.285908 52.47 20.42
Reproducibility 0.0139528 0.083717 15.36 5.98
Operator 0.0139528 0.083717 15.36 5.98
Part-To-Part 0.0760485 0.456291 83.73 32.59
Total Variation 0.0908224 0.544934 100.00 38.92

Number of Distinct Categories = 2

• % R&R (for Design efforts)

• % P/T (for acceptance to spec efforts)
• BE CAREFUL - watch for low Distinct Categories! Must
be at least 5 for Design use!
46
 Distinct Categories

LCL
LSL UCL
USL

100% Noise

LCL
LSL
UCL
USL

Attribute Data
LCL
LSL
UCL
USL

Low Grade Variable Data

47
Graphical Output

48
 Gage R&R X / R Chart

• The Range chart

can help identify
inadequate
discrimination

D4=2.575 for 3 trials

Avg. Range=0.0567
D4*Avg.R=0.1459

Based on Noise of the Gage

• We want to see variability in the X chart outside the Control Limits

• This indicates Part-to-Part variability
• If there was none, you probably did not get samples that cover the
normal range in production
49
 X-Chart Indicators

If the average for each operator is different, the reproducibility is

suspect

We want more averages to fall outside the control limits… but
consistently for all operators
– This indicates more part-to-part variability which is what we want

We want to see the majority of the points on the chart outside the
control limits
– If this is the case and the R-Chart is in control, then we will be able
to determine the percent of the design or process variability that is
consumed by the measurement system

50
 R-Chart Indicators

Suspect inadequate Discrimination if:

– the range chart has less than 5 distinct levels within the Control
Limits
– 5 or more levels for the range but more than 1/4 of the values are
zero

Repeatability is questionable if the range chart shows out-of-
control conditions

If the range for an operator is out-of-control and the others are
not, the method is suspect

If all operators have ranges out-of-control, the system is sensitive
to operator technique

51
 Operator-Part Interaction Plot

• A line is plotted for each operator by part

• Significant interactions are indicated by diverging or crossing lines
between operators
• We desire parallel lines for all operators, all parts
• If there exists Operator-Part interactions, we need to understand and
resolve them
52
 Components of Variation

• A graphical representation of the data discussed before

• We want the Gage R&R bars to be as small as possible, driving the
Part-to-Part bars to be larger

53
 By Operator

• The By Operator graph shows the average value (crossed

circle) and the spread of the data for each operator
• We want the grouping to be similar across all operators and a
flat line across the means for the operators
54
 By Part

• The graph shows the average (crossed circle) and spread of the
values for each part
• We want to see minimal variability for each part and large
variability between parts (large part-to-part variation)
55
 P/T and GR&R Metrics -
Guidelines

Number of
Distinct
P/T R&R Categories
%Study %Contribution (Discrimination
Ratio Variation (Ratio of Variances) Index)
Red

30% 28% 7.7% 5

Yellow

8% 14% 2% 10
Green

R&R % Contribution is more statistically correct than %Study Variation

56
 Exercise

Take a look at the next worksheet New Gage

Tolerance RANGE for this file is 1.4 (values represented are
multiplied by 1000 for ease of data entry as before)

Run the Gage R&R study

Take 5 minutes and draw your conclusions on:
– Gage acceptability for Design
– Gage acceptability for Product Acceptance
– Improvements made to gage system

57
 Exercise

How to do a Run Chart on

the data in the New Gauge
worksheet and select Part,
Operator and Thickness.

58
 Gage Run Chart
Gage Run Chart of Thickness by Part, Operator
R eported by :
G age nam e: T olerance:
D ate of study : M isc:

1 2 3 4 5 O perator
D oug
7.7 Rich
Rick

Mean 7.6
Thickness
s

7.5

6 7 8 9 10

7.7

7.6 Mean

7.5

Ope rator
Panel variable: Part

The chart shows the average values for each Sample by Trial by Operator
Look for anomalies - we want flat lines at the same value for all operators!
59
 Measurement System Analysis
Questions

Written inspection/ measurement procedure?

Detailed process map developed?

Specific measuring system and set-up defined?

Trained or Certified Operators?

Instrument calibration performed in a timely manner?

Tracking Accuracy?

Tracking R&R?

Tracking Bias?

Tracking Linearity?

Tracking Discrimination?

Correlation with supplier or customer where appropriate?

60
 Measurement System Analysis
Questions

Have you picked the right measurement system? Is this
measurement system associated with either critical inputs or
outputs?

What do the precision, accuracy, tolerance, P/T ratio, %R&R and
trend chart look like?

What are the sources of variation and what is the measurement
error?

What needs to be done to improve this system?

Have we informed the right people of our results?

Who owns this measurement system?

Who owns trouble shooting?

Does this system have a control plan in place?

What’s the calibration frequency? Is that frequent enough?

Do identical systems match?
61
 Variables MSA

How will a “good” measurement system look?

How will a “bad” measurement system look?

What is the impact to the customer if the system is “bad”?

62
 GR&R Exercise

The Department of Health has decided to let you check your own
waste water cleanliness provided you can measure the water
volume with acceptable repeatability and reproducibility

Due to the nature of the materials being measured, if you touch
the container while measuring you will be exposed to potentially
high levels of biological and chemical toxins.

If you touch the container while measuring, you will not be
permitted to perform your own measurements, costing the
company thousands of $$ in labor and testing by outside
laboratories
63
 GR&R Exercise

Equipment needed
– 10 plastic containers
– 1 measurement device

Specifications
– See instructor

Procedure
– Have each of three operators independently measure the height of
the water three times (use proper MSA techniques), WITHOUT
touching the container and following the order of the worksheet
• REMEMBER - Maintain one operator per trial!
– Record the measurements in the “MSA Exercise” worksheet in the
MSA.MPJ MINITAB file.
– The measurement order of the containers is randomized (i.e.,
1,3,7,5,2,...) for each operator

Analyze the results and present to class
64
 Summary

Measurement systems are important to analyze BEFORE

embarking on Design activities

Be careful when picking samples - watch for correct sub-
grouping and sample size requirements

Analyze the measurement system for Operator, Part, and Trial
effects

Make sure that the gage system has enough discrimination to be
useful in determining different levels in the measured attribute

Always generate a GR&R Report to document findings, methods,
and improvement opportunities

Total Variation includes Measurement Error - try to minimize the
controllable error in the gage system

65
 Appendix

Variation Sources and GR&R Study Matrix Design Methods

66
 Categories of Measurement
Variation

Reference Environment
Standard

Measurement Measurement Measurement

Set-Up Equipment Subject

Total
Measurement
Variation
Definition of Procedures Software
Measurand Calculations

Physical Personnel
Constants

67
 Example Sources of
Measurement Variation

Measurement Set-Up
– Fixturing/part location
– Contact angle/Cosign error
– Flexure
– Probe type/size

Reference Standards
– Gage blocks/rings/pins
– Ball bar/hole plate
– Torque transducer

68
 Example Sources of
Measurement Variation

Measurement Equipment
– Range
– Sensitivity
– Resolution
– Linearity
– Stability
– Frequency response
– Digital interface

69
 Example Sources of
Measurement Variation

Environment
– Temperature
– Humidity
– Vibration
– Electrical power quality
– Note: Environmental sources of variation can many time be very
dependent on time of day or shift.

Measurement Subject
– Cleanliness
– Surface finish

70
 Example Sources of
Measurement Variation

Definition of Measurand
– Referenced the correct datum
– GD&T call out

Physical Constants
– Temperate coefficient of material

Measurement Procedures
– Number of points measured
– Location of measurement points
– Settling time
– Checking calibration prior to measurement

71
 Example Sources of
Measurement Variation

Personnel
– Properly trained
– Fail safes

Software/Calculations
– Algorithms verified
• Example calculating a cone
– Filtering
• Frequency filtering
• Sampling frequency

72
 MINITAB Gage R&R Data Matrix

Assume we want to set up a table for a 3 Operators, 10 Sample

per trial, 2 Trials gage study

We want to create a MINITAB file with the following columns
– Part or Sample
– Operator
– Trial
– Measurement

There are two ways to create the data table

First, let’s set up the table to have the correct Trial / Operator /
Sample combinations using the Make Patterned Data procedure

73
 Data Matrix Development

The general format for generating a design for:

# Trials = a
# Operators = b
# Samples = c

Trials The syntax to the left

(1:a)bc are the MINITAB
Session commands for
Operators
setting up the table
a(1:b)c
Samples We will generate the
table with pull down
ab(1:c) menus instead

Let’s look at this example. In this case a=2; b=3; c=10

74
Making Patterned Data

75
Pull Down Menus
Trials
(1:a)bc

1:a

bc

76
Operators
a(1:b)c

1:b

77
Samples
ab(1:c)

1:c

ab

78
MINITAB Gage R&R File

79
 Coding Variables

You can change the

Operator values to the
actual person’s name and
still perform the analysis
using a Coding procedure

80
Coding Variables

81
 Last Checks...

Before running your study, make sure that you have the right
number of cells for each operator and sample

To do this we use Cross Tabulation

82
Cross Tabulation

83
 Table Statistics

The table should contain equal counts for all operators and
all parts before continuing

Tabulated statistics: Trial, Operator

Rows: Trial Columns: Operator

Fred Joe Mary All We look for equal

cell counts in all
1 10 10 10 30 combinations
2 10 10 10 30
3 10 10 10 30
All 30 30 30 90

Cell Contents: Count

84
 Mathematical Terms

GRR – Repeatability and Reproducibility

EV – Equipment Variation

AV – Appraiser Variation

PV – Part Variation

85
 Mathematical Relationships

Gage R&R
– Total variation of measurement system, GRR = EV 2 + AV 2
equipment and appraiser

Total Variation
– Total variation of study
TV = GRR 2 + PV 2

86
 Mathematical Relationships

% of Equipment Variation
– Ratio of gage and fixturing variation [
% EV = 100 * EV
TV
]
compared to total variation

% of Appraiser Variation
– Ratio of appraiser variation compared to
total variation
% AV = 100 * AV[ TV
]

% Gage Variation
– Ratio of measurement process variation
compared to total study variation
[
%GRR = 100 * GRR
TV
]

% Part Variation
– Ratio of variation of the parts compared
to total study variation
[
% PV = 100 * PV
TV
]

87