NCSL International Workshop & Symposium |Metrology in Motion

August 26-29, 2019 | Cleveland, Ohio

Managing Calibration Risk for Medical Devices

Speaker/Author: Harry C Spinks

Company: TechTrology LLC
6801 15th Street CT N, Oakdale, MN 55128 USA
Phone: 651-252-1147 Email: Harry@techtrology.com

Abstract
Calibration is a complex subject and requires training and experience to understand it. This paper
provided and introduction to some of the risks associated with calibration. It mostly covered
risks to the management of calibration. There is risk associated with every measurement
performed, which is why your organization needs expertise in metrology (measurement science)
and calibration.

1. Introduction
Calibration is important to manufacturing and in some cases it is critical.
In the case of medical devices calibration can be critical to the life of the patient.
ISO 13485 is the quality management standard for medical device organizations. And for many
medical device manufacturers, they also need to meet federal regulations such as 21 CFR Part
820, which is a law and is legally enforceable. ISO 13485 is voluntary and not legally
enforceable.

The 2016 version of ISO 13485 states to apply a risk based approach to the control of the
appropriate processes needed for the quality management system.

Calibration is one of those processes and is referred to in section 7.6 control of measuring and
monitoring equipment.
ISO 13485:2017 7.6 requires monitoring and measuring equipment to provide evidence of
conformity of product to determined specifications.

What is Calibration?
The simple definition is comparing an unknown to a known. A device being calibrated is
compared to a more accurate calibration standard. The calibration standard is compared to a
more accurate calibration standard and so on, until you get to your country’s national
measurement institute (NMI).

1
 NCSL International Workshop & Symposium |Metrology in Motion
August 26-29, 2019 | Cleveland, Ohio

2. Risks
Risk#1 – Conformity to specifications
How do we know that equipment is capable of determining conformity of product to
specifications?
Step 1, we need to know the product specs which usually are found on the product print or
drawing. Not all specs are on a print. If it’s an electrical parameter the specs may be in an
engineering specification document. In any case, we’ll call the document containing the specs a
print.
Step 2, identify the equipment type necessary to meet the measurement specifications. Electrical
(DC, AC, etc.), pressure, temperature, etc.
Step 3, how accurate does the measurement need to be to ensure product conformance? +/-
0.001, 0.01, 1, etc. NOTE: resolution of the measuring device is not accuracy.
Step 4, identify the specific piece of equipment needed to meet the measurement specifications.
Signal analyzer, micrometer, vernier, CMM, multimeter, etc.
Step 5, determine if the organization has the equipment on-hand. If not, do they have a suitable
substitute? Check the equipment list from the calibration provider. Consult with the calibration
provider whether they are internal or external.
Step 6, if the organization doesn’t have the necessary equipment, begin the process to procure
the equipment. Consult with your calibration provider.
Step 7, determine how to measure the parameter. See your metrology engineer or Metrologist.
Step 8, determine how to ensure the measurement device is accurate (calibrated or verified) and
traceable to national standards.
This seems like a simple process, but it isn’t. There is risk to product associated with each of
these steps. The second paragraph is section 7.6 requires that the organization document the
process.
This process requires the coordination of the design engineering, manufacturing engineering,
quality engineering, equipment engineering, and metrology/calibration. Your organization may
not have all of these groups, but whichever groups are in your organization, they should all be
working together. They cannot be involved sequentially or significant process risk will occur.
When you outsource calibration, the service provider needs to know the requirements for every
piece of equipment they calibrate. This would require you to know what specs you need and
convey them to the supplier. Do not rely on “manufacturer’s specifications”.

2
 NCSL International Workshop & Symposium |Metrology in Motion
August 26-29, 2019 | Cleveland, Ohio

I repeat do not rely on calibration to “manufacturer’s specifications.” You assume that when the
supplier calibrates your equipment they are doing it to the manufacturer’s specification. They
may state that it meets manufacturer’s specs, but it might not meet yours, especially in medical
device companies.
Why? How could this be possible if you’ve requested them to meet the OEM specs, the ones
you are looking at?
Here’s one example. You need the device calibrated from 0 to 100 volts. The calibration
provider (supplier) calibrates the device at 90 volts (for whatever reason). It happens. The FDA
looks at your cal cert and sees that it’s only calibrated to 90 volts and you use it at 98 volts. You
will receive a finding for this. The evidence is that the device is not traceable because the full
range was not calibrated.
This is a favorite of auditors because it’s easy to detect. The supplier will argue that if the device
is good at 90, then it’s good at 100. And they are probably correct. They based their cal on the
design of the device, not on what is stated in the manufacturer’s specs.
There are many examples of this occurring. To minimize this risk, you need to review a
calibration certificate for this device prior to using this supplier. If you have hundreds of
different pieces of equipment, it can be a very time consuming effort.
You should have a documented process and forms associated with the specifications for the
equipment. And a process for communicating those requirements to the supplier, without
providing them with product specifications.
Of course, you can state that it you’ve determined that the supplier’s method of calibration is not
a significant risk and that they were selected on the basis of their scope of accreditation to ISO
17025.

Risk# 2, Out of Tolerance

section 7.6 – “In addition, the organization shall assess and record the validity of the previous
measuring results when the equipment is found not to conform to requirements. The
organization shall take appropriate action in regard to the equipment and any product affected.”
Step 1, When measuring equipment is found out of tolerance (OOT), it fails calibration, then
every measurement that equipment made (for the associated parameter) since the last calibration
shall be reviewed to determine if the OOT affected product and to what degree. This assessment
and the results shall be recorded. This is commonly known as a nonconformance (NC) or
nonconforming event.
If the equipment was only used to measure one parameter on one product, then there is only one
measurement to evaluate. If it was used to measure 10 parameters on 10 different products, then
there are 100 measurements to evaluate.

3
 NCSL International Workshop & Symposium |Metrology in Motion
August 26-29, 2019 | Cleveland, Ohio

The OOT is reported to the product or process owner (e.g. manufacturing engineering, quality
engineering, etc. – dependent on the organization’s procedures). The process owner evaluates the
impact to product and takes appropriate action. Typically, the department responsible for
calibration does not have the training or expertise to evaluate the product impact.
Note: There is not a requirement in ISO 13485 or Part 820 for the calibration department to track
the nonconformance. Their requirement is to report the OOT to the process owner and verify
that the process owner received the notification. They may record the NC tracking number in
their calibration system as evidence that the OOT was reported.
Step 2, “the organization shall take appropriate action in regard to to the equipment affected.”
The metrology/calibration department or equivalent will need to evaluate the OOT to determine
why it occurred (if possible) and if it is preventable.
The equipment may have drifted out of tolerance over time. This can be determined by
reviewing previous calibration data to see if there is a trend. If so, then other pieces of this
model of equipment should have their calibration data reviewed to see if it is trending toward the
specification limited.
Or, the equipment may have shifted significantly and suddenly due to a traumatic event such as
being dropped or over-ranged. Over-ranged is when an input is applied that isabove the range of
the equipment. For example, on the 10 volt range, 50 volts is applied. Or on a 1kg balance, 10
kg is applied.
More significantly, if the equipment that was OOT is used to calibrate other equipment instead of
directly measuring product parameters, then every piece of equipment that was calibrated by the
OOT equipment will need to be evaluated to see if it was significant enough to affect them. This
is known as reverse traceability. In addition, if the OOT was significant to the calibrated
equipment then any and all product measured with that equipment will need to be evaluated (an
NC to the product/process owner).
For example, if a meter calibrator is found to be OOT in 10 volt DC range, then every device that
was calibrated (on the corresponding volts DC range) with the OOT meter calibrator will need to
be evaluated.
There may be other actions taken in regard to the equipment. This could include guard-banding
the specifications of the equipment. This would flag the equipment as needing adjustment or
repair if it drifted but hasn’t gone out of tolerance yet. Explaining guard banding is beyond the
scope of this paper.
It may be determined that the equipment should be replaced with newer, more accurate or stable
equipment.

4
 NCSL International Workshop & Symposium |Metrology in Motion
August 26-29, 2019 | Cleveland, Ohio

Risk# 3, Overdue for calibration.

This is one of the easiest risks to prevent, yet it one of the most frequent audit findings associated
with calibration. The reason for this is that it is the easiest requirement for an auditor to detect, so
they are always looking for it.
Section 7.6 “As necessary to ensure valid results, measuring equipment shall: (c) have
identification to determine its calibration status.”
This requirement is interpreted to mean that all calibrated measuring equipment will have a
“calibration label” attached to it. That’s not what the standard requires, but that’s the way it’s
interpreted.
Not all equipment can be labeled (weights, embedded measurement devices, etc.), in which case
the status needs to be readily available. This could mean that the label it on the case the item is
stored in or that there is on an equipment list nearby that lists the equipment and it’s status.
Typically, status refers to it’s calibration due date. If the date due has passed then the equipment
is considered “out of calibration” and should be removed from service.
It should be calibrated as soon as possible to determine if it is in or out of tolerance. If it’s in
tolerance no action is required pertaining to the possibility of affected product. If it’s out of
tolerance then the nonconforming event process needs to be activated.
As I said, this is the easiest to detect. This occurs more often when a month/day/year calibration
due date is used. If the equipment is due on May 10th, 2019 and it’s May 15th, the equipment is
overdue.
One method to minimize this risk is to use a month/year calibration due date on the calibration
label, commonly referred to as the “cal label” or “sticker.” Using this method the equipment isn’t
overdue until the first day of the following month. If the equipment is due May 2019 (05/2019),
then it isn’t overdue until June 1st, 2019 (06/2019).
Some organizations want the equipment to be due on a specific date for scheduling purposes. In
that case, the month/day/year will be on the label. But, you can modify your policy document to
state that it isn’t overdue until the first day of the following month. The time between the due
date and the first of the following month is often referred to as a “grace period.” If that’s what
your policy states, then the auditor cannot issue a finding. Your organization has assessed the
risk associated with the grace period and has found it an acceptable risk.

Risk# 4, Traceability
Section 7.6 - “As necessary to ensure valid results, measuring equipment shall: a) be calibrated
or verified, or both at specified intervals, or prior to use, against measurement equipment
standards traceable to international or national standards; when no such standards exist, the basis
used for calibration or verification shall be recorded.”

5
 NCSL International Workshop & Symposium |Metrology in Motion
August 26-29, 2019 | Cleveland, Ohio

This is the second easiest requirement for an auditor to evaluate. The auditor records calibration
ID numbers and due dates from equipment calibration labels. They return to the calibration
department/coordinator and request to see the calibration records for the equipment. They check
the calibration certificate for the traceability statement, which in the US is “traceable to NIST.”
For example, the organization calibrated the process equipment with their own calibration
standards. The auditor will review the calibration certificate for the process equipment and write
down the standards used to calibrate it. Then they will look at the calibration certificate(s) for
the calibration standard(s) used. The calibration standards due date should have been “in cal”
(not overdue) when they were used. And they should indicate that they are traceable to NIST
(national or international standards).
What auditors should know is the definition of “traceability”. Just because the calibration
certificates states it is traceable to NIST doesn’t mean that it actually is traceable.
According to the VIM (JCGM200)
Metrological traceability: property of a measurement result whereby the result can be related
to a reference through a documented unbroken chain of calibrations, each contributing to
the measurement uncertainty
Refer to the VIM for more information on calibration and metrological traceability.
Each of the calibration standard’s certificates should have the measurement uncertainty stated or
a TUR (test uncertainty ratio) from an accredited calibration provider. An auditor may just look
for the traceable to NIST statement and be satisfied. While measurement uncertainty is required
for traceability, they don’t enforce it, probably because they are unaware of it or don’t
understand it.

Risk# 5, Measurement Uncertainty (MU)

VIM definition: non-negative parameter characterizing the dispersion of the quantity values
being attributed to a measurand (what’s being measured), based on the information used.
This definition is simple in it’s definition, but more complex in it’s application. For non-
metrology staff, you do not need to know everything about MU. But, you should know how to
recognize it and use it.
MU is determined by the process and the equipment used. Since the MU is typically 25% or less
of the accuracy, the lower the uncertainty the better. The measurement uncertainty of the
calibration provider’s process can be found in their scope of accreditation. And they need to be
accredited to ISO 17025 if you want accredited and traceable calibrations and uncertainties.
The calibration certificate for your equipment would have the uncertainty or Test Uncertainty
Ratio specified. Provided, you have contracted with your calibration provider for an accredited
calibration. Just because your provider is accredited doesn’t mean that they are providing you
with an accredited calibration. Many providers charge more for an accredited calibration so the

6
 NCSL International Workshop & Symposium |Metrology in Motion
August 26-29, 2019 | Cleveland, Ohio

medical device company requests data only in order to save money. And because they don’t
understand the need for MU.
Some calibration providers do not put the MU for each calibrated parameter, but give a statement
of compliance to a 4:1 TUR (test uncertainty ratio). We’re not going to dive into what a TUR is,
just know that 4:1 is okay. Not great, just okay. In many cases the provider has a TUR that’s
better than 4:1, but they don’t tell you what it is unless it’s less than 4:1. They may state the
TUR overall because specifying the uncertainty for each test point may be costly for them
because their calibration computer system doesn’t compute it for them.
3. Scope of Accreditation
Every accredited calibration provider has a scope of accreditation (scope). They may have it on
their website or it can be downloaded from the website of their accreditation body (A2LA,
ANAB, NAVLAP, PJA, etc.)
Evaluating the provider’s scope can be time consuming and you have to understand what
measurement parameters you need.
For instance, you need temperature devices calibrated. You review the scope of the prospective
providers and find that one of the providers does not have temperature on their scope. This
should disqualify this provider since they are not accredited in one of the areas you need.
Another provider has uncertainties for temperature that are 0.02 degrees F and another provider
is 0.2 degrees F. One provider is has uncertainties for temperature that are 10 times better than
the other. This doesn’t happen too often, but you see the difference.
You would need to review all the measurement parameters you need for each calibration
provider. The process used to select an external provider must be documented and you must
have records to show that the evaluation was performed. This is a requirement of ISO 13485.
And it’s one that every auditor evaluates. Not necessarily for calibration, but they do audit the
process and records.

4. References
VIM, JCGM 200:2012, https://www.bipm.org/en/publications/
Guides in Metrology section.