Advanced Quality Auditing

An auditor’s review of
risk based thinking, lean auditing
and data analysis
 Lance B. Coleman
 ASQ Senior Member, CQE, CSSGB, CQA, CBA
 ASQ Lean Enterprise Division Chair-elect
 ASQ Instructor for CQA Exam Preparatory Course
 Exemplar Global Principal QMS Auditor
 Chair US TAG 302
 Voting Member US TAG 176
 AAS EET, Southern Polytechnical University
 Author, Advanced Quality Auditing: An Auditor’s Review of Risk
Management, Lean Auditing and Data Analysis (Quality Press 2015)
 Author, The Customer Driven Organization:
Using the Kano Model (Productivity Press, 2014)
The Three Pillars of Auditing

The audit program can be thought of as a

platform that rests on the three pillars of
 Assessing compliance
 Assessing the effectiveness of how risk is
managed
 Helping to drive continuous improvement
An audit program needs all three columns for
that program to be truly robust.
Why consider risk based quality auditing?

 Audits become repetitive, mundane

 If QMS is working well, no major NCs
 Management begins wondering about the value
and becomes complacent
 Financial auditors have been doing risk-based
audits since Sarbox, COSO, etc. were issued
Some Real World Drivers to Change Focus to
Risk Management

• Enron, Tyco, WorldCom, …

• 9/11/2001, 26/11/2008, …
• Lehman Brothers, AIG, …
• Madoff, Stanford, …
• Dog food, peanut butter, spinach, …
• Firestone tires, Toyota acceleration, …
• Vioxx, Celebrex, …
• Heart valves, stents, …
• Bird flu, ebola, zika virus…
Defining Risk
 Risk – combination of the probability of occurrence of
harm and the severity of that harm.
ISO 14971:2007
 Risk – the effect of uncertainty on objectives
ISO 31000:2009
 Risk – the effect of uncertainty
ISO 9001: 2015
Related Terms
Risk Management – Overall process for identifying,
classifying, assessing, mitigating and monitoring risk

Risk assessment –Overall process comprising a

risk analysis and risk evaluation to determine the
type of risk and how potentially bad it is

Risk mitigation – Those measures taken to prevent

the occurrence of a hazardous event or failing that,
minimizing the impact of the event.
Residual risk – Risk remaining after risk
mitigation measures have been taken.
Identifying Risk

Where does risk lie in our process?

1. Complexity of the process
2. Complexity of the product
3. Criticality of the product
4. Process location of the product
5. Newness of the product
6. Newness of employees
7. History of the process
Managing Risk
How do auditors help manage risk?
1. Audit frequency
2. Sample size
3. Complexity of audit plan
4. Containment
5. Corrective action plans
6. Effectiveness verification
7. Special training
8. Assessing the risk management program
Managing Risk
How do we formally integrate risk based thinking into
the audit program?
 By tying finding classification to risk
assessment
 Let finding risk dictate actions taken
 Let finding risk dictate report distribution
 Establish feedback loop to risk management
program
Risk Based Thinking

 Definition is unclear in ISO 9001:2015

 We do it naturally all the time

 Needs formalization
Risk Based Thinking

Risk-based thinking enables an organization to determine the

factors that could cause its processes and its quality
management system to deviate from the planned results, to
put in place preventive controls to minimize negative effects
and to make maximum use of opportunities as they arise
ISO 9001: 2015 – 0.1 General
Risk Based Thinking

A process can be thought of as an activity that

transforms inputs into outputs, in this case
perceptions

PROCESS INPUTS Risk RISK MANAGEMENT

Based
• Inspection data • Risk Model
• Audit Findings Thinking • Risk Management Plan
• Management Review • Reporting Structure
• Test data • Feedback Loops
• Continuous
Improvement
• Operator feedback
• Gembawalks
• And so forth…
Applying Risk Based Thinking to Action Planning

Noncompliance Count
maintenance 1
operator error 5
complaints 1
product failure 3
records 4
calibration 2
cGDP 8
training 1
faulty documents 3
Applying Risk Based Thinking to Action Planning

Noncompliance Count
maintenance 1
operator error 5
complaints 1
product failure 3
records 4
calibration 2
cGDP 8
training 1
faulty documents 3
Applying Risk Based Thinking to Action Planning

Noncompliance Count Impact Risk Ranking

maintenance 1 2 2
operator error 5 3 15
Complaints 1 3 3
product failure 3 4 12
records 4 2 8
calibration 2 2 4
cGDP 8 1 8
training 1 2 2
faulty documents 3 2 6
Applying Risk Based Thinking to Audit Findings

Negligible Minor Major Critical

Impact
(1) (2) (3) (4)
Improperly Violation of Violation of customer Noncompliance
completed forms internal procedure requirement or that is itself a
and records or work internal requirement. hazard or may
(Information still instruction; Systemic or chronic lead to hazardous
retrievable) Current practice failure of QMS condition. Direct
QMS that meets requirement. Multiple violation of ISO
requirement is not related minor standards or
accurately violations. Cause great cGMP. Absense of
documented harm to other required procedure
operations in the or record
company.
Applying Risk Based Thinking to Audit Findings

Failure Likelihood Estimation Chart

Probability
Likelihood Definition
Rank
Very Low Unlikely to happen, rare,
1
remote
Low Can happen, but not
2
frequently
High Likely to happen, often,
3
frequent
Very High Very likely to happen,
4
more often than not
Applying Risk Based Thinking to Audit Findings

RISK
IMPACT
MATRIX Negligible Moderate Marginal Critical
L (1) (2) (3) (4)
i
Very Unlikely
k (1)
e
l Unlikely
i (2)
h Likely
o (3)
o Very Likely
d (4)

Low Risk Medium Risk High Risk

Applying Risk Based Thinking to Audit Findings

Risk Finding Classification

High Critical/Major
Medium Minor/Major
Low Minor
Risk Based Quality Auditing (RBQA) Types

 Directly audit the risk management (RM)

program itself
 Conduct RBQA of aspects of the QMS or
of the QMS as a whole
o Standalone risk management audit of
QMS processes
o Incorporate risk management into
existing audits
Auditing the RM Program

For a truly robust risk management program, the

following as a minimum should occur:
1.The program should encompass all aspects of a
product life cycle from design to end-of-life disposal.
2.Data from external as well as, internal sources
should be captured and analyzed and the risk
model updated as necessary.
3.Teams when formed, should be cross functional in
nature, in order to model the broadest range of risks
 Auditing the RM Program
1. First confirm all three of the items from the
previous slide are occurring.
2. Confirm that results from the risk management
program are reported as necessary to appropriate
levels of management.
3. Confirm that existing risk management procedures
and work instructions are followed.
4. Ensure that organizational training supports the
risk management program
5. Confirm that adequate resources are supplied to
meet the goals of the risk management program
Where does risk come from?

Assessment of risk, like other business concerns,

is open to interpretation. Risk can come from:
 Known and planned activities
 Unknown events and behavioral outliers
Risk is often an expected result of critical and/or
complex products and processes, as well as
changes (to product, processes, equipment,
people, …)
Risky Behavior

How do we know risky behavior or situations

when we see them?
 Variance from industry norms
 Employee concerns
 Established feedback channels
 Identified in risk management plan/risk
register
 Risky Behavior

Sometimes you will just know it when you see it…

Conducting RBQA of the QMS

1. Identify area hazards and risks

2. Classify hazards and risks

3. Are risks accounted for in the risk model?

4. Are controls/mitigations in place?

5. Are controls/mitigations adequate?

Risk is the CompassTM Model1

 Enablers – Activities or controls such as clear

work instructions, operator training program,
calibrated/maintained equipment instituted in
order to insure that a process is carried out
properly.
 Risk – those hazards or conditions such as
newness of the product, complexity of the
process, and poor lighting that work against
successful implementation of a process

1 – Developed in 2005 by Denis Devos of Devos & Associates

Risk is the CompassTM Model1
Input: Enablers Risks
Department
Needs a
Staff-member
Requisition Form #123, Job Requirements may be vague
Manager completes a Requires a complete justification
personnel requisition and full job-description Timing may be too short to react

Requisition is Business Plan projects No forecast for new hiring

reviewed/approved by Staffing needs: budgets in place
Human Resources HR reviews the Requisition Skill may be in high demand
for completeness

HR posts the job Job Posting Boards Not reaching the right target group
opening and recruits Succession Plans
outside the company HR staff has experience Outside recruiters not fully
and a network of contacts understanding our needs

Resumes are received Hiring Procedure 18-01 Candidates do not always tell the truth
and Reviewed, Skill of the HR interviewer
Interviews Conducted Dept. Manager participates Experience may not be applicable to
HR knowledge of governing this job
employment law HR interviewer may not understand
the technical requirements of the job
Offer made and
candidate is hired

1 – Developed in 2005 by Denis Devos of Devos & Associates

Risk is the CompassTM Model1

Offer made and

candidate is hired
Orientation & training
During probationary
period
Orientation Package #18-02
Employee Handbook #18-03
Training Procedure #18-04
Training Procedure #18-04
Job Description
Employee Training Plan
Employee is a poor “fit”
Employee doesn’t perform duties the
“company way”
Employee may not have all the skills
that he/she said
Manager may not conduct training

Performance Evaluation Lack of objective criteria or standard

Evaluation and Form #18-abc method for evaluation
acceptance
S as full- Skill of Department Manager Training ineffective
time employee HR Follow-up to ensure the Employee not treated fairly
evaluation is completed Evaluation may be late: unsuitable
Output: employee not fairly dealt with
Fully trained,
Effective, productive employee

1 – Developed in 2005 by Denis Devos of Devos & Associates

ISO 9001: 2015

4.4 The organization shall determine the

processes needed for the quality
management system and their application
throughout the organization, and shall:
e) Address the risks and opportunities…
Risks and Opportunities

“Not all change is improvement but all

improvement is change”
Chuck Anger

 Opportunities for improvement and growth

come with change.

 With change comes risk that must be identified

and mitigated.
Lean & Risk Management – The Connection

 By eliminating wastes Lean allows risks to be

more visible.

 It is important to understand that risks can

come from improvements as well.
Adding Value Through Auditing

The audit program provides value to the

organization by
 Assuring that corrective and preventive
actions are carried out effectively in response
to audit findings
 By identifying opportunities for improvement
(OFI) during audits
 By identifying process waste that can be
eliminated
 Waste

waste n. Anything that takes time,

resources or space but does not add to the value
of the product or service delivered to the
customer.
Value

value n. Always taken from the perspective of

the customer. Value is that which the customer
would be willing to pay for. For a process to have
value the following three conditions must be true.
 The customer is willing to pay for it
 A transformation must take place
 Must be done right the first time
THE 8 WASTES = D.O.W.N.T.I.M.E.
Defects
Over production
Waiting
Non-utilization
Transportation
Inventory
Motion
Excess processing
Improvement

Three opportunities to incorporate Lean Six

Sigma tools into the audit program
• Value Stream Mapping
• SIPOC Diagrams
• Control Charting
Improvement
Value Stream Mapping
• Value add vs non-value add activities
• Potential process bottlenecks
• Opportunities to match cycle to takt time
SIPOC Diagrams
• Identify implicit needs
• Identify input value streams
Control Charts
• Trending
• Training
• Equipment Discrimination
Traditional audit practice
Auditing using Value Stream Map
SIPOC Diagram

SIPOC
Process Name: SIPOC Date:
Process Owner: SIPOC Rev.:

Facilitated By: Notes:

Input Provided By:

Suppliers Inputs Process Outputs Customers

(Providers of the (Resources required by (Top level description of the activity) (Deliverables from the (Stakeholders who place the requirements on the
required resources) the process) process) outputs)
Requirements Requirements

Inputs

Process

Output
 EDART SIPOC Diagram
Suppliers
Providers of the required
inputs / resources to ensure
the process executes as
planned.
Materials Group
Inputs Process Outputs Customers
Resources required Top level description of activity. Deliverables from the Any organization that receives an output
by the process to process. or deliverable from the process.
obtain the intended Note: Deliverables can Note: Can also capture systems/databases
output. be hardware, software, that receive outputs, information, data, etc.
systems, services, data,
information, etc.
Requirements Requirements
Molded Parts Accepted parts
Material Visual attribute specs- Press Operator
MQC

Operations/Maintenance Press Dimensional specs – Production

MQC Inspector
Visual attribute specs-
MQC
Engineering Press LNH
Process graphs
Production/Engineering Labor Per settings Engineering
Rejected parts
Quality Instructions None Recycling
We are not analyzing
parts to see if they
should be rejected.
Quality Specifications

EDART Settings IQMS scrap data None Site

Management
Engineering Per part EDART What should our
profile maximum allowable
(How developed?) EDART scrap be?
Is the scrap data that I
am receiving valid?
Audit Checklist Development Matrix1
SYSTEM EVALUATION (VALIDATION) CONTINUOUS IMPROVEMENT

PERFORMANCE VERIFICATION

Reference Reference Performance Statements

Positive Practices
Document Standard Standard Observations Records Lean Auditing
Opportunities
Paragraph Requirement Paragraph(s) Interviews
1. Government 1. State the requirement 1. What is the requirement? 1. Do we do what we said 1. Are required records 1. Any positive practices 1. Is the process yield as
regulation 2. Any related findings 2. What do our documents say? we would? available? that may be transferred expected?
2. ISO Standard from most recent 3. Do required documents 2. Are employees 2. Are records complete and to other operations? 2. How does the process
3. Contract audit? exist? effectively trained? accurate? 2. Any opportunities for yield compare to similar
4. Purchase 3. Working environment 3. Are records consistently improvement? processes?
Order safe and clean? filled out? 3. Any identified wastes?
4. Work environment 4. Are good documentation 4. Any non-value-add
suitable for operations? practices followed? steps?
5. Are adequate resources 5. Can cycle time be
provided to achieve reduced?
goals?
A B C D E F G

1 - Auditing Beyond Compliance By Janet Bautista Smith

Visual Workplace
Visual (Risk) Management
Visual Device

A visual device is an apparatus, mechanism, item

or thing that influences, directs, limits or controls
behavior by making information vital to the task-
at-hand available-at-a-glance, without speaking a
word
Spotting Adverse Trends

AA BB CC
88 10
10 12
12
99 11
11 13
13
55 77 99
66 88 10
10
10
10 10
10 10
10
11
11 11
11 11
11
77 77 77
88 88 88
12
12 10
10 88
13
13 11
11 99
99 77 55
10
10 88 66
Responding to trends

$20,000
+ scrap
ISO 9001: 2015

9.1.3 Analysis and evaluation

The organization shall analyse and evaluate

appropriate data and information arising from
monitoring and measurement…

..NOTE Methods to analyse data can include

statistical techniques.
 Responding to trends

1. One data point is an incident – should be

verified and noted
2. Two data points may be a coincidence – verify
and document
3. Three data points starts to look like a trend
Trends of three data points in a row or even 3 out
of 4 data points in one direction or another should
be reported.
Spotting Adverse Events

Data Shift
16
14
12
10
8
6
4
2
0
1 2 3 4 5 6 7 8 9 10 11 12

Data UCL LCL Nominal

Data Spread
16
14
12
10
8
6
4
2
0
1 2 3 4 5 6 7 8 9 10 11 12

Data UCL LCL Nominal

Spotting Adverse Events
 Auditing process monitoring data review

 What do you do when an adverse trend is

encountered?
 What do you do when an out of control condition is
encountered?
 How do you know what to do when an out of control
condition or adverse trend is encountered?
 Is there a structured program in place for review of
and response to adverse trends/conditions in data?
 What training is provided in SPC and data
analysis?
Validation report review

Is the data that was specified in the protocol in

the report?
Have all of the required signatories signed off?
Have all the success criteria been met?
If all of the success criteria has not been met,
were appropriate procedures followed?
Any red lines crossed on the graphs?
Key Thoughts Recap

1. Risk based thinking occurs when we filter the data

that we encounter throughout the workday through
the lens of risk – impact, likelihood, detectability and
vulnerability and respond accordingly.
2. The three pillars of a robust audit program are
compliance, improvement and risk management.
3. Graphical depictions of data are much more easily
interpreted and have a more powerful impact in
presentations
4. Lean helps manage risk by eliminating wastes that
make environmental hazards easier to spot.
5. It is important to understand that improvement
activities can contain risks that should be identified
and mitigated
Key Thoughts Recap

1. Risk based thinking occurs when we filter the data

that we encounter throughout the workday through
the lens of risk – impact, likelihood, detectability and
vulnerability and respond accordingly.
2. The three pillars of a robust audit program are
compliance, improvement and risk management.
3. Graphical depictions of data are much more easily
interpreted and have a more powerful impact in
presentations
4. Lean helps manage risk by eliminating wastes that
make environmental hazards easier to spot.
5. It is important to understand that improvement
activities can contain risks that should be identified
and mitigated
Concluding Thought

Don’t let the boundaries of your potential

be framed by the limits of your knowledge
Lance B. Coleman WCQI 2016
References

Advanced Quality Auditing

ISBN: 978-0-87389-913-0
Quality Press 2015

Auditing Beyond Compliance

Quality Press 2012
ISBN-13: 978-0-87389-840-9

Performance Metrics:
The Levers for Process Management
Quality Press 2013
ISBN-13: 978-0873898508
 To learn more…

About Lance B. Coleman

• lance@fullmoonconsulting.net
• www.fullmoonconsulting.net
• www.asq.org/quality-press/

About the topic:

• Learn About Quality
www.asq.org/learn-about-
quality/
• Knowledge Center @
www.asq.org/knowledge-center/