Regulated

Computerised
Systems

Risk
Assessment

Page 1/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
CONTENTS

1 PREFACE

2 GOAL
2.1
SCOPE
2.2
OBJECTIVES
2.3
REQUIREMENTS

4
4
5
5

3 RISK ASSESSMENT MODEL AND METHOD

3.1
TIMING
3.2
PLANNING PHASE
3.3
PERFORMING PHASE
3.4
REVIEWING PHASE
3.5
FOLLOW-UP PHASE

6
7
8
10
14
15

4 CONCLUSION

15

Page 2/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems

PREFACE

This White Paper describes the process followed by ps_testware during Risk
Assessments intended to target the validation effort for cGxP and Part 11 regulated
computers systems. Risk assessments identify the possible risks and threats posed by
a computer system to the business and to compliance with the rules and regulations
defined by regulatory organisations. The Risk Assessment identifies the risks that the
system poses to product safety, efficacy and quality, and the risks that it poses to
data integrity, authenticity and confidentiality. "Product" refers to human or veterinary
drugs, biological products, medical devices or radiological products.
Process validation, included system and software validation, is a requirement of the
current Good Practice (cGxP) regulations of the Food and Drug Administration (FDA)
and the European Union (EU). The Quality System regulation requires that when
computers or automated data processing systems are used as part of production or
the quality system, the manufacturer shall validate computer software for its intended
use according to an established protocol. For example cGMP for Finished
Pharmaceuticals (FDA, 21 CFR Part 210 and 211), cGMP for Medical Devices (FDA, 21
CFR Part 820) and GLP for Non-clinical Laboratory Studies (FDA, 21 CFR Part 58).
In addition to the above validation requirement, computer systems that implement
part of a manufacturers production processes or quality system (or that are used to
create and maintain records required by any other cGxP regulation) are subject to the
Electronic Records and Signatures regulation (FDA, 21 CFR Part 11). This regulation
establishes additional security, data integrity, and validation requirements when
records are created or maintained electronically. The additional Part 11 requirements
should be carefully considered and included in the requirements for any automated
record keeping system. System validation and software validation should demonstrate
that all Part 11 requirements have been met.
Even for commercial off-the-shelf software the FDA suggests that end users should
conduct functional testing of software that covers all functions of the program that the
end user will use. For both off-the-shelf software and custom-built software, risk
assessment is a tool that can justify limits being placed on the testing of systems and
functions. It is impractical to completely validate every aspect of a cGxP computer
system. Therefore the effort should be focused on critical areas such as risks to
regulatory compliance, the business, and project completion. By assessing the
processes and computer system for risks the effort can be focused on the most critical
parts of the system.
In the next chapter, the scope, objectives and the requirements of a risk assessment
are defined.
The third chapter explains the methodology and model that ps_testware uses to
conduct a risk assessment. The stages of a project at which a risk assessment should
be conducted, and the steps to be taken within each of the four phases of the risk
assessment are discussed.

Page 3/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems

The Risk Analysis document, that is the result of a risk assessment, will show
if a computer system requires validation;
what aspects of the computer system or process are critical to product and
patient safety;
what aspects are critical to business and;
what corrective actions and preventive measures are needed to realise a
computer system that meets the expectations of the regulatory organisations.

GOAL

A principal objective of risk assessments is to identify and evaluate the risk factors
involved in the development, implementation and enhancement of a computerised
system (CS) (e.g. computerising a manual process, the purchase and implementation
or enhancement of a cGxP computer system). Risk assessment will be used to
determine the scope and extent of the necessary validation processes and any
allowable exclusion from them, ensuring that the appropriate Software Development
Life Cycle (SDLC) phases, documentation, and level of testing are applied. Choosing
the right validation strategy depends on understanding the risks and their
consequences.

2.1 Scope
The scope of a risk assessment includes the factors that threaten compliance with the
regulations applicable to a cGxP computer system and those that may have a negative
business impact. The following topics are generally within the scope of a risk
assessment.
Business Processes (cGxP);
Electronic records, signatures and audit trails (21 CFR Part 11);
System Development and implementation methodology, including completion of
documentation to required standards;
System Configuration, System Security and Change Control;
Standard Operating Procedures (SOPs) for system operation, system
management, support, etc;
Protocols for Installation Qualification (IQ), Operational Qualification (OQ) and
Performance Qualification (PQ);
User training / coaching and training records;
General system issues such as system performance, maintenance, backup and
recovery, disaster recovery, data retention and network management.
This is not a limited list and will be adapted to the specific situation.
The risk assessments made prior to and during the validation of a system will normally
examine the above mentioned topics to determine if there is a potential impact on
compliance with the regulations or a potential impact on the business. Any threat
assessed as having impact on regulatory compliance will be within the scope of the
validation plan.

Page 4/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
2.2 Objectives
Because resources, time and money for validation are often limited, risk assessments
are conducted to identify how "much" validation is required for a computer system.
The risk assessment objectives that can be achieved and recorded in a risk analysis
document are:
Identification of which computer systems require validation;
Identification of the possible threats to cGxP and Part 11 compliance and to the
business;
Evaluation of the threats to see if they are applicable;
Estimation of the likelihood of the threat occurring and being detected;
Evaluation of the potential impact of each threat, should it occur.
After the identification of the possible threats, the likelihood of them occurring and
being detected, and their potential impact (regulatory and/or business), corrective
actions and preventative measures must be defined.
The risk analysis document can be a very significant means of defining and justifying
the approach taken to the validation of the system (in whole or in specific aspects of
it). It can also enable the Validation Process to be as efficient as possible in terms of
time and cost by optimising the effort involved.

2.3 Requirements
Conducting a risk assessment is a multi-disciplinary task. The responsibilities for the
risk assessment will be dependent on the organisational structure and the processes
that are being assessed. The senior person responsible for Quality Assurance should
be the owner of the Risk Assessment, the Risk Analysis document and the Validation
Process as a whole. Other key people in the organisation (such as the Business and
System Owners) should also be approvers of the Risk Analysis document. These
people have also great added value as information source. Examples of other
personnel who may have important input for the risk assessment are:

Users (Operations Personnel) can provide a lot of information on the technical

and operation processes and procedures. They are responsible for the
investigation and evaluation of the risks that are identified as a result of the
technical and operational implementation.

Software Application and Infrastructure specialists should be able to

provide a lot of information on software and hardware related issues. They are
responsible for the investigation and evaluation of those risks identified as a result
of software and hardware implementation and therefor important for the risk
assessment.

Developers and Software Vendor because they know the software

application, should be able to provide information on the used development of the

software release and test methodology and can identify possible gaps, risks, etc.

Customers can identify important requirements (business) for the process and
are therefor a possible source for the risk assessment.

Page 5/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
The above is not a complete list of the personnel who may have important
information for the risk assessment or who could be members of the risk assessment
team.
These personnel will normally have important non-validation roles to play in one or
more aspects of the SDLC. They are also likely to be responsible for important
documentation in the overall validation process. Examples of that documentation are:
Business or Process Model (BM/PM);
User Requirements Specification (URS);
Functional Specifications (FS);
Design Specifications (DS);
Test Plans, Cases or Data;
Standard Operation Procedures (SOPs).
That documentation, if already available and approved, is of major importance and is
a major input to the risk assessment process. If the documentation does not exist or
is not completed to the required standards that in itself represents a risk to regulatory
compliance.

RISK ASSESSMENT MODEL AND METHOD

The intention of this chapter is to explain the assessment model and methodology
used by ps_testware. The risk assessment methodology can be divided into four
phases: Planning phase, Performing phase, Reviewing phase and Follow-up phase.

Planning # CS

Performing CSn
Performing CS3
Performing CS2
Performing CS1
Review CSn
Reviewing
Reviewing CSC2S3
Reviewing CS1

Follow-up

- V(M)P
- Actio
ns
on

Page 6/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
Figure 1: The ps_testware Risk Assessment model
Figure 1 shows the ps_testware Risk Assessment model. This methodology consists of
4 essential layers:

Planning phase: The goals of the Planning phase are to define the scope of the

risk assessment and to identify the resources and the time required. The output is
a prioritised list of systems to assess, defined risk factors, an established risk
assessment team and allocated time, resources and material.

Performing phase: The goal of the Performing phase is to conduct the system

specific risk assessments. The output is a draft risk analysis document with a list
of (possible) regulatory and business risks presented by specific computer
systems including recommended corrective actions and preventative measures.

Reviewing phase: The goals of the Reviewing phase are the review and

approval of the draft risk analysis document. The output is a risk analysis
document that can be published and circulated, and be used to justify to
inspectors the decisions taken concerning the scope and the depth of validation
activities.

Follow-up phase: The Follow-up phase consists of the development and

implementation of a plan to deal with the identified risks, threats and defined
actions. The output is a Follow-up Plan that is incorporated in the next issue of
the Validation Plan, or in a Test Plan. If the risk assessment is part of the total
validation process, the Follow-up phase can be a basis for creating or amending
the validation plan, adapting the validation activities or analysing the risks while
executing validation activities.

These 4 phases are explained in more detail in the sections 3.2 to 3.5, but first the
timing and number of risk assessments necessary is discussed.

3.1 Timing
Because the validation process should start at the beginning of the SDLC, the risk and
threats to non-compliance and priorities are likely to change throughout the project.
Therefore the risk assessment should be conducted at several stages of the project.
The number and timing of risk assessments should be documented in a validation
(master) plan. Figure 2 shows examples of good moments to conduct risk
assessments (RA).
The best time to conduct a first risk assessment is during the completion of the
Business Model and User Requirements. Other moments are after a Supplier
Assessment (Audit), after completion of Functional Specifications, after the test phase
and when the system undergoes major changes.
An efficient way of working while conducting a risk assessment is to combine it with a
first review of the existing documents (Design Qualification (DQ) or Design Review).
This involves planned and systematic documented reviews of documentation
throughout the SDLC. These reviews validate the deliverables for constancy and
correctness against standards, identify (possible) problems and propose the necessary
corrective actions.

Page 7/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
Validation
report

URS

RA

RA

PQ protocol
and results

Testing of the URS

Validation
Plan/Strategy

Testing of the FS

FS

OQ protocol
and results

RA
Hardware DS

Testing of Hardware specs

IQ protocol
and results

RA
Integration
testing

Software DS

Software
Module Spec

Module
testing

RA
Implementation

(change
control)

Building

Project Planning

Figure 2: V-model for validation and risk assessments

3.2 Planning phase

The first step of the methodology is to identify the scope of the risk assessment. This
phase emphasises organisational considerations rather then technical implications.
There are 7 steps in the Planning phase. The steps are explained in detail below. The
first three steps consider the importance of an approved and documented assessment
process.
1. Identify the relevant written approved procedures for conducting risk
assessments on computer systems. If they do not exist, they have to be written
and approved before the Risk Assessment is started. This chapter contains the
basic information necessary to develop written procedures concerning risk
assessments.
2. Identify the risk/threat factors, define the threat criteria levels (for likelihood
and impact) and determine the threat categories (for priority). Examples of
criteria levels and categories are discussed in step 4, 5, 6 and 7 of the Performing

Page 8/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
phase. The risk factors are associated with (the possibility of) a failing computer
system and may include the regulatory impact (cGxP), safety concerns, business
concerns (costs, time and human resources), and other factors (see examples
below). The risk factors should be described as fully as possible to allow correct
assessment of the threat criteria levels.
Regulatory Impact includes integrity of regulated data, security, and product
quality focus. Consider the current regulatory expectation for validating such a
system. If the system impacts regulated data, or is used to assist in making regulatory
decisions, computer system validation is a regulatory requirement. Business
Concerns include company reliance on the system, data for decision making, the
establishment of contingency plans, and protection of assets. Safety Concerns
include consumer safety and environmental hazards. Other factors to consider
include:

Complexity of hardware, application, and system software configuration

management;

Detail of the change control documentation;

In-house manpower (number and skill set);
New and pending regulations, and related regulatory guidelines (for example 21
CFR Part 11).

3. After creating and approving the necessary procedures, and defining the risk
factors, the risk assessment process will be defined in an approved document,
such as a validation master plan. One of the biggest risks an organisation can take
is failing to define their approach to risk assessment and validation. Sometimes a
risk assessment is conducted as a basis for the (draft) validation master plan and
so it cannot be defined in an approved Validation Master Plan. Then the risk
assessment will be part of a policy statement or quality assurance. The number
and timing of further risk assessments will be documented in the validation plan.
If there are several operational computer systems in the organisation, the following
steps will be followed to identify regulated computer systems and rank them by
priority for the Performing phase.
4. An inventory of all computer systems utilised by the organisation will be
generated and maintained. For each computer system the available participants
and documentation that can be of use for system specific risk assessment in the
Performing phase will be identified.
5. From the inventory list created and maintained as defined in step 4, systems that
require validation based on the regulatory impact and business risks will be
identified. Here a checklist is used with 15 question that can be answered with
Yes or No. If one of the questions is answered with Yes the computer
system should be validated and a system specific risk assessment should be
conducted. This checklist is not the only method of doing this, but it does give a
good first indication. For each computer system that requires validation,
prospective validation or retrospective evaluation needs are identified and the
current validation status is recorded on the inventory list.

Page 9/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
6. All the systems in the inventory list will be categorised. To categorise the systems,
the GAMP Categories are used. The GAMP Categorisation is a tool for making
some broad assumptions regarding the risk levels based on the nature of the
system and offers a high level starting point for determining the validation strategy
for the specific computer system. GAMP uses 5 categories, namely: Operating
Systems, Firmware, Standard Packages, Configurable Software and Custom
Software. Each category has his own kind of validation strategy. Reasonable
complex systems are likely to be comprised of several components of different
categories.
7. For those systems with regulatory impact, a (numerical) exercise can be conducted
to prioritise the system specific risk assessment and further validation activities.
This prioritisation takes place on the outcome of step 5 and 6, but depends on the
number of involved computer systems and the organisation. Important aspects to
consider for prioritising are, for example, system criticality to the products or
services, the business and patient; industry distribution of the software (custom
made, for sector, for many industries or off-the-shelf software); regulatory impact
(number of Yes on checklist), and the vulnerability at downtime. It is important,
for future inspections, to document the assessment process and the prioritisation
sequence that will be followed, including justifications for decisions stating that
systems do not require validation.
From the prioritised list of regulated computer systems that must be assessed, the
participants of a risk assessment team can be established and time, resources and
material can be allocated. The results of the risk assessment prioritisation can be
reflected in the validation master plan.

3.3 Performing phase

After defining and approval of the risk assessment methodology and the identification,
and prioritisation of the computer systems that must be validated, the risk
assessments can be performed for each system.
The risk assessment for systems identified as having regulatory impact will result in a
draft risk analysis document with a list of regulatory and business risks / threats posed
by the system, and the associated recommended actions. Though it is possible to
create one risk analysis document for all identified systems, one risk analysis
document per system is preferable.
The 9 steps required to perform a system risk assessment are stated below. These
steps have to be performed for each computer system that has to be validated.
1. The first step is to identify the business processes that the computer system
should support. The base for this process identification will be mainly the User
Requirements Specification and a Business or Process Model. Other documentation
such as Functional Specifications or Design Specifications, and SOPs, if available,
can be used. These documents will be used to identify the system functions and
their sub-functions, including any dependencies between them.

Page 10/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
On the completion of the first step, the identified business processes and functions
can be assessed against the risk factors determined in step 2 of the Planning phase.
Only the risk factors for the regulatory and business threats are assessed.
2. Identifying threats presented by cGxP functions and Part 11 criteria is the next
step. Checklists are used for cGxP and Part 11 requirements. The risks and threats
that may be identified are concerned with the quality and safety of the product,
and incorrect data (e.g. for the support of regulatory submissions, Part 11). Each
identified process or function in step 1 is looked at and an assessment made on
their cGxP and Part 11 impact. The outcome of the assessment of each function
will be stated in the risk analysis document. If the assessment determines that
several processes and functions have no cGxP or Part 11 impact than the
justification for this outcome will also be stated in the risk analysis document.
3. The third step is to identify threats to business criteria. Risks and threats that may
be concerned with incorrect or incomplete data for decision making, operation, and
administration, the business's reputation, poor yields in manufacturing, late
deliveries, late payments to suppliers etc. Each process or function identified in
step 1 is reviewed and an assessment is made of the business risks. The outcome
of the assessment of each function will be stated in the risk analysis document.
After the identification of the possible threats (regulatory and/or business) their
potential impact is assessed, and the likelihood of them occurring is evaluated. These
threat criteria levels are defined in step 2 of the Planning phase in this document.
Corrective and/or preventative measures that are necessary to address threats to
compliance are stated, for later inclusion in the validation plan.
4. The likelihood of the risks occurring are determined. The likelihood of a risk
occurring ("likelihood criteria") can be classified, for example, as one of:
High, the threat probably will occur;
Medium, some occurrences are expected;
Low, unlikely to occur;
Negligible, eliminated through previous actions or procedures.
Sometimes it is impossible to estimate the likelihood of a threat occurring. In that
case the likelihood should be classified as High.
5. The impact of the threat is assessed ("Impact Criteria").
Critical, this threat will be a serious non-compliance or business risk if it
occurs. The likelihood of it occurring must be negligible;
Medium, will cause a non-compliance or business risk if it occurs but can be
corrected or recovered;
Low, will not cause a non-compliance or business risk.

Page 11/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
6. The threat levels are classified. Table 1 shows the 4 possible levels of threat
classification and the levels are explained in detail below the table.

High

Medium

Low

Negligible

Impact
assessment

Threat
likelihood

High

Medium

Low

Table 1: The threat classification

H threats are classified as high level threats, because of the high impact and are
very likely to occur.
M threats are classified as medium level threats but are still very important,
because a threat with high or medium impact can occur, even if it is very unlikely.
L threats are classified as low level threats because of their medium impact and a
low likelihood to occur.
N are not stated as (possible) threats because they have low (no) impact or are
unlikely to occur.
7. After threat level classification, the threats are categorised by priority. This is done
by introducing the likelihood of detection factor. Criteria to be used for
likelihood of detection are:
High, detection of the threat is perceived to be highly likely;
Medium, detection of the threat is perceived to be reasonable likely;
Low, detection of the threat is perceived to be unlikely.
The GAMP guide suggests frequencies of detection that correspond to the above
classifications.

Page 12/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
Together with the threat classification the likelihood of detection is used to prioritise
the threat.

High

Medium

Low

Negligible

Detection
likelihood

Threat
classification

Low

Medium

High

Table 2: The threat priority categories

Table 2 shows the threat priority categories. These 4 categories are explained in more
detail below.
Category 1 threats have high priority. These threats are difficult to detect and have
a high threat level. These threats should have corrective and preventive actions.
Category 1 must have a strong validation focus.
Category 2 threats have a lower priority then category 1 but are still very important.
Corrective and preventive actions should be in place also. This category still has a
high focus on validation.
Category 3 threats have a low priority because the threats have mostly a low threat
level and are very likely to be detected. The threat can be corrected or recovered, but
correcting and recovering procedures should be in place. These threats have a lower
emphasis on validation.
Category 4 is not stated as (possible) threats. The assessment determined that
these processes and function are no threat to compliance and to the business and
have low or no emphasis in validation. But the justification of this outcome should be
stated in the risk analysis document. This could be useful for future inspections and
justify the exclusion out of the validation plan.
8. The next step after setting the priority is addressing actions to the identified
threats. Examples of preventative or corrective actions for threats are:
Ensure security is sufficient for the set-up of an item master such that they
can only be updated by authorised personnel;
Written procedures (SOPs) that should control all cGxP processes;
Specific test protocols should be written to validate that all cGxP functions and
processes work correctly and data is protected from unauthorised change;
Develop and validate audit trail reports;

Page 13/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
Implement procedure that stores electronic versions in a format that prevents

alteration, supported by procedures that ensure retention of all such

documents for minimum of 15 years;
There should be a written procedure for installing the software;
All network access rights must be controlled through written and approved
procedure;
User SOPs should be written, reviewed and validated during OQ and PQ.
Training materials should be written, reviewed and validated during PQ.

9. The completion of the draft risk analysis document records the results of the
assessment, and all defined preventative and corrective actions for the high
priority risks. The contents of the risk assessment document should include:
Introduction, a short description of the assessment and the computer system
included the GAMP categories of software used in the system;
Scope, which aspects are within the scope of the risk assessment;
Objectives, and what is the summarised outcome of the risk assessment;
Inputs, which documents and participants were used for the risk assessment;
Business processes, which identified functions have been assessed;
Risk criteria, what criteria were used for the likelihood and impact.
The most important constituent of the risk analysis document is the list of potential
threats to regulatory compliance and the business, together with an assessment of
the likelihood and impact of each risk event / threat, and a prioritised assessment of
each risk event / threat. These can be divided in three parts:

Threats to cGxP and business processes;

Threats to Electronic records, Signatures and Audit Trail (Part 11);
Threats to other risks like Security, System Configuration and Change Control,

Data Retention including backup and restore, Support and Administration,

SOPs and Training.

3.4 Reviewing phase

After the Performing phase has been completed and the draft risk analysis document
has been created, the document must be reviewed, approved and issued. To do this,
several formal reviews of the document must be conducted by the risk assessment
team and other participants in the risk assessment process. There should be an
approved procedure for conducting such reviews.
After the review(s), the risk analysis document may be modified based on the
corrections received and comments made during the review process. When the
modifications are completed the document will be reviewed again. The process of
modifying and reviewing ends when the assessment team agrees upon the end result
of the risk analysis document. When the assessment team agrees upon the end result
the risk analysis document should be formally approved.
After the Reviewing phase, the risk analysis document is available for release to those
on the distribution list and as input to the validation process (e.g. in the justification of
the approach to validation, its focus, and its depth).

Page 14/16
Copyright 2006 ps_testware

Risk Assessment for Regulated

Computerised Systems
3.5 Follow-up phase
The Follow-up phase is concerned with the development and implementation of a plan
to deal with the identified risks, through the execution of the actions defined in the
risk analysis document. As part of the validation process, the Follow-up phase will be
a basis for the creation or amendment of the validation plan, and consequently the
validation activities.

CONCLUSION

Risk Assessment is a tool to assist in defining and following a better and efficient
validation process. The assessment gives a clear and reliable picture of all the risks
and threats to the business and towards compliance of cGxP and 21 CF Part 11
requirements. The identified risks and threats are assessed and corrective actions and
preventive measures are defined.
The principal benefit is that a structured process of conducting risk assessments
produces a solid base for determining and justifying the scope and extent of, and
exclusions from the validation process. Through a detailed risk analysis document that
identifies the risks, their consequences, and the necessary preventative and corrective
actions, the right validation strategy can be chosen.

Page 15/16
Copyright 2006 ps_testware

Quality is (y)our
business

ps_testware n.v. Leuven (B)

Tel. +32 16 35 93 80
Fax. +32 16 35 93 88

ps_testware b.v. Gorinchem (NL)

Tel. +31 183 699 773
Fax. +31 183 699 774

info@pstestware.com
www.pstestware.com

Page 16/16
Copyright 2006 ps_testware