Office of Science and Engineering Laboratories Excellence in Regulatory Science

Assessing Credibility of Computational Modeling and Simulation Results 
through Verification and Validation: Application to Medical Devices
ASME V&V40 Subcommittee on Verification and Validation of Computational Modeling for Medical Devices

Background/Overview Model Risk Credible for the COU?

Computational modeling is used throughout the entire product life cycle
of a medical device to provide information related to the performance,
safety and effectiveness. Unlike other types of physical models,
computational models can be used to assess aspects of in vivo
performance without subjecting patients to potential harm or
unnecessary risk. The credibility or trust in the predictive capability of a
computational model is of paramount importance due to the potential
consequences of an incorrect assessment about the performance of the
medical device. Model credibility can be established through
verification and validation (V&V) activities. Although methods for
V&V are well-established, guidance on the adequacy of the V&V
activities is lacking. The activities of the ASME V&V40 subcommittee
are to provide guidance on ensuring that V&V activities establish
sufficient model credibility. Given the inherent risk of using a
computational model as a basis for predicting device performance, a
risk-informed credibility framework has been developed.
We present a framework for establishing and assessing the credibility,
the trust obtained through the collection of evidence in the predictive
capability of a CM for a context of use (COU). The COU is the specific
role and scope of the CM&S used to address a question or decision, or
claim as evidence. The framework, referred to as the risk-informed
credibility assessment method, is presented below in Figure 1.
Model risk is the possibility that the CM&S leads to an incorrect
decision that results in patient harm or undesirable non-patient-related
impacts. Model risk is the combination of the influence of the CM&S
(CM&S influence) and consequence of an adverse outcome resulting
from an incorrect decision (decision consequence).
CM&S influence is the contribution of the CM&S to the decision
relative to other available evidence.
Decision consequence is the significance of an adverse outcome
resulting from an incorrect decision.
Consequences will typically be considered in the context of potential
harm to the patient. However, non-patient-related impacts may also be
considered, such as delayed patient access to medical devices, impact on
the clinician, financial loss, or increased time to market.
The relevant risk management procedure (e.g., ISO 14971) for the
medical device may be used to identify the severity and occurrence that
inform the decision consequence.
Table 1: The aspects of the verification, validation and applicability
processes and the associated credibility factors.
In the guide, we present the definitions for each credibility factor. An
example is provided in the box below. Associated with each credibility
factor is an example gradation of various activities to achieve the
appropriate rigor for supporting the CM for the COU. The gradations
foster planning and comparing the activities that can impact model
credibility, and can be adapted for a particular COU.
The credibility of the CM for the COU is determined by a review of the
completed V&V activities and outcomes, with consideration given to the
established credibility factor goals, model risk, COU, and any additional
knowledge gained during the V&V process. A review of the rationale
indicating that the completed activities are sufficient to establish
credibility commensurate with the model risk is integral to the
credibility assessment. This may include the individual assessment for
each credibility factor. Deviations between the V&V plan and the
completed activities should be part of the supporting rationale and
contribute to the totality of the V&V activities that are ultimately used to
determine model credibility. It is possible that upon completion of the
V&V activities, the credibility goals might not have been met as initially
planned; however, the CM may still be sufficiently credible based on the
rationale developed. It is suggested that an organization’s internal
review processes be used to facilitate this assessment. It is
recommended that the reviewing participants have the knowledge and
experience appropriate to assess the CM credibility.
An example work flow of the CM credibility assessment is provided in
Figure 4.

Relevance of the validation domain to the COU: The factor summarizes the
relative proximity of the COU domain to the points in the validation domain. An
example gradation of applicability, from lower to higher credibility, is:
• Portions of the COU domain fall outside the boundary of the CM
assumptions; thus there is limited applicability to the validation domain
(COU3 in Figure 3).
Figure 1: The risk-informed credibility assessment process. • There is minimal or no overlap between the COU domain and the points in
the validation domain, but the domain of the CM assumptions encompasses
the COU domain (COU2 in Figure 3). Figure 4: Example work flow for assessing credibility of the CM
The foundational element of the framework is model risk, the possibility
• The COU domain encompasses multiple points in the validation domain
that the CM&S may lead to an incorrect decision that might result in an Figure 2: Model Risk Map (left) represents how CM&S influence and If the desired level of credibility is not achieved, the CM may not have
(COU1 in Figure 3).
adverse outcome, such as patient harm. Model risk will be used to decision consequence dictate model risk; the color map (right) shows to be abandoned altogether. These are possible next steps:
• The COU domain is entirely within the validation domain and encompasses
establish the adequacy of the V&V activities for the CM&S to be used another mechanism for how to define gradations for influence and multiple validation points (see the overlapped region of COU1 and the 1. Conduct additional V&V activities: For example, additional
in the COU. consequence to determine model risk. Validation Domain in Figure 3). activities regarding the comparator could be conducted.
2. Change the CM: This might involve modifying the code, solution
formulation, system configuration, system properties, boundary
Context of Use Credibility Factors Figure 3: conditions and/or governing equations.
Illustrative
3. Reduce the influence of the CM: The influence of the CM, and thus
The context of use defines the specific role and scope of the CM&S used Model credibility can be established through the collection of V&V examples of
model risk, may be reduced by performing other activities (e.g.,
address the question of interest. It is a detailed and complete statement evidence and by demonstrating applicability of the V&V activities to three COU
domains physical tests, clinical data) to gather additional evidence.
of how the output from the CM will be used to answer or inform a support the use of the CM for the COU. Applicability is a
relative to one 4. Modify the COU: The COU could be modified to lower model risk
question of interest. It should also include a description of what will be complementary component of the V&V process, which can be
validation and reduce the credibility needed for the COU.
modeled and how. The COU describes the specific use of the CM&S discretized into the following components: code and calculation
domain and the If none of the above are feasible, the CM may need to be abandoned.
for assessing one or more specific aspects of the device and/or device verification, computational model for validation, experimental
boundary of
performance. It is important to note that for medical devices, the COU comparator for validation, and then validation assessment. The assumptions for
for the CM&S is distinct from the indications for use or intended use of credibility factors, presented in right-hand column of Table 1, are the
the device. The intended use is a description of how the device itself key aspects of the risk-informed credibility assessment method that
a simple two
parameter CM
Contact Information
will be used clinically, whereas the COU is a description of how the enable the practitioner to determine the rigor needed for each step in the To join the interested parties list, please email Ryan Crane at
CM&S will be used to answer a question of interest about the device. V&V process and to demonstrate applicability. CraneR@asme.org. For information regarding the Credibility Strategy,
please email Carl.Popelar@swri.org or Tina.Morrison@fda.hhs.gov.
For membership, please contact ARau@exponent.com
ASME V&V 40 SUBCOMMITTEE — VERIFICATION AND P. Afshari, Depuy-Synthes Spine K. K. Debus, Cd-adapco H. Jin, Medtronic, Inc. W. A. Olson, Ethicon Endo-surgery
VALIDATION IN COMPUTATIONAL MODELING OF B. P. Baillargeon, Dassault Systemes Simulia Corp M. Dharia, Zimmer Biomet A. Kiapour, 4WEB Medical Inc. T. L. Rossman, Mayo Clinic
MEDICAL DEVICES D. Bardot, Medical Device Innovation Consortium S. Eswaran, Abbott Vascular L. Knudsen, Syncroness P. Saffari, Endologix
A. Bestelmeyer, BD Technologies C. Funkhouser, Baxter Healthcare Corporation S. Kulkarni, VEXTEC Corporation C. Scotti, W.L. Gore
C. Popelar, Chair, Southwest Research Institute J. Bischoff, Zimmer K. Genc, Simpleware Inc. D. Levine, Zimmer, Inc. R. Swift, Cook Research Inc.
T. M. Morrison, Vice Chair, US Food and Drug Administration J. P. Bodner, Medtronic Corp M. Goodin, Simutech Group X.M. Li, Consultant P. Tomaszewski, Depuy Orthopaedics Inc
A. C. Rau, Vice-Chair, Exponent, Inc. S. Cheng, Integra Life Sciences I. Guler, Boston Scientific Corporation X Liu, Stryker Orthopaedics T. Zhao, Edwards Lifesciences
R. L. Crane, Secretary, ASME B. D. Choules, MED Institute A. Gupta, Google Inc. B. A. Lurie, W.L. Gore A. U. Nair, Alternate, BD Technologies
R. Chow, Consultant P. Hariharan, US Food and Drug Administration R. Marinescu, Smith & Nephew N. R. Rebelo, Alternate, SIMULIA Western Region
J. C. Coburn, US Food and Drug Administration W. Hary, Heartflow J. Mast, Hill-Rom, Inc.
C. Corrales, Baxter Healthcare Corporation M. Horner, ANSYS, Inc. L. Mulugeta, Independent