Sei sulla pagina 1di 6


An Overview of Software Testing in Medical System and Medical Device Testing

M Vijaya Deepti

Department of Computer Science

Stratford University, Virginia

Abstract--- The development of reliable software is given

utmost importance now-a-days especially in the field of

medicine where reliability of the software developed is

achieved through what is called as testing. Testing is a

process of verifying and validating the software to ensure

that the specifications and requirements of the end user

are met and that it is working according to the system

design. Software testing of the medical devices is

increasing important to check the efficacy and safety of

the devices before they are implemented on the patients in

real time. History has seen many disastrous medical

system failures which lead to the death of a large number

of people. So medical practitioners make it a mandatory to

check the medical devices and system to ensure the proper

delivery of the tangible product to the customer. In this

paper an attempt is made to explain the various steps

involved in testing of the medical devices and what

elements have to be tested mainly. It also explains about

the learning experiences about software testing and its

management plans which include the important phases of

software verification and validation.

Keywords: Qualification Testing, Verification, Validation,

Water-fall Model.



Health care professionals rely on software to simulate

anatomical and physiological elements of the human

body for purposes of training, prototyping, and decision

making. Software can also be used to simulate medical

processes and protocols to measure cost effectiveness

and resource utilization. Whereas much of the software

engineering research into simulation software focuses

on validation (determining that the simulation

accurately models real-world activity), to date there has

been little investigation into the testing of simulation

software itself, that is, the ability to effectively search for errors in the implementation. Developers and testers

must ensure that newly developed products or product enhancements meet functional and performance requirements and that those products are reliable and able to operate consistently under peak loads [1]. The traditional approach for development projects is to perform some unit testing throughout the development

M N V S S Kumar

Department of Electronics and Communication Engineering

Andhra University, India

process but to leave the bulk of the testing effort until

the project nears completion. Although this is the most

common method for developing and delivering

products, it is fraught with danger. First, it is difficult to

estimate the time and effort it will take to remove all

defects once a product is completed, and an inaccurate

estimation may lead to shipping delays. Second, the

cost of finding problems at the end of a project may be

extremely high if fixing them means reworking (or

worse yet, redesigning) parts of the application.


The developers must outline four plans by the end of

the user requirements (UR) review to define how the

development will be managed in the Software

Requirements (SR) phase. These are the:

Software Project Management Plan

Software Verification and Validation Plan

Software Quality Assurance Plan

Software project management plan

Every project must produce a software project

management plan. The plan should declare the

objectives of the project. These may take the form of

statements of deliverables and delivery dates, or of

statements of required functionality, performance and

quality. The objectives should be prioritized. SQA staff

should examine the statement of objectives and check

that all plans are consistent with them. The plan should

define the life cycle approach to be used in the project,

such as waterfall, incremental delivery or evolutionary.

The choice of life cycle approach is crucial to the

success of the project. SQA should check that the life

cycle approach is appropriate and has been correctly

tailored to the project. A process model, based upon the selected life cycle approach should be described in the plan. The process model defines all the major activities.

For each activity it should define:

entry criteria;




exit criteria.

ISSN: 2230-7818

@ 2011 All rights Reserved.

Page 80


Software verification and validation plan Verification

It is the process of determining whether the products of

a given phase of the software development cycle fulfill

the requirements established during the previous phase or the process of evaluating software at the end of the software development process to ensure compliance with the software requirements.


V&V plays a large role throughout the Waterfall Model

of phased software development. It is essential to the

development of quality medical software [2]. During

the Requirements phase, a plan is produced for

establishing verification and validation of the medical

software. A method for configuration control is

established. All documents are reviewed for

compatibility with the V&V plan. Verification will be

the overriding activity at all stages of the software

development. Validation will be addressed in the final

stages. During the Design phase, the Architecture is

compared, or traced, to the Systems Requirements. A

Hazard Analysis is produced that identifies the known

risk areas of the software. This analysis is critical to the

development of quality medical software. By

identifying the areas of software whose failure could

cause the potential for serious injury or death, the test

plan can be modified to ensure correctness [3]. These

identified critical areas should always receive additional

and mention in the V&V plan During the

Implementation phase, the code is verified as it is

produced. In order to ensure completeness, the code is

traced to the design, linking design specification to

coded module. All verification defects are resolved.

By the end of the UR phase, the software verification

and validation plan needs to include a plan for verifying

the outputs of the SR phase and an acceptance test plan.

The intention of software Verification is to make sure

that the work products are working properly and

consistently. Verification techniques used in the SR

phase include reviewing, formal proof and tracing.

Plans for reviews should include intermediate and

formal reviews. Intermediate reviews are used to verify

products, and later to clear up problems before formal

review. The primary formal review is the Software

Requirements Review. SQA would also evaluate the

Verification processes by finding the bugs and defects

that were missed by SQC (Software Quality Control) during Verification. SQA should examine the software verification and validation plan and check that the review procedures are well-defined, the right people are involved, and that enough reviews are held [4]. Documents that have not been reviewed are just as likely to contain problems as code that has not been tested. SQA staff should examine the software

verification and validation plan and check that the verification approach is sufficiently rigorous to assure the correctness of the software requirements document. Formal methods for specification and verification of software are often necessary when safety and security issues are involved. The software verification and validation plan needs to define how user requirements will be traced to software requirements. SQA staffs often have to check the traceability matrix later in the

project, and it is important that traceability procedures

are easy to use [5]. Verification answers the queries like

are we building the product correctly and Validation

answers the question are we building the correct

product. Validation explains that the end product

gratifies the intended purpose when placed in the right

environment while Verification refers to the actual

developing of the product according to specification

The term Acceptance Test could also apply to

Validation, in most cases the Acceptance test is carried

out by a different group of people from the SQC team

that performed Verification, as the product was being



A. Unit and Integration testing

Since the accuracy of the dose calculation algorithms is

the main safety issue in a treatment planning system,

the unit testing was started from them. It was not

enough to test the user interface only since the

correctness of the calculation phases had to be checked

also in intermediate levels. The software qualification

engineer rebuilt the dose calculation algorithms from

the specifications without consulting how they were

implemented in the actual code to be tested. The rebuilt

dose algorithms were compared with the actual code by

using various input data sets. This approach introduced

the need for so called stubs, i.e., code that had to be

written in order to simulate the data that the function

under test needed, and drivers, code that called the

functions under test [6]. The output data from both the

reference code and code under test was then compared

automatically. The same approach was used at two

higher integration levels by combining the smallest test

units into larger entities.

B. Qualification testing

Software requirement specification (SRS) was used as the starting point for the development of qualification test specifications. Application’s functionality was grouped to testable entities and test cases were created to test every feature specified in the SRS to ensure the conformance of the system. More tests were created based on the results of hazard analysis to focus on safety issues and the user’s manual was used to create

ISSN: 2230-7818

@ 2011 All rights Reserved.

Page 81


test specifications about the usability of the whole system. Writing these test specifications took about two man months. The test input data of functional tests were created by using the equivalence class partitioning: The input data was divided into classes that were expected

to behave in the similar way and one value of each class was used as an input for the test. Also the boundary and other special values were used as test data. Testing was done with both valid and invalid values. Furthermore,

real clinical data was used to test the usability. The

testing of dose calculation algorithm accuracy was

continued in the qualification phase: by comparing the

output with published experimental and simulation

results, and physical measurements. This way the

suitability of both the chosen calculation model and its

parameters was verified. Customer acceptance

procedures, quality assurance procedures, the user’s

manual and installation guides were tested, either by

reading them through or using them as a user would do.

Attention was paid also to the usability aspects and

communication between the user and the application.

All the faults found were entered into a discrepancy

report database. Tests were rerun every time there was a

reason to believe that the changes in the new version of

the application could cause a change in the function

under test. Before the final release all the tests were

rerun once more.



In testing a medical device, there are five major

elements to cover. The first is the display of the unit.

The graphical display must be tested to ensure all pixels

are working, and the output on the display is correct for

the functioning of the device. Vision tools are used to

perform this type of testing. The second area is battery

testing [7]. Most devices require battery power even if

only for data backup purposes. Some modular

instrumentation including Digital Multimeters and

Switching are used to perform battery tests. The third

area is power. The unit must be tested to ensure the user

does not receive a shock from the power supply. The

fourth area is Electromagnetic Emissions/Immunity

testing. These tests ensure that the medical device does

not radiate EMI or malfunctions from susceptibility to

EMI. RF tools are used to perform pre-compliance

testing. Finally, many medical devices use electronics that must be tested for functionality. A set of modular instruments that can test the electronics in medical

device are given as follows. The digital thermometer is an example of a medical device requiring testing. The tests include the following:

1. Displays – LCD displays must be verified for pixel on/off, brightness, and contrast.

2. Battery – current/voltage draws must be tested as

well as the life cycle of the battery under maximum usage conditions.

3. Temperature accuracy – the accuracy of the

temperature readout must be calibrated and measured.

4. EMC – electromagnetic emissions must be tested to

ensure the unit is not radiating EMC signals beyond the

allocated spectrum.

5. Internal Self Tests – must be performed to verify

proper functioning.

6. Computer interfaces – for those devices designed to

connect to a computer, the interfacing (serial, parallel,

etc) must be tested.

Other outputs – if the digital thermometer generates

analog voltage, analog current, frequency, or alarm

signals, these must be tested for accuracy. In addition to

testing the basic functionality of the device, other tests

may be performed. These include safety, environmental

testing such as shock, vibration, thermal, humidity

testing, etc, biocompatibility and sterility testing, and

life-cycle testing.


In the previous section we outlined a rigorous software

development methodology suitable for medical

software [8]. In the present section we amplify on

requirements for the testing of medical software. First

we introduce some useful distinctions:

- Testing is nominally the process of executing the

software and comparing its behavior to expected results

and predefined criteria.

-Testing may be performed by the developers and/or the


- Acceptance is the process of determining whether or

not a software product satisfies contract requirements.

Clients make acceptance decisions. Certification is the

process of stipulating that a software product has

required properties that may not be fully testable.

Certification may be performed by an organization

other than the developer or client. Testing often

provides the only basis for acceptance and certification.

In this section we review the objectives of testing,

discuss test methods, and comment on the current state-

of-the-art in software testing with examples drawn from

our experience in testing medical software.

Test Objectives

In a narrow sense the objective of software testing is the discovery and elimination of "bugs." There are many examples of serious losses attributed to software bugs in supposedly "safe" systems. A software bug is suspected in a recent series of deaths in therapeutic radiology. Bitter experience with programs of even modest size quickly teaches that the total elimination of

ISSN: 2230-7818

@ 2011 All rights Reserved.

Page 82


such errors is impractical. Even if all of the bugs could be detected and eliminated there would remain other areas of concern. Major test objectives are defined in terms of the specific tasks or functions which the software must perform. Criteria include (1) the ability to perform the task completely and (2) the ability to perform the task efficiently (quickly or with a minimum

of storage) as well as (3) the ability to perform the task free of errors. Other test criteria may be economic. For

example, the testing of prototypes may establish that

the fully developed system will be too costly [9]. This

is particularly true of the fast-paced commercial

medical software market.

Test Methodology

We have emphasized the importance of methodology in

evaluating "good practice" for medical software. When

testing complete builds or conducting complete system

testing three approaches may be characterized and they

are structural, functional and random. These

techniques provide complementary strategies for

developing test cases. Functional testing methods

typically require the preparation of a matrix of

numbered functional and/or performance requirements

vs. tests where it is established. Random testing

methods proceed from the recognition that in practical

software systems, the number of possible program paths

grows rapidly beyond the ability of the tester to

exercise exhaustively. But a meaningful sample of the

system behavior may be obtained with a test generator

that varies the input in a random manner. This approach

is more helpful when the aim is to probe vulnerability

to security breaches, system crashes, or system

degradation rather than the accuracy of calculated

results. The approach is also useful in validating

software reliability models based on statistical


Test Conduct

From the beginning of computer programming it was

recognized that the testing phase, in particular, is as

much an art as it is a science. This recognition proceeds

from the simple fact that even for small systems the

number of possible program paths is far too large for

the tester to exhaust. Thus the tester may design a test

plan in which the requirements vs. tests matrix is

complete, that is, every requirement is tested at least once, but in the test description or test procedure it quickly becomes apparent that only a small fraction of all possible combinations of input values and conditions is exercised. Recognition of this situation has led to the concept of random testing mentioned previously. Refinements in the concept of random testing attempt to attach priorities to test cases on the basis of frequency- of-use, severity of error and other criteria. Such an

approach is crucial for the large software systems under development today. To reduce the chance for bias, system testing is today generally performed by an independent test team. A subset of the system tests is usually chosen as the basis for customer acceptance tests when the acceptance tests are not designed by the customer. Alternatively, the customer may contract with an independent testing organization to design and perform acceptance tests. The independent test team

lies outside the line authority of the project's system

engineering and software development departments.

Moreover, a product assurance office oversees the

testing organization and in a rigorous environment, the

product assurance office will function in parallel with

the project management and report above the project




A defibrillator is a device implanted into the body and

under cardiac arrest conditions provides a shock to the

heart to reestablish a beat. It also provides a pulse

generation to pace the heart. A pacemaker provides

primarily the later function. Pacemakers and

defibrillators consist of complex digital circuitry that

must interface to its exterior environment through

analog connections. A telemetry channel provides

interfacing for device diagnostic purposes after

implantation. Defibrillators must provide a kilovolt

level shock to the heart without damaging the internal

circuitry of the device. Because the device is implanted

it must have a substantial battery life. The test challenge

relates to making precision measurements on complex

digital circuitry that has been stimulating and interfaced

through analog signals. While this is not a traditional

mixed-signal test, signal analysis is a key element of

testing the device. The second challenge is the wide

range of signals to measure. The shock voltage reaches

kilovolts and amperes, but must run on nanoamps and

sense microvolts. For pacemakers, the primary function

is generating a pace pulse to ensure continuous heart


This pace pulse consists of a waveform signal, whose

rise time, drop, and recharge pulse amplitude yields

information about the condition of the digital circuitry.

Figure 1 shows a graphical representation of the pulse

signal to be measured. Test instrumentation for the pace pulse test consists of a signal source and a digitizer as shown in Figure 2. The signal source provides the stimulus representing the heartbeat. A digitizer reads the response from the device representing the pacing pulse from the device to the heart. The sensing test uses the signal source to generate Haversine waveforms that simulate the electrical activity of a heartbeat [7].

ISSN: 2230-7818

@ 2011 All rights Reserved.

Page 83


External shock tests the sensing circuitry after an external shock has been applied.

Recharge Pulse Recharge Amplitude Reference Pulse Width (50% Point) Pulse Drop Amplitude (As % of
Recharge Pulse
Pulse Width
(50% Point)
(As % of Pulse)

Pace Pulse

Figure 1: Pace pulse from a pacemaker.

This simulates the condition in which an external

defibrillator has generated a shock to a patient with a

pacemaker implanted.

To perform this test, relays are used to switch high

voltage capacitors onto the leads. With an external

voltage applied, the device is tested to confirm its

ability to detect heart rhythms.

Additional tests include the following:

1-Battery Voltage

2-Lead impedance

3-Safety core

4-Gross Fail Shock

5-Timm – Leads

6-Sense Amp Differential Gain

7-Sense Amp Common Mode

8-Temperature Measurement

9-Battery usage

10-Sensitivity Refractory Testing

High PG Lead Defibrillator Voltage Scaling Output Switching Isolation Signal Sense Digitizer Amp Circuit
PG Lead

Figure 2: Instrument Setup for Pacemaker Test


Testing medical software includes testing for an automated blood chemistry analyzer, a picture

archiving and communications system, a medical data network, a telephony system which records messages for radiologists, and large hospital information systems. This convinced that medical software practitioners must

follow a rigorous engineering methodology with strong

project management to deliver safe and effective

systems. Factors like the personnel assigned to the

projects, the complexity of the hardware/software

interface, and the computer language used are different

for each system. While specific problems are unique,

certain classes of problems tend to recur. In particular,

it was found that:

1. Cost and schedule overruns in earlier parts of the

project limited the funds and time available for


2. The end product was not exactly what the end-

users wanted even though it rigidly met the original

specifications and

3. Small technical problems caused large cost and

schedule impacts when not controlled. Examples of

these problems and solutions are discussed below.

Most of them could have been avoided by adopting

a rigorous engineering approach to software




Software testing is very important phase in the software

development cycle because it checks whether the

particular software design is free of bugs and defects.

Although all software developers face common cost

and quality concerns, medical software poses special

challenges to developers and regulators. So software

testing reduces the cost and time consumption by

detecting the errors in advance. Software testing is very

important in the field of medicine as an error in medical

instruments costs a persons life. Medical device testing

is becoming increasingly important as it determines the

safety and efficacy of devices. So an overview of

software testing is brought out here and it is clearly

explained for testing medical devices by considering

examples like Pacemakers and Defibrillators.


[1] “Testing Implantable Medical Devices”, J. Max Cortner,


Guidant, Evaluation Engineering, June 2004, page 38. Y.-S. Ma, J. Ofutt, and Y. R. Kwon. MuJava: An automated


class mutation system. Journal of SoftwareTesting, Verification and Reliability, 15(2):97{133, June 2005. B. Beizer, “Black Box Testing”, New York, Wiley, 1995.

[4] C. Kaner, “Software Negligence and Testing Coverage”,

Software QA Quarterly, Volume 2, No. 2,

ISSN: 2230-7818

@ 2011 All rights Reserved.

Page 84


Van Dyk J., Barnett R.B., Cygler J.E., Shragge P.C., “Commissioning and quality assurance on treatment planning

computers”, Int. J. Radiat. Oncol. Biol. Phys. 26: 261473,1993 [6] J. Callahan, F. Schneider, and S. Easterbrook. “Automated software testing using model-checking”, Technical Report NASA-IVV-96-022, NASA Independent Verification and Validation Facility, Fairmont, WV, August 1996.

C. Murphy, G. Kaiser, L. Hu, and L. Wu. Properties of machine



learning applications for use in metamorphic testing. In Proc. of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE), pages 867{872, 2008.

[8] E. J. Weyuker. On testing non-testable programs. Computer

Journal, 25(4):465{470, November 1982.

[9] “Model-based testing and validation of control software with

reactis”, Technical report, November 2003

ISSN: 2230-7818

@ 2011 All rights Reserved.

Page 85