Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Testing Methods
By George N. Brower
Analex Corporation
❖
S
tructural testing encompasses levels discussed below.
three critical phases of soft- ❝Structural Structural testing should be done
ware development and test- at the unit, integration, and system
ing; yet, one or more of these phas- testing assures the levels of testing.2 (As used herein, a
es is often deliberately bypassed, “unit” is the smallest separately
overlooked, or performed in a less
program’s compilable – or equivalent – ele-
than rigorous manner because either statements and ment of code, such as a procedure,
the technical advantages are not subroutine, class, method, or data-
fully considered or, more often, the decisions are base table.) Structural testing
cost and schedule benefits are not assures the program’s statements
appreciated. While structural test- fully exercised and decisions are fully exercised by
ing is required by the FDA for code execution. For example, it
medical devices of moderate and by code confirms that program loop con-
major level of concern, this testing execution.❞ structs behave as expected at their
1
Figure 1 Figure 2
Software Verification Versus Configuration for a
Software Validation Testing Structural Unit Test
Driver Simulating
Concept “Compute Y”
(Unit-Under-
Product Test)
Specifications Get Formatted
ABC
Verification
Software
Requirements
Stub Simulating
Verification Stub Simulating “Notify Operator of
“Format ABC” Invalid Data”
Design Performance
Qualification
Figure 3
Incremental Integration Structural Testing
Compute Y
Notify Operator
Open Files if Device X
Format Notify Operator is Off-Line
ABC of Invalid Data
Notify Operator
2nd Function 3rd Function
of Any Errors
1st Function
Output Error
Messages
ing system should be structurally tested first. This utilities for the next structural test, and so on. All
portion of the code itself should be broken down previously tested functions should be regression
into functionally cohesive packages if it is large tested, as appropriate.
and/or complex; otherwise, it can be structurally
tested as an entity. The second portion of the code The Nonincremental Approach
to be tested is normally the unique input/output
section. If there are diverse input/output devices, As used herein, a nonincremental approach is one
these may be structurally tested separately. But it is in which the complete software package, as it is
often best to select a thread that includes both the intended to be deployed, is totally built and then
ability to input data and to see the resulting output structurally tested. The nonincremental approach to
for each structural test. The third step is to select integration-level structural tests may reasonably be
and structurally test a functionally cohesive portion used for small programs and is often the best ap-
of the application and the utilities needed to sup- proach for third party, independent testers. Although
port that application. (Remember: The units select- in most circumstances integration-level structural
ed for integration structural testing, and this is testing is the purview of the software developers and
especially true for the utilities, should be verifica- not, for example, that of the computer system vali-
tion tested first – which includes unit testing – prior dation team, at times these third-party testers are
to any integration-level structural testing.) Then, asked to support a program in this manner. An
select the next functionally cohesive portion of the immediate problem that most often surfaces, howev-
application software and associated application er, is due to budget, schedule, and/or personnel con-
straints, testers independent of the software develop- both using the product specifications (which in turn
ment team cannot, in a cost-effective manner, select, may include the user’s manual and other labeling)
compile, and link individual software functions. and software requirements for the test development
However, they can build and load the fully integrat- and the acceptance criteria. Although, at the system
ed program, select and test a function (ignoring the level, a knowledge of the design may be necessary
remainder of program while maintaining a structur- to determine how to set up a test for a specific func-
al-tester’s point-of-view), and then select the next tionality (for example, an output error message that
function for test. One advantage to this approach is a file cannot be found versus an output error mes-
since a majority of the code may be executing dur- sage indicating there is invalid data in a file), that
ing each test, the need for regression testing of the design knowledge should not be used in construct-
previous functions is diminished. A disadvantage to ing the acceptance criteria of the tests – such as the
this nonincremental approach is if a function is boundaries of legitimate data. That information
found to not work properly, finding the cause of the should be derived from the specifications and
problem may be more difficult precisely because so requirements.
much of the code is executing.
At the integration level, structural testing uses the The Importance of Each Testing Phase
architectural high-level design to develop the test
cases and to derive the acceptance criteria. (During As with each verification, integration, and qual-
the integration test phase, functional, black-box test- ification activity, each structural testing phase finds
ing should also be performed.) unique problems that otherwise may be either very
difficult to find at best or, at worst, may be left
System-Level Structural Testing undetected during pre-release testing, waiting for
the error to surface in the field under actual use.
The system-level structural tests must be run on Using a step-wise approach to testing finds prob-
the fully integrated software package. However, lems and solutions faster and with a greater guar-
whether the software functions for the integration antee of producing a solid software product than if
structural tests discussed above are built incremen- any phase is skipped or not performed with rigor.
tally or all at once prior to conducting the integration In this context, rigor means (a) planning all tests
structural tests, the system-level structural tests must and reviewing the plans for completeness, correct-
still be run. The primary differences between inte- ness, and consistency; (b) writing the procedures
gration level and system level structural tests are the sufficiently well so they may be reviewed and,
tester’s frame of reference for developing the test when necessary, rerun (e.g., following software
inputs and determining the acceptance criteria of the updates); and (c) maintaining a written record of
test outputs. An important secondary difference in the tests performed and the results so those may
integration structural testing is that the code may be also be reviewed for confirmation that all tests
altered (e.g., to create test drivers and stubs) or the have been completed. 3 In performing the above
normal code execution flow may be altered (e.g., three tasks, program management is supported in
running the software in a de-bug compiler mode that all of the activities, from planning through
where the execution can be halted, a data value execution and reporting, can be scheduled, budget-
changed, and the execution restarted, allowing for a ed, and, therefore, managed.
variable’s maximum or minimum value, or a failure Figure 4 illustrates, for each structural test phase,
or other perturbation to be inserted). In system struc- the documentation against which the software is test-
tural testing, the code must not be changed, and the ed.1 In that figure, software development implementa-
program execution flow must not be altered. Rather, tion is shown on the left side, going from product
all test inputs must be accomplished by control of specifications to code implementation. Software
the actual or simulated environment. (structural) test is shown on the right side, going from
At the system level, structural testing is also best unit test to system test. The documentation for each
performed along with functional, black-box testing, test phase is illustrated by the horizontal arrows. This
general approach is followed for any software devel- of hidden, difficult-to-find software problems in the
opment and testing life cycle approach, be it waterfall, succeeding test phases that can easily disrupt any
spiral, or iterative. For example, if the spiral approach schedule and budget. ❏
is used, a piece of the product specification is deter-
mined, the software requirements for that piece are
developed, the high-level and detailed design, and About the Author
then the code for that piece are developed, and then the
testing is accomplished starting at the unit level, work- George N. Brower has published several articles
ing through the integration level, and finally working and has given presentations at numerous confer -
up to the system level. ences on his areas of expertise: software develop -
ment and software Independent Verification and
Validation (IV&V) testing. For the past 11 years, he
Figure 4
has served as Deputy Director of the Denver office
Structural Test Verification of Analex Corporation, providing software engineer -
Documentation ing services for computer systems validation and
outsource software development. Analex-Denver is
Product Specifications System-Level an ISO 9001 4 registered company that is the recipi -
and Structural ent of the 1995 Supplier of the Year Award for IV&V
Software Requirements Test testing and the 1996 recipient of the James S.
Cogswell Industrial Achievement Award. In 1998
Analex received a second award for IV&V testing of
Integration-Level critical software. Brower can be reached by phone
Top-Level Design
Structural Test toll free at 1-888-262-5391, by fax at 303-730-2057,
or by e-mail at brower@analex.com.
Unit-Level References
Detailed Design
Structural Test
1. Guidance for FDA Reviewers and Industry, Guidance for the
Content of Premarket Submissions for Software Contained in
Medical Devices, CDRH, May 29, 1998.
2. Guidance for Industry, General Principles of Software Val-
Implementation idation (Draft Guidance) Version 1.1, June 9, 1997.
(Coding) 3. FDA Current Good Manufacturing Practice (cGMP) Final
Rule; Quality System Regulation, 21 CFR Part 820.30, Design
Control, fully effective June 1, 1998.
4. International Organization for Standardization, 1994. (Analex
is registered for independent software verification and valida-
Again, the preceding discussion, as well as all tion, software development, systems analysis, and hardware
aspects of this paper, applies to all software applica- prototyping.)
tions, whether the software is for a process control
system, medical device, software supporting a qual-
ity system, or any other software, such as that used
for simulators or even business systems.
Summary