Software Testing

Testing is a set of activities that can be planned in advance and conducted to find errors
within a document.

A strategy for software testing must accommodate low level tests that are necessary to
verify that a small source code segment has been correctly implemented as well as high
level tests that validate major system functions against consumer requirements.

Verification and Validation-

Verification-Verification refers to the set of activities that ensure that software correctly
implements a specified function i.e. refers to code

Validation- Validation refers to different set of activities that ensure that the software that
has been built is traceable to customer requirements.

Testing provides the last bastion from which quality can be assessed and errors can be
uncovered. Quality cannot be tested only at the end; it should be incorporated at every
step in software testing.

Stakeholders in testing process-The software developer is always responsible for testing

the individual units of the program, ensuring that each performs the function or exhibits
the behavior for which it was designed.

Developer also conducts integration testing- a testing step that leads to the construction of
the complete software architecture.

The role of the independent test group (ITG) is to remove the inherent problems
associated with testing. Developer and ITG work closely throughout the software project
to ensure that through tests will be conducted.

Software testing(parts) can also be viewed as spiral model that can be divided into the
following four phases-

a) Unit Testing- Unit testing begins at the vertex of the spiral and concentrates each
unit ( i.e. component) of the software as implemented in software code
b) Integration Testing- The focus is on design and construction of the software
architecture.
c) Validation Testing- Requests established as part of the software requirements
analysis are validated against the software that has been constructed.
d) System Testing- The software and other system elements are tested as a whole.
e) Object Oriented Testing- It involves testing the classes for data and then classes
are integrated into an object-oriented architecture, a series of regression tests are
run to cover error due to connections and collaborations between classes and side
effects caused by addition of new classes.
Details of Testing

Unit Testing- Unit testing focuses verification effort on smallest unit of software design-
the software component or module.

Using the component-level design description as a guide, important control paths are
tested to uncover errors within boundary of the module. The unit test focuses on the
internal logic and data structure within boundaries of the component.

Local data structures are examined to ensure that data stored temporarily maintains its
integrity during all steps in an algorithm’s execution. All independent paths through the
control structure are exercised to ensure that all statements in a module have been
executed at least once.

Among common errors in unit testing are-

1.) Incorrect arithmetic precedence
2.) Mixed mode operations
3.) Incorrect initialization
4.) Precision inaccuracy
5.) Boundary value failures

For testing a unit a driver program must be developed. Driver is nothing but the main
program that accepts test data, passes such data to the component and prints relevant
results. Stubs serve to replace modules that are subordinate to the component to be tested.

Integration Testing-

Integration testing is a systematic technique for considering the software architecture

while at the same conducting tests to uncover errors associated with interfacing. The
objective is to take unit test components and build a program structure that has been
dictated by design.

Types of Integration-

1. Incremental Testing- Incremental testing is the antithesis of the big bang

approach. The program is constructed and tested in small increments where errors
are easier to isolate and correct.
2. Top down Integration Testing- It is an integral approach to contribution of the
software architecture. Modules are integrated by moving downward through the
control hierarchy, beginning with the main control module. It is again of two
types-
a) Depth-first integration- It integrates all components on a major control
path of a program structure. Selection of a major path is somewhat
arbitrary and depends on application specific characteristic.
 b) Breadth first integration- It incorporates all components directly
subordinate at each level moving across structure horizontally.

3. Bottom-up integration testing- Bottom up integration testing begins construction

and testing with atomic modules (i.e. components at lowest levels) in the
program. Because components are integrated from the bottom-up testing required
for the components subordinate is always available.

4. Regression Testing- Each time a new module is added as part of integration

testing, software changes. New data flow paths are established, new I/O may
occur and new control logic is invoked. As functions are added there is a need to
test them to avoid any side effects to the whole software. Regression testing may
be done manually by re-executing a subset of test cases or may use an automated
software tool.

5. Smoke testing- It is a testing approach that is mainly used while software products
are being developed. It can be applied to complex, time-critical software products.
The following steps are involved-

a) All software components are integrated into a build. A build includes all
data files, libraries, reusable modules and engineered components.
b) A series of tests are designed to expose errors that will keep the build
From properly performing it’s functions.
c) The build is integrated with other builds and the entire product is smoke
tested.

6. Sandwich Testing- Selection of integration testing strategy depends on the

software characteristics and the project schedule. A combined approach called
sandwich testing uses top-down integration tests for top levels and bottom-up
approach for bottom levels.

Integration testing must always identify critical modules which have high level of
control, definite performance requirements, are complex and are error-prone.

7. Object-Oriented software testing strategies (Unit and Integration Testing)

i) Unit testing- An encapsulated class is the focus of unit testing. Operations

within the class are smallest testable units. Also the operation should be
tested in each of the classes within which it is used.

ii) Integration Testing- There are three different approaches for integration
testing an OO system.
a. Thread based testing- It integrates the set of classes required to
respond to one input or event of the system.
 b. User based testing- First the independent classes called the server
classes are tested followed by the dependent classes called the stub
classes.
c. Cluster testing- A cluster of collaborating classes is exercised by
designing test cases that attempt to uncover errors in the
collaboration.

Validation Testing (for system requirements)

Validation testing tests the software for software functions in a manner that can be
reasonably expected by the customers. These expectations are defined in the software
requirements specification which contains the visible attributes of the software.

A test-plan outlines the classes of tests to be conducted and a test procedure defines
specific test cases. Both the plan and procedure are designed to ensure that all
performance requirements are attained, documentation is correct and usability
requirements are met.

An important element of validation process is a configuration review. The purpose of

the review is to test the software configuration and the necessary details to help in the
support phase.

Alpha and beta testing-

Acceptance testing- When customer software is built for a customer, a series of

acceptance tests are conducted to enable the customer to validate all requests. An
acceptance test can range from an informal test drive to a planned and systematically
executed series of tests.

Alpha testing- Alpha testing is conducted at developer’s site by end users.

Beta testing- The beta testing is conducted at end user’s site. Beta testing is a live
application of the software in an environment that cannot be controlled by the
developer.

System Testing-

It involves testing everything including hardware, software, people and information.

System testing is actually a series of different tasks whose primary purpose is to fully
exercise the computer-based system. Although each test has a different purpose all
work is to verify that system elements have been properly integrated and perform
allotted functions. It involves the following steps-

a) Recovery testing- Recovery testing is a system test that forces the software to fail
in a variety of ways and verifies that recovery is properly performed. Automatic
recovery, reinitialization, checkpointing mechanisms, data recovery and restart
 are evaluated for correctness. If recovery requires human intervention, the mean
time to restart (MTTR) is evaluated to determine whether it is within acceptance
limits.

b) Security testing- Security testing verifies that protection mechanisms built into the
system will infact protect it. The tester may attempt to acquire passwords through
external means, may attack the system with custom software designed to break-
down any defence that may have been constructed, may overwhelm the system
denying service to others, purposely cause system errors, may browse through
insecure data.

c) Stress testing- Stress testing executes a system in a manner that demands

resources in an abnormal quantity or frequency or volume. For example special
tests may be designed that generate ten interrupts per second, input data rates may
be increased, maximum memory utilization tests may be done, excessive disk
reading tests may done.

d) Performance testing- Performance testing is done to test the run time performance
of the software within context of the integrated system. It is important to monitor
execution intervals, log events and sample machine states on regular basis.

e) Debugging- Debugging occurs as a consequence of successful testing. When a

test case uncovers an error, debugging is an action that results in the removal of
the error. The debugging process begins with the execution of a test-case.
Debugging strategies include brute force (using memory dumps and output),
backtracking (step wise tracing), and cause elimination (formulating a reason and
testing it).

Testing Tactics

Definition- Software testability is how easily a software program can be tested.

Characteristics for testing a system-

a) Operability- If a system is designed and implemented with quality in mind,

relatively few bugs will block the execution of tests, allowing testing to
progress without fits and starts.

b) Observability- Inputs provided as part of testing produce distinct outputs

c) Controllability- The better we can control the software the more the testing can be
automated and optimized. Controlling the software and hardware variables is
important.

c) Decomposibility- The software system is built from independent modules that can
be tested independently.
 d) Simplicity- There should be functional simplicity , structural simplicity and
Code simplicity.

e) Stability- Changes to the software are infrequent , controlled when they occur and
do not invalidate existing tests.

f) Understandability- The architectural design dependencies between external and

, internal and shared components are well understood.

Black Box and White Box testing-

Black Box test examines some fundamental aspect of a system with little regard for the
internal logical structure of the software.

White Box testing involves examination of the procedural details, logical paths through
the software and collaborations between components are tested by providing test-cases
that examine specific set of conditions and or loops.

1. White Box Testing- White box testing sometimes called glass-box testing is a test
design philosophy that uses the control structure described as part of component-
level design to derive test cases. Using the white box testing method the software
engineer can derive test-cases that cover areas like check all independent paths,
exercise logical decisions, execute loops and use internal data structures.

White-Box testing methodologies

a) Basis-path testing- Basis path testing is a testing technique proposed by
McCabe. For basis path testing we use-
Flow Graph- It depicts logical control flow. We can derive a flow graph
from a flow chart. Each statement is a node and control structures control
the flow path.
b) Independent Path-n independent path is any path through the program that
introduces at one set of processing statements. It can be derived from a
flow-graph. For example
Path 1 : 1-11
Path 2 : 1-2-3-4-5-10-11
c) Cyclomatic complexity- It is a software metric that provides a quantitive
measure of the logical complexity of a program. Cyclomatic complexity
defines the no. of independent paths in the basis set of a program and
provides an upper bound on the number of tests. It can be computed as
No. of regions
V (G) = E-N+2 where E is no of edges and N is no of nodes
d) Deriving test cases- It involves steps like- drawing a flow graph from a
flow chart, determining cyclomatic complexity, determine set of linearly
independent paths and determine test cases for each set of paths.
 e) Graph Metrics- To prepare a software tool we can store the nodes of flow
graph and compute the independent paths to derive test cases.
f) Control structure testing- It involves condition testing in which a test case
is designed that check the logical condition.

It involves data flow testing which is a method that selects test paths of a
program according to location of definition and use of variables.

A definition-use (DU) chain of a variable X is {X, S, S’} where definition

of variable x in statement S is live at statement S’ i.e. there exists a path
from S to S’. All definition-use chains should be covered at least once.

It also involves loop-testing that focuses exclusively on loop constructs

covering different types of loops like simple loop, nested loop,
concatenated loop and unstructured loops.

2. Black Box Testing-

Black Box testing also called behavioral testing focuses on the functional
requirements of the software. Black box testing attempts to find errors in the
following categories- incorrect or missing function, interface errors , errors in data
structure, performance or behavior errors, initialization or termination errors.

a) Graph based testing methods- A graph is created which is a collection of

nodes that represent objects, links that represent relation between objects
, node weights that describe properties of a node, link weights that
describe some characteristics of a link. This link can be a directed link,
symmetric link, bidirectional link or parallel link.
Objects to be tested can be document window, document text etc.

b) Transaction flow modeling- It involves finite state modeling, data flow

modeling and timing modeling.

c) Equivalence Partitioning- Equivalence partitioning is a black box testing

technique which divides the input domain into classes of data from which
test cases can be derived. Equivalence partitioning strives to define a test
case that covers a class of users, thereby reducing the total number of test
cases that can be developed.

d) Boundary value analysis- A great number of errors occur at the boundary

Of the input domain. BVA leads to a selection of test cases that exercise
boundary values. It complements equivalence partitioning because it leads
to selection of test cases at the edges of a class.
 f) Orthogonal Array Testing- Orthogonal array testing can be applied to
problems in which the input domain is relatively small but too large to
accommodate exhaustive testing. If there are three input X, Y and Z
instead of taking one input at a time, it is possible to consider all different
permutations i.e. 3^3=27. It’s like testing all sides of a cube.

3. Object Oriented Testing Methods-

Object oriented testing is mainly used for testing object oriented testing software.
It involves the following methods-
a) Fault Based Testing- The objective of fault-based testing within an OO s
System is to design tests that have a high likelihood of uncovering
plausible faults. To determine whether faults exist , test cases are designed
to exercise the design or code.
b) Integration Testing- Integration testing looks for plausible faults in
Operation calls or message connections. Integration testing applies to
attributes or operations. Testing should exercise the attributes to determine
whether proper values occur for distinct types of object behaviour.
c) Testing class hierarchy- Tests have to be done for both the base and
Derived classes.
d) Scenario based testing- Scenario based testing concentrates on what the
User does. This means capturing tasks that the user has to perform then
applying them and their variants as tests. Scenarios uncover interaction
errors, but to accomplish these test cases must be more complex and more
realistic than fault based tests.
e) Testing surface and deep structure- Surface structure refers to externally
Observable structure of an OO program that is the structure that is visible.
Deep structure refers to internal technical details of an OO program. That
is the structure that is understood by executing the design and the code.
f) Testing at class level-
Random Testing- If a class contains a number of operations, a variety of
operation sequences may be generated randomly. E.g. for banking
application Test case r1 : open.setup.deposit.close
Partition Testing- Partition testing reduces the number of test cases
required to exercise the classes by partitioning the input domain. Inputs
and outputs are categorized and test cases are designed for each category.
g) Inter-class test case design- Test case design becomes more complicated
As integration of object oriented system begins. It is at this stage that
collaboration of classes must begin.
Multiple-class testing- For each client class use the list of operations to
generate a series of random test sequences. The operation will send
messages to other server classes.
For each message that is generated determine the collaboration classes and
the corresponding operations in the server class.
For each operation in the server object determine the messages that it
transmits.
 For each of the messages determine the next level of operations that are
involved and incorporate these into test sequence.
h) Tests from behavior models- The state diagram for a class can be used to
Help derive a sequence of tests that will exercise the dynamic behavior of
a class and those classes that collaborate with it. The tests to be designed
should achieve all state coverage.

4. Testing for specialized environments- Unique guidelines and approaches are

Required to test different environments.
a) Testing GUI’s- Finite state modeling graphs may be used to derive a
Series of tests that address specific data and program objects involving
GUI.
b) Testing client-server architecture- Testing client-server software occurs
At three levels. i) Individual client applications are tested in a
disconnected mode. ii) Client applications and server software are tested
together. iii) Complete client/server architecture including network
operations and performance are tested. Other tests include application
function tests, server tests, database tests, transaction tests and network
connection tests.
c) Testing documentation and help facilities- Documentation testing can be
Applied in two steps- i) Review and inspection examines the document for
editorial clarity ii) Live test uses documentation in conjunction with the
actual program.
d) Testing for real time systems- Comprehensive test case design methods
For real-time systems continue to evolve. The steps involved are – i) Task
testing tests each task independently. ii) Behavioral testing simulates the
behaviour of real time system and examines its behavior as a consequence
of external events. iii) Inter-task testing tests asynchronous tasks that are
known to communicate with one another and testing them with different
data rates and processing load to determine if inter-task synchronization
errors. iv) Software and hardware are integrated and full range of system
tests is conducted in an attempt to uncover errors at the software/hardware
level.

Product Metrics

1. Software quality- Software quality in conformance to explicitly stated

Functional and performance requirements, explicitly documented development
standards and that are expected of all professionally developed software. A few
important factors should be kept in mind-
a) Software requirements are the foundation from which quality is
measured. Lack of conformance to requirements is lack of quality.
 b) Specified standards define a set of development criteria that guide
The manner in which software is engineered.
c) There is a set of implicit requirements that often goes unspecified e.g.
Ease of use.

2. McCall’s quality factors-

a) Correctness- The extent to which a customer satisfies it’s specification
And fulfills customer’s mission objectives.
b) Reliability- The extent to which a program can expect to perform its
Intended function.
c) Efficiency- The amount of computing resources and code required by a
Program to perform its functions.
d) Integrity- The extent to which access to software or data by unauthorized
Persons can be controlled.
e) Usability- The effort required to learn, operate, prepare input for and
Interprets output of operations.
f) Maintainability- The effort required to locate and fix an error in a
Program.
g) Flexibility- The effort required to modify an operational program.
h) Testability- The effort required to test a program to ensure that
It performs its intended function.
i) Portability- The effort required to transfer a program from one
Hardware/software environment to another.
j) Reusability- The extent to which a program can be reused in another
Application.
k) Interoperability- The effort required to couple one system to another.

3. Quantitative View-
a) Measure- Measure provides a quantitative indication of the extent ,
Amount, dimension, capacity or size of some attribute of a product or
process.
b) Metric- A quantitative measure of the degree to which a system,
Component or process processes a given attribute.
c) Indicator- An indicator is a metric that provides insight into the software
Process, software project or the product itself.
d) Characteristics- Metrics can be characterized by five activities-
i. Formulation – The derivation of software measures and metrics
appropriate for the representation of the software being considered.
ii. Collection- The mechanism used to accumulate data required to
derive formulated metrics.
iii. Analysis- The computation of metrics and application of
mathematical tools.
iv. Interpretation- The evaluation of metrics in an effort to gain insight
into quality.
v. Fault detection- Detect faults using developed metrics.
4. Goal oriented software measurement- GQM emphasizes the need to-
a) Establish an explicit measurement goal.
b) Define set of questions to be answered to achieve the goal.
c) Identify the metrics that help to achieve the goal.

5. Attributes of effective software metrics-

a) Simple and computable
b) Empirically and intuitively persuasive
c) Consistent and objective
d) Consistent in the use of units and dimensions
e) Programming language independent
f) Effective mechanism for high quality feedback

6. Metrics for analysis model

a) Function point metric- It is used for measuring the functionality
Delivered by a system. The metric is used to-
i) Estimate cost or effort for design and testing
ii) Predict no. of errors
iii) Forecast no of computations
It depends on-
i) No. of external input (EI)
ii) No. of external outputs (EO)
iii) No. of external enquiries (EQ)
iv) No. of internal logical files (ILF)
v) No. of external interface files (EIF)
FP= Count * [ 0.5 + 0.01 * Σ Fi ]
Fi= Value adjusted factor depending on other factors
b) Metrics for specifying quality- Analysis model can also be used to test
Completeness, correctness, understandability, verification, internal and
external consistency, achievability, conciseness, traceability,
modifiability,
Precision and reusability.
Nu=Nf+Nnf
Nu= Total requirements
Nf= No of functional requirements
Nnf= No of non-functional requirements
Specificity of requirements Qf= Nui/Nt
Nui= Common or unambiguous requirements
Completeness-
Qz= Nu/ [ Ni * Ns ]
Nu= No. of unique functional requirements
Ni= No. of input stimuli
Ns= No. of state specifications
7. Metrics for design model-
a) Architectural design metrics- These are also called black box metrics and
These do not require knowledge of any software component. The design
metrics are-
i) Structural complexity-
S(i)= f out(i)^2
Fout= No. of outputs
ii) Data complexity-
D(i) = V(i) / [ fout(i) + 1 ]
V(i)= No. of input and output variables
iii) System complexity-
C(i)=S(i)+D(i)
S(i)= Structural complexity
D(i)= Data Complexity
b) Metrics based on tree diagram of module structure-
Size= n + a
n= no of nodes
a= no of arcs
arc to node ratio= a/n
Another form of metric is
S1= total no of modules
S2= no of modules whose correct function depends on source of
Data input
S3= no of modules whose correct function depends on prior
Processing
S4= no of database items
S5= total no of unique data items
S6= no of database segments
S7= no of modules with single entry and single exit
Program structure D1=1 (OOPS) D1=0 (Structural)
Module independence D2= 1- S2/S1
Modules not dependent on processing D3= 1- (S2/S1)
Database Size D4= 1- S5/ S4
Database compartmentalization D5= 1- S6/ S4
Module entry and exit characteristics DSQ = Σ wi Di
c) Metrics for object oriented design-
Size, Complexity, Coupling, Sufficiency, Completeness,
Cohesion, Primitiveness, Similarity, Volatillity

8. Class oriented metrics-

a) Ck metrics suite ( Chidamber and Kemerer)-
i) Weighted methods per class (WML)- Assume that n methods
Of complexity C1, C2,……..Cn are defined for class C. The
specific complexity metric that is chosen should be normalized so
that nominal complexity for a method takes on a value of 1.0.
WMC= Σ Ci
 For i= 1 to n
ii) Depth of inheritance tree (DIT)- This metric is the maximum
Length from the node to the rest of the tree. More the depth of
class hierarchy more is the complexity.
iii) Number of children- The subclasses that are subordinate to a
Class are considered its children. As NOC increases
complexity will increase.
iv) Coupling between object classes- CBO is the number of
Collaborations listed for a class on its CRC index card. As
CBO increases the reusability of a class increases and so does
its complexity.
v) Response for a class (RFC)- It is a set of methods that can
Potentially be executed in response to a message received by
an object of that class. As RFC increases so does design
complexity.
vi) Lack of cohesion in methods (LCOM)- LCOM is the number
Of methods that access one or more of the same attributes. If
LCOM is high so is coupling. Cohesion should be high where
as LCOM should be low.

b) MOOD metrics suite- Metrics for OO design are quantitative in nature.

i) Method inheritance factor (MIF)- It is the amount of inheritance
Used.
MIF= Σ Mi(Cj) / Σ Mi(Cj)
Ma(Cj)= Md(Cj) + Mi(Cj)
Ma(Cj)= the number of methods that can be involved in the
association with Cj.
Md(Cj)= the number of methods declared in class Cj
Mi(Cj)= the number of methods inherited in Cj.
ii) Coupling Factor (CF)- Coupling can be defined in the
following way-
Cf= Σ iΣ j is_client(Ci,Cj) / ( Tc^2-Tc)
I= 1 to Tc and j= 1 to Tc
Is_client=1 if relationship exists between Ci and Cj
=0 otherwise
c) OO metrics by Lorenz and Kidd
i) Class Size- no of operations and no of attributes NOA.
ii) Component level design metrics- It involves five concepts
and measures. Data Slice ( data values), Data tokens
( variables) Glue tokens ( attached to data slice) SuperGlue
tokens (attached to every data slice) Stickiness ( no of data
slices bound)
iii) Coupling metrics- Dhama proposed a metric for module
Coupling that encompasses data and control flow, global
coupling and environmental coupling. The formula is-
M= Di + ( a * Ci) + D0 + ( b * C0 ) + Gd + ( c * Gc) + w + r
 Di= no of input data parameters
Ci= no of input control parameters
D0= no of output data parameters
C0= no of output control parameters
Gd= no of global variables used as data
Gc= no of global variables used as control
W= no of modules called or fan-out
R= no of modules calling the modules under consideration

d) Complexity Metrics- The most important complexity metric is

Cyclomatic complexity. Operation oriented metrics proposed are-
i) Average operation size (OSavg)- Although lines of code
Could be used as operation size, the LOC measure suffers
from a set of problems.
ii) Operation complexity (OC)- The complexity of an
operation can be computed using any of the complexity
metrics as in conventional software.
iii) Average no of parameters per operation (NPavg) - The
larger the number of operation parameters, the more
complex the collaboration between objects.
iv) User interface design metrics- A typical GUI uses layout
entities- graphic icons, text, menus, windows etc.The
metrics are the time required to achieve a specific operation,
no of operations required, no of data or content objects.

9. Metrics for source code-

Metrics for source code include
Program Length N= n1 log n1 + n2 log n2
n1= no of distinct operations
n2= no of distinct operands
N1= total no of operator occurrences
N2= total no of operand occurrences
Program Volume V = N log ( n1 + n2 )

10. Metrics for testing-

Program Level PL= 1 / [ n1/2 * N2/n2 ]
Effort L = V/PL
Metrics for object oriented testing include- lack of cohesion in methods
(LCOM),
Percentage of public and private variables (PAP), public access to data members
(PAD), no of root classes (NOR), fan in (FIN), no of children (NOC), depth of
inheritance (DIT)