2008 15th Asia-Pacific Software Engineering Conference

Early Estimate the Size of Test Suites from Use Cases

Qu Yi, Zhou Bo, Zhu Xiaochun
College of Computer Science, Zhejiang University, Hangzhou, P.R. China
yibear@zju.edu.cn
{bzhou, alwaysforyou}@zju.edu.cn

Function Point Analysis (FPA) [2] are both famous

models for system sizes estimation. However, both of
them focused on software size and development effort
instead of testing size. Only few models focused on
testing effort estimation for high-level effort not to
mention individual test cases.
This paper introduces the early estimation of test
suites size from test case number prediction. In this
paper, an empirical study on early estimation of testing
size is presented. The key point for the estimation is
the test case number prediction model [TCMP].
This paper is organized as follows. In Section 2, an
overview of related works on existing models for
software size and their applications are given. In
Section 3, use case verification points and test case
number prediction model are introduced in detail. In
Section 4, a results obtained will be analyzed and
evaluated. Finally, conclusion and future work will be
presented in Section 5.

Abstract
Software quality becomes an increasingly important
factor in software marketing. It is well known that
software testing is an important activity to ensure
software quality. Despite the important role that
software testing plays, little is known about the
prediction of test suites size. Estimation of testing size
is a crucial activity among the tasks of testing
management. Work plan and subsequent estimations of
the effort required are made based on the estimation of
test suites size. The earlier test suites size estimation
we do, the more benefit we will get in the process of
testing.
This paper presents an experience-based approach
for the test suites size estimation. The main findings
are: (1) Model of use case verification points. (2)
Linear relationship between use case verification
points and test case number. The test case number
prediction model deduced from the data of real
projects in a financial software company.

2. Related Work
Over the past two decades, several research surveys
focusing on software project effort and schedule
estimation have been proposed [4]. Lots of classic
models and popular techniques can be found in [1], [2]
and [3]. A few classic and related models or
approaches are presented here.
The first one is Line of Code [1] which is one of
most famous estimation models. It is a software metric
used to measure the size of a software program by
counting the number of lines in the text of the
programs source code. LOC is typically used to
predict the amount of effort that will be required to
develop a program, as well as to measure
programmers productivities once the software is
finished. Since it is designed to estimate the cost of
developing system, it is hard to directly adopt this
model to testing effort estimation especially in the

1. Introduction
Software testing is an important activity for
ensuring the software quality to be high. More and
more companies have realized that it is critical and
important to ensure products quality to conform to the
clients expectations. In software life cycle, testing
evaluates the product to ensure that it meets the actual
purpose.
Many organizations assign a separated department
for testing nowadays. Test managers take
responsibility for plans schedules and resources. They
should be able to estimate the testing size for the
purpose of implementing software testing management
activities.
Quite a lot models and techniques were created for
estimating the size, complexity, effort and cost of
software development. Line of Code (LOC) [1] and

1530-1362/08 $25.00 2008 IEEE

DOI 10.1109/APSEC.2008.62

487

Authorized licensed use limited to: Zhejiang University. Downloaded on September 17, 2009 at 11:59 from IEEE Xplore. Restrictions apply.

limitations. The disadvantage is that the measure of

size is based on how well test suites are prepared. It
cant predict the size of testing in the early stage.
Inspired by the researches and surveys mentioned
above, we conducted Test Case Number Prediction
model to estimate test suites size in an early stage.

early stage. It gives us the idea about the estimation of

the test suites size.
Functional Size Measurement (FSM) methods
intend to measure the size of software by quantifying
the functional user requirements. A FSM method holds
a logical external view of the software from the users
perspective and evaluates the amount of functionalities
to be delivered. The most commonly used FSM
method is Function Point Analysis (FPA). FPA [2] is a
model-based estimation method. It measures the size of
applications by calculating the complexity of system
functionalities and counting the function points (FP).
The ISO/IEO organization has recognized FP potential
by creating the functional size metric standard [5].The
measurement is then used for estimating the effort to
develop it. Capers Jones proposed a testing estimation
model based on FPA [6]. In his method, the number of
test cases can be determined by function points. The
formula is
1.2
NumberOfTestCases = (FunctionPo int s )
Then the effort in personal-hours is calculated based
on a conversion factor obtained from historical data.
Although FP is a proved size metric, several authors
have found it necessary to adjust function points to
present development contexts, among Karner[7], and
Anda et al.[8], who worked with use case points;
Harput et al.[9], Fetcke et al.[10], and Uemura et
al.[11], who separately calculated function points from
object-oriented models. One of the most serious
practical limitations of existing FPA methods in FSM
is difficult to calculate.
The Use Case Point Analysis (UCP) [3] is an
extension of FPA and estimates system sizes based on
use case specifications. Suresh Nageswaran reported a
test effort estimation using UCP [12]. In the method,
technical complexity factors for testing are proposed
and calculated as the technical and environmental
factors to adjust Unadjusted Use Case Point (UUCP).
Then a conversion factor is introduced to estimate
testing efforts to be multiplied by the Adjusted Use
Case Point (AUCP). UCP-based methods take testing
activities including test planning, test design, test
execution and test reporting as a whole, but it doesnt
give any estimation for each individual test unit such
as a test case or a test suite.
In 2007, Eduardo Aranha and Paulo Borba reported
an estimation model for test execution effort based on
the test specifications [13]. In the model, they defined
and validated a measure of size and execution
complexity of test cases. This measure is obtained
from test specifications written in a controlled natural
language [14]. Every step in every test case is analyzed
based on a list of characteristics to obtain its execution
points. The model seems great but still there are some

3. Test Case Number Prediction from Use

Cases
In software engineering, a test suite, also known as
a validation suite, is a collection of test cases that are
intended to be used as input to a software program to
show that it has some specified set of behaviors. Test
suites are used to group similar test cases together. A
test suite is designed according to the functionalities of
a use case. The question is whether there is any
relationship between the size of a use case and the size
of a test suite. If the answer is yes, test case number
will be predicted. It is useful for testing size and test
effort estimation.

Figure 1. A big picture of the model

In this section, we will present an experience-based
approach on use case verification points model (UCVP)
which measures use case by counting each individual

488

Authorized licensed use limited to: Zhejiang University. Downloaded on September 17, 2009 at 11:59 from IEEE Xplore. Restrictions apply.

applicable and where they are implemented. The

structure of use case is shown in Figure 2 Use Case.

use case and test case number prediction model based

on UCVP.
This paper presented a model for use case
verification points and test case number prediction
based on UCVP. The big picture for the model is
shown in Figure 1.

3.1 The Size of Use Case versus Test Cases

Number
3.1.1 Theoretical Grounding of Use Case
Use function points (FP) to develop a method to
measure size is because Unified Modeling Language
(UML) is widely used and that the object oriented
paradigm is extensively accepted. The advantages are
that the use case textual description is written early in
the lifecycle of a system or application, and many
people are using them [15].
Jacobson defines the use case as a special way to
use the system by running some part of its
functionality; each use case constitutes a complete
string of events triggered by an actor and it specifies
the interaction, which takes place between the actor
and the system [16]. With regard to measure the size of
use case by estimating the weight of each use case,
transactions and entity objects were used as size
notions [15].
In the use case model, the client requirements are
defined. A list of use cases-which defines the scope of
the application may be written. Then classified
descriptions of each use case will be made. With this
information, the use case designer defines the GUI
specification, the inputs and outputs of the system.
Finally, draw the entity relationship model madel. All
of them are the advantages of utilizing the use case
model as s resource for size calculation.
From the system perspective, a transaction is the
key in the complete flow of the system. In fact, one use
case can have one or more transactions, depending on
the complication of the use case and the criterion used
by the person who writes the use cases. Karmer took
this into consideration in his definition of use case
points, where he defines the size of use cases in terms
of the number of transactions in them [15]. Meanwhile,
entity objects are taken into account as key elements
which handle the persistent information for business
flow in use case model. In addition, special
requirements are inevitable elements in use case model.
The special requirements do not only include defined
transactions but also composed of entity objects. They
are specific to a use case and applicable to the specific
use case. Use cases will clearly refer to SRs that are

Figure 2. The Structure of Use Case

3.1.2 Theoretical Grounding of Test Suite

A test suite is a collection of test cases that are
intended to be used as input to a software program to
show that it has some specified set of behaviors. Test
suites group similar test cases together. A test suite
often contains detailed instructions or goals for each
collection of test cases for later system configuration to
meet. It is composed of a set of test cases.
3.1.3 Theoretical Grounding of Test Case
Table 1. Example of a test case written in a

Step
1
2
3
4

controlled natural language

Description
Expected Results
Access login page The login page is
presented
Input user id
The
user id
is
presented
Input password
The password is
presented by * sign
Click Login page
The application will
be displayed

Tests are usually specified in terms of precondition,

procedure (list of test steps with inputs and expected
outputs) and post-condition [17]. Problems such as
ambiguity, redundancy and lack of writing standard
will occur if use natural language for test case design.
All of problems could be avoided using controlled
natural languages. The controlled natural language
(CNL) [14] is a subset of natural language with

489

Authorized licensed use limited to: Zhejiang University. Downloaded on September 17, 2009 at 11:59 from IEEE Xplore. Restrictions apply.

restricted grammar and lexicon which will help to

write sentences written in a more concise and standard
way. Table 1 shows an example of test procedure
written in a controlled natural language defined for
financial software.

As shown in above formula,

Si means the step

th

number of the i transaction, EO means the number

of entity objects, and SR means the number of special
requirements.
Due to our assumption, test case number prediction
can not be derived from UCP. UCVP is just the case
for estimation each individual test unit (test suite or
test case).

3.1.4 Use Case VS. Test Case

Nowadays testing is involved in an early stage of a
project. Once Use cases are prepared and published,
software tester start test suites preparation. From
functional tests perspective, every step of every
business transaction should be verified. Every entity
object in the application system is also required to be
tested. At the same time, special requirements are
required to be verified definitely. To validate and
verify the requirements in a use case, a test suite is
designed accordingly. In a test suite, test cases are
designed for all verification points. System responses
have been defined in use cases and they should be
included in verification. Also, every entity object and
its attributes should be covered in test cases in order to
be verified. Meanwhile, testers should pay more
attention to special requirements in testing process. In
test case design, test steps can be mapped to events
triggered by actors when expected results can be
related to system responses. It conforms to a
transaction which is described in a use case. For each
entity object, a test case is designed to verify the
attributes and rules. And it is considered to validate a
special requirement to meet special business logic as
well.
According to above analysis, we can assume that
test case number is directly correlative with the
measure of transactions, entity objects and special
requirements in a use case.

3.3 Test Case Number Prediction

To validate the assumption, we conduct a study in a
software company who provides IT solutions to top
financial organizations.
The first task is to calculate UCVP. The historical
data are collected from 4 projects which use the mature
use case model with complete information. All the
projects are composed of 63 use cases using the
formula [1] to calculate.
For test case number, it is a straightforward measure
of a test suite. We found out the relationship between
verification points of use cases and number of test case
which are designed to test use cases functionalities.

3.2. Use Case Validation Points

The Use Case Point Analysis (UCP) is an extension
of FPA and estimates system sizes based on use case
specifications. UCP-based methods are applied in the
entire testing process including test planning, test
design, test execution and test reporting. Unfortunately,
UCP doesnt estimate for each individual test suite.
The model to measure use case by counting each
individual use case is called Use Case Verification
Points (UCVP).To calculate the UCVP, we count the
steps for each transaction, the number of entity objects
and the number of other special requirements which
are not covered by transactions and then add the
numbers together.

Figure 3. Test Case Number vs. UCVPs

We discovered linear relationship between UCVP

for corresponding use case and test case number of a
test suite as figured in Figure 3.

4. Evaluation and prediction quality

The magnitude of relative error is used to evaluate
the accuracy of project estimations [18]. It is defined as
the absolute value of the quotient of the difference
between the actual effort and the estimated effort and
the actual effort. We introduce here two metrics.

UCVP = Si + EO + SR

[1]

490

Authorized licensed use limited to: Zhejiang University. Downloaded on September 17, 2009 at 11:59 from IEEE Xplore. Restrictions apply.

MRAE =
Where:
z

technical factors since test case specification is much

similar with requirement specification. This gives great
support to test manager for planning and resource
allocations. Meanwhile, it also ensures that the test
schedule is correct and exact in the every early stage of
the project.
Last is how much does the study cost? We should
separate the tasks into two parts. One is the calculation
of UCVP. In fact, it is easy to count the point after the
calculation rule is ready. The other is to count test case
number for each test suite. It is convenience to count
the case number. To sum up, it doesnt spend too much
time for calculation.
To sum up, it is meaningful to figure out the linear
relationship between UCVP and test case number for
test effort estimation because of low cost and test case
number prediction once use case is ready. However,
quite a few following up sessions could be investigated
including:
z An early estimation approach on test execution
effort
z Metrics for early test effort estimation
z Automation tool development for the model
z Methodology for reduction test suite

Mean Relative Absolute Error (MRAE)

estimatedi actuali

actuali

abs
i

T = number of tests

PRED: Prediction Quality

PRED(q) = k

K is the number of tests whose magnitude of

relative error is less than or equal to q, and n is the
number of tests in evaluation set.
Conte, Dunsmore, and Shen suggest that an
acceptable level of error is something less than or equal
to 0.25 and PRED (0.25) is at least 0.75 [18].
In our calculation method, the MRAE is 0.20 and
the PRED (0.25) is 0.76 which are acceptable
according to original metrics listed above. In this way,
we can predict test case number of a test suite from the
corresponding use case. It helps test manager to
estimate test effort and resource allocation.

5. Conclusion and Future Work

Acknowledgment

This paper presents a model for use case

verification points and test case number prediction
based on UCVP. The results from this study on the
linear relationship between test case number and
UCVP support claims that the test case number
prediction helps early estimation testing size.
However, there are still some open questions which
are worth discussion.
First, why focus on early estimation of testing size?
As we known, test activities are more and more
population in nowadays. Software testing is a
systematical work which generally contains test
planning, test design, test execution, test reporting as
well as test management. And it is an important stage
in software life cycle. How to estimate testing size in
early stage? No matter the FPA and UCA, both of
them dont support planning for each test unit. The
study in this paper supports the size of test suite
estimation at a very early stage as soon as the use case
specifications are prepared.
Second is the advantage of test case number
prediction. About Test Case Number, we believe that
the case number of a suite is correlated with the
requirement size, especially when test cases are drafted
following some standard such as CNL and
requirements are documented in use case format. So it
is possible to estimate the size of a test suite even in
test requirement analysis phase. After calculating
UCVP, we dont need to worry too much about

This work is supported by the Quality Assurance

teams of State Street Technology Zhejiang. Related
projects are collaboratively run by Zhejiang
University, China and State Street Corporation, USA.
The content of this paper represents the opinions
and conclusions of the authors only and do not
necessarily represent the position of State Street
Corporation or its subsidiaries, affiliates, officers,
directors or employees.

References
[1] B. Boehm, E. Horowitz, R. Madachy, D. Reifer, B. Clark,
B. Steece, W. Brown, S. Chulani, and C. Abts, Software
Cost Estimation with COCOMO II., Prentice Hall, 2000.
[2] D. Garmus, and D. Herron, Function Point Analysis,
Measurement Practices for Successful Software Projects,
Addison Wesley, 2001.
[3] Capers Jones, Applied software measurement,
McGraw-Hill, 1996.
[4] Bergeron, F. and J.-Y. St-Arnaud, Estimation of
Information Systems Development Efforts: A Pilot Study.
Information & Management, 1992, pp. 239-254.
[5] ISO/IEC, Functional size measurement, www.iso.org.
[6] Capers Jones, Applied software measurement,
McGraw-Hill, 1996.
[7] KarnerG(1993), Metrics for Objectory, Diploma Thesis,
University of Linkping, Sweden.
[8] Anda B, Benestad H, and Hove S, A multiple-case study

491

Authorized licensed use limited to: Zhejiang University. Downloaded on September 17, 2009 at 11:59 from IEEE Xplore. Restrictions apply.

of effort estimation based on use case point, In: Fourth

international symposium on empirical software engineering
(ISESE2005), IEEE Computer Society, Australia, November
1718 2005, pp. 407416.
[9] Harput V, Kaindl H, and Kramer S, Extending function
points
analysis
to
object-oriented
requirements
specifications, 11th IEEE international software metrics
symposium, Como, Italy, September 1922 2005.
[10] Fetcke T, Abran A, and Nguyen T, Mapping the OOJacobson approach into function point analysis, Technol
Object-Oriented Lang Syst, Tools 23, pp. 192202.
[11] Uemura T, Kusumoto S, and Inoue K, Function points
measurement tool for UML design specification,
Proceedings of the sixth international software metrics
symposium, Boca Raton.
[12] S. Nageswaran, Test effort estimation using use case
points, 14th International Internet Software Quality Week
2001, June 2001.

[13] Eduardo Aranha, and Paulo Borba, An Estimation

Model for Test Execution Effort, 1st International
Symposium on Empirical Software Engineering and
Measurement, 2007.
[14] R. Schwitter, English as a formal specification
language., In Proceedings of the 13th International
Workshop on Database and Expert Systems Applications
(DEXA02), pages 228232, 2002.
[15] Gabriela Robiolo, and Ricardo Orosco, Employing use
case to early estimate effort with simpler metrics,
Innovations Syst Softw Eng, 2008.
[16] Jacobson I, ChristersonM, Jonsson P and vergaard G ,
Object oriented software engineering, Addison Wesley,
Harlow,1992.
[17] P. Jorgensen, Software Testing, A Craftsmans Approach,
CRC Press, second edition, 2002.
[18] Fenton N, and Pfleeger S, Software metrics, PWS
Publishing Company, Boston.

492

Authorized licensed use limited to: Zhejiang University. Downloaded on September 17, 2009 at 11:59 from IEEE Xplore. Restrictions apply.