Software Reliability

Models
2
Reliability Models
usage
assessing product reliability
estimating the number of latent defects at the
time when the product is released
helps planning resources for maintenance
the variable being studied
the defect number/rate in a time interval
the time between failures
3
Expression
the variable being studied has a form as
below
y=f(x
1
,x
2
,...,x
k
)+e
y - dependent (studied) variable
x
i
- independent variables
attributes of the product/development process
e.g., the complexity of the project, the team's
training level, etc.
e - error
4
Static and Dynamic Models (1)
static models
the coefficients of the independent variables are
estimated based on data from previous
processes
dynamic models
the coefficients of the independent variables are
estimated based on data collected from the
current process
thus, the resulting model is product-specific
5
Static and Dynamic Models (2)
static models are better suited for testing
hypotheses rather than estimating reliability
dynamic models - 2 categories
applicable for the whole development process
e.g., the Rayleigh model
applicable for the testing phases
e.g., reliability growth models
The Rayleigh Model
7
The Weibull Distribution (1)

8
The Weibull Distribution (2)
m - shape parameter
c - scale parameter
t - time
in theory, may be any independent variable
standard distribution - the area below the
probability density curve is 1
the probability density curve approaches 0
asymptotically
9
Usage
many reliability processes evolve according
to the Weibull distribution
corresponding to various values of m
examples: electron tube failures, river
overflows, etc.
software engineering
CDF - cumulative defect arrival pattern
PDF - defect rate (density) over time
10
The Rayleigh Model
particular case of the Weibull distribution
m = 2
2
c
t
m
=
probability density grows until a moment t
m
,
then decreases asimptotically to 0
11
Usage in Software Engineering (1)
in most cases, the life cycle of a software
project observes the Rayleigh distribution
including the defect removal pattern
in practice, the probability functions are
multiplied by a factor K
where K is the total number of defects
so the area below the probability density curve
is K
12
Example




GA - General Availability phase
when the product is available for customers
we are interested in estimating the defect rate
during this phase
13
Assumptions
applying the Rayleigh model in software
quality relies on two basic assumptions
1. the defect rate observed during the
development process is positively
correlated with the defect rate in the field
2. if more defects are discovered during the
early phases, fewer defects remain in the
later phases
14
Assumptions (continued)
assumption 1
a higher rate of defects discovered means that
more defects were introduced
if defect removal effectiveness remains the same
thus, it is normal that more defects are revealed
in the field as well
assumption 2
if the PDF curve reaches the maximum value at
an earlier moment, then the area below the
curve is smaller in the final part (the GA phase)
15
Predictive Validity
refers to the accuracy of the estimates made
based on the model we use
how to achieve it
ensure the accuracy of input data
in many projects, defect monitoring is not well
implemented in the early phases
permanently compare the estimates to actual
results
useful for the subsequent phases
16
Predictive Validity (continued)
the Rayleigh model often underestimates
the defect rate in the field
the value of parameter m may be too big if
input data are incomplete
a Weibull distribution with m less than 2
(but still close to 2) may be better suited for
modeling the software process
e.g., 1.8
Reliability Growth Models
18
Exponential Model
particular case of the Weibull distribution
m = 1
( )
t -t/c
e - 1 e - 1 t F : CDF = =
( )
t - t/c -
e e
c
1
t f : PDF = =
=1/c - instantaneous failure rate
the parameters to be estimated for
describing the model: K (total number of
defects) and
19
Cumulative Distribution Function
20
Probability Density Function
21
Usage
models the defect rate during the testing
phases for many software projects
assumption
the testing effort is homogeneous throughout all
testing phases
if the testing effort is homogeneous, a
normalization procedure needs to be applied
22
Normalization
analyze the number of person-hours per
week allocated for testing
compute its average value (n) for the entire
testing process
divide the testing effort into n person-hours
units
the defect rate for the first n person-hours unit
is assigned to the first week
proceed similarly for the second unit and the
second week and so on
23
Reliability Growth Models
used for testing phases
they describe the evolution of the number of
defects discovered throughout the testing
phases
they are all based on the exponential model
the dependent (studied) variable may be
the time between failures
the number of defect arrivals
24
Time between Failures Models
the time between two consecutive failures
grows as defects are fixed
the time between failures i-1 and i follow a
certain distribution
the parameters of the distribution depend on the
number of latent defects remaining after failure
i-1
25
Jelinski-Moranda Model (1)
assumptions
the number of failures at the start of testing: N
failures occur at random
all defects have equal contributions to failures
the time necessary for fixing a defect is
negligible
each defect fix is perfect
26
Jelinski-Moranda Model (2)
let us consider the interval t
i
between failures i-1 and i
the instantaneous failure rate function
during interval t
i
is of this form

( ) ( ) ( ) 1 - i - N t Z
i
u =
u - proportionality constant
the instantaneous failure rate
is constant between failures
decreases by u after each defect fix
27
Littlewood Model
similar to the Jelinski-Moranda model
but the defects have different contributions
to failures
we say the defects have different sizes
during the testing process we expect a
decrease in
the number of defects
the average size of the defects
28
Goel-Okumoto Imperfect Debugging
Model
also derived from the Jelinski-Moranda
model
but considers that defect fixes may be
imperfect
i.e., they could introduce new defects
( ) ( ) ( ) 1 - i p - N t Z
i
=
p - the probability of imperfect debugging
- the failure rate per defect
29
Fault Count Models
the number of defects in a time interval
decreases as defects are fixed
the time interval used may be
CPU time
calendar: hour, week, month, etc.
the parameter to estimate - the number of
remaining defects
30
Goel-Okumoto Nonhomogeneous
Poisson Process Model (1)
the number N(t) of defects observed by time
t is modelled as a nonhomogeneous Poisson
process
the failure rate depends on time, following
an exponential distribution
( ) { }
( ) | |
( )
0,1,2... y , e
y!
t m
y t N P
t m -
y
= = =
( ) ( )
-bt
e - 1 a t m =
( ) ( )
-bt
abe t m' t = =
31
Goel-Okumoto Nonhomogeneous
Poisson Process Model (2)
parameters
m(t) - the expected number of failures observed
by time t
(t) - failure density
a - the expected total number of failures to be
observed
b - defect detection rate
32
Goel-Okumoto Nonhomogeneous
Poisson Process Model (3)
the model is an application of the
exponential model
m(t) corresponds to the cumulative distribution
F(t)
(t) corresponds to the density probability f(t)
parameters a and b correspond to parameters K
and , respectively
parameter a is treated as a random variable
33
Musa-Okumoto Model
similar to the previous model
the later a defect is fixed, the smaller effect
it has on the reliability of the software
it is thus a logarithmic Poisson process
( ) ( ) 1 ln

1
t m
t 0
+ =

0
- initial failure intensity
u - rate of reduction of failure intensity
34
Delayed S Model
considers the time necessary for failure
analysis
the curve of the cumulated number of defects is
S-shaped
the maximum value is not reached at start, as in the
exponential model
still a nonhomogeneous Poisson process
( ) ( ) ( )
t -
e t - 1 - 1 K t m

=
- error detection rate
K - total number of defects
35
Inflection S Model
errors are no longer considered as
independent
the more failures are detected, the more
undetected failures become detectable
again, a nonhomogeneous Poisson process
( )
t -
t -
ie 1
e - 1
K t m

+
=
i - inflection factor
36
Model Evaluation Criteria
predictive validity
the ability to predict the failure behavior or the
number of defects based on currently available
data
simplicity
collecting data - simple and inexpensive
the concept does not require extensive
mathematical background
easy to implement by computer programs
37
Model Evaluation Criteria (cont'd)
quality of assumptions
the assumptions are plausible and likely to be
met in practice
capability
the ability to estimate quantities that are useful
for managers, software engineers, users, etc.
applicability
the model can be equally applied on projects
that differ in size, structure, functions, etc.
38
Reliability Modeling Process (1)
data examination
the kind of available data: number of defects vs.
time between failures
the time unit that is used (CPU time, week, etc.)
data reliability
select one or several models that fit the
available data
estimate the parameters of the model
39
Reliability Modeling Process (2)
test to which extent the model (with the
estimated parameters) fits the data
if the model does not fit the data, another model
must be chosen
make reliability predictions based on the
model
predictions are assessed based on other
information
e.g., through the analysis of similar products
40
defect arrival patterns differ between the
testing phases and the maintenance phase
testing phases are specifically focused on
finding defects
in the field, defects arise more rarely
thus, defect density is different
such diference is called compression factor
Test Compression Factor
41
Test Compression Factor (continued)
the value of the compression factor may
vary depending on
the testing environment
the customer usage profile
if the software is more complex, defects
tend to arise later
so the curve of the probability density with
respect to time exhibits a slower evolution