2006 CSTE CBOK –

Skill Category 1 (part 1)

9 March, 2006
Presenter:
Donovan Bean
Agenda

Vocabulary
Why Do We Test Software?
Multiple roles of the Software Tester
Vocabulary

Knowledge of a profession and the

ability to convey the knowledge
unique to a profession to others
Quality Assurance versus Quality
Control
Cost of Quality
Software Quality Factors
Quality Defined
Quality

Working definitions:
Producer’s view – Quality of product
meets requirements
Customer’s view – Quality of product “fit
for use”/ meets customer’s needs
 Quality Assurance vs. Quality Control
Preventive methods
Planned, systematic activities necessary to
provide adequate confidence that products
and services will conform to specified
requirements and meet user needs
Staff function, responsible for implementing
the quality policy defined through the
development and continuous improvement
of software development process
Activities that establish and evaluate the
processes that produce products
Measures processes to identify
weaknesses, then correct these
weaknesses to continually improve the
process
Relates to all products or services that will
ever be created by a process
Helps establish processes and
measurement programs to evaluate
processes
Identifies weaknesses in processes and
improves them
Responsibility of management
Detective methods
Process where product quality compared to
applicable standards with action taken
upon detection of nonconformance.
Line function where work is done within a
process to ensure that the work product
conforms to standards and requirements
Activities that focus on identifying defects in
the actual products produced.
Activities begin at the start of the software
development process wit reviews of
requirements, and continue until all
application testing is complete
Relates to specific product or service
Verifies whether specific attributes are or
are not in a specific product or service
Identifies defects for the primary purpose of
correcting defects
Responsibility of team/worker
Quality Assurance vs. Quality Control (cont.)

QA = QC over QC because it
evaluates whether QC is working
QA staff don’t do QC except to
validate that the QC process is
working
The Cost of Quality
All costs above producing the product
right the first time.
Quantifies total cost of prevention,
appraisal & production of software
Includes costs of assuring delivered
product meets established quality
goals
Includes costs of preventing,
identifying & correcting defects
The Cost of Quality (cont.)
Prevention Costs
Establishing methods & procedures
Training
Tool acquisition
Quality planning
Costs incurred prior to build
Appraisal Costs
Comparing completed product to requirements
Inspections
Testing
Reviews
Costs incurred after build but prior to ship or moved to
production
Failure Costs
Costs associated with delivering a defective product to user or
deploying to production
Repairing products to meet requirements
Cost of operating faulty product
Damage caused by faulty product
Help desk operations
The Cost of Quality (cont.)
Identification and correction of defects account for
a majority of the Cost of Quality
Production costs can be minimized by focusing on
preventing defects
Eliminate rework and build inspection into the
process to attain the goal of optimizing the
production process
Continuous testing applied to the development
process can reduce the cost of quality
The IT QA group is responsible for identifying,
quantifying and developing programs that
minimize prevention, appraisal & failure costs
Software Quality Factors
Attributes of a system that pose a business risk if
desired and absent
When defining your testing scope, risk factors are
the basis and/or objective of your testing efforts
Testing software quality factors are the objectives
of many of these tests
Improving the quality of the software product is the
objective of applying these quality factors to a
software development program
Once determined, the software quality factors
need to be communicated to the development
team, documented. The CBOK recommends
conducting a briefing to communicate the inclusion
of the quality factors
Software Quality Factors (cont.)
Correctness
Reliability
Efficiency
Integrity
Usability
Maintainability
Testability
Flexibility
Portability
Reusability
Interoperability

Examples/definitions from participants…

CBOK Definitions on pp. 47
Software Quality Factors (cont.)
Identify the important software quality factors using the Software
Quality Factor Survey (pp.47) soliciting decision makers for input
Procedures for completing the survey:
Consider basic characteristics of the application
Each system is unique
Evaluate basic characteristics Fig. 2 pp. 48
Consider life cycle implications
Quality factors can be grouped by 3 life cycle activities
– Product operation
– Product revision
– Product transition
Perform trade-offs among the tentative list of factors
Consider the interrelationships
Some synergistic, some conflicting
Identify the most important factors
Reorganize list by importance
Provide explanations for the choices
Document your rationale for the decisions made
Review the example on pp. 50
How Quality is Defined
Based on customer satisfaction
Must be defined before it can be realized
Corporate sponsorship a must
5 perspectives of quality
Transcendent – I know when I see it
Product-based – Desired features included
User-based – Fit for use
Development/manufacturing-based – Conforms
to requirements
Value-based – Cost is acceptable
How Quality is Defined (cont.)
Patrick Townsend’s Quality View
Quality in Fact Quality in Perception
Doing the right thing Delivering the right product

Doing it the right way Satisfying our customer’s

needs

Doing it right the first time Meeting the customer’s

expectations

Doing it on time Treating every customer with

integrity, courtesy and
respect
More Definitions of Quality
Meeting customer’s requirements the first time, every time
Conformance to requirements set
More than absence of defects
Requires controlled process improvement
Only achieved by continuous improvement of all systems
and processes in an organization:
Production
Design
Development
Service
Purchasing
Administration
Customer interactions
Quality can only be seen through the eyes of a customer
Excellence
Excellence is the measurement or degree
of quality
Definitions of quality and excellence a
starting point for quality policy implemented
by management
Management must agree on a definition of
quality and the degree of excellence
desired
The customer is the most important person
in any process
Why Do We Test Software?
Six concepts that explain why we test
software
Developers are not Good Testers
What is a Defect?
What is Quality Software?
Why Does a Development Process Produce
Defects?
Reducing the Frequency of Defects in Software
Development
An effective development process that
minimizes defects
How is quality defined?
Developers are not Good Testers

Disadvantages to checking your own work:

Misunderstandings not detected
Improper use of development process not
detected (may not be understood)
Developer “blinded” into accepting bad system
specs and coding falling into same trap that
introduced the defect
IT folks overly confident in ability to produce
defect-free work
Developer may be tempted to improve system
instead of dedicating time to testing
What is a Defect

A defect is an undesirable state

Must define desirable state before
comprehending an undesirable state
Quality = desired state
To understand a defect we must
understand quality
What is Quality Software?
IT’s view: Requirements Met
Customer’s view: Fit for use
Two software quality gaps
User’s view of quality
^
User gap
v
Producer’s view of quality as specified
^
IT gap
v
IT’s view of quality as delivered
What is Quality Software? (cont.)
Potential role of software testing is to help close the gap
between what is specified and what is to be delivered (IT
gap) and the gap between what is delivered and what is
desired (user gap)
The McDonald’s effect
Closing the IT gap helps in providing consistency to the
customer. A BigMac looks and tastes the same in Jersey as it
does in Munich, New York and Paris
Understanding the needs of the customer helps close the
user gap
Customer surveys
Joint application development
More user involvement during build
Software testing goals must be defined. One must ensure
that the software meets requirements and is considered fit-
for-use by the customer
Why Does a Development Process
Produce Defects?
Ideally, a software development process should
produce the same output each iteration of a
process
Variability is the “enemy” of quality
Concept of maturing a development process is to
mitigate or reduce variability
The process of measuring and reducing variability
is statistical process control (SPC)
Variation is quantified with summary statistics
using mean and standard deviation.
If a process remains constant and outputs are
within the acceptable, set variance it is said to be
in control
Sources of Variation
Measurement components
Counting
Sampling
Material components
Forms
Suppliers
Machine components
Office equipment
Computers
Software
Method components
Procedures
Policies
Accounting practices
People components
Training
Experience
Attitude
Aptitude
Environment components
Temperature
Humidity
Noise level
Lighting
Special Causes of Variation
Special causes not typically present in the
process
If exist, the process is unstable
Special causes must be eliminated to bring
the process into a state of statistical control
Read Brian Joiner’s input on:
Special causes of variation pp. 58
Strategy for eliminating special cause variation
pp. 59
Common causes of variation pp. 59
Strategy for reducing common causes of
variation pp. 60
Variation (cont.)
Those working in the process (employees) responsible for
reduction of special causes of variation.
Those working on the process (managers) responsible for
reduction of common causes of variation
Reducing common causes requires process or system
changes
85-94% of problems in an organization are system problems
Concept of statistical control allows us to determine which
problems are common causes and which are external or due
to special causes of variation
A stable process may not be acceptable
A process is “incapable” if its variation due to common
causes results in operation of the process beyond
specifications and must be improved
Tampering

Deming definition:
“Action taken on a stable system in
response to variation within statistical
control, in an effort to compensate for
this variation – the result of which will
inevitably increase the variation and will
increase cost from here on out”
Process variation is expected and is
not a reason for tampering
Reducing the Frequency of Defects
in Software Development
DOD’s attempt to make software
development more effective relying on
research from Carnegie Mellon University’s
Software Engineering Institute (SEI)
resulted in the Capability Maturity Model
Defines five levels of maturity as a process
moves from level 1 to 5, the variability in
the process is reduced
For a level 5 organization, the cost
difference of producing a function point of
logic is 100 times that of the cost for a level
1 organization
Five Levels of Process Maturity in a
Quality Management Environment
Multiple Roles of the Software Tester

People Relationships
Scope of Testing
When Should Testing Occur?
How the Test Plan Should be
Developed
Testing Constraints
People Relationships
Attitudes that shape a negative view of testing and testers:
Testers hold up implementation
Give them less time and reduce the chance of finding defects
Testers as debuggers
Testers to blame for defects found in production
Testers don’t need training, only developers do
Variables that affect the testing process:
The development process
Software risk
Customer/user participation
Testing process
Tester’s skill set
Use of tools
Testing budget
Resource constraints

People side of software testing has been ignored over the more process
related issues like test planning, test tools, defect tracking etc.
People Challenges
Training
Relationship building
Tool use
Getting managers to understand testing
Communicating to users about testing
Making time for testing
Testing “over the wall” software
Trying to hit a moving target
Fighting a lose-lose situation
Having to say “no”
Essential Testing Skills
Test planning
Automated and manual tool use
Test execution
Defect management
Risk analysis
Test measurement
Designing a test environment
Designing effective test cases
Solid testing vernacular
Understand what to test
Understand who does what types of tests
Understand when to test
Understand how to test
Understand when to stop testing
Broadened Scope of a Tester
Test to determine whether or not the user’s
needs have been met regardless of the
specifications
Find defects early in the software
development process where they’re
cheaper to fix
Remove defects of all types prior to going
into production
Identify weaknesses in the development
process so the processes can be improved
and thus mature the process
When Should Testing Occur?
Too often testing after coding is the only verification technique used
to determine the adequacy of a system
If testing constrained to a single phase at the later stages of
development, testing can consume 50% of the development budget
An error found toward the end of the life cycle must be paid for 4
times:
1st – develop erroneously
2nd – test to detect the error
3rd – wrong specs, coding and docs replaced with correct versions
4th – system must be retested
Incorporating testing at each phase lowers cost and increases
quality
2/3rds of all detected system errors attributed to errors in design
phase see Table 5. Life Cycle Verification Activities (pp. 68) for the
recommended testing process
When Should Testing Occur?
(cont.)
Table 5. Life Cycle Verification Activities (pp. 68):
When Should Testing Occur?
(cont.)
Activities that should be performed at each
phase of the life cycle:
Analyze the structures produced for internal
testability and adequacy
Generate test sets based on the structure of the
phase
Additional steps for design and
programming:
Determine that the structures are consistent
with structures produced during previous
phases
Refine or redefine test sets generated earlier
Developing the Test Plan
Figure 9. Example of a High-Level Test Plan (pp.
Select and rank test 71)

objectives
Identify the system
development phases
Identify the business
risks associated with
the system under
development
Place risks in the
matrix
Testing Constraints
Budget and schedule
Incomplete statement of work
Changes in technology (pp. 75)
Limited tester skills
Software risks (pp. 77)
Software defects
Software design defects
Data defects
Finding defects (why hard to find, and
circumstances causing defects pp. 79)