Experience-Based and Exploratory Testing - Itkonen

How Do Testers Do It?
Experience-Based
Experience
Based and
Exploratory
p
y Testing
g
Ohjelmistojen testaus
15.11.2010
Juha Itkonen
S b
SoberIT
Agenda
Intelligent Manual Testing
E
Experience
i
b
base ttesting
ti
Exploratory testing
Ways of Exploring
Session Based Test Management
Touring
g testing
g
Intelligent Manual Testing Practices

Examples of empirically identified testing
practices
Benefits of Experience Based Testing
Juha Itkonen - SoberIT

2010
2
Research has shown:
Manual Testing
1. Individual differences in
testing are high
2. Test case design
techniques alone do not
explain
l i th
the results
lt
T
Testing
ti that
th t is
i performed
f
d by
b
human testers
Stereotype of manual testing
Executing detailed pre-designed test
cases
Mechanically following the step-bystep instructions
Treated as work that anybody can do

2010
3
In practice, its clear that some testers

are better than others in manual
testingg and more effective at revealingg
defects...
Image: Salvatore Vuono
Experience is invaluable in software

testing
Domain experience
Technical system experience
Knowledge and skills gained in the application domain area

How the system is used in practice, and by whom
Wh t are the
What
th goals
l off the
th users
How is the system related to the customers (business) processes
How the system was built
What are the typical problems and defects
How is the system used and all the details work
How things work together and interact
Testing experience
Knowledge
g of testing
g methods and techniques
Testing skills grown in practice
Juha Itkonen, 2009 SoberIT

4
Software testing
is creative
ti and exploratory
l
t
work
ork
requires skills and knowledge
application domain
users processes and objectives
so
some
e level
e e o
of technical
ec ca de
details
a sa
and
d history
soyo
of the
e app
application
ca o
under test
requires certain kind of attitude

2010
5
Testers
Tester
s Attitude
People tend to see what they want or expect

to see
If you want to show that software works
correctly, you will miss defects
Testers
Tester
s goal is to break
break the software
software
Reveal all relevant defects
Find out any problems real users would
experience in practice
Testing is all about exceptions, special cases,

invalid inputs, error situations, and
complicated unexpected combinations
Photo by Arvind Balaraman

2010
6
Testers
Tester
s Goal
Explore, investigate, and measure

Provide quality related information for
other stakeholders in useful form
Testers attitude is destructive
t
towards
d the
th software
ft
under
d ttest,
t but
b t
highly constructive towards people
Photo by Graur Codrin

2010
7
My viewpoint: Experience Based

Manual testing that builds on the testers

tester s experience
knowledge and skills
Some aspects of testing rely on testers skills

during testing
e.g., input values, expected results, or interactions
Testers are assumed to know what they are doing

Testing does not mean executing detailed scripts
Focus on the actual testing work in practice

What happens during testing activities?
How are defects actually found?
Experience-based and exploratory aspects of software
testing

2010
8
Exploratory Testing is creative testing

without predefined test cases
Based on knowledge
g and skills of the tester
1. Tests are not defined in advance
Exploring with a general mission

without specific step-by-step instructions on how to accomplish the mission
2. Testing is guided by the results of previously performed tests and the

gained knowledge
g
g from them
Testers can apply deductive reasoning to the test outcomes
3. The focus is on finding defects by exploration
Instead of demonstrating systematic coverage
4. Parallel learning of the system under test, test design, and test execution
5. Experience and skills of an individual tester strongly affect
effectiveness and results

2010
9
Exploratory Testing vs. Scripted Testing

ET is an approach
Most of the testing techniques can be used in exploratory way
Exploratory testing and (automated) scripted testing are the ends
of a continuum
Pure scripted
( t
(automated)
t d) testing
t ti
Freestyle
y exploratory
p
y
bug hunting
High level
test cases
Chartered
exploratory testing

10
Manual scripts
Scripted vs. Exploratory Testing
In scripted
p
testing,
g, tests are first
designed and recorded. Then they
may be executed at some later time
y a different tester.
or by
In exploratory testing, tests are

designed and executed at the same
time and they often are not
time,
recorded.
A mental model of the product
Model
M d l off what
h t th
the product
d t iis and
dh
how it
behaves, and how its supposed to
behave
11
Tests
Tests
P d t
Product
James Bach, Rapid Software Testing, 2002
Lateral thinking
Allowed to be distracted
Find side paths
p
and explore
p
interesting
g areas
Periodically check your status against your mission
12
Scripted vs. Exploratory Tests

Mine-field analogy
gy
fixes
bugs

13
Two views of agile testing

eXtreme Testing
Automated unit testing
D
Developers
l
write
it ttests
t
Test first development
Daily builds with unit tests always
100% pass
Functional (acceptance) testing
Customer-owned
C
Comprehensive
h
i
Repeatable
Automatic
Timely
Public
Focus on automated verification

enabling agile software
development
Juha Itkonen
14
SoberIT
Exploratory Testing
Utilizes professional testers skills
and experience
Optimized to find bugs
Minimizing time spent on
documentation
Continually adjusting plans, refocusing on the most promising risk
areas
Following hunches
Freedom,, flexibilityy and fun for
testers
Focus on manual validation
making testing activities
agile
Agenda
E
Experience
i
b
base ttesting
ti
Exploratory testing
Ways of Exploring
Touring
g testing
g

practices

2010
15
Some ways of exploring in practice
Freestyle exploratory
testing
i
Unmanaged ET
Functional
F
ti
l testing
t ti off
individual features
Exploring high level test
cases
Exploratory regression
testing
by verifying fixes or
changes

2010
16
Session-based
exploratory
l
testing
i
Exploring like a tourist
Outsourced exploratory
t ti
testing
Advanced users, strong

domain knowledge
Beta testing
Session Based Test Management (SBTM)

Bach, J. "Session-Based Test Management", STQE, vol. 2, no. 6, 2000.
http://www.satisfice.com/articles/sbtm.pdf
Charter
Time Box
Reviewable Result
D b i fi
Debriefing

2010
17
Session Based Testing

Session-Based
a way to manage ET
Enables planning and tracking exploratory testing

Without detailed test (case) designs
Dividing testing work in small chunks
Tracking testing work in time-boxed sessions
Efficient no unnecessary documentation

Agile it
itss easy to focus testing to most important areas based on
the test results and other information
Changes in requirements, increasing understanding, revealed
problems, identified risks,
Explicit, scheduled sessions can help getting testing done

when resources are scarce
When testers are not full-time testers...

2010
18
Exploring like a tourist

a way to guide ET sessions
Touring tests use tourist metaphor to
guide testers actions
Focus to intent rather than separate
features
This intent is communicated as tours in
different districts of the software
James A. Whittaker. Exploratory Software Testing, Tips, Tricks,

Tours, and Techniques to Guide Test Design. Addison-Wesley, 2009.

2010
19
Districts and Tours
Business district
Guidebook tour
Money tour
Landmark tour
Intellectual tour
FedEx tour
After-hours tour
Garbage collectors tour
Collectors tour
Lonely businessman tour
Supermodel tour
TOGOF tour
Scottish pub tour
Hotel district
Rained-out tour
Coach potato tour
Historical district
Bad-Neighborhood tour
Museum tour
Prior version tour
Tourist district
Seedy district
Saboteur tour
Antisocial tour
Obsessive-compulsive tour
Entertainment district
Supporting actor tour
Back alley tour
All-nighter tour
2010
20
James A. Whittaker. Exploratory Software Testing, Tips, Tricks,

Tours, and Techniques to Guide Test Design. Addison-Wesley,
2009.
Examples of exploratory testing tours

The Guidebook Tour
The Garbage Collectors Tour
Use user manual or other

documentation as a guide
Test rigorously by the guide
Tests the details of important
features
Tests also the guide itself
V i ti
Variations
Bloggers tour, use third party advice as guide

Pundits tour, use product reviews as guide
Competitors tour
Choosing
gag
goal and then visiting
g
each item by shortest path
Screen-by-screen, dialog-bydialog, feature-by-feature,
feature by feature,
Test every corner of the software,
but not very deeply in the details
The All-Nighter
All Nighter Tour
Never close the app, use the

features continuously

21
keep software running

keep files open
connect and dont disconnect
dontt save
don
move data around, add and remove
sleep and hibernation modes ...
Agenda
E
Experience
i
b
base ttesting
ti
Exploratory testing
Ways of Exploring
Touring
g testing
g

Examples of empirically identified
testing practices

2010
22
Empirically observed practices from industry

Testing, not test case pre-design
Practices work on different levels of abstraction
Many practices are similar to traditional test case design techniques
Many practices are similar to more general testing strategies,
heuristics, or rules of thumb

2010
23
Overall strategies
Structuring testing work
Guiding a tester through features
Detailed techniques
Low level test design
Defect hypotheses
Checking
g the test results

2010
24
Overall strategies
Exploring weak
areas
Aspect oriented
testing
User interface
exploring
Exploratory
Top-down
functional
exploring
Simulating
ga
real usage
scenario
Smoke testing
b iintuition
by
i i and
d
experience

2010
25
Data as test
cases
Documentation
based
Exploring highlevel test cases

Checking new
and changed
features
Detailed techniques
Testing input
alternatives
Testing
boundaries and
restrictions
Testing
g alternative
ways
Input
Covering input
combinations
Exploring against
old functionality
Testing to-andfrom the feature
Simulating
abnormal and
extreme situations
Exploratory
Persistence
testing
Comparing with
another application
or version
Comparing within
the software
Comparison
Feature
interaction testing
Checking all the

effects
Defect based
exploring
End-to-end data
check

2010
26
Basic Objectives in Testing Activities

Exploring: Guiding tester through the functionality
Coverage: Selecting what gets tested and what not
Oracle: Deciding if the results are correct
Risks: Detecting specific types of defects
P i iti ti
Prioritization:
S l ti what
Selecting
h t tto test
t t first
fi t

2010
27
<exploratory strategy>
Exploring weak areas
Description: Exploring areas of the software that are weak or

risky based on the experience and knowledge of the tester.
Goal: Reveal defects in areas that are somehow known to be

risky. Focus testing on risky areas.
complicated
coded in a hurry
lots of changes
coders' opinion
testers' opinion
based on who implemented
a hunch...

2010
28
<exploratory strategy>
Top-down functional exploring
Description: Proceeding in testing by first going through typical

cases and simple checks. Proceed gradually deeper in the details
of the tested functionality and applying more complicated tests.
Goal: To get first high level understanding of the function and then
deeper confidence on its quality set-by-step.
set by step.
Is this function implemented?

Does it do the right thing?
I there
Is
th
missing
i i ffunctionality?
ti
lit ?
Does it handle the exceptions and special cases?
Does is work together
g
with the rest of the system?
y
How about error handling and recovery

2010
29
<documentation based strategy>
Using data as test cases
Description: Pre
Pre-defined
defined test data set includes all relevant cases and
combinations of different data and situations. Covering all cases in a predefined test data set provides the required coverage.
Testing is exploratory, but the pre-defined data set is used to achieve systematic
coverage.
coverage
Suitable for situations where data is complex, but operations simple. Or when
creating the data requires much effort.
Goal: To manage exploratory testing based on pre-defined test data.

Achieve and measure coverage in exploratory testing.
Example: Different types of customers in a CRM system

system.
User privileges
Situation, services, relationships
History,
y, data

2010
30
<comparison technique>
Comparing within the software

D
Description:
i ti
C
Comparing
i similar
i il ffeatures
t
iin diff
differentt
places of the same system and testing their consistency.
Goal: Investigating and revealing problems in the
consistency of functionality inside a software; help
decide if a feature works correctly or not.

2010
31
<input technique>
Testing to-and-from the feature

Description:
D
i ti
Test all things that affect to the feature

Test all things that get effects from the feature
Goal: Systematically cover the features

feature s interactions
interactions.
Reveal defects that are caused by a not-the-mostobvious relationship between the tested feature and
other features or environment.

2010
32
Ways of utilizing IMT Practices

Training testers
Guiding test execution
Test documentation and tracking
Test patterns for different
situations

2010
33
Training Testers
T
Testing
ti practices
ti
are good,
d experience
i
b
based,
d
knowledge for intelligent testers
Named and documented
Give common terminology and names that can be used to
discuss how the testing should be done
By learning these practices a novice tester could do

better job
Compared to just go and test around

2010
34
Guiding Test Execution

P
Practices
ti
ttogether
th with
ith a hi
high
h llevell ttestt d
documentation
t ti
can be used as a test design
Tester can choose applicable practices when doing
exploratory testing
More
o e co
conscious
sc ous dec
decisions
so s
Better idea what the tester is actually doing
Easier to maintain focus what am I going to achieve?

2010
35
Test Documentation and Tracking

Testing
g practices can be used in test specifications
No need for detailed descriptions for the tester
Tester knows what to do
Other people know what has been done
Test planning and design can focus on high level

structure and coverage issues
Not to teaching experienced tester how to test ;-)
Example:
p
Use exploring high-level test cases to cover the functionality
Apply Testing input alternatives and Testing boundaries and restrictions
practices for each function
In addition, use Comparing with another version practice to test that new
GUI components correctly provide the existing features
2010
36
Test patterns
T
Testing
ti practices
ti
could
ld b
be ffurther
th d
developed
l
d
Testing pattern will provide set of good testing practices
For a certain testing problem and motivation
With a certain testing goal
Describing
esc b g a
also
so the
e app
applicability
cab y (co
(context)
e )o
of the
e pa
pattern
e

2010
37
Agenda
E
Experience
i
b
base ttesting
ti
Exploratory testing
Ways of Exploring
Touring
g testing
g

practices
Benefits of Experience Based

T ti
Testing

2010
38
Strengths of experience based testing

Testers skills
Utilizing the skills and experience of the tester
Testers know how the software is used and for what purpose
Testers know what functionality
y and features are critical
Testers know what problems are relevant
Testers know how the software was built
Risks,
Risks tacit knowledge
Enables creative exploring

Enables fast learning and improving testing
Investigating, searching, finding, combining, reasoning,
deducting, ...
Testing intangible properties

Look and feel and other user perceptions
Juha Itkonen, SoberIT - 2009
39
Strengths of experience based testing

Process
Agility and flexibility
Easy and fast to focus on critical areas
Fast reaction to changes
g
Ability to work with missing or weak documentation
Effectiveness
Reveals large number of relevant defects
Efficiency
Low documentation overhead
Fast feedback

40
Challenges of experience based testing
Planning and tracking

How much testing is needed
needed, how long does it take?
What is the status of testing?
How to share testing work between testers?
Managing test coverage

What has been tested?
When are we done?
Logging and reporting

Visibility outside testing team
or outside individual testing sessions
Quality of testing
How to assure the quality of testers
tester s work
Detailed test cases can be reviewed, at least

41
RESULTS OF A CONTROLLED STUDENT EXPERIMENT
What is the benefit of

d i i the
designing
h test cases b
beforehand?
f
h d?
Juha Itkonen, Mika V. Mntyl and Casper Lassenius
"Defect Detection Efficiency: Test Case Based vs. Exploratory Testing, ESEM, 2007.
Research problem
Do testers
performing manual functional testing
with pre-designed test cases
find more or different defects
compared
p
to testers working
g
without pre-designed test cases?

43
Research questions
H
How
d
does using
i pre-designed
d i
d ttestt cases affect
ff t
1 the number of detected defects?
1.
2. the
e type
ype o
of de
detected
ec ed de
defects?
ec s
3. the number of false defect reports?
False reports are:
incomprehensible
duplicate
reporting a non-existent defect

44
No difference in the
total number of detected defects
Testing
approach
ET
TCT
Feature
Set
Number
of defects
Found defects
per subject
FS A
44
6 28
6.28
2 172
2.172
FS B
41
7.82
2.522
Total
85
7.04
2.462
FS A
43
5.36
2.288
FS B
39
7.35
2.225
Total
82
6.37
x = mean and = standard deviation
2.456
Difference shows 0.7 defects more in the ET approach

T-test significance value of 0.088
Not statistically significant

45
Differences in defect Types

Compared
C
d tto TCT
TCT, ET detected
d t t d
More defects that were obvious to detect

More defects that were difficult to detect
More user interface and usability issues
More low severity defects
These are descriptive rather than conclusive findings

Not statistically significant

46
TCT p
produces more false defect reports
p
Testing
Approach
ET
TCT
Feature
Set
False defects
subject
FS A
1.00
1.396
FS B
1 05
1.05
1 191
1.191
Total
1.03
1.291
FS A
1.64
1.564
FS B
2.50
1.867
Total
2.08
1.767
per
TCT produced on average 1.05 more false

reports than ET
per subject and testing session
Difference is statistically significant with a
significance value 0.000
Mann-Whitney U test
47
False reports are:

incomprehensible
duplicate
reporting
ti
a nonexistent defect
Conclusions
1 Th
1.
The data
d t showed
h
d no benefits
b
fit ffrom using
i pre-designed
d i
d
test cases
in comparison to freestyle exploratory testing approach
Defect type distributions indicate certain defect types might be
better detected by ET
But no significant differences
2. Test-case based testing produced more false defect

reports
Perhaps test cases made testers work more mechanically
leading,
g e.g.,
g to higher
g
number of duplicate
p
reports
p

48
Questions and more discussion
Contact information
Juha Itkonen
juha.itkonen@tkk.fi
+358 50 577 1688
htt //
http://www.soberit.hut.fi/jitkonen
b it h t fi/jitk
Juha Itkonen - 2009 SoberIT

49
References (primary)
Bach, J., 2000. Session-Based Test Management. Software Testing and Quality Engineering, 2(6). Available at:
http://www.satisfice.com/articles/sbtm.pdf.
Bach, J., 2004. Exploratory Testing. In E. van Veenendaal, ed. The Testing Practitioner. Den Bosch: UTN Publishers,
pp 253
pp.
253-265.
265 http://www.satisfice.com/articles/et-article.pdf.
http://www satisfice com/articles/et article pdf
Itkonen, J. & Rautiainen, K., 2005. Exploratory testing: a multiple case study. In Proceedings of International
Symposium on Empirical Software Engineering. International Symposium on Empirical Software Engineering. pp. 8493.
Itkonen JJ., Mntyl,
Itkonen,
Mntyl M.V.
M V & Lassenius,
Lassenius C.,
C 2007
2007. Defect Detection Efficiency: Test Case Based vs.
vs Exploratory
Testing. In Proceedings of International Symposium on Empirical Software Engineering and Measurement.
International Symposium on Empirical Software Engineering and Measurement. pp. 61-70.
Itkonen, J., Mantyla, M. & Lassenius, C., 2009. How do testers do it? An exploratory study on manual testing practices.
In Empirical Software Engineering and Measurement, 2009. ESEM 2009. 3rd International Symposium on. Empirical
Software Engineering and Measurement, 2009. ESEM 2009. 3rd International Symposium on. pp. 494-497.
Lyndsay, J. & Eeden, N.V., 2003. Adventures in Session-Based Testing. http://www.workroomproductions.com/papers/AiSBTv1.2.pdf. Available at: http://www.workroom-productions.com/papers/AiSBTv1.2.pdf .
Martin, D. et al., 2007. 'Good' Organisational Reasons for 'Bad' Software Testing: An Ethnographic Study of Testing in
a Small Software Company
Company. In Proceedings of International Conference on Software Engineering
Engineering. International
Conference on Software Engineering. pp. 602-611.
Whittaker, J.A., 2009. Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design,
Addison-Wesley Professional.

50
References (secondary)
Agruss, C. & Johnson, B., 2005. Ad Hoc Software Testing.
Ammad Naseer & Marium Zulfiqar, 2010. Investigating Exploratory Testing in
Industrial Practice. Master's Thesis. Rnneby, Sweden: Blekinge Institute of Technology. Available at:
http://www bth se/fou/cuppsats nsf/all/8147b5e26911adb2c125778f003d6320/$file/MSE 2010 15 pdf
http://www.bth.se/fou/cuppsats.nsf/all/8147b5e26911adb2c125778f003d6320/$file/MSE-2010-15.pdf.
Armour, P.G., 2005. The unconscious art of software testing. Communications of the ACM, 48(1), 15-18.
Beer, A. & Ramler, R., 2008. The Role of Experience in Software Testing Practice. In Proceedings of Euromicro
Conference on Software Engineering and Advanced Applications. Euromicro Conference on Software Engineering and
Advanced Applications
Applications. pp
pp. 258-265
258 265.
Houdek, F., Schwinn, T. & Ernst, D., 2002a. Defect Detection for Executable Specifications An Experiment.
International Journal of Software Engineering & Knowledge Engineering, 12(6), 637.
Kaner, C., Bach, J. & Pettichord, B., 2002. Lessons Learned in Software Testing, New York: John Wiley & Sons, Inc.
Martin, D. et al., 2007. 'Good'
Good Organisational Reasons for 'Bad'
Bad Software Testing: An Ethnographic Study of Testing in
a Small Software Company. In Proceedings of International Conference on Software Engineering. International
Conference on Software Engineering. pp. 602-611.
Tinkham, A. & Kaner, C., 2003a. Learning Styles and Exploratory Testing. In Pacific Northwest Software Quality
Conference (PNSQC). Pacific Northwest Software Quality Conference (PNSQC).
Wood, B. & James, D., 2003. Applying Session-Based Testing to Medical Software. Medical Device & Diagnostic
Industry, 90.
Vga, J. & Amland, S., 2002. Managing High-Speed Web Testing. In D. Meyerhoff et al., eds. Software Quality and
Software Testing in Internet Times. Berlin: Springer-Verlag, pp. 23-30.

51

Experience-Based and Exploratory Testing - Itkonen

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Experience-Based and Exploratory Testing - Itkonen

Caricato da

Copyright:

Formati disponibili

How Do Testers Do It?

Intelligent Manual Testing Practices

Benefits of Experience Based Testing

Juha Itkonen - SoberIT

Research has shown:

Juha Itkonen - SoberIT

In practice, its clear that some testers

Experience is invaluable in software

Technical system experience

Knowledge and skills gained in the application domain area

Juha Itkonen, 2009 SoberIT

requires certain kind of attitude

Juha Itkonen - SoberIT

People tend to see what they want or expect

Testing is all about exceptions, special cases,

Juha Itkonen - SoberIT

Explore, investigate, and measure

Juha Itkonen - SoberIT

My viewpoint: Experience Based

Manual testing that builds on the testers

Some aspects of testing rely on testers skills

Testers are assumed to know what they are doing

Focus on the actual testing work in practice

Juha Itkonen - SoberIT

Exploratory Testing is creative testing

Exploring with a general mission

2. Testing is guided by the results of previously performed tests and the

Testers can apply deductive reasoning to the test outcomes

3. The focus is on finding defects by exploration

Instead of demonstrating systematic coverage

Juha Itkonen - SoberIT

Exploratory Testing vs. Scripted Testing

Juha Itkonen, 2009 SoberIT

Scripted vs. Exploratory Testing

In exploratory testing, tests are

Scripted vs. Exploratory Tests

Juha Itkonen, 2009 SoberIT

Two views of agile testing

Functional (acceptance) testing

Focus on automated verification

Intelligent Manual Testing Practices

Benefits of Experience Based Testing

Juha Itkonen - SoberIT

Some ways of exploring in practice

Juha Itkonen - SoberIT

Advanced users, strong

Session Based Test Management (SBTM)

Juha Itkonen - SoberIT

Session Based Testing

Enables planning and tracking exploratory testing

Efficient no unnecessary documentation

Explicit, scheduled sessions can help getting testing done

Juha Itkonen - SoberIT

Exploring like a tourist

James A. Whittaker. Exploratory Software Testing, Tips, Tricks,

Juha Itkonen - SoberIT

Districts and Tours

James A. Whittaker. Exploratory Software Testing, Tips, Tricks,

Examples of exploratory testing tours

The Garbage Collectors Tour

Use user manual or other

Bloggers tour, use third party advice as guide

Never close the app, use the

Juha Itkonen, 2009 SoberIT