Testing: Luca Viganò

Testing
Luca Vigan
Ingegneria del Software, AA 10/11, 04.05.11
The slides in this section are partly based on the courses Software Engineering by Prof. Peter Mller, ETH Zurich, Software Engineering I by Prof. Bernd Brgge, TU Mnchen and Software Engineering by Prof. Jan Vitek, Purdue University
The software testing process
Ian Sommerville 2006
!Software Engineering, 8th edition. Chapter 23
Slide 8!
7. Testing Basics Basics
Why Does Software Contain Bugs?

Our ability to predict the behavior of our creations is imperfect
- Software is extremely complex - No developer can understand the whole system
We make mistakes
Unclear requirements, miscommunication Wrong assumptions (e.g., behavior of operating system) Design errors (e.g., capacity of data structure too small) Coding errors (e.g., wrong loop condition)
Peter Mller Software Engineering, SS 11
7. Testing Basics
Goal of Testing
An error is a deviation of the observed behavior from the required (desired) behavior
- Functional requirements (e.g., user-acceptance testing) - Nonfunctional requirements (e.g., performance testing)
Testing is a process of executing a program with the intent of finding an error A successful test is a test that finds errors
Definition: testing
To test a software system is to try to make it fail
7. Testing Basics
Limitations of Testing
Testing can only show the presence of bugs, not their absence. [E. W. Dijkstra] It is impossible to completely test any nontrivial module or any system
- Theoretical limitations: termination - Practical limitations: prohibitive in time and cost
7.1 Testing Basics Test Stages

7.1 Test Stages 7.2 Test Strategies
Test Stages
Analysis System Test
System Design
Integration Test
Detailed Design
Unit Test
Implementation
10
Unit Testing
Testing individual subsystems (collection of classes)
Detailed Design Model Subsystem Code
Unit Test
Goal: Confirm that subsystem is correctly coded and carries out the intended functionality
xUnit widely used testing frameworks

xUnit frameworks allow testing of different elements (units) of software, such as functions and classes. The main advantage of xUnit frameworks is that they provide an automated solution with no need to write the same tests many times, and no need to remember what should be the result of each test. Examples ! JUnit for Java ! NNnit for .NET ! CppUnit for C++
JUnit: Overview
Provides a framework for running test cases Test cases !! Written manually !! Normal classes, with annotated methods Input values and expected results defined by the tester Execution is the only automated step
17
Test Execution
Execute the test cases Re-execute test cases after every change
- Automate as much as possible - For instance, after each refactoring
Regression testing
- Testing that everything that used to work still works after changes are made to the system - Also important for system testing
18
Eight Rules of Testing

1. Make sure all tests are fully automatic and check their own results 2. A test suite is a powerful bug detector that reduces the time it takes to find bugs 3. Run your tests frequently every test at least once a day 4. When you get a bug report, start by writing a unit test that exposes the bug 5. Better to write and run incomplete tests than not run complete tests 6. Concentrate your tests on boundary conditions 7. Do not forget to test exceptions raised when things are expected to go wrong 8. Do not let the fear that testing cant catch all bugs stop you from writing tests that will catch most bugs [M. Fowler]
19
Integration Testing
Testing groups of subsystems and eventually the entire system
Subsystem Code Subsystem Code Subsystem Code
Software Architecture Integration Test
Goal: Test interfaces between subsystems

20
Steps in Integration-Testing
1. Select a component to be tested
- Unit test all the classes in the component
2. Put selected components together

- Make the integration test operational (drivers, stubs)
3. Do the testing
- Functional testing, structural testing, performance testing
4. Keep records of the test cases and testing activities 5. Repeat steps 1 to 4 until the full system is tested
21
Integration Testing Strategy

The order in which the subsystems are selected for testing and integration Typical strategies
- Big-bang integration (non-incremental) - Bottom-up integration - Top-down integration - Sandwich testing - Variations of the above
Call hierarchy
A
22
Big-Bang Strategy: Example

A B Dont try this!
C
D E F
Whole System
Integration Test
G
23
Bottom-Up Strategy
Strategy
1. Start with subsystems in lowest layer of call hierarchy 2. Test subsystems that call the previously tested subsystems 3. Repeat until all subsystems are included Pros - Useful for integrating object-oriented systems and systems with strict performance requirements
Cons - Tests the most important subsystem (UI) last
24
Bottom-Up Strategy: Example

B
B
E B,E,F A C D E F G
Whole System
D,G
G
25
Top-Down Strategy
Strategy
1. Start with subsystems in top layer of call hierarchy 2. Include subsystems that are called by the previously tested subsystems 3. Repeat until all subsystems are included
Pros - Supports test cases for the functionality of the system
Cons - Writing stubs is difficult: Stubs must allow all possible conditions to be tested - Possibly very large number of stubs required
26
Top-Down Strategy: Example

B A B A,B,C, D E F
C
D
E F G
Whole System
27
Sandwich Strategy
Combines top-down with bottom-up strategy
The system is viewed as having three layers

A target layer in the middle A layer above the target A layer below the target Testing converges at the target layer
How do you select the target layer if there are more than three layers?
- Try to minimize the number of stubs and drivers
28
Sandwich Strategy: Example

F
Target layer
B B,E,F E Bottom / target D,G Whole System F G C D
E
B G D A C
A,B,C, D
Top / target
29
Sandwich Strategy: Discussion

Pros - Top and bottom layer can be tested in parallel - Fewer drivers and stubs needed (target layer instead of driver for bottom layer and stub for top layer) Cons - Does not test the individual subsystems thoroughly before integration
31
System Testing
Testing the entire system
Requirements Specification Entire System System Test
Goal: Determine if the system meets the requirements (functional and non-functional)
32
System Testing Stages

Entire System Functional requirements Non-functional requirements Clients understanding of requirements User Environment
Functional Test Performance Test Acceptance Test Installation Test
33
Functional Testing
Goal: . Test functionality of system
- System is treated as black box
Test cases are designed from requirements analysis document

- Based on use cases . - Alternative source: user manual
Test cases describe

- Input data - Flow of events - Results to check
Classification: by available information

White-box testing !! To define test cases, source code of IUT is available Alternative names: implementation-based, structural, glass box, clear box
Black-box testing !! Properties of IUT available only through specification Alternative names: responsibility-based, functional
A comparison
White-box
IUT internals Focus Origin of test cases Typical use Who?
Black-box
Knows internal structure No knowledge & implementation Ensure coverage of many Test conformance to execution possibilities specification Source code analysis Unit testing Developer Specification Integration & system testing Developers, testers, customers
34
Performance Testing
Stress Testing
- Stress limits of system (maximum number of users, peak demands)
Volume testing
- Large amounts of data
Configuration testing
- Various software and hardware configurations
Compatibility testing
- Backward compatibility with existing systems
Security testing
- Try to violate security requirements (red team)
35
Performance Testing (contd)

Timing testing
- Response times and time to perform a function
Environmental testing
- Tolerances for heat, humidity, motion
Quality testing
- Reliability, maintainability, and availability
Recovery testing
- Systems response to presence of errors or loss of data
Usability testing
- Tests user interface with user
36
Acceptance Testing
Goal: Demonstrate that the system meets customer requirements and is ready to use Choice of tests is made by client
- Many tests can be taken from integration testing
Performed by the client, not by the developer
37
Acceptance Testing (contd)

Majority of bugs is typically found by the client, not by the developers or testers Alpha test
- Client uses the software at the developers site - Software used in a controlled setting, with the developer always ready to fix bugs
Beta test
- Conducted at clients site (developer is not present) - Software gets a realistic workout in target environment - Potential client might get discouraged
Alpha and beta testing

Alpha testing The first test of newly developed hardware or software in a laboratory setting. When the first round of bugs has been fixed, the product goes into beta test with actual users. Beta testing A test of new or revised hardware or software that is performed by users at their facilities under normal operating conditions. An interesting example: proportional testing of Gmail.
38
Independent Testing
Programmers have a hard time believing they made a mistake
- Plus a vested interest in not finding mistakes - Often stick to the data that makes the program work
Designing and programming are constructive tasks

- Testers must seek to break the software
Testing is done best by independent testers
39
Independent Testing: Responsibilities

System Test
Performed by independent test team

- Exception: Acceptance test performed by client
Performed by independent test Integration Test team Performed by programmer

Unit Test - Requires detailed knowledge of the code - Immediate bug fixing
40
Independent Testing: Wrong Conclusions

The developer should not be testing at all
- Test before you code
Testers get only involved once software is done
Toss the software over the wall for testing

- Testers and developers collaborate in developing the test suite
Testing team is responsible for assuring quality

- Quality is assured by a good software process
7.2 Testing Basics Test Strategies
41

7.1 Test Stages 7.2 Test Strategies
42
Testing Steps
Select what will be tested
What parts of the system? What aspects of the system?
Select test strategy
What integration strategy? How is the test data determined?
Define test cases
What are the test data? How is the test carried out?
Create test oracle

What are the expected results? Defined before executing tests
43
Example: Solve Quadratic Equation

void roots( double a, double b, double c ) { double q = b*b 4*a*c; if( q > 0 && a != 0 ) { numRoots = 2; double r = Math.sqrt( q ); x1 = (-b + r) / (2 * a); Fails if a==0 and x2 = (-b - r) / (2 * a); b*b4*a*c == 0 } else if( q == 0 ) { numRoots = 1; x1 = -b / (2 * a); } else { b b 2 4ac x= numRoots = 0; 2a } }
44
Strategy 1: Exhaustive Testing

Check UUT for all possible inputs
- Not feasible, even for trivial programs
void roots( double a, double b, double c ) { }
Assuming that double represents 64-bit values, we get (264)3 1058 possible values for a, b, c Programs with heap data structures have a much larger state space!
45
Strategy 2: Random Testing

Select test data uniformly
void roots( double a, double b, double c ) { double q = b*b 4*a*c; } if( q > 0 && a != 0 ) { Fails if a==0 and b*b4*a*c == 0 } else if( q == 0 ) { numRoots = 1; x1 = -b / (2 * a); The likelihood of } else { } selecting a==0 and b==0 } randomly is 1/1038
46
Random Testing: Observations

Random testing focuses on generating test cases fully automatically Advantages
- Avoids designer/tester bias - Tests robustness, especially handling of invalid input and unusual actions
Disadvantages
- Treats all inputs as equally valuable
47
Strategy 3: Functional Testing

Use analysis knowledge to determine test cases that check requirements
Given three values, a, b, c, compute all solutions of the equation ax2 + bx + c = 0
Two solutions
a 0 and b2-4ac > 0
One solution
a = 0 and b 0 or a 0 and b2-4ac = 0
No solution
a = 0, b = 0, and c 0 or a 0 and b2-4ac < 0
Test each case of the specification

48
Functional Testing: Observations

Functional testing focuses on input/output behavior
- Goal: Cover all the requirements
Attempts to find
- Incorrect or missing functions - Interface errors - Performance errors
Limitations
- Does not effectively detect design and coding errors (e.g., buffer overflow, memory management) - Does not reveal errors in the specification (e.g., missing cases)
49
Strategy 4: Structural Testing

Use design knowledge about system structure, algorithms, data structures to determine test cases that exercise a large portion of the code
void roots( double a, double b, double c ) { double q = b*b 4*a*c; Test this if( q > 0 && a != 0 ) { case } else if( q == 0 ) { and this case } else { and this case } }
Error might still be missed, for instance, when case is tested with a==1, b==2, c==1
50
Structural Testing: Observations

Structural testing focuses on thoroughness
- Goal: Cover all the code
Not well suited for system test

- Focuses on code rather than on requirements, for instance, does not detect missing logic - Requires design knowledge, which testers and clients do not have (and do not care about) - Thoroughness would lead to highlyredundant tests
51
Testing Strategies: Summary

Functional testing Goal: Cover all the requirements Black-box test Suitable for all test stages Structural testing Goal: Cover all the code White-box test Suitable for unit testing
Random testing Goal: Cover corner cases Black-box test Suitable for all test stages
7. Testing Basics
52
Summary
Main objective
- Design tests that systematically uncover different classes of errors with a minimum amount of time and effort - A good test has a high probability of finding an error - A successful test uncovers an error
Secondary benefits
- Demonstrate that software appears to be working according to specification (functional and non-functional) - Data collected during testing provides indication of software reliability and software quality - Good testers clarify the specification (creative work)
GUI Testing
Why is GUI testing hard?

!! GUI "! Graphics: easy for humans, hard for machines "! Themable GUIs "! Simple change to interface, big impact !! Network & Databases "! Big effort to set up environment !! Computers !! Operating Systems !! Applications !! Data !! Network "! Reproducibility
Why is GUI testing hard?

!! In the old days things were easy "! Stdin / Stdout / Stderr !! Modern applications lack uniform interface "! GUI "! Network "! Database "!
Minimizing GUI code

!! GUI code is hard to test !! Try to keep it minimal !! How?
Model-View-Controller
Views
VIEW
Model
A = 50% B = 30% C = 20%
Model-View Controller
Model View Controller (2/2)

Model
Change Notification ! Encapsulates application state ! Exposes application functionality ! Notifies view of changes State change
View
! Renders the model ! Sends user gestures to controller ! Allows controller to select view Feature calls Events
View selection
Controller
! Defines application behavior ! Maps user actions to model User gestures updates ! Selects view for response ! One for each functionality
Test Data
8. Test Data Functional Testing
8. Test Data
8.1 Functional Testing 8.2 Structural Testing 8.3 Automatic Test Case Generation 8.4 Dynamic Test Generation 8.5 Mutation Testing
Applications of Functional Testing

Black-box test a unit against its requirements
Functional test Acceptance test Test interfaces between subsystems
System Test
Integration Test
During test-driven development, when code is not yet written
Unit Test
8. Test Data
8.1 Functional Testing
8.1.1 Partition Testing 8.1.2 Selecting Representative Values 8.1.3 Combinatorial Testing
8.2 Structural Testing 8.3 Automatic Test Case Generation 8.4 Dynamic Test Generation 8.5 Mutation Testing
Finding Representative Inputs

Requirement not implemented
Divide inputs into equivalence classes

Requirement implemented incorrectly
- Each possible input belongs to at least one of the equivalence classes - Goal: some classes have higher density of failures
Failure No failure
Requirement implemented correctly
Choose test cases for each equivalence class
Equivalence Classes: Example

Given a month (an integer in [1;12]) and a year (an integer), compute the number of days of the given month in the given year (an integer in [28;31])
month
Month with 28 or 29 days month = 2 Leap years
year
(year mod 4 = 0 and year mod 100 0) or year mod 400 = 0
Months with 30 days

Months with 31 days
month {4, 6, 9, 11}

month {1, 3, 5, 7, 8, 10, 12}
Non-leap years
year mod 4 0 or (year mod 100 = 0 and year mod 400 0)
Invalid inputs missing

Equivalence Classes: Example (contd)

month
Month with 28 or 29 days month = 2 Leap years
year
(year mod 4 = 0 and year mod 100 0) or year mod 400 = 0
Months with 30 days

Months with 31 days Invalid
month {4, 6, 9, 11}

month {1, 3, 5, 7, 8, 10, 12} month < 1 or month > 12
Non-leap years
year mod 4 0 or (year mod 100 = 0 and year mod 400 0)
Partitioning seems too coarse
Equivalence Classes: Example (contd)

month
Month with 28 or 29 days month = 2 Standard leap years
year
year mod 4 = 0 and year mod 100 0
Months with 30 days

month {4, 6, 9, 11}

month {1, 3, 5, 7, 8, 10, 12} month < 1 or month > 12
Standard nonleap years

Special leap years Special nonleap years
year mod 4 0
year mod 400 = 0 year mod 100 = 0 and year mod 400 0
Sources of Information
Use analysis knowledge to determine test cases that check requirements
Analysis Model Functional Model Analysis Object Model Dynamic Model
Input / output behavior, valid and invalid input Data manipulated by the system
System states and protocols
10
Using the Functional Model

The functional model describes the input-output behavior of the whole system
- Valid and invalid inputs, entry conditions - Expected results, exit conditions
Basis for functional system testing
11
Using the Functional Model: Example

Actor steps
1. Authenticate (use case Authenticate) 3. Client selects Withdraw CHF
Inputs
System Steps
2. Bankomat displays options
4. Bankomat queries amount

5. Client enters amount
Expected outputs
6. Bankomat returns bank card 7. Bankomat outputs specified amount in CHF
12
Using the Analysis Object Model

The analysis object model contains the main concepts manipulated by the system, their properties and relationships
- Useful to determine equivalence classes - Useful to set up state of objects
Relevant information
- Classes and attributes - Subtypes - Aggregations and multiplicities
13
Using the Analysis Object Model: Example

Book
Test for super- and subclass Library Member borrow( Copy ) 0..1 Test both cases 0..1 0..* 0..* Borrowable
1..*
Copy Recall: library members may borrow at most 6 items (12 for staff members)
Staff Member
borrow( Journal )
Journal
14
Equivalence Classes
Consider the operation member.borrow( copy )
member copy
Library member
Staff member Invalid library member Invalid staff member
05 borrowed copies
0 11 borrowed items more than 5 borrowed copies more than 11 borrowed items
Copy on shelf
Copy borrowed by library member Copy borrowed by staff member Invalid
copy has zero library members

copy has one library member copy has one staff member null
15
Using the Dynamic Model

Sequence diagrams describe protocols for object interactions
- Benefit for testing is similar to use cases - Especially useful for integration testing
State diagrams describe state-dependent behavior

- Different states typically require different equivalence classes - State defines valid input and expected output - Expected output includes successor state - Useful for protocols, GUIs, and objects
16
Using the Dynamic Model: Example 1

:Sensor signal( floor ) opt <<Control>> :Controller
<<Entity>> :Engine
<<Entity>> :RequestPool
Test both cases
stop( ) served( floor )
[ floor = nextStop ]
nextStop := getNext( ) alt setIdle( )

start ( direction )
[ nextStop = ] [ else ]
Test both cases
17
Equivalence Classes
Consider the operation controller.signal( floor )
controller Serve only request Serve first request floor = nextStop, and there is no further request floor = nextStop, and there are further requests
Serve no request
floor nextStop
18
Using the Dynamic Model: Example 2

Test valid and invalid operations [ pool is empty ] request( floor )
Idle
Requested Stop
request( floor )
[ pool is not empty ] / choose nextStop
signal( floor ) [ floor = nextStop ]
Moving
Test all cases signal( floor ) [ floor nextStop ]
request( floor )
19
Equivalence Classes
controller Reach floor controller is in state Moving and floor = nextStop controller is in state Moving and floor nextStop
Keep moving
Invalid state
controller is in state Idle or RequestedStop
20
8. Test Data
21
Selecting Representative Values

Once we have partitioned the input values, we need to select concrete values for the test cases for each equivalence class Input from a range of valid values
- Below, within, and above the range - Also applies to multiplicities on aggregations
Input from a discrete set of valid values

- Valid and invalid discrete value - Instances of each subclass
22
Boundary Testing
Given an integer x, determine the absolute value of x
x Valid all values
int abs( int x ) { if( 0 <= x ) return x; return x; } Negative result for x==Integer.MIN_VALUE
Large number of errors tend to occur at boundaries of the input domain

Overflows Comparisons (< instead of <=, etc.) Missing emptiness checks (e.g., collections) Wrong number of iterations
23
Boundary Testing: Example

Select elements at the edge of each equivalence class (in addition to values in the middle)
- Ranges: lower and upper limit - Sets and collections: empty set
month Month with 28 or 29 days Months with 30 days Months with 31 days Invalid month = 2 month {4, 6, 9, 11} month {1, 3, 5, 7, 8, 10, 12} month < 1 or month > 12
There is only one value Choose all values Choose 1 and 12 plus one more Choose MIN_VALUE, 0, 13, MAX_VALUE
24
Boundary Testing: Example (contd)

Choose for instance -200.004, -4, 4, 2012, 400.008 Choose for instance -200.003, -1, 1, 2011, 400.009 Choose for instance -200.000, 0, 2000, 400.000
year Standard leap years Standard nonleap years Special leap years Special nonleap years year mod 4 = 0 and year mod 100 0 year mod 4 0
year mod 400 = 0

year mod 100 = 0 and year mod 400 0
Choose for instance -200.100, 1900, 400.100
25
Parameterized Unit Test for Leap Years

[ Test ] public void TestDemo29( [ Values( -200004, -200000, -4, 0,4, 2000, 2012, 400000, 400008 ) ] int year ) Only one All selected values for { value leap years and special int d = Days( 2, year ); leap years Assert.IsTrue( d == 29 ); } Expected result
Analogous test cases for February in non-leap year, months with 30 days, and months with 31 days
26
Parameterized Unit Test for Invalid Inputs

Expected result: an exception
All selected [ Test ] invalid values [ ExpectedException( typeof(ArgumentException) ) ] for month public void TestDemoInvalid( [ Values( int.MinValue, 0, 13, int.MaxValue ) ] int month, [ Values( -200100, -200004, -200003, -200000, -4, -1, 0, 1, 4, 1900, 2000, 2011, 2012, 400000, 400008, 400009, 400100 ) ] int year ) { int d = Days( month, year ); All selected } values for year
27
Coverage of State Diagrams

We define coverage criteria to measure how thorough a state diagram is tested Path Coverage: execute each possible path
- Not feasible with many nested conditionals - Impossible for most loops
State Coverage: visit each state

- A minimum criterion
Transition Coverage: execute each edge

- Thorough testing
28
Coverage Example
request( floor )
Idle
[ pool is empty ]
Requested Stop
request( floor )
[ pool is not empty ] / choose nextStop
signal( floor ) [ floor = nextStop ] One test for state coverage
Moving
signal( floor ) [ floor nextStop ]
request( floor )
29
Coverage Example (contd)

One test case for state coverage
controller controller is in state Moving and floor = nextStop controller is in state Moving and floor nextStop controller is in state Idle or RequestedStop
Reach floor
Keep moving Invalid state
Two test cases for transition coverage: empty and non-empty pool Not required for state coverage
30
8. Test Data
31
Combinatorial Testing
Combining equivalence classes and boundary testing leads to many values for each input
- Twelve values for month and 17 values for year in the Leap Year example
Testing all possible combinations leads to a combinatorial explosion (12 x 17 = 204 tests) Reduce test cases to make effort feasible
- Semantic constraints - Combinatorial selection - Random selection
32
Eliminating Combinations
Inspect test cases for unnecessary combinations
- Especially for invalid values - Use domain knowledge
month Month with 28 or 29 days month = 2 month {4, 6, 9, 11} month {1, 3, 5, 7, 8, 10, 12} month < 1 or month > 12
Test all combinations with year
Months with 30 days

Behavior is independent of year
Reduces test cases from 204 to 17 + 4 + 3 + 4 = 28

34
Selecting Object References

Objects are different from values because they have identity
a1 = new Account( 1000 ); a2 = new Account( 1000 ); a1.transfer( a2, 500 ); a1 = new Account( 1000 ); a1.transfer( a1, 500 ); Might behave differently (e.g., deadlock)
When selecting test data for objects, one has to consider object identities and aliasing Referenced objects lead to combination problem
35
Selecting Object References: Example

member Library member Staff member Invalid library member 05 borrowed copies 0 11 borrowed items more than 5 borrowed copies
Empty list, list of 1 and 5 (11) copies, list without duplicates, list with duplicates
List List of of 6 (12) 6 (12) copies copies and very large collection
Invalid staff member
more than 11 borrowed items
This is a case of combinatorial testing since it combines the Library Member and the collection
36
Semantic Constraints for Objects

Object invariants restrict the possible instances of a class diagram
- Expressed as comment or OCL constraint
In our example, let assume the following invariants

Library Member 0..1 0..* Copy
- A Library Member m contains a Copy c in its collection of borrowed items if and only if cs Library Member is m - For each Library Member, the collection of borrowed items contains no duplicates - For each Library Member, the collection of borrowed items contains at most 6 copies (12 items for staff)
37
Selecting Object References: Example (contd)

Empty list, listlist of 1 ofand 5 (11) 5 (11) copies, copies without list withou duplicates
member
Library member 05 borrowed copies
copy
Copy on shelf copy has zero library members
Staff member
Invalid library member Invalid staff member
0 11 borrowed items
more than 5 borrowed copies more than 11 borrowed items
Copy borrowed by library member

Copy borrowed by staff member Invalid copy is borrowed by member, member, copy is not borrowed by by member member
copy has one library member

copy has one staff member null
List of 6 (12) copies

38
Roots Example
a b any value c any value
Valid
Invalid
any value
infinity, NaN
infinity, NaN
infinity, NaN
Boundary testing: a, b, c { Double.MIN_VALUE, -5, 0, 5, Double.MAX_VALUE }
53 = 125 test cases for valid inputs

39
Roots Example (contd)

Look at dependencies between inputs

No solution
Two solutions a 0 and b2-4ac > 0
One solution
a = 0 and b 0 or a 0 and b2-4ac = 0
a = 0, b = 0, and c 0 or a 0 and b2-4ac < 0
Semantic constraints on combinations

Partitioning seems too coarse
40

Two solutions
Linear equation (Truly) quadratic equation
One solution
a = 0 and b 0
No solution
a = 0, b = 0, and c 0
a 0 and b2-4ac > 0 a 0 and b2-4ac = 0
a 0 and b2-4ac < 0
Not all inputs are covered: a=b=c=0

41

Given three values, a, b, c, compute all solutions of the equation ax2 + bx + c = 0; report an error if all three values are zero
Two solutions
Linear equation (Truly) quadratic equation Invalid input
One solution
a = 0 and b 0
No solution
a = 0, b = 0, and c 0
a 0 and b2-4ac > 0 a 0 and b2-4ac = 0
a 0 and b2-4ac < 0
a = 0, b = 0, c = 0
42
Roots Example: Summary

Classifying the combinations according to semantic constraints did not reveal any irrelevant test cases But we did identify an omission in the specification
- It is common that testers clarify the specification
One option is to manually choose a manageable number of test cases such that there is at least one test case for each semantic constraint
- Note that omitting test cases might leave errors such as arithmetic overflow undetected
43
Semantic Constraints : Discussion

Semantic constraints potentially reduce the number of (interesting) test cases
- They also help increasing the coverage
But too many combinations remain

- Especially when there are many input values, for instance, for the fields of objects
44
Influence of Variable Interactions

Empirical evidence suggests that most errors do not depend on the interaction of many variables
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1 2 3 4 5 6 Medical Devices Browser Server NASA GSFC
Vars 1 2 3 4 5 6
Medical NASA Network Browser Server Devices GSFC Security 66% 97% 99% 100% 29% 76% 95% 97% 99% 100% 42% 70% 89% 96% 96% 100% 68% 93% 98% 100% 20% 65% 90% 98% 100%
Network Security
Interactions of two or three variables trigger most errors
45
Pairwise-Combinations Testing
Instead of testing all possible combinations of all inputs, focus on all possible combinations of each pair of inputs
- Pairwise-combinations testing is identical to combinatorial testing for two or less inputs
Example: Consider a method with four boolean parameters

- Combinatorial testing requires 24 = 16 test cases - Pairwise-combinations testing requires 5 test cases: TTTT, TFFF, FTFF, FFTF, FFFT
Can be generalized to k-tuples (k-way testing)

46
Pairwise-Combinations Testing: Complexity

For n parameters with d values per parameter, the number of test cases grows logarithmically in n and quadratic in d
- Handles larger number of parameters, for instance, fields of objects - The number d can be influenced by the tester
Result holds for large n and d, and for all k in k-way testing
47
Pairwise-Combinations Testing: Example

Two solutions One solution a = 0 and b 0 a 0 and b2-4ac > 0 a 0 and b2-4ac = 0 a = 0, b = 0, c = 0 No solution a = 0, b = 0, and c 0 a 0 and b2-4ac < 0
Three parameters, five values each

- Double.MIN_VALUE, -5, 0, 5, Double.MAX_VALUE - 53 = 125 test cases for combinatorial testing - 25 test cases for pairwise-combinations testing
Bug is still detected (depends only on a and b) Some cases depend on three parameters, e.g., invalid input
48
Pairwise-Combinations Testing: Discussion

Pairwise-combinations testing (or k-way testing) reduces the number of test cases significantly while detecting most errors Pairwise-combinations testing is especially important when many system configurations need to be tested
- Hardware, operating system, database, application server, etc.
Should be combined with other approaches to detect errors that are triggered by more complex interactions among parameters
49
Functional Testing: Summary

Functional Requirements, Analysis Model
Independently Testable Feature
Equivalence classes, boundary testing, coverage
Representative Values
Exhaustive enumeration, semantic constraints, pairwise combinations
Test Case Specification
Test Cases

Testing: Luca Viganò

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Testing: Luca Viganò

Caricato da

Copyright:

Formati disponibili

Testing

The software testing process

Ian Sommerville 2006

!Software Engineering, 8th edition. Chapter 23

7. Testing Basics Basics

Why Does Software Contain Bugs?

Peter Mller Software Engineering, SS 11

Peter Mller Software Engineering, SS 11

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

7. Testing Basics Basics

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

7.1 Testing Basics Test Stages

Peter Mller Software Engineering, SS 11

xUnit widely used testing frameworks

7.1 Testing Basics Test Stages

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

Eight Rules of Testing

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

Software Architecture Integration Test

Goal: Test interfaces between subsystems

7.1 Testing Basics Test Stages

2. Put selected components together

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

Integration Testing Strategy

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

Big-Bang Strategy: Example

7.1 Testing Basics Test Stages

Cons - Tests the most important subsystem (UI) last

7.1 Testing Basics Test Stages

Bottom-Up Strategy: Example

7.1 Testing Basics Test Stages

Pros - Supports test cases for the functionality of the system

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

Top-Down Strategy: Example

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

The system is viewed as having three layers

7.1 Testing Basics Test Stages

Sandwich Strategy: Example

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

Sandwich Strategy: Discussion

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

Peter Mller Software Engineering, SS 11

7.1 Testing Basics Test Stages

System Testing Stages

Functional Test Performance Test Acceptance Test Installation Test

7.1 Testing Basics Test Stages

Test cases are designed from requirements analysis document

Test cases describe

Classification: by available information

7.1 Testing Basics Test Stages

7.1 Testing Basics Test Stages

Performance Testing (contd)

7.1 Testing Basics Test Stages