Test Requirements

David Capocci, CQA, CSTE
Test Requirements: The Basis of Testing

David Capocci, CQA, CSTE Sr. QA Systems Analyst SAFECO Corporation davcap@safeco.com
1
Agenda
Defining Test Requirements (TR)
What, Why, Where
Fitting TRs into the testing picture

Whats within our testing process Generating TRs
Organizing & Decomposing TRs Test Requirements Hierarchy Samples Setting the stage for measuring test coverage
2
Background
Test Requirements Hierarchy is a term coined from Rationals SQA Team Test software. The principle of identifying, organizing, and measuring test requirements is universal to many test processes and methodologies Much of this in-service is distilled from:
Rational methodologies (we are an SQA Team Test house after all) QAI Workbench model Seminar topics from
19th annual International Software Testing Conference Star 98 West Conference
3
Ed Kits Software Testing in the Real World
Test Requirements
What exactly is a Test Requirement? Why identify Test Requirements? Where does a Test Requirement come from?
Defining TRs: What, Why, Where
What exactly is a Test Requirement?

Identifies the WHAT of testing
What needs to be tested, AND What are you going to validate about it
Includes both normal and error conditions Covers business rules, functionality, nonfunctional standards Do NOT have case specific data values assigned to them yet (data appears in test cases, the How of testing) examples
5 Defining TRs: What, Why, Where
Example 1: Testing the inserting of a record to a table

Test Requirements Identified (among others): Validate that you can insert an entry Validate that insertion fails if entry already present Validate that insertion fails if table already full Validate that you can insert an entry to an empty table (initial)
These are test requirements NOT tests because they do not describe the data element being inserted The data is irrelevant at this level, it will appear in the test cases used to cover these test requirements Validate you can insert John Doe is a test case not a test requirement
Defining TRs: What, Why, Where
Why identify Test Requirements?

Its what QC owns in our workbench: Requirements-based or Function-based testing Its the basis for establishing the completion of testing Helps determine the scale of the testing effort Governs the types of resources you will need Serves to identify automation strategies you can use Becomes a roadmap for your testing effort Can be a tool for leverage and dialog with developers and business analysts Dev Team can sign off on them (Verification!)
Where does a TR come from?

Traditional: Business Requirements, functionality, internal logic
Marketing specs, Functional specs, Technical specs
Reality:
Interview Analysis, Non-Functional Checklists (standards & compliance), Use Cases (from business scenarios and users), Discovery during testing, any other deliverables from previous workbenches (diagrams, modeling, flowcharts, etc.)
How do Test Requirements relate to the Test Plan?

Traditionally, the Test Plan has represented what was going to be tested, even including test cases. Paradigm is shifting: Test Plan should relate what your testing process (and deliverables) will be for a particular project. A Test Plan should build confidence in the testing process for a project, making approaches clear. A Test Plan can include the Test Requirements However, if the Test Requirements are too lengthy, they should become their own document: A Test Requirements Hierarchy Defining TRs: What, Why, Where
Agenda
Defining TRs: What, Why, Where Fitting TRs into the testing picture
10
Drilling down: Where test

requirements fit into the picture
Business Requirement
1
Generates
Test Requirement
1
Generates
Test Scenarios/ Cases
Executes/Runs
1
Test Procedure/ Script

11 Fitting TRs into the testing picture
Drilling Down
Test Requirement Test Scenarios/ Cases Test Procedure/ Script
First, Lets look at this relationship: Whats within our testing process
Then well look at this relationship: Gernerating TRs from what feeds into our testing process
12
ATM Example: Practice Writing Test Requirements

Business Requirements:
- ATM must do withdrawals - Withdrawals are between $20-$300 - Withdrawals are in $20 multiples
Group Exercise! 1. Limit the scope to these 3 requirements. 2. What will you validate (test for)? 3. Are there any implied requirements that may not be written out?
Whats within our testing process
13
Example 2: Testing Withdrawals on an ATM

Test Requirements Identified (among others):
Validate that a withdrawal option is available "Validate that a withdrawal of a multiple of $20, between $20-$300 can be done" "Validate that <$20 is not allowed" "Validate that >$300 is not allowed" "Validate that $20 multiples >$300 is not allowed" "Validate that non-$20 multiples between $20-$300 not allowed" "Validate strange numeric amounts/combinations not allowed (all zero's, all 9's, 20.0000)" Validate that the withdrawal received is equal to the amount requested "Validate that a valid withdrawal amount must be below the account balance
14
These are test requirements NOT tests because they do not describe the data element being used (like $20, $40, $60, $1) The data is irrelevant at this level, it will appear in the test cases used to cover these test requirements
Test Scenarios/Cases for Validate that a withdrawal of a multiple of $20, between $20-$300 can be done
Case # P/F $ entered Expected Results

WD01 WD02 WD03 WD04 WD05 : WD13 WD14 WD15
15
Actual Results
Pass Pass Pass Pass Pass : Pass Pass Pass
20 40 60 80 100 : 260 280 300
$20 withdrawn $40 withdrawn $60 withdrawn $80 withdrawn $100 withdrawn : $260 withdrawn $280 withdrawn $300 withdrawn
Test Procedure & Script for previous example

Procedure:
Step 1: Insert Card Step 2: Enter PIN Step 3: Select Withdraw option Step 4: Enter dollar amount Step 5: Validate amount received
Script: (in
pseudo-code )
Do until EOF until end of data file Input data record Senddata CARDINFO to Cardfield Senddata Enter Senddata PIN to PINFfield Senddata Enter Senddata W to SelectionField Senddata AMOUNT to DollarField Senddata Enter If ErrorMsg > 0 then print ErrorMsg Print DollarAMTgiven Loop
16
Agenda
17
The Workbench Concept

Rework
Product Input
DO
Check
Product Output
Entrance Criteria
Standards Tools
Exit Criteria
Our workbench is called Generating Test Requirements

18
Generating TRs
Entrance Criteria for Business Requirements to generate Test Requirements

Visible ? Clear? (unambiguous) Complete? Consistent? (conflicting requirements must be prioritized) Reasonable? (achievable) Measurable? (quantifiable) Modifiable? (will it change or is it stable?) Traceable? (the source is known) Dependent requirements identified? Testable? (given current environment, resources, skills)
19
Generating TRs
Exit Criteria for Test Requirements

Can another tester create test cases/scenarios from these? Does a Test Requirement specify what is being tested and what about it we are validating? (Clear?) Are the Test Requirements
Complete? Consistent? (conflicting requirements must be prioritized) Reasonable? (achievable) Measurable? (quantifiable for measuring test coverage) Modifiable? (will it change or is it stable?) Traceable? (the source is known) Testable? (given current environment, resources, skills)
Do the test requirements cover the complete scope of the project? Are all the test requirements verified and signed off by the Development Team?
20
Generating TRs
When creating Test Requirements (Do)...

Use action verbs & words
- Validate that - Check for - Test that
Trace them back to the source Remember that different applications arrange in different ways
- Think of MF, batch, C/S, web, e-commerce, GUI, etc. - Use testing considerations checklists that generally cover what kinds of things should be considered when testing your specific situation
Make your Test Requirements document a living 21 document Generating TRs
Also...
Maintain a level of balance between too much & too little
- Too High level: wont be useful, vague, cant generate test cases from it. - Too low level: Over-process, over documentation, no productivity - General rule: 5-7 levels deep in an outline format
Organize them!
22
Outline/Hierarchy format recommended Look at how the BA breaks down the project into areas Look at how the PA breaks down the project into areas Organize by functional areas Organize by types of testing (Function vs. System vs. NonFunctional)
Generating TRs
Agenda
Organizing & Decomposing TRs

Test Requirements Hierarchy Samples Setting the stage for measuring test coverage
23
Distinguishing the types of testing.

I. Function-Based Tests II. User Interface Tests III. Security Tests IV. Installation Tests V. Configuration Tests VI. Performance Tests
(Response)
IX. Stress Tests (breaking point:

memory, resources)
X. Resource Usage Tests XI. Documentation Tests XII. Compatibility Tests XIII. Recovery Tests XIV. Serviceability Tests and others *III - XIV are all Systemsbased tests
Organizing TRs
VII. Load Tests (simultaneous

users, lots of small transactions)
VIII. Volume Tests (Big transactions)

24
Organizing by Functional areas.

Most testers perform function-based, or requirements-based tests
- Tests on functionality that the system will provide - Usually scenario/case based - includes normal and alternate (invalid) cases
25
Organizing TRs
Organizing by Functional areas.

Most testers also perform User Interface Style Tests
- These are generally separate from the functionality that the software will provide - Usually encompass the architectural standards & compliance (like Windows Design Standards) - Also includes tests of navigation, menus, admin functions (like printing, saving)
26
Organizing TRs
Agenda
27
Remember this?Drilling down

Test Requirement Test Scenarios/ Cases Test Procedure/ Script
28 Fitting TRs into the testing picture
Decomposing: Drilling down within a Test Requirement

Business Function
Test Requirement
Keep the function-based perspective in mind!
Tasks within the Function Transactions to perform a task Data Entry Types for transactions Field Validation
Test Scenarios/ Cases Test Procedure/ Script

29
Test Requirement Decomposition

Business Function Tasks within the Function Transactions to perform a task
The types of data entry the transaction can process: Add, Change, Delete, Inquire The "what" that the software is intended to do, like "Rent a Car to a customer" The tasks involved: 1) query car rates, 2) check customer policy coverages, .... The database transactions necessary to perform the task.
Data Entry Types for transactions Field Validation
The edits on the data included within the different types
30
Decomposing TRs

Business Function Tasks within the Function
High level Functional Areas: usually from Functional Spec type docs, or BA work
Transactions to perform a task

Lower level Functional Areas: usually from Technical Spec type docs regarding internal logic, or PA work
31
Data Entry Types for transactions Field Validation
Decomposing TRs

You can start on the high level functional areas early into the project & build in lower level areas as they become available Any level can contain a test requirement which can also be made up of (or broken down into) lower level test requirements
32
Decomposing TRs
Business Function Level

Try to identify groups of functions or functions connected by similar themes
file management functions, printing functions, help functions, car rental functions, reservation functions, ticket purchase functions, claim reporting functions
Be sure all areas of the system are covered. If something is left out or doesnt fit into a group, it becomes its own group. It may be easier to identify functional areas by window instead of by function.
33
Function
Task
Transaction
Trans Data Type
Field Validation
Decomposing TRs
Business Function Level

At this level, the idea is seeing the connections, integration, and interactions of the systems functionality. May not necessarily be identifying a test requirement at this level as much as just identifying the functional area.
34
Function
Task
Transaction
Trans Data Type
Field Validation
Decomposing TRs
Task Level
Break down each Function area into the tasks within the function For complex tasks, it is possible to break them down further into sub-tasks Some Business Functions can not decompose into further tasks (example: Check Writing function)
35 Function
Task
Transaction
Trans Data Type
Field Validation
Decomposing TRs
Transaction Level
From this level down, we start to address the internal things that occur to make the functions and tasks happen Identify any logical transactions that ties the task to the database or any other transactions necessary to perform the task. Identify any data processing, calculation, data formatting type transactions
Note: A screen or window may cause the execution of several different logical transactions
36 Function Task
Transaction
Trans Data Type
Field Validation
Decomposing TRs
Transaction Data Type Level

Identify which of the four types the transaction can become: Add, Change, Delete, Inquire It is entirely possible that a transaction can be multiple types. If a transaction is only one type, you can wrap this level up into the higher level.
37 Function Task Transaction
Trans Data Type
Field Validation
Decomposing TRs
Field Validation Level

Most testers like to jump directly to this level. Its the most obvious at times. Field validation covers all edits & cross edits on fields and data. Be careful of the detail you document at this level. Remember the separation between test requirement and test case.
38
Function
Task
Transaction
Trans Data Type
Field Validation Decomposing TRs
Field Validation Level

Not all test requirements (especially at this level) fit well to be documented in this manner.
- Example: Testing all the stated properties of windows objects - Use Validate that the properties of all the objects in this window match the properties listed on the Object Properties Reference Table in Appendix B upon initialization of the window - Dont list each property check as a separate test requirement if it can be summarize under one test requirement - This is a judgement call YOU make for your given project.
39 Function Task Transaction
Trans Data Type
Field Validation Decomposing TRs
Example 3: Rental Car Application

1. Validate that a Rental can occur. 1.1 Check Customer policy coverage 1.2 Query Car availability Lets look at 1.3 Query Car rates the lower levels for this one 1.4 Open a Rental ticket 1.4.1 Validate that a customer record can be entered 1.4.2 Validate that credit card approval is obtained 1.4.3 Validate that status on the car record is changed from waiting to rented Then well try it 2. Billing Function on this one 3. Reservation Function
40
Decomposing TRs

1. Validate that a Rental can occur. 1.4 Open a Rental ticket 1.4.1 Validate that a customer record can be entered 1.4.1.1 Validate that a new customer can be added to the customer table 1.4.1.1.1 Validate that the first name is all alpha 1.4.1.1.2 Validate that the age is > 21. 1.4.1.1.3 Validate that the phone number is numeric 1.4.1.1.4 Validate area code is an existing area code number. 1.4.1.2 Validate changing an existing customer
41
Decomposing TRs

1. Validate that a Rental can occur. 1.4 Open a Rental ticket 1.4.2 Validate that credit card approval is obtained
fill in the lower level test requirements!

First, Identify any sub-areas (further tasks, or even separate transactions within this) Then, Identify the lowest level field validation test requirements (think about what is typically involved with credit card authorizations)
42
Decomposing TRs
What did you come up with?

1. Validate that a Rental can occur. 1.4 Open a Rental ticket 1.4.2 Validate that credit card approval is obtained _________________________________________ _________________________________________ _________________________________________ _________________________________________ _________________________________________ _________________________________________ _________________________________________ _________________________________________
43
Decomposing TRs
Possible Test Requirements...

Function 1. Validate that a Rental can occur. Task Transaction 1.4 Open a Rental ticket 1.4.2 Validate that credit card approval is obtained
1.4.2.1 Validate the expiration date is a valid future date 1.4.2.2 Validate the expiration date is not within 1 month of expiring. 1.4.2.3 Validate that the CC# is 12 digits 1.4.2.4 Validate that the $ amount is <= credit balance available 1.4.2.5 Validate that an authorization # is received.
44
Decomposing TRs
Agenda
45
TRH Samples
Lets look at a few examples of how test requirements can be organized:
- Sample 1: Excerpt from a Personal Finance app - Sample 2: actually organizing Function requirements, but good representative of the concept - Sample 3: QBS, from Rationals sample project (adds types of testing into the hierarchy) - Sample 4: another view of the Personal Finance app - Sample 5: A mainframe batch system
46
Test Coverage Measures Test Requirements are the what of testing & are the basis for establishing the completion of testing TRs give us the point of measurement for test coverage Each TR should receive a Priority, Risk, and Weight Each TR should be tracked for Verification (3) and Validation (%)
47
Test Coverage Measures
Summary & Recap

Defined: What, Why, Where Described: How TRs fit into the big picture Decomposed: Organizing TRs & Generate TRs into a measurable format Demonstrated: Sampling some TRH Determined: The Measurement Connection
48

Test Requirements

Caricato da

Informazioni sul documento

Descrizione originale:

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Test Requirements

Caricato da

Copyright:

Formati disponibili

David Capocci, CQA, CSTE