STEP AUTO Colloquium on Test 12/10/2007

Automation ,6th Dec '07

Automation beyond testing tools

Srinivasan Desikan

© 2007 Hewlett-Packard Development Company, L.P.

The information contained herein is subject to change without notice

Question

How many programmers does it take to change

a light bulb?

Automation many times gets the wrong set of people for the
job
• Either a test engineer who can’t program
• Or a Programmer who can’t test
• Or a person who is neither programmer nor a tester,
given to automation saying “Automate if you have nothing
else to do”
December 10,
2 2007

Automation beyond Testing Tools ,

Srinivasan Desikan -HP 1
STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07

Automation Survey – Why you are not automating?

No belief Not aw are

0 6

Inadequate
Features
33 Expensive
61

The job of automation include understanding where tools can help and where
it can’t…
The job of automation starts in creating value on top of commercially
available tools as any tool will not meet 100% of your requirements…
December 10,
3 2007

Automation Survey – What did you automate?

Easy
10

Suitable
39
Regression
28

All
23

• Only Low-hanging fruits are plugged

• 23% got into “trap” as
• There is no end to testing ; so 100% is not possible
• There may not be ROI if all are automated (Waste of effort)
• We have a long way to go to say “we are automating complex test
cases” that gives us great business value
December 10,
4 2007

Automation beyond Testing Tools ,

Srinivasan Desikan -HP 2
STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07

Automation Survey – What tools you used?

OS Features 35

In house 45

Open source 42

COTS 78

0 20 40 60 80 100

• 35+45+42+78 = 180%; Therefore people on an average use 1.8 tools

which is not a good number to get a good coverage on automation
• There is heavy dependence on COTS and awareness and usage of
Open source/in-house tools to improve

December 10,
5 2007

Automation Survey – Difficulties in automation

More efforts 29
25
More time 17
35
Skilled people 48
45
Usage 17

0 10 20 30 40 50 60

• Automation is kept in the freezer because it requires more efforts and

more complex
• Lack of training available on automation. Lack of usage knowledge
infact is a great difficulty in automation
• Eventhough industry produced the required numbers for automation,
the skills of people has a long way to go (Quantity Vs Quality)
December 10,
6 2007

Automation beyond Testing Tools ,

Srinivasan Desikan -HP 3
STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07

Automation Survey – Satisfied with automation

No
0
Not yet High
29 27

Just
44

• Only 27% of people satisfied with automation ; More we do

automation more we get satisfied with it
• This percentage drops further below if we exclude the people we
were part of the automation project ; Business value out of
automation as perceived by mgmt and outside testing team is very
low
December 10,
7 2007

Automation Survey – Quality of automation

• Only 2% of people share their automation suite with people outside the
team (they are not easy to use, and have potential quality issues)
• 95% of people say that “They repeat the tests manually when
automated tests bring up defects”; this shows
− Trust level of test suite is very low
− It defeats the purpose of having automated the tests
− More needs to be done to make the quality of test suite better
− Quality requirements for automation should be better than or equal to those
criteria set for products
• Only 1% people share their test suite with their customers
− Automation is not done with the perspective that “customers also do testing
and it should save their time too”
− Not many realize that automation is yet another business opportunity (Many
of the commercial tools in market today were in-house tools before)

December 10,
8 2007

Automation beyond Testing Tools ,

Srinivasan Desikan -HP 4
STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07

Automation beyond testing tools & Test cases

• Test reports are automatically prepared by automation scripts
• Test defects are automatically filed in defect tracking system
• Test metrics are updated/created by automation scripts
• The test results are updated automatically by automation
• Traceability matrix updated with mapping to requirements and test
cases
• Keep looking at defect status and selecting test cases automatically for
regression testing (Keeping the track of what defects corresponds to
what test cases)
• Calculating code coverage and test coverage
• Suggesting the improvements in test management
− A list of test cases that were never executed for a long time
− A list of test cases that always failed (Then why execute them again and
again)
− A list of test cases that was not executed for long (to execute them atleast
now or delete them from database)
− December
A list10,of test cases that always passed (Do we need to really execute them
9 again and again)
2007

Survey results – Where test engineers spend

time?
5% 5% 10% Automation
5%
Test execution
15%
10% Test Setup

Reports

Waiting for build

test case design

10% Test plan

5% Review s
35% Others

• Very little investment on automation

• Automation in lab setup will give better ROI

December 10,
10 2007

Automation beyond Testing Tools ,

Srinivasan Desikan -HP 5
STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07

References

1. Test automation survey results – Jayakumar. K.R, AmitySoft,

Chennai (Necessary approvals obtained)
2. Book: “Software Testing – Principles and Practices”,
Srinivasan Desikan & Ramesh Gopalaswamy, ISBN:
817758295X

Disclaimer: The views expressed in slides and in the presentation are my own and not that
of my employer
December 10,
11 2007

Questions?

December 10,
12 2007

Automation beyond Testing Tools ,

Srinivasan Desikan -HP 6
STEP AUTO Colloquium on Test 12/10/2007
Automation ,6th Dec '07

Thank You !

December 10,
13 2007

Automation beyond Testing Tools ,

Srinivasan Desikan -HP 7