Sei sulla pagina 1di 11

July/August 2005 Feature Article

Best Practices in Software Test Automation


http://www.testfocus.co.za/featurearticles/july2005.htm

What are Best Practices?


Best practices are guidelines and advice on the best way to do something; collected over time
and based on experience with previous projects. Best practices in an organisation should
come from the bottom of the organisation up, rather than being mandated by management.
These best practices might even differ from organisation to organisation, but certain key
factors are present in successful organisations and projects. Discussed in this article are
twelve of the most important practices that can assist you in ensuring successful
implementation of functional test automation on your projects and in your organisation.

Best Practice 1: Know your objective


There are many good reasons for doing test automation.
Test automation can save time, make testing easier, and improve the testing coverage. It can
also help keep testers motivated; I can list a page of other benefits that can be derived from
doing functional test automation. However, it is not likely that your organisation will need to
do all these things at the same time. Different groups typically have different hopes and ideas
of what they want test automation to offer them. These need to be stated, or else
disappointment is likely.
A clear objective to work towards during test automation is highly important. It provides us
with a compass heading towards which planning and implementation will be directed.
Eventually, the results of the test automation implementation will be compared against the
original objective to establish whether we did reach our goal or not.
Mistakes made...

"Test automation is the answer to our testing problems"


Test automation assists in the testing process, and can therefore provide value to a test
team. Certain benefits can be derived, but we need to know and understand exactly
what they are without having false expectations or hopes. Automation is not a
replacement for an inadequate test process or an inefficient test team. Address the
testing problems prior to embarking on a costly test automation implementation.
Automation's objective is definitely not to solve all testing problems in existence.

"Since we have limited time to test, let's use test automation."


Test automation takes 3 to 128 times longer to implement than manual testing, and
therefore it should be properly planned to achieve the expected benefits. If time is of

the essence, consider adding more resources to the test process (this also has limited
benefits, and might not always be a solution) rather than embarking on the lengthy
process of implementing test automation. Functional test automation is definitely not
a quick solution to a test project in distress. Multiple other solutions are better suited
to address this need.

Best Practice 2: Test Automation requires a manual test process


"Success in test automation requires careful planning and design work, and it's not a universal
solution. ... automated testing should not be considered a replacement for hand testing, but
rather as an enhancement." (James Bach [pg. vii] in his foreword: Software Testing with
Visual Test 4.0, Thomas Arnold II, IDG Books Worldwide Inc. Foster City, CA.: 1996. ISBN
0-7645-8000-0.)
The words of James Bach and many others are quite clear on this subject. Test automation
will not give you a correct testing process or methodology. It will not enforce a methodology
or process. Test automation merely supports the test process; it will not replace the manual
test process.
Mistakes made ...

"We need to implement testing in our projects. Let's buy a tool and automate the
testing."
You can script to your heart's content, if you do not have a proper manual test process
in place the benefits of test automation will never be seen. It will only assist you in
doing the incorrect process a lot faster. Test automation merely assists and enhances
the test process itself. Ensure that a proper test methodology is implemented before
attempting to add test automation.

"We can just add the test automation to our current test process and start automating
our current test scripts."
Test Automation requires test cases to support the automation process. If my test cases
do not incorporate the all-important logon to the SUT (system under test), the
automation script will need a lot of manual intervention, and would therefore be more
manual than automated. The current test process might have to be changed, or
adapted, to be implemented for the tool suite selected. Review and adapt the test
process to ensure maximum benefit is realised from the tool suite to be used in your
test process.

Best Practice 3: The earlier the involvement in the Software Development Life
Cycle (SDLC) the greater the benefits
We all know how testing has its own phases aligned with the phases of the SDLC. The earlier
testing gets involved in the life cycle of the project the better. Many articles have been written
on this subject, and it holds true. Since test automation supports the testing process, the test
team can get involved very early in the life cycle of a project. If the test team uses a test
automation tool suite, they will most likely have a requirements management tool or test

management tool included in the suite. These tools can assist the test team in various ways
from an early stage in the development life cycle, even before a line of code is developed.
The more mature techniques and methods of test automation are closely tied to the testing
methodology being used. An early start to the test life cycle, at the start of the development
project life cycle, is vital to achieving the best results, and ensuring maximum return on
investment.
Mistakes made

"The test automators cannot get involved in the project before the developers release
the first build."
A software development project does not start with code development. Since
functional test automation is also a development project, as we will discuss in Best
Practice 9, it follows the same phases as a software development project and does not
start at the coding phase. Proper planning and design are essential to the success of the
functional test automation implementation, and need to be conducted hand-in-hand
with the test life cycle.

"Testing is only a single phase in the SDLC and therefore testers and test automation
only need to be part of the project for the testing phase."
This 'prehistoric' misconception in the IT industry has been proven one of the major
reasons for bad software quality.

Best Practice 4: Test Automation is a fulltime effort


When people are allowed to work on test automation on their own time, or as a back burner
project whenever the test schedule allows, test automation becomes a sideline effort. Test
Automation does not get the time and focus it needs.
Getting into automated testing requires preparing not only yourself, but also your company
and environment. This requires focus and dedicated resources.
Mistakes made

"We have a test automation tool. Can the test team see if they can use it?"
It is clear that the expectation and an understanding of test automation is lacking. I
can guarantee that the tool implementation will not be successful, and would result in
almost no benefit to the test team. The test team is usually confronted with manual
test work for development projects that does not meet the code delivery date for
testing. The test automation implementation would probably be of the capture-recordand-playback form, and therefore the benefit would be low.
Test Automation will require training for the staff involved with it. Time is required to
design and implement the architecture. Referring to the development nature of
functional test automation implementation, the team will have to learn a new skill set.

Test automation implementation requires dedication and focus to ensure success. In


this instance, it is better to not implement test automation than to allocate ad hoc time
for experimentation.

Best Practice 5: Select the correct product or suite of products


Buying the wrong tool is listed as one of the major challenges in test automation, because no matter what kind of process, methodology, or organisation you have - if the tool is not a
good technical and business fit, it will not be used.
We know that a good process and organisation are also essential for test automation.
However, if the tool will not function at a basic level, the people who should use the tool, and
thereby gain the benefit from the tool, will not use it.
Unfortunately, too few people do adequate research before buying a test tool. Adequate
research includes defining a set of tool requirements based on what the intended users of the
tool need to accomplish, developing a set of evaluation criteria by which candidate tools will
be judged, and taking the experience of other people who have used the tools into
consideration.
Select a tool that will allow you to implement automated testing in a way that conforms to
your long-term testing strategy and test methodology.
Mistakes made

"A vendor demonstrated a tool to us and it seems as if the tool will add value. I think
we should buy it."
Take time to define the tool requirements in terms of technology, process,
applications, people skills, and organisation. Then look at multiple tool vendors and
select the right fit for your circumstances. A good idea would be to do a proof of
concept project with the tool that best fits your criteria.

"We purchased the whole suite of tools from the vendor, but are only using the
functional test automation tool."
Test Automation tools are expensive, and to justify the Return On Investment (ROI) it
is normally not effective and efficient to use only the functional test automation tool
in the product suite. Other benefits can be derived from using the full suite of
products; for example, proper defect management, metric reporting on test status, etc.
Purchase what you need, or use what you have fully.

"The yearly license fees are so expensive we cannot afford to use the tool suite any
more."
Too often test automation tools end up as 'shelf-ware' because of the cost of the
licenses compared to the benefit derived. Refer to mistake two above for one of the
reasons why the ROI is not being realised. Before signing the purchase order, be
aware of the impact of annual licensing fees.

Best Practice 6: Get executive management commitment


Test automation can yield very substantial results, but requires a substantial commitment to
be successful. Do not start without commitments in all areas, the most important being
executive management.
'Thomas R. Arnold II, VP at ST Labs, Inc., sums it up fairly well:
"I would like to say up front ... that no test automation tool is a silver bullet. Automation
takes time, effort, and commitment from all involved, including an understanding from
management about the realities of what automation can and cannot do." ' 1
Management support is needed in designing and deploying test processes that will support the
effective use of test tools, reinforce the role and use of automated test tools in the
organisation, and allow time for tools to be integrated in the testing process.
Without management support, the entire test automation effort is at risk. If management does
not - clearly and consistently - show their support for test automation, people will be less
inclined to show interest in using the tools. This is a major concern, especially considering
that overcoming the learning curve of some tools requires dedication.
Perhaps the greatest challenge seen in management support is balancing the high expectations
of tool benefits against the time, effort, and discipline it takes to implement the tool.
Management may become impatient about the lack of tool progress and shift their support to
other initiatives.
Mistakes made

"We thought these tools were going to solve your testing problems."
When making the case to management for acquiring test tools, present the challenges
as well as the benefits. Make sure that management understand their influence on how
others will accept automated test tools.

"We purchased a tool for you 3 months ago, and yet we cannot see any benefits in the
project."
Communicate to management from the start that it takes time and planning to build a
firm foundation of people, processes, and the right tools. Keep management informed
of tool progress, and any issues that arise.

Best Practice 7: Select the appropriate technique/s for the project


As maturity levels improve in the functional test automation field, we see a decrease in cost
for test automation implementations, and an increase in benefits derived for the project or
organisation.
The increase in maturity levels has provided us with more mature techniques for
implementing test automation. These techniques vary from the simplest record-and-playback

to the most mature, viz., keyword or action based. These techniques each have their own
benefits and drawbacks, appropriate for certain specific circumstances.
During a small data conversion project, a few lookup data tables' contents in the old system
need to be compared to the new lookup data tables' contents after the batch conversion. If this
can be done via the enquiry functions of both systems, the fastest method might be recordand-playback. This is a once off test of the conversion, and writing specific scripts to do this
might take longer than the record-and-playback method. In terms of cost and timing, the
record-and-playback technique would be the most appropriate for this project. This is one of
the few instances where I would suggest using the record-and-playback technique; most of
the time the benefits of this technique are few compared to the more mature techniques like
data-driven and keyword test automation.
It is important to know, and be able to use, multiple techniques appropriately. If we do not use
the various techniques when appropriate, our test automation will not provide us with the
benefits we are aiming for.
Mistakes made

"Record-and-playback is sufficient for our needs."


The benefits of this simplistic technique are few, and its application is widely
misused. More mature techniques are available with exponentially more benefits in
terms of ease of use, cost saving, and time saving if implemented correctly.

"We have created our test cases for the project and are approaching the release of the
first build. Can we implement keyword test automation before the end of the project."
Keyword test automation is a very advanced test automation technique, and is closely
tied to the testing methodology being used for the project. The preparation work for
implementing keyword test automation requires more time than some other
techniques. The framework needs to be established, and test design needs to support
the method. I would not suggest implementing the method from scratch at this late
stage of the project.

Best Practice 8: Do not attempt to automate all tests


Test Automation does not work well for all situations or tests. Some tests are better left for
manual execution. If a test will not be repeated more than three times, it is probably not a
good candidate to automate. It takes a lot longer to automate a test than to execute it
manually. The benefit of the automated tests is linked to how many times it would be
repeated. For example, multiple builds, or testing on multiple platforms, or executing the
same test for multiple data sets.
Software that is tested manually will be tested with a randomness that helps find bugs in more
varied situations. Since a software program usually will not vary, automated testing may not
find some bugs that manual testing will. Automated software testing is never a complete
substitute for manual testing.

Proper analysis needs to be done to identify the correct test cases to automate to ensure ROI
for the effort. How to select tests for automation is a subject on its own, but the following
suggestions can be helpful:
1. Perform a proof-of-concept (POC) on a cross section of test cases for the system to
determine technical applicability and return on investment (ROI) metrics.
2. Perform an ROI analysis for each testing regimen to select automation candidates.
3. Perform a risk analysis on the automation candidates to prioritise what should be
automated first.
If the tests selected for functional test automation are identified and prioritised, the test cases
with the best potential to provide benefit will be automated first. If time constraints affect the
test automation delivery, at least we know the largest possible benefit would be derived.
Mistakes made

"It was so much fun that we automated all the tests."


It might have been fun, but the ROI was definitely not optimal. Time spent on
automation of certain tests would never be recovered. A general rule of thumb would
be never to automate more than 60% of your test cases to ensure maximum benefit
with the lowest investment cost.

Best Practice 9: Manage the automation as a development project


Test Automation development is a software development process, although it is seldom
treated that way. Following software development practices can make the difference between
the success and failure of an automated testing project.
We need to run test automation projects just as we do our other software development
projects. The following pertains to test automation projects, and is true for development
projects:

Projects need developers dedicated to developing test automation.

Test automation automates a task in which the programmer is probably not an expert.
Therefore, expert testers should be consulted and provide the requirements.

Test automation benefits can be seen if we design our approach before we start
coding.

Test automation code needs to be tracked and safeguarded. Therefore, we need to use
source code management.

Test automation will have bugs. Therefore, we need to plan to track them, and test for
them.

Users will need to know how to use it. Therefore, we need user documentation.

The type of work done during test automation is software code development, and since these
are programs, they must be managed in the same way that application code is managed.
Mistakes made

"When the code is complete, give it to the testers so they can implement test
automation."
Why would we follow a phased approach to software development, but not with test
automation script development? We should enforce the same stringent processes and
phases on test automation scripters as on developers and the scripts need to be
tested.

"You don't need to follow all the phases of the software development life cycle with
test automation."
The fact that we are writing software to test software, and not to perform some
business scenario, does not mean the method is different. What is important for the
one is important for the other; both are software development projects.

Best Practice 10: Use the correct resources


One area consistently missed by organisations desiring to automate testing is the staffing
issue.
Automated testing is different from other types of testing:
1. Most tools use procedural languages to create their scripts.
2. These tools are very similar to IDEs (Integrated Development Environments).
3. Debugging scripts is the same as debugging traditional software.
and therefore, resources are going to need:

academic-level knowledge of procedural coding languages,

basic knowledge of coding practices and procedures, and

in-depth knowledge and practical experience with the tool being used for the test
automation.

Because of this and because 'record-and-playback' (which generally does not require learning
the back-end scripting) is not a valid way to automate most tests, the use of these tools
becomes like that of a development effort. Given that such is the case, there must be support
within the organisation to realise this and hire staff accordingly, as well as giving the
necessary time for the development.

To accomplish this, a Test Automation Tool Specialist, or similar, position must be created,
and staffed with at least one senior-level programmer. It does not really matter in what
languages the programmer is proficient; what is important is that this person must be capable
of designing, developing, testing, debugging, and documenting code.
Test Automation is a combination of testing and development. To mentor and teach your team
and resources, use consultants as appropriate to bootstrap your effort. Learn as much as
possible from theses consultants to enable you to avoid the stumbling blocks, and proceed
successfully with your automation effort after they depart.
Mistakes made

"We sent our testers on a test automation course, and they still can't implement proper
test automation scripts."
Learning the test tool Integrated Development Environment (IDE) does not make you
a good scripter. A 3-day course will not teach you programming principles. Mentoring
by a developer can accelerate the learning process. Add a person to the test team who
is a test scripter. This person should be comfortable working with code, and be able to
take the basic test designed by a test analyst and convert it into an automated script.

"The testers did attend the vendor training."


The vendor training will train your staff in the product. Teaching someone Microsoft's
development studio will not make them a code writer or programmer. The person will
merely know how to use the IDE to write code. It is going to require more than just
vendor training to turn your staff into test automators. I suggest hiring a test
automation consultant to assist with getting the test team out of the starting block. The
consultant can train and mentor your staff, and teach them how to avoid the most
common pitfalls. Start the team with simple basic scripting concepts, and add
complexity later.

Best Practice 11: Develop for maintenance


Without proper automation architecture planning and design the test team will soon find itself
with hundreds or thousands of test scripts, thousands of separate run result files/logs, and the
combined work of maintaining the existing scripts for various versions of the system, as well
as creation of new scripts for new enhancements. Soon they will find themselves with a
maintenance nightmare.
To avoid the maintenance problems that so many organisations succumb to, the following
pointers can help:

Create a reusable architecture.

Develop generic, cross application initialisation and configuration parameters.

Create reusable, cross application functions.

Design test scripts by assembling components.

Avoid the creation of disposable automation through proper planning and design of your test
automation implementation.
Mistakes made

"We did not design with maintenance in mind."


If you did not start your test automation with a design for maintenance, you have
probably already become a victim of the maintenance nightmare. Review your current
scripts and determine what amount of script/code snippets can be reused if you
implement a different approach (maintenance focused). If the amount of rework is
large, it might be better to restart your effort with the correct approach in mind.

"No standards were followed in script creation."


This will also result in difficulty during the maintenance phase; or if the resource that
created the scripts leaves for greener pastures. Coding and scripting standards allow
for ease of maintenance for the whole team. Different people can understand each
other's scripts with ease, and when someone leaves the team it is not the end of the
world. When test automators return to scripts that they have not worked on for a while
it is also easier to understand what they did the first time round. This leads to a
reduction in script maintenance time.
Start to implement standards and good coding practices for the test automators or
scripters in your project/organisation. It is never too late to start with this practice.

Best Practice 12: Review and improve implementation process after each
project
Development practices suggest having 'post-mortems' at the end of a project. This holds true
for the test automation project as well. The following information will surface during a 'postmortem':

New ways to deal with specific issues that were discovered during the project may be
suggested.

Experiences gained will provide insight into changes required for the current process
or architecture in use.

Ideas may emerge that may enhance and make the current process more efficient.

Metrics to be reviewed may provide valuable information for the enhancement of


processes.

Further training or mentoring needs may emerge.

Use the information from the 'post-mortem' to learn and adapt your processes and best
practices to gain maximum benefit from your test automation implementation.
Mistakes made

"We are still implementing test automation in the same manner as we did on our first
test automation project in the organisation."
It is clear that the team/organisation is not learning from its experiences. Learning
from our experiences allows us to change our ways for the better, adapt processes for
maximum benefit, and make life easier for ourselves. The test team needs to review
what works for them and what does not, and adapt their methods to improve for future
projects.

"What benefits did we derive from this automation effort."


This should be easy to answer if we followed Best Practice 1: Know your objective.
We set out with a goal in mind and, while we implemented the test automation, we
worked towards our goal. At the end of the project, we should know if we achieved
what we set out to do. If we cannot answer, it is time to implement Best Practice no.
1.

Conclusion
Once you determine your company profile, perfect your processes, establish test specialists,
give the team members appropriate testing tools, and follow the best practices laid out in this
article; your company can realise the benefits of automated software testing. When compared
to manual testing, properly applied automation will result in higher quality products, lower
risk to your company, faster approval, and decreased time to market.
The higher level you reach on the automated software testing maturity ladder, the more
benefits you will realise. Whatever level you choose, however, keep in mind a major lesson
of the past few decades: No matter what tools you buy, your largest investment by far will be
in the processes and people you put in place to use those tools.
Use the best practices as specified by those who have travel-led the path to successful test
automation implementation, and adapt these practices to fit your organisation's individual
needs by learning from your test automation implementation.
Henk Coetzee

Potrebbero piacerti anche