Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
the essence, consider adding more resources to the test process (this also has limited
benefits, and might not always be a solution) rather than embarking on the lengthy
process of implementing test automation. Functional test automation is definitely not
a quick solution to a test project in distress. Multiple other solutions are better suited
to address this need.
"We need to implement testing in our projects. Let's buy a tool and automate the
testing."
You can script to your heart's content, if you do not have a proper manual test process
in place the benefits of test automation will never be seen. It will only assist you in
doing the incorrect process a lot faster. Test automation merely assists and enhances
the test process itself. Ensure that a proper test methodology is implemented before
attempting to add test automation.
"We can just add the test automation to our current test process and start automating
our current test scripts."
Test Automation requires test cases to support the automation process. If my test cases
do not incorporate the all-important logon to the SUT (system under test), the
automation script will need a lot of manual intervention, and would therefore be more
manual than automated. The current test process might have to be changed, or
adapted, to be implemented for the tool suite selected. Review and adapt the test
process to ensure maximum benefit is realised from the tool suite to be used in your
test process.
Best Practice 3: The earlier the involvement in the Software Development Life
Cycle (SDLC) the greater the benefits
We all know how testing has its own phases aligned with the phases of the SDLC. The earlier
testing gets involved in the life cycle of the project the better. Many articles have been written
on this subject, and it holds true. Since test automation supports the testing process, the test
team can get involved very early in the life cycle of a project. If the test team uses a test
automation tool suite, they will most likely have a requirements management tool or test
management tool included in the suite. These tools can assist the test team in various ways
from an early stage in the development life cycle, even before a line of code is developed.
The more mature techniques and methods of test automation are closely tied to the testing
methodology being used. An early start to the test life cycle, at the start of the development
project life cycle, is vital to achieving the best results, and ensuring maximum return on
investment.
Mistakes made
"The test automators cannot get involved in the project before the developers release
the first build."
A software development project does not start with code development. Since
functional test automation is also a development project, as we will discuss in Best
Practice 9, it follows the same phases as a software development project and does not
start at the coding phase. Proper planning and design are essential to the success of the
functional test automation implementation, and need to be conducted hand-in-hand
with the test life cycle.
"Testing is only a single phase in the SDLC and therefore testers and test automation
only need to be part of the project for the testing phase."
This 'prehistoric' misconception in the IT industry has been proven one of the major
reasons for bad software quality.
"We have a test automation tool. Can the test team see if they can use it?"
It is clear that the expectation and an understanding of test automation is lacking. I
can guarantee that the tool implementation will not be successful, and would result in
almost no benefit to the test team. The test team is usually confronted with manual
test work for development projects that does not meet the code delivery date for
testing. The test automation implementation would probably be of the capture-recordand-playback form, and therefore the benefit would be low.
Test Automation will require training for the staff involved with it. Time is required to
design and implement the architecture. Referring to the development nature of
functional test automation implementation, the team will have to learn a new skill set.
"A vendor demonstrated a tool to us and it seems as if the tool will add value. I think
we should buy it."
Take time to define the tool requirements in terms of technology, process,
applications, people skills, and organisation. Then look at multiple tool vendors and
select the right fit for your circumstances. A good idea would be to do a proof of
concept project with the tool that best fits your criteria.
"We purchased the whole suite of tools from the vendor, but are only using the
functional test automation tool."
Test Automation tools are expensive, and to justify the Return On Investment (ROI) it
is normally not effective and efficient to use only the functional test automation tool
in the product suite. Other benefits can be derived from using the full suite of
products; for example, proper defect management, metric reporting on test status, etc.
Purchase what you need, or use what you have fully.
"The yearly license fees are so expensive we cannot afford to use the tool suite any
more."
Too often test automation tools end up as 'shelf-ware' because of the cost of the
licenses compared to the benefit derived. Refer to mistake two above for one of the
reasons why the ROI is not being realised. Before signing the purchase order, be
aware of the impact of annual licensing fees.
"We thought these tools were going to solve your testing problems."
When making the case to management for acquiring test tools, present the challenges
as well as the benefits. Make sure that management understand their influence on how
others will accept automated test tools.
"We purchased a tool for you 3 months ago, and yet we cannot see any benefits in the
project."
Communicate to management from the start that it takes time and planning to build a
firm foundation of people, processes, and the right tools. Keep management informed
of tool progress, and any issues that arise.
to the most mature, viz., keyword or action based. These techniques each have their own
benefits and drawbacks, appropriate for certain specific circumstances.
During a small data conversion project, a few lookup data tables' contents in the old system
need to be compared to the new lookup data tables' contents after the batch conversion. If this
can be done via the enquiry functions of both systems, the fastest method might be recordand-playback. This is a once off test of the conversion, and writing specific scripts to do this
might take longer than the record-and-playback method. In terms of cost and timing, the
record-and-playback technique would be the most appropriate for this project. This is one of
the few instances where I would suggest using the record-and-playback technique; most of
the time the benefits of this technique are few compared to the more mature techniques like
data-driven and keyword test automation.
It is important to know, and be able to use, multiple techniques appropriately. If we do not use
the various techniques when appropriate, our test automation will not provide us with the
benefits we are aiming for.
Mistakes made
"We have created our test cases for the project and are approaching the release of the
first build. Can we implement keyword test automation before the end of the project."
Keyword test automation is a very advanced test automation technique, and is closely
tied to the testing methodology being used for the project. The preparation work for
implementing keyword test automation requires more time than some other
techniques. The framework needs to be established, and test design needs to support
the method. I would not suggest implementing the method from scratch at this late
stage of the project.
Proper analysis needs to be done to identify the correct test cases to automate to ensure ROI
for the effort. How to select tests for automation is a subject on its own, but the following
suggestions can be helpful:
1. Perform a proof-of-concept (POC) on a cross section of test cases for the system to
determine technical applicability and return on investment (ROI) metrics.
2. Perform an ROI analysis for each testing regimen to select automation candidates.
3. Perform a risk analysis on the automation candidates to prioritise what should be
automated first.
If the tests selected for functional test automation are identified and prioritised, the test cases
with the best potential to provide benefit will be automated first. If time constraints affect the
test automation delivery, at least we know the largest possible benefit would be derived.
Mistakes made
Test automation automates a task in which the programmer is probably not an expert.
Therefore, expert testers should be consulted and provide the requirements.
Test automation benefits can be seen if we design our approach before we start
coding.
Test automation code needs to be tracked and safeguarded. Therefore, we need to use
source code management.
Test automation will have bugs. Therefore, we need to plan to track them, and test for
them.
Users will need to know how to use it. Therefore, we need user documentation.
The type of work done during test automation is software code development, and since these
are programs, they must be managed in the same way that application code is managed.
Mistakes made
"When the code is complete, give it to the testers so they can implement test
automation."
Why would we follow a phased approach to software development, but not with test
automation script development? We should enforce the same stringent processes and
phases on test automation scripters as on developers and the scripts need to be
tested.
"You don't need to follow all the phases of the software development life cycle with
test automation."
The fact that we are writing software to test software, and not to perform some
business scenario, does not mean the method is different. What is important for the
one is important for the other; both are software development projects.
in-depth knowledge and practical experience with the tool being used for the test
automation.
Because of this and because 'record-and-playback' (which generally does not require learning
the back-end scripting) is not a valid way to automate most tests, the use of these tools
becomes like that of a development effort. Given that such is the case, there must be support
within the organisation to realise this and hire staff accordingly, as well as giving the
necessary time for the development.
To accomplish this, a Test Automation Tool Specialist, or similar, position must be created,
and staffed with at least one senior-level programmer. It does not really matter in what
languages the programmer is proficient; what is important is that this person must be capable
of designing, developing, testing, debugging, and documenting code.
Test Automation is a combination of testing and development. To mentor and teach your team
and resources, use consultants as appropriate to bootstrap your effort. Learn as much as
possible from theses consultants to enable you to avoid the stumbling blocks, and proceed
successfully with your automation effort after they depart.
Mistakes made
"We sent our testers on a test automation course, and they still can't implement proper
test automation scripts."
Learning the test tool Integrated Development Environment (IDE) does not make you
a good scripter. A 3-day course will not teach you programming principles. Mentoring
by a developer can accelerate the learning process. Add a person to the test team who
is a test scripter. This person should be comfortable working with code, and be able to
take the basic test designed by a test analyst and convert it into an automated script.
Avoid the creation of disposable automation through proper planning and design of your test
automation implementation.
Mistakes made
Best Practice 12: Review and improve implementation process after each
project
Development practices suggest having 'post-mortems' at the end of a project. This holds true
for the test automation project as well. The following information will surface during a 'postmortem':
New ways to deal with specific issues that were discovered during the project may be
suggested.
Experiences gained will provide insight into changes required for the current process
or architecture in use.
Ideas may emerge that may enhance and make the current process more efficient.
Use the information from the 'post-mortem' to learn and adapt your processes and best
practices to gain maximum benefit from your test automation implementation.
Mistakes made
"We are still implementing test automation in the same manner as we did on our first
test automation project in the organisation."
It is clear that the team/organisation is not learning from its experiences. Learning
from our experiences allows us to change our ways for the better, adapt processes for
maximum benefit, and make life easier for ourselves. The test team needs to review
what works for them and what does not, and adapt their methods to improve for future
projects.
Conclusion
Once you determine your company profile, perfect your processes, establish test specialists,
give the team members appropriate testing tools, and follow the best practices laid out in this
article; your company can realise the benefits of automated software testing. When compared
to manual testing, properly applied automation will result in higher quality products, lower
risk to your company, faster approval, and decreased time to market.
The higher level you reach on the automated software testing maturity ladder, the more
benefits you will realise. Whatever level you choose, however, keep in mind a major lesson
of the past few decades: No matter what tools you buy, your largest investment by far will be
in the processes and people you put in place to use those tools.
Use the best practices as specified by those who have travel-led the path to successful test
automation implementation, and adapt these practices to fit your organisation's individual
needs by learning from your test automation implementation.
Henk Coetzee