2

Agile Test Automation

a Behaviour Driven Approach Ray Hua

1. A journey of test automation 2. Some lessons learned

2 years ago Manual testers

No automation

Now

BDD
Skilled testers

Some automation

A commercial tool Some contractors A project delivery approach Half a million dollars

Some automation that didnt work

An in house solution

A mix of open-source & commercial tools In-house R&D One automation engineer

BAU delivery

Some automation that didnt work

In house solution

QA team on board

Start redesigning test framework Start training testers

Some automation that didnt work

In house solution

QA team on board

Grow regression suite and upgrade technology

7 projects 90% functional coverage Migrated to Selenium/Grid 2, more VMs

Code migration completed by testers

Some automation that didnt work

In house solution

QA team on board

Grow regression suite and upgrade technology

Behaviour Driven Development

Executable requirements BDD engine for .net SpecFlow BA/PM on board

BDD automation, scripted by testers

C# Visual Studio TFS

Nunit
Resharper Selenium/Grid Team City Vmware

Behaviour Driven?
A specialized version of TDD Name of test methods should be sentences Readable when test fails A ubiquitous language for analysis Executable acceptance criteria

Tools
Java JBehave .Net SpecFlow Ruby Cucumber/RSpec

A BDD scenario:
Given the account is overdrawn And the card is valid When the customer requests cash Then ensure a rejection message is displayed And ensure cash is not dispensed And ensure the card is returned.

BDD engine generates stubs (SpecFlow):

[Given(TheAccountIsOverdrawn)] Public void TheAccountIsOverdrawn (){}
[And(TheCardIsValid)] Public void TheCardIsValid (){} [When(TheCustomerRequestsCash)] Public void TheCustomerRequestsCash (){} [Then(EnsureARejectionMessageIsDisplayed)] Public void EnsureARejectionMessageIsDisplayed (){}

[Given(TheAccountIsOverdrawn)] Public void TheAccountIsOverdrawn (){ setupAccount(overdrawn);} [And(TheCardIsValid)] Public void TheCardIsValid (){ setupCard(valid);} [When(TheCustomerRequestsCash)] Public void TheCustomerRequestsCash (){ aCustomer.Setup(standard); customer.requestCash();}

The team loves

Readable test report for PM with colours indicating pass, fails and in-progress
IntelliSense for BA to write stories

Tool generates method stubs to help Testers focusing on what to test next.

Challenges
Unskilled testers
Test framework Unstable tests

Unskilled testers
Start training with motivated testers Set examples Learn by following, then creating Give enough time Always have helps available

Test framework
Use established patterns Leverage application code Make it earlier to understand Naming and conventions

Unstable tests
Decouple tests from build pipeline Categorize tests Leverage application code Wait and retry and verify

Catches
An additional layer = more code
More code = more maintenance Given/When/Then format can be too verbose

Tips
Categorize tests Stabilize tests Speed up tests Design testable architecture

Categorize tests
Warm up tests < 1 minute Smoke tests < 5 minutes Regression tests < 15 minutes Daily regression tests < 1 hour Unstable tests

Stabilize tests
Ensure atomicity Control the environment (system, application and data) Run tests in random order Wait, retry and verify

Speed up tests
Run tests in parallel [Selenium Grid] [SauceLabs] One code base, use application code where necessary Smaller tests

Design testable architecture

Built-in testability Automation hooks E.g. Client side MVC/API/server side MVC Lots of unit tests Less integration tests Even less end-to-end tests

Most Importantly
Keep tests Green (its the heartbeat)

and fix them when they fail

ray.hua@myob.com