TestNG

and improving use of JUnit

Dec 11, 2008

C. Gordon Huffman

Agenda (not chronological)

TestNG testing framework
Improving use of JUnit
Testing integrated classes vs. 1 class

What is TestNG?

Automated testing framework

NG = Next Generation
Similar to JUnit (especially JUnit 4)
Not a JUnit extension (but inspired by JUnit)
Designed to be better than JUnit, especially
when testing integrated classes
Created by Dr. Cdric Beust (of Google)
Open source (http://testng.org)
3

History
sUnit for Smalltalk (~1998, Kent Beck)
JUnit (~2000, Kent Beck & Erich Gamma)
Latest 3.x: 3.8.2 (March 2006)

TestNG (~2004)
Latest: 5.8 (March 2008)

JUnit 4.x (2006)

Latest: 4.5 (August 2008)
4

TestNG philosophy
Use more Java and OO features
Feature-rich (JUnit: simplicity is valued)
Support testing integrated classes (e.g.,
by default, dont create a new test class
instance for every test method).
Separate compile-time test code from runtime configuration/data info.
5

Basic Testing Terms

Test class

Class(es) under test

(Production class)
CUT (class under test)
SUT (system under test)

Rhythm of an Automated Test

Setup (most of it in a separate method)
Exercise (call) production class(es)
Assertion(s); true => silence; false => fail
Teardown (if needed, in a separate method)
7

Test class/method (JUnit 3 vs. 4)

// JUnit 3
import junit.framework.TestCase;
public class MyTestClass extends TestCase {
public void testSomething1() throws ... { ... }
public void testSomething2() throws ... { ... }
}
// JUnit 4
import org.junit.Test;
public class MyTestClass {
@Test public void aTest1() throws ... { ... }
@Test public void aTest2() throws ... { ... }
}
8

Test class/method (JUnit 4 vs. TestNG)

import org.junit.Test;
... or
import org.testng.annotations.Test;

// JUnit 4
// TestNG

public class MyTestClass {

@Test
public void aTestMethod() throws ... { ... }
}

TestNG Assertions
import static org.testng.Assert.*;
import org.testng.annotations.Test;
public class MyTest {
@Test
public void myTestMethod() {
// ... Possibly some setup
// ... Call production method(s)
assertTrue(boolExpression);
// ... more assertions
}
}
10

Assertions: TestNG vs. JUnit 3/4

JUnit 3 assertions inherited by base TestCase
JUnit 4: similar to TestNG
TestNG assertEquals() assertion signatures
are different than JUnits:
JUnit: [msg,] expected, actual
TestNG: actual, expected [, msg]

Can use JUnit 3/4 assertions (library issues)

Can use TestNGs copy of JUnit 3 assertions
(org.testng.AssertJUnit): migration aid.
JUnit4/TestNG assertions richer than JUnit 3;
each contains features that the other doesnt.
11

Expected Exceptions
JUnit 4 (can simplify test code)
@Test(expected = AThrowable.class)

TestNG (can have more than one exception)

@Test(expectedExceptions = AThrowable.class)
Or ...
@Test(expectedExceptions = { T1.class, ... })

12

TestNG Groups

Each test method is tagged with any # of groups.

@Test // no groups
@Test (groups = group1)
@Test (groups = { g1, g2, ... })

A group therefore contains any # of test methods.

Can also annotate test class this way; test method groups are union
of any explicit test method groups, and any of its test class.
Groups can span classes.
Groups can also be externally defined (TestNG xml configuration
file).
A group is identified by a unique string (dont use white space).
There are no pre-defined group names.
E.g., slow, db, ui, unit, integration, broken.unknownReason,
check-in, week-end, functional.billing

13

TestNG Groups (continued)

TestNG community suggests hierarchical
names from more general to less. E.g.:
db.table.CUSTOMER
state.broken.fix-in-progress

Design group names so that you can

select them with prefix patterns.
Groups complement other features (as
well see).
14

Setup/Teardown (JUnit 3/4)

JUnit 3
protected void setUp()
throws Exception { ... }
protected void tearDown() throws Exception { ... }

JUnit 4 (also, @After*; annotation package: org.junit)

@BeforeClass public static void before1() ...
@Before
public
void before2() ...
Multiple per class (order not defined)
These 4 annotations are inherited (order is [kind of] defined).

15

Setup/Teardown (TestNG)
@BeforeMethod
@BeforeClass (no need to be static)
@BeforeGroups ({group1, })
@BeforeTest
@BeforeSuite
And @After*
16

TestNG run-time concepts

A org.testng.TestNG instance runs tests.
Invoked by IDE, ant, or on command line.
Run params specified as setters, or in xml.

Hierarchy of suite/test/class/method.
A TestNG run executes 1 or more suites.
Can be transparent, e.g., run 1 test method via Eclipse.
17

testng.xml
Controls test runs
Optional
File can have any name (.xml suffix advised)
TestNG creates testng-failed.xml, for easy re-run.

Root: <suite> (then <test>/<classes>/<method>)

Suite can also point to a list of additional xml suite files.
Some of the available specifications are described next.

18

testng.xml (continued)
Which potential test methods to run: suite/test: list of
java package name patterns to include/exclude; test: list
of fully qualified class names to include; within a class,
optional (default = include all) list of method names to
include/exclude; test: list of group name patterns to
include/exclude (and can define new groups for this file
with a list of existing group names).
Parameter name/value pairs: Suite/test: (primitive
values) for @Parameters (test values override suite
values).
JUnit semantics: test-level flag (default: false);
migration aid.
19

Disable Tests
TestNG
@Test(enabled = false)
Add to a group which is excluded
Exclude in other ways in testng.xml

JUnit 4: @Ignore, @Ignore(reason)

Can also annotate test class.
Runners report # ignored
20

Test Timeouts
TestNG
@Test(timeout = 1000)
testng.xml <suite|test> time-out attribute
JUnit 4
@Test(timeout = 1000)

21

The Costs of Testing

Time, money, schedule, production scope.
Doubled (very roughly) code base.
Maintenance (production and test).

Sloppily written tests??

Is test code quality as good as production?
Slow tests?
True buy in from business sponsors/owners,
project managers, user reps?
22

High View of Test Benefits

High View: using tests to best advantage

Do the thing right
Do the right thing (acceptance tests)
Safety net for modifications/refactoring
Defect localization (1 fail is more info than 100)
Documentation/training
Minimize costs of testing
This view leads to better tests
23

Integration vs. Unit Tests

Unit test: tests 1 invocation of 1 method of 1
isolated class with 1 thread
Integration test: tests more than 1 method
invocation of an isolated class, often much
more, or more than 1 thread
Distinction: unit test vs. a test written in JUnit
Examples
Test hits db
Life-cycle of an object (e.g., a customer order)
Testing units by testing integrated classes
24

Integration vs. Unit (continued)

Should both be tested?
Unit
How big do you want the holes in your
refactoring safety net?
How many permutations can integration tests
handle?

Integration
If the units work independently, will they work
together?
25

If Unit Tests are Really Integration Tests

Slow tests.
False sense of security for refactoring safety net.
Permutations of business rules likely not
covered.
Poor defect isolation: 100 tests failing is less
information than 1 test failing.
High maintenance for test/production
modification.
Counter to the goals of TDD and Agility.
26

Test Method Dependencies

JUnit 3: Antithetical concept; each test method
gets its own test class instance; high isolation;
unit oriented by intentional design

TestNG:
@Test(dependsOnMethods = {m1, m2, ...})
@Test(dependsOnGroups = {g1, g2, ...})
E.g.: configuration dependencies; production lifecycle

JUnit 4: tests ignored if assumptions (similar to

assertions) are violated: assumeTrue(bool),
assumeNoException(Throwable), assumeNotNull(refs
), assumeThat() high-readability method
27

Parallel (multiple threads)

@Test(threadPoolSize = #,
invocationCount = #,
timeout
= #)

testng.xml:
<suite|test
parallel
=tests|methods|none
thread-count=# (5); 1 if not parallel
time-out
=# ms (default: 0=none)
...>

Testing production code with multiple threads (unless

not supported); stress testing; speeding up tests!
28

Test Method Parameters I

@Test
@Parameters({name1, name2})
public void aTest(String name1, int name2) {
...
}

Values come from testng.xml (<test> is

checked first, then <suite>).
Only basic types (String, [Bb]ool, #s, ).
29

Test Method Parameters II

@Test(dataProvider = myDataProvider)
public void aTestMethod(/* params! */) {
...
}
@DataProvider(name = myDataProvider)
public Object[][] getSomeTestData() {
...
}
Runs (possibly multiple: first []) parameterized tests.
Multiple parameters: second []
Return type can also be Iterator<Object[]>
Data provider itself can take 0, 1, or 2 args: Method, and test context
30

Test Class Factories

@Factory
public static Object[] myTestFactory() {
// return instances of test classes
}
Those instances will also be interrogated for @Factory
methods, and so on. (Each such method invoked once.)
Example use: you want some of your test classes to
have non-standard constructors; the factory constructs
them.
31

Migrating JUnit 3 => TestNG

TestNG tool that modifies source code by
adding imports, @Test, @BeforeTest,
@AfterTest, and an optional list of desired group
names; tool accessible via command line, ant, or
Eclipse/IDEA.
Can use testng.xml <test junit=true>
for tests that need further migration analysis.
Thats not necessarily enough; for other
warnings and tips, see Appendix D of [TestNG].
32

More TestNG and JUnit 4

TestNG and JUnit 4 each have other
features not mentioned here.
For more TestNG features, see [TestNG].
For more JUnit 4 features, see JUnit 4
docs and the source code (e.g.,
Hamcrest).
33

Reference
[TestNG] Next Generation JavaTM Testing;
TestNG and Advanced Concepts; Cdric Beust and Hani
Suleiman; Addison-Wesley; 2008

34