Comparison of the Features of GUI Testing Tools
A.Askarunisa, D.Thangamari
nishanazer@yahoo.com, thangam@tce.edu
Abstract- Testing software manually is a labour intensive
process. Efficient automated testing significantly reduces the
overall cost of software development and maintenance. GUI test
automation is a major challenge for test automation. Different
kinds of automated tools are available in market for various types
of GUI application testing. This paper performs an analysis of
various GUI tools based on some specific features so as to
enable the tester to choose an appropriate tool to satisfy his
requirements and to perform efficient testing of GUI
applications. And also this paper proposes GUI Automation testing
technique to test GUI-Based java programs as an alternative to the
CR technique. This technique develops GUI-event test specification
language for GUI application written using java swing APIs, which
initiates an automated test engine. Visual editor helps in viewing the
test runs. The test engine generates GUI events and captures event
responses to automatically verify the results of the test cases. This
includes the test case generation, test case execution and test case
verification modules. The testing efficiency is measured by
determining coverage metric based on Code coverage, while may be
useful during Regression Testing. This paper uses Abbot and JUnit
tools for test case generation and execution and Clover tool for code
coverage. We have performed tests on various GUI applications and
the efficiency of this technique is provided.
Keywords—Abbot, Capture Reply, Code Coverage, GUI Testing,
Unit Testing
I. INTRODUCTION
Graphical User Interfaces (GUIs) have become an
important and accepted way of interacting with today’s
software. There are different types of GUI testing [4]
v i z . Manual testing, Static Analysis, Usability testing and
Functional testing. The manual testing is user verification of
requirements using some screenshots and
sample inputs and outputs. The static testing is a type of
unconventional testing i.e. testing performed on an
application when it is not executed. It is also called
Verification or in process reviewing. Usability testing
satisfies user requirements. It captures the user actions and
gets the data. Functional testing satisfies the functional
requirements which generally have a separate list of
requirements for GUIs from which test c a s e s a r e
d e v e l o p e d . The t e s t c a s e s a r e executed and the
output is compared with expected results. It is divided into
two states namely verifier and level of testing. Verifier is
used for verification of the actual output with manual
output. The level of testing includes the functional testing
such as; Changed property set (Operators), GUI relevant
property set (Specification) and Complete property set
(Toolkit/Language). The most reliable way to ensure that
a GUI works as expected is through functional testing.
Unlike unit tests, which focus on testing classes in isolation,
functional tests confirm whether the system behaves as
expected by the user. GUI tests should be able to simulate
user input to verify that the application will behave as
expected in front of the user [1],[15].GUI tests should
have a reliable way to find GUI components. GUI tests need
to tolerate changes in the position, size and layout of any
GUI component. Changes in the appearance of an
application must not break any GUI tests. Writing GUI
test programs is very tedious (they are always
lengthy and difficult to read).Thus, CR (Capture and
replay) script technique becomes an attractive
technique. With a CR tool, a test designer interacts
with the GUI of the Application Under Test (AUT)
and all the events are recorded in a test script. Several
approaches are available for preparing test scripts. Tools
are mainly based on Automation Testing approaches.
Some of the GUI Testing approaches are manual testing,
and Static analysis and functional testing.
In or de r t o p er fo rm f u n c t i o n a l t e s t i n g
different kinds of automated tools [3] are available in
market. It will be quite hard for the tester to choose an
appropriate tool for satisfying his requirements. In
running automatic tests for the GUI application, the tester
saves much time, especially when he is in a huge
production house and needs to be multi-tasking This
paper is organized as follows: Section 2 describes the
different approaches on GUI Testing, including a
description of the importance of Automation testing.
Section 3 presents the categories of Automation
Testing. Section 4 presents the d i s c u s s i o n an d
analysis of various GUI Testing tools. Section 5
presents the analysis of selected tools for GUI testing.
Section 6 discusses why Abbot tool is the best option
for GUI application testing. Section 7 presents the
implementation of the Abbot tool through which test
case generation, execution, verification and code
coverage are done. Finally, Section 8 gives the
conclusion and future enhancement.
II. APPROACHES IN TESTING GUIS
Testing a GUI application seems to be easy at
first. You don’t have to write any code, you just have
to click and see if it works. This native approach works
only for GUIs of limited complexity. Even quite simple
software has surprisingly complex GUIs and the amount
of work required to test the software will grow fast. It
is also quite difficult to envision all ways in which a
GUI can break, and sooner or later the testers will grow
so accustomed to testing the same GUI that they
will start to miss opportunities to make it fail.
A. Automation
Multiple a v e n u e s o f a p p r o a c h h a v e been
used in the attempt to conquer the test automation
problem. Since maintaining GUI tests are expensive,
some have tried to automate the generation of tests.
Other approach to the same problem is to model the test
cases so that they can be automatically repaired when
changes are done to the User Interface (UI)[6]. Both these
solutions are usually based on the modelling of the UI as
a set of events and states, which are then combined into”
components”, which can be tested separately. The
common feature is that these events sometimes have to
be tested in sequence like cut-paste. Generally in auto
generated tests the event sequences could grow to infinite
length. Some limit has to be imposed and according to
studies there is no point in testing very long event
sequences except in special cases. From studies [5 ],[8],[11]
it is clear that event coverage (each event tested at least
once) guarantees very high statement coverage even at
sequence length of 1. Lengthening the sequence makes the
statement coverage approach 100%.
B. Why write automated tests?
Automated tests m a k e s u r e t h a t c o d e works
(specification) and shows the intended use and the behaviour
of the code. Through this things do not get wrong
(integration) and mistakes do not reappear (regression) i.e.
They verify bug fixes. Most of the automated tests detect
unused code (code coverage) and delete obsolete code or
improve tests. Automated tests generally measure the UI
responsiveness (performance). This is the only way to check
the daily performance.
III. C ATEGORIES OF AUTOMATION TESTING
There are excess of tools available in the market to test
GUI applications. These tools can be categorized in the
following ways viz.
A. Random Testing
Random testing is also called adhoc testing. In this
technique, the tester can give the out of coverage data and
check the stability of the application whether the system
performs smoothly or not. Random testing is black box
technique which requires no partitioning of the
component's input domain but simply requires test inputs
to be chosen from input domain at random. It is good for
finding when system crashes. No effort in generating test
cases. The updates of GUI are independent. It increases
confidence on the software when running several hours
without finding errors. It is” Easy” to implement. It is not
good in finding other kinds of errors. It is difficult to
reproduce the errors. The output is unpredictable for some
test cases.
B Unit Testing Frameworks
Unit test frameworks [18] are a key element of
popular development methodologies such as extreme
Programming (XP) and Agile Development. But unit testing
has moved far beyond extreme Programming; it is now
common in many different types o f Ap p l i c a t i o n
development. Unit tests help ensure low-level code
correctness, reduce software development cycle time,
improve developer productivity, and produce more robust
software. It is very flexible. It has an automatic test
execution. It supports TDD (Test Driven Development). The
GUI test libraries can be used for GUI testing (essentially
reduce GUI to API testing).The test cases are programmed
manually, but usually has more lines of testing code than
application code.
C. Model based Testing
A model is usually an abstract, partial presentation of the
system under test's desired behavior. The test cases derived
from this model are functional tests on the same level of
abstraction as the model. These test cases are collectively
known as the abstract test suite. The abstract test suite
cannot be directly executed against the system under test
because it is on the wrong level of abstraction. Therefore
an executable test suite must be derived from the abstract
test suite that can communicate with the system under test.
This is done by mapping the abstract test cases to concrete
test cases that are suitable for execution. In the case of
online testing the abstract test suite exists only as a
concept but not as an explicit artifact. Model testing can
be easily adapted to changes. It requires a formal
specification/model. Small changes to the model can
result in a totally different test suite.
D. Capture and Replay (and script)
Most common testing tools use record playback /
Capture replay techniques [4]. A test designer interacts
with the GUI, generating mouse and keyboard events. The
tool records the user events, captures the GUI session
screens, and then stores the session usually as a script. The
tester later plays back the recorded sessions to recreate the
events with different inputs. An automatic test case
generator can provide a higher level of support, but a
programmer must code it for all possible decision points
in the GUI. Automated or not, the time consuming record
playback approach easily misses important GUI
decisions.GUI testing requires a clear methodology that
employs tools and techniques integrated to use a
standardized GUI representation. This integration ensures
that all test results are compatible with each other.
Programmatic technique can efficiently reduce the
memory occupied. The drawback of this technique is that
the tester needs to develop his own code for testing which
is a time consuming process. The Testing would be more
efficient if it combines the features of capture/replay and
programmatic techniques. This combination of tool is the
best choice for all GUI applications.
IV. S URV E Y OF VA R I OU S GUI T E S T I N G T O O L S
We have collected nearly ten different GUI tools[13]
that mainly come under the category of the Capture-
Replay technique and freely downloadable. The details of
the tools are listed below:
Abbot [5]: Abbot is a framework of Java library that
provides methods to reproduce user actions and examine
the state of GUI components. It is an improvement upon
the very rudimentary functions provided by the
java.awt.Robot class (A Better 'Bot'). The framework can
be invoked directly from Java code or accessed without
programming through the use of scripts. It is suitable for
use both by developers for unit tests and Quality
Assurance for functional testing.
Jacareto [9]: Jacareto is a capture and replay framework
for Java applications and applets. It is very flexible. There
are two front ends (Clever PHL is a GUI front end,
Picorder is a command line front end) included in it.
Jemmy [2]: Jemmy [2] is a Java TM library that is used to
create automated tests for Java GUI applications. It
contains methods to reproduce all user actions which can
perform on swing and AWT components (i.e. button
pushing, text typing, and tree node expanding).Jemmy
Test is a program written in Java which uses the Jemmy
API to test GUI applications.
JfcUnit [7]: JfcUnit enables developers to write test cases for
java Swing based applications. JfcUnit provides XML
recording and playback. This allows users to quickly and
automatically generate/edit scripts to drive the testing. The
XML API's are open and allow developers to define their own
XML tag handlers.
Marathon [10]: Marathon is a testing framework for GUI
applications developed using Java/Swing. Marathon composes
of recorder, runner and editor. The test scripts are composed
of python code. Marathon focuses on end-user testing. One
need not know Java or Swing to record test scripts using
Marathon. It is kept intentionally simple, flexible and provides
facilities to extend the functionality available through python
scripts or through writing ones own component revolvers.
QF-Test [14]: QF-Test is a highly flexible professional cross-
platform tool for Java and Web GUI test automation (Swing,
Eclipse / SWT+RCP, Web start, Applets, ULC).Supports
regression and load testing tolerant to GUI changes and can
handle even complex custom GUI objects. Modularization
allows creating reusable, robust tests that really pay off.
Q1 [17]:Q1 is a tool for developing GUI and HTML
automated tests. It was designed to provide great control
while maintaining low cost to make it affordable to small
companies and individual developers.
Test Agent [20]: Test Agent focuses on making User
Acceptance Testing more efficient and it enables
developers and QA to meet deadlines without
compromising quality. Some of the key features include
capturing and recording all data on the applications'
pages, automatically detecting and capturing standard and
custom content errors.
GUI dancer[19]: GUI dancer uses a sophisticated, yet
easy-to-use specification method which eliminates all
programming from test creation. A variety of powerful
features allow the extensive reuse of editable test
elements, resulting in tests which are quick to create,
highly adaptable, and which require only minimal
maintenance.
Squish [16]: Squish is the cross-platform, automated GUI
test and web testing tool. It supports testing applications
based on the following GUI technologies: Trolltech Qt
and Qtopia, Web/HTML/Ajax in IE, Firefox/Mozilla,
Safari and Konqueror, Java Swing/AWT, Java
SWT/Eclipse Rich Client Platform (RCP), XView.It
implements dedicated support for the mentioned toolkits
and technologies and its controls to allow creating robust
test scripts.
Thus there are many tools available in the market for
testing GUI applications with varied features.
V. T OO L S S E L E C T E D FOR A NA LY S I S OF GUI TESTING BASED
O N SP E C I F I C AT IO N S
For the purpose of analysis, we have chosen five
different tools which are freeware tools that can easily be
downloaded. The details of the tools along with its advantages
and disadvantages are given below:
A. Jemmy [2]
Jemmy is a tool that allows one to create automated
tests for Java GUI applications. Tests are written on Java,
using Jemmy as a regular Java library. No recording, no GUI,
no XML, - all the work being done in Jemmy is dedicated
to test stability, so it could be used for big, complicated
and "dynamic" Java GUI applications.
Advantages
• Has a Java library that is used to create User
Interface (UI) automated tests.
• Provides support covering all Swing
components.
• Have two modes of action reproducing (Robot
and event dispatching).
• Has a Pre/post verification method.
Disadvantages
It does not support System test and Acceptance test.
B. Jacareto [9]
Jacareto is a capture and replay tool for programs
written in Java. One can capture actions on applications
and replay them later on (like macros). Jacareto can be
used for many purposes such as Creation of animated
demonstrations, Creation of macros, Qualitative and
quantitative analysis of user behavior etc.
Advantages
• Uses virtual machine registration
• Batch file and XML script used to start Java
virtual machine, start the test tool, and load the
application’s classes. (Main method of application called
by test tool as required.)
• Jacareto has to discover all instances of
subclasses of Component (including Components for
Swing)
Disadvantages
• It has difficulty in launching various Java
applications that have dependencies on some external
libraries to handle the look-and-feel of the application.
• It did not seem compatible with it, as several
applications we selected for test case capture used it and
would fail to launch when we would attempt to perform a
capture in Jacareto.
C. JfcUnit [7]
JfcUnit is an extension to the popular testing
framework JUnit [12]. This document assumes that one is
familiar with the usage of JUnit. A JUnit extension that
supports writing GUI tests for Java Swing applications.
Advantages
A library that allows one to test swing applications. It
simulates button presses, ant test focus etc. A nice
solution to a hard problem.
Disadvantages
Tests require writing Java and XML code, which
adds complexity and consumes time. As a result,
applications specially designed to test Swing user
interfaces takes time.
D. Marathon [10]
Marathon is a general purpose tool for both running
and authoring acceptance tests geared at the Java
platform. Included with marathon is a rich suite of
components that helps to interact with the application at
the User Interface (UI) Level. To aid with the regression
testing of existing applications, Marathon comes bundled
with a recorder to capture events. These events are then
converted into a valid Marathon test which can subsequently
be played back.
Advantages
adds only a handful of well defined scripting elements to the
script runtime - either Jython or JRuby.
Disadvantages
It develops product that obviously do not mature. Web
testing is not available. Documentation and support is not at
sufficient level.
E. Abbot [5]
Graphical user interface (GUI) testing is a potentially
problematic area because constructing effective test cases is
more difficult than the corresponding application logic. The
roadblocks to effective functional GUI testing are:
• Traditional test coverage criteria like "80% coverage
of the lines of code" may not be sufficient to trap all the user
interaction scenarios.
• End users often use a different user task interaction
model than the one conceived by the development team.
The framework easily integrates with the JUnit test
harness and therefore, during application development, the
functional GUI tests can become a part of the test suite. All
these features of Abbot make it an effective framework for
rapidly creating a comprehensive test framework.
Advantages
• Abbot builds upon the java.awt.Robot class to
provide an automated event generation and
validation framework for Swing GUI components.
VI. A NA LY S I S OF T H E S E L E C T E D T O O L S B A S E D
C HA R A C T E R I S T I C S
For the analysis of the selected GUI tools, we have
considered tool features that are classified as,
A.General Features
The general features are categorized as
• Supported Platforms: Platform Independence for
application
• Supported OS:Independent for all operating system
• Testing principle: GUI component development as
well as composite GUI development
• Testing Structure: Efficient application of planning
by first creating a hierarchical model
• Robustness of tests: Improving the entire system's
safety and robustness
• Component recognition: Unique mechanism for
specifying and locating components
• Test Management: Easy integration with third party
solutions
• Reporting: Maintained the bug reports. Bug
Tracking and Reporting
• Extensibility / Scripting: The solutions to support
new custom GUI components
B. Usability Features
The usability features include
• User friendly: People will tend to buy systems that
are more user-friendly
Marathon is the ability to write test cases even before the
application under test is built. Marathon, itself,
• Accessibility: People should be able to access
GUI objects
• Installation: Installing the GUIs tool without
any interrupt
• Documentation: Extensive manual and tutorial,
online help
• Programming Knowledge Required: Requires
any previous knowledge about tool
• Rapid script: It will be available for only with
Global GUI map file concept.
• User Actions: User interact with GUIs
components
• Component finding Technique: Each
component is centralized and can be easily
updated
C. Test Execution Features
The Execution features includes
• Test Scripts: It can make test suites easier
• Test Case Preparation: Test case must be
complex and follow a path through the GUI
where the operations are performed in a specific
order
• Test Modification: Reused for many related
tests.
ON S P E C IF I C
• Object mapping: logical names and physical
properties of GUI mapping tool
• Recording: Store the user actions
• Play back: Play the stored user actions
• Event Mechanism: To implement the
synchronization of user actions
• GUI Map Editor: To save unsaved entries in a
local buffer by the test engineer
• Error Handling: Maintain all user actions and
reuse for future
• Maturity: It is a key variable in deciding what
type of automation is considered
• Complexity: Software needs to be tested
because untested code is a potential source of
bugs
• Context Sensitive Recording: Records the
operations are perform on specific application by
identifying specific GUI objects work
• CSS script: It is a simple mechanism for adding
style
D.Kinds of Testing Performed
This feature includes
• Unit Test: It looks similar to the JUnit
GUI.Building unit tests and functional tests
(AWT Unit/ Swing Unit)
• System Test: client/server and web based
systems deliver system functionality using a
GUI.
 • Flexible Testing: It's enough for general desktop • Running Modes: It is provide Robot class most
GUI automated testing. basic programmatic support, no support to find
• Acceptance testing: It should be done at the widgets
business logic level • Component Lookup: It should best way to
• Image testing/comparison: It can find image guarantee reliable
sample and then focus mouse on it
TABLE 1 ANALYSIS OF GUI TOOLS BASED ON SPECIFIC FEATURES
REQUIREMENTS ABBOT MARATHON JEMMY JFC UNIT JACARETO
General Features
Supported Java, Java swing, Java,Xml Awt /swing
Java, java swing Java, applets
Platforms Xml Java Swing components
Record/ Play Record/Play
Dynamically
back, back Programmatic
Testing principle Capture events Capture Replay
Programmatic Programmatic GUI
play back later
GUI GUI
Has well-defined
Has a scripted Is a Net Beans An extension to Groups record
Testing scripting
Java GUI testing independent the JUnit elements - build high-
Structure elements to the
framework. module framework level structures
script at runtime

Testing Robust    
CSV or Xml or
Reporting Html or Text Html or Xml Xml CSV or Xml
Text
has Unique
Rare(new
Extensibility / Component Jython/Python
operator) / high New events from
Scripting (ID) Scripts - Libraries/Call Special event types
level xml tool basic events
Call Directly java Java Code
(jelly)
Code
Usability Features
Depends on
User friendly Yes Yes Complex Yes
Design
Very fast - 17 Very fast - 16 Install Jar file Install Jar file
Installation/ Size 8 MB
MB MB -1MB only
Documentation Available Available Available Available Available
Programming
Java Swing and Java Swing and
Knowledge Java Java Swing, Xml Java
Xml Xml
Required
Test Execution Features
Xml and Java
Test Scripts Python/Jython Xml (Jelly) Xml Xml
Swing
Usability Net Usability Net Not allowed. Usability Net
Test Abbot Molecular
Performance Performance Increases the test Performance testing
Modification Testing
testing methods testing methods complexity methods
Record/ Play
Use Abbot Robot Jython No No Regenerate Events
back
Event Robot (Default)/ AWT Control/ Awt (Default)/ Awt (Default)
AWT/Robot
Mechanism AWT Program control Robot /Robot
Script Editor Marathon Map
GUI Map Editor - JFC Editor Jacareto Editor
(Costello) Editor
Error handling
Yes Yes Yes Yes Yes
(Automated)
CSS script Available Available Available Available Available
Context Sensitive Script Enviro
Using JUnit Using JUnit Using JUnit Using JUnit
Recording -nment and JUnit
High Degree Higher Higher
Maturity Low Level High level
Granularity Granularity
Types of Testing Performed Features
UnitTest Yes No Yes Yes No
System test Default Yes No No Yes
Flexible Testing Yes Yes Yes No Yes
Acceptance test Available Available Not Available Not Available Available
 Image testing/
Supported Supported Supported Supported Supported
comparison
Running Modes 1 2 1 1 2
Component Component
Fuzzy Expects Classes -
Lookup Specific
GUI Object Custom Event
API Design Jython Operator objects -
Script Classes
 A detailed comparison of all these features for the five basic operations such as Add, Sub, Mul and Div and also
selected GUI Testing tools are as shown in Table 1. From the Scientific Operations such as Sin, Cos, Tan, Log, sqrt ,
table it is very clear that, Abbot, Jemmy, and JFCUnit provide etc. The program also contains Radio Buttons like
ways to unit-test GUI code. Abbot and Marathon provide Hexadecimal, Octal, Decimal and Binary. The calculator
higher-level scripting actions more suitable for, Acceptance takes inputs from both standard and scientific modes,
testing. JFCUnit has started work on some level of scripting where the scientific users may be categorized as technical
support. Abbot and of user actions on a system under test for users, students and business users. Fig .1 shows the GUI
further manipulation by hand or later playback. Abbot model application being implemented. All the operations
provides both robot (default) and AWT modes, and throws in are performed by click events .
a little Programmatic control while the other tools do not
support these features. Jemmy and JFCUnit use AWT mode
by default but provide a robot version if needed. Marathon
uses a mix of AWT control and Java’s Accessibility libraries
(programmatic control).
Abbot uses a dedicated class to represent GUI
components (since the components don’t necessarily exist
when you first need to reference them). It stores a range of
attributes to be used when looking for a matching component
in the hierarchy, and does a fuzzy lookup to avoid spurious
errors due to component movement or minor changes to the
GUI. The lookup mechanism is very general due to the fact it
is used by the scripting layer which has no apriori knowledge
of the GUI hierarchy or what needs to be looked up. The Figure1: The GUI Calculator Application
framework also provides a number of utilities to facilitate
inspecting the GUI hierarchy itself. Jemmy provides a few
different lookup classes that finds component based on A. Testing with Abbot
different criteria built into the class; the programmer needs to In the Calculator Application, test case generation
pick the right lookup class based on knowledge of the makes use of the core components like Component Test
hierarchy. JFCUnit provides component-specific lookup Fixture () and Component Tester () in Abbot and also
classes. These are used in conjunction with a test helper class writes the test cases using actionClick (), SelectMenuItem
to look up a desired component. Marathon mainly expects to (), actionDelay (),etc. By making use of these
find components by name. components, test cases are generated. The assert method
Abbot also provides for extending the basic tester classes compares the data values obtained from calculator
for new components, and has a well defined method for program with the data values obtained by the automatic
naming those new actions so that they are automatically execution of the test cases. The click event for
available at the scripting level. It also provides extensibility hexadecimal option button and the corresponding enabled
for converting strings from scripts into arbitrary classes, and buttons are shown in Fig 2.
introducing new individual steps into scripts.
Jemmy has rather a high hurdle for introducing a new
operator. Jemmy has a higher-level xml-based scripting tool
called Jelly that facilitates higher-level application
actions.JFCUnit requires to define new event types for
components derived from the basic ones. Abbot provides both
a script environment and a JUnit fixture, both of which handle
setup and teardown of the complete GUI environment. Scripts
also support launching a single application instance and
including other scripts.
Thus compared to the other four tools Abbot is
considered best when the applications are mainly Java swing
oriented applications. Using this tool, test cases could be Figure 2: Automatic Execute of User Input Sequence
executed , positive and negative test cases identified, coverage
of the test cases could be found, etc. , which cannot be done
1). Test Case Generation
by other GUI tools.
The next section provides the implementation of the Test case Generation and Execution is done
abbot tool through which the generated test cases were using Abbot and Junit Tool. Abbot is a framework for
executed and verified. driving java UI components programmatically. Costello is
a script editor and a launcher which accompanies Abbot.
VII. I MPLEMENTATION They are based on the same capture/ playback/script
principle that marathon uses, but Abbot can also be used
In this paper, the GUI model of Calculator programmatically. The framework can be invoked
Application, written using Java Swing is considered. The differently from Java code. It is suitable for use both by
scientific Calculator Application contains a collection of developers for unit tests and QA for functional testing.
Standard Buttons and Scientific Buttons. It is used to calculate The framework easily integrates with the JUnit test
the Arithmetic and Scientific data values. It performs several harness and therefore, during application development, the
functional GUI tests can become a part of the test suite. All
these features of Abbot make it an effective framework for
rapidly creating a comprehensive test framework. Collections
of test cases are generated by using Abbot and JUnit Tool. The
test cases contain the sequence of user Input Events.
Fig 3 Test Case Generation using Abbot Tool
This framework shown in Fig3 chooses a specific model of
the GUI application. This model satisfies the GUI application
under test which is also the input of the Test Case generation.
Collections of test cases are generated by using JAR (Abbot
and JUnit)..The source code of test case generation for the
calculator application is shown in Fig 4.
Figure 4: The Source code of Test Case Generation
2) Test Case Execution
Test cases from the repository are executed one by one
automatically using Abbot and JUnit Tools as shown in Fig 3.
Fig 3 Test Execution by JUnit Tool
The Test case contains the sequence of user Input events. The
test cases are run with test runner. A test designer interacts
with the GUI and generates mouse and keyboard events.
Depending upon the test cases, the events are executed
automatically. The source code of the test runner is given
below.
Public static void main (String [] args)
{ Number of Files, Number of Packages.
TestHelper.runTests (args, CalculatorTest.class);
}
3) Test case Verification:
The expected results of various test cases are manually
determined and stored in the GUI model of Testing Oracle.
The Testing Oracle contains the expected state of sequences
for each application. When the test cases start running, the
start timer is initialized. The events are generated
automatically and the actual results from the tool are verified
with the expected results. Pass/Failure testing report is
generated accordingly. Fig 6 shows the positive and
negative test case verification.
Figure 6: Test case Verification
B.Code Coverage with Clover
Clover [21] uses source code instrumentation,
because although it requires developers to perform an
instrumented build; source code instrumentation produces
the most accurate coverage measurement for the least
runtime performance overhead. As the code under test
executes, code coverage systems collect information about
which statements have been executed. This information is
then used as the basis of reports. In addition to these basic
mechanisms, coverage approaches vary on what forms of
coverage information they collect. There are many forms
of coverage beyond basic statement coverage including
conditional coverage, method entry and path coverage.
1) The Clover Coverage Measurement
Clover uses these measurements to produce a Total
Coverage Percentage for each class, file, and package and
for the project as a whole. The Total Coverage Percentage
allows entities to be ranked in reports. The Total Coverage
Percentage (TPC) is calculated as follows: TPC = (BT +
BF + SC +MC)/(2*B + S + M) where BT - branches that
evaluated to "true" at least once BF - branches that
evaluated to "false" at least once SC – statements covered
MC - methods entered B - total number of branches S -
total number of statements M - total number of methods.
Table1 displays the report title and the time of the
coverage contained in the report. The header displays
metrics for the package files or project overview which is
currently selected. Depending on the current selection, the
metrics include all or a subset of: Number of Lines of
Code (LOC),Number of Non-commented Lines of Code
(NCLOC),Number of Methods, Number of Classes,
C.Event Coverage
Event Coverage is determined by the total number of
click events generated by the user. For Eg table3 shows
that Single event is generated for Basic button, 11 events
are generated for calculating the area of circle (pi*r*r).
D. Code Coverage
Code coverage and event based performance
analysis are done in the module. Table 4 shows that, Code
coverage tool is used for statements, branches and loop.
Manual analysis is done for event interaction and events
 and the relevant data is collected. Coverage metrics are
collected for different units of application.

TABLE:2 VIEW OF ALL TEST CASES VALUE USING CLOVER COVERAGE TOOL
Area
TestCase State Bran Metho Classes LOC NCLO Total Cmp Avg Stmt / Methods / Total
ment ches ds C Cmp Densit method methods classes Coverage
y Cmp (in %)
Circle 280 132 21 1 625 483 138 0.49 6.57 13.33 21 54.3
Rectangle 293 132 24 2 667 513 141 0.48 5.88 12.21 12 53.6
Parallelogram 291 132 24 2 666 511 141 0.48 5.88 12.12 12 53.6
Triangle 298 132 24 2 672 518 141 0.47 5.88 12.42 12 54.1
Trapezoid 297 132 24 2 672 517 141 0.47 5.88 12.38 12 54.1
Total 339 132 28 2 738 570 145 0.43 5.18 12.11 14 55
Surface
Rectangle 312 132 25 2 690 535 142 0.46 5.68 12.48 12.5 41.6
Prims 304 132 25 2 681 527 142 0.47 5.68 12.16 12.5 42.7
Cylinder 303 132 25 2 676 525 142 0.47 5.68 12.12 12.5 53.3
Cone 300 132 25 2 671 522 142 0.47 5.68 12 12.5 53.3
Sphere 294 132 24 2 668 513 141 0.48 5.88 12.25 12 51.7
Total 373 132 28 2 775 602 145 0.39 5.18 13.32 14 53.6
Volume
Rectangle 291 132 24 2 666 511 141 0.48 5.88 12.12 12 55
Prims 293 132 24 2 669 513 141 0.48 5.88 12.21 12 55
Cylinder 296 132 24 2 671 516 141 0.48 5.88 12.33 12 55
Cone 297 132 24 2 672 517 141 0.47 5.88 12.38 12 55
Sphere 299 132 24 2 677 519 141 0.47 5.88 12.46 12 55.5
Pyramid 295 132 24 2 671 515 141 0.48 5.88 12.29 12 55
Total 351 132 29 2 762 586 146 0.42 5.03 12.1 14.5 55
 TABLE: 4 VIEW OF ALL TEST CASES CODE COVERAGE TOOL
Test Case Stmt Branch Loop Strict
Condition
Basic 59.4 2.3 10.1 6.0
Minus 63.9 10.5 8.7 7.7
Add 63.9 9.3 8.7 7.7
Mul 63.9 11.6 8.7 7.7
Div 63.9 12.8 8.7 7.7
Mod 67.1 15.1 10.1 9.4
Hex 61.6 11.6 14.5 19.7
Dec 60.7 9.3 13.0 7.7
Oct 61.6 11.6 14.5 15.4
Bin 61.6 14 13 9.4
Area
Circle 68.9 36.0 10.1 14.5
Rectangle 63.9 12.8 8.7 11.1
Parallelogram 63.9 12.8 8.7 10.3
Triangle 65.3 17.4 8.7 13.7
Trapezoid 65.8 18.6 8.7 15.4
Surface
Rectangle 64.4 14.0 8.7 13.7
Prims 65.3 16.3 8.7 13.7
Cylinder 68.9 34.9 10.1 16.2
Cone 68.9 34.9 10.1 15.4
Sphere 67.6 30.2 10.1 9.4
Volume
Rectangle 63.9 11.6 8.7 8.5
Prims 64.8 15.1 8.7 10.3
Cylinder 68.5 34.9 10.1 14.5
Cone 68.5 34.9 10.1 12.8
Sphere 68.5 33.7 10.1 12.8
Pyramid 64.4 15.1 8.7 11.1
Total 97.7 93 34.8 94.9
TABLE: 3 VIEW OF ALL TEST CASES EVENT COVERAGE

Test Plan Event Execution Execution

(in sec) Delay
(1000 sec)
View
Basic 1 1.684 2.699
Scientific 1 1.763 2.714
Hex 1 1.342 2.371
Dec 1 1.295 2.309
Octal 1 2.262 2.356
Binary 1 1.342 2.324
Area
Circle 11 3.588 4.555
Rectangle 7 2.434 3.463
Parallelogram 6 2.262 3.26
Triangle 12 3.447 4.446
Trapezoid 12 3.401 4.415
Volume
Rectangle 6 2.278 3.291
Prism 8 2.036 3.65
Cylinder 11 3.525 4.633
Pyramid 10 2.995 4.009
Cones 11 3.588 4.556
Sphere 14 4.119 5.085
Surface
Rectangle 26 6.006 6.989
 Prims 18 4.524 5.506
Cylinder 17 4.68 5.647
Cones 13 3.916 4.898
Sphere 9 3.183 4.165
E.Coverage Report tools, though they are completely automated, are not
Coverage XML Report generalized for all kinds of GUI applications. In future,
The clover xml report task generates a full HTML report a new tool that satisfies most of the mentioned features,
with sensible default settings. It is also generated prior to tests a variety of GUI applications and improves the
generation of the full report. coverage for user interactions can be developed.
Coverage PDF Report
The clover pdf report task generates a PDF report with REFERENCES
sensible default settings. It is also generated prior to
generation of the full report.
[1] A. M. Memon. Gui testing: pitfalls and process. IEEE Computer,
35(8):87–88, 2002.
The different explorer view of coverage report for all
completed test cases is shown in fig 8, 9.
[2] http://jemmy.netbeans.org
[3] Brad A. Myers. User interface software tools. ACM Transactions
on Computer-Human Inter-action, 2(1):64{103, 1995.}
[4] Capture-Replay Tool, http://soft.com
[5] T. Wall. Abbot java gui test framework,
2004.http://abbot.sourceforge.net/
[6] Tamas Daboci,Istvan Kollar,Tamas Megyeri,”How to Graphical
User Interfaces” IEEE Instrumentation & Measurement
Magazine,Sep 2003
[7] The JFCUnit project’s homepage http://jfcunit.sourceforge.net/-
[8] A. M. Memon, M. E. Pollack, and M. L. Soffa. Using a goal-
driven approach to generate test cases for guis. In 10 Proceedings
of the 21st international conference on Software engineering, pages
257–266. IEEE Computer Society Press, 1999
[9] C.Spannagel.Jacareto,2003. http://sourceforge.net/ projects/
jacareto
[10] A Thoughtworks production. probably the most recent
project.http://marathonman.sf.net.
Fig 8: Project Coverage Report in XML Format
[11] A. M. Memon, M. E. Pollack, and M. L. Soffa. Hierarchical
gui test case generation using automated planning. IEEE Trans.
Softw. Eng.,
[12] E. Gamma and K. Beck. Gui tools for
junit,2004.http://www.junit.org/news/ extension/gui/index.htm
[13] Atif M Memon. Advances in GUI testing. In Advances in
Computers, Ed. by Marvin V.Zelkowitz, volume 57. Academic
Press, 2003.
[14] QF-Test - The Tool for Test Automation of Java/Swing and
Eclipse/SWT/RCP applications. http://www.qfs.de/ en/qftest/
download.html.
[15] A. Memon, A. Nagarajan, and Q. Xie, “Automating
Regression Testing for Evolving GUI Software,” J. Software
Maintenance and Evolution: Research and Practice, vol. 17,
[16] http://www.froglogic.com/squish/
[17] Q1 is a tool for developing GUI and HTML. http:// www.
3d2f.com/programs/11-164-q1-download.shtml
[18] P.Hamill, Unit Test Frameworks, O’Reilly, 2004
[19] http://www.bredex.de/en/guidancer/first.html
Fig 9: Coverage Report in PDF Format [20] Clover coverage tool www.atlassian.com/software/

VIII. CONCLUSION AND FUTURE WORK

In this paper we have analysed the functionalities of

five different automated GUI tools based on some specific
characteristics. This analysis will be of great use to the tester
in order to choose an appropriate tool based on his
requirements. In this paper we have used Abbot tool. This is
easy to learn and use, and provides some unique features that
can make GUI development more productive. We have
developed the method for automatic test case generation,
execution, verification and calculating the coverage based on
statement, method and branch for GUI applications. These