[Project Name]

Usability Test Plan

[Version]
[Creator Name]
[Delivery Date]
Table of Contents
Table of Contents ................................................................................................................. 2
Document Overview ............................................................................................................ 3
Executive Summary ............................................................................................................. 3
Methodoloy ........................................................................................................................ 3
!artici"ants ....................................................................................................................... #
Trainin ............................................................................................................................ #
!rocedure ......................................................................................................................... #
$oles .................................................................................................................................... %
Ethics ................................................................................................................................ &
'sability Tas(s ..................................................................................................................... &
'sability Metrics .................................................................................................................. &
Scenario Com"letion ........................................................................................................ )
Critical Errors ................................................................................................................... )
*on+critical Errors ........................................................................................................... )
Sub,ective Evaluations ..................................................................................................... )
Scenario Com"letion Time -time on tas(. ....................................................................... )
'sability /oals ..................................................................................................................... 0
Com"letion $ate .............................................................................................................. 0
Error+free rate ................................................................................................................... 0
Time on Tas( -TOT. ........................................................................................................ 0
Sub,ective Measures ........................................................................................................ 0
!roblem Severity ................................................................................................................. 0
1m"act .............................................................................................................................. 0
2re3uency ......................................................................................................................... 4
!roblem Severity Classification ....................................................................................... 4
$e"ortin $esults ................................................................................................................. 4
2
Document Overview
This document describes a test plan for conducting a usability test during the
development of [web site name or application name]. The goals of usability
testing include establishing a baseline of user performance, establishing and
validating user performance measures, and identifying potential design
concerns to be addressed in order to improve the efficiency, productivity, and
end-user satisfaction [add or delete goals].
The usability test objectives are:
To determine design inconsistencies and usability problem areas
within the user interface and content areas. otential sources of
error may include:
o !avigation errors " failure to locate functions, e#cessive
$eystro$es to complete a function, failure to follow
recommended screen flow.
o resentation errors " failure to locate and properly act upon
desired information in screens, selection errors due to
labeling ambiguities.
o %ontrol usage problems " improper toolbar or entry field
usage.
&#ercise the application or web site under controlled test conditions
with representative users. 'ata will be used to access whether
usability goals regarding an effective, efficient, and well-received
user interface have been achieved.
&stablish baseline user performance and user-satisfaction levels of
the user interface for future usability evaluations.
[(dd a paragraph that summari)es the user groups that the application or
*ebsite will be deployed+launched to, the user groups that will participate in
the usability test and the number of participants from each user group that are
e#pected to participate. ,ndicate whether the testing will occur in a usability lab
or remotely and the e#pected date range for testing.]
Executive ummary
5-ummari)e specific details of the usability test for the given application or *eb
site. describe specific functions to be evaluated. -ummari)e the usability
goals.]
/pon review of this usability test plan, including the draft tas$ scenarios and
usability goals for [web site name or application name], documented
acceptance of the plan is e#pected.
!et"o#olo$y
['escribe briefly the number of participants, the setting of the usability test
sessions, the tools used to facilitate the participant0s interaction with the
application 1e#., browser2, and the measures to be collected, such as
demographic information, satisfaction assessment, and suggestions for
improvement.]
3
Partici%ants
[Thoroughly describe the number of participants e#pected, how they will be
recruited, characteristics of their eligibility, and e#pected s$ills+$nowledge.]
The participants0 responsibilities will be to attempt to complete a set of
representative tas$ scenarios presented to them in as efficient and timely a
manner as possible, and to provide feedbac$ regarding the usability and
acceptability of the user interface. The participants will be directed to provide
honest opinions regarding the usability of the application, and to participate in
post-session subjective 4uestionnaires and debriefing.
['escribe how the team will select test participants to meet stated
re4uirements. &#plain if participants will have certain s$ills and+or bac$ground
re4uirements, if they will be familiar with the evaluation tas$s, or have
e#perience with performing certain tas$s.]
Trainin$
['escribe any training provided as an overview of the *eb application or *eb
site.] The participants will receive and overview of the usability test procedure,
e4uipment and software. ['escribe any parts of the test environment or testing
situation that may be nonfunctional.]
Proce#ure
[/sability 5ab Testing]
articipants will ta$e part in the usability test at [put the name of the testing
lab here] in [location here]. ( [type of computer] with the *eb site+*eb
application and supporting software will be used in a typical office environment.
The participant6s interaction with the *eb site+*eb application will be
monitored by the facilitator seated in the same office. !ote ta$ers and data
logger1s2 will monitor the sessions in observation room, connected by video
camera feed [describe if lab has one-way mirror or video feed]. The test
sessions will be videotaped.
[,f the facilitator is seated in a control " describe the environment and the
e4uipment and how communication is supported.]
The facilitator will brief the participants on the *eb site+*eb application and
instruct the participant that they are evaluating the application, rather than the
facilitator evaluating the participant. articipants will sign an informed consent
that ac$nowledges: the participation is voluntary, that participation can cease
at any time, and that the session will be videotaped but their privacy of
identification will be safeguarded. The facilitator will as$ the participant if they
have any 4uestions.
articipants will complete a pretest demographic and bac$ground information
4uestionnaire. The facilitator will e#plain that the amount of time ta$en to
complete the test tas$ will be measured and that e#ploratory behavior outside
the tas$ flow should not occur until after tas$ completion. (t the start of each
tas$, the participant will read aloud the tas$ description from the printed copy
and begin the tas$. Time-on-tas$ measurement begins when the participant
starts the tas$.
7
The facilitator will instruct the participant to 8thin$ aloud6 so that a verbal record
e#ists of their interaction with the *eb site+*eb application. The facilitator will
observe and enter user behavior, user comments, and system actions in the
data logging application [describe how these metrics will be recorded if a data
logging application is not used.]
(fter each tas$, the participant will complete the post-tas$ 4uestionnaire and
elaborate on the tas$ session with the facilitator. (fter all tas$ scenarios are
attempted, the participant will complete the post-test satisfaction
4uestionnaire.
[9or :emote Testing]
articipants will ta$e part in the usability test via remote screen-sharing
technology. The participant will be seated at their wor$station in their wor$
environment. ;erbal communication will be supported via telephone.
The facilitator will brief the participant and instruct that he or she is evaluating
the *eb site+*eb application, rather than the facilitator evaluating the
participant. articipants will complete a pretest demographic and bac$ground
information 4uestionnaire. -essions will begin when all participant 4uestions are
answered by the facilitator. The facilitator will inform the participant that time-
on-tas$ will be measured and that e#ploratory behavior outside the tas$ flow
should not occur until after tas$ completion.
The facilitator will instruct the participant to read aloud the tas$ description
from the printed copy and begin the tas$. Time-on-tas$ measure will begin. The
facilitator will encourage the participants to 8thin$ aloud6 and that a verbal
record will e#ist of the tas$-system interaction. The facilitator will observe and
enter user behavior and comments, and system interaction in a data logging
application.
(fter each tas$, the participant will complete the post-tas$ 4uestionnaire and
elaborate on the tas$ session. (fter all tas$s have been attempted, the
participant will complete a post-test satisfaction 4uestionnaire.
&oles
The roles involved in a usability test are as follows. (n individual may play
multiple roles and tests may not re4uire all roles.
Trainer
rovide training overview prior to usability testing
'acilitator
rovides overview of study to participants
'efines usability and purpose of usability testing to participants
(ssists in conduct of participant and observer debriefing sessions
:esponds to participant0s re4uests for assistance
Data (o$$er
:ecords participant6s actions and comments
<
Test Observers
-ilent observer
(ssists the data logger in identifying problems, concerns,
coding bugs, and procedural errors
-erve as note ta$ers.
Test Partici%ants
rovides overview of study to participants
'efines usability and purpose of usability testing to participants
(ssists in conduct of participant and observer debriefing sessions
:esponds to participant0s re4uests for assistance
Et"ics
(ll persons involved with the usability test are re4uired to adhere to the
following ethical guidelines:
The performance of any test participant must not be individually
attributable. ,ndividual participant0s name should not be used in
reference outside the testing session.
( description of the participant0s performance should not be reported to
his or her manager.
Usability Tas)s
[The usability tas$s were derived from test scenarios developed from use cases
and+or with the assistance of a subject-matter e#pert. 'ue to the range and
e#tent of functionality provided in the application or *eb site, and the short
time for which each participant will be available, the tas$s are the most
common and relatively comple# of available functions. The tas$s are identical
for all participants of a given user role in the study.]
['escribe the application0s test setup up such as special development
environments or test databases. concurrent development activities that may
impact the test application0s availability or performance. and impact to real
data or wor$flows outside the testing situation.]
The tas$ descriptions below are re4uired to be reviewed by the application
owner, business-process owner, development owner, and+or deployment
manager to ensure that the content, format, and presentation are
representative of real use and substantially evaluate the total application. Their
acce%tance is to be #ocumente# prior to usability test.
['escribe the scenarios and groups of participants whom will attempt to
complete tas$s and documented in sufficient detail to warrant customer sign-
off. 'escribe how typical and encompassing these scenarios are in the overall
scope of tas$s that the application or *eb site will support.]
Usability !etrics
/sability metrics refers to user performance measured against specific
performance goals necessary to satisfy usability re4uirements. -cenario
completion success rates, adherence to dialog scripts, error rates, and
=
subjective evaluations will be used. Time-to-completion of scenarios will also
be collected. [include or delete any metrics not used in the planned test]
cenario Com%letion
&ach scenario will re4uire, or re4uest, that the participant obtains or inputs
specific data that would be used in course of a typical tas$. The scenario is
completed when the participant indicates the scenario0s goal has been obtained
1whether successfully or unsuccessfully2 or the participant re4uests and
receives sufficient guidance as to warrant scoring the scenario as a critical
error.
Critical Errors
%ritical errors are deviations at completion from the targets of the scenario.
>btaining or otherwise reporting of the wrong data value due to participant
wor$flow is a critical error. articipants may or may not be aware that the tas$
goal is incorrect or incomplete.
,ndependent completion of the scenario is a universal goal. help obtained from
the other usability test roles is cause to score the scenario a critical error.
%ritical errors can also be assigned when the participant initiates 1or attempts
to initiate2 and action that will result in the goal state becoming unobtainable.
,n general, critical errors are unresolved errors during the process of
completing the tas$ or errors that produce an incorrect outcome.
Non*critical Errors
!on-critical errors are errors that are recovered from by the participant or, if
not detected, do not result in processing problems or une#pected results.
(lthough non-critical errors can be undetected by the participant, when they
are detected they are generally frustrating to the participant.
These errors may be procedural, in which the participant does not complete a
scenario in the most optimal means 1e.g., e#cessive steps and $eystro$es2.
These errors may also be errors of confusion 1e#., initially selecting the wrong
function, using a user-interface control incorrectly such as attempting to edit an
un-editable field2.
!oncritical errors can always be recovered from during the process of
completing the scenario. &#ploratory behavior, such as opening the wrong
menu while searching for a function, [will, will not 1edit rocedure2] be coded
as a non-critical error.
ubjective Evaluations
-ubjective evaluations regarding ease of use and satisfaction will be collected
via 4uestionnaires, and during debriefing at the conclusion of the session. The
4uestionnaires will utili)e free-form responses and rating scales.
cenario Com%letion Time +time on tas),
The time to complete each scenario, not including subjective evaluation
durations, will be recorded.
?
Usability -oals
The next section describes the usability oals for [web site name or application
name].
Com%letion &ate
%ompletion rate is the percentage of test participants who successfully
complete the tas$ without critical errors. ( critical error is defined as an error
that results in an incorrect or incomplete outcome. ,n other words, the
completion rate represents the percentage of participants who, when they are
finished with the specified tas$, have an @output@ that is correct. !ote: ,f a
participant re4uires assistance in order to achieve a correct output then the
tas$ will be scored as a critical error and the overall completion rate for the tas$
will be affected.
. com%letion rate of [/0012enter com%letion rate] is t"e $oal for eac"
tas) in t"is usability test3
Error*free rate
&rror-free rate is the percentage of test participants who complete the tas$
without any errors 1critical or non-critical errors2. ( non-critical error is an
error that would not have an impact on the final output of the tas$ but would
result in the tas$ being completed less efficiently.
.n error*free rate of [4012enter error*free rate] is t"e $oal for eac"
tas) in t"is usability test3
Time on Tas) +TOT,
The time to complete a scenario is referred to as @time on tas$@. ,t is measured
from the time the person begins the scenario to the time he+she signals
completion.
ubjective !easures
-ubjective opinions about specific tas$s, time to perform each tas$, features,
and functionality will be surveyed. (t the end of the test, participants will rate
their satisfaction with the overall system. %ombined with the
interview+debriefing session, these data are used to assess attitudes of the
participants.
Problem everity
To prioriti)e recommendations, a method of problem severity classification will
be used in the analysis of the data collected during evaluation activities. The
approach treats problem severity as a combination of two factors - the impact
of the problem and the fre4uency of users e#periencing the problem during the
evaluation.
5m%act
,mpact is the ran$ing of the conse4uences of the problem by defining the level
of impact that the problem has on successful tas$ completion. There are three
levels of impact:
Aigh - prevents the user from completing the tas$ 1critical error2
B
Coderate - causes user difficulty but the tas$ can be completed
1non-critical error2
6ow + minor "roblems that do not sinificantly affect the tas(
com"letion -non+critical error.
're6uency
9re4uency is the percentage of participants who e#perience the problem when
wor$ing on a tas$.
Aigh: 3DE or more of the participants e#perience the problem
Coderate: FFE - 2GE of participants e#perience the problem
5ow: FDE or fewer of the participants e#perience the problem
[9or studies with less than ten participants in a group, the percentages may
to be adjusted. 9or e#ample, for a study with B participants the low fre4uency
should be F2.<E 1F+B H .F2<D]
Problem everity Classification
The identified severity for each problem implies a general reward for resolving
it, and a general ris$ for not addressing it, in the current release.
everity / - Aigh impact problems that often prevent a user from
correctly completing a tas$. They occur in varying fre4uency and are
characteristic of calls to the Aelp 'es$. :eward for resolution is
typically e#hibited in fewer Aelp 'es$ calls and reduced redevelopment
costs.
everity 7 - Coderate to high fre4uency problems with moderate to
low impact are typical of erroneous actions that the participant
recogni)es needs to be undone. :eward for resolution is typically
e#hibited in reduced time on tas$ and decreased training costs.
everity 8 - &ither moderate problems with low fre4uency or low
problems with moderate fre4uency. these are minor annoyance
problems faced by a number of participants. :eward for resolution is
typically e#hibited in reduced time on tas$ and increased data
integrity.
everity 9 - 5ow impact problems faced by few participants. there is
low ris$ to not resolving these problems. :eward for resolution is
typically e#hibited in increased user satisfaction.
&e%ortin$ &esults
The /sability Test :eport will be provided at the conclusion of the usability test.
,t will consist of a report and+or a presentation of the results. evaluate the
usability metrics against the pre-approved goals, subjective evaluations, and
specific usability problems and recommendations for resolution. The
recommendations will be categorically si)ed by development to aid in
implementation strategy. The report is anticipated to be delivered to the
roject /%' %ontact by [enter date].
G