Usability Test Report

The Catamaran

Jonathan Ouimet, Alex Lebedev, Greyson Palmieri, Nubia

Colussi, Felipe Dalmau
24 October 2016

Table Contents
Front Matter
Cover Page
Table of Contents
Introduction
Executive Summary

1
2
3
3
Body

Methodology
Results
Conclusions

4
6
7
Back Matter

Attachments
Test Reader Comments / Recommendations
Writers Guide for Testing Draft
Likert - Scale Test Questions
Observations Sheet for User Test
Follow Up Interview Questions for Readers

Introduction
The purpose of the instruction set is to be able to successfully and
efficiently put together a Lego catamaran. Visuals and written instructions
are used to guide the readers through each of the steps to build the Lego
catamaran. The main elements that compose the catamaran (the hull, the
centerpiece, the sail, and the mast) are divided into separate parts. The final
part includes putting each element together to construct the catamaran.
For the user test Jonathan printed each part of the instruction manual.
There were five parts, each one for a different part of the catamaran. Before
Jonathan gave the instruction set to the participants, each participant was
provided and asked to fill out a Likert-Scale. After completion, the group was
handed the instruction manual and asked to construct the catamaran.
Jonathan took notes on the parts and steps that participants were having
trouble. After completion of the catamaran, the participants filled out the
remaining portion of the Likert-Scale. After completion of the Likert-Scale a
group interview was conducted. The questions and answers were recorded.
The participants also provided additional information on what they thought
could be improved. This data was noted on a different piece of paper which
was also found in the attachments.

Executive Summary
A usability test was conducted in Bachelor Hall Room 252 on
Wednesday, October 5th 2016. The test took approximately 12 minutes. The
purpose was to build a catamaran out of Legos using the set of instructions
that were provided.
The Redhawks, a group of five college students, were between the
ages of 19 to 21. It was determined that two of the five had an intermediate
understanding of legos and how they worked. The other three students
claimed that they had no prior experience using legos. It took approximately
15 minutes for the group to properly and correctly complete the task.

Overall they found the instructions easy to follow. All five participants
completed the test on this day. Of the five participants, three were female
and two were male. All of the participants ranged in age from 19-20 years of
age. Each participant was white/Caucasian. The Redhawks struggled on a
few parts of the instruction set and recommended the following:
Fix labels so that all of the pictures have labeled pieces
Phase two needs to align better with the pictures that were provided
The instructions that were given could have been presented clearer
The rest of the document includes the methods that were conducted
within the usability test that was given. This includes how the testing group
conducted the project and the results that were pulled from it. The
demographics of the participants are described, and the evaluation that took
place is listed as well. In the results section, the time it took the participants
to complete the task as well as the errors that occurred during the time the
usability test was taking place was recorded.

Methodology
The participants were recruited randomly on the Miami University
college campus.
A usability test which lasted approximately 15 minutes was given to
the group as a whole. The group was given the test and asked by an
administrator to start. The administrator told the participants that they
should attempt to finish building the Lego set by themselves, and only if they
were completely lost would they obtain outside assistance.
The group of participants got together in a controlled environment and
were handed a Lego set (the catamaran), and the corresponding instructions
manual and asked to build the Lego set. The participants sat down and an
administrator provided a Likert-Scale for all participants to complete. After
completion of the Likert Scale, the administrators allowed them to begin. The
administrator took notes while the participants were constructing the Lego
set. The participants finished Phase 1 of the Lego in three minutes. They had
no problems during the construction of this phase. It took participants four
minutes to finish Phase 2. Likewise, there were no major issues as the
participants constructed the Catamaran. They viewed several steps twice,
4

but had no major issues. Phase 3 took one minute to complete, obviously,
this was viewed as one of the simplest steps. Phase 4 also took
approximately one minute and participants had no problems completing this
step. They did not look nor hesitate to complete the phase. Phase 5 took two
minutes to finish. There was some confusion on the main image, as it was
found that it did not match perfectly with the image provided in phase 2.
After the test was completed the participants filled out the same Likert-Scale
to see if their view of the Lego set was different afterwards. The participants
then went into an interview phase where the entire group was asked their
thoughts on multiple aspects of the catamaran and the instruction set.
The group was individually asked to complete a Likert-Scale before and
after the completion of the Lego set. The Likert-Scales completed are
attached. The interview session questions are also attached with the answers
from the test group included.

Participants
There were a total of five participants. The participants were known as
the Redhawks and reviewed the Catamaran instruction set on October 5th.
All five participants completed the test on this day. Of the five participants,
three were female and two were male. All of the participants ranged in age
from 19-20 years of age. Each participant was white/Caucasian. Each
participant worked on at least one phase of the construction. The table below
displays exactly how many participants took part in the construction of each
phase.

# of
Participant
s
Completin
g This
Phase

Phase 1

Phase 2

Phase 3

Phase 4

Phase 5

Table 1. Provides the information of how the participants delegated their

catamaran construction. Some participants worked on more than one phase.

Evaluation Task
The task scenarios were constructed by the Catamarans. The scenarios
are listed below:
Task
Task
Task
Task

1:
2:
3:
4:

Complete the Likert-scale

Construct the Lego Catamaran with the use of Instructions
Recomplete the Likert-scale after constructing the catamaran
Answer the questions of the group interview

Results
Jonathan Ouimet recorded the participants to complete the tasks without
prompting.

Phases

Easy to
Somewhat Neutral
comprehen easy to
d
comprehen
d

Phase 1

Phase 2

Somewhat
difficult to
comprehen
d

Very
difficult to
comprehen
d

Phase 3

Phase 4

Phase 5

X
Table 2. Task completion success rate.

Time on task:
Phase 1:
Phase 2:
Phase 3:
Phase 4:
Phase 5:

3 minutes
4 minutes
1 minute
<1 minute
2 minutes

Errors:
Jonathon Ouimet recorded the errors made by the participants.
6

There was only one task where errors were made and it was during the
building of the Center Piece. The issue during this part had to do with the
inconsistency of pictures in our instruction set. The effect of this was mild
confusion on Part 2, Step 4 of the instructions and we simply removed the
unnecessary piece.
There were no other errors within the instruction set or during the
construction of the Catamaran. The Redhawks constructed Parts 1, 3, 4 and 5
without making any Critical Errors.

Overall Metrics
The test readers liked the images in every step. They said that being
able to see how the Legos should look after every step helped them with the
construction of the Lego catamaran.
What readers liked the least was that in the written instruction we
describe the pieces that are needed for each step with a numerical
identification. But for them to recognize the numerical identification, they
had to go to the beginning of each section, which had the appendix with the
image of every piece and their numerical designation. They found this to be
slightly inconvenient.
The test readers recommended that to solve the problem of having to
look at the index at the beginning of every section, to be able to recognize
what piece the written instruction mention. We should include an image of
the piece with it numerical identification, at the bottom of the step.

Conclusions
The participants found that the construction of the Catamaran went
rather well. They had no major issues. The inconsistencies in the pictures
were found and corrected. The final draft for the construction of the
catamaran is finished and provided along with this report.

Follow Up Group Interview Questions and Answers:

1.
a.
2.
a.
3.
a.
4.
a.
5.

Which phase was the easiest to follow?

The last phase (Assembly of the Catamaran)
Which Phase was the most difficult to follow?
Phase 2 (does not align with the picture from that phase)
Was it easier to follow the pictures or the written steps?
The pictures
Would you have labeled things differently?
No, effective labeling. Maybe numbers by each piece in all of the pictures
Was it beneficial to have the phases separated from each other the way that
they were?
a. Yes
6. Do you have any recommendations?
a. Include the numbers by each piece in all of the pictures instead of just the
first one

Writers Guide
Testing Drafts
Goals of Your Test
This test is to see how well proficient Lego users are able to build the
Catamaran. This test also sees how efficiently and quickly the catamaran can
be built given a set of easy to follow instructions.
The persuasive objective of this instruction set is to allow people to see
how easy it is build the catamaran. In doing so, we hope that more people
will buy the catamaran and find pleasure in constructing it for themselves.
There are no parts that we are insure how well they will work, but the
entire report will be tested in order to get a second round of opinions from a
neutral third party.

Test Readers
Our target readers are children between the ages of 5 and 15.
Although we cannot find readers of this age group, we are choosing readers
with little to no Lego experience. This will allow for a very similar, low level,
skill set, and accurate results. The only important difference between our
target readers and the readers who will actually be testing the draft is that
those who are testing the draft are about 10 years older than the average
age of our target reader.

Testing Your Drafts Usefulness

The test readers will be asked to construct the catamaran without any
outside assistance. The simulation will occur in a class that has been
designed to feel like a home, the area in which we expect most of our
targeted audience to actually build the Legos. We will be observing our test
readers by taking notes as they construct the catamaran. We will conduct a
survey after the construction of the catamaran that way we are able to know
what they were thinking during the construction of the Lego Boat. The
interview questions are drafted on a separate piece of paper. They focus on
which parts gave the test readers the most difficulty.

Testing Your Drafts Persuasiveness

10

We will ask the test readers how they felt throughout the construction
of the draft. We will avoid biasing their responses by providing open ended
questions with neutral words.

11