BUILD Usability Report

www.buildinboston.
org
Usability Report

Gabrielle LaFavre
December 2015
ENL_620: Writing Computer User Documents
Prepared for Professor Gulbrandsen
LaFavre Usability Report 1

Table of Contents

Executive Summary3
Introduction.6
Test Plan and Procedures...............................................................................................................7
Tasks.7
Findings...10
Recommendations..16
Questionnaire Analysis.17
Conclusion....18
Appendix A: Personas19
Appendix B: Heuristic Evaluation..20
Appendix C: Test Plan...45
Appendix D: Pre-Test Questionnaire..58
Appendix E: Post-Task Questionnaires..59
Appendix F: Post-Test Questionnaire.61
Appendix G: Moderator Script.63
Appendix H: Permission Forms65
Appendix I: Participant Questionnaires.68
Appendix J: Test Log89


Executive Summary

Purpose

This report contains findings from a usability study of BUILD Bostons website,
www.buildinboston.org. This study was conducted to assess the users experience interacting
with the site, and the research focuses on whether BUILD Bostons website is unique,
persuasive, and credible enough to convince visitors to become engaged with the organization
beyond a single website visit. This study evaluates whether BUILD gains supporter through
their online presence.

Process

This study followed three target users as they conducted four primary tasks and evaluated their
experience with the site through self-reported surveys, a series of questionnaires, and physical
and verbal reactions, as recorded by the test evaluator. Users were asked to:

1. State their initial reactions to the homepage, as well as how they think they would
proceed with the site.
2. Locate and evaluate company mission, history, and partnerships to assess BUILDs
credibility.
3. Find volunteer opportunities and reflect on whether provided information is adequate.
4. Uncover how to connect with the BUILD community, including current volunteers and
BUILD staff.

Additionally, all participants were asked to consider the following scenario as they proceeded
through the four tasks:

Imagine your work colleagues have started volunteering with a youth entrepreneurship
nonprofit called BUILD Boston. These colleagues, including your boss, have mentioned what a
wonderful experience it is for them. You decide to visit the organizations website to discover:
What BUILD stands for; What BUILD does; Who BUILD benefits; and How to get involved.

Findings

This usability test produced a range of valuable insight into how users engage with BUILD
Bostons website. The following includes a shortlist of major positives and problems that users
experienced.


Positives
Aesthetically pleasing, professional-looking visuals, including logo, photographs, and
video.
Adequate content to explain company mission, expressed through clear and concise
language.
Volunteer section organized opportunities usefully, based on skills and time
commitment, making it easy for busy users to digest the content.
Consistent, well-marked menu banner with clearly understood labels for each sub-
category.
Intuitive navigation and overall ease of use on site. Successful user interface.

Problems
MAJOR:
o Users found it difficult to understand what volunteers do, what qualifications are
required, and where volunteers actually lend their time.
o Link to National Site does not warn users that they are leaving the current
webpage, and users were not aware that BUILD was a national organization with
a Boston chapter.
o Users were wary to join mailing list online, and would prefer additional options
to connect with the organization beyond the mailing list.
MODERATE:
o Locating contact information was too challenging, and users were disappointed
that there were no staff bios or photos.
o Confusion over organizations purpose, in that users did not realize students
created businesses and sold products; users were unsure of what employee
positions, such as E1 Program Manager, meant.
o Users thought the site lacked visuals, and could benefit from more photos of
students and their products
MINOR:
o Users believed emphasis on social media feeds at the bottom of the homepage
was distracting, and reported that they wished for more information about the
company in this space instead.

Major Recommendations

1. Add more photos throughout the site, and update staff information page to include
short bios, email links, and employee photographs.
2. Consider streamlining the homepage by deleting one of the social media feeds, and
instead adding a short paragraph about the organization so that users do not have to
navigate to other pages to understand what BUILD does.
3. Bolster the Volunteer section to include a FAQ and an application form, just as the
National site provides.

4. Consider adding a discussion board so that site visitors may read testimony or ask
questions to staff and volunteers, as an alternative to the mailing list.
5. Generally, add more information about BUILD: on the employee page, before the
BUILD Boston Story: Cristhian video, and on the Student Business page.
6. Make the link between BUILD Boston and BUILD National explicit throughout the site,
and also provide feedback for visitors that click on the National link, so that they
understand they are leaving www.buildinboston.org
a. Additionally, consider making the link to the National site look more like a
hyperlink or button, rather than just another sub-section of the menu banner.

While the website remains easy to navigate and contains consistently labelled menu categories,
users had some difficulty understanding the content of the site. Applying a few updates listed
above will ensure that more site visitors continue to develop a relationship with BUILD Boston
beyond their visit to the online page. As with all usability testing, further studies will continue
to refine and improve new versions of www.buildinboston.org.


Introduction

This report contains the findings and recommendations from the usability study of BUILD
Bostons website. The report includes the following information:
Methodology to collect data
Test results, including analysis and recommendations
Recommendations for future testing
Appendices that include documents from the test, as well as data collected

Test Purpose

The purpose of this report is to reveal the results of a usability test of BUILD Bostons website.
The data collected evaluates the users experience interacting with www.buildinboston.org. The
data records how users completed four primary tasks, and iterates their self-reported
observations while interacting with the website.

This test focused on whether the website is unique, persuasive, and credible enough to
convince site visitors to become engaged with the organization beyond a simple website visit.
My research evaluated how BUILD gains supporters through their online presence; BUILD must
contend with many other nonprofit organizations in Boston. I sought to find whether their
website ultimately converts site visitors into BUILD supporterswhether that be financial
donors, mentors, student interns, or vocal advocates.

Test Objectives

The study assessed the user experience, and evaluated how target users undertake typical tasks
and familiarize themselves with the site. Through assessing four primary tasks, the test results
produced both qualitative and quantitate data. This study focused on content and credibility,
while also considering the effect of navigation and design. The test studied how users locate
company information, assess the organizations credibility, uncover volunteer opportunities,
and engage with the BUILD community.

Test Background

Before conducting a user test, I identified a major research question and created a persona,
which is a compilation of characteristics to form one imaginary user which represents a target
group of users. Creating a persona allowed me to conduct a Heuristic Evaluation, which means
that I was able to use the site as the persona and evaluate how the website conforms to a set of
guidelines, or heuristics, of usability.


Persona

After conducting surveys from target users, I created two personas based on qualities that
emerged from the research. I made use of the second persona, Eva Kim, to evaluate the
website through her perspective. Eva Kim is a 32-year-old financial professional, who hears
about BUILD through her company of employment and seeks to become involved in mentoring.
The persona of Eva Kim can be found in Appendix A.

Heuristics Report

Using a set of heuristics from Grice et al., I conducted an evaluation of www.buildinboston.org.
The results of this evaluation conveyed issues, ranging from minor to major, which necessitated
the call for a formal user test with three actual participants. The results of the Heuristics
Evaluation can be found in Appendix B.

Test Plan

After creating the persona and administering the Heuristic Evaluation, I designed a test plan to
outline methodology and to guide users through the testing process. The test plan contains pre-
test questionnaires, four primary tasks, post-task questionnaires, and a post-test questionnaire.
It is available in Appendix C. Based on the test plan, I recruited test participants who reflected
the professional characteristics of the persona Eva Kim.

Testing Process and Tasks

Following a successful pilot test, the formal user testing took place between November 20th and
November 29th, 2015. Three participants were asked to complete the following four tasks:

1. State their initial reactions of the homepage, as well as how they think they would
proceed with the site.
2. Locate and evaluate company mission, history, and partnerships to assess BUILDs
credibility.
3. Find volunteer opportunities and reflect on whether information provided is adequate.
4. Uncover how to connect with the BUILD community, including current volunteers and
BUILD staff.

While also considering the scenario:

What BUILD stands for; What BUILD does; Who BUILD benefits; and How to get involved.

Methodology

This section outlines the process and methods used to conduct the usability tests and
systematically collect data to analyze for the purpose of providing recommendations. The
following is a list of sub-sections within the Methodology Section:
Participant Recruitment
Test Limitations
Video Permission Forms
Moderators Script
Test Questionnaires

Participant Recruitment

I selected participants who represented the qualities of a BUILD Boston target user group:
financial professionals with an interest in volunteerism and/or philanthropy. To reflect the
characteristics of the persona Eva Kim, I sought business professionals who were interested in
nonprofits on a general level, and who were temporary or permanent residents of
Massachusetts. To identify candidates, I contacted friends, family, and academic contacts. The
following table outlines the participants who were involved with this study.

First Name Age Profession

Katie 23 Professional Writing M.A. Student/Professor

Rachel 25 Healthcare Management

Jennifer 33 Realtor
Table A: Participant Demographics

Test Limitations

As this was a smaller-scale user test with a tight deadline, I was unable to recruit any male
users. Additionally, I only tested the tasks on three users. This test could be administered again
with more participants in the future to provide more results. There were no technological
malfunctions during this test.


Video Permission Form

Before taking the test, each user signed a permission form allowing me to video and/or voice-
record their testing session. Each participant consented to this, and copies of the forms are
found in Appendix D.

Moderators Script

The moderator used a script to ensure continuity during each testing session. The script
followed the same sequence of tasks for each participant. The moderators script can be found
in Appendix E. Asking participants to use the site as they would normally, the moderator also
encouraged users to think out loud, and explain their thoughts and choices as they interacted
with the site. Appendix C contains the Test Plan with scenarios that users were asked to
complete, in table format.

Test Questionnaires

I administered three types of questionnaires: pre-test questionnaires, post-task questionnaires,
and post-test questionnaires.

The pre-test questionnaires gathered relevant background information about each
participants experience with, and opinion of, nonprofit websites overall. Post-task
questionnaires measured responses to each task, and allowed participants to put their
thoughts about various aspects of BUILD Bostons website into writing. Post-test
questionnaires allowed users to comment on their overall experience with BUILD Bostons
website, as well as any last thoughts they may have.


Findings

The Findings listed below include both positive elements of the website, as well as problems
ranging from minor to major. By measuring participants qualitative and quantitative reactions,
I was able to paint a detailed picture of www.buildinboston.org and its value to users. These
findings are organized into three categories: Aesthetics, Content, and Navigation. After
analyzing this data, I was able to deduce a number of recommendations for further
improvement.

Positive Elements

The following details positive reactions, as elicited by our users:

Aesthetically pleasing, professional-looking visuals, including logo, photographs, and
video.
Adequate content to explain company mission, expressed through clear and concise
language.
Volunteer section contained useful organization about opportunities based on skills and
time commitment, making it easy for busy users to digest the content.
Consistent, well-marked menu banner with clearly understood labels for each sub-
category.
Intuitive navigation and overall ease of use on site.

The following tables display the findings, along with user observations. These findings are
organized into three categories: Aesthetic findings, Content findings, and Navigation findings.


Positive Findings:
Aesthetics
Category Findings User Observations
Photos High quality; Visually appealing; Friendly looking students, good speed
Professional and photos on the homepage slideshow
The colors were well integrated
throughout. The website was modern
Website colors and logo Bright colors, looking, with short paragraphs that are
Eye-catching logo easy to read, and bullet points

The logo is warm and inviting
Videos Well made, professional video I thought it made the charity look really
good. Well done video

Content
Understanding of Good idea of what BUILD does I liked that they clarified this isnt a
organization and clarity based on websites content; vocational education program, because
of message Clarity of language and the organizations name is misleading
message
Volunteer information Users appreciated the I like that they have time commitments
breakdown of volunteer right under, so you know what you are
opportunities based on time getting into.
and skills

Navigation
Menu Clarity Banner menu clearly marked, The menu has clear tabs that lead you
well understood; Consistent to the information that you are looking
throughout the website for
Overall Navigation Intuitive navigation; overall clear website, overall quite easy to
ease of use navigate

navigation bars on the top of the page
are helpful


Problems

Using a three-part ranking system from minor to major, in which:
1. Minor ProblemAffects the user experience but is not a priority; should be fixed when
time permits.
2. Moderate ProblemHinders the users ability to navigate.
3. Major ProblemProhibits the user from accomplishing tasks.
I was able to gauge user obstacles while interacting with www.buildinboston.org. There were a
few areas which proved difficult for participants. I have outlined the problems in table format,
with screen shots to illustrate as necessary. The negative findings are organized into three
categories: Aesthetic problems, Content problems, and Navigation and User Feedback
problems. The findings conclude with recommendations to ameliorate these problems.
The following tables illustrate the problems and user observations, while also providing
relevant recommendations.

Negative Findings:
Aesthetics

(Severity) Findings User Observations Recommendations
Users noticed the
homepage slideshow I think they should Add more photos of students and volunteers
Lack of Visuals only scrolls between put a picture on this throughout the site; consider adding another
moderate two photos; page next to video on homepage
everyone. I like to put
Staff page does not a face to the names Update staff information page to include bios
have photos of and photos
employees
The bottom of the
homepage is I dont like seeing
dedicated to social social media right on
media; users thought the front page. I Add a brief paragraph about the organization
Social media displaying both would rather know and its needs prominently on the homepage,
links tweets and Facebook more about the and consider removing the tweet feed OR the
minor posts takes up too organization before I Facebook feed from the bottom
much space which see all the tweets. It is
could be devoted to a waste of space in my
BUILDs mission or an opinion
appeal to volunteers

Content

Are volunteers onsite Follow format of BUILDs national volunteer
teachers? Do you page, which is more explicit, and outlines
need to be trained? volunteer qualifications and responsibilities
As a volunteer, do you http://build.org/join-us/volunteer/
need to create lesson
plans? Also, add an application form and mentor FAQ
Users unsure of what link within Volunteer Section, like National
volunteers do and Do volunteers get does:
Volunteers what qualifications formal training? Is http://build.org/wp-
role they require, even there an orientation? content/uploads/2015/08/FY16-Mentor-
major after visiting the How is a volunteer FAQ.pdf
Volunteer section qualified for this
position? On the National site, there is a link to Carinas
email so that volunteers may contact her. This
This looks like you should also be available on Bostons site:
can only help if you
are an entrepreneur
or professional, so
maybe they should
put other
opportunities here
too so no one feels
excluded
The where is
unclearBoston, but Follow BUILD Nationals model of explaining
not specific as to each aspect of the BUILD program
which school sites (http://build.org/what-we-do/program-
details/)
Where to Users unclear where Is it at BUILDs office
volunteer? volunteers work with or do you meet at
moderate the students school? Probably
depends on students
schedule and what
they are comfortable
with

Do you meet at the
school, library?

It was challenging to
find contact
No links to staff information, not clear Link staff emails below each name and
emails; No bios or who current mentors position;
Locating photos of staff; and partners are
contact Contact Us page is Make office contact information (address,
information an email feedno phone number) more prominent for users
moderate direct way to get in it was difficult to find who prefer to call on the phone;
contact with current who the mentors are,
volunteers except what they think about Consider adding a discussion board area, or a
through calling the program, and similar way for users to engage with current
BUILDs office. especially difficult to volunteers and ask direct questions
find testimonials from
the students. In fact, I
couldnt find any
stories from students
I didnt understand Emphasize how BUILD is an actual business
that this is a student incubator, rather than a simulation, more
Understanding Users did not realize business site prominently on the website, perhaps on the
student students sold homepage
businesses products This is not at all what
moderate I thought it was going
to be. I thought it was Provide a short text introduction on the page,
going to be about before the BUILD Boston Story: Cristhian
building things video, so users understand what BUILD
provides to students before they watch the
video

What is an E1
Program Manager?
Understanding No information about No idea what that Add information about employee positions
employee employee positions means. If I was going underneath each employees name
positions to volunteer, I would
Minor want to know what
each person does

I was missing a break
down of their work,
not just what they do
but also how


Navigation and User Feedback
Im so confused
about what just
happened. Im really
confused about how
to get back to the
homepage and I
National Site Link probably wouldnt go
does not warn users back to the website at
they are leaving this point
BUILD Boston; lack of Add information about the national chapter
explanation about I ended up at the on BUILD Bostons homepage to situate and
Link to BUILD Boston as a national website and I alert users that BUILD is in fact a chapter of a
National Site chapter of a national didnt even know larger organization
major organization; users there was a national
unprepared and website. And then I Create a more prominent link to the national
confused by the couldnt get back to site, one which does not look like other links
understated link the Boston page on BUILD Bostons site. It will then be
user does not immediately evident that this national link will
recognize that this in the mission take users away from their current page
link will take you statement it doesnt
away from the site, say it is part of a
even after being national organization,
directed away and I just assumed it
was local

Now I feel like I am
starting all over. I
would say forget
them, and not to be
coldhearted but this is
really bad
Im wary of signing
up onlineI dont
Users wary to sign up want spam. I would Include an option for users to send direct
Joining mailing online; rather learn more messages beyond the Contact Us section
list Didnt know how from the website than
major quickly they would sign up Consider creating a discussion board so that
receive a response (if wary users may ask questions (thus remaining
at all) I have an issue with involved) without signing the mailing list
joining mailing lists
because I should be

able to call or email to
get involved, not just
sign up for a list

I dont want to join
the mailing listI
would just send them
a direct message or
call
Table B: Negative Findings

Condensed List of Recommendations

1. Add more photos throughout the site, and update staff information page to include
short bios, email links, and employee photographs.
2. Consider streamlining the homepage by deleting one of the social media feeds and
adding a short paragraph about the organization so that users do not have to navigate
to other pages to understand what BUILD does.
3. Bolster the Volunteer section to include a FAQ and an application form, as the National
site provides.
4. Consider adding a discussion board so that site visitors may read testimony or ask
questions to staff and volunteers, as an alternative to the mailing list.
5. Generally, add more information about BUILD: on the employee page, before the
BUILD Boston Story: Cristhian video, and on the Student Business page.
6. Make the link between BUILD Boston and BUILD National explicit throughout the site,
and also provide feedback for visitors that click on the National link, so that they
understand they are leaving www.buildinboston.org
a. Additionally, consider making the link to the National site look more like a
hyperlink or button, rather than just another sub-section of the menu banner.

Post-Task and Post-Test Analyses

While reflecting upon this data, it is important to remember that most users give feedback that
is more positive than what they experienced during testing, and this is known as acquiescence
bias. The qualitative findings here are mostly positive, despite some instances of frustration.


The following scale was used to gauge the responses of the participants.

1. Very 2. Difficult 3. Neither 4. Easy 5. Very easy
difficult difficult
nor easy

Participants were asked how difficult or easy they think the site will be to use.
All participants responded : Very easy (3 / 3)

Participants were asked how difficult or easy it was to find company history and mission.
Two participants responded : Very easy (2 / 3)
One participant responded : Very difficult (1 / 3)

Participants were asked how difficult or easy it was to locate volunteer opportunities.
All participants responded : Very easy (3 / 3)

Participants were asked how difficult or easy it was to find info about the BUILD community.
All participants responded : Somewhat difficult (3 / 3)

Participants were asked how difficult or easy the site was overall (final impressions).
Two participants responded : Somewhat easy (2 / 3)
One participant responded : Very easy (1 / 3)


Conclusion

This user test identified positives and problems of BUILD Bostons website, as accessed by a
group of target users. BUILD Boston, overall, remains an efficiently constructed and
navigationally-friendly website. Users were able to easily find where information was located,
and were satisfied with their ability to move through the website.

Users struggled a little in terms of content. They wished to see both more visual and verbal
content describing BUILD and the organizations work. In particular, the Volunteer section could
benefit from a FAQ page and a link to a discussion board or similar content to facilitate the
exchange of information and the fostering of community. Lastly, the connection to BUILD
National needs to be explained more explicitly and prominently. Users need to receive
feedback when they navigate away from BUILD Bostons page via the National link, and would
also like the link to be more distinctive.

While BUILD Boston should re-test following any and all modifications, the organization should
also strongly consider conducting further testing with a broader pool of participants.

Thank you for allowing me to examine the website. The usability test has been a beneficial and
enriching experience, and I hope that these findings will provide a positive course of action for
BUILD Boston.


Appendix A Persona

Persona: Financial Professional

Fictional Name: Eva Kim

Job Title/ Major Responsibilities: Investment and Accounts Manager, Wellington Management.
Job responsibility includes advising senior-level management with financial investments for
clients on a case by case basis.

Demographics: 32, Unmarried, No children

Goals and Tasks: She is detail-oriented and impeccably well-organized. She is a well-regarded
mid-level employee who works frequently with many different colleagues within the
organization. One of her main concerns is advising management and working with clients to
understand the best possible investment for their finances. She spends her work time: advising
management, consulting with clients, preparing memos and briefs, and assisting colleagues.

Environment: She is comfortable with using the Internet, as well as with all business software
required for her position. She uses social media fairly infrequently, but does have some social
media accounts. She uses email extensively and it is her main method of communication,
although she does consult print media and newspapers occasionally.

Quote: I would be willing to help with that


Appendix B Heuristic Evaluation
www.buildinboston.org
Heuristic Evaluation

Gabrielle LaFavre
October 2015

Table of Contents
Introduction..3
Research Question...3
Personas...4
Scenarios..4
Tasks5
Methodology....6
Heuristics.6
Ranking System...7
Evaluation Findings.8
Overview of Evaluation Findings....8
Positive Findings10
Negative Findings..11
Summary of Recommendations.18
Conclusion.18
References..18
Appendix....19

Introduction
I have conducted a heuristic evaluation of BUILD Bostons website, www.buildinboston.org,

using Grice et al.s Heuristics for Technology-Mediated Communication (Grice et al.).
Heuristics, a strategic group of principles applied by an evaluator, uncover the strengths and
weaknesses of a websites usability and usefulness. The heuristic evaluation uncovers potential
problems which the evaluator can further explore through a formal user test. This heuristic
evaluation uses the persona Eva Kim, developed out of formal interviews with target users, to
test BUILD Bostons website.
Research Question
My research question is whether or not BUILD Bostons website is unique, persuasive, and
credible enough to convince site visitors to become engaged with the organization beyond a
simple website visit. To gain supporters, volunteers, and donors, BUILD must contend with
many other nonprofit organizations in Boston. Does their website ultimately convert site visitors
into BUILD supporterswhether that be financial donors, mentors, student interns, or vocal
advocates?
This heuristic test will prioritize content and credibility over navigation and design, although the
latter categories will also have an impact on how users interact with the site, and thus what
opinion they form of BUILD Boston. I seek to understand if the website is both usable and
useful, and how the website conveys BUILDs credibility while fostering an engaged
community.

Personas
Eva Kim is a mid-level Investments and Accounts Manager. She is 32, unmarried, and uses the
internet approximately six hours a day. She is proficient using various internet browsers, email
platforms, financial software, and social media sites. She is well-regarded within her company,
which is an associated partner of BUILD. Becoming more engaged with her community, socially
interacting with colleagues outside of work, and meeting other like-minded people all motivate
Evas web usage.
Evas employer has a relationship with BUILD, but she personally seeks to connect with the
community, increase relationships with work colleagues, and meet other Boston-area
Professionals, both for networking and for friendship, through working with BUILD Boston.
Scenario
While at a work meeting one morning, Eva learns about a company that her employer sponsors,
called BUILD. Two esteemed colleagues discussed their experiences volunteering as mentors
with BUILDs entrepreneurship program. Eva hopes to advance within the company, forge
stronger connections with colleagues outside of work, and meet other civic-minded
professionals, so she decides to find out more about BUILD. By browsing BUILDs site, Eva
hopes to find out what BUILD does, which groups of people BUILD supports, what
opportunities exist for volunteers, and information about the companys credibility.
Eva Kim is exploring nonprofits in a work-related context; her colleagues have recommended
BUILD to her and have suggested that she become involved. Eva is a positive person, but lacks a
lot of timeif things dont work, she will move on. Eva prioritizes detailed information and
organizations with transparent organizational costs and salaries. As a meticulous and organized
person, Eva becomes slightly frustrated when websites do not match her high standards. Eva
fully represents a typical site user because she is a financial professional seeking general
information about the company.

Tasks
Eva visits the BUILD website for information about volunteering. To do so, she researches what
volunteers do, who they help, and what hours they commit to. She also hopes to discover if the
company is credible: she researches how they distribute their funds, what other organizations
BUILD is associated with, and if they have high overhead costs. Eva finds BUILDs website
through a Google search, and her pathway takes her from the homepage, to the About section,
through Partners, Events, Contact, and finally to the Volunteer section. Her specific
tasks are listed below.
As with many BUILD Boston site visitors, Eva:
1. Seeks BUILD informationThe large majority of users visit the BUILD website to
gather general information about the company and its history. Target users spend time
browsing for general information and asking themselves: what more can I find out about
BUILD? Eva seeks BUILDs history, mission, and any other relevant information.
2. Assesses organizations credibilityUsers research if the organization is credible,
therefore influencing their decision to become engaged with the nonprofit. As Eva uses
the website, she assesses BUILDs professionalism and partnerships.
3. Discovers opportunities to volunteerThis is a major task for users: discovering when
to become involved, where volunteer opportunities occur, and who the volunteers work
with. Eva hopes to find more information about potential involvement.
4. Locates information about communityUsers, including Eva, hope to form a
connection and engage with their community, which includes BUILD students, staff,
volunteers, and donors.
After using BUILD Bostons website, Eva will determine whether or not to become involved
with the organization on a deeper level.
In this report, the Methods section describes the evaluation process, including which heuristics
were applied, and the Findings section details both positive and negative findings. The report
concludes with a Summary of Recommendations and a Conclusion.

Methodology
Heuristics
To evaluate the BUILD Boston website, I have applied Grice et al.s heuristics. These heuristics
allow for thorough evaluation of BUILDs website; they prioritize content and usefulness over
design and usability. As I am evaluating BUILD Bostons ability to convert first-time site
visitors into returning supporters, both content and usefulness form the basis of my research
and Grice et al. allowed for a closer examination of such. Grice et al. also complements a
narrative approach to evaluation, and so these heuristics were most appropriate for this study. I
focused my research on whether the website is easy to understand and facilitates information
retrieval, and Grice et al.s task-oriented heuristics were relevantespecially because this set
includes a category for storytelling, both visual and verbal. Lastly, Grice et al.s heuristics
account for a persona with changing motivations and identities, which characterizes many of
BUILD Bostons visitors. The persona, Eva Kim, can be considered a produser, or
participatory audience, who prioritizes websites that are task oriented, accurate, and visually
effectiveall of which Grice et al.s heuristics outline and explore.

Grice et al.s Heuristics
Below are select criteria which I studied while evaluating BUILD Bostons website. For a complete list of
Grice et al.s heuristics, consult the Appendix at the end of this report.
Readiness / pre-use
Style appropriately suggests author authority / professionalism
Apparent value of communication / motivation is to engage
Technological requirements for access are minimized
Communication (appears to be) crafted with audience in mind, for a known context
Required background knowledge is available (unless intentionally excluded)
Navigation
Readability (for example, text large enough to read)
Similarity / compatibility with familiar tools
Clarity of control mechanisms and interactive objects
Flexibility and comfort with communication modes
Clear, efficient, and effective communication protocols
Meaningful categorizations
Meaningful hierarchy of media and text
Consistency of visual cues
Minimal syntactical complexity
Experience
Emotionally gripping / involving the affective domain
Incorporating rich communication modes matching user accessibility needs
Evoking confidence in the technology
Incorporating an appropriate degree of personalization
Displaying appropriate chunking of information
Visually supporting an immersive experience
Action / post-use
Call to action / next steps or additional information available.

Ranking Systems
Throughout the evaluation process, I ranked the findings based on a four-part scale:
4. Minor ProblemAffects the user experience but is not a priority; should be fixed when
time permits.
5. Moderate ProblemHinders the users ability to navigate.
6. Major ProblemProhibits the user from accomplishing tasks.
7. Positive FindingPositively enhances the users experience with the site.

Overview of Evaluation Findings
Positive Findings
Use is logical/straightforwardIntuitive and consistent menu banner, back to top

buttons, and icons to connect via social media on all pages of this website, which follow
internet best practices.
Experience is consistentEach individual section of the website contains the same
visual clues and user interface, which allows user to navigate the site mostly-intuitively.
Professional qualityAll areas of the website look polished and professionally made.
Video and photo slideshows on certain pages create an engaging aesthetic.
Visuals stimulate emotional involvementPhotos and slideshows on select pages
display students and volunteers, and add to companys ethos and pathos.
Motivates users to move through informationVolunteer, Subscribe, and Donate
buttons all change color (are highlighted) when a user hovers over them with a mouse,
which encourages the user to explore these sections.
Negative Findings
Engaging visual(Major Problem) Lack of visuals on the homepage, only a banner for
BUILDFest Event. Homepage could benefit from slideshow of students and volunteers.
Visuals stimulate emotional engagement(Major Problem) See above.
Provides background and informationHomepage does not provide adequate
information about the organization and its mission. User must navigate to another part of
the site to access this information.
Engages user and creates connectedness(Major Problem) There are no links for
current, prospective, or former members to converse digitally, and the website also lacks
volunteer testimony for potential users to read.
Support interactions among users(Major Problem) Website lacks opportunities for
interaction and channels to distribute user experience or contributions.
Experience is customizable(Moderate Problem) Website corrals user into either
volunteer or donor role through the buttons at the top of each section; does not account
for or allow user with both roles.
Simple ways to accomplish tasks(Moderate Problem) Must navigate multiple
dropdown lists on the menu bar, and user cannot press general category headings.
Welcoming design and experience(Moderate Problem) The homepage lacks a
welcome or general company information, which would positively encourage users to
remain on the site and explore other sections.
Motivate users to move through information(Moderate Problem) Lack of
information and welcome on the homepage, and there are multiple steps necessary to find
specific information; this discourages users, especially those that do not have knowledge
of BUILD or previous ties with the organization. These types of users are more likely to
move on to another non-profits website.
Minimal barriers(Moderate Problem) No visible search box.

Use is logical and straightforward(Minor Problem) When the user clicks on National
Chapter section, it takes the user away from the BUILD site and onto the BUILD national
website, not just in another tab.
Design for usability(Minor Problem) User experiences issues with the menu banner
while scrolling down the page within the site. The menu banner is large, but when the
user scrolls down past the fold, it suddenly becomes smaller. When user hovers over the
menu banner categories, it becomes a dropdown list, which obscures the text on the page
underneath.

Evaluation Findings
Positive Findings
There are several areas where BUILD Bostons website conforms to Grice et al.s heuristics.
1. Use is logical/straightforward
Intuitive and consistent menu banner, back to top buttons, and icons to connect
with social media found throughout all pages of this website.
Use of site follows Internet best practices.
2. Experience is consistent
Each section of the website contains the same visual clues and user interface.
Bright colors; Links for each site are consistent; Back to Top button
ubiquitous.
This allows users to navigate the site seamlessly.
3. Professional quality
All areas of the website look polished and professionally made.
Bright colors; professional video; functioning links.
Video and photo slideshows on select pages foster an engaging aesthetic.
4. Visuals stimulate emotional involvement

Select photos and slideshows display students and volunteers.
Compelling video on Our Mission Page; Photo of students with Senator
Markey; Student Business section includes information on each student-
team, which encourages users to buy products.
Where included, these visuals develop companys credibility and emotional appeal.
5. Motivates users to move through information

Volunteer, Subscribe, and Donate buttons all change color (become highlighted)
when a user hovers over them with a mouse.
This encourages the user to explore each section.

Negative Findings
There are many areas where BUILD Bostons site does not conform to Grice et al.s heuristics. I reviewed each of
the four tasks for each heuristic, and have reported on where issues arose. For each heuristic, I have listed the
associated task, the severity of each issue, relevant details, recommendations, and a screenshot where appropriate.
1. Design# 1 Task
Heuristic Finding and Image
for Recommendation
Diverse
Users
a. Minor Issue
Instead of opening a
Use is Locate new tab, clicking on
logical and BUILD National Chapter
straight- info (1) link takes user away
forward from the site
completely.
Seek info Recommendation

about
Allow clicking on
BUILD
communi
National Chapter
ty (4) to open in a new tab,
rather than
navigating user
away from the page.
b. Moderate Issue
Pigeon-holes user
Experience Locate into either volunteer
is BUILD or donor. Doesn't
customizable info (1) support user who is
both a volunteer and
a donor. Does not
allow for multiple
Discover identities.
volunteer
opportun
-ities (3)
Recommendation
In addition to
Volunteer and
Donate icons,
create another
button labeled
Advocate which
merges some
characteristics of
both identities.

2. Design Task Finding and Image
for Recommendation
Usability
a. Minor IssueMenu
banner slightly inhibits
Follows All tasks users navigation. Too
usability many dropdown
guidelines
options for this
heading.
Recommendation
Address and amend
issue.
b. Minor Issue Menu Larger:

banner at the top of
Design is All tasks each page is a large
well heading, but as you
organized
and
scroll down, it
readable suddenly becomes
smaller, which is
slightly disarming.
Smaller:
Recommendation
Address and amend
issue.
c. Moderate Issue
When user hovers over
Simple All tasks one of the banner menu
ways to categories with their
accomplish
tasks
cursor, the menu
becomes a dropdown
box that obscures the
text below it.
Recommendation
Address and amend
issue.

3. Makes Task Finding and Image
Users Recommendation
Feel
Welcome
a. Moderate IssueNo
welcome statement or
Welcoming Locate broad overview of the
design and BUILD company on homepage.
experience info (1)
Recommendations
Add brief statement
Assess about BUILD or
credibility BUILDs mission so
(2) that users can
understand what the
organization is as soon
as they arrive on the
homepage.
b. Major IssueLack of
visuals on homepage.
Engaging Locate Apart from banner
visuals BUILD
info (1)
about BUILDFest
Event, no other pictures
Assess of students, staff, or
credibility volunteers. Only a few
(2) photos throughout the
entire site.
Discover
volunteer
opportuniti
Recommendation
es (3) Add more visuals of
students and volunteers
to increase pathos of
website.
c. Major IssueNo
visuals on homepage.
Visuals All tasks
stimulate Recommendation
emotional
Add photos of students
engagement Only links to
and volunteers.
Consider adding a BUILDs social
slideshow of students media.
and their products on
the homepage.

4. Set the Task Finding and Image
Context Recommendation
a. Major IssueLack of
background
Provides All tasks information on the See photos in Heuristic #4 above.
background homepage.
and
introduction
Recommendation
Include a small
paragraph or a few
sentences about
BUILD on the
homepage, so users do
not have to search other
pages to understand
what BUILD does.
b. Moderate Issue
Because there is a lack
Motivates All tasks of information on See photos in Heuristic #4 above.
users to homepage, an
move
through
unmotivated user (one
information without an agenda)
may navigate to a
different nonprofit
website if they do not
immediately locate
information that they
seek.
Recommendation
Include background
about organization on
homepage.

5. Makes Task Finding and Image
a Recommendation
Connect-
ion
a. Major IssueLack of
volunteer testimonials
Engages Discover on Volunteer link,
user and opportuniti
es to
and no place for a
creates
connected- volunteer member or current
ness (3) volunteer to sign in.
Seek info Recommendation

about Add volunteer
BUILD
community
testimony to the site,
(4) and add an interactive
element to website
such as a discussion
board or member log-in
area.
b. Moderate IssueNo
search box on the site
Minimal All tasks to quickly find specific
Barriers information.
N/A
Recommendation
Add a search link or
box.
c. Moderate issue
Apart from one video
Uses Locate on the Our Mission N/A
storytelling BUILD section, the site lacks
to immerse info (1)
user
storytelling elements.
Assess
credibility Recommendation
(2) Include a blurb on the
homepage about the
Seek info organization and its
about mission. Emphasize
BUILD
community
that BUILD is not a
(4) simulation, but rather a
real-life entrepreneurial
accelerator.

6. Task Finding and Image
Supports Recommendation
interact-
ions
among
Users
a. Major IssueNo
interaction
Easily All tasks opportunities, such as a N/A
accessible discussion board.
interaction
opportunity
Recommendation
Include discussion
board or member log-in
section to encourage
information and
experience exchange.
b. Major IssueNo
ways to share ideas or
Easy ways All tasks experiences associated N/A
to distribute with BUILD.
users work
Recommendation
Increase user
engagement and foster
an inclusive
community by adding
interactive user sharing
platforms.
c. Major IssueNo
guidelines on how to
Clear All tasks interact with the larger N/A
protocols BUILD community.
for
interaction
with others Recommendation
Create links or
information about the
BUILD community and
how to engage with it.

7. Plans Task Finding and Image
to Recommendation
Continue
Engage-
ment
a. Major IssueApart
from the blog and
Relation- purchasing an event
ship with ticket once a year, there
site remains All tasks
ongoing
is little reason for a
volunteer or donor to
return to the site after
they are involved with
the organization; there N/A
are no ongoing online
elements to BUILD.
The website simply
informs.
Recommendation
Offer an interactive
virtual component to
complement the
volunteer and donor
experience, which will
increase feelings of
community.

Summary of Recommendations
In general, BUILD Bostons website would benefit from increased visuals, videos, and
volunteer testimony. Target users, such as the persona Eva Kim, seek emotional appeals and
visual displays to better understand what obstacles a nonprofit undertakes, as well as what goals
it accomplishes. By including more pictures on the homepageperhaps a slideshow of students
with their productsusers visiting the site will be more likely to explore other pages, rather than
navigating away to another non-profit. To allow for users with multiple identities, BUILD
Boston could create another button labeled Advocate in addition to the pre-existing
Volunteer and Donate Icons. Additionally, the other major area for website improvement
would be to create an interactive sharing and discussion platform, such as a member or
volunteer sign in area. This would greatly facilitate two major goals: to attract and retain
prospective volunteers and to foster an inclusive community. This is because potential volunteers
will benefit from reading volunteer testimony, and will also be able to directly ask questions
about the BUILD Boston experience. Concurrently, this interactive community area would allow
current BUILD volunteers, staff, and corporate partners to exchange ideas, advertise events, and
create inclusivity.
Conclusion
Heuristic evaluation of BUILD Bostons website revealed several issues which inhibit user
experience. I will further evaluate major and moderate issues in a formal user test to follow. By
delving into these issues through real user tests, I expect to identify further recommendations for
website improvement and increase the user experience.
References
Grice, Roger A. et al., "Heuristics for Broader Assessment of Effectiveness and Usability in
Technology- Mediated Technical Communication." Technical Communication 60.1
(2013): 3-27. 2 Nov. 2015.

Appendix
I. Heuristics and sub-items II. Operationalized metrics
1. Design for diverse users

User is confused < > User understands everything
Product Metric: Use is logical and straightforward.

Behavioral Metric: User understands the interface without
a. Recognize that nothing is intuitive to
assistance, does not get
everyone
confused.
Survey Metric: User describes experience as logical or
intuitive.
Greater confusion for some groups of users < > Diverse users
understand
b. Design for the inevitability of diverse Product Metric: Experience is consistent across user types.
audiences Design elements have the same meaning for all users.
Behavioral Metric: User (type) not stumped by the design.

Survey Metric: User describes experience as easy to follow.
User is limited by design < > User has options
Product Metric: Experience customizable for different users;

c. Provide users with options for
customization does not hinder design use.
differential experience using different
views or levels
Behavioral Metric: User is able to customize with ease/finds
and enjoys a suitable view. Survey Metric: User rates
customization highly.


2. Design for usability

Confusing non-traditional design < > User recognized standard elements
a. Follow standard usability
Product Metric: Design follows usability guidelines.
guidelines
Behavioral Metric: User understands the design based on other
experiences. Survey Metric: User describes experience as a familiar one.
User disoriented or led astray < > User easily perceives site content
b. Enforce readability (font
large enough to read; break up Product Metric: Design is well organized and easy to navigate.
blocks of text) Behavioral Metric: User finds what he or she is looking for in a timely
manner. Survey Metric: User describes experience as efficient.
Design perceived to be standard < > Design perceived to be enhanced
c. Use professional quality
Product Metric: Appearance and content suggest professionalism to user.
design components
Behavioral Metric: User prefers design vs. other designs.
Survey Metric: User describes experience as professional.
Highly unfamiliar < > User experiences familiarity where expected
d. Follow general conventions
Product Metric: The design is organized and consistently familiar.
where available
Behavioral Metric: User is more comfortable with the design vs. others.
Survey Metric: User describes the experience and familiar and enhanced.
Many navigation complications < > Quick, free user motion throughout
e. Offer simple ways to do
Product Metric: Components are in correct locations. Links work.
what users want to do
Behavioral Metric: User efficiently navigates through site/design. Survey
Metric: User describes experience as uncomplicated.

3. Make users feel welcome

User feels put off or unwelcome < > Users feel welcome
Product Metric: Design and experience feels welcoming and

friendly.
Behavioral Metric: User lingers/spends more time in initial,
a. Make users feel welcome
welcoming screens or
areas.
Survey Metric: User describes experience as welcoming or
inviting.
Users is intrigued by visuals < > User is annoyed by visuals
Product Metric: User is engaged by visuals, not distracted by

b. Use visuals to draw users in them.
Survey Metric: User describes visuals as enhancing the experience

or as highly useful and helpful.
User is engaged by sounds < > User is distracted or annoyed by
sounds
Product Metric: Sounds are used constructively.

Behavioral Metric: User stays focused, finds sounds useful or
c. Use sound to enhance experience engaging, is not
distracted or put off by sounds.
Survey Metric: User describes sounds as helpful, useful, or

enhancing the experience / understanding of the content.
User unresponsive to design < > Appropriate user emotions are
triggered
Product Metric: Visual elements stimulate user emotional

d. Engage the affective domain with
engagement. Behavioral Metric: User responds to visual language,
visual language (color, icons,
is drawn in.
symbols)
Survey Metric: User describes visual language used as engaging,
enhancing the
experience, or in terms of appropriate emotional response.

4. Set the context

User feels unready or unprepared < > Users feel prepared
Product Metric: Experience has appropriate precursor activities

that allow for familiarization.
a. Design activities that allow users to
become prepared for the experience
Behavioral Metric: User encounters an appropriate introductory
experience that supports what follows.
Survey Metric: User rate preparation as useful or helpful.

User lacks context to perform < > User has sufficient background
b. Provide users introductory context Product Metric: Background information needed is provided.
Behavioral Metric: User is not puzzled at any stage.
Survey Metric: Users rates their contextual readiness as high.
User has no drive to continue < > User moves smoothly through
Product Metric: Experience motivates users to familiarize

themselves with the interface, moves them smoothly through as
they are ready.
c. Motivate users to move through any
Behavioral Metric: User responds to incentives, increases
necessary initiation
familiarity or demonstrates proficiency, and moves through the
experience.
Survey metric: User finds the introduction worthwhile, is not

frustrated or unprepared at any stage, or describes initiation as
enhancing.
User spends a long time on setup < > User passes through setup
quickly
d. Limit setup time to a small portion
Product Metric: Setup is quickly completed by any user.
of the total experience
Behavioral Metric: User is not confused at any stage of setup.
Survey Metric: User perceives setup as taking a reasonable or
minimal amount of time.

5. Make a connection
User feels detached < > Users feel drawn in
Product Metric: Users can relate to elements of the experience.

Behavioral Metric: User is focused on the product. User takes
a. Engage people in what is going on; less time to learn. User is
create connectedness
immersed in the experience.
Survey Metric: User rates the connectedness of the

experience highly, or describes it as immersive.
Users get stuck < > Users overcome barriers quickly and
easily
Product Metric: Barriers are minimal; universally identifiable

and easily grasped hooks offer routes through any necessary
b. Understand potential barriers and offer barriers.
users identifiable ways to overcome them
Behavioral Metric: User does not encounter design barriers, or
easily overcomes obstacles.
Survey Metric: User perceives experience to be barrier-free.

Users describe hooks they encounter as easily understood.
User uninvolved, rejects premise < > User is drawn into
story/encounter
Product Metric: Story is worked into experience seamlessly.

c. Use well-crafted storytelling to Behavioral Metric: User is invested in story and encounter,
immerse users in the encounter does not want to leave
experience.
Survey Metric: User rates storytelling highly, describes
encounter as immersive.

6. Support interactions among users

User feels isolated from other users < > Users interact
Product Metric: Experience contains easily accessible interaction

opportunities. Behavioral Metric: User encounters chances to interact with
a. Create opportunities for others. User interacts
users to interact
with others.
Survey Metric: User rates the experience as very interactive. User is happy
with the quality of interactions present.
Users cannot share creations with others < > Users share their creations
Product Metric: The experience includes easy ways to distribute user work.
b. Allow users to share what Behavioral Metric: User utilizes the sharing options.
they create Survey Metric: User rates sharing options highly or reports sharing to be a
key part of
the experience.
User is confused re: interaction < > User understands sharing procedure
Product Metric: The experience embeds obvious protocols for interaction

c. Provide clear protocols with others. Behavioral Metric: User recognizes and makes use of
for interaction with others interaction procedures easily
and without errors.

Survey Metric: User rates the interaction procedures as obvious.

7. Plan to continue the engagement

User is stuck in the past < > User is ready to continue
Product Metric: Relationship with product is ongoing, can persist

beyond a single experience or task.
a. Design for the next engagement
Behavioral Metric: User is drawn into/stays with experience, is
willing to return to it.
Survey Metric: User rates continuity of engagement highly, spends

more time with the experience.
User is stuck deciding < > User understands what to do next
Product Metric: The experience includes beneficial guidance for the

user and clear action options.
b. Make calls to action clear
Behavioral Metric: User makes easy progress from action to action.
Survey Metric: User rates calls to action as clear and easily
understood.
User has no interest in continuing < > User pursues deeper
connection(s)
c. Invite users to continue
connections past the current
Product Metric: Progress beyond any given point is available to the
encounter
user. Behavioral Metric: User moves deeper into the experience.
Product Metric: User rates access to further experiences highly.

Appendix C Test Plan

www.buildinboston.org
Test Plan

Gabrielle LaFavre
October 2015


Table of Contents

Purpose..3
Problem Statement and Test Objectives... 4
User Profile..5
Methodology and Tasks..6
Preparation
Testing
Think Aloud Protocol
Session Length
Questions, Tasks, and Scenarios.. 7
Project Timeline... 13
Evaluation Data..13
Quantitative Data
Qualitative Data
Deliverables..13
Appendix A: Video Consent Form....20


Purpose

The purpose of this report is to plan a testing session for BUILD Bostons website,
www.buildinboston.org. The usability test will allow collection of data which will evaluate the
users experience interacting with BUILD Bostons site. This data will record how users complete
four primary tasks, whether they encounter any obstacles, and their self-reported feelings
while interacting with the website.

This report will uncover whether the website is unique, persuasive, and credible enough to
convince site visitors to become engaged with the organization beyond a simple website visit.
My research question continues to evaluate how BUILD gains support by means of their virtual
presence; BUILD must contend with many other nonprofit organizations in Boston. Does their
website ultimately convert site visitors into BUILD supporterswhether that be financial
donors, mentors, student interns, or vocal advocates?

The following test plan provides:
The problem statement and test objectives
User profiles
Testing methodology
Tasks and scenarios
Testing facility and equipment
Evaluation methods
Project deliverables
Questionnaires
Participant screening questionnaire
o Pre-test questionnaire
o Post-task questionnaires
o Post-test questionnaires
Moderator script
Video permission
User persona


Problem Statement and Test Objectives

This usability study of www.buidinboston.org will assess the user experience with BUILD
Bostons website. It will evaluate how target users undertake typical tasks and familiarize
themselves with the site. Through assessment of four primary tasks, the report will produce
qualitative and quantitative data about the ease of user experience, any obstacles which inhibit
use, and self-reported feelings about the user experience.

The primary objectives of this study will prioritize content and credibility over navigation and
design, although the latter categories will also have an impact on how users interact with the
site, and thus what opinion they form of BUILD Boston. In particular, I will evaluate the ease of
locating company information, the credibility of the organization, uncovering opportunities to
volunteer, and the presence or absence of a BUILD community.

This study will evaluate, and how the website conveys BUILDs credibility while fostering an
engaged community through the following tasks:

1. Seek the company history and mission of BUILD Boston
2. Assess the organizations credibility and professionalism through community and
business partnerships
3. Discover opportunities to volunteer
4. Locate information about the BUILD community for networking purposes

I will evaluate each task considering the following underlying criteria:
Navigation: are all users able to use the site and complete tasks easily?
User satisfaction: which aspects of the site does the user like? Dislike?


User Profile

I have developed a user profile to reflect one of BUILD Bostons target users. The user profile
was used to create a user persona, included in Appendix E. I will test select participants based
on the user profile characteristics listed below.

General user characteristics:
Aged 21-40
Business Professional or Young Professional
Temporary or Permanent Resident of Massachusetts
Has interest in volunteering and/or philanthropy
Uses the internet regularly
Motivated to give back to society and increase community engagement

Participants must be employed professionals with an interest in volunteerism and/or
philanthropy, and should be located in Massachusetts, as BUILD Boston relies heavily upon
locally situated volunteers and donors.


Methodology and Tasks

Usability testing on BUILD Bostons website will focus on major and moderate issues raised
during heuristic evaluation. I will test www.buildinboston.org with three participants who meet
user characteristic requirements. One additional participants will function as alternate, in the
case of cancellations. All participants will perform a series of tasks using the site, and their
actions and responses will be recorded.

Preparation
To ensure a successful user test, I will conduct a walkthrough practice of the test plan and
procedure with a tolerant participant. This participant will be unfamiliar with the website, but
will not need to pass the screening questionnaire to qualify for this run-through.

After the run-through, I will conduct a pilot test which will use the first of the recruited
participants. If I do not need to make major changes to the test plan following this pilot, I will
then include the results of the pilot in the final report.

Testing
The three participants will undergo a user test over the course of one week, November 21-
November 29. Each test will include documentation of analysis and post-test questionnaires.

Think Aloud Protocol
I will ask all participants to perform think aloud protocol as they complete the associated tasks.
To accomplish this, I will encourage participants to verbalize their thoughts, actions, feelings,
concerns and joys that result during their experience on www.buildinboston.org. This will allow
me to record the actions and reactions of users as they encounter obstacles or successfully
complete tasks.

Session Length
Each session will last for approximately one hour, and will allow users to complete the
following:

Introduction and pre-test questionnaire (5 minutes)
Scenarios, including brief post-task questionnaires after each scenario (40 minutes)
Closing with post-test questionnaires (10 minutes)


Questions, Tasks, and Scenarios

The tasks created for each scenario result from a combination of the heuristic evaluation results
and the concerns over how the website converts site users into organizational supporters.

The primary areas of concern are:
The sites contentability to locate sites information
The organizations credibility and professionalismassessing professionalism and
partnerships
Ease of locating volunteer opportunities and means of organizational engagement
The ability to identify, associate with, and interact with a community for networking
purposes

While considering:
Navigational simplicity and conventionalityis the site easy and intuitive to use?
User satisfactionis the site user-friendly, or are there inhibiting obstacles?

Scenario:
What BUILD stands for; What BUILD does; Who BUILD benefits; and How to get involved

The following tables provide information that will help guide testing, including:
Question: What the user should answer based on concerns raised during the heuristic
evaluation.
Task: The task performed in order to answer the question.
Scenario: The scenario which I will provide the participant during testing.


Question:
User Satisfaction and Navigational
Simplicity:

Does the homepage welcome user and
encourage user to remain on the site
and explore further? First impressions.
Task:
Getting a feel for the organization, as
well as understanding its purpose and
ethos.

Ask user how the site makes them feel;
what are their first impressions, what
is the tone/expectations of this site?
Scenario:
Look around the homepage and tell me
what you think about it. What do you
think you can do on this website?
Where would you go to get started?

Question:
The sites content

Can you easily find
the organizations
history and mission
statement?

What does the
organization stand
for? What does it
actually do?
Task:
Choosing which
website section, or
combination of
sections, to visit in
order to
understand BUILD
Bostons ideology
and daily activities.
Scenario:
You wish to
understand more
about BUILD before
you decide to
volunteer.
Where can you find
organizational
information? Find
the companys
mission and
history.

Go ahead and read
the content,
please.

What does BUILD
Bostons website
tell you about this
nonprofit?


Question:
Ease of locating
volunteer opportunities
and means of
organizational
engagement

Can user find
opportunities to
volunteer? Does the
user know where,
when, and why to
volunteer with BUILD
Boston?
Task:
Find where the
information on
is located.
Scenario:
Can you find
information about
and means to engage
with the organization?

Is the information
complete based on
what you readdo you
understand when,
where, and how
volunteers engage with
the organization, as
well as what they
actually are able to do?


Question:
The ability to identify and
interact with a community

Task:
Find a means to interact
with the greater BUILD
Boston community
whether that be schools, N/A
students, employees,
business partners, or other
volunteers.
Scenario:
Imagine you still have
particular questions to ask
current volunteers and
BUILD employees.

Additionally, you hope to
read testimony from
students and current
BUILD volunteers. Where
would you go to ask your
questions and locate this
testimony?


Question:
(if time permits)
The organizations
credibility and
professionalism

Is this website
professional? Does BUILD
Bostons website allow
users to form a positive
judgment of the
organization? Can users
find community and
business partnerships?

Task:
Evaluating the ethos and
credibility of BUILD Boston
through the
professionalism (or lack
thereof) of the website.
Scenario:
You heard your co-workers
discussing BUILD, and they
mentioned that BUILD
Boston has a partnership
with your company. You
would like to verify this
partnership by checking
their partners online. You
would also like to see
which schools BUILD works
with.

How can you complete this
task? Please find
community and business
partners now and evaluate
them.


Project Timeline

11/10Workshop Test Plan Draft
11/16Pilot User Test
11/24Submit Test Plan
11/21-29Conduct User Tests
12/1Summary of Findings
12/8Submit Final Report

Evaluation Data

Quantitative Data
I will collect both quantitative and qualitative data during the user test. The quantitative data
will measure feedback on users responses to questions. Questionnaires will include a set of
standardized questions for usability studies.

Qualitative Data
I will monitor users throughout the test, and will collect data on remarks, non-verbal body
language (such as sighs), and facial expressions. This will form the qualitative data for the
report, as well as administering post-test surveys.

Pre-test, Post-task, and Post-test questionnaires are available in the Appendix section of this
document.

Deliverables
I will conclude this study on December 8, 2015, and will produce a formal written report to
follow.


Appendix D Pre-Test Questionnaire

Screener: Thank you for participating in this usability test. I appreciate your time, and will use
data from this study to provide valuable feedback. Before we begin, I am going to ask you a few
questions.

Screener: Have you ever used a particularly frustrating website? What were the most
frustrating aspects?

Answer:

Screener: Are there any websites that are enjoyable for you to use? What are their
characteristics?

Answer:

Screener: If you are looking for an organization to volunteer with, what information is most
important for you to locate on their website?

Answer:

Screener: Which nonprofit website have you visited in the recent past? What did you like about
it? What didnt you like? How did you use the site?

Answer:


Appendix E Post-Task Questionnaires

Task One
Please rate your impression of how difficult or easy you think this site will be to use. Circle the
option below.

1. Very difficult
2. Somewhat difficult
3. Not difficult nor easy
4. Somewhat easy
5. Very easy

Please write why or why not you think it will be easy to use:

What is the likelihood that you would continue using this site? Would you navigate to a
different nonprofit website based on your experience and impressions of BUILDs site?

Task Two
How easy was it to find the company history and mission based on where you navigated?

1. Very difficult
4. Somewhat easy
5. Very easy

Did it take you more or less time than you expected to locate this information?

Did the organization seem credible and professional? Why or why not?


What does the website tell you about this nonprofit?
Task Three
How easy was it to find volunteer opportunities?

1. Very difficult
4. Somewhat easy
5. Very easy

Did the information that you found provide a detailed picture of what volunteers do and how
they become engaged with BUILD Boston?

Based on what you found on the website, how would you actually begin the volunteering
process?

Task Four

How easy was it to find information about the BUILD community, such as networking or
testimonials?

1. Very difficult
4. Somewhat easy
5. Very easy

Do you feel like you were able to understand who is part of the BUILD community, as well as
how to engage with this community?

Based on what you found on the website, how would you get in contact with current
volunteers?


Appendix F Post-Test Questionnaires

To what extent did you:
(Use any descriptive words that apply)

1. Find this website easy to use? Provide an example.

2. Find this site pleasant to use? Provide an example.

3. Were able to locate information that you needed quickly and easily? Provide an
example.

4. Find this website professional, well organized, and persuasive? Provide an example.

5. Find BUILDs website contained content about the organizations history, mission, and
activities, which painted a compelling picture of this nonprofit?

6. Understand who is part of the BUILD community, as well as how to engage with this
community? Explain.

7. Think you would return to this website and/or recommend it to a friend?


8. Be willing to become involved with BUILD in some capacity, even just re-visiting the
website?

Please rate your final impressions of using BUILDs site?

1. Very difficult
4. Somewhat easy
5. Very easy

If your actual experience using BUILDs site differed from your first impressions, please
comment on what changed your impression:


Appendix G Moderator Script

Introduction
Hello, my name is Gabrielle LaFavre. I am a graduate student at UMass Dartmouth. I am
conducting a usability test on BUILD Bostons website. I will guide you through this testing
process, and if you need any assistance, please ask. I appreciate your assistance with my test,
and I am not testing you but rather how the site works for you. I am not associated with BUILD
and therefore will not be offended by any comments you may have. I am reading this script so
that I can provide the same information to each participant.

Camera and videotaping permission
I will be videotaping your test for my research purposes, so that I may refer back to the footage
in case I cannot write something down. With your permission, I will only use your video until I
complete my final report, and then I will destroy any video footage. Additionally, I will only use
your first name if necessary during the user test process. Please sign and date this consent form
after you have finished reading it. [Present consent form]

Test Information
As I mentioned, this is a test of the BUILD Boston website, not a test of your capability or
capacity as a user. This test will demonstrate how real people use the website. I will learn what
works or doesnt work based on the obstacles and successes that you experience, as well as the
comments that you provide.

I am going to ask you to Think Aloud. This means that I want you to tell me what goes through
your mind as you use this site. Feel free to say things like I am clicking on or I dont like this
because or I didnt think this would be here Here is an example of Think Aloud Protocol in
action [Show video] [Make sure sound is turned on]

I will tell you certain scenarios to complete, and when you feel that you have completed the
task please let me know so that you can take the brief post-task survey. Then we will continue
until we have finished the user test.

Before we begin, please fill out this pre-test questionnaire. [Complete questionnaire]

Now we can begin. Please remember to think out loud and when you think you are done, say
Im done.

I am going to read the imaginary scenario of your use:


What BUILD stands for; What BUILD does; Who BUILD benefits; and How to get
involved.

Task #1:
Look around the homepage and tell me what you think about it. What do you think
you can do on this website? Where would you go to get started?

Task #2:
You wish to understand more about BUILD before you decide to volunteer. Where
can you find organizational information?
Find the companys history and mission, and go ahead and read the content, please.
What does BUILDs website tell you about this nonprofit?

Task #3:
Where would you find information about volunteer opportunities and means to engage
with the organization?
Is the information complete based on what you readdo you understand when, where,
and how volunteers engage with the organization, as well as what they actually are able
to do?

Task #4:
Imagine you still have particular questions to ask current volunteers and BUILD
employees, and also hope to network with them. Find how to contact current
volunteers and staff.
Additionally, you hope to read testimony from students and current BUILD volunteers.
Please locate these testimonies.

[Participants complete tasks and post-task questionnaires. They then complete the post-test
questionnaire]

Final Questionnaires
This is the last thing I will ask you to do. Here is the final questionnaire.

After Testing
Thank you so much for assisting me today. I learned a lot from your feedback, and really
appreciate your time.


Appendix H Executed Video Permission Forms



Appendix I Participant Responses to Questionnaires

Pre-Test Questionnaire

Thank you for participating in this usability test. I appreciate your time, and will use data from
this study to provide valuable feedback. Before we begin, I am going to ask you a few questions.

1. Have you ever used a particularly frustrating website? What were the most frustrating
aspects?

Answer:
Yes. When I couldnt find what I was looking for, or was unable to complete the goal or task that
led me to the website.

2. Are there any websites that are enjoyable for you to use? What are their
characteristics?

Answer:

Yes. Websites with lots of pictures, and the option of gathering more textual information when
needed.

3. If you are looking for an organization to volunteer with, what information is most
important for you to locate on their website?

Answer:

Mission statement, role as a volunteer, who/how I will be helping, expected time commitment
(is there flexibility?)

4. Which nonprofit website have you visited in the recent past? What did you like about it?
What didnt you like? How did you use the site?

Answer:

Boys and Girls Club Cheney Branchto look up specifics about the branch. It was fine, but not
that interactive.
Post-Task Questionnaires

Task One
Please rate your impression of how difficult or easy you think this site will be to use. Circle the
option below.

6. Very difficult
9. Somewhat easy
10. Very easy

Please write why or why not you think it will be easy to use:

Big pictures, little amounts of text, obviously clickable buttons that seem appropriately named.

What is the likelihood that you would continue using this site? Would you navigate to a
different nonprofit website based on your experience and impressions of BUILDs site?

I would definitely click farther into the website, especially with the clickable squares that say to
get involved, etc. They make it easy to explore further. The menu at the top, especially the
about button, is where I would go next.


Task Two
How easy was it to find the company history and mission based on where you navigated?

6. Very difficult
9. Somewhat easy
10. Very easy

Did it take you more or less time than you expected to locate this information?

I was unable to find it.

Did the organization seem credible and professional? Why or why not?

Yes.

-New York Times Article
-Opportunity to meet a senator
-variety of students in the pictures, which means that the program is probably successful
somewhere

What does the website tell you about this nonprofit?

-Goal is to help students build business relationships/skills
-Goal is to cultivate young entrepreneurs
-Wants you to volunteer and or donate


Task Three
How easy was it to find volunteer opportunities?

6. Very difficult
9. Somewhat easy
10. Very easy

Did the information that you found provide a detailed picture of what volunteers do and how
they become engaged with BUILD Boston?

Kind of. The reader gets the basics on the volunteer page. You know the mission of the
volunteers, and you know the time commitment. The where is unclearBostonbut not
specific as to which school sites. The volunteering blurbs also dont give any information
as to what volunteers are expected to do, make lesson plans? Do they get formal
training? How is a volunteer qualified for this position?

The most clear was the event-based volunteer.

Based on what you found on the website, how would you actually begin the volunteering
process?

I guess I would have to sign-up. I wouldnt want to sign-up because if I figure out I am not
qualified or not interested in volunteering then I am stuck with annoying emails.


Task Four

How easy was it to find information about the BUILD community, such as networking or
testimonials?

6. Very difficult
9. Somewhat easy
10. Very easy

Do you feel like you were able to understand who is part of the BUILD community, as well as
how to engage with this community?

I was able to understand what kinds of students (students like Christian according to the video),
but not the types of people who volunteer (are they current business professionals?
Entrepreneurs? Or business students?

Based on what you found on the website, how would you get in contact with current
volunteers?

I would probably just call Build Boston and ask to get in contact with someoneor email if I
couldnt get through on the phone.


Post-Test Questionnaire
To what extent did you:
(Use any descriptive words that apply)

9. Find this website easy to use? Provide an example.
Mostly yes. For obvious information like: about. It was difficult to find more in-depth
information about volunteering.

I also ran into a snafu, where I ended up at the national website (I didnt even know there was a
national website) and couldnt get back to the Boston page.

10. Find this site pleasant to use? Provide an example.

Yes. The colors were appealing, lots of pictures.

11. Were able to locate information that you needed quickly and easily? Provide an
example.

Yes: ex. mission statement
No: ex. Volunteering specifics

12. Find this website professional, well organized, and persuasive? Provide an example.

Yes. The colors were well integrated throughout. The website was modern looking, short
paragraphs to read, and bullet points.

13. Find BUILDs website contained content about the organizations history, mission, and
activities, which painted a compelling picture of this nonprofit?

The mission, the video, and the examples of student projects (which was the most compelling
part to me).

14. Understand who is part of the BUILD community, as well as how to engage with this
community? Explain.

I understand the students are underserved and I dont know particularly about the
volunteers.


15. Think you would return to this website and/or recommend it to a friend?

If a friend was looking for a volunteer position like this one, I would recommend it. If I was
looking for a volunteer position I might return if I didnt find something I liked better.

16. Be willing to become involved with BUILD in some capacity, even just re-visiting the
website?

Perhaps.

Final Post-test Questionnaire
Please rate your final impressions of using BUILDs site?

6. Very difficult
9. Somewhat easy
10. Very easy

If your actual experience using BUILDs site differed from your first impressions, please
comment on what changed your impression:

N/A
















Appendix J: Test Log
Rachel Notes:
TASK 1 & 2
Like slideshow
Good speed and photos
I can see the logo. It is warm and inviting
Dont like social media right on front
Rather know more about org before I see tweets. Waste of space
Would go to menu and go to our mission to see what they do
Reads mission, likes it. Clear, concise, understands what they do.
I get the point of having a video but I wouldnt watch it. I would rather read
First paragraph= know exactly what they do
Like that they clarify that it isnt vocational education because name is misleading
Would go to programs because would want to know more
Went to student businesses
Looked a little confused reading it
But said individual pages dedicated to each business, looks at another tab
Reads it and smiles. Says its cool
Wants to know how they select students
Goes to school partners
Reads list
Now goes to corporate partners ooh pretty good people
Want to know how to become a mentor
What does builds website tell you about this?
Work in Boston, with students in high school, support students starting own business
and creates entrepreneurship among kids who wouldnt have economic venture
opportunity without build. Mentor must have experience with starting or running
business
TASK 3
To find volunteer info went to menu, volunteer button
They look for mentors, so assume that is the volunteer-ship
Join the BUILD family
Great blurb, looking for professionals, entrepreneurs and business people, talks about
the community
Says volunteer responsibilities
The How Can you get Involved heading is good
Likes that they are three concrete types of volunteers
Likes that it has time commitment and what the responsibilities are

Skill-based is flexible; looks at what you have to offer and how you can most benefit the
students and not fitting you into a pre-conceived role
And then extra sets of hands at events, seems well organized
I definitely understand when and how; where seems quite flexible. Is it at BUILDs office
or do you meet at school? Probably depends on students schedule and what they are
comfortable with.
Next step would be to join mailing listbut issue with it because dont need to join
mailing list to get involved should be able to call and ask not just join list
Is there a training before?
Is there orientation? Do you need qualifications?
TASK 4
To contact staff and volunteers, go to our staff, see if links for emails
Looks confused
No links to staff
Should have a bio and what they do. What is E1 Program manager? No idea what it
means. If I was going to volunteer, I would want to know what each person does
I assume Carina is person to speak with because volunteer coordinate but cant contact
her
Took her long time
Contact us is just email feed. No direct way to get in contact with current volunteers
that I can see
Clicks on BUILD national, confused. Oh, interesting it is part of a national organization
Knew to use back button on browsers
Interesting, in mission statement doesnt say it is part of a national org, I just assumed it
was just Boston org, not until I saw BUILD national that I realized it was national
Clicks on national page volunteer section
BUILDs national volunteer page is more explicit. There is an application
There is a mentor FAQ page that is really cool. Should be on Boston Page!!!
Yeah this should totally be on BUILD in Boston page
You would think national org would send website guidelines to each branch
National site join us section is there anything about volunteers here
Saw Boston enquires carina down at the button with email links. Should be on Boston
site
I would email email her if I knew to go there but honestly I just completely stumbled
upon it
I was missing a break down of their work, not just what they do but also how. Not just
on homepage but somewhere I could find it
Oh look there is E1 so now I understand her job. I like BUILD nationals website better.
More information
This is so much more helpful
On that NYT banner interesting because it says it is a promising solution for the dropout
rate. Why is to help with kids staying in schoolonly just realized that. I like that it
explicitly says it.

Testimoniesfrom students I would go to student businesses, I think? Umm okay well
this is just profile of each business ummm not really there which is where I thought it
would be
Perhaps it is in the news maybe? No there isnt really
Okay maybe blog. Its not clear where I would find it and I would definitely stop looking
now. It is taking me too long

Jens Notes
TASK 1
I would go to the top and go to drop down menus and see what interests me first
I like when it has little dots at button so you can scroll back to moving picture menu
I like the tweets. Thats cute
Are these both tweets? Yeah okay. But it is actually fb
Like that it has address and homepage right there
I would click on volunteer first to see what they want from me. If they want too much I
wouldnt be interested
Looks at about tab
Says it has local stuff so I would click on all of them and read them
Reads our story. Clicks on BUILD video didnt load at first
Reads our program
Sad video so far. Sees that it is a real business program and says OOH that is not at all
what I thought it was going to be. I thought it was going to be building things. Even after
seeing the homepage she didnt get what it did. The kids look happy
So is it teaching them how to build a business?
Ooh okay so they have a competition and then they win. I see.
I thought it made the charity look really good. Well done video. I think maybe they
should tell you a little more about what it is first and then show you the video so you
understand what they are doing.
Oh okay this tells what it is. Maybe I should have read this first or they should have put
this before the video
Oh so It is an elective in the school wow. So they work with the schools
I think that all looks really good so that all describes it well
Then she clicks on our staff
I think they should out a picture on this page next to everyone
Clicked on BUILD national site
Clicks on About us
I think this is weird because now their homepage looks totally different this time around
I dont like how the website looks different now. Definitely not good.
The menu categories are even different descriptions underneath
Now I feel like I am starting all over. I would say forget them and not to be coldhearted
but it is really bad

Goes to donate section
$5000 is too much for the average person. Can choose specifically where money goes so
that is good. Pretty straightforward.
TASK 2 question:
Who the heck knows because its just going to change for the next person
TASK 3
Big box that says volunteer
Nice that it gives you options to sign up based on what you are interested in
I think that if they have any basic needs they should put this here bc this looks like you
can only help if you are entrepreneur/profession so maybe they should put other things
here so no one feels excluded
This is good right up front it says time commitment and months
Okay I think this is really good how it gives you detail
Nice that there is a flexible option for people who cant commit
Describes it well and gives you option
When and how yes, I understand
Where: hold on. Looks puzzled. Not so much where. Yeah I do you meet at school,
library?
TASK 4
Contact page
Okay to this just seems exactly like join our mailing list but you could put in a message.
Thats good. Oh and there is address and mailing list. I am the type that would just call
I would just call number and ask how to contact volunteers
Locating testimony
Goes to partners
Corporate partners
Not helpful
Goes to blog
I would skim one blog post
Back to school with BUILD
This is good because it tells what goes on and recent history but there are no
testimonials
So if they are in here they are not super easy to find
Maybe I would just go to twitter because it is on the homepage

BUILD Usability Report

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

BUILD Usability Report

Caricato da

Copyright:

Formati disponibili

www.buildinboston.

LaFavre Usability Report 1

LaFavre Usability Report 2

LaFavre Usability Report 3

LaFavre Usability Report 4

LaFavre Usability Report 5

LaFavre Usability Report 6

LaFavre Usability Report 7

LaFavre Usability Report 8

LaFavre Usability Report 9

LaFavre Usability Report 10

LaFavre Usability Report 11

LaFavre Usability Report 12

LaFavre Usability Report 13

LaFavre Usability Report 15

LaFavre Usability Report 16

LaFavre Usability Report 17

LaFavre Usability Report 18

LaFavre Usability Report 19

LaFavre Usability Report 21

I have conducted a heuristic evaluation of BUILD Bostons website, www.buildinboston.org,

LaFavre Usability Report 22

LaFavre Usability Report 23

As with many BUILD Boston site visitors, Eva:

LaFavre Usability Report 24

LaFavre Usability Report 25

LaFavre Usability Report 26

Use is logical/straightforwardIntuitive and consistent menu banner, back to top

LaFavre Usability Report 27

LaFavre Usability Report 28

4. Visuals stimulate emotional involvement

5. Motivates users to move through information

LaFavre Usability Report 29

Seek info Recommendation

LaFavre Usability Report 30

b. Minor Issue Menu Larger:

LaFavre Usability Report 31

LaFavre Usability Report 32

LaFavre Usability Report 33

Seek info Recommendation

LaFavre Usability Report 34

LaFavre Usability Report 35

LaFavre Usability Report 36

LaFavre Usability Report 37

I. Heuristics and sub-items II. Operationalized metrics

1. Design for diverse users

Product Metric: Use is logical and straightforward.

Behavioral Metric: User (type) not stumped by the design.

Product Metric: Experience customizable for different users;

LaFavre Usability Report 38

2. Design for usability

LaFavre Usability Report 39

3. Make users feel welcome

Product Metric: Design and experience feels welcoming and

Product Metric: User is engaged by visuals, not distracted by

Survey Metric: User describes visuals as enhancing the experience

Product Metric: Sounds are used constructively.

distracted or put off by sounds.

Survey Metric: User describes sounds as helpful, useful, or

Product Metric: Visual elements stimulate user emotional

experience, or in terms of appropriate emotional response.

LaFavre Usability Report 40

4. Set the context

Product Metric: Experience has appropriate precursor activities