1

2
Getting feedback

From observation to feedback

The Third Part of the Usability Cycle
•Gathering information(i.e. user types, market needs) >
•Design(i.e. applying design principles, using personas and
use cases) >
•Gathering feedback(i.e. removing your ego and make the
best application you can for your users)
Interface Hall of Fame or Shame?

Contributed by Yasmin Sewak

Interface Hall of Fame or Shame?

http://web.mit.edu
Contributed by John Warwick
Interface Hall of Fame or Shame?

Start to submit these to my email with the subject “Interface Hall of Fame or Shame”
and I’ll pick one or more for each lecture to discuss.
Iterative Design Process
•Ideo’s human centred design process:
http://www.designkit.org/resources/1
Getting feedback
Types of feedback
Feedback types?
• (Paper) Prototypes
• Surveys
• Interviews
• Usability tests

• Other (see Usability Engineering)

• Focus groups
• Log analysis
Setting up for feedback
Receiving criticism is not fun
•Nobody enjoys getting feedback.
•However it is essential. In most of the your tutorials you
were constantly asked to review your peers’ work.
•It is humbling to know that you will not be able to hit a
home run right off the bat.
Embrace feedback
•It is your best tool.
•As programmers we are often enamoured with
programming challenges. This is fine, we are not telling
you to be enamoured with users.
• However you need to ensure that your application is on
track in the design and business sense by involving your
users.
Knowing when to listen to your user
•Users’ feedback is not the rule.
•The key is to know when feedback is useful or relevant.
•If there is one user out of 100 who is adamant that a
scrolling marquee is in vogue right now, you would
probably not take that into consideration.
•Unless you REALLY like a scrolling marquee…
•Which in that case you should really not…
Types of user
•Information Overloader.
–These will be very eager to tell you absolutely everything.
–Even if they are not relevant.
–So you will need to sift through the noise.
Type of user
•The Control Freak
–These users will try to be very pedantic about controls.
–They would also try to run the tests or surveys.
–Or create their own questions.
Type of user
•The Devil’s Advocate
–These will try to find problems wherever they could.
–You will need to see if they are actual problems.
Negativity is not productive
•Users can be negative.
•Well, developers can be negative too!
•Sometimes the best thing to do is to look at what works
and go from there.
Feedback
Leverage on:
•Social Media
•Friends,
•Peers,
•Competitors.
•In fact, talk to ALL of them.
User Research Methods
User
experience
research
methods
What UX methods to use and when?
Requirement stage
• Who are the Users and Customers?
• Survey
• Persona
• Contextual Inquiry
• Stakeholder Interviews
• Competitive Analysis
• Quality Function Deployment
• What are users trying to do?
• Task analysis
• Top tasks analysis
http://www.measuringu.com/blog/method-when.php
3-dimensional
framework
•Attitudinal vs.
Behavioral
•Qualitative
(formative) vs.
Quantitative
(summative)
•Context of Use
Formative vs summative

RMIT University©2013 CS&IT 25

Formative vs summative
•Do Formative Usability Testing at the start of the design
phase, testing with paper-prototypes and similar. Do this to
discover insights and shape the design direction.
•Do Summative Usability Testing during latter half of the
development phase, testing with actual working
prototypes. Do this to determine metrics (time on task,
success rates), which you would then compare test results
from improved versions against.
•Usability test protocol usually employed
RMIT University©2013 CS&IT 26
Formative vs summative (evaluative)
•Formative usability testing
–beginning stages of the design process
–provides valuable insights of where users have difficulty reaching
their user goals with the technology (website, desktop GUI
design, hardware product) or service.
•Summative usability testing
– a Quality Assurance (QA) type of test
–usually performed later in the development process.
–formal user acceptance testing before the product is released to
the target audience
RMIT University©2013 CS&IT 27
Formative vs summative
Formative --- > Build --- > Summative (evaluative)

Ethnography Ethnography
Interviews Interviews
Surveys Surveys
Cultural Probes Focus Groups
Focus Groups Diary Studies
Diary Studies Experience Sampling Studies
Experience Sampling Studies Interaction Logging
Studying Similar Products
Interaction Logging of Past Product / Early Prototype
Studying Past Product Documentation

RMIT University©2013 CS&IT 28

User Research Methods
Observations
Why can’t we just ask users what they
want?
•“You can observe a lot by just watching” (YogiBerra, MLB
Player / Quote Machine )
“If I had asked my customers what they wanted, they would
have said a faster horse.” (Henry Ford, car inventor)
•So, often it’s better to watch what people do than to only
design solely for what they say
• Especially when what people say != what they do
Ethnography
•In anthropology, ethnography developed as a way to
explore the everyday realities of people living in
small-scale, non-Western societies and to make
understandings of those realities available to others.
• Today, ethnography is applied to the study of people
and social groups in specific settings within large
industrialized societies, such as workplaces, senior
centers, and schools…
(Jeanette Blomberg, Anthropologist, IBM Research, Mark
Burrell, Psychologist, Microsoft Corporation)
3 Principles of Ethnography
•Holistic: tiny details into big picture context; attempt to look at
things broadly—understand context
•Natural settings: Directly observe in the things that you’re trying
to study
•Descriptive: focus on recording behavior—analysis comes later
Look at what users really do
Look at what users really do

http://www.knowledgeproductivity.com/inspiring-ideo-method-cards
Ask users to help

http://www.knowledgeproductivity.com/inspiring-ideo-method-cards
Diary Studies
Diary Study
•Users are requested to record feedback and/or events on
a specific part of a product experience.

•Can be on any medium, paper, voice, videos or audio

recordings.

•Usually lasts a week to two weeks.

•Daily support from the UX personnel running the study.

Diary Study
•Diary is then analysed by a consultant.

•Debriefing at the end of the study with the user.

•Usually lasts a week to two weeks.

•Daily support from the UX personnel running the study.

Diary Study
•Diary is then analysed by a consultant.

•Debriefing at the end of the study with the user.

Diary Study
•Advantages
•Rich data, covers responses over time.
•Context rich data, user experience may be related to
external and internal aspects.

•Disadvantages
•Resource intensive - time/$$
•Self reporting bias.
•Highly reliant on user participation.
Surveys

Challenges
Recap
Surveys
• You fill so many out.
• Tempting to think you know how to write them
• Creating a useful and reliable survey is an art form.
• You not only have to know which questions to ask, but
how to frame the questions using concise and impartial
words.
Survey
• Survey
• Coverage error
• Do you get to everyone using your system
• Sampling error
• If you only talk to a sample, some error
• Nonresponse error
• Survey the correct people, but what if not all
answer?
• Measurement error
• People don’t answer accurately
Measurement error
• Ask factual questions people often answer themselves?
• Generally no problem

• Ask questions people need to think about?

–The way you ask the question could influence the
answer
Measurement error
• Social desirability (answer so as to be viewed favourably)
• Self administered
• Interviewer
• Acquiescence (bend to the “view” of the question)
• “Individuals are more to blame than social conditions for
crime and lawlessness in this country”
• 60% agreed
• “Social conditions are more to blame than individuals for
crime and lawlessness in this country”
• 57% agreed
Surveys

Recommendations
Types of Questions
Open ended questions where you ask a question and leave space for the
user to write freeform answers. This is used if you would like to know more
about specifics that you were not confident of and to explore other aspects
of the problem.

Closed ended questions where you provide the answers and ask the
users to select one or more. Use this if you know for sure what are the
parameters and variables you are looking for, and also if you want to use a
scale like the SUS or other metrics.
Start with a screener
Screeners are important questions to help you target the
correct audience for your surveys. Start by asking if they
used the site/app on that specific platform. Remember
that this assignment requires that you survey current
users and not potential users.
Do not lead the user.
It is not a good idea to project negative or positive sentiments in your
questions.

For instance, instead of:

Trying to navigate between courses in blackboard is confusing, what
do you think?
you can say:
How often do you switch between courses on blackboard? Did you
have any problems switching?
Do not waste time.
Every question should provide answers that would play a
part in your analysis. Ask yourself this for every question.
“what can I do with the data?”
You are not studying the features but how easy it is
to understand and access those features.

Although you are indeed trying to find out if the users are happy with the
app, you will need to be more specific about what you ask in order to
produce a useful and actionable report.
For instance, asking if friends would recommend an app to their friends will
not help you identify the drawbacks of the interface. You are more
interested in the “why”s.
Your questions should be succinct, direct and does
not contain jargons.
Especially not the guidelines you learn in this course. Nobody Not many
know what is affordance or Nielsen’s thingamajig.
Keep to the point, and use natural common language so that your users
does not have to do guess work.
Also maintains their confidence.
If you want to ask for temporal data (how often do
you…), define the timeline (...within the last month).
For instance, instead of:
How often do you visit ABC.xyz? < Not so good
you can say:
In the past 6 months, how often have you visited
ABC.xyz? < Better
Don’t make the user feel tested.

For instance, instead of:

I was very confused. (Agree/disagree)
you can say:
The app made it very confusing to navigate around.
(Agree/disagree)
Users
•How many?
•Not very helpful answer: Depends.
•A little more helpful: What sort of information are you
trying to obtain? How much value would this information
bring?
•Who?
•Depends
•See earlier lectures
Nielsen suggests
How many users needed?
•Using 5 users to find 84% problems with each of three
design iterations
•Using 15 users to find 99% of problems with one design
iteration

(Nielsen)

RMIT University©2013 CS&IT 58

Which is better?
•Is the 5 user assumptions correct?

Faulkner, Laura. “Beyond the Five-User Assumption: Benefits of Increased

Sample Sizes in Usability Testing.” Behavior Research Methods, Instruments, &
Computers 35, no. 3 (August 2003): 379–83.
(Find paper in Blackboard)

RMIT University©2013 CS&IT 59

How many users?

•The bigger issue is actually approaching the right users.

•Have a good mix.
•Balanced users
•Do not test them with fellow developers. You will end up
spending hours debating the merits of using some CSS
API like bootstrap or semantic UI…
Card sorting – How many users?
•For most usability studies, Nielsen recommends testing 5 users
(https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/), see lecture
slides week 5
•But Tullis and Wood’s study on card sorting recommends 20-30
users (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.514.3907)
•Based on Tullis and Wood’s study, Nielsen suggests it should be
15 users (https://www.nngroup.com/articles/card-sorting-how-many-users-to-test/ )
3-dimensional
framework
•Attitudinal vs.
Behavioral
•Qualitative
(formative) vs.
Quantitative
(summative)
•Context of Use
Formative vs summative
•Do Formative Usability Testing at the start of the design
phase, testing with paper-prototypes and similar. Do this to
discover insights and shape the design direction.
•Do Summative Usability Testing during latter half of the
development phase, testing with actual working
prototypes. Do this to determine metrics (time on task,
success rates), which you would then compare test results
from improved versions against.
•Usability test protocol usually employed
RMIT University©2016 CS&IT 63
Formative vs summative (evaluative)
•Formative usability testing
–beginning stages of the design process
–provides valuable insights of where users have difficulty reaching
their user goals with the technology (website, desktop GUI
design, hardware product) or service.
•Summative usability testing
– a Quality Assurance (QA) type of test
–usually performed later in the development process.
–formal user acceptance testing before the product is released to
the target audience
RMIT University©2016 CS&IT 64
Formative vs summative
Formative --- > Build --- > Summative (evaluative)

Ethnography Ethnography
Interviews Interviews
Surveys Surveys
Cultural Probes Focus Groups
Focus Groups Diary Studies
Diary Studies Experience Sampling Studies
Experience Sampling Studies Interaction Logging
Studying Similar Products
Interaction Logging of Past Product / Early Prototype
Studying Past Product Documentation

RMIT University©2016 CS&IT 65