BENCHMARKING

101
Introduction 3

What is UX benchmarking? 5
Why UX benchmarking is so valuable 9

What is competitive UX benchmarking? 11

Why competitive UX benchmarking is so valuable 14

What can you UX benchmark? 15

When and how often should you benchmark? 17

How to conduct UX benchmarking 19

1) Identify your areas of focus 20
2) Identify the metrics to capture 21
3) Verify your tasks 24
4) Build your script and study 25
5) Include post-study questions 26
6) Soft launch your study 26
7) Choose your sample and sample size 27
8) Analyze results 29

How to get executive buy-in for benchmarking? 31

Scorecarding 33
UserZoom’s qxScore 34

Conclusion 39
Introduction
Anyone who works in user research knows their work can uncover an array of
user insights which can inform business and design decisions, and ultimately
help create a better experience for users. Hooray! You’re the best. Pat on the
back for you and your significant UX spirit animal. Ours is a great-horned owl.

However, it can be difficult to quantify anecdotal experience, especially if

you’re mainly running moderated tests and getting qualitative insight (i.e.
people telling you why your product is rubbish or great).

But how can you communicate this data effectively and efficiently to
stakeholders? How do you tie an ‘improved user experience’ to improvements
in your company’s overall key performance indicators (KPIs)? You worked hard
for those insights, dammit, and you should be rewarded!

And even more importantly, how can you measure user experience
improvements over time and therefore justify more testing to make your
product even more awesome and user-friendly?

One way is through UX benchmarking!!!

(You probably guessed this already, because you looked at the front cover.)

Quantifying the impact of UX research through benchmarking can help prove

the value of research, tie user experience to business objectives and engage
stakeholders.

Phew, problem solved. Now let’s all clock off early and get a strawberry
milkshake.

3
BUT WAIT!!! How can you benchmark experience? Aren’t experiences
subjective? How can you place a value on the subjective? This is the stuff that
made Descartes suffer from ‘exploding head syndrome’.

Don’t worry - yes, subjectivity is a challenge, and as a result so is identifying

and tracking meaningful metrics. But if you implement a valid benchmarking
strategy, and select the right metrics to track, you’ll be well on your way to
benchmarking glory and your head will remain relatively intact.

What to know how it all works? Step right this way…

4
What is UX
benchmarking?
What is UX benchmarking?

Benchmarking creates a baseline for understanding the current user

experience on your website, app or any digital product.

And the sooner you get started, the sooner you’ll be able to measure
and improve the user experience. You may also hear this referred to as
longitudinal benchmarking - which is basically measuring over time.

If you need a simple example of this, you’ve come to right ebook author.

When you start benchmarking you need to create a baseline. A starting point.
Otherwise you’ll have nothing to measure against.

You won’t know how fast you can run 100 metres until you actually put on
a pair of running shoes and persuade a confused friend to measure the
distance and time for you.

Ready, set, go!

Now you have a baseline! Well done. Let’s put that in our little
scorecard here…

MY 100 METRE RUN:

Attempt # Time

1 Made it 75 metres before stopping for a chunky kit-kat and a lie-down

But it’s a hollow victory, as you have nothing else to compare it to. As far as
you’re aware the world record for running 100 metres is stopping 3/4 of the
way for a chunky kit-kat and having a lie down.

What if instead you decided to make a few changes to your lifestyle that could
directly influence your 100 metre run time? Quit the kit-kats, practice with a
few smaller runs, actually warm-up first. Then you could run the 100 metres
again, make a note of your time, add it to your scorecard then you can see
exactly how your changes have directly influenced your efforts.

6
What is UX benchmarking?

MY 100 METRE RUN:

Attempt # Time

1 Made it 75 metres before stopping for a chunky kit-kat and a lie-down

2 Made it 87.5 metres but was significantly less crampy afterwards

Then you carry on doing this over time, noting down your results,
monitoring your progress and making changes based on your increasing (or
decreasing) times.

So boom, this is Longitudinal Benchmarking. And with this kind of

benchmarking you’ll be busting out 15 second 100 metre-long sprints sooner
than you think!

As for UX benchmarking, the principle is the same.

You pick a product, feature or tool you want to measure and improve on your
website or app - let’s say it’s the checkout section on your retail site. You then
run some testing to get some baseline usability metrics.

We’ll discuss methods of testing and metrics in more depth from page 19 -
but for this example let’s just say we ran a basic usability test where we timed
how long it took a group of 20 test participants to find a specific item and
purchase it through checkout. We’ll also look at average number of pages
they visited, and whether the task was successful.

You’d then input these figures in a table and then you will have your baseline
data to begin tracking this particular user experience over time.

1st test (date) 2nd test (date) 3rd test (date) 4th test (date)

Task success 94%

Average time on task 1.5 mins

Average number of page views 4 pages

7
What is UX benchmarking?

Perhaps within this initial test you got some in-the-moment feedback from
your participants where they mention feeling “overwhelmed by so many
options during the checkout.” You could then make some changes to the
user experience based on this research (for example, simplifying or removing
some options), then you run another round of testing and see how you
perform against your baseline.

REMEMBER: keep all the details of the test study the same for each round
of testing otherwise it won’t be an accurate benchmark. For instance - the
item being purchased, the device/browser being used, the number of test
participants, the type of task and any questions you might ask.

You then start to build a picture like this (which is essentially a scorecard, and
we’ll talk more about that on page 33)...

1st test (date) 2nd test (date) 3rd test (date) 4th test (date)

Task success 94% 95% 100%

Average time on task 1.5 mins 2 mins 1.5 mins

Average number of page views 4 pages 5 pages 4 pages

As you make changes and run each round of testing, you’ll begin to
see problem areas for improvement. You’ll then see exactly how your
improvements are affecting these metrics, and if you tie these metrics
to wider company KPIs (improve revenue, improve ease of use, improve
customer satisfaction) your user research can be seen as an invaluable tool in
the organization.

8
What is UX benchmarking?

Why UX benchmarking
is so valuable
Why would you undertake UX benchmarking in the first place? Here’s our
executive-friendly cut-out-keep guide to why UX benchmarking is so rad.

Engage stakeholders and secure buy-in. While video-based qualitative

insights are compelling, you may struggle to get buy-in from more data-driven
stakeholders. Therefore you may get increased stakeholder buy-in when
you run studies specifically with benchmarking in mind, because the higher
sample sizes increase statistical significance, and this kind of robust data
brings much more confidence.

Identify product strengths and weaknesses. Benchmarking across your

key customer and user journeys on your digital products enables you to
understand specifically where your strengths or weaknesses lie. This can
help you to work on reinforcing your competitive advantages with further
developments or changes to your messaging.

Inform better design decisions. Making changes to a digital product without

measuring the outcomes quantitatively (i.e. numerically) means that you are
operating in a knowledge void. If changes are made without the context of
knowing if they’re making an experience better or worse, changes become
reactive and based on gut instinct. Dirty Harry operated purely on gut instinct,
and although he got results, you’re not Dirty Harry. I’m sorry you had to hear
it from this downloadable UX ebook first.

Validate designs against business objectives. Benchmarking key user

journeys on your digital products prior to and after a significant change will
enable you demonstrate the impact of user research on the most important
business KPIs, such as conversion rates, basket abandonment or even the
bottom line.

Demonstrate the positive impact of your efforts. Adding quantitative,

measurable metrics to your UX research not only helps support your findings
in the moment, it will also help you understand trends and provide context to
design and business decisions in the longer term.

9
What is UX benchmarking?

Of course if you REALLY want to knock your stakeholders’ tiny weird trainer
socks off, you could appeal to their more competitive nature and show them
EXACTLY how you compare to your fiercest rivals.

Which brings us neatly to our next chapter…

10
What is
competitive UX
benchmarking?
What is competitive UX benchmarking?

A huge part of optimizing the user experience of your website is knowing how
it performs in comparison to your competitors.

It’s like the 100 metre sprint example from page 6, sure you can measure and
influence improvements and declines in your own performance, and that’s
good for you! Pats on the back for you and your patronus once again (great
horned owl is taken, hands off) but you’re not going to win any trophies if
you’re not measuring against your opponents.

Now I could wrap up this chapter right now, as that was an utterly watertight
metaphor - but I remember that the free market doesn’t operate on a trophy
based system, so I guess I’ll have to explain a bit more.

Competitive UX benchmarking is a way to compare some metrics of

your product with your direct (or sometimes indirect) competitors.

This is great because your boss may often say to you, “Ooooh I like this fancy-
pants new feature that COMPETITOR X has launched, let’s copy it” and even
though your own gut might be saying, “No bad idea, bad idea, wrong!!!” and
“When’s lunch?” you don’t really have a concrete argument against copying
the feature. So you just steal it, because they’re the boss and you haven’t
discovered the joys of competitive UX benchmarking yet and breaking for
lunch feels increasingly like a foreign country.

But thankfully now you’re reading this 101 ebook, you can say, “Good idea
BOSS X, I like what you’re suggesting, but before we spend too much money, let’s
build a prototype of the feature and UX benchmark it against our competitors,
then we’ll see if our users actually like it or not?”

That way you’ve either saved the company some money and proved yourself
very wise, or you’ve backed up your boss’s idea with concrete data and
proved them as very wise, and hopefully in both cases boosted the value of
your user research efforts. There are many more good reasons for doing
competitive benchmarking, and you’ll find a fuller list on page 14

12
What is competitive UX benchmarking?

Comparing a prototype or concept with a competitor’s similar feature isn’t the

only thing you can do. With competitive benchmarking you can test anything
that’s live on your site and compare with any number of competitors or
non-competitors (after all, you may find inspiration from outside your own
industry).

You could compare the usability of your checkout against another, or you
could compare softer metrics such as asking participants, “What is your
perception of this company?” before and after the task.

You could conduct competitive benchmark as one snapshot in time, or

perform longitudinal competitive benchmarking, where you monitor the
performance of your product against your competitors over a longer
period of time.

Just remember that if you’re comparing a particular tool or functionality - such

as adding a lawnmower to a shopping basket - you have to be confident that
you and your competitors will continue selling lawnmowers over the length
of your study period. After all, it might just be a phase that FOREVER 21 is
going through.

Also bear in mind, the more competitors you benchmark against, the more
expensive, time consuming and unwieldy the data becomes. So try to keep
the group small and meaningful.

13
What is competitive UX benchmarking?

Why competitive UX benchmarking

is so valuable
Why you would undertake competitive UX benchmarking in the first place?
Here’s another executive-friendly cut-out-and-keep guide to why it’s so rad.

It provides measures of how you’re performing directly related to your

competitors. This creates one heck of a compelling argument when you
can say to your stakeholders and executives, “This is how we’re performing
directly against our competitors and this is what we can (or have done) to
beat them!”

You’re getting an entire industry view. This is really helpful, as you can be
sure your customers aren’t just your customers (I know, heartbreakers) but
also your competitors’ customers. Therefore their expectations of your site
and its UX are shaped by this.

Identifying best in class examples. Sometimes this might be you,

sometimes it’s not. Either way, it would be good to know, so you can either
celebrate or optimise your stuff so you can meet and exceed these SO
CALLED best in class examples.

Understand if a competitor’s new feature is worth replicating and

iterating. Sure it may look shiny and fancy, but maybe it’s also a huge waste
of time. Copying competitor features is a favourite topic of executives, so
anticipating these conversations with this type of research can help you save
time, money and a few heated discussions.

14
What can
you UX
benchmark?
What can you UX benchmark?

So you know you want to benchmark something, but what exactly can you
benchmark?

ŠŠ Your website, app or any other digital product

ŠŠ Any of the above versus those of your competitors’
ŠŠ Different iterations of prototypes through development
ŠŠ Different versions of your live website or app
ŠŠ A prototype of a feature versus a live feature of a competitor
ŠŠ User journeys from one point on your website to another

Or a combination of the above – basically anywhere your users and

customers are accessing your site or product online can and should be UX
benchmarked.

This applies to external sites as well as internal sites or intranets, because you
want to ensure your online properties are as streamlined and optimized for
your internal users as they are for your customers. Just imagine if Amazon
had the same UX as your employee expenses portal? *Shudder*

16
When and
how often
should you
benchmark?
When and how often should you benchmark?

If you’re benchmarking just your own product over time, you can can begin
any time from concept, to live working product, to every iteration that follows.

As for how often? We work with many different customers who have different
intervals in which they run their benchmarking. Sometimes it’s annually,
sometime’s biannually and sometimes quarterly or even monthly. It really
depends on your development cycle and how often you’re rolling out
design changes.

We would also recommend testing before or after launching a major update

or redesign that may not fall within your regular intervals, as you don’t want
to find that something isn’t working six months after launch, because that’s
when your timetable says you’re due another round of tests.

Now that you know the concepts behind longitudinal and competitive UX
benchmarking and appreciate the value they can bring, you’ll need some
practical guidance on how to actually run benchmarking.

To unlock this guidance, one of you needs to send the following trading cards
to our San Diego head office and await further instructions: Marvel 1994
Fleer Ultra #15, 28, 70, 82, 109, 115, 129, 138 & 140 and the 94-95 Topps
Basketball Card of Muggsy Bogues (#69).

UPDATE: thank you, all received in near-mint condition. Please proceed to the
next stage of learning

18
How to
conduct UX
benchmarking
How to conduct UX benchmarking

1) Identify your areas of focus

Before you begin, you should identify the areas of focus for your research.
Do you want to monitor and improve a user journey or is it a specific task or
functionality you want to look at? OR do you want to gauge the overall user
experience of your site?

When setting your research goals you need to take your time. Designing
your first benchmarking study is a critical step and not one to be rushed
through. You could use past research that you’ve already conducted. Or
you’re probably already aware of some known problems, whether through
customer feedback or just general frustrations you’ve discussed internally.
These could be addressed first.

It’s also important to stay focused. It’s really tempting to ask users
absolutely everything just because you have their attention, but all your
questions should be tied into helping you answer your top objectives.

It’s important that you have a plan for every piece of data you collect,
otherwise you’ll just waste time and effort.

For most companies, they typically track their most important tasks,
for instance:

ŠŠ How easy or difficult it is to purchase a product

ŠŠ How users respond to a redesigned home page
ŠŠ Is it easier or more difficult for customers to use a new
navigation scheme?

It’s not always just about the tasks and their respective usability issues -
you also have an opportunity here to understand the needs of your
customers, as well as their expectations - the more qualitative, opinion
based, emotional stuff. Therefore we recommend adding a small set of survey
questions to help monitor these changes over time as well.

20
How to conduct UX benchmarking

2) Identify the metrics to capture

After identifying your area of focus, it’s important to choose metrics that
reflect your objectives and the overall KPIs of your business.

What’s the difference between KPIs and metrics?

KPIs (key performance indicators) reflect the overall goals of your business -
such as revenue growth, retention, or increase user numbers. Metrics are the
all measurements that go towards quantifying these higher goals.

Typical metrics you could capture include these task-level behavioural

measurements (“behavioural” basically means ‘what they did’):

Task Success
Task Time
Pageviews
Problems and Frustrations
Abandonment Rate
Error Rate
Number of Clicks

21
How to conduct UX benchmarking

In addition you might want to capture attitudinal metrics (this means ‘what
they said’):

Loyalty using scores such as SUS or NPS (more information over the page)
Usability or ease of use
Credibility taking things like trust, value and consideration into account
Appearance “oooooh pretty!” or “OW MY EYES!!!” etc.

These metrics can also be wrapped up in an overall metric known as

SUPR-Q, which is an eight item questionnaire for measuring the overall

quality of the website user experience. You can read details about SUPR-Q
at www.suprq.com.

It may be worth capturing other attitudinal-style questions pre and post

task, to understand the effect of your digital experience on the overall brand
perception.

22
How to conduct UX benchmarking

GAJ (Guide to Abbreviated Jargon):

Your cut-out-and-keep guide to the common ways to capture attitudes in a

quantifiable way.

NPS: Net Promoter Score. This is a survey you can include at the end of
your UX tests. NPS helps you measure loyalty based on one direct question:
“How likely on a scale of 0 - 10 would you recommend this company/product/
service/experience to a friend or colleague?” The actual score is calculated by
subtracting the percentage of customers who responded with a score of 0 to
6 from the percentage of customers who scored you 9 or 10.

SUS: System Usability Scale. For every usability test carried out, users
complete a short questionnaire and a score is derived from that. It’s on a
Likert scale (a 5 or 7 point scale that offers a range of options, from one
extreme attitude to the opposite, with a neutral attitude at the midpoint). This
helps ascribe a quantitative (or numerical) value to qualitative opinions.

SUPR-Q: Standardized User Experience Percentile Rank Questionnaire.

This is an 8 item questionnaire for measuring the quality of the website
user experience, providing measures of usability, credibility, loyalty and
appearance.

CSAT: Customer Satisfaction Score. This again measures customer

satisfaction, but doesn’t have the strict question limit parameters of NPS or
SUPR-Q as you can ask anything from one single question to a full length
survey. Results are measured as a percentage. Pro: unlimited customisation.
Con: the people who actually take the time to fill in a full-length survey are
only likely to either love or hate your product.

23
How to conduct UX benchmarking

3) Verify your tasks

As you build your study, you should carefully test each task to make sure that
the metrics can be captured consistently.

When competitive benchmarking, bear in mind that another company’s

website can be prone to change without you expecting it, so you should
carefully monitor both while building your study and while the study is live to
ensure your task still makes sense.

When you set up any usability test it’s necessary to not only define the tasks,
but determine what success or failure looks like. This means you’ll need to
figure out how you validate the success of your task.

ŠŠ Validation by URL - this can be used if task success is dependent

upon reaching a specific page or finding content available on a specific
page. When conducting usability testing, study participants are asked
to perform a series of tasks on a website. They are presented with
instructions for each task and are taken to a specific start URL to begin
the task. Validation can occur on the last page of the process. If people
can navigate to that specific pre-programmed page URL, they will be
marked as a success.

ŠŠ Validation by question - if you’re testing a trading card website and

you ask participants to find current financing incentives for a Muggsy
Bogues rookie card, then task success can be validated by the following
question, “What are the current financing offers and incentives available
for a Muggsy Bogues rookie card in the 94404 zip code?” Participants
are presented with a list of pre-defined answer choices. Those who
successfully complete the task should be able to answer the validation
question correctly.

24
How to conduct UX benchmarking

4) Build your script and study

Once you’ve decided on the digital products you want to test and the metrics
you want to capture, you can start writing your study script.

For benchmark studies, study scripts should be consistent, easy to follow,

and avoid any leading questions that could induce bias.

To reduce the potential for inaccuracy in your results, the questions and
tasks used in your first baseline study should be the same for every
subsequent study. If you’re benchmarking competitively, your tasks should
be achievable across any of your competitor’s properties.

For example, if you’re testing a user journey across fashion retailers where
a user is asked to find men’s pre-distressed jeans that cost between $180
and $240, ensure that all the websites you test actually sell jeans that fit that
criteria. In a scenario like this, if you’re looking to carry out regular, ongoing
benchmarks, it may be sensible to choose popular user journeys that are
unlikely to disappear from your competitors’ sites in the future. Therefore you
probably shouldn’t be as specific as shopping for $200 ripped jeans.

Any tasks you include in your study should be as simple as possible

and feature a specific objective. Tasks such as “find a pair of women’s blue
jeans” and “find your nearest store” are good examples of specific, actionable
tasks that mirror real life scenarios without giving them any guide as to how
to complete the task.

However, make sure you provide an end point for the task. For example: “Find
a pair of jeans you would buy and stop on the product page for those jeans”
or “Find a pair of jeans and stop after you add them to your shopping cart.”
If you don’t provide an end point, the time on task will not be an ‘apples-to-
apples’ comparison across participants.

25
How to conduct UX benchmarking

5) Include post-study questions

Your benchmark study should be supplemented with questions that help you
further quantify the experience and enrich your data.

UserZoom automatically collects a range of quantitative behavioral data, such

as time on task and number of clicks, and supplementing this with attitudinal
data can provide you with a full view of the user experience.

Such questions can include rating scale questions - such as those mentioned
earlier in the NPS or SUS scores - and these will capture Ease of Use or
Satisfaction. You can also ask comprehension questions that validate whether
users understood the information presented to them.

6) Soft launch your study

Benchmark studies deliver increasing levels of value as you run them again
and again over time. That’s why it’s particularly important to QA (quality
assure) your study and run several dry runs with a small sample of users
before your first benchmark study.

This minimises the risk of needing to change your study script or structure in
the future, and gives you the assurance that you will be comparing apples to
apples. Or if running tests on a non-grocery site, pick your own metaphor.

26
How to conduct UX benchmarking

7) C
 hoose your sample and
sample size

Your research is only as good as your participants, so there are a few

things to consider when sourcing users to carry out your tests.

In most cases, you’ll want to choose participants who match your target
market. Consider basic demographics (like age, location and income) as well
as criteria like whether they are already customers, how often they shop
online, or other relevant factors.

However if you’re trying to understand how well your site is set up for
customer acquisition, then your existing customers aren’t the best people to
give you feedback, because they’ve already bought into your company, service
and its products.

Not all people are equally qualified to give you valuable feedback, so it’s
really important in this case to develop a good screener question.

A screener is an opportunity for you to have bit more control over who carries
out your test before they begin. This will also help you filter out anybody who
wouldn’t necessarily be right for it.

For instance, if you’re testing the navigation of a website that has a sole
audience of civil engineers and all the wording on the site is only relevant to
that industry, there’s little point in recruiting a load of people who literally just
this second had to Google the term ‘civil engineers’.

Therefore you’ll want to find out what level of knowledge a participant has
around civil engineering terms. ‘Blind’ screener questions are best for this,
as these aren’t ‘yes/no’ questions, which prospective participants can easily
guess the ‘correct’ answer to in order to make it through the screening
process. Instead they ask more specific questions and provide multiple
options. For instance: “What civil engineering qualification do you have?” then
provide a list of courses and qualifications.

27
How to conduct UX benchmarking

How many participants do I need?

Benchmarking studies are useful for showing overall trends and surfacing
specific, actionable insights.

Therefore it’s important to test with high quality, relevant participants, and
it’s also important to test with a significant number, so that you can have
confidence in your data. We’d recommend at least 50 participants. If you
start going beyond 100+ you’ll start seeing diminishing returns, and your
budget and time stretched needlessly.

Maintain your sample size

While you don’t have to run your studies with the same participants every
time, you should use the same number of participants. If you establish your
baseline study with 50 participants, then each subsequent study should also
use 50 participants.

If you’re running several benchmark studies over time, it isn’t necessary

to test with the same people every time, in fact that would likely be
counterproductive to good research. However, it’s important to test with
the same number of participants each time. This provides you with
comparable data sets so you’ll have the same level of confidence in your
results each time.

The validity of the data as you measure changes over time is only
as good as the consistency you use. And that means using your same
screener question(s), the same sample size, the same criteria for your
panelists and the same proportions of demographics.

Keeping proportions across competitors is important. You don’t want a much

higher number of millennial participants testing competitor A compared to
competitor B. This is sometimes overlooked but is incredibly important to
help keep your data relevant.

28
How to conduct UX benchmarking

8) Analyze results

Here’s an example of the results of a competitive UX benchmark that we ran

recently on four banking websites.

Bank of America Chase Citibank Wells Fargo

Key Performance Indicators
n=50 n=50 n=50 n=50

Task Success 64% 60% 72% 70%

Average Time on Task 2.59 min 2.75 min 4.31 min 2.37 min

Average # of Page Views 4.1 pages 4.3 pages 7.6 pages 3.8 pages

Did not encounter problems or frustrations 44% 62% 34% 38%

Provided just the right amount of information 64% 84% 50% 64%

= highest proportion / fastest time / lowest # of pages = lowest proportion / slowest time / highest # of pages

This overview table shows task success, average time on task, number
of pageviews, number of people who didn’t encounter any problems or
frustrations and whether each bank provided just the right amount of
information - all of this data helps us understand how easy or difficult it was
for users to discover information about various bank accounts on offer.

Citibank had the highest success rate at 72% success rate, which on the
surface is great, but if you look a little deeper, they have the lowest score
on all the other metrics. So clearly task success alone does not tell the
complete story.

Chase on the other hand had the the lowest success rate at 60%, but their
time on task, average pageviews and number of users who did not encounter
any problems or frustrations, all performed significantly better than Citibank.

29
How to conduct UX benchmarking

You could then argue that Chase actually has the better overall user
experience, despite the lower success rate. But you would only surface this
information by balancing these attitudinal and behavioural metrics, and
taking time to study all the data.

The success of your benchmarking and the ease of your data analysis relies
on good study design and having a plan for exactly what you’re going to do
with the data you surface.

Read between the lines

If you’re planning on conducting several rounds of competitive benchmarking,

it’s important to look out for the relative changes in results, not just the
absolute change in your score. This is especially true if you’re comparing with
competitors of a different scale or brand reputation than yours.

If you have a high level of market share, and users are familiar with
your website, then your absolute performance against a new up-and-
coming competitor who is less well known may not be the most relevant
thing to note.

Comparing your overall experience with that of your competitors will be an

interesting exercise, and may provide the type of engaging content that gets
stakeholders interested in research.

However, the insights that provide you the direction and pointers on where
to improve can be found within your task-level tests. Analysing relative
differences in time on task, ease of use, and task success for different tasks
will provide the context of where to investigate further.

30
How to get
executive
buy-in for
benchmarking?
How can you talk to your executives about the importance of benchmarking?

We love the idea of ‘democratizing UX’ throughout your organization. It’s

one of the key trends of 2019. As demand for user research increases, but
budgets remain the same, UX leaders will have to find a way to spread user
research education, share results and win stakeholder buy-in in order to
remain user-focused AND stay ahead of the competition.

It’s a tall order, but there are many ways to persuade the people who control
your budgets to invest in UX benchmarking.

Firstly, there’s the cut-out-and-keep guide to the value of benchmarking from

page 9 which we hope you’re still clutching in your hands.

If you don’t have the budget for full-blown longitudinal benchmarking, you
could run a small competitive benchmark, where you just use a handful
of participants to run a couple of quick, top-level tasks to show how you
compare to competition in a specific known area that appeals to your
executives. Then you show them this simple data and make a case for
running more long-term, robust studies.

Here are a few more ideas for sharing your work throughout the organization...

Share early planning - one way to begin is by starting early and

communicating with colleagues what you’re planning to benchmark in one of
your daily stand-ups. Ask about which products or services have been tested
before or collate suggestions of what to test. Capturing the right questions
means you’re capturing the insights your stakeholders are more interested in,
which makes sharing the research more purposeful and engaging.

Involve other teams when running research - you can involve other teams
in the research as it is happening. There’s nothing more motivational than
seeing an actual human being struggle through a usability test on your own
website. Stakeholders don’t even need to be in the same room as they see
this, as you can run remote moderated tests from any location.

Share user insight summaries - you can share your user research successes
with a wider audience via email, or another team messaging platform. You
don’t have to dive into the complexities, instead share quick wins, key quotes
and insights.

32
How can you talk to your executives about the importance of benchmarking?

By sharing your insights with as many people and teams as possible, you
can help democratise user research throughout your company and create
advocates for UX testing.

Of course, actually sharing your work in a simplified, easy to understand form

is half the battle. Especially with executives who only have a few minutes
available between meetings, and therefore don’t want to wade through
endless pages of data. And you probably don’t have the time or inclination to
do that either.

One way to simplify this data is by using scorecards.

Scorecarding

UX researchers need to be data-driven, but keeping a handle on that data can

be difficult. With benchmarking you could be balancing a number of different
behavioural and attitudinal metrics, and if you’re looking at competitors too -
these data-sets can easily be quadrupled in number.

A useful tool for collecting and tracking data is a UX scorecard. This is a simple
table where you track your chosen metrics after every study. Once you set up
a scorecard, you’ll get into the habit of entering data, so you always know how
you’re performing in any given point in your product’s development.

33
How can you talk to your executives about the importance of benchmarking?

UserZoom’s qxScore

At UserZoom we have our own UX scorecard program, called the qxScore.

This is an experience score that combines various measurements, collecting

both behavioural data (such as task success) and attitudinal data (such as
ease of use, trust and appearance) - the purpose of this is to create a single
benchmarking score for your product.

qxScore

BEHAVIOR ATTITUDE
What they do What they say, feel

OVERALL TASK SUCCESS

Task 1 success % Appearance

Task 2 success % Ease of Use
Task 3 success % Loyalty NPS
Task N success % Trust

34
How can you talk to your executives about the importance of benchmarking?

This single UX score is a simple, clear and persuasive tool for communicating
user research results to stakeholders, and should help with getting
future buy-in.

Once you’ve entered your results into our UX scorecard calculator, we’ll
generate a qxScorecard that looks like this…

If you’d like to know more about how UserZoom can help measure, manage
and action your user research data, get in touch!

35
Dana Bishop’s
10 expert
benchmarking
tips
Dana Bishop’s 10 expert benchmarking tips

Here at UserZoom we have a secret weapon when it comes to UX

benchmarking in the form of Dana Bishop. Dana is one of our Senior
Directors of UX Research, whose 25 years of experience has been invaluable
in leading countless impactful UX benchmark studies, so I thought I’d leave
you with Dana’s own personal 10 tips for benchmarking success!

01. Stay focused. Be clear and specific when you define your
benchmarking goals. Don’t set too many goals or ask questions that are
not directly tied back to helping answer your top questions.

02. Get started measuring today! Your first benchmark you run will
create a baseline for understanding the current user experience on
your website or app, and from which you can measure against in
the future.

03. Once is not enough! Keep measuring to track how the experience
changes over time—both at regular intervals, and after rolling out
major updates or redesigns.

04. Keep in step with your users. Include survey questions to help
keep abreast of changes in feature prioritization, usage, needs, and
expectations.

05. The greatest insights are derived from real users. But, you need
more than just real users; you need to measure feedback from your
users and/or intended target audience on an ongoing basis.

06. Consistency is imperative to yield reliable comparative and

longitudinal data. Use the same screener and keep proportion even
across all competitors each time you run your benchmarking study.

37
Dana Bishop’s 10 expert benchmarking tips

07. Don’t miss out on the many benefits of expanding your testing
universe. Competitive analysis offers a way to interpret your usability
standing, feature set, and more, in your industry. And identify best-in-
class examples to emulate.

08. Carefully consider which competitors you use. Include the ones you
look at and discuss the most, the ones you feel you’re falling behind,
and the ones that you think you can learn the most from.

09. Plan accordingly! Because of the larger sample sizes in benchmark

studies, it can require more time in the field.

10. Work smarter, not harder. Know what you will do with each piece of
data you collect. If you took the time up front to carefully identify the
questions, concerns, areas of interest, and purpose of the research and
designed your study with data analysis in mind, then it will be a breeze!

38
Conclusion
UX Benchmarking is a really powerful exercise as it enables you to quantify
the impact of your user research and subsequent design and development
changes, and allows you to tie these into the changing user experience over
time.

This can be an important step in establishing a more concrete ROI of user

research, and the impact of you and your team.

In addition, competitive benchmarking allows you to survey your product’s

landscape from the perspective of users, and identify areas of strength and
weakness within your own product. This can be incredibly persuasive when
looking for organizational buy-in.

No matter the type of benchmarking you conduct, if it’s conducted properly

you’ll be able to establish tangible metrics around the user experience of your
key journeys, and you’ll have more insight to deliver to the stakeholders and
executives within your business.

NOW it’s time to clock off and grab that strawberry milkshake.

39
Why conduct UX benchmarking with UserZoom?

Why conduct UX benchmarking

with UserZoom?

ŠŠ Fast and cost effective: Convince stakeholders by conducting large-

scale testing quickly and cost-effectively with UserZoom’s remote
unmoderated testing methodology – no moderation or expensive lab
equipment needed.

ŠŠ Study templates: When you create your first Benchmark study within
UserZoom you can use it as a template for all your future benchmark
studies, allowing you to quickly and easily test iterations.

ŠŠ Get the complete picture with both types of data: Get actionable
insights by collecting both quantitative and qualitative data in a single
study to better understand your users’ experiences.

ŠŠ Quick results: As soon as your study is active you can begin collecting
results in real time, allowing you to see feedback as participants
complete your study.

ŠŠ Recruit your target audience: Whether it’s testing participants who are
geographically dispersed or whether you have a specific target audience
in mind, UserZoom’s flexible recruiting options allows you to recruit the
right participants into your studies.

ŠŠ Easy survey building. UserZoom’s survey engine is capable of handling

all of the standardized scoring and many types of measurements via
many question types, including customizable matrix questions and
NPS scoring. Plus you can easily save these question for future use as
templates.

40
Why conduct UX benchmarking with UserZoom?

ŠŠ Benchmarking all devices: Whether you want to Benchmark your

website, mobile site, mobile app, or prototypes for all the above,
UserZoom’s platform has you covered.

ŠŠ Benchmark against competitors: With UserZoom you can test

your competitor’s websites, allowing for a direct comparison of key
usability metrics.

UX benchmarking is about letting real users help you reveal pain points on
your own site and competitors’ sites. When you benchmark KPIs on your
site vs. competitors over time, you keep the competition in view and benefit
from data and inspirations that can drive future enhancements to your site –
ultimately leading to an industry-leading user experience.

41
 Want to learn
more about
UserZoom?
Let us show you how we’ve helped hundreds of
companies unlock competitive user insights with our
user experience testing platform.

Schedule a call

USA: +1 866-599-1550
UK: +44 (0)203 948 1360

San Jose Denver London Manchester Madrid Barcelona

www.userzoom.com marketing@userzoom.com 866-599-1550
THE UX INSIGHTS COMPANY HQ: 10 Almaden Blvd #250 San Jose, CA 95113 USA