1

Walmart.com Usability Study

By Robert Bialecki Usability Design and Testing May 5, 2011

Table of Contents

Executive Summary.....3 Introduction..4 Target User Analysis....5-6 Participant Demographics6 Methodology7 Results8-13 Recommendations14-16 Appendix..17-25

Executive Summary
I recently conducted a usability test on www.walmart.com using five participants who had no prior experience with the website. For more information about the test, see the Introduction section (p. 4 ) and Methodology section (p. 7 ). Based on the comments of the test participants, I have compiled a list of recommendations for enhancing the usability of www.walmart.com. The recommendations are listed in order of importance, based on the results of the usability test. 1. Decrease the size of the on-screen pop-up that asks if users want a service plan when they add a product to their shopping cart. 2. Make the In-Store Events link easier to find, possibly in the top navigation bar. 3. Turn the New Customer? text in the upper right hand corner of the page into a link. 4. Include an Add to Cart button on the pictures and brief descriptions in the search results for products. 5. Reduce the number of images and ads on the homepage. 6. Turn the Remove link into a button, like the Add to Cart button. For more information about the recommendations, see the Recommendations section (p. 14-16).

Introduction
Wal-Marts website (http://www.walmart.com) was the focus of this usability project. Wal-Mart is one of the largest retailers in the world. The main purposes of its website are to promote and inform consumers about its products, to allow consumers the option of buying products online, and to provide a forum for consumers to offer feedback and have their questions answered. Because of the wide variety of products and services it has to offer, Wal-Marts primary audience is typical American consumers. Their audience comes in both genders and many different ethnicities and age groups. However, due to their reputation of having everyday low prices, Wal-Mart is particularly popular among middle and lower class families. The objectives of this usability test were to evaluate the usefulness and level of difficulty of four common tasks that users of the website might do. Additionally, test participants pointed out aspects of the website that made the tasks more difficult to complete.

Target User Analysis

Persona
Name: Age: Location: Kathy Johnson 35 La Crosse, WI

Occupation: Stay-at-home Mom Status: Married, 2 Children

Kathy Johnson is a 35-year-old stay-at-home mom who lives in La Crosse, Wisconsin. Her husband, Karl, is an accountant at a local business. The couple has twin 10-year-old boys, Tanner and Taylor, and the family lives in a modest suburban house. Kathy has been a loyal Wal-Mart customer her whole life. As a young girl, she loved accompanying her mother on the weekend shopping trip. Today, she recreates those memories by taking her twin boys with her every weekend. As a stay-at-home mom, Kathy leads a very busy life. Between maintaining a tidy household, getting the kids ready for school, and preparing meals for her family, she also often travels to many nearby cities and towns, since her boys are on the traveling soccer team. One thing Kathy has always loved about Wal-Mart is that she can find clothes, groceries, toys, and much more all in one store. Kathy has access to high-speed Internet because her husband must also work at home sometimes, and the children need a computer for their homework. Kathy would estimate she spends about 6 hours per week using the Internet. In her leisure time, Kathy maintains a blog and keeps in touch with friends from high school on Facebook and Twitter. Kathy also loves to shop online, where she often finds better deals than those found in stores. Although she loves to shop both in-stores and online, Kathy is very frugal, and Wal-Marts everyday low prices are what has kept her coming back after all these years. Key Characteristics: Is a busy person Conclusion: She would like to find the information shes looking for as quickly as possible, without having to deal with a lot of clutter. She would be interested in buying WalMart products online She would like to know if theres a

Shops Online Often travels to nearby cities and towns

6 Wal-Mart in her travel destination. She wont have trouble downloading large files. She might like to receive Wal-Mart updates and information on those sites. She would be interesting in knowing about any coupons, discounts, deals, and promotions that Wal-Mart has to offer.

Has access to high-speed Internet Uses social networking sites Likes to save money

Demographics of the Participants in this Usability Study Gender: Ages: 2 male, 3 female 60, 55, 41, 31, 24

Methodology
For this study, I selected five participants who passed the screening process. Participants needed to fit the following profile: Be older than 18 years of age Have more than one year of experience using the Internet Be familiar with Wal-Mart Have no negative opinions about Wal-Mart (the company or its practices) Have never visited Walmart.com The participants were asked to complete four common tasks that new users of the website might do. For the purpose of consistency, each participant in the study took the test in the same makeshift usability lab. The lab consisted of a room with: a desk, a computer with high-speed Internet access, a comfortable chair, and adequate lighting. I monitored the usability test for each participant and took notes on their procedures and thoughts as they completed each task. The tasks were assigned in a different order for each participant so that the participants gradual experience with the website didnt skew the results of the study. Due to resource limitations, the test sessions were not recorded on audio or video. Participants were asked to complete a pre-test questionnaire, post-task questions after each task, and a post-test questionnaire.

Results
Table 1 lists usability problems that participants encountered during the completion of the usability test. In addition, the table lists the number of participants who experienced the problems, participant comments in regard to the problems, and the severity level of the problem. If 4 or 5 of the participants experienced a particular problem, its rated high severity. If 3 participants experienced a particular problem, its rated medium severity. If 1 or 2 participants experienced a problem, its rated low severity.

Table 1 Usability Problems

Usability Problem A pop-up ad for adding a service plan appeared after participants added an item to their cart. Participants clicked Close rather than Continue, which required scrolling down to find. Number of Participants who Experienced Problem Participant Comments Do I want to add a service plan? No. (closes pop-up) Okay so did it add the item to my cart or not? Oh, I guess it must have because the My Cart tab now says 1. The link is too small and inconspicuous. - Participant 2 I probably would have found it [the link on the home page] sooner if there werent so many pictures distracting me. I assumed clicking on New Customer would take me directly to the registration. I thought Sign in was for established customers. Severity

3 out of 3*

High

Participants had trouble finding InStore Events

3 out of 5

Medium

New Customer? on the home page is not a link that allows you to create an account.

3 out of 5

Medium

9 There is no Add to Cart button on the pictures and brief descriptions in the search results for products. Participants had to click on a particular item in order to get the Add to Cart button. There are too many images and ads on the homepage.

3 out of 5

Wheres the Add to Cart button?

Medium

2 out of 5

I dont like how you have to scroll down pretty far to find a lot of the important information at the bottom of the page.

Low

The Remove link is That link doesnt harder to find than the 2 out of 5 Low stand out very well. Add to Cart button. After entering an address to get driving directions to a store, Wheres the written participants saw a 1 out of 5 instructions? Oh, Low map but had to scroll there they are. down to get the written directions. * The service plan pop-up appeared only for users who searched for major appliances and electronics in Task 3.

10 Tables 2 through 5 show the ratings participants gave for the difficulty level of the task, the amount of time it took to complete the task, and the likelihood that they would use each task/feature.

Table 2
Task 1: Create a user account at www.walmart.com. Difficulty Level of Completing This Task
Very Easy Number of participants who gave this rating Somewhat Easy Neither Easy nor Difficult Somewhat Difficult Very Difficult

The Amount of Time it Took to Complete This Task

A Lot Shorter Than Expected Number of participants who gave this rating Very Likely Somewhat Shorter Than Expected About the Amount of Time Expected Somewhat Longer Than Expected A Lot Longer Than Expected

The Likelihood That They Would Use This Task

Somewhat Likely Neither Likely nor Unlikely Somewhat Unlikely Very Unlikely

Number of participants who gave this rating

11

Table 3
Task 2: Get driving directions from your home to the nearest Wal-Mart store. Difficulty Level of Completing This Task
Very Easy Number of participants who gave this rating Somewhat Easy Neither Easy nor Difficult Somewhat Difficult Very Difficult

The Amount of Time it Took to Complete This Task

A Lot Shorter Than Expected Number of participants who gave this rating Somewhat Shorter Than Expected About the Amount of Time Expected Somewhat Longer Than Expected A Lot Longer Than Expected

The Likelihood That They Would Use This Task

Very Likely Somewhat Likely Neither Likely nor Unlikely Somewhat Unlikely Very Unlikely

Number of participants who gave this rating

12

Table 4
Task 3: Add a product of your choosing to your online shopping cart and then remove it from your shopping cart Difficulty Level of Completing This Task
Very Easy Number of participants who gave this rating Somewhat Easy Neither Easy nor Difficult Somewhat Difficult Very Difficult

The Amount of Time it Took to Complete This Task

A Lot Shorter Than Expected Number of participants who gave this rating Somewhat Shorter Than Expected About the Amount of Time Expected Somewhat Longer Than Expected A Lot Longer Than Expected

The Likelihood That They Would Use This Task

Very Likely Somewhat Likely Neither Likely nor Unlikely Somewhat Unlikely Very Unlikely

Number of participants who gave this rating

13

Table 5
Task 4: Find a list of events taking place at your nearest Wal-Mart within the next two months. Difficulty Level of Completing This Task
Very Easy Number of participants who gave this rating A Lot Shorter Than Expected Number of participants who gave this rating Very Likely Somewhat Likely Somewhat Easy Neither Easy nor Difficult Somewhat Difficult Very Difficult

The Amount of Time it Took to Complete This Task

Somewhat Shorter Than Expected About the Amount of Time Expected Somewhat Longer Than Expected A Lot Longer Than Expected

The Likelihood That They Would Use This Task

Neither Likely nor Unlikely Somewhat Unlikely Very Unlikely

Number of participants who gave this rating

14

Recommendations
Based on the comments of the test participants, I have compiled a list of recommendations for enhancing the usability of www.walmart.com. The recommendations are listed in order of importance, based on the results of the usability test.

1. Decrease the size of the on-screen pop-up that asks if users want a service plan when they add a product to their shopping cart. Reason: After selecting Yes or No, users have to scroll down to see the Continue link. The test participants all clicked on the Close button in the upper right hand corner and werent sure if the product was added to their cart or not.

2. Make the In-Store Events link easier to find, possibly in the top navigation bar. Reason: Test participants had a difficult time finding the link, as it required them to scroll half-way down the homepage and sort through less relevant content.

3. Turn the New Customer? text in the upper right hand corner of the page into a link. Reason: New users currently have to click on Sign In to create an account, which implies that the use already has an account. (See Figure 1)

Figure 1

15 4. Include an Add to Cart button on the pictures and brief descriptions in the search results for products. Reason: Test participants had to click on a particular item in order to be taken to a screen with an Add to Cart button. Some wondered why the button wasnt already in the search results. (See Figure 2) This change would be useful for users who already know exactly what theyre looking for.

Figure 2

5. Reduce the number of images and ads on the homepage. Reason: The homepage requires quite a bit of scrolling to get from the top to the bottom of the page. Since there is relevant content throughout the page, this would make that content easier to find.

16 6. Turn the Remove link into a button, like the Add to Cart button. Reason: Some test participants found the Remove link hard to find because the text is small and its surrounded by other links. (See Figure 3)

Figure 3

17

Appendix: Usability Protocol

General Script
The screening process: Excuse me, could I have a moment of your time? If no, Okay then, sorry to bother you. If yes, Hi, my name is Robert Bialecki. Im a student at the University of Wisconsin-Stout, and Im conducting a study for my Usability Testing class. Would you be interested in participating in my study? If no, Okay then, sorry to bother you. If yes, Okay, thank you. First, Id like to ask you a few questions to see if you would qualify for the study. (Ask the questions from the Screener Form and record the responses) Thats all the questions I have for you. Thank you! If you are selected to participate in this study, what is the best way for me to contact you to schedule a session? Before the test: Hello, and welcome to the usability study. Again, my name is Robert Bialecki. If you need a beverage or a restroom, theyre right over there. As you know, Im conducting this study for my Usability Testing course at UW-Stout. The purpose of my study is to evaluate the Wal-Mart website. Specifically, I will ask you to complete a series of common tasks on the Wal-Mart website that new users might use. This usability test will help me evaluate the ease and usefulness of the tasks. Before we begin, Id like to go over this consent form with you. If you have any questions or concerns regarding the consent form or the study itself, please let me know right away. (Go over the form with the participant, allowing him/her time to read and sign) So do you have any questions or concerns? Okay, everything looks good. I would also like you to fill out this questionnaire before we begin. (Allow participant time to complete the questionnaire.) Okay, looks good. The test: So heres how the test will work - I will give you four tasks to complete on the Wal-Mart website. Each task will require you to start at the home page you see currently displayed on the screen. I will be beside you as you complete each task, but I will not be able to tell you how to complete the tasks. I will be taking notes on your procedures, thoughts, and my own observations as you complete each task. There is no time limit on completion of the tasks, but if at any point you decide a task is too difficult, you may give up, and we will move on to the next task. I would also like to encourage you to think out-loud as you complete each task. Say out-loud whats on your mind as you complete each task. The thoughts could be about anything related to the task or the website. For example, you might say Okay, so next Im going to or I think it would be easier if or This is frustrating because I know the process of thinking out-loud may feel

18 weird at first, but your thoughts will be very beneficial to my study. Do you have any questions at this point? Okay then, here is your first task

After the each task: Youre doing a great job. Again, please remember to think out-loud. Do you have any questions so far? Okay, now before we move on to the next task, Id like you to fill out this short questionnaire about the task you just completed. (Allow time for the participant to read and fill out the questionnaire) Great! Thank you! Now lets move on to the next task. Please return to the home page. Your next task is After completing the test: Well, thats the last task. Youve done a very good job, both with the tasks and thinking outloud. Theres one last thing Id like you to do. Please fill out this questionnaire about the test overall. (Allow time for the participant to read and fill out the questionnaire) Okay, thats all I have for you. Do you have any last questions? Again, Id like to thank you for your time and participation in this study. Have a nice day!

19

Screening Form
Candidate Name: ______________________ Qualifies?

Yes

No

Screening Questions: 1. Are you older than 18 years of age? Yes (continue) No (terminate) 2. Do you have more than one year of experience using the Internet? Yes (continue) No (terminate) 3. Are you familiar with the company, Wal-Mart? Yes (continue) No (terminate) 4. Do you have any negative opinions about Wal-Mart (the company or its practices)? Yes (terminate) No (continue) 5. Have you ever visited Wal-Marts website, Walmart.com? Yes (terminate) No (continue)

Cell Phone # _______________________ Home Phone # ______________________ E-mail _____________________________

20

Consent to Participate in Wal-Mart Website Usability Study

Title: Walmart.com Usability Study Moderator: Robert Bialecki

Description: This is a usability study of Wal-Marts website, Walmart.com. The purpose of this study is to evaluate the effectiveness of the website in order to help better meet the needs of Wal-Marts consumers. You will be asked to complete several tasks that users of the website would commonly do, as well as answer some questions that pertain to your experience while interacting with Walmart.com. You will be observed only by the moderator who will be taking notes; however, you will not be recorded by video or audio. Time Commitment: Your involvement in this study will take no longer than one hour. Risks: There are no adverse risks involved in the participation of this study. This study is solely for educational purposes in the completion of a college course project. Confidentiality: Your involvement in this study will be held strictly confidential by the moderator. No potentially identifiable information will be shared with anyone for any reason in any aspect of this study. Right to Withdraw: Participation in this study is strictly voluntary. There will be no consequences should you choose not to participate in this study. Furthermore, you may withdraw from this study at any time and for any reason, even if you have already agreed to participate. Moderator Robert Bialecki University or Wisconsin-Stout Student (608) 783-0529 bialeckir@my.uwstout.edu Statement of Consent: By signing this form, you acknowledge that you have read and understood the information above and that you agree to participate in the Walmart.com Usability Study.

_______________________________________________ Signature

__________________ Date

21

Pre-test Questionnaire

1. Have you ever visited the website of any retail store before? Yes

No
If you answered Yes, please answer the following two questions: Which site(s) did you visit?

Are there any aspects in particular that you liked or disliked about the site(s)?

2. Have you ever purchased anything online from a retail store before? Yes

No
If you answered Yes, please answer the following two questions: Which retailer(s) did you purchase from?

Was your experience a positive one? Why or why not?

3. What sorts of information do you feel should be available on a retailers website? Please list what you feel is most important.

4. What sorts of features do you feel should be available on a retailers website? Please list at what you feel is most important.

22

Tasks

1. Create a user account at www.walmart.com 2. Get driving directions from your home to the nearest WalMart store. 3. Add a product of your choosing to your online shopping cart and then remove it from your shopping cart. 4. Find a list of events taking place in the next two months at your nearest Wal-Mart store.

23

Post-task Questionnaire (same for each task)

1. Rate the difficulty level of completing this task: Very Easy Somewhat Easy Neither Easy nor Difficult Somewhat Difficult Very Difficult

Please explain your rating:

2. Rate the amount of time it took to you to complete this task: A Lot Shorter Than I Expected Somewhat Shorter Than I Expected About The Amount of Time I Expected Somewhat Longer Than I Expected A Lot Longer Than I Expected

Please explain your rating:

3. Rate the likelihood that you would use this feature/task: Very Likely Somewhat Likely Neither Likely nor Unlikely Somewhat Unlikely Very Unlikely

Please explain your rating:

How might this task be made easier?

24

Post-test Questionnaire

1. Overall, rate the average difficulty level of the tasks: Very Easy Somewhat Easy Neither Easy nor Difficult Somewhat Difficult Very Difficult

Please explain your rating:

2. Rate your overall satisfaction with the Wal-Mart website: Somewhat Satisfied Neither Satisfied nor Dissatisfied Somewhat Dissatisfied Very Dissatisfied

Very Satisfied

Please explain your rating:

3. Would you recommend Walmart.com to friends or relatives if they asked? Why or why not?

25

Checklist
Before participant arrives: Clear the desk of any unnecessary clutter or potential distractions

Access the Internet and go to the Walmart.com home page Make sure a pen is available for questionnaire completion Have pen and paper ready for note taking

Welcome: Introduce myself and thank participant for participating

Show participant where to find beverages and a bathroom Escort participant to the testing room

Before the test: Explain the purpose of the test.

Review the Consent Form with the participant, allow time for him/her to read and sign Ask the participant if he/she has any questions or concerns Give the pre-test questionnaire

Instructions: Explain the procedure of the test

Ask participant to think out-loud during test; explain what it is and give examples

After each task: Offer reassurance if participant seems to have had difficulty with task

Remind participant to think out-loud, if necessary Give post-task questionnaire Ask participant to return to the Walmart.com home page

End of test: Give post-test questionnaire

Thank participant for participating