UsabilityTestReport: TheCysticFibrosisFoundationCFF.

org

EmilyLazo CaronGarstka December3,2012

Introduction TheCysticFibrosisFoundationswebsiteaimstoprovideaccurateandcomprehensivehealth informationforthosebattlingwithcysticfibrosis,ortofamilymembersandfriendsofthosewith cysticfibrosis.Thegoalofthefoundationistofacilitatethedisseminationofinformationtothose whoneedit:patients,familymembersandfriends,anddonorstothefoundation. Thisusabilitytestwasconductedinordertodeterminetheextentthattheinterfacefacilitatesa usersabilitytocompleteroutinetasksbasedonvariousinformationneeds. ExecutiveSummary Weconductedthisusabilitytestusingaliveversionofhttp://www.cff.orgusingWindowsInternet Explorer6.0locatedonaWindows7PCintheITlaboftheUniversityofTexasSchoolof Information.Thetestadministratoranddataloggerwerepresentinthetestingroom.Each sessionwasaudiorecorded.Wecapturedeachparticipantsnavigationalchoices,task completionrates,comments,overallsatisfactionratings,questionsandfeedback. Theuserswererecruitedlargelyfromaconveniencesample,assixoutofsevenoftheusers werefellowstudentsattheSchoolofInformation.Ourparticipantshadavarietyofexperience lookinguphealthinformationontheweb.5outof6participants(83%)usedthewebtosearch forhealthinformationatleastonceamonth. Seventestswereconducted,includingonepilottestandsixregulartests.Eachindividual sessionlastedapproximately20minutesandincludedanintroductionandinformedconsent, backgroundquestionnaire,fivetaskstobeattempted,andanexitsurvey. Accordingtousercomments,themajorityofparticipantsfoundthewebsitetobeclearand straightforward.83%thoughtthewebsitewaseasytouse,andthesamemajorityfoundthat theywereabletodoeverythingonthishealthsitethattheyhadbeenabletodoonotherhealth sites. Thetestidentifiedonlyafewminorproblemsincluding: Textheavyinterface Confusionsurroundingthedifferencebetweenacysticfibrosiscarecenterandachapter locationoftheCysticFibrosisFoundation InconsistentinterfacedesignforGreatStridesinformation ConfusionsurroundingsortingofchaptersinthedropdownmenuforfindingaGreat Stridesfundraisingwalk:listedalphabeticallybychapternameinsteadofbygeographic location Thisdocumentcontainstheparticipantfeedback,satisfactionsratings,taskcompletionrates, timeontask,errors,andrecommendationsforimprovements.Acopyofthescenariosand questionnairesareincludedintheAttachmentssection.

Methodology Thisusabilitystudygatheredassessmentdataabouttheeffectivenessofwww.cff.com.Inthis study,sixparticipantworkedthroughfivetasks.Eachsessionwasapproximately30minutesin length.Fiveminutesofeachsessionwasusedtoexplainthesessionagendatotheparticipant, outlinetheresponsibilitiesofbeingaparticipant,andreviewconsentforminformationwiththe participant.Duringtheremaining15minutesofthesession,participantscompleteda backgroundsurvey,thefivetasks,andanexitsurveycoveringtheiroverallfeelingsandthoughts towardthewww.cff.org. TestEnvironment ThisusabilitystudywasconductedattheUniversityofTexasintheSchoolofInformation computerlab.Morespecifically,theclassroomatthebackofthecomputerlabwasused becauseitwasaquietspacefreeofoutsidedistractions.Thiscontrolledenvironmentwasused inanattempttocontrolandeliminateenvironmentalfactorsthatmightinfluenceorimpactuser behavior.ThetestportionoftheusabilitystudywasconductedonaWindows7PCusing InternetExplorer6.0connectedtotheuniversity'shighspeedinternet.Thebrowserwasreset aftereachsessioninordertoeliminatetheprevioususersbrowserdata.Additionally,the browserwassetupwiththenecessarytestmaterialsbeforethenextsessionbegan.Alltest participantsusedthesamedesktopsetupinordertomaintaincontroloverthedigitaltest environment. Duringthetests,themoderatorwaspositionedtotheleftoftheparticipantwhilethenotetaker wasstationedtotheparticipantsright.Thedistancebetweenmoderatorandparticipantwas closeenoughsothemoderatortoseethescreenbutfarenoughawaytogivetheparticipant theirownspace.Thesameistrueforthenotetakersposition. Whilethesessionswerenotdigitallyvideorecorded,theywereaudiorecorded.Adigitalvoice recorderintheformofanAppleiPhonewasusedtotapetheresponsesandreactionsthe participantshadwhileusingthewww.cff.org. Sessions Theparticipantsampleusedinthisstudywasaconveniencesample.Thetestmoderator recruitedparticipantsfromtheINF385PUsabilityclass.Eachparticipantwasrecruitedin persontotakepartintheusabilitystudy.Intotal,thereweresevenparticipants(fivefemalesand twomales)whorangedinage(2742).Oneparticipantwasusedforapilottest,andtherefore thatparticipantstestresultswerenotincludedinthefinaltestsamplesresults. Eachindividualsessionlastedapproximatelytwentyminutes.Duringthesession,thetest moderatorexplainedthetestsessionandaskedtheparticipanttofilloutabriefbackground questionnaire.Participantsreadthetaskscenariosaloudandtriedtofindtheinformationonthe website. Afterthelasttaskwascompleted,thetestadministratoraskedtheparticipanttocompleteand

exitsurveywhichconsistedofratingthewebsiteoverallbyusinga4pointLikertscaleforeleven subjectivemeasures.Theparticipantthenhadtotextuallyinputtheirpersonalthoughtsand feelingsaboutthewebsiteinapersonalresponsesectionofthesurvey.Thesequestionswere structuredasoverallwebsitequestions: Whattheparticipantlikedmost Whattheparticipantlikedleast Recommendationsforimprovement

Finally,theparticipanthadtoselectthebestanswerona5pointLikertscaleforhowthey generallyfeltaboutusingcff.organdsearchingforthespecifiedhealthinformation. Participants Allparticipants,withtheexceptionofone,werestudentsintheINF385PUsabilityclassduring thefallsemester.Sevenparticipantswerescheduledovertwotestingdates.Sevenoutofseven participantscompletedthetest.OneparticipantwasinvolvedintestingonNovember9thandsix onNovember12th. Friday,November9 Meet,setup Session1(Pilotsession) Debriefing Revisesessionscriptifneeded Revisetasks Monday,November12 Session2 Session3 Session4 Session5 Session6 Session7 Debriefing Ofthesevenparticipants,twoweremaleandfivewerefemale.Theagesoftheparticipants rangedfrom2742yearsofage. TechnologyUse Participantsoutlinedtheirtechnologyuseonthebackgroundquestionnaire.Allparticipants includedinthestudyownamobilephoneaswellasalaptopcomputer.Inadditiontothat,all participantsselectedSeveraltimesadayforparticipatinginthefollowingbehavior: Sendingandreceivingemails Readingnewsitemsonwebsites

Accessingsocialnetworkingsites Fourparticipants(67%)saytheysendorreceivetextmessagesontheircellphoneSeveral timesadayand2participants(33%)saytheysendorreceivetextmessagesontheircell phoneOncedaily. Overall,oursampleparticipantsusedigitalwebtechnologiesdailyandcanthereforebe categorizedastechnologicallyproficientinwebbrowsingandwebinformationseeking. EvaluationTasks/Scenarios Testparticipantsattemptedcompletionofthefollowingfivetasks: 1.FindoutmoreabouttheFoundationanditsgeneralpurpose. 2.Findinformationoncysticfibrosistherapies. 3.FindoutifthereisacysticfibrosiscarecenterinTexas. 4.FindaCysticFibrosisFoundationchapterinAustin,Texas. 5.FindoutwhenthenextGreatStridesfundraiserwillbeinAustin,Texas. Results TaskCompletionSuccessRate AllparticipantssuccessfullycompletedTask2(findinformationoncysticfibrosistherapies)and Task4(findaCysticFibrosisFoundationchapterinAustin,Texas).Fiveofthesix(83%) participantscompletedTask1(findoutmoreabouttheFoundationanditsgeneralpurpose)and Task3(findoutifthereisacysticfibrosiscarecenterinTexas).Fourofthesix(67%)of participantswereabletocompleteTask5(findoutwhenthenextGreatStridesfundraiserwillbe inAustin,Texas).

TimeonTask Wecollectedthetimeontaskforeachparticipantfromthesessionsdigitalrecordings.Some taskswereinherentlymoredifficulttocompletethanothersandisreflectedbythemediantime ontask. WhilethemediancompletiontimesforTasks1,2,3,and4werefortyfivesecondsorbelow,the mediancompletiontimeforTask5,whichrequiredparticipantstofindoutwhenthenextGreat StridesfundraiserwillbeinAustin,Texas,was101seconds.Completiontimesrangedfrom5 secondsto103seconds(approximatelytwoandahalfminutes)withmosttimesfewerthan60 seconds.

Errors Thenotetakercapturedthenumberoferrorsparticipantsmadewhiletryingtocompletethetask scenarios. Errorswerecategorizedaseithernoncriticalorcritical.Noncriticalerrorsinvolveconfusion, followingincorrectlinks,andor/userfrustration.Criticalerrorspreventthecompletionofataskor resultinincorrectdata. NoncriticalErrors 1.Backtracking 2.Initiallyselectedincorrectroute 3.Foundcorrectinfobutdidnotacceptit 4.Foundinfothroughanalternativepath 5.Foundinfobutcontinuedtosearch

CriticalErrors 1.Couldnotcompletetask 2.Hadtoaskforhelp 3.Believedtaskwascompletewhenitwasnot

ExitSurvey Alloftheparticipantsfoundthatthewebsitewasveryprofessional(onascaleof1 Professionalto4Unprofessional)forbothitsuserinterfaceandcontent.Sixparticipants (100%)ofusersratedthewebsiteassafetouse.Safeinthisinstancewasanextensionof trustworthyandcouldbeappliedtotrustworthinessofinformationprovidedandtransparencyof thesitecreators.Mostoftheparticipants(67%)ratedthewebsiteasbeingveryhighqualitywhile theremainingparticipants(33%)ratedthewebsiteasbeinghighquality.Althoughparticipants encounteredsomedifficultyandconfusionwithcertaintasks,theparticipantsaverageratingfor howclearthewebsitewas1.5.Thisisparticularlyinterestingsince1stoodforClearand4 stoodforConfusing.Halfoftheparticipants(50%)ratedthesiteasa1andhalf(50%)ratedthe siteasa2.Noparticipantsratedthesitea3ora4,whichindicatesthatthesitewasnot confusing. Halfoftheparticipants(50%)agreedthatthewebsitewasEasytouse.Twoparticipants(33%)

ratedthesitea2foreaseofusewhichindicatesthatwhilethesitewasforthemostparteasyto use,therewassomeconfusiononcertaintasks.Oneparticipant(17%)ratedthesitea3for easeofuseandnoparticipantratedthesitea4.83%ofparticipantsfoundthesiteEasyto Moderatelyeasytouse,while17%foundthesitetobeSlightlycomplextouse. Threeparticipants(50%)ratedthewebsitea1forlikability.Threeparticipants(50%)alsorated thewebsitea2forlikability.Likabilityinthisinstancereferstooverallfeelingstowardthe website.Thereforetheparticipantsexpressedstrongtomoderatefeelingsoflikingthewebsite whilenoparticipantsexpressedstrongtomoderatefeelingsofdislikingthewebsiteasawhole. OverallRatings Afteratasksessioncompletion,participantsratedthesiteforthreeoverallmeasures.These measuresinclude: 1. Relevancyandusefulnessofsitecontent 2. Thesitefacilitateshealthinformationgathering 3. Easeofuse Thefollowingbargraphsshowthethreeoverallratingquestionsandhowtheparticipants responded.

IthoughttheinformationIgotfromthewebsitewasuseful.

IcandoeverythingIwouldexpecttobeabletodoonahealthinformationwebsite.

Overall,thesitewaseasytouse.

Asisevidentfromthegraphs,almostalloftheparticipantseitherAgreedorStronglyagreed withthemetricstatements.Therefore,itisreasonabletosaythattheparticipantswere consistentlysatisfiedwiththefunctionalityofthesite.Theoneparticipantwhoratedthesecond statementasNeitherdisagreeoragreerelatedtothemoderatorthathedidnotknowwhata healthinformationsiteshouldincludeandcouldnotdecidewhethertoagreeordisagree. Likes,Dislikes,ParticipantRecommendations Uponcompletionofthetasks,participantsprovidedsurveyfeedbackforwhattheylikedmost andleastaboutthewebsiteintheexitsurveyaswellasrecommendationsforimprovingthe website. LikedMost Thefollowingcommentscapturewhattheparticipantslikedmost:

Whataretwothingsyoulikedmostaboutthesite?

Participant1

"Selfexplanatorytermsfornavigationtabsclear Clearandsimplecontent" "Ilikedthewaythemenuspoppedupwhenyourolloverthetopnavigation.I feellikethatwouldsavemealotoftime! Ithoughtthenavigationlabelswereprettyclear." "Thesitedidnothaveanoverwhelmingamountofmenuoptions.Itemswere easytofindevenwhenIgotofftrackabit.Thesitedidnottakelongto navigate.Theuseofcolorsandimageswasnotdistracting." "Thelabelswereveryclearandeasytounderstand NavigationwassimpleandbroughtmetowhereIexpectedtogo" "decentlayout,prettysimplenavigation" "Itfeaturedinformationforthefirsttimevisitorprominentlyandpresentedit clearly. Accomplishingthechosentasksseemedfairlystraightforward."

Participant2

Participant3

Participant4 Participant5 Participant6

LikedLeast Thefollowingcommentscapturewhattheparticipantslikedtheleast: Whataretwothingsyoulikedleastaboutthesite? Participant1 "TheGreatStrideSection(locatingacity) Locatingachaptervs.locatingabranchoffice" "There'salotofinformation,andthereforealotofchoices.Itseemedlikealot ofnavigationoptionswereduplicatedinthelefthandmenu. ThewalkwebsitelookedsubstantiallydifferentfromtherestoftheCF website.Thatwasalittlejarring." "Idon'tthinkIlookedatthesitelongenoughtofindsomethingthatIdidn'tcare for." "Itwasmoredifficulttofindspecificinformationaboutthewalkwithoutactually beingaparticipant Someofthelinklocationswithintextfieldswerenotaseasytoaccess" "Fndinglocalresourceswasnotintuitiveastheywerelistedbytitle alphabeticallyratherthanlocation" "Thelinkstofindingchaptersandcarecentersarerelegatedtothebottomof theleftsidebaringraytext.Thesecouldbenotvisiblewithoutscrollingona smallerscreen. Maybesomethingmoredynamiclargerfeaturedimagesofkidsthatreally encouragesmetowanttohelpout.

Participant2

Participant3

Participant4

Participant5

Participant6

Onthe"Findafundraiserbychapter"dropdown,thechaptersareorderedby alpharatherthanalphabystatethenalphabychapter."

RecommendationsforImprovement Thefollowingcommentscapturewhattheparticipantslikedtheleast: Whatwouldyouliketoaddtothesite? Participant1 "Awellorganizedlocateawalkmenu" "Itwouldbecooltohaveafeaturethatdetectsyourlocationandautomatically providedyouwithalistoflocalchapters,carecentersandevents.Orevena singlezipcodequestionthatcouldbringupalistofallofyourlocal resources." "Fromthelocalchaptersite,IwenttochaptereventstofindoutaboutGreat Strideseventhoughtherewasahugelinktoitintheleftnavbar.Theresults frommysearchforGreatStridesdidnotshowanything.PerhapsiftheGreat Strideswerealsolistedinthesearchresults,thatmightbehelpfultosome users." "Easierbacknavigation" "Nothing" "Ididn'tgetachancetocheckduringthetest,butfeaturedstoriesaboutkids withCysticFibrosis?(Okay,there'sone.Maybemore/larger?Maybeaslide showofclickablevideos?)Featuredimagesofsuccessful,enjoyable fundraisingevents. MoreprominentdisplayofFinda...links."

Participant2

Participant3

Participant4 Participant5

Participant6

Recommendations Thissectionprovidesrecommendedchangesandjustificationsdrivenbytheparticipant successrate,behaviors,andcomments.Eachrecommendationincludesaseverityrating.The followingrecommendationswillimprovetheoveralleaseofuseandaddresstheareaswhere participantsexperiencedproblemsorfoundtheinterface/informationarchitectureunclear. Toreiterate: AllparticipantssuccessfullycompletedTask2(findinformationoncysticfibrosis therapies)andTask4(findaCysticFibrosisFoundationchapterinAustin,Texas).Five ofthesix(83%)participantscompletedTask1(findoutmoreabouttheFoundationand itsgeneralpurpose)andTask3(findoutifthereisacysticfibrosiscarecenterinTexas). Fourofthesix(67%)ofparticipantswereabletocompleteTask5(findoutwhenthenext GreatStridesfundraiserwillbeinAustin,Texas).

EventhoughsomeerrorsoccurredwhenparticipantstriedcompletingTask1andnoerrors occurredforTask4,therecommendationswefocusoncomefromTask3,4,and5. Task3Recommendations,Justification,andSeverityRating Task3requiredparticipantstofindoutifthereisacysticfibrosiscarecenterinTexas. Change Addaworkingdefinitionfor carecenter. Justification Severity

Threeparticipantswere High confusedbetweenthe functionofacare Includeaprominentsearch centerandthefunction featuretosearchlocallyfora ofachapter.Thesite carecenter. wouldbenefitfrom definingthefunctionof bothand/orusing differentterminology. Additionally,beingable tosearchforacare centerlocallyis essential.Currently, carecenterinformation iscategorized regionallyandnot categorizedlocally. Participantcomments includedcategorizing carecenterinformation inamoreconcise mannersoitiseasier tofind. Task4Recommendations,Justification,andSeverityRating Task4requiredparticipantstofindaCysticFibrosisFoundationchapterinAustin,Texas. Change Addaworkingdefinitionfor chapter. Moreprominentlydisplaythe Findachaptersearchtool. Justification Severity

Threeparticipantswere High confusedbetweenthe functionofacare centerandthefunction ofachapter.Thesite

wouldbenefitfrom definingthefunctionof bothand/orusing differentterminology. Additionally,beingable tosearchforalocal chapterisessential. Currently,chapter information iscategorized regionallyandnot categorizedlocally. Participantcomments includedcategorizing chapterinformationina moreconcisemanner soitiseasiertofind. Participantsalso suggestedplacinga Findachaptersearch optioninamore prominentspotonthe homepage. Task5Recommendations,Justification,andSeverityRating Task5requiredparticipantstofindoutwhenthenextGreatStridesfundraiserwillbeinAustin, Texas. Change AddGreatStridesLocatea WalkinQuickLinksor mainnavigationmenu. Startpageatthetop. Addsearchfeaturethat retrievesalistoflocal chapters,carecenters,and events. Justification Severity

Participantssuggested High placingaGreatStrides Locateawalklinkin theQuickLinks menu.Thiswould facilitatefaster informationretrieval withouthavingtogo throughadditionalsteps tosatisfythe informationneed. Additionally,whenthe

GreatStridespage opens,itneedstoload fromthetopofthe page,notthemiddle. Themiddlepageloadis confusingtousers. Oneparticipant suggestedthatthere shouldbeageneral localsearchthatwould retrievelocalchapters, carecenters,and eventsallatonce.This hasthepotentialto satisfymultiple informationneeds simultaneously.

Conclusion MostoftheparticipantsfoundCFF.orgtobewellorganized,comprehensive,cleanand uncluttered,veryuseful,andeasytouse.Havingacentralizedsitetofindinformationiskeyto manyifnotalloftheparticipants.Implementingtherecommendationsandcontinuingtowork withusers(i.e.,reallaypersons)willensureacontinuedusercenteredwebsite.

DocumentA:BackgroundQuestionnaire

DocumentB:Script SessionIntroduction Thankyouforagreeingtotakepartinourresearchstudy.Mynameis[Caron/Emily]andthisis mypartner[Caron/Emily].Todaywearegoingtoberunningthroughacoupletasksusingthe CysticFibrosisFoundationswebsite. Therefore,wereheretolearnabouthowusersnavigatetheCysticFibrosisFoundation's website. Duringthesession,Iwillaskyoutousethewebsitetodoavarietyofthingsandwillobserve youwhileyoudothem.Wewantourtesttobeasrealaspossiblesoasyoudothesethings, pleasetrytodowhateveryouwouldnormallydowhenseekinginformationonawebsite. Pleasetrytothinkoutloudwhileyoureworking.Justtellmewhateverisgoingthroughyour mind.Pleaseknowthatwerenottestingyou,andthereisnosuchthingasawronganswer.You willhelpusunderstandwhatworksordoesntworkonthesitebydoingthis. Also,pleasebecompletelyhonestwhenexpressingyourthoughts.Imjustheretomoderatethe study,sonothingyousayaboutthiswebsitewillupsetmeorhurtmyfeelings.Ifsomething aboutusingthesiteishardordoesntmakesense,besuretoletmeknow.Also,ifsomething worksgreatorsurprisesyouinagoodway,besuretotellmeaboutthattoo.Wewantallofyour feedback. Now,beforewegetstarted,IhaveastandarddisclaimerthatIsaytoeveryone: Thisstudyiscompletelyvoluntary.Bylisteningtoyourthoughtsandobservingyouractions,we hopetomakeusingtheCysticFibrosisFoundationswebsiteeasierandmoreenjoyable. Nothingyousayordowillbeusedtoevaluateyou.WeareonlyevaluatingtheCysticFibrosis Foundationswebsite.Also,nopersonallyidentifiableinformationgatheredduringthisstudywill besharedwithanyoneoutsidetheprojectteam.Onceagain,thisstudyiscompletelyvoluntary. Youcanstopthetestatanytimeforanyreason. Doesthatsoundokwithyou? [Ifyes]Great.Pleasereadthroughandsigntheconsentform.Thesecondconsentformisfor recordingpurposes.Pleaseunderstandthattherecordingwillbeconfidentialandwillbe destroyedaftertheusabilitytestingiscomplete. Thewholesessionwilltakeabout20minutes.Doyouhaveanyquestionsbeforewebegin? [Answeranyquestions.]

TaskMatrix Great.Letsgetstarted. First,pleasefilloutourpreliminarysurveybeforestartingthetaskportionofourtest. (pullupopeningquestionnaire) Task Nowletsbeginthetaskportionofthetest.Foreachtask,readitaloudandthencompletethe tasktothebestofyourability.Donotmoveontoanothertaskuntilyouhavecompletedthe presenttask.If,forwhateverreason,youareunabletocompleteatask,justletusknowand youcanmoveontothenexttask. Anyquestions? (handthemtasksheets) Letsbegin!Pleasereadthefirsttask. (foreachscenario,periodicallyask"Pleasetellmewhatyousee")

ClosetheSession Havingusedthesitenow,wewouldlikeyoutocompleteaclosingquestionnaire. (pullupclosingquestionnaire) (onceparticipantisdonewithquestionnaire) Thankyouforparticipatinginouruserstudy!Yourinputisvaluabletous.

DocumentC:Tasks Tasks

YouareunsurewhattheCysticFibrosisFoundation'spurposeis.Findoutmoreaboutthe Foundationanditsgeneralpurpose.

Nowthatyouknowalittlebitaboutthefoundation,youwonderwhatinformationthefoundation hasavailableoncysticfibrosistherapies.Findinformationoncysticfibrosistherapies.

TheCysticFibrosisFoundationprovidesfundingforandaccreditsmorethan110cysticfibrosis carecenters.YouarecuriousifthereisacenterlocatedinTexas.Findoutifthereisacystic fibrosiscarecenterinTexas.

TheCysticFibrosisFoundationhasmorethan75chaptersandbranchofficesacrossthe country.YounowwanttolocateaCysticFibrosisFoundationchapterlocally.FindaCystic FibrosisFoundationchapterinAustin,Texas.

YouwanttosupporttheCysticFibrosisFoundationbyparticipatinginaGreatStridesfundraising walk.FindoutwhenthenextGreatStridesfundraiserwillbeinAustin,Texas.

DocumentD:ExitSurvey