Gui clint'5.1,'l: I)cu,loy lislsol ,tsutrr t,dl(,\t,rir,s llurttn, t,ntttl .tllrl .rclusiur Guideline5,14: M.rintai spttcitrlq bcttttct,ttllsrt)l,r cttl.,!

t,rit,s tltttl is consistent Tuitlt,ncasurcmc t i tcnt Closed-Ended Nominal Guideline5.75: Ask respondents to runk only a fezu itemsrtt rttrr L rather than a long list Guideline5.76: Auoid biasfrom unequalcornpatisons Guideline5.77: Randomize responseoptions if there is cotrtr,ttr about order effects Guiiletine5.78: llse forceil-cioice questions insteadof check_ult that-apply questions Guiileline5.79: Consider using ilifferently shaped ansr,,r,t spaces(circles and squares)to help respontlt,ttt: distinguish betzaeen single_and multiple_an stlt t questiot s Closed-Ended Ordinal Guideline5,20: Choose an appropriate scale length_in generrrl, limit scalesto four ot fioe categories Guideline5,21: Choosedirect or constnrct_specificlabels to iur_ prove cognition Guideline5.22: Proztidescalesthat approximatethe actual distri_ bution of the characteristicin the population Guidelitre5.23: Proaide balanced scales uhere categoies are rels_ oely equal distancesapflft conceptually Guideline5.24: Considerhozooerbally labeling and aiiuatta dis_ playing all rcsponse categories may inl.luencc ansT0ers Guideline5.25: Carefully eaaluate the useof numeic labels flnd their impact on meqsulement Guideline5.26: Align responseoptions oerticalh1 in one column or hoizontally in one rozo and strioe for esual .1ist ance betueen categori es Guideline5.27: Place nonsubstantiae options at the end of the scaleand separatethemfrom substantiaeoptions

C H A P TE R

From Questionsto a Questionnaire

Tttlnt ts a huge differencebetweena list of unordered survey questionsand lhe questionnaire that delivers them to respondents effectively. The design of a questionnairemust consider how to motivate someonewho receivesit to becomea respondentand complete the survey.It must also avoid many measurement problems, ranging from unintended order effects to unnecesThis chapterdescribes how to transition from rarily high item nonresponse. I list of questionsto a respondent-friendlyquestionnairethat will maximize rcsponseand minimize measurement error. Many of the challenges of transforming questions into a questionnaire are . lllustrated by the now infamous butterfly ballot experiencefrom the 2000 presidentialelection.A dramatic controversy emergedover whether some votersin Palm Beach County,Florida,had mistakenlyvoted for PatBuchanan for president while intending that their punch on the butterfly ballot be for the Democraticnominee,Al Gore (seeFigure 6.1).C. R. Fox (2000)showed that about 2,800 unintentional votes were recorded for Buchanan, whereas incompletepunchesand 11,000 2,300 double puncheswere recordedthat invalidated intended votes for Gore. Theseargumentsare supported by data showing that the votes recorded in Palm BeachCounty for Pat Buchanan in that election were proportionally much greater than those recorded in other Florida counties based on the number of registered Reform Party voters, votes for Bush,total votescast,and votesfor Buchanan in the 1996 primary (see Figure 6.2).Inaddition, Wand et al. (2001) concludedfrom a detailedpostelection analysisthat theseanomalieswere not found among absentee voters in Palrn BeachCounty who did not use the butterfly ballot and that Buchanan'ssupport in Palm BeachCounty tended to come flom more Democraticprecincts. 151

| ' ,r

llirr\t(-)r

r,r r , ' r , ,

t, ' \ (-)r r',tt' rr.rrr\ttrl

I tttttt l l ttt

l ttttl

l r, // ()i /r",//rIrrri rl ,r'

I 1]

Figure 6.'l

Palm Beach C(nlrly tnrltotllybilllol

( lt l( 'sli( ) lllr . lilf s is t 't t t t t suall'eople ' l l rc rr sc ol Pt t r r lt l, ir r s lor t r t , t t kit t f l ( '|l( 1) t lt ll( 't lll( 'lll, t \ ( ( '[ ) t wh( ll t lr ey v( 't c, dnd f ( ) r m any rl o rtt)t t vPir ', r llv r olt.rs thc prt'r'iotts t'lt'r'tiotrwas tikely ;r distant memory Thus, the at()[ v()t('rswas likety dividc'd between making sure they Punched lr,rrti()n tlrt,b.rllrt corrcctly and that they registeredtheir choice correctly' I M()st \,ot(,rs knew which candidate they planned to vote for before they wt'rrI into the voting booth, so they probably stopped reading choices ont t' tlrcy found their preferred candidate. r llt'st'arch has shown that peoPle typically start to read in the upper leftIr.rrrtl corner of the left-hand page and proceed down the page, much as tlrt'v would read a book. Thus, George Bush's name was likely seen first, lrrlkrwed by that of the second listed candidate, Al Gore. , lt is unusual tO list the candidates for one office (or answer categories in ,r srrrvt'y) in two different groupings on two different pages. Voters as wr.ll as qucstionnaire respondents are used to having all answer choices irr a sirrgle, defined group on the same page. r llrt' lines around candidates on each page visually suggest that the canrlirl,rtes on the first page should be considered as a separate grouPing Ilorrr clnclidates on page 2. . ll is ('ven more unusual to have answer choices on two separate Pages "slr,rlr," a column of answer spaces in the middle between them, so that .rraw(,rs must be punched to the ri8hi of candidates listed on the left gr,rgt.,rnd to the left of candidateslisted on the right page. r Wlrt'n some people who wanted to vote for Gore saw his name and Ilirrl io figure out how to vote, their eyes were likely drawn to the word " l )('nr()cratic" and the line irnmediately above it, which led directly io box in the answer column. It was logical for a voter to mark llrl st,trrrrtl llr,rl st,tonci box because the first box in the column was for Bush, the rrrb I it . r r r | Ir,r cr r r didat e. . ( )r( (, tlr(' st'eoncl holc was punched and voters pulled away their hand, rl n,orrltl not bt'surprising that some voters noticed the arrow in the l)r,rr(\'r'.rlic sp.rcc and considered the possibility thai they may have probably marked l'r I r( llr\l th(' wr()ng hole. Ttr remedy the problem, some llrr,shrrrrl lrolt'ltsicle the Gore "rectangle," thus producing a "double
" l rl l l l ( l l

Based on these analyses,there can be little doubt ihat the desigrrof tlrr' | ' rrt terfly ballot produced (a) many votes for a candidate for which rolt'r",lr'I not intend to vote and (b) many votes that could not be countcd al .tll llr, t r,'r, consequences of these errors for the outcome of the 2000 presidcn fi.rI r'1, ' make this perhaps the most dramatic and consequentialexamplc clt'r' I 'l I " " 'r ll design of an electionballot, which is quite similar to a survey qtr('sti{)rrrr,rrr' i slik elv t hat anum b e ro fc h a ra c te ri s ti c s o fth e bal l otcausedthevol i rrl i ,rr" r in 2000 counties in all Florida Figure6.2 Votesfor Buchanan Re presidential to the numberof registered relative election, form Partyvoters.
4000 3500 1000 2500 2000 1500 1000

Palm Beach Counl\

r r ll' r l\r r ' I r l ll, 1,"

'n

r lt l. rrrrliLr.lylh.rt somt'onc will raise his hand in a polling place and ,r,,1 lor ll)(.,rll('nti()rrof poll workers to get a new ballot. Most voters by having to announce to votrng precrnct n'orrlrl lik'lv I't'r.rrrl,.rrr.tsscrl \\'r,rLr.t:,llr,rl llrcv ttt,tv lt.rt r.'ntatit' a nt islrke cln the ballot ln addition, ,r l r.\v\ ol( , t r -|lr , r l lr . t r ( 'llr r t t lilt l il w. r s. tPPr opr iat t t ' o m ar k t wo punches ( ' \ i( l( 'llliaI car lt lit lat cwer e list ed , r r r . r I . t t l( l \ 'i( l ,rr,rtr. , rl, . , , llt . r | ' rr' t I ' t tI l) r t s, is llol st t r Ptisit r r llhat double Irrl l rr. l) r 'r r r lr r , r lr , '. l, r , i lot t lt , r . r 't r ', t r 'ot til l', 1 llt r "'( '\ lt . l" vllr ". lr r t lllt r ll. r r r . r t t . rrrrrrl r r . , l, r r or r lr r r r r r r lt

l5, l

l r H o n t ( - ) t t ' , t t o l,,

tr t.\

( - tt,t .,lr r ^ tN \lr l

lltt| 'l lr t ,ex pt ' r ic nec o l l l r(.l )rtl l (.rl l yl r.r l h rl i rr l ' ,rl rrr l l t.,tt l t i l l trstr..rtt' s,utV oI tl r tl (l (' s l i (| n n.l i r(' l | 1)nl us s ( ) c i. r tcw rli tl r tl t' r,t.IrP i rrl ],t .l l i st ()l (l l r(.\l t{ ,| l ., Pr(tblem s First, respondentsmust und('rstltrd ltow tr: n.gistt'r thcir rt'sPotrst,s givt,n llrl appropriate technology.For questionnairt'sthe technology uscd nray rarr11. from optically processed paper questionnaires that may requirc marks kr 1,,. made in a particular way and in a particular location (e.g.,fill in thc bubl,L, or place an X in a box) to web surveys where some respondents mny 1,,. unfamiliar with navigating between pages and registering their rcsPonsr-, correctly. People often have to learn the technology of how to record tlr(.,, responses at the same time they are trying to understand the question ,rn,l decide on a response. Second, designing a questionnaire involves ordering the questions irr,r logical way so that the effects of earlier questions on later questions ,u,. diminished. The presidential preference question is particularly complex lrl cause both the presidential and vice-presidential candidates were displavr., I as one choice and also because there were 10 groups of candidates to chorrs| from, even though people are used to thinking of only two maior party can(ii dates. Had this complex question appeared later in the ballot, it is likely ih.r r people would have already learned the ballot format and, therefore, wotrrrr have been less likely to make such a significant error Finally, respondents answering questionnaires or ballots are respontlirrli to subtle visual clues as to what constitutes a question and how they shorrlrl answer itbased on how they rypically process written information on a pagr, The butterfly ballot was particularly poor in this regard, making it difficul t t(,l voters to know where the presidential preference question ended and l.t()u, the answer spaces were associated with the names of candidates. A detailt.,l analysis of how visual design played a role in the butterfly ballot has bccn presented elsewhere (see Dillman, 2007, pp. 463472). In this chapter, we discuss three fundamental aspects of questronn.rrrl design: the effects of technology, deciding question order, and using visrr.rl design principles to construct questionnaires. We present guidelines for or dering the questions and visually designing questionnaires that are comnron to both mail and the Web. Becauseof the different technologies used to errn struct mail and web questioruraires, we then introduce specific guiclt,lirrr,.' for developing each. Finally, we propose means of prctcsting mail arrd r,r,r,lr surveys/ a processthat we believe involvcs mort'sirtrilaritit's than differt,rrrr", for these two survey modes, and tht' .rinr ol wlrit.h is lo Proviclca final t lrcr k to prevent problems such as thost' lh.rl rxr rrllcrl w,illr llrt'bLrltt'rfly b.rllot

litlnnltlt

ltltA* t l)illtttnt

l5S

H()W

Ili(

llN( ) l( ) { ' \

l\ li\ hlli

A l }| | l r l i l l l l N (

t lillclt'lrts

A t lirst b lrrsh , wr.lr,r r r r l r r r , r r lr I I || ' ' , | || || | | | ,| | | | ,ii t t |r t y , t | ] l | ( ! t . 1 0 l t , t \ , {. t ( n t t ( , | l l , t l ( )rr lltc W. l' . ' ||||' , ' ||' ||||r , , r r r lr r . , r , , h' r lllr . l \ t l l ( , , t 1 , t l i l l ( , , r . ! , t t r . r r , , t , ,

In ,r,ri l tl l , r t is ir c'it it 'r r t. r r r t lsr , lr lr r r r (r l( ) . ( , lr st e, r d,pdpcr qucst ionnair es ar e ollr.lr l)ritlt(,(i irr btxrklt,t loll.rt l() lrt.lP respe16gp1" iravigate between the 11,., Wt,[r i5,15 inexpensive as con;uchng in black ltrtgcs. Usirrg colol q;11 dl(l whik', but n(.,t s() for papcr, where each questionnaire must be printed, Wr.h srrrvt'ys can use answers to early questions to ask follow_up quesuons dnrl (,v('r't cusk)mize the question wording (e.g.,,,When you began work at 'Srrlrrrr,' how Iong was your daily commute?,,), whereas paper questionnaires ldnrrot. In addition, branching constructions must be e*plicit u.rd for .e_ r|lr rtrrlt.nts,hr follow in mail-questionnaires, whereas for-web "asy sr.v.y".espo._ rh.nls ne(.d not, as a general rule, even know that branching has tak"a, piu.". (.ilrstructing web questionnaires involves learning howio program items I llrnl they appear on the respondent,s screen in the same way they appear rttr lht' designer's screen. Moreover, web programrning will be transmitted Iu urllwirre and hardware that may be quiie different aid through electronic llrr('cti()ns that can range from quite limited and slow to incredibly fast, and lhr,sc rl ifferences can have substantial effects on the final appearance of the tlufnlronnaire. Developing paper_surveys requires leaming about pnnting t'd1lrtbilitiec, optical scanning and image requirements (wlich vary widelv su rvey.organizarions),.postal regulations that constrain whai respon_ lllljlnrrc,ln rh.nls receiveand respond to, and processingof the questionnaires once lhr.y on' returned. ()ther seeming differences between mail and the Web no longer exist. For lt.tnlplc, both technologies now allow variation to be built into question_ ndlr|',s l()r specific recipients. Through inteltigent printing, one maii respon_ rlrnl nright be asked the following question referring to a!% increase in local ldrr,s, wlrereas another is asked the same question but with the 5% changed frt ll)'11,: "Would you be willing to pay '#yo'more in local taxes if a new nlultapurp()se basketball arena were to be built in Seattle?,, Respondent iden_ lllk'irliorr rumbers can then be used to identify which version of a questron Pd(h r('spondent completed so one can evaluate the effects of the differeni rlroltion w()rding. In addition, the order in which questions are asked or in whI h rcsrr.nsr' c.rtegories provided can arso be varied and even randomized lry lrotll nl(d('rn maii and web survey design technology. Now, even small I rv(,y ()rgilniz.rtions can obtain these capabilities. lt is ,rlso inlPortitnt to recognize the substantial differences that now ex_ . hl lrr lhc t'lcctronic world of surveying. Handheld devices are proliferating wlllr In'rrx,ndorrssPecd,raising the prospect that certain surveys will be an_ twr,r(\i, t)Ul rrol on a trdtlitional dcskkrp or laptop computei with a large rl l' ,rrltliti.rr, litarrrc )r; firt's art' t'mcrging is a suistitute to rnterac^'r''. llv| wlh strtvcyr lo 1.1, 'r pr.inlirrg, 1;',.r,,' sharing, arrtl rcspgnding, especiallyin l l x' w orl (l ol lr r r sir r r . ss. ,rrcl r r r l|. ir , l wllr sr r r yqy*ar " aL, , s, r m . . t im cs em bed_ rl l rl w i l l ri rr r. r r r , r il r r r r , ss, r 1,tr , r rilit , r lr .n{, t lr , t vinll t ( )( . li( . k r.r.l, ( }n a link or go t o l l tl W ' l r l n It r r vir ll , t t r . , I on, ! r . llr r r , , n, r , l, , t r r rr l r r , r il sr r r . vr , ys, r rll{r u w, , ) l l) ( , s1.

lr){}

lrrnrN(-)l

l ' , l t ( ,N' ,

l( ,,\

( - ) l !.' ll( ' NNAll{ l

( )rrl'tittr litr'()irr':;l wtr:; l\7 (lucsti()r1n.rire (!rsy lo n,lvill,tl(' .tnd complctr:.Iiust is cltcourtrgcd through p,ryiirgattcntionkr dt't,rilancl[.rymaking the questionnaire look and seem inrportantand professional. All of the issueswe addressin this chapter,from thc order of questions to the visual designofthe questionnaire and pretesting, drc important to designing a good questionnairethat encouragesresponse and reducesmeasurement errot ORDERING THE QUESTIONS

trrtlslt ttt liott tlt peoplc rrtr gcncral tcrms tlrat describc tirilorirrgcluestionnairt' the move who can answer in different places and situatiot'tsttsing diffcrcnl technologies. The differences that currently separate web from mail survey construction tend to mask equally large similarities in how people respond to visttal stimuli. Both the Web and mail use visual channels to communicate with r(' spondents. Nearly a decade of research has shown remarkable consistencics in how visual layouts of questionnaires produce similar answers on PaPer and on the Internet. For example, whether scalar questions are asked in linerr' or nonlinear formats, whether needed instructions about how to respond ap prt'a r with response cateSories rather than only in the stem of questions, and wlrt,ther tluestions are asked in a check-all format versus a forced choice for nr.rt ,rtl .rffect irnswers similarly across web and mail surveys (Dillman,2007)' Moreovet thc obicctives of good questionnaire design remain the samt' across tlrcse two modcs. The first obiective is to reduce nonresPons(' Responclent-fricndly questionnaires have been shown to imProve overall response rates, but only to a moderate degree (Dillman et al., 1993) More inr portant, some research has shown that making a questionnaire respondclrl iriendly is most likely to improve response among people who are least likcly to respond to surveys, thus helping to reduce nonresponse error' For exanr ple, in a 1992 test of resPondent-friendly U.S. Decennial Census questiotr naires, response was improved by only 2.9 percentage points in areas of tho United Stateswith high response to the previous Census compared to 7 5'Z' itt areas with low response (Dillman et al., 1993).Specific design choices, such os which question to ask first, can also contribute significantly to the reductiott of nonresponse error by getting people to whom the survey questions do n()l apply to retum their uncompleted questionnaires ()l In addition to obtaining acceptable response rates, another obiectivc questilnr error' Good measurement good questionnaire design is to reduce naire design helps encourage all respondents to read and process questiorrs and their component parts completely and in the prescribed order, antl it minimizes the influence of one question on the measurement of subseqtr(rrt items. Poor design of individual questions and of the questionnairc carr rrr crease item nonresponse by causing questions to be overlooked becausc rrl where they are located or other design properties. Effective questionnairt' tir' sign, in contrast, can help reduce item nonresponse. In addition, poor cit'silitt can result in some questions or response options being interpreted d iff('r('tlll\ than intended. Thus, respondent-friendly questionnaires should hclp gtrirlr' respondents as they comPlete the questionnairc. ttto The design of the questionnairc,whiclr is intt'nticrl kr t'trcottr't14t'atrtl t' rl l t' ttl gt' t' l t' tttcl l l :' lht' srri ,tl th n ' t' o l i rrv o k t' s a l l ti va t e people t o r cs p o n tl , l )v tl l ' l I ( ' hap tt' r i tttl ' trrl t' t.' w ' l r(l s r' ,ttt i tt tl t' s i l i tl 2 . Q ttl s l i o trtr,t d i sc t r s s t ' rin 'l i tttl l t' s l i ttl i ' ttl rl srttt' tl l \' l l l l | | rrl l ' l l l l l l rl rrr" l " ,tP| )r' ,rt i trg lht ' t 1t t l' s liot ttt,ti rr' tl rr' i l l rl ti l l l { i tt l l rt' r,ttl v c y (,r l r l | t. t{\l tl (t' rl l ry ttttrl l trg, iak , r l willr J t tl rti ( ,rsr, ot

A questionnaireshould be organized much like a conversation,which typically evolvesin accordance with societalnorms (Schwarz,1995). Mosi convt'rsationstend to follow a logical order in which people respond to what other people are saying and contributing to the conversation.If someone jumps to a new iopic immediately after every responseyou give, it appears that ihey are not listening to or caring about what you said. Consider for n moment receiving ihe foilowing questionnaire,which was proposed for a sclf-administered survey.We have presentedit here in abbreviatedform, but with the order of the questionsunchanged: . . . . . . . . r r o o . What was your total family income in20o7? Do you like to play golf? What is your opinion on global warming? Are you married? How many times have you gone bowling during the past year? What is your political party preference? Do you favor or oppose these measures to reduce environmental pollution? What is your occupation? Please describeyour favorite recreationalactivity. How adequateis your presenthealth care? Which political party doesthe bestjob of promoting economicgrowth? How old areyou? Has your health gotten better or worse during the past year?

Inr.rgincthe difficulty of trying to respond to thesequestionsin the current 0r'tlt.rwhere questionsabout personalcharacteristics, political issues,recreirtirnl, nnd health care are all intermingled and where the questionsare not lltr)up('(l in thc way your knowledBeof the topics is likely to be organized. (itt//rlittr 6.I: Crotp Ralatcd thatCouer Topics, andBegin with Similar Questions I ikrl7to l\ Snlicnt to Nenrly All Respondents Qtir'rlirrrs ( itott|irtg rll.rlt'tl tlrrt'stions rnlkt's it casierfor respondents to answerand Jl)l)r()xitrr.lt('\.ttt .tcttt,tl lltott'r'lost,ly conv('rsati()n. swiiching betweentoPtlr,rtlx\)ltl(,'s,rls!vr,rs,rr('['ss Ir. rtr',tns likcly to l.rcwr'll tlroughtout, as to r,r'ok.loP ol llrt' lrrrrrl Ir,w lol'i(s ,lr(,nrrr('liLr,lt' rt'sltorrscs. lrr ,rddition,

(-trrr.,rr()N', lrilt lrH(|nr tr) /\ (-)l r',||()NNA||rl suchas in constantly changing k)pics lrack and forthwithin l r;rrr.slionrr,rirt,, makcsit appr'.rr tlrc originalquestionorder listedpreviously, that no effort was made to order the questionsin a meaningful way (i.c.,the questionnaire appearsunprofessionaland thereforeunimportant). Once related questions are grouped, ii is often best to begin with questions that are most salient and interesting to the potential respondentsand then move to questions Heberlien & Baumgartner,1978). that are less salient (Groves et a1.,2006; Asking the more salient questionsfirst will help get respondentsto commit to the questionnaire. The following exampledemonstrates the importanceof ordering questions and beginning with salient questionsthat will be of interest to people who will be asked to respond. One of us was once asked to provide advice for a particularly difficult survey of licensedcommercialsalmon fishermenthat focusedmostly on the size of their boats and investmentin their equipment. After much discussionwith the survey sponsor,an introductory sectionwas added to the survey that asked about the fishermen's views of the future of salmon fishing, whether they would advise young people to enter this occupation,and other issuesthat were describedas the "gut" issuesfacing rate for this study of individuals, many of whom the business.The response had not completedhigh school,was well over 50%. Guideline 6.2:Choose the First Question Carefully No single question is more crucial, especiallyin web surveys,than the first one, as it is most likely to determine whether people will respond to th(' survey or choosenot to participate.This is not the place for a question that is long, boring, difficult to understand, tedious to answer/ or poteniially embarrassing. Whereasa mail respondentmay be able to page through thc entire survey to get a senseof length and content, the first question in a wetr design) must stand alone as an survey (especiallyone with a page-by-page invitation to continue to respond The first question should apply to everyone and be easy to read, conr prehend, and answer.In addition, it should be interesting and reflect thc purpose of the questionnaire as it has been previously explained to tlrt' respondent.A first question that is applicableto respondents communicak's to them that the entire questionnaire is relevantto them and encourages tht'ir' participation. Likewise, a first question ihat is interesting will cncorrragr' response by increasingthe perceivedrewards of completing thc srrrvcy arrrl arrdint('r('slir)l'., reducingthe perceived costs. In additionkr beingapplicablt' a first questionthat is simplehelpsto sct the framcworkthat tlrc n'sl ol tlrr' questions will be relative ly t'asyto answt.ritntl will not rer.ltirt' a lol ol t'llott pt'rcr,ivctl cosls.Attri litt,tlly,,r from potential responclt'nls, thrrs n.tllrcirrg firstrltrt'slion llrr'(lr('slio||n,lir'r'.r', th,rlis clcarly corrncr'lt,rI lo lltr,IrrrPost'ol (rrll.r( it w.tscxPl,tittrrl in | l(,ttr,rs llx, srrr\'r'v ,t llr'ltttliol r ottrisk'ttr v lii\,('s .rttrl rttt ltr,lp ptrrtnul(, lruhl,

( )r'rlr,ri rru//rr, Qra'slirrrrs159 Irrtlrt,salntolr lislrr,r.rrr.rrr t,r,rrnplt,discussed in the previousguideline, the lilst tlucstionaskt.tiwlrt'tlrr,r. tho resp()ndent felt that the benefitsof being ,r s.rlmon fisherman werc gctting better,getting worse, or staying about the samc. This was a simple closed-ended question that all respondentscould arrswereasily and that would be interesting to the salmon fishermen being surveyed,which would help encouragethem to continue with the survey. Although it is desirableto choosea first questionthat applies to everyone, is casy to answer,is interesthg, and connectsto the implementationmateriitls, specialsituationsmay override the searchfor a first question that meets thesecriteria. The most common is when respondentsmust meet eligibility r(,quirements up front in order to complete the survey. In these instances, wt' use the first question to determine eligibility and to inform both those cligible and those not eligible to retum the questionnaireif they are compltting it by mail (eligibility criteria are also sometimescommunicated in tlrc cover letter-see Chapter 7). Asking people to completea questionabout cligibility and then return the questionnaire even if they do not completethe rt mainder of the questionsis helpful for understandingcoverage, sampling, lnd nonresponse issues.Otherwise,the ineligible person would be counted rs a nonrespondent.If they are answering a web questionnaire,those who [rc determined to be ineligible are politely thanked for their willingness to porticipateand informed of their ineligibility. Qttilaline6.3:Place Sensitizte or Potentially Objectionable neartheEnd euestions lh( nJ Questionnairc In.lddition to choosing the first question carefully, it is also important to plncc sensitive or potentially obiectionablequestions near the end of the qu(.stionnaire after respondents havehad an opportunity to becomeengaged witlr thc questionnaire,to establish rapport with the suweyor, and have drtriwt'red the more salientand interestingquestions.Respondents who have nlrc,rrlyrespondedto severalquestionsand spent 5 to 10 minutes answering lhc tlut'stionnaire are less likely to quit if asked potentially obiectionable quostions.Moreover,some questionsmay seemlessobjectionable in light of .lu(,sli(tnsirlready answered, and placing sensitive questionsnear the end Avoi(ls interrupting the flow of the questionnaireas would happen if they wr,rr.,rskt.cl abruptly at the beginning or in the middle. pretesting can help ucrrlify qtrr,stions that pcople might objectto answering,such as thoseabout lnrrrrrrr', st'xrral bt'havior, criminalactivity, medicalhistory and so on. (;xtth'litn' 1,..1: ,4rl Qrft,s/irrrr a/ror rl I'.ttrttls in tfu OrdcrtlrcEaents Occurred Att .llrrrt slr'rrltl,rls. lr1' (l.csti(nrs 111.1111' r()()r(l(,r irr a way thatwill be logical Iu lltl tlr.;rrrttrIr'ttt. ln l),trli(ttl,lIl\\)ltl(' lit](l il ('i)si(.r k) rcspondto quesllo n ,rlrorrl r,vr.nl\ i|l llrr.rrrr||,r. lltr,r,\,r,nls lr,rl;r,rrcrl. lirr cxantplt,, if asking
r t $ 'l I n , o l {l r .f'l ti xt', ,r l r ||r r l Ir r ,r 'i r r r |i r tr l ( l ||tr .||l r ,r r r l l 1 o y1 1 1 1 ,1 1si 1 l i o r r s, i l l xr

br,holFlullo wnll llrnruSlr p$rltkxrrwlth rurpmd$nbln lhr dlllgfdnt

l{r0 lirr,r\1 Ql l:, (rN:,tl|,l (-)rtr,rr rru,lrrrr lrl ot ltrrr tlrt'Pr('s('t)l chronokrgicirl ordor (i.c.,from the pastto thc pr('s('nt is ht'lPfulbec,tttst' the past).Asking abouteventsin the order they occurrt'cl autobiographicalmemories are often hierarchically linked in a nctwork so that remembering one event can facilitate accuraterecall of the next cv('rl (Belli, 1998). ln addition, more accurateand complete rccrll in the sequence about eachposition can be facilitated by having respondentsanswer wh('r( they worked, in what position, the nature of the position, and how long tht'r were in that position before responding to questionsabout what they likctl most or why they left the position. Guideline 6.5:Aaoid Unintended OrderEffects Question Many surveyorsthink of eachof their questionsas standing alone;howevt'r, r('spond('nts oftendraw on surrounding questionsasthey attempt to interpr('l lntl answcr a given question.The effectsof earlier questionson answersl(' latcr tlr.restions arc rcferred to as question orderefects.Altho'tg! the caust'ol ir tlucstiorr ordcr cffect can vary, the outcome is usually one of two typt's: to questionsbecomemore differe'nt; a cutlntst tfftct, whereby the responses whereby the responses becomemore similar. Both ol or an assiuiltfion affcct, thesetypes of effectsbecomeincreasinglylikely to occur when the questi(nrs are closer to one anothet both in terms of topic and in terms of physical proximity on the page or screen. Figure 6.3 summarizesvarious causes of assimilationand contrasteffects. As the figure illustrates,questionorder effectscanoccurwhen early questions influence the cognitive processingof later questions (i.e., a cognitiuahnt,l ordereffect\ or when early questionsinvoke a socialnorm that affectsthe wav later questionsareanswered(i.e.,a normatiae-based ordere/ect).The followirrli examplesillustrate how eachof the effectsin Figure 6.3can occur in a survt y Priming Researchers randomly assigned respondentsto the National Health lrt('r' in an attempt to me.rsr.r'r' view Surveyon Disability to six condition checklists the prevalenceof chronic conditions in the populaiion. After the checklisls were administered,all respondentswere asked questionsabout disabilili('s, including whether they had a disability and, if so,what causedit. ln n's1'orrs,' ls to the questionabout what causedtheir disability,nearly 49'Z'of responclt'rr who had previously been asked about sensoryimpairments reportt'tl srrr'lr conditionsas the causeof their disabilitycomparctlto only 4l'X' ol tltosl impnirnr('nts. Tht' sarlc pntlcn)lr'l( | who had not beenaskedaboutsenst)ry ()vt,rlll, typcs of chrorrit trrncliliorrs. rt'sporrtlr,rrls wlro foranumberof other reportedhaving c lisabilitit's w('r('rror(' likr'ly to atllilrrrk'llx'il rlis.rlrility tht.yh,rrlh.r.rr,rslrlrl ,rlrrrrrl irr tlrr,inlclvilw tlr,rr to contlitions l)r'(,viously ('li',lor,'r, l(),rll(,rrr.)tiv('trrrrtliliorrr llll|(l) llrr.r',rlv rtrr,.,liotr,, Ir|.|| ||, { r,| |,I i tI

( ) , r / r 'r iI , ( lr r '( ) r / r 'slr r r r s l( )I sourcsof questionorder lloura 6.3 Common cognitivo-and normative-based tlf.ot8.

I l r ttr f\r! , r' s b r' g I', Ir r r x l !r nlr! rrl l h rt rs l h.n tr !r . i (. . s s rh l c

l{csll(nrdenlspcrcc'vc

Conshincy: Respondentsanswe. a similarly to the fiBt in

Avoiding Respondents attempt selecnng somc

'lems

Nonn: P.esentationof

Nom: Prcsentationof

fttllr liliorrs rrron,ar'tt'ssiblc for considerationin the later questions,a priming i Jl r\ | l l r,rt r cst r lt r r lin assinr ilat ion. l'nt tVtpr't prcscnk'tl willl two qucstions:"How would you describe ller;rlttrh.nls wr.r'r. yrtttr rrr,rl l i, r 11'f,"r r r r l"l kr w r vot r lt lvor r s, r y t hings ar t 't ht 'sedays?" Answer s or r whit 'h wir s asked f ir st . l wr r r lur . sli{r r , v, r I ir r l lilr . , r lly t lt 'Pcr r r lir r g ht l l rr,nr, i lu( . r , 1i( s, li( r r iagc wr t svcr y l l rr.rr r , r lr i, r 1ii' r | $', r s, r sI r \ | lir st ,7ll': i, | |lr lil r r r a W l rr,rr 1r s11; , , 11111 l'' I r r 'llf ||"t t in' lt , r |||\ , ) ,, t r r l 52'i' wI r t l ot t lr t sit y llt t 'y Itnl l l , 11 "r

l{ r: l

lrx0M

Ql

t . i l t ( r N' ,

t( | A L h

t' ,lt( | NNAtxl

( ) r r l'll, / , \l/ li'( ) t ir , r li0r r r l( r . 1 ) r r r t l I ) olt . lr or r t , st lor . r r r t l t lust wor t lt y ( M t xr r e, 2002) .Wher eas Itt,ol )l c(( ) . 1'1, ltt lltr,Prt'viorts t'rarrrPlr'[11rP11' k,rrtlt.r.l to focus on tlre similarities between ( lirrlorrarrti( iort', r'r'str lting irr a n assimilationeffect,here they tended to focus lrr llrc rliflt'rt.nct's bt'tween (iingrich and Dole, resulting in a contrast effect. Itr lrolh of thcse examples, the standard formed by the subiect in the first (lr r.rli()n was applied to the second question, resulting in an anchoring effect. 5rl'lrurlit,tl M,rsorr,Carlson, and Tourangeau (1994) asked respondents how they would rllsr'r'ibc tlrc economic situation in their communities over the next 5 years how thcy felt about the economic situation in their state over the next 5 '||r(l yr,,rls. l'lrcy found that more people (7%-70%) said the state economy would Xcl lx'ttcr when the state economy was presented before the community econonry question. The reason for this difference is that once people answer the lirsl question, they tend to "subtract" out reasons used to justify their answer lrorrr tlrcir reasoning on the second question. In this particular case, when llx'r1trt'stion about the community economy came first, new industry was rr l,r( k)r that strongly influenced respondents to say that their community's ('(()nouly would get better. New industry also strongly influenced responses ithrrrt thc state economy when that question came first, but when thc state r,((nr()myquestion came after the community question, new industry did rr0t pl,ty as large of a role for the state question (i.e.,it was subtractctl out of tl.rc t'rrrrsitlcrations). The result was a subtraction effect,whercby inrportirnt con, litlt,rations used to answer an earlier question are taken out of consirlera tiorrs hrr n l.rtcr question. Noru tf Ertenhandedness Slu(l(,nts at Washington State University were asked two questions about lhr,r'orrst'quences of plagiarism. One asked whether a student who had plagi,trizcrl should be expelled. The other asked whether a professor who had Pl,rgi rized should be fired. When the professor question was asked first, 34% ln(iic.rh'd on the following question that a plagiarizing student should be exlrr'llt,tl. I fowever, when the professor question was asked second, only 217o Inrlica ttrl that tht'student should be expelled (Sangsier,1993).The likely cause ol this rliffcrcncc is that once the students had judged someone from outside llrt.ir gloLrp harshly for plagiarizing (i.e., by recommending firing), they then Ir,lt llr(,y nrust npply tlrc samc standard to someone within their group. This ()l ,rdjrrstinglnswcrs to be evenhanded or fair was first noticed lrllr,rtot)r('nr)rr (llyrrran & Sl.rt'atsley, Irr ,r r'l,rssitslurlv in l().18 1950) in which respondents wr,rr' ,rsktrl wlrr.llrr.rlorlrrrrrrrisl r('l)orlors shotrld bc allowed to report on vi .,i l s l o l l r r , I lr r it r r l St , r lr . s, r r r twlt l r . llr r . rt J. S.r t Jr or lt 'r sshould be allowed t o tt' l \rtl rrr lisils lr r llt r , $r r ir 'l llr r ir r r r Wlt lt t t lt t . t ; t r t 'slior r , r hot r tcom m unist rr,l ,orl (,r', r r 'lr r lr r l' ol \ r t t r l, lo , llr r . llr r r lr . , llil, r lr , r w, , r s, r skr r lir l sl, only i7'7,

t() tht'gent'ralt;utstiotr.llttl wltt'lt tht'gt'ttt'r,tl w(,r('v(,ryhappy in rcsP()llse thatthcy wert'very happy.l lrrrs rtnly 3tl'X, answered first, wtrsasked clucstion it appears that when the marriage question was asked first, responclt'ttls' thoughts about the happinessof their marriage were carried over into tht'ir generaliudgments about how things were going (Schuman& Presset 19ltI ) In another example, three groups of undergraduate students took P.rrl in a study in which they were asked about their general well-being. ()rtt' group was simply askedabout their generalhappinessand life satisfactiorr Another group was askedto write about a life event that had made them ft't'l good beforebeing askedthe questionsabout well-being.The third group r'r'as asked to write about a life event that had made them feel bad before bt'irr1i who wrote about positiv(' askedabout their well-being.Overall, ParticiPants life eventsreportedbeing more happy and satisfiedwith their lives, where.rs those who wrote about negative life eventsreported lesshappinessand lill respondents In both of theseexamples, satisfaction(Schwarz& Clore, 1983). from an early question or task into their answt'r carricd ovt'r consiclerations formntion for.r later question,resulting in an assimilationeffect.

1997containedtwo questions: A Gallup poll conducted in early SePtember "Do you generallythink Bill Clinton is honestand trustworthy?" and "Do yotr wcrt' generallythink Al Gore is honestand trustworthy?" Vvhenrespondents was honest and trust' indicated that he first,50% Clinton asked about Bill wortht and 60% then answeredihat Al Gore was honest and trustworthy indicated that he was honest arrd trusl When asked about Gore ffust,68o/" worthy, artd 57%answeredthat BilI Clinton was honest and trustworthy. lrr had to answerabout Clinton basetl the first order (Clinton first), respondents but in the secondthey could use Al Gort' on their own frame of reference, as a standard againstwhich to judge Bill Clinton. When this was the casc, Clinton benefited(*7 percentagepoints) from the Gore standard. Becaust Clinton and Gore at that time were generally viewed as a team, it aPPe.trs witlr adjustedtheir answerfor Clinton to make it consistent that respondents The result was an assimilationeffect. their answer for Gore (Moore,2002). Resultsfrom an earlierGallup poll (M arch27-29,1995)show how anchor' ing can also lead to a contrasteffect.In this poll respondentswere asked to and qualities describedNcwt indicate whether a number of characteristics Gingrich (then Speakerof the House) and Bob Dole (then SenateMaiority of respondentssaid tlr.rt Leader).\44renGingrich was mentioned fitst, 41o/o after Bob Dolt', when he was mentioned but trustworthy, and was honest he llavirrg llob l)olt'as trustworthy. he was honest and that only 33%indicated (iingrit lr llrirrking led to fcwt'r pt'opk' which to c()mParc a standardagainst ).L' l.t l whl'lt llc w'tr lht'oPprxik'haPPt'rrtrl ittrtltrtlstw()rtlly was hotlt'st
I isk'tl fiIsl,6{)'li' in di. , t l( \ l llt , t l ll( ' w. t s lt ot t ( , s l . t t r r l l l r r ', l w '| l l l l y , l r t t l w l t I t t (iin* r'ir'lr lru tlrrl llt r ' \ t , t t t ( l, t f t l , t llt it lr t wlt t t lt lr c r r ', t ', , r 't t r l ', t t t 'r l , {'\ '( 'l l l l l r t l r '

l(' l

I rri '\r (-)r r '.||, , \ . .

r r , \ ( - ) r r ' , I |r , N\ \ I il

( ,i '//l i ,/l r l ( o tttttto ttVi ttu tl Sl i "'l l /l l s

l(15

o l {( . s p( ) nd( ' nts s. li tl y t.s .I l o w t.r,t.t, w l rt' tr l h i s tl rrt' si i orr l ol l ow t' ti tl tt' r1rl sl r,,rr abottt U.S. reP()rtcrsrep()rtilrt ()lr trips t() tlr('S()vi(,tUni(nr, 73'r1, s,r(l !r.', Consistency Dillehay and Jemigan (1970)administered three questionnairespt'rl,rirrirrtl t. policies toward criminals to three sets of students. One was strongly l)r.r'.',, I , toward leniency for criminals, another was strongly biased t()w.rrd h,rlslr rr,, for criminals, and the third was constructed to be neutral. All of thc strr llrrt , were then asked to complete scalesmeasuring their opinions about cn n rrI r,I l , The results indicated that those who initially completed the questiorrrr.rir r' I'r ased toward leniency later displayed more lenient opinions toward crirr rirr,r1.. than did those who initially receivedthe harsh or neutral questionnairr,. llrrr'. it appears ihat these students, after being swayed by a biased questiorrrr.rrr, into supporting a very lenient stance toward criminals, adjusted thcir .rrr swers to opinion questions about criminals to be more consistent with tlr, rr original responses. Aa)idi tt s Extrantanass As part of an experiment, a group of students were told the study tht'y u', r, taking part in was concemed with language and the structure of spcer'lr. llr,, students were then introduced to the topics of euthanasia and reducetl t r',r irr ing for doctors through a questionnaire that asked them whether thqy st(x'rl for or against these controversial practices. Half of the students wt'rt. tlrlrr informed that they would be participating in a face-to-face interacti()n $ rtll another participant about either euthanasia or reduced training for tlot tor.. They were told the topic they would be discussing as well as the positi,'rr the other participant would take. The other half were informcd th.rt tlr,,r would be listening to a recording of another participant's views on orrr.,,1 the two topics, and they were then informed of their particular topit ,rrrrl the other participant's views. All of the students were then told that rrr,'r, information was needed before beginning and were asked to compl('t(',r :,'.1 of four questions on each topic. The results from these questions showt'rl tlr,rI students who expected to have to talk in person with anotht'r plrti( il',!rl about their topic rated that topic more moderately than thosc who crpcr lr', I to only have to listen to a recording (Cialdini, Levy, Herrnan, & livcrrl'r.,1., 1973).When confronted with a morc social situir[ion, rcspontlt'rrts,rvoir|r',1 presenting themselvesas extremc by nrorlt,r,rtingtlrt.ir .rnswt'rs. r,lhtls, ,r lrrly ol rt'st..rrclr In addition to thesequcstiorrordr.'r' lr,rscrrrt'r I 11., th a t s um m ar iz es t h c c ffrrl s rrl ,rs k i rl i 1 i t,rrr.r,ol url rnl .rry i l t' rrrs, sr, ,r' , l r rl "How woul, . l y ( ) r rr ' ,l l (' tl l (' o v r,r' ,rl r;r,rl l rty ' o l l rl r. l r yorrr' (rrl l l l tntl v " ' , l ' l or ,t n tr| l l ' ,' t rrl ' rIr,rtl tr r l | | tr,ri ttr r| l r l r ,rs hl r.r.l | ,,| | rrl t().r n( l , lll( ' r ' , r s k irrl,rl i rrrrrt l o .rtls , t r lr r t , rlion, , rr rrl l ' r l t ' rr l r,,l l " l :' tu ' l tr."l l .t\ r, l ol rrrl l l r,rl l l rl r,rrl rrrr,rrl ' l rr ' r;ttr, s lion lr , r t r l.lr , r I ' r,' ul tr,,l l rrrrl t I' t rr' ,,| ' rrrrrI r' t t| ,, w l r.tr ,r' rhr.r rrr',r l l rl l l I l rr' l r

drtmain wllt'tl'tsLt'tl tlrtt'slirrtts 'lltt r tltt'sltt'cific rplt ilir rkttll,titt 'tttrlltigltr'r lr)()5) (Wrllit:& 5,rlti('1, (l rr.slion\ ''';,;, on questionorder effects Ltcn conducted it ,i ;it.';'"c.rrch th.rt h'r's urr\l,rt('stlr('rlt'vcltrPmcntofwebsurveys'but'forthemostpart'onecan surveys as in mail surveys' 1..,..,; ,n.*' cfft'cts ttr be very similar in web can magnify their web-surveys lkrwr,vcr,th('way that surveyorsProgram presenting atl ot-th: questronson ,,, ,rri,rin'rir.'ordtir effects.For example, ihroughout the allows the respondent to scroll back and forth ,,,ui (much as questions ".r,,"n between to identify relationships .,,ru,'y, ,r,lotlngit "asier construction using page-by-page i. ,t,"'.ur" with a mail survey). In contrast, them through forthand back moving u,t),rr,lt('sthc questions and makes fewer expect would one case' this if it is altowed at all' L"I ;'l';..';,,;;;;-", to cay the aSit lS more difficult for the respondent effects, ttrder ttttt.slitln -ln fact' Tourangeau screens across **-nry of Previousquestions ,l u,,,lt, ,,n,.t when they appear correlated highly more showed that items are rt rrl.(2(X)4) screens' across-several sPread logr.llr('r()n one screenas opPosedto being to quesrelevant is irx,th.'r programming capability of web suweys.that quesin which and vary' the order ll,,r ,,r.ler effectsis the atility to c;ntrol' to order questionsin the same lk,rr,,,rrepresentedOne can presentthe same of respondents can vary th; order for randorn subsamples uit ,l '"p",i.r*o that are questions two of - .an rangefrom fliPPingthe order th. ,,rde, Vnrvi,'rq (alquestions the of to randomlvorderingall orderiffects ;;l;:,ii;; reason)' ";;i"ce the latter without good ;ltl;;,;|];;' ;";"r"lly would not recommend ' give-and-takeof regular connormal s,ltr*nrr'ilqso) detailedhow, in the take into account things they vrr",rti,rrrs,people tend to give answers that questionsare lesscomplete' hnvc alrr',r.lysaid l hus, answersto individual of those questions probably ,,i t,,". nLrt,,'krstand alone, than the writers larger with questionsthat imi,,,,',*,"U. Altlrough order effectsare probably that effectsalso occur evidence itt,"ii,ti'tt f,,llow one anothet therei; Iimited to recit arc widely seParatedConsequently' is imPortant wlr,,rt ,1,,|.'.ti,,n" be viewed as a compilation of ,fi,rfr,"".trfy on th.rt a questionnairecannot ''"A"p"rrtlcrrt questionsthat have no effectson one another'Each ,'i,r,jri"u'fy of its individual contentbut d r,|ilion|l .rstb(,(,vnlu.ttednot only on the basis mearung' contextthat oftenadds or subtracts trr tht'lar1;er llr,, wirlr rr'g,rrtl it like thosedescribed Previously' questions inclucling ,uu',tii,i.ip,rtt's Wlu,rr one with half the questionnaires tlldy lx' ill(li( iolls l() c()llsld('rconstructing findings of in reports recognize r, or nt a minimum ll with ,rrrothe lr,r ,rrrtl trlr.r. answers' on respondent order influence tll (lrl('sli(nr 1lp, Ponr'ilrilily ( l{lrAllN(; n ( ()MM()N VISUAL STIMULUS

people irrflucnces layotrt ittrr'l tlt'sigtr llow vistt.tt rr ( ll,rl)l('r.l rLr.r.rl Wr,rllr,,
g tti cl cl i n e s |tIrIi ttIi l t r sl l l vl Y ( l l l ( 'sl i ( n l h "l tr ' l w t' Pr l r vi tl t'tl w l t r , t l l r t , v ,| ||' |( 't,I) |

l('t '

|tr

n r ( - ) ! ; . , 1 1 1 1 1 .1 ,, r r ,.\

( _ ) t t,.| | ( | NN

\| il

{ / i1r lr , r lr / ( r t ut unuVit r ut lSlir t t t t lt r s lt 7 design. thotguidequestionnaire Flguro6.4 Visualdesigncor)copls

l hrtt: slagcsoJ visusl procctslng process quickly scanthepageandpreattentively lhtu, ldgt h|,t)ut: In lhc lirst slcp.rcspondcnls understanding ofthe basic hirsicvisualproperlics, suchascolor and size,to gain a general processing. This stage is dominated by bottom-up lilyoutoflhe page. by the information step,respondents beginto organiz lnlir'ntlltionott:anitalior:In the second from the individualvisualelements sogmenting thepageinto basicregions, differentiating grcupsor relationships The stage involvesboth amongelements. hackground. andperceiving prcattentive processing, move from bottom-up to top-down andaltentive andrespondents process'ng. individualquestions is on the taskof answering l (* &mpletion: ln thefinal step,attention wheresurvcyrespondents focuson a smallerareaof the pageandbeginattentively processing proccssing occursin this ofeach individualquestion. Top-do$'n the components prior knowledge influence intetpretation. stage wherethe surveycontext and respondents'

l o l t lr t . r , is r r , rpr l ( ' s (' ltn .t t i ()rr o l i rrtl i v i rl rra (l l u (,sti ()ns. l n thi s t.h,tptr,lw t,rrrn ti n u e t ht r t dis c us si o nb u t i n tr()d u c cl ti ti i ti o n a l vi sual dtsi gn corrccl .rts i o l rr,l ;, surveyors understand how rcspondentsperceivecntire qucstionn.rirt, P,r1i,.,, organize.informationon the page, and navigate the survey questionnairr,. Ensuring that every respondent perceives the survei questions in tlr,. same way is considered essential for obtaining quality iurvey data. I lorr evet creatrng a common stimulus for all respondents is far more difficrrlt rrr self-administered surveys because the respo;dents, rather than intervicw(,r.,. control the order in which they answer the questions and whether they Pr'., cessall of the questions and their component parts. Even though intervrewr.r ., are not present to guide respondents through the questionnai"re, effectivc I.r sual design can help respondents gain entry into the page so they will \.\,.urr to respond to the questionnaire and will understand how the informatio. i,. organized. In addition, visual design can be used to encourage respondent., to process all of the questions in the desired order and to naiisate betwu,rr sectionsand individual questions.

Allontion and visual processing

How Do ResrollorNrs M,cre StNss op Illpon\4anoN pnesrNrlp rN A SURVEY QuEsrrouNalns? When a person is presented with visual information, whether in a newspapt,r, web_site, or survey questionnaire, many separate actions take place vt,r.1. quickly as the eye takes in the information and the brain processes rt to m.k(, sense of the page or screen. The ways in which people perceive and attencl to survey information are determined by innate tendencies of visual infornr,r tion processing as described in recent work by palmer (1999), Hoffman (2(X).t ). and Ware (2004). People do not assign meaning to information all at onl. Instead, they process and give meaning to visual elements and therr pro1.rt.r. iies in multiple steps, although often quiie quickly. The way people proccss visual information can be divided into three stages: (1) understancting rc basiclayout of the page, (2) orgaruzing the inforriation on the page, and ( t) focusing on completing the task. During the first two stages, the informatiorr on the page is processed on a global level, and in the third stage, thc ftx.trs t: to processing parts of the page in more detail. In Figure 6.4, w,. 1u..oy"9 define the three stages and the types ofprocessing that occur ilr th.r,,stug,,. When respondentsfirst view a paper or web quc.stionn.r irc, th(,y (lui(.kll scan the page and notice visual properties, such as col,,r arrtl sizc, lo rrrr derstand the basicpage layout. This stage occurs mostly prior. to corrsr,rrus attention at the level of preattentiutprocrssirr,q., wh,,n. .,rr.,,.l,rirr, ,r 1icrrt,r.,rl urr derstanding of the sceneby noticinll c(,rt.tirrvisu.rl t.ol)(,t.t i(,s1t..1i, rrrrr1l,,,r, l) si ze ,s hape,c ont r as t ,t n c l .s rrrt,, rrrl .r h trr..rrrtli rrfr .' si l y,t.tr.; l l r,rt i ,st,' rrl.rrl . fro m ot her it r f ( ) r r na l i o rr o rr l l rt,P,rg t,.Irr,ItItIi ti ('rI,Il rrs sl ,rl i r.rs r l r rrrri rr,rl r.r I tty b o llo t / / / , / r i) r r ' ss i l /(, w l rr.t.r,o rrl \, l l tr, r,rs rr,rl srr,rtr. tl ,.r.l lrrr| | rrr.rrr r.r, rrrrrv i ttl o tt t t , t lipr r is t r . l r . i ' r.r I

Broad,rapidvisualanalysis of the entirefield ofavailableinformation l'tt ottcnti|eprocessing: to certain to in laterstages according lhat determines which visualelements areattended propcrties tbeydeviat from other and "standout" because that arenoticed subconsciously i|lfbrmation on thepage. andthe focusis wheretle visualfield narrows Conscious visualprocessing ,4ttottit'(processing: on r few elements that entervisualworkingmemoryandaremoreeasilyrecalled. duringattentive when respondents arefocused on individualsurveyquestions Ii,r(d/ r,i.,w: in width. pr'occssing, or 8 to 10characters their field of view shrinksto about2 degrees alone,and is quickly processed by the visualsystem lh,ltt,,?tt) lro&lsing: V isualinformation perception. itselfinfluences r)nlythc visualstimulus '11,t, and is processed based on the contextof the situation l n turssing: Y isualinformation '', prior experiences, thc vrcwcr'sculturalknowledge, andexpectations. into what is objectandwhat is t\tnl t), ir,t,tutir)r: Thc orSanization of the visualscene I i$utt /.t:i orderof anddetermining b:rlJkgr(nrn(1. dctcrmincd by the contounofthe visualelements, with figuresattended to first. rlrcnliofiduringprocessrng.

Irr llu' ittlirnn!iott urvtnizatittn stage, respondents divide the page into basic f r,l{i()lrsn( ( (fr(lirrg kr tht'ir shared visual properties, a process called segmentalirll. ( )rrtr. llrt' p,rgr' is rlividcd into regions, the contours and boundaries help rlistirrgtrishfigure frorn ground and differentiate individual llrr,rr,sl'orrrlt,rrl vlur,rl r,h,nr.nls ll),rl ,rlr.rrst'din ftrrthcr visual processing.Then, respondents rlls tl r)r1)rrg v isu,rlt'lcments,using the Gestalt grouping lrr.girrlo pt,rrr,ilr' plr'()r ( ( ir ouping inf or m at ion helps speed Iri l rr1r[' r,,r s, lislr r ssr r l ir r lr . r pt r , r '. 1. . r t 'xit m plt ', r cspondent s | | | | l r| rr| ' r, r ||r ll , t ||r l it t t lt t r r vr "' r or t t lr t t 'lt t 'lt sit t t tlir r r l 'r t t t t il, t t 'r r r t t lt it sllh, t l , t t r ' kx, t t t 't l cllr scin ttti gl tl l r| gtr r lo l'r 't r|'t t 'r 'r 'h't t t |. lt l'r (lllll(|tlr, [r rl 8r(rtllt' I ltin il rttl ilr-[relwt'etr lrrt)tlIllly, rur h dr d |r'l ul lr,rl|tlll|r!

I nll

I'R( )M L't rt !i

o Nr i tr ) AL l| | tn i

( r NNAtRl.

( )r?tri t rl ('tt fiut\t Vi::t!tlSlir!|ltr

ll''(l

sl,lA('wlr(,r('tlrt.r'lt'rrrt'rrts tll,tt .I.(' Ptr'.rlt(,t.ttivt,ly irr Sl,rgt,I ,rrrtl Pr.111,1,1,q11 r'.rrly in Stagt'2 art' now netiv('lynttondedto orrtl t,ognilivr,ly pr()c(,ss(,(.1. Mcaning is detcrminedby the visual scene,but lop-rlorrrrr irlstr lrr1)tssirrr begins, so that the context of the situation and respondents, culturrl knowledge, prior experiences, and expectationsalso influence the meanrng assigned to visualinformation. After respondents perceiveand organizethe basic visual elementsof the questionnaire, they begin the third stage,taskcompletiorz. It is during this stagt, that respondents shift their focus from the entire page to a much smaller are;r (the foveal view) for more focused processing and that they first begin to read the text on the page in any detail. During this attentireplocessing, rcspan_ dents sequentiallyattend to the componentsof eachindividual question (tht. question stem, any additional instructionsor definitions, the answer spaces and/or response options),and thesecomponentsenterworking memory and are more easily remembered. How respondentsperceivethe information in the questioninfluencestht. remaining four steps of the responseprocessas describedby Tourangeau (1992): comprehendingthe meaning of the question,recalling important in_ formation, forming a judgment, and reporting a response. The task comple_ tion stageis dominated by top-down processing,where the survey contexr and the respondents'prior cultural knowledge and experiencesinfluenct. how they interpret the visual information. For example,symbolssuch ascurrency symbols can have different meaningsbased on cjtural expectatrons, and symbols such as boxes and circles have different meanings in web than in mail surveys becauseon the Web, checkboxes allow multiple responses and circular radio buttons only one. In Figure 6.5,we demonstratehow respondentsprocessa questionnairt, by showing images of what respondents perceive in each of the three stages. During the first stage,respondentsnotice the basic horizontal and vertical layout of inforrnation on the page;distinguish changes in contrastand color; and notice the dark, medium, and light gray and white areason the questiolr, naire page (the Stage1 image in Figure 6.5).Respondents perceivemultiplt, dark gray featuresof different shapesand sizes(a circle,rectangle,and fivc small squares), three medium gray rectangles of different sizesat the top ol the page,and white spaces in the lower right and bottorn areasof the pagc. We show two imagesof Stage2 in Figure 6.5 to differentiateearly Stagt,2, where preattentive and bottom-up processingstill dominate, late Stagc 2, where more active attention begins and respondentsuse top-down prcct,ss. ing. In early Stage 2, respondents usethe boundariesand contrastto s(,snt(.nl the page into two regions. Once ihe page is segmr.nted, r(rspond(,nls l()(.lls on the darkergray and white elements and lct thc light gray rt,ct,tlt, irrtrrllrr. background (theStagc 2a imagt').ln thist,,trlystagr,, r(,sP()n(lcnts rrolict, t,r.t,rr
morc ('l(tme nts lo cltt. r J ir r r J if f t ' r t ' 1t , r r . r ' , t s 6l llr r , pit 9 1 , , 1 1 1 ,l1 lr1il.r,,1.yrrr1s 1r z r . " i ,

(tlrr,St,rgt'2b rlt,tPr,s,,rrrrl lnrlr',rsl. ln l,rk.Sl,rgt,2 inr.tge), responclents attend to lhcsc variotrst.lt'rrrt.rrls, thcm irnd using prior knowledgeand Br(rupirrg lxP('ri('nc('s t()lssigrrrn(\)rrirrg to thcm. Rcspondents distinguishthe header rcgion with thc logosidcrrtifying the sponsorand the contactinformation, the lil lc of the survey,and the instructionsto respondents. In the questionregion, perceive heading rt'spondents the section and instructions,five questions, (nr(.with four subcomponent (respondents parts, and the answerspaces may nlsobriefly notice the "office use only" areain light gray at the bottom of the

prrgrr.
ln the final task completion stage,respondentsfocus their attention on pt'rceiving and comprehendingthe words in the first question stem so they ciln then retrievethe relevantinJormation,formulate a judgment, and report in the accompanyinganswer space(Stage I response 3 image in Figure 6.5). 'l'lren respondentsmove on to focus on processingand answering the next {,lucstron. Once respondentscan easily get into the pages,they can then focus on nnswering the individual questions.The following guidelines help make questionnaires nurethat respondents to self-administered can understandthe basiclayout of the questionnairepagesand the organizationof information contained within them in order to process questions and navigate through thcm in the intended order. Guidctine 6.6:Establish Consistency in theVisualPresentation of Questions Pages and Screens), ind UseAlignmentand Vertical Spacing to HeIp ktcross Rtspondents OrgmizetheInformation on thePage Wc have seenmany questionnairesin which it is not clear to respondents whcro they should begin and how questionsare organized.In these questionnaires,respondents have to searchthe page to figure out how to navigate bctwcen questions.In addition to presenting the componentsof the questiorr consistentlyand creating clear grouping and subgrouping (discussed in Chapter 4, Guidelines4.10-4.12), displaying questionsconsistentlyacross (lur.stionnaire pages and screenscan help respondentsas they move from to questionand page to page. rlr.rcstion 'l ht' (lestalt principle of pragnanz, introduced in Chapter 4, posits that visunI inf()rmationthat is regular is easierto processand remember.In addiliorr, rcpcatcdobservationin cognitiveinterviews of respondents to both web ilrxl p,rp('rqucstionnaires shows a rhythmic quality in people'sresponse path.rns. l hc rcsp(ndent goessomewhat automatically from the answer space ol or)('(lu('sti()n k) thc bcginningof the nextappropriate question. Arranging (lu(,ili(nrs sirrrilar'ly with {hc prrts in the sameorder for eachitem helpsrerpottrlt'ttls lrc tttot' t'llicilrtt in sctlut,ntially proctssing the informationin the ( )tttt tcs|onrlt'trls r;ttr'slirrtttt,titr'. .trt' inkr lhc rhythm of rcsponding, devialhrrs llorrrllrislrrlrrr,rl olk.rrh.,rrl lo nrisul(l(,rslilt(lings. Iigtrrc6.5(Stagt'2b)

Li t:

',,', l:,,
'u)
0)

o
4

A
5 i

R Y
E

Fi
9rE
5..:

!
fr

o o
(!

AE ;i

w
*' ,,

$ffl dl
. "

a
tlj

7) t:

B2 Eg -g

?!l

JE .t;

E
; r, ;,i ,;
tl I '7,1

&
o) 'c o
(!

$
II

;.*

n,t' .r. 1,. 3,i

.g r!
q) (!

" ,.\

. ? {.

9
e qJ
I o o o st
Lr.r !

iu ffirtniiiili ri ti
i::r l'di

E
ro to
|'t tt

8 .E
I

.E9 3E FE

il :t
s

ig

rl
li

IE

;: !.:

il

ti
Ii |l

ii
I

d ; ID
tl I

iil$ ii, &liiiiuii

177

irir ti

l: i

::

170

\ t, tltr

)r

/ , rrl r, , l

\ t l,tl

,ltnLttll

l,

Fi g u r c 6 t i

l x , r r | lllo l

Ir

...1 it l,

t,

\,t

t,

r!, 1,t

,.t , nr.||

r, 1,,,rr,lrl , ,r vhcr t 't ht . r ' r 1, , , . , rr lr , t r l, lr lr r lr , r , r r , lr r ( l ir lr r t Jl I t , r 1.1003) ( ( ) r l fl .l l ' r,i 1... , llr r ', l|l( r \ lr ( i I I rr' , ,rI I 1 ( ) ns\ \ 'r 'r ) | r ( \ spaces, bet ( ) r L. I Tlo\ 'lng I , l l r l | ( \l , lr r r , \ lr ( ) r r 1, l l ,,rrl r or r cr1r ) ! \ 'nper p, r lle r lslion is slr ( sur \ , r . vs, it is helpf r . ll , , r silr m an\ ' \ \ , cLr ,' Ir r,, | ' { r' .r ( lr , 1r r t ''lionin t hc s, t t r r c placc on t hc scr censo r esp( ) ndent s know rr l l r, l ol o r Lr st lr t 'ir , r t t ( 'nt iononelchpagc. Als( ) , indcnt ingt hc'. lt s\ \ 'cr spaces Lndenr ( 'nt lr t he( luest ionst cr n,as sho\ \ 'lti11 Figur e6. 6, ' r r' .l )()ns ( r ', r ir . L( ) r ies r,,l t,| l l rr' l plr r luhf r ] t her eisonlYolr cclucst ionpcr pagL. bccauser espolr dent s 1,,r, lr.rrr. io nrove their cyes .lll th(' \\,av acrossthc scrL'en to percelle the 't t trl r.rl l , displaving r luest ionsconsist cnt lyt hr oughout t hc.qucst ionnair e ,r i ri ni n g qu( , st ions. r r r r t. ht l 'il cor lespolt ding. t nst vc.spaces r or cat egor ies ' r,,1 rtr,,rl l 1l rt 'lPsr esponclent or s ganizc t hL. inf onr at ion on t he page. lnd m ain Lrr ,r s(.n\a ()f order as thcv motr. icross (lucsti(ns. Perhaps even nrore rnl,or t.rrt, n \ ('rtic.rl.rlignnrent ht'lps orcotrragc rc.sponclcnts to processthe ,;rr, ,tronsin t he ir r t enclecl or cler , r nd dccr L- it scs f hc.likcliht r t d t lr at it em s will
1,,.rrr,rLlrr t tt'n tl l r r i sse d . r .ttt,lr ltrtt o.l: LlsL' Cttlor tttLl Ctt tt'Ltst Lo I I(lP /ifsl){)//r/r,//ls /{1rr!/ri.:r, 1r(' I rutl,tturuls oi ll( QttL'stit)I5 tutd frf Nrr.'r,{//irrrlrrl/ I)nllt llltou,:lt lltL'
'l | \\ ( r5l 1.la()\.

,. celbq2.1r,

: lto* dt,iilbl.ia

WSU .r . pt ce to go lo sgloot? Wolld yos say_.

,rnL1 Figure 6.6 dispJayt'xanrplcsof ir mail and of a !\,ebquestiorln.rirL, \\.lr(,r( Llr.rcstions art displaled c()nsistcntl! on thc p..lgeitnd across p(rgesto h(,11, respondcnts.ts theV nar,igatebt.tr\'ecltqu(,stions.ln(i pagL.s. adc lit ionk r di s p i a v i rrgg u t.s ti o n s c o n s i s tc : rtl r,, al i gni ng thcm i s.r f.rt,rr, r n - .ltool fttl in hclping rtsponclcnts to pfocL.ss lnrl org.tnize the qucstions ()lr ilr(. pagc (l-irli.r't'll, Holden, & Butler., 2(X)ii). !{hcn pr(,scntinll rnLrlti}rle Llucsii()n on th('p.t8c or scrccn, \\,c suggcst aligning rltrestiorrs |erticitll\,; so th.t ll), questiolt slentsitll bt'gin .tltug it citntnton vertici.tl linr-.. Th is yertical align nrr,rr I is consistcltt u,ith thc nattrrai re.rrlingrtrcierin ntanl languages,n.her-c,pr,,, plt'begJn in thc'trppt'r leftJrdnd colncr of tht page, movc horizorrt.rlll ,rcr.o, the pagt'. ancl thcn continuc this proccss \,erticallv t1or,r,n tht pagc. \\t,,r1.,,, sullgc.stplacing mor.. sp.lcc betl,r'tcnrluc.stions than betr,vtc,n thcir.sLrlrtorrl poncnt parts to rcintorce thc of the subc()mp()nent p.,r rts, (l is(.U.\( !l Erouping i n C luideline- 1. 11\{ . c c a n n o t e n p h a s i z L .e n o u gh the effe,cti ve trst,ol l ,l ,rri spaceln thc qucstionrrairefor helpiltg rcsp(ntdcntsorganizc,tirr., ilfor-nt.rt r,rr on thc pagc ancl gror-rp relatcrl intornrl]tion. O nc e t ht ' quc s t i o l t s te n ts .rrt' ,tl i g rrr.c oln tl tc lci t si rl r.rrl l l rr, l ).l S (...rr,.\\, I sp a c es c anbeloc itte rl a l o n t.trro th t,r.c o rtrn ro rrrcrti r.rl l i l ,.tLr tl rt.rti ,l l r,.rr,l I o ttth c ' P , r gt ' , . t s s lrr)\\' l )i n l :i s U() f(5 (\l .ri i r,l l ,) \l t,rrrrt i , \ \\,l ,r,t(rrr,l tri
tttg lltr',trts\r( r \ l ) i l (( ,\ ( ) t t.{ ,\l) ,) tr .,( . ( r tr ,r lo r r r .. r r L,l l ,,,t,, ,, ,,, L,r |, rrr,l , I j ,.r t,,r rr trl l , I i rr ' , r 1 1 I 1i ; , 1L , 1, ( | | I r ...iL ,| | ..i,.| | I ' ' I r ,r .r , r r r r | ,| , r r tr

t r 'l . r i s . r po 'cr fu l kxr l fo r l r r 'l p i n g PL \) p l ( '( ) r - tn n i ,/( i Ir i ( l Ir r r .r Ii ( ) |r .r r ( l u n ( l ( r . l . r r r , l l l r r , n r r ',r n i n g o fsl - r tci fi cr i su ,r l e l ( 'i r ( 'n i s( l i r l $ r 'l l t l ,r l ..r ( ) ( ) ",\\,l l ( . ) {) ( ) l ) L r r r l Lo l o r co n si stcn tl \, th r o u g h o u t th e .l r .r csti o r r r ,r i l L,.r n l r L ,l I r r ''l 'o r r ,l cn l ' , l , . . r r l r i r l t n ti f,,, r cl ,r tcd i n fo r m .tti o n a n cl c.tl t u tsf tl t( 't,l sl .. r r i .tn sr r cr i l g tl tr . , 1 r r , . t r o r r '. (o l o r c,r l t,tl so c( ) r ttr i L r u tct( ) r ( 'sl r ( ) n d a n In ]( r Ii \.]|i ( ) n .r r 1 ( l tl r t o r tr a l l , r . r r . r l . r 1 , pr '.r l o f th c r l tr csfi o r r n a i fc. l \ Il e r 'tc( ) l ( ) f c.l l l - to t b c L tscr l ,.( ) !ttfn st c.l t l ', \ , r i ( ( l i rtr i r a r sca l e \ l r r l l r l l r sl r .r r i r r i l ',r ckg r o tr n d ca n h cl p r L .sp o n d cn ts i n sc\ cr a l \\'a \,s. Fi r st, rl , r, ,rlr'\ .r f( r.li()n irr \\ lrich tht rt'sponctcnt can focus attcntion. Thc' ust'of , r 1 1 r , 'r . .l o ,r sr i sl Pr o r - r 'ssi n g i s r l i scu ssccl i n C u i ctc.l i n t.6 .E. Sc'co n d ,.t sh a d c( t l ', r , l r t r o r r r r rl ,r l l ( ) \\\ l ( ) r - th f L l so o i r th i te .r n sl 't'r sp a ce s,th e b e n e fi to f w h i ch i s , l r , r . , . , r ', 1 l ', 1 ,,r r .Ii n a l l r ;b l a cki o n tco l o r fo r th e te xtc.r n e a si l Vb e p e r ce i ve cl , , r r r r , r r r r 'l r o l r r l l r l l r sh ,r d r r l b .r ckg r o u n cl s so th a t r e sp o n d e n ts c.tn e .tsi l y se e r r r , l r , r , i l l rr 'l ( \l ( ) l l l ) ( ,r 1 tl r 'sti o n a n cl i ts cOD tp On e n t p i tr ts. I r l l r l , r .,l r ,r ,l i ., r ) l l ) l l tr '\ .l tr l g r ( \'n s n r r ,o ft( 'n u se d .ts b a ckg r o u n d cn l o r s rr rr\,\ lr.j,l, | , r 1, r rr r, I r , , ' , |, 'r r ! i l . r ! , r t.i ! lr r,l I j rl l ,r r , rr r' .r' l l ri i ,r ,( 'n l ( r r ( 'p l ( '.l si l r g n l r !l c.r l n ti n g . Ofte n ti m e s d a r ke r , , 1 l l r , l ,,r , l :i r ,'r r r r ,l ,o l o r ( o r ,r tl .r l kcl g r a r ') c.tn th e n b e u se ,d fo r o r r ,1 l r r 'r ||r Io ||l l ,l ||r 'r r r r :L 'r l l r t tl tt'sr r r vc\.( ) r s. l j o r e xa m p l e , .l r r ,l , ,,t I'l L r , r r r l r l 'r r r ,r ,,l l o l l l 1 q l r .1 1 L 1 '1 1 1 1 1 1 1 r 1 ,.r th ne t1 n r ,,,,1 , r ,'r ,l .r l r ' r l r \ 1 ,,1 r ,,i l ,r l l r , 1 ,, r .,( 'r ) l ) U t l ( ) ,1 ',,r r ,l r r r l

( i ! I i r t r r |r l L ,r 'tr rn t,tkc U r r , l r r ,l , ,.r r r l ', r r , ,1 l ,'r .l l i r .tr ,\ | | I | ( )| I I I , I I L

r l ' r l ', 1 ,, ,,r ,', ,,r ,, ,,,,1 r r L | i ,,r r l r I ,,l l ,,r ,! | !ri!l,l' ,,,, r l ,'l l l ,

l7. l

|tt

v ( J t t r ' t t r tN.'

l( r A ( - ll l"tl( ) NNAllll

( tt'ttli r n ( uunttutVi:;ntlSlttttttltts 175 or ll ()n('tinl(,(c.!i,., s()nr(. l,rrr)rrrs nray conrplcte nll of the sections they (iu) without corrsrrllirrg remaining nrrrrtls artd then leavethe sections to lrt, cornplt'ted oncc they havc compiled the relevant information). The first page of this questionnaire,shown in Figure 6.5 (Stage2b), has three the first region contains the title, sponsorship information, and rt'1;ior.rs: instructions; the second region groups several related questions H(.ncral (,rntl labels that area "Section A"); and the third region groups several rpaccs for agencyuse only (seeGuideline 6.10 for more information about visually deemphasizingthis type of information). The regions are defined tusingenclosure,where a large rectanglewith contoured edgesgroups the n,latedinformation. Contoured edgeshelp theseregionslift off the page and the objectsof visual attention (Hoffrnan, 2004).In addition, lightly lrt,eome shading is used to demarcatethe two regionson which respondents colore'd trt'edto focustheir attention,whereasno shadingis usedfor the agency-only Information to help this arearecedefrom respondents'attention. llcgions can also play a powerful role in helping respondentsorganize Information in web surveys, even when only one question is shown per page. pagelayout is used where a consistent l;igure6.6showstwo questionscreens dcrossscreensand related information is grouped into three main regions. 'l'hekrp region includesthe title and sponsorof the survey,the middle region eontainsthe questionand componentparts aswell asthe navigationbuttons, nnd the bottom region includesinformation about how to contactthe survey lponsor. Thesethree regions are visually defined by enclosing the related Itr,mliin a rectanglewith different background shadingand making changes ln thc fonts used for the text. 'l'hc top regionis repeatedfrom oneweb pageto the next and is likely tobe after the first screenor two, similar to the "banners" lgnorcd by respondents Incluclcd on many web pages. Enclosing the components of the question hl .r eommon region helps respondentseasily processthem as a Broup. In arldition, a colored background is used to encouragerespondentsto focus lhoir ttention on this areaand on the task of answeringthe question.Lastly, lht, lrotkrm region usesa white rather than shaded background to help de('t'r,rui(' its visual importance but still provides the contact information for NrFrrrclt'ntswho needit. Cognitive interviews have shown that respondents using this screenlayout do not recall wlro t ornpletetla 25-itemquestionnaire hnving sct'nany information in the survey about how to contactthe sponsor. ( jrlrlr'llrrr' ('rrrsislttttly ldantifytheBeginning and/or 0,1); of EachQuestion Srr'1,r,,, (l(,sigrr('rs.rlmost irlwaysusc numbersor lettersto identify Alllrolglr l),rp(,f l't,tvt'oftt'n becnrt'luctant to do thisfor two wcb tk,sigtrcrs $dllrrrcwrlrrr,slirrrr, ( )trr. (r bv -l),ll{(' n lsl ru( tiotr is ttsed, numbcrsarenot is llr,rlw lrt,tt ftrdrolts, P,tgt' rIrr,ntiotr is Dr',rl lrrrirttsc t',rr,lt rlttcsliotr irppcirrs lrrr'Lrrowll1p, wlrir'lr ttl.t,rlrrl

witllitt ,r lightly colorccl are encl()s('tl white nnswersP.lces Wlrt'n snraller or shaded area they tend to bc Perceivedas a figure, or tlle clements th'lt people focus their attention on, and the larger colored area recedesinttr ihe background (Dillman, Gertseva,& Mahon-Haft, 2005;Hoffman, 2004)' in white on a lightly coloredbackConsistentlydisplaying all answerspaces ground wili, therefore, help resPondents quickly and clearly identify where ihey should record their answers and can also help them identify the navigational paih in the early stagesof visual processing'Moreover,white answer more easilyfocus their mouse,pencil, or pen on can help respondents spaces to constrain respondents the space,and in paper surveysthey can encourage theiimarks within the answer spaces,thereby enhancing legibility and facilappear in white itating optical processing'Lastly,having all answer sPaces have completed if they to see responses quickly scantheir helps respondents all of the question(s)on the Page 2b) and 6.6 show a mail and web survey that both used Figures6.5 (Stage a black font color with white answer spacesagainst lightly colored green backgrounds(although the imagesare shown here in grayscale)' throughthe Useof in Regions Information 6.8:VisuallyGroupRelated Guideline ContrcstandEnclosure to easily proGrouping information into larger "chunks" helps respondents Dividing (Lidwell 2003) et al', cessand;rganize the information on the page the page into visually seParate regions through the use of contrast and enclosure helps respondents to perceive the elements within the region as more related to one another than to other elements outside the region Regions can be used to group related information, such as the title and introduction to the However, regionsneed to survey,or to grouP related questionsinto sections. be chosencarefully so they aid in the completion processlt is imPortant not to group questionsor other information into arbitrary regionsthat have little or no meaning to resPondents. contain sectionsof questionsthat are more related Many questionnaires to one i.,othe. than they are to items located elsewhere'For example, the in both English and Spanish 2010DecennialCensuswill use questionnaires another for English, and by and By using one background color for Spanish placing boundaries around the questionswith the different languages,thf i.gior,i b.co-e utt effectiveguide as to who needsto fill out eachpart of tht' questionnaire. Another effective use of regions is fronr lllt' 2l)-P'lU('U s l)('p'lrtnr('nl long Sttrvt'y, of AgricultureAgricultural llt'sottltt' M,ttt,t11r'tttlttl 't 2 5-hoLtr
( l l ) t 'r 'l l i ( n l s ( paPe r survey tha t as k s d( ' lJ il( \ l r lt l( ' hli( t t t r '' llr r r ll l l 'l r l l l 'r r ) t l [ r l t r l l i l l l l x r f l 'l l l l i l l l l l i s relatccl clttcstio rls illlo r t ' f ' , iolls , t t t r l "r ' r lllt t ' r t " t " 'r l r i 't l l y *111 yq 'y bcc,tttsc sot t t r . s |r li t t ' , , lo t t l t t r ' r ' r l lr r lr t 'l i l h r l I t l l t l t v s l ) ( \ 'i l i ( l y l ) {'s llf r , ll, l lr r , l l l l r r l t r t t l t t t , t s | ) , 't t l r . t r I t lr r r , , , r , 0l l,ln ||crs, ,trrrl llr r . r . r , r

; iiitEsg*si lalgl#iages+i+s

'11+i?i\tzl+ti1+111t{

i:1i11i2,,'11=tfttri1lE

=='i,.=?iii;iE? iiliE iiiiiiEgEiiiii ii i g; 7{zitzl trz Flgaii igglglgU liliiuiAEiaig

(-)tt.,tt,'N., l7l4 lrHorrr to,\(-)r r.,loNN \l1r rothertlrnnthe blackfont userllirr tlrt'tttrr.slions, ,rrrtl ,rn.hrt,.rlt,d orrlsirit.ol the answerspaces. In addition,agency-only irrlorrrration irr tlretlucstionrr,r ir.r, in Figure6.5 (Stage 2b) and contactinformationin Figure6.6.rrcpresenl(\l in a gray font against a white background and are located at the bottorrr of the page to help this information recedefrom the respondents'attenti()n. As theseexamplesdemonstrate, severalpropertiescanbe manipulated at ()n(, time (e.9.,contrast,location,and font style) to help deemphasize informatiol that none or only a few of the respondentswill need. In addition, layering manipulations by bolding and increasingthe size of a word or phrase cnn make the word or phrase appear more visually promineni, emphasizing its importance. For each survey, we recommend setting rules for when to use vari(tus visual properties (underlining, capital letters,italics, bolding, reverseprjnt, size of texL color,etc.)so that information is presentedsimilarly throughoLrt the questionnaire.Often a construction"rule book" describingwhich prop erties will be manipulated to display certain types of information can nerl' ensure consistencyin how visual information is presented,particularly f()l longer or more complicatedquestionnaires. For example,when redesigning the Agricultural Resource ManagementSurveymentioned earlier,surveyors createda rule book to guide the consistent and regular use of visual inform.r tion, which then could be applied by all thoseworking on the questionnairt, (Dillman et al., 2005).Many of the rules for visual presentationmay be tht, same from one questionnaire to the next; however, some rules are created for. specificissuesin a suwey and may not apply for other questionnaires, an(i thus constructionor design rule books are likely to vary from questionnairt, to questionnaire, Guideline 6.11:AaoidVisualClutter It is important to view the questionnairefrom the respondent'sperspecti\'(, and eliminate or reduce the prominence of any information that is not nt,t essaryfor completing the survey.We often seequestionnaire pagesdesignt,rl suchthat too much information is jammed onto the pagesand different itcnrs on the page competefor the respondent'sattention.Somedesignersmay trs,, only an 8-point font so they can fit more questionson the page. Others sr.,, questionnaires asan opportunity to promote branding and include logosarrtI text that prominently identify the sponsor(s), the client, the type of softw,rr| used, the secudty features,and other extraneousinformation on tht,wr.l, pages.Variously sized logos and imagescompetefor respontlents'att('nlion and often distract them from completing the task at lranrl. Thereareseveral waysto reduce cluttcron tht' p,r1ir.. lir.rlrrr irr;iIIIt'rrrrrrrIrcr
,,;1,q,,, o f i tcm s on t he pag c l n c l i n c re a s i n gl h (' n t)l ()l n t rrl l rl ,111l r,rrr l rt.l I rl sP()t t dent gl s in ( ' nt r y i n k r t l rr,| .rg r' ,rn rI o rl l .u r i z r.l l rr.rrr l r l t,tl tl l Itr,sr.nl l rI l rI 't tl tr'ttt .ll is lr , r llir ' t r l, trl y i n l l tr)tl ,l tl l o r' r,n l o \' r.,rr\,r r,rl (' ,rrrl r\ (rrl i ,ly ttru,, ot

( ,r 1 r l i ,,,(r l ( r tttttttL ttt Vt,ttl

Sl i u u l u ::

l7r)

o l l t r , r i n l o f n r ,r l i ( ) r l l t,r l ttt,tv tl t,tw ,tl tt'ttl i o tt a w i ty l ttr tl th c ta sk tl f a n sw e r i n g lltl tlrr,sliorrs.rtrtl lh,t I nr,rv nr,rl(' il rttort'tl ifficult to rcspond. Complex grouplngr rrl rltrcstiotts, sttr'lr as grir.ls atttl nratrixes (and particularly unnecessary

lltrn within grids or matrixes),also interrupt the flow of the questionnaire Enrllt'nrl kr introduccmorecomplexity,asPeoplehaveto connectinformation drfosscolumnsand rows. Also, reducing the variety of different elementson hclps to reduce complexity.For example,including five questions lltr. 1r,rge type may be fine for one Page,but including five questionsthat llr(.snnre ol prl lt'syrondents to respond in five different ways and that include different lylr's o[ t'lements (e.g., answer sPaces,check boxes, drop-down menus, oxl ,r slider scale)makesit much more difficult for respondentsto Perceive wlr(,rf to begin and how to navigatebetweenquestions. Arrotlrerstrategy for reducing clutter is to selectivelyemPhasizeonly a visual information is processedin llnritt'ri amount of information. Because n,ldti(nrto the other information on the page, it is recommendedto highllght or visually emphasizeless than 10% of the total visual scene(Lidwell of highlighted 2(X)3), as emphasis is less effective as the Percentage !l 'r1., and their variety of elements As a greaternumber and htkrrnr,rtionincreases. prolx,rti('sare manipulated, it becomesmore difficult for the viewer to disllnguish the oncs that deviate from the other elementson the Page (Ware, l(104),lirr example,if red is used to draw attention to a particular word or the it will becomemore difficult for the respondentto easily Process ;rhrrrst', OverIlrdng('irr color as the number of other colors on the page increases. tll, rt'ducing visual clutter improves the visual appeal of the questionnaire, pcoplc's motivation to respondand helping them to focus on the Ittltr,.rsirrg the questions. ldll ol ,rrrswcring of Matrixesand TheirComplexity l)nnh'liur o.! 2: Minintizathe LIse Mdlri\(,s r('presentone of the most difficult question formats to answer for Irirst, they require Peopleto match information in rows with lltfr,r,r'r',rsorrs. tltx,ili(nrsin trrlumns(or vice versa),a task that is quite complex.Second, lltr lr.rtrrcslto fill out a matrix and the instructions for how to do so are of the task. of the comPlexity in part because olh,n (lilli( ull to rurct'rstand, asto whether (1,llx,slructrrrc it up to the respondent 'l'hlt of the matrixleaves and fill in answersprimarily in columnsor rows or kr t,rviprk,tll(' rrr,rlrix of all threeof the previous because 0[ both. Morcover, r|I||r, (rrrtl)in,rli()rr ,rrt' rnort' likely to be missedwhen they are arrangedin tllllI rlli('s, ilt'rrrs in a vertical arc posedsequentially rlttestions l dlrl\r,! tlr.rr wltr'rrirrdivitltt,tl irrlrililrt'r' itt'ttt ttontcspttnst'. ittl', r'r,srtll Inyorrl,
l {r , rl r r l tl sr ',ttr l t o tt l l tl Atttt'r 'i r i ttt ( i r ttttr tr ttr i l y Sttr vcy h a s sh tl w n th a t a 't'l ,l l i l r tl sh i P l ( ) ( ) n e a n ( ) th e t l w 0 1 , , r 1 ',n r'r ,r l tt\ l l t,r l ,r sl r l r r t tt,ttttr ',1 ',r 'ttt|l 'r ,,r 1 1r', r , l l r r l ' l l v , . r r r ,l t,t,r 'l o r ,r l l o l l l r r ' l r l o IL ' l i vi tr l i i r t l l r t'tcsl r r tl tl cttl 's l r o L tscI t l , l I , r r ' , I rr, , ' ' , , t l 0 $ 'r 't ttt'ttl ||'\l 'r tl t',t' l 'tl r ' ,tl tr l l i r r "tt( .t i l |ttt tl o tttl .sl ) {l tl st' l r tr

I U{}

|ltr(}lt Qt

r.,

( ' N., t( , ,\ ( - ) tlr ., r } NNA| | il

( t r nliuI t t ( nnuut \ t Vt t ' t t l 5lr / , / r / / l/ s lul processinglroth horizontally and verl'0ltrllr( t(\l s() ,rs l() (,n(()ur.rll(.(.,rsy itnd ean irnl.rrovcthe quality of responses.An example lhnlly is t,sst,rrti,rl ttJ d ltrrlri\ rrst'd irr an intlividual-pcrson, general public survey to encourlf,t, llrouglrtful consideration of only two items of information is provided Itt lrlprrlt,('.7.n morc complete explanation of a similar matrix is provided f l ,wlx.ft' (l)illman,2008).Examplesof other effectivematrix layouts are provlrlr,rlhy l)illnran et al. (2005)andMorrison, Dillman, and Christian (inpress). lbulr g.z Exampleol a matrix format used etfectivelyin a householdsurvey on (x t|trunrly rssues.
l0 I'hri( thlnli for a minut sbout thre rclatives who do not liv with you but with whom you (oornrt lcrto mosl freqoently, rnd rosrder thesequ$tions. It may help to list their first nrme l nl tl Nhl n the l i n D rov i ded. Rclrtiv | Relative 2 Relative 3 V

so l l' t ( ' it c m st hln d tx ' s n s k i n g l ()r n ,s p ()n s (' s t() I)(.nt,t(l (' kr.rl l tl trr.sl i rrrr,. l ,,r one person at a time, as is ilcl'ti('v('tl lry placitrg gLrcslions lor t.,rrlr rrrlrrrlrr.r r,l ihe household in their own vt'rtical column (i.e.,not a rn.rtrixiC h('stnr I t, .'lloliI The latter method of asking for all information for one pcrs()nlt J I ir)rr'(l,r| | 1r,,,, the navigational path for answering rather than lcaving it up t() lh(' 11 :.1'r,rl dent as to whether to work vertically or horizontally to compl('t(' t lr(' n r,I tr | , Even with these substantial difficulties in mind, we pnrpost' rrlr, rn11 rather than eliminating the use of matrixes because in some survcys llrr. r rrr ptexity of the information asked for can best be communicated irr srrrrl,l, matrixes, and some survey populations are used to thinking abor.rtin l0I rrr,r tion presented in this format (e.g., accountants responding to establislrrrr,.rrl surveys). However, we have seen matrixes that required the resp()n(l('[t t,, remember as many as seven different specifications (through the use ()l srrl, headings in both columns and rows and additional definitions or instrrrt trorr booklets) in order to fill in a single cell of the matrix (Dillman, 2000a, p. .l.t tr Most people can only remember four to five pieces of information at or, r, and the likelihood of errors being made when so much information nr rrst lr, connected is enormous. The Web has had conflicting impacts on the use of matrixes. On thc orrl hand, the fact that many web pages are built using tables has encouragul tlr,. use of relatively simple matrix formats, although they are often excccclirrl,,lr long. In addition, inappropriate shading and the failure to suppress gritllirrr.'. may make matrixes difficult for respondents to process. For example, rrrr,. survey we saw recently had 19 items on a single screen, and each item w,rs to be rated on a scale ol7 (poor) to 10 (excellent).That's 190 radio buttons on or rl screen! Even though their overall structure is quite simple by comparisorr t,' some matrixes, the length of many such rnatrixes is highly burdensomt' .rrr,l tedious for respondents, and connecting information from rows and colrr rrrrr', is still quite difficult for some. On the other hand, the Web has decrcasul tlrr. use of very complex matrixes with multiple specifications for each colurrrrr and row, The Web is simply not conducive to such matrixes becaust'tlrr.1 would require both horizontal and vertical scrolling in order to fu lly conrplr. hend the questions being asked. Such scrolling adds an addition,rl layt'r.r'l complexity that has negative effects on response. Paper surveys, t.sptr.i.rll1 those used for business surveys, have exhibited a tendency to usc frrrnr id,rlrh matrixes in order to reduce the number of pages and costs of mailing. lrr,rs much as many business surveys that use matrixes arc now nratlt'.rvail,rlrlr. in a printable format or paper altemative, the reducctl tcntt'lrr.y lirr trsirli exceedingly complex matrixes on thc Wt'b is rt'tltrcirrg llrcil rrsr.()n l)!rl\'l as well. W hc n m at r ix t ' s ca n n o t b c a v o i d c d , to n s t i o rt s r.l l orl ' , rl rorrl rl l rr. rrr,rrh' lo c()n t r ( ) lt lr ( ' nt r nr lr t ,r o f l )i (' (r,s()l i n k )l l r,rti (n r l l r ,rl rrr.,l l rr' I rrrr r,,,,,r,r 1 rrr or ' tl t't k r P lov it it ' , t s ittl g l r.,rtrs w r,r' . l l rr.tr,,r.rrl\ t,,rnl l ,r\rl l ,, l l r,rl ,l r. Mrrrr' o rr,r,

v
llr|l n.|||c/lnftlsls l ol l knt{l )

w
(name)

1)t'|l hl i rol$l l l e l l v e tr Ys h l ,cw l rl r) trNo |'hrhl oo?... ..........,...,...

O Ys EN o

DYs trNo

l l nor rborl how fs r l,fiihton/( hrkrkrn do l i rl l l w i '.... ................... -Mi l es A l rorl hol l ok l l s l hl s rrl al l v{i '.........................

-Mi l es

A pl ro!knnl r' l ). how

tr l i v c ry (l ay tr livcry wccl

O Every day O Every week E Lessthan oncea

tr Every day O Every week E Lcssthan oncea

l omnrrrl ({l ( w l l h l hl s C l .c s sIhrn on.c a trl .l l rr?...... ...................

visil tr l ' rrs (nrrl nui l rorl |rnrnkrt r * l l h l hl r tr I' ,rs l ,rl D l r nl rrl rrhl l rr. w hl (h ol tl r( tr( fl l |hon. fnll"|rlrs roo tri( 'lo OI x l ., rel c phoD c trt0rlol l !||?. .,..,,..,,..,,..,..

tr tr tr E

Postalmail E-mail C el l phone Othertelephone

E Postalmail trCell phone EOther telephone

h l l h rrl .l l r( l r rrrr ol l hr r.nrr o' Irol rrrl ' na. or Irn| l l r rr ro||f r,l rrhr

O l es O t'1,' O l )(i ) rl n,,w

t r No
D l)on't know

tr Yes ENo
OD on' t k now

(Jucslionnaires" (pp. l6l-175), by \r,rrr, , I lrr. | ,rprr rrrrrl I'ryr lroIrpv o! ( r'nslnrcting l tr lnttnrtrt'tt,tl llurlhrtl tl ,\urvtv tlthnhthryt. Ia. D. de Leeuw,J. J. f A lltlltrrrrrr. l l r '\ , r r r l l l :\ I )| | || tIr|| | { | | |', ) ,.!( l l ) 8 N , r t\ \i 'r L l '\yr l x'l 'tIv l 't!.r \

Itl ?

lr t r or r Q r r r , .r to r' r' ,ro ,r (-)trr.,r ro l t rrri r ( ; ull) lil. lN l ,s w l ,l i tr()t( l )ti s t(;N tN (; Q t]l .s I t()N N A MA {l S . n N t)

t)trrrltltttt:, Mtiltttrl lNtblt6slirrrrrrrri /rUI )r.rt{,1,,1{ |r,s llJl} with v('rys p('(ili(dir('eli()r'ts to grritlt, lht.nr. lrr atlditiorr, it is rrotpossible to lnc()rpor.rt(, .lnsw('rs l()c,lrlier(lu('stions into Iaterquestions in orderto make laterclucstions clt'arcrirnd ('ilsicrt().lnsweras is the casewith web surveys ilnd computer-assisted telcphoneinterviews.Neither is it possibleto provide ilutomatic totals or feedbackwhen answers to questions are inconsistent with answersto previous items (e.g.,when current age is inconsistentwith ycar born). Also, later questionscannot be completely hidden until earlier questions are answered. In addition, it is easyfor paper respondents to peruse the entire questionnaireto get a senseof topics covered and length, which may be more difficult to do on the Web. Cuideline 6.13:Determine whether Keypunching or OpticalImngingand Scanning Will Be Used, andAssess theLimitations ThatMay Impose on gni ng and Process[ng Dcsi ion n a i res Quest (i.e., entering the data from a questionnaireinto an electronic Keypunching spreadsheet)has becomean increasingly significant cost for those usrng paper questionnaires. It is now possibleto avoid much ofthat costby using software to design, scan, and processquestionnairesoptically; however, the use of optical scanningplacescertainlimitations on the design of questionnaires. In thepast,optical questionnaires were not respondentfriendly asthey were not visually pleasing,nor were they easy to completebecause of the limitations imposed by the technology.For example, the answer spacesneeded to be printed in red or other bright inks that contrastedwith the black ink used for the answeroptionswith which they were grouped,and therewere limitations on which parts of each page could be used for answer categoriesas well as what kind of mark needed to be made by the respondent.Significant improvements have occurred in optical scanning design software and in the capabilitiesof scannersduring the past decade.In addition, there is no t'vidence that well-designed questionnairesthat can be optically scanned will obtain lower responserates compared to other questionnaires. Several optionsare availablenow so that responses canbe accepted on any part of the page(exceptforthe extrememargins),and respondents arenotas constrained in how they can respond. Efficiencyin processinghas advancedsignificantly as software has made it possible. to image the entire questionnaireinto a computer. The imaged information can then be processed efficiently by software that is set to make probnbility tlccisionson the quality of marks, acceptingthe likely onesand flaggingthc r.1trt'stioruble oncsfor a computer operatorto examine and make jttr,igrrrclrls alrotrllh(' K'sp()n(l(,nt's intcntions. Scanning cntirequestionnaires
itttprovcs lltr.sptlrl ol p1111'1,pii11g [)(\..1!ls('1ril1ypf tftt respgnsescan be , r lc t 'asily vicwat r lc f or Ir(xtss(' (l l ry llr r , sr r llw, ur 'r t ( ) lir ilt ) , , t r r t lllr r . ir r r , r gt 's vr,ti l i r,tli ott wlt r , t rt t . r lr onsr , r , ', ult ol lr r .f r \ 'ol'lil( \ 1.

The guidelines we havc pr('s('rrt(.(l uP t() tllis l.roint on ortlcrilrg thc rlut,s tions and creating a common visull stinrulus apply k) both mail and wclr questionnaires. We have highlighted how some, of thcse guidelines may o(, implemented differently in paper and web surveys; however, the common principles underlying these guidelines are the same for both modes. Becausc of the underlying technological differences in mail and web surveys anrl how people relate to these different technologies, we now turn to discussing guidelines for questionnaire construction that are specific to either mail or. web surveys.

Tr rr-: Mlrl

QuesrloNNArRE

When the first edition of this book was published in 197g, the possibility of using graphical design features effectively for guiding respondents through questionnaires was quite limited. paper questionnaires were typically pro_ duced by typewriters, which did not allow for bold versus light type and on which shapes (e.g., check boxes) were difficult to create. Vertical spac_ ing, dotted lines to connect text (e.g., opinion items on the left side of fage with answer choices on the right), and hand-drawn lines were amonq the few features that could be easily used in most questionnaires. Black-on-white printing was often the only practical choice. When the second edition was published in 2000, computers had made possible the use of varied shapes, fonts, and graphical displays. However, printing remained predominantly black on white partly because of the higher costs associated with color printing. We are now in an era when full-color printing has become widely available and less expensive, so even smaller organizations can use color in their questionnaires. Moreover, the linkine of computers with printers now offers the possibility of intelligent printing, whereby variable inJormation can be inserted into questionnaires while they are being printed so that each paper questionnaire can, in essence, be a unrquc document identified by an assigned identification number. The availabilitv of so many different possibilities for printing and improvements in the tech_ nologies for achieving the possibilities now give surveyors choices that wer!. considered impossible less than a decade ago and have increased the use of mail surveys in combination with other modes. Color can be used effectively in both web and rnail surveys, and rotating and completely randomizing ()r rcsponscoptions are also possiblein both nrorlt,s. questr(,ns l l owt ' v t ' r , r l, r il sl rrv t' y sl l s o h a v c c e rta i n l i m i t,l ti ()nstl r,rl rl i sl i rrgrri sh tl ri s trrtL , lr or r r ot ll' r s . l l p (\)p l (' ,l r(,to [rt,tl i n ,c tt,dl o (l i l l r.r,rrl l rrl l ow _rrprl ,rrrs tk'Pr, r r r lin6 r r |r r r rllr t,i t,rrrs w c r k ra Pn rl i (.u l ,tr(l u (,sl i orr,l l rr.t,rrrrrrl l \, 1)t(,\,i (l (\l

l11{

lrx()trt Qt

t : ' t t r r Nr '

lt|

( - ) l l.' ll( lNNAll{ l

tttrtili's lll5 \,lhh'lutt't' Mtil ttt'l Wd, Qrttslrr,l lnt I lr'r,itl,tttj in the long proct.ssirrg nature,which results bt,earrsr.ol llrr.irk'ss(r)nlr()ll('d interfere with ihe large marks that linc lftcr thc point rt,vt'r's,rl l.rrrxlucing reading of nearby .lllsw(,r spaces.ln addition, it has been found through that rectangles encourage intensiveobservationOfcompletedquestionnaires with nearbyresponse the useofcheckmarks thatproduce similar interference areas. A significani barrier to the development of effective optical questionnaireswas the entrenchedbelief that respondenthandwriting would be too divergent to ever allow the reading of most people's handwriting. In retrospect this turned out to be a caseof defining a problem incorrectly. lnstead, to write moreclearly,for examplebyprinting peopleneededto be encouraged The insteadof using cursive. use of narrow lines or tick marks to segmentanswer boxes into a seriesof connected one-characterspacesencouragespeoPle to print letters rather than use cursive (Dillmary 1995).It has also been observed that the use of colored backgrounds to surround white answer spaces encourages further improvements in writing by helping people stay inside for printed However, it is important that spaces the desired answer spaces. and numbershavegreaterheight than width to accommodate characters People's natural way of writing. Thus, good questionnairedesign has becomea words, and numbers. The way to facilitate intelligent reading of characters, to open-endedquestionsis farecordingand processingof longer responses cilitated by providing larger blank spacesin which people can write their responses in cursive, which for many people is faster than printing. Once the entire field is imaged into a computet a trained data entry person can simply processall of the answersto a particular question as a group, typing or coding one answer after another (Dillman, 2000a). Formats, andChoose PaperQuestionnaires in Booklet Gttidcline 6.'14: Construct Based uponPrinting andMailing Consideratinns I'ltysicol Dimensions 'lb put it simplt questionnaires should be presentedin conventionalformats thirt peopleare used to handling when reading severalpagesof information. In U.S. culture, the most common format is a book or booklet with pages that arc tallcr than they are wide and that people are used to opening and n,,rclingfrom left to right, iop to bottom (and turning pages from right to with pagesthat that deviate from this, such as questionnaires l('ft). Ir()rnrats rrrt.printctl on both sidesand stapled in the upper left comer or pageswith lirltlsarrcl unusualshapes, shouldbe avoided. nrrrltil.rlt' one of us was Wlrt,n workirrq on thc U.S. DccennialCensusin the 1990s, forms that could be unfoldedso that ,rltt'rrrativt'(nr(Lpirge ,rsktrl lo rlt'sigrr Alr'otrLlbt'stdnnt'clin a singleoperation. r l!,w(,fslo ,rll 0l llrr'rlrrcsliorrs (intlrrding folds, wr'rr, r,rlr.nsivt'ly tt,str,tl accordion nr,rrry lolnr.rl:. llrorrl',lr witlr a singlc r.olllrrlrlrllirht ir lr,rll,rr(l llx,n ltt r;tt,rtk'nl,,tltd,tbrxrklt'l r|| wt'll rtr ,t r.itttl'lr' lrrrolh'l(l )illrrr,rrr, ottl l',tgr,), tt tu' tr('tLt'rl I lacksorr,

and writing thatallctrespondent spaces ol snswer designs Flgure6.8 Graphical moreellective scanning makeoptical
fill the to complclely rcspondents Smaltcirclcsor ovalsencourage or Pencll with a Pen space thatarc less themakingofcheckmarks Smallboxesencourage for opticalscanning' difficulties andmaycause constained lo respondents encourages An optimallysizedbox (3/16 square) ofcheckmarks. write X's within thebox instead of thdesired instead people to makechecks encoumges A rectangle X's. check(or tick). theuseofa constnined A largeoval encoumges because cultures thana box in certain it is moreacceptable Moreover, a badmarkin school. with receiving an X is associated to write in cusive' respondents encourage blankspaces Undivided to for opticalimagesystms the resultofwhich is to makeit harder conectly. rcadtheresponse the useof spaces encourage Partially or completelysegmented whichcanbe teadmoreeasilyby optical ofcursive), printing(instead imagesystems. pinted againsta coloredbackgroundencourage spaces Segmented to rest ct marksto within the answerspaces respondents makeil moredlfficuhlo \4ritechamcters spaces square Seqmented ralio. heighGto'\tidth ,ninrmber. Lecau,.ollhe unnatural requiresunnatumlhandshifts' Too much spacingbetweenspaces thus making it more difficult to prcvide an answer'

a , d

rM1
II

FFITF.TA

|lElfJ:l
FI'I'ET6-I

EEEEg

Although different software packages require different kinds of marks' it one provides send is important to recognizethat the shapeand sizeof spaces about what kind of mark is desired. For example, as shown in u ^"""ug" Figure 6.8,a small circle (or oval) answer spacetends to encourageresponentirely with a Pencilor pen,whereaslargeovals tencl denis to fill in the space to encouragetick or checkmarks. Checkmarks are a Preferredmark in some with receivinga [rac]mark in sclrtxtl' of X's being associated culfuresbecause X 'ts is dt'sitrrl itt olllt'r t ttltttrt's'a an to use When one wants respondents tht' rtst'0f sttclrtrt'tlls llrrw|vr.t' 't sttt'tllt'rlrttx fr" squareb<lxenctlurigt's .rllr.rtItlrr"ltil" tlt|lsttt,tll as lt'sl)orrtlr.tIl:, nrarks of elrt,r..k tht,rrsc t]ncorrr,rqes
}l ' lrox wilh tll(, lrrrtt0r r rol llr r , r . lr t lf k r t r , rr ' |\ .s r r r lr rt r , t th n , l t t , l l t , t t l r l l l l t l t 'l l | l l l i ( l {

lf{ {r

lrttrrtrt(-)l t',ll(,N ' . lt ' \ ( - ) l' l"t l

NNAI I { l

t)rrirltlirt'r' rtt Mnilnt,l Wd,l)ttt,:litnmirts ll\7 lrttI )r'ritrti l ltissizt' brxrklt,t is more commonly intog" x l2"cl.rspt'rrvckrPcslolrrt,rilittg. for remakc's it easier .rrrriits largcrsizt'oftcr-t lry rt'sPorrrlt,rrls, cne()untered printer, one's options t() g() to .r professional sp(nd('nts to usc. lf one chtxrscs ,rre expandcd,and cven morepaper and color optionsmaybe available.How('ver,it is important to assess on printtheeffectofthe sizeofthe questionnaire in more detail in Chapter Z,Guideline 7.8. irrgand mailing costs,as discussed on Will BeArranged LayoutandHow Questions 6.15:Decide Cuideline Question F,ach Page Oncethe size of the booklet has been decided, the next step is determining the graphical layout of individual pages.In the legal-sizefolded format shown in Figure 6.9, the questionsare aligned in a single column on each page. If the larger size booklet is used with questionnairePagesthat are 8.5" x LL", the page can be divided into two columns so that the span of type being read does not extend entirely acrossihe page. The example in Figure 6'10 shows relatively short questions for which the two-column format works well becauseit allows more questions Per Page and contributes to more of eachquestion (seeGuideline 4.19in accuratereading and comprehension Chapter4). A one-columnformat can alsobe used with the larger sizebooklet,particularly when longer or more complex questionsare included; however, this format can be difficult for some respondentsas lines of text extend all the way acrossthe page and a larger eye motion is neededto move from the end of oneline to the beginningof the nexi. One benefitof this format is that it can with which are usually constructed morecloselyemulateweb survey screens, ln addition to the width of the screen. read all the way across questionsthat each using a one-columnformat, the exampleshown in Figure 6.11encloses questionin a separateregion, more similar to how questionswould appear web survey.This format was developedfor a mixed-mode in a page-by-page is discussed in considerablymore depth in Chapter 7. survey that All three of thesedifferent formats can be developedusing common word processingprograms that are available to most surveyors. The examples shown in Figures6.9through 6.11incorporatethe guidelinesfor visually designing questionnairepagesdiscussedearlier,although in somewhatdifferent ways. Overall, the most important decisionsto make when constructing paper questionnaires are how the questionnairewill be presentedto responclcnts(in a booklet format or not), what size the questionnairewill be, and will be arranged on the page. how tht'tluestions (irouping cluestionson one page in paper questionnaires presentsboth ()ne risk itics. problemis that thereis a considerable prol,k'nrs lrrrl t4rportttrr well from the I lr('y(lo ttrt e,rrryttvcrirrftlrmation llt,ttwltt'ttPcopk'tu rrr|.tg('s, suchas rt'latt'tl qtl('stions, is llt,rlsotrrt'litrtt's A srrrrttrl l-rlr'vious PitHt'. Prohlt'ttt
, r r l t i t 's o l rl t tct i cs,r l r o ttl r r ttr "s i o l r r ". p l ti l ttr ts,tl l l t r 'r tl l t'g c g r i ttl tt,t l i o tr , tr u 'tl to

i t t t l t t 's t ' p r o j t 'e t s ' l' , rv kr,, & Sclta t'lt'r',l() ( llt ) A v r t r t c wlt , ll lllls lf ' lt ( \ l P 'l r t i ( i P 't t t l ( [ l( ' s t i o l l t l 'l i r t 's that wortlcl ttol lt ' lol( l( ' ( l h' lllr l, r t t k ' t t t r l ,tlt t ' t otrst'rvittg 1'r.tg c s maps' 1.rt'oplc likc ltxrk (ltl('st nrlll ires i( lit into cnvclopt:s, tlotitul llt,tl !vltr'rr form lt is oriSinal trcat them likejmaps lnd arc Lttt.tblt'to relirld them in thcir than landscape format' ,rlso important to print qucstiolls in a Portrait rather reinforcc the vertical to helps it wide, it is When the paper is longcr than

format and conventionalreading Patterns' related ttr When using booklets,it is important to factor in severalissues and mailing differently sized booklets lt is imPortant to designing,Printing, "one usesbooklet construction,pageshave to be added or realie th"aiwhen paPer (i e 4 deletedin units of four. The decisionto add or deletea sheetof ' the quesprinting and for the costsof mailing pages)often has consequences although iioi-aire". Many officeprinters can now print and staplebooklets' cotunon paper size requirementsmay need to be met' For example' some two with (8 x 14") 5" paper legal-size on print questionnaires printers individso that booklei -ill ,id", iold, and then staPlethe resulting pug"" or, 'uaiqrrestiont "u.h uire (seeFigure 6'9)' Thesequestionnaires Pagesare 8.5" x 7" mail envelopes can then be foldeJlengthwise to fit into regular #10business for correspondence' used commonly (4.125"x 9.S"),whichis an envelopesize x 17" A more common format printi the questionnaire pages using 11" fit and booklet paper,which can then be folied into a conventional8 5" x 11" paper questionnaire madefromlegaFsize of booklet Figure6.9 Example
(8.5" x 14").

!t.

Dr--*Fr.l.-...-b ... , Y! !!

e s?,P3T#95?*'er.. bF-aF-r...F.h
1f uf u' t r dJ d4Fdd t fu,.*&'* _' k &* l: ibd&dI dI iI I L--""' {lr* ^arr4 |*'.,9b nhdtdqFEnE' h'd! I!

itil'-

rtt. :-fi!::ir:?5$f5?*-'.

: Lt",:fi.--*"*-**

lli)i

l r:, rrr(-)r t.. t, ,

( , \ t r r r . l, '

\ t.l

t,hl tl tl tIi

l ttt l tt

1' .nttt' .:A l i rri ,rrr,/ l \i i ,(l r/,

//,IL,/,I//,.

llil,

F igure 6.1 0 ltonnatre.

Lxirttr plr ol, r r |l) ||lr l, , , ,|||I ||||| |r i|{ |{ 'I |( ) | I Ii [ ) {} 1 )

I I l ( '( , k l '. 1 ques

] l o sl {Il o ( l l o o n l u l a te a p a g e - b yI xr r r ttp ||'r r l,r l ,.r l r ! ( l U r :J( ) r l r l i tl l ( ( Flgure6.ll ( l u o sl r ( ) r ) r i r r o ( 8 .5 1 1 b o o kl e l ) ;r r tx( xl r l xxl o sl u d y l ( ) r w o l ) l)ir()()

l" lbr.5 F.rdid.

hlE -.d...lolvliir.

t lrt tlltR.tu.l U$d.|l rly, rtrr ib yi! hoC. b i.? . Sdrn .a.

grb Unr*

Qt.

Aprrorl.i.t

t now mmY to.i

h.v. tou llvt{ h uro L.rbao|l.cLllnot

.r..?

Scn rto.*al - tLtrDd B lh.Lhb : 8.r.rat rd.ar*

lrat .?dr.ddda rs ati.. dolct d$.rd

Ql.

os..[,

nor ..tlftn.lt

tos rlth

llrhg h $rt rt .?

7. I*ldg-qrFc|h lcr. sa*hc|3cir (rr6nyuic. ..aane rffi$

O sdEltEtsdf.d O SodE*tat dilra't rfi.d O very drs&fi.d

qdt d ralrle. a ll.rnJ[Fqr-t a-na j rir a nrfi,lfi|tticad a{?

:: cd
Q3. flo, .tttct d do you S!.d to tX. l..dsto|r-Cb.rtb. ..6?

L It s|li.l !sr!rlral a lttl-i.,o.!.d-(.d rrddFo r6aL bEnlrPr&nmdllry,|.!:.r o,rntrD( Mld.t

rryo! !* lr..talar tu, r rlre tor,ro9a!rl.a tdl

O So@*h.a atl'd'd O srgnw.{lad'ed o, I{.r ,t .rr .(dld

tLlrodi!-r...
!ol|*lt|6a.lnl

tnc..*h
torrt iod Qa. l'l|'{n, ri. p|.tfvc h.r !.(o|!|. ft . pL{! to lhr.? o sdwn r b.tt r nr.rdr t6tt r d worr. do you $lrt lrritto.r"cL.l(.{r,n

., ftidT.!.a tu tEla! h $|.n yo, n3r lvrd calr9 lra Fa F... EEF, rbb r-,.,

O. S{6+slwe

y.n.onad.. yqr.l5 & Oo

!.rlll.ais

Can|r

Qt

flat, mudr b.tLr

or vr'rt.

do roo $lna $.

Ldl

dow

n5 b.com

i. th. p.i

fiv.

..

O s*erEt O. Sdrqlrt

a.tt r 6'!.

Q5. tlolf muct b.G.r o. !ror*

do to! think

O sdnwh.t b.u.r O sdBh.t @lge

"loul.r'c:"Thc lnflucncc olCraphicaland Symbolic LanguagcManipulationson Responscs to Sclf-Administered by L. M. Christianancl D. A. Dillnran. 2001. PuhliL Questions." Oltinion Quarterl1,.68( I.). pp. 5lt 8l.

be grouped on one page or in one scction. In thes(' situ.rl i()lrs,il is irrrP()rt.trrl t() desig n qu e.stion na ir es s o t hat t he ent ir t ' s c t lt r t ' nr 1' ( i r r l \ ' \ i r '\ \ 'r '( l ( , r ( ) n ( ' l ) , r l l ( ()r on t\,vo f,tcirtg pa gt ' s . llt ' qt r ilir r g t c s Por t t ll' r r t s l t l l U l 1 r l ', r r r , , , r r r t l r i r r l l r t . s r . \ (' rlllr'nC('\ ()f s(\'li{)r r \ ( , lr r ir r l( r ' r ' r I l llr r ' r r r . r r r r l r r l,, r , I l l r ' r | | , , | , ( | | I L , t i , l ! l l r 'n sct l' t.tl t'r'|.rIr'rIi1 rrr' , |r or r ' , r r |r ' , 1lllI ' r ' r r I r r , , |( ' ||( , r |, I '. I r , I rr r rr , l , , ,r rr r r ' \ ' ,. l l r r . t

Sur\\t)rk tbr Randon Household \,,ra,, \\rll rNlr\.(l Nl(xic(Mril,lnlcrncl)Pft)ccdure a. n dA O'N ci l l . . . M . C l h r i sr i a n . . Sr n ythL \ t . \ . ,, , 1t l r r ( ;( |l r r i r lI'r r l r l r r '"h r l ) ,\. I) i l l r D r n ..lD of the Amerlcan ul lhc llnnulllconference 'l)(lfi NIir\.\.\\ I )rl(r|l\. I \ l'ir|]!r l)f\rscnlcd l ir'r l 'r l Jl r (0 l ,|l Io |l l {r \e i r tr l r \ r r r r . r t r , ,r

l()(l

ljRo\t

( - ) t t . , l t { r N., tr | ,\ ( - ) t | ' ,| | o NNA

{l

titiltlittt"

Mtil nu'l Wrh lnl'| ilir)rr,rrlirr's l()l /rrr l)r'r,i1rrirl,1

ttt't'dto scroll togt'tlrt't on so tlrat rt'sltotrtlt'ttts arc oftt'trgrottP<'rl (iuitlcline 6.23for a tliscussion (st'e of thc down to vicw thc latcr (lu('stil)ns paging of wcb surveys). In contrast, placing information on different pages allows items to be separatedwhen there is concern about respondentsreferring back to earlier questionswhen answering later onesor when surveyorsprefer that one question does not immediately impact the context for another related question. In sum, it is important to contemplateindividual page content as both a problem and an opportunity for which a balanced solution needs to be found. Unfortunaiely, responding to this challenge is sometimes made more difficult when using booklet designswhere two pagesare often viewable at a time; however,thesefacing pagesare more conducivefor complex seriesof questionswhere multiple pagesare neededto presentall of the questionsh the sequence. Contrast, Size, Proximity,andPagination Guideline 6.1.6: UseSymbols, to Help Respondents When Designing Branching Instructions Effectiaely Execute Them Correctly One of the most significant differencesbetween web and paper questionnaires is that branching instructions for paper surveys must be executed by the respondentso they need to be made explicit. Getting somepeople to skip not to questionsthey should not answerwhile convincing other resPondents skip ahead remains one of the most perplexing problems facing rnail survey designers.TWo experiments-the first, a classroomexperiment wlth 1,267 respondents; arrd the second, a field test of about 12,500households in the 2000U.S. DecennialCensus-have now been conductedon how to achieve this goal (Redline et a1.,2003).The first experiment involved a specially constructedquestionnairethat was four pages long and included 52 questions, 24 of which contained branching instructions asking respondents to answer different follow-up questions depending on which answer category was initially selected. The secondexperimentwas built basedon the lessons leamed in the first and was embedded in the Census Long Form, which contained 53 questions,19 of which included branching instructions (Redline, 2005). Dillman, Carley-Baxter, & Creecy, In both studies,a combination of factorsdesignedto get people to follow branching instructions correctly were tested. One procedure, the detection method, involved tsing an arrow symbol in a large,dark font to direct people from their answereither to the next questionor to an appropriateinstruction W<rrds were used that was written in a larger,darker font than the questions. who shouldand to remind resprontlt'trls at the beginningof the nextquestion "lf Ycs").A sccondPtrxltltrtr',lltt'/tlt,r'ttliott who shouldnot answcrit (c.g., it's ttt mcthLtd, movtd thc lrrswt'r boxcsflorrt tltr' lt'll ol llr' ,rtt,rvr'tr',tk'got trrttltllrr'lrr,tllrl *'tllrltrtltr'r'klnrtt(l('nl'. th(' riglrtso th,rtskip irrstlrrr'tiorrs

designsused in classinstruction ol dilbroDl branching Flgure6,'t2 Examplos Census censustsl in the 2000 Decennial and national roomxperimonl
Clrsroon Error Rste l%. CeNus Error Rate tY!)

'l r(!lnrenls:

1
l,r.!cntiotr method: w.rning; shifi answcr h,x lo right; and larger. !, ttd

dld tilt Fnon w.tl tt . 1 LA5IYIAN,1919, loh or hdr*.t rt jr drn ? O ves O No+ co iort dld tlt ,.$i tro* b l!!,9 b. tlo{ noy r.tt

22.3

23'8

b .tr.d( ibr . '60 ro' lrlrudrd Atr.rnLi: i.dn!.. d!. qEd.. bdd. .tt yq .Ba al .t . . l^'r YE r, ItNl, i[d ti& Fs rr m, ri.? Ft d lrasn|e i.t! drd |na ,.rd r, b O . > C o FS t srl t. 1t!tt

merhod: o Dere(rion
Amws.largerand holdcr fonrs plus verbsl feedbacl

irfrH-t3:#ort'ton*!'t't'

r O yes lo 3t * O No+Go dr h dU rhh F{r b, 0t y*l Hd nE!, r..lt 1*tr aqnt ea.J 6'fJq\ o.k! tt* bE .d atibrv

Note: Eror raies shown here include cornmissioneEors (notskipPing aheadwhen direcied) and omission efto6 (skippingahead when nol directed).Advanced verbalwamins uscdonlv in the classroom Expcriment Slari;idlly significaDtdiffemce in total ero. ratesbetwee. the conlrol fom and each ofthe lwo mclhortt with o<.01 for both xDenments.

wrlh Pcrli)rmancc VisualDesignon Respondents' of Alternative Jbrrce. "The Influence { l) llctl(pp. l9(r). by 179 in Self-Administered lnstructions llranching Questionnaircs" D. DillmJn.J |ltingc. ilnd Ii R. Groves. linc andD A. Dillman,in Srrvqr Nonresponse. Hoboken, NJ:wiley. l.ittlc(Eds.),2002,

foveal view when focusing on the answer box. These instructions were written in larger, darker print and were placed against a darker background 'l he control instructions were written in the same font as the questions, as slrown in Figure 6.12. In the Decennial Census experiment, the detection method reduced the errots (\'e , ltroportion of branching errors significantly, lowering commission rrrrt skipping ahead when directed to do so) from 71% to 1'3% and omission r,rnrrs (i.c'., skipping ahead when not directed to do so) frorn 5% to 4%. The l()l,ll (,rror ratcs (commission and omission error rates combined) were lower lirr both the clctcction and prevention groups compared to the control group, whr,rc t'rrors wt'n' madc nearly 26ul' of the time (see Figure 6.12). The concluriorrs tlrirwtr frttnr tht'se experimcnts were that both of these methods were follow thc desired navigational path. They rcspotrderrts r.llirtivc irt lrt'l1.ring (' ()f lh(' stlbst,lntiill('ffectsthat the combined use of symrrlv r pn rvirk' t,virlt'n( pr ow or r lir r li,sizr ',r r r r lr asl, . r t t t l plt r xint it y cnn hav( 'or r t ask- or ient ed Irrrl s, m t lst liot r s wor t lt i it r slr t t t br , t t t t llit t g llt , t l , t lst t t t 'r 't ', t h'r l r |st ', t t , lt llt is rr.ri si ng. ( llt 'dlir lt ' t lx'r r t 'xt ot t r 'l) , lt I 'lo lr ot t t lr llt t r lcr l llt cy l l Lr.l v l t t llnot r r l r vlr lt r

1611,11111 l()2 liH(,ltQtrt :, ( rN., tr r .r Qrrl ,,111 'lhest' ct.ll.,2(X)5). fintlirrgs h,rvt' lt,tllo lh('prnctic('ol rrrakirrg slrorrg tlt,sign ()r't effortst<> avoid h.rvinH the last tlrrestior-r .r p.rg('branch any rcrip()ltdcnt somewhere otherthan to the first tlucstion on the next p.lgc,evcn if a slight reordering of questionsis neededin order to avoid that situation. Guideline 6.17:Create Interesting andlnformatil.)e Flont andBackCoaerPages thnt Will Hare WideAppealto Respondents In the 1990sa national survey being designed at the U.S. Census Bureau was being proposed as a replacementfor the DecennialCensusLong Form, which every 10 yearshad collecteddetailed demographicinformation for 1 in 6 households. The long form made it possiblefor virtually all jurisdictions in the United States,even quite small ones, to obtain information about the education,occupations, commuting behavior,and income of the people who lived there.The new survey that was proposedto replaceit was to be mailed by the CensusBureauto severalhundred thousandhouseholdseach year.The large title placed on the cover was "The Continuous MeasurementSurvey." When it was pointed out to the director of the CensusBureau that the title, while technically correct,was unappealing to most people, a decision was quickly made to changeit to "The American Community Survey," which it remains today. Often, when surveys are designed, they are assigned functional names that may describe the purpose from the perspective of the surveyors but that have no meaning whatsoeverto respondents. Titlesneed to describewhat a survey is about but also have broad appeal to the respondentswho receive it. At the same time, care must be taken not to use a title that makes the questionnaire appeal to some !4pes of respondents while being a tum-off to others.Thus, one would not title a survey on political views "The Problems with the Current PresidentialAdministration." That would seem likely to attract people who do not like the president and turn off those who do. This challengeof choosing a title and cover page that appealsto sample respondents was faced in a recent survey of a small metropolitan area, the focus of which was to understand key issuesaffecting the quality of life in the area and use of communication technologies, particularly Internet and cell phones (Dillman et al., 2008).The title of this survey became"Lewiston and ClarkstonQuality of Life Survey,"as shown in Figure 6.13.A brief statement describing the study in general terms as well as contact information for the sponsor were also included on the front cover, as was a color picture of the region depicting where the two communities meet, an image that is imrnediatelyidentifiable to anyone living there. The back coverof the questionnaire thankccl rcspontlt.trls lirr (\)trrpl('tinll the questionnaire and askcd for any atlclitit:nal <onrrrrr.rrls ,rlrorrl llrt' slrrtly. The address to which rt'spontt'r'rls slrorrkl rrr,ril llrr,r;rrr,rltrrltr,titr.w,rr,tls() n'pt'att'tl on thc b,rck p,rgr'. Arlriiliorr,rl trrIrr plrolor, ol ollrr,rk,y ,lrlrr\'ls r)l

r,)ttiltlitrr"' /rrr llr'r'irrtirit M il tt,l Wt'ltlrtt''tilrofr,,rliri's l()ll wt 'r c st 'lt r lt r l lol t lt l b, r tk c( ) v( 'rt ( r r ( ) lllit lt l( 't lx', lPlr t ', t llr t l rotl r trrrt r r r r ut r it it 's sUrv('v t('sp()11d(tlts. It is inrportant kr dcsign tlut'stittttn irt'c()vcrs thilt Itxrk itrterestitrgarrtl kr irs widc a spcctrum of the survey ptlpulation as possible l ht' trrvcr ,r1.rpeal can understand, (b) iclentily t lrt' should (a) have a clear title resPondents PnBcs sporrsor, and (c) encourage respondents to want to oPen thc qucstiotrtr,rir(' ,rnd complete the survey. It is also important for the questionnilir('e()v('rs to connect to the other implementation materials, as is discussccl for wt'lr opcning screens in Chapter 7 (Guideline 7.25). t',uidclinr 6.18: Auoid llating QuestionsSide by Sideon a PaXrro tlnl lkspondentsAre Asked to Answer Tuo Questionsat Once lrr Chapter 4 we discussed the importance of not asking respondcnts kr arr swer more than one question posed as a single question. Here w(r nr(' cr)rr ccrned with the related, yet distinctly different, issue of arrangitrg two or more questions on a Page (i.e., page layout). Questionnairt'tlt'sigtrt'ts,ttt' often tempted to ask respondents to simultaneously answ('r lw() or rrrot('(lil f('rent questions that are arranged side by side on a pa8e. lir sotttt' t'r h'ttt llt is t' tk'silitlt'rsrl it I practiceis a holdover from the days when paper qucstitttltr,lit everything possible to reduce questionnaire page lcnglh by sitrtttll,tttt'rrltrly introducing two questions that used some of the sanlc itrlot trt,tliotr,rlrrI lllt'tl displaying them so that resPondentswere encouraged to Ptr)(('ssltrrllr irl lllt' irr w'lrlrlt l (ltr('slirrrr $ametime. For example,Figure 6.14shows a multiplc-1.r,tt women in farm and ranch families were first asked how oltt'rr tltt'y rIr r',trlt plr t tt lltotr' nnd then were askedif they would prefer to do eachiob less,tht' s.ttttt', ,r r'('slx)rr(lt'lrl re(luif(' Such questionsexhibit two problems. One is that thcy hr make a choicebetween toggling back and forth betwecn it('nrsor.rrl\w('l ittli each item over again to itlswt'r Iltt'oiltt't onc question and then reprocessing path, whereas others go a clifft'rt'trtrottlt', itt one respondents choose Some (.tisence exposing themselves to a different sequence.ln aclciitiotr,lrtrltrt'ttl tr.stingof such questions has led us to the conclusion that th('y an'Llillit rrll In this particular ('xnnrPl(', I l'1, and morc likcly to pn)duce item nonresponse. in lcast one ite'm rcspondcnts left at of Question 11 (frequcncy of tloirtg lltt' job) unlnswerccl,lncl fcwcr than 1'2,skipped the qu!'stion entircly. Ilowcvt't, j5'2, of rcsponclt'nts lcft at leastont'itt'm in Qucstion 12 (prt'ferrt'tl fnrlttt'ttr'y) ttnirnswt,tcd,itntl, r'vr'tr with th('ildditiotlitl remintler kl go back ittttl ittrswt't 5'l' , skippr r l it alhr gcllr t 'r . l l ri s r;rrt'slior r Alllrotrgh trtrrlll)('r()l Pilll('s is l('ss ()f il c()llc('rll with wt'[r stlrv('ys, llx' t lt 'sigt r t 't slo Irol i zorrt , r ( l )| i (' tI I,| | i()tl ( ) [ t t r t t t Pt t lt 'rs( t ( t 't ] s sonl( 'lilt l( 'sl( 't r r [ r ls l l l ,r{ r,rIr( . slionssir lt , lr y sir lt ' in t llis t t l, t t }t l( 't .lllr 'l, r lgt 'r 't lt 'sigt t isst t t 'llr , lt I sit t ; ilt 'r 't 't lit , t lr t , lvipll iot t , t rl i srrrrrrir l', rllr . r is I r ', r lir r . is llt r . t lt , r it 'r 'lr r lr 'sigt r , t r r lv r t t t , t llr r l t r lolL ( lt r 't lr r r t t h't t lr , t t r ' r t , l ll t cs1lot t r l l r tl ttot t ; ilt llr r ' t ( 'r li( t l, tit ( ' llr , ;r,r r,rr tl t.rt tlt ly I lxr '! r k ( lt t lsliot L t t t ||t l t I I I t ' t tr Ir'r I rrt rIr' t'

l' )l

,'..t()r

,,

I ut'l'

ltlt'

! 't I \'

F ig u r c 6 . l 3 I x ; r rtr l) 1 r ",r ) l r lll| r ,,,lljr l,Ixl (los q r r o ( l l o i r p l x t i t l l( ) :,il r l) [' r ' r lx:t:l

itl) lxltl

l (J l r()nl ,l l l {l l )i l (:k (;()v('r l );t(l (]:j

1i gtfr ( !

6.1 3

\l

t rt t l t t t t t t \ l l

Lewistonand Clarkston Qualityof Life Survey

Thanksagainfor completlngth,s suraV!

It yo! hlva .ny .ddi5on l thougtrB .boi* any .f th theh hr. .bove lopicr or the $rrvey ittelt, plc.t.h!r

soci.l.nd

Economi. s<ience Rsearch cGnte. W.rhington St t Uhive6ity PO aox 91801 Pull|tt.n, wA9915+t8o1

.: "Will a Mixed-Modc(N{.lil,lnlernet) Pfoccdurc work li}| RandomltouscholdSLrr ^\'.)r//r vey sol.lh eCen eraPublic l l b) D. A. Dillnun. . l I ) . Snr ) , l hL - M ( h f i s t i r n . r r ) d A O 'N c i t l . 2(X)lt. Ma),.Ne$ Orleans. LA. Pitlcr lirc\cnlcrlut thc rnlrrirl torrli.rL fL L ot lll(. \rrencitlr ' . \ ' . 1. 1. A *,'u i.rti,'rr,'rr Puh lr .( ) l\ ' r ' r . ' rjl,

l r r r \ \ 'r r

( - ) L r '.r ( ,\\\

rrl

, . . . I I r,rr r , r , l rr . ri I r'. l l r ,' .r r l r cr r l o t l tl h a s b t't't- ttu r n ctl u p si d e d o l vn b y lrr 1 l l r , | | r l r , ',1 fl , l r ,'r r ,'l r r r 'l r '.tr tr |t. rr,l,.rrrr,t N r ,r v l tt( ) r ( ' stl r \'( '\'|s th a tl o vt'r a r e l r e 'i t- tg sttl r r '\s l l r r ' l r t',r r ttt' o l i l , , , r r , l r r , l , ,l l ,,r .r r r '., ,'l tl r , l ,'u , r , l o l r r .r r l \ r l l l r

r 'l l r ,r l ,'r r ,, r l t L r ttttr l r ,,l r t,,r r r .r n l 'r r r r ( r l ) l ( .l r |r o tr r l tl It|,Ir ttt

t v ll

l tt( rlt t-rl |.

( r N" to

( _ ) l I,i

T T NNA ( I

(;titltltnr',/"r llr,,r/\'ll|,t Mtil tnl Wtb llu*l tiutuit t:t ltlT Ics1rrtttit'nls, atttinr,rrV r:,r't,, rlo rrolrrPrlatc tlrt'ir[rrowst'r's),.t til]l)irlll P0int hirslrct'ttreaclrt.ti lrtr rrr.rrll As the ex.rrrrPl('s IIrlorr1iIrorrt this PoIrrl,rliorrs. btrrk hlvt' showtt,s()nr(' slt\'(.ys(iut now be donc cntirolyorr tlrt' Wt,b1,i,;16out c()ncern aboutpt'oyrlt' h.irrg trrrable to respond. Along with tlresent'w caprabilities, thereare also challt'ngt's tll,lt arre yet to be solved. The Web remains an unsafe environment antl orl'that many respondentsfear. The anonymity of the Internet allows pcop.'k' kr pretend to representother legitimate organizations,such as banking and credit card companiesor governmententities,to try to gain access to financial or other sensitiveinformation, to send bogus offers of rewards for going to certain web sites,and to encouragepeople to purchaseproducts that are never sent and sometimes do not evenexist.Thus,surveyorshave to be carefulto ensure ihat iheir survey requestsare not viewed suspiciously or thought to be an effort to trick the respondentinto taking a computer action that will deliver a computer virus or worm. Techniques for doing this are discussed in more detail in Chapter 7. To provide context for understanding the guidelines that follow, it is useful to start with a brief introduction to the Intemet and how web surveys are conducted.The Internet consistsof many independentnetworks around the world that use a standard protocol (TCP/IP) to exchangeinformation electronically.The World Wide Web (www) uses a standard hypertext transport protocol (http) to exchange information over the Internet,and web pages are formatted using hypertext markup language (HTML) and other programming technologiesand stored on web servers.Users,or potential respondents/ connecttheir computers to this network through a telephone cord, cablewire, or satelliteand through softwareon their computerscalled a brouser(e.9.,Internet Explorer,Firefox,or Safari).Once respondentsopen their browsers, they can connect to a specific server and web page by entering a web addressor uniform resourcelocator (URL). That server then sendsthe web page associated with that URL to the user's computer in the form of code,which the browser translatesinto a viewable web page. A web survey is made up of a web page or seriesof web pagescontaining survey questionsformatted most commonly in HTML that are stored on a scrverso that respondents with the proper URL canaccess them through their computersand the Internet. Suwey sponsorssend requests, often by e-mail but also by mail or telephone,to the person (or unit) from whom a response is desiretlrnd provide the URL for the web survey (and often an individual itk'ntificatiorr code).Aftcr respondents enter the URL into the browser's adtln'ss[T.rr, tlrt'y ,rrt'r,rsrrally first routcd to a welcomeor introductoryscreen lh,rlbrit'lly tlt'stlilrt'sllrt'sulvt.yirntl,rsks thcm t(, proceed with the question()rrrr,n,slrorrrllrrls tt,titr'. llliirr lll.srrlvt'y and ,rnswcr tht (lucstions, their rlnrw('ts ,r'r.slrrl l',rr k to tlrr,wr.lrrt.rvr.r ,rllt,rllrt,vrrn,ss lhr"'srrbmit"or "rx,\1"l'rllor llrr.rr rr.',1'ur'(", rrr,ry lrr.r'r.r,ir.wr.rl lrv tlrr,srrlt,r,y sollw,tn'arrtl

Flgure6.14 Examplo ol doubloquostion gridlormat

I t. Pl.e ldl(.t. rL{tEr t{ r.rd.rtt, (.-tm.[t, or xhi rto r.h IrF d ro.l th.d bator, f . .tp. or rorr rr not dom d ydr t nnlrrb, ptr brrl Dc Nor Apd, .ra .lodru ro rb nd

'fi
n@Dg, d!ri.S, p&d!s * tr4iE! !rii&6. !..tbtt r qbdti(k 4pltiq Di!b8 ,r.!. !!.ri Dotra tudrcit rt||t.trl rrtt!.rr Crir! fd hls Doir8 &m/tch rdt *i6 lag O.ii.! cdd. qlw. Ohir!4lrlli.a F.dbS.dlc V&idbldrL 8ddb8, d.6o ii& o.4rflila 6dG f,tejrs tuE Et--d. n.;6Tc4oi!saa FBIE 6 Mii'gnja.qlbht !d6 lr&te||i!!Fod&tt 9odLa-& drd{ h-.... <. trra t|9drli!! 6. f-dr.et r(t o( dt6 eile ft.ldo aeid fo. n0il}u Cery i( cni&a ( .Hdly lrtrity nslhqs i.rolira-o|hE i$/tlnleht.s i . : l:: a, f at :: I : ll i a

L I t.l T

:t
:: -C CN

r.
:l

ktr

t:

Ml.r

:.

t::i l :

r ,:
C l:

:,

uu
at

r.
f:

D nc ar,r
a. ': :acl :LL CEC L:f
F-

a T L

tr

tn a: nc
i: : fn :t

f.

,. n

I :l tl C

t: |] U t.: ! |: ]

nc

[rf t: l: {t r l ] tfctl {_ - L l :

;:-

{ : n |]
Lrt onc

:C

rr.Mi.!rouh.rrn\|h-drdon,.d.ar.a@to|t.!n t**r.rd,*n*,* De co||!rE rb.t&cr yq F.t rio.t *tJob tr l} re jqd,., DoF rlr,ou

.tr;b

dc

Source: Doing Gender lyhen Home and Work Are Bluted: ll/omen and Sex-Atvpical Tastrs in FamilyFarning,by J.D. Sm',th,2007,Pultman.WAr WashingtonSrare Universiiy.Unpublished doctoml dissertation.

quickly by largenumbersofpeople and at low cost.Thousands or eventensof thousandsof questionnaires can be completedin a single day and the results released. As a result, as discussedin Chapter 1, many surveyshave shifted from being an interviewer-respondent conversation over the telephone or in person to being a visually delivered requestover the tntemet. However, in practice,there are many barriers to web survey success. Although web surveys were conducted a decade ago when the second edition of this book was written, there were many significant technicaland social barriers to their widespread use, and responserates were often low. Only a third of householdsin the United States had Intemet connections, connectionswere very slow (it sometimestook many minutes for questionsand answersto be sentback and forth) or would break off entirely,and therewas huge variation in the technologyused by both respondents and surveyors. Much has changedsince 1998,and the maiodty of U.S. householdsnow have access to the Intemet many of them through high-speedconnections, as discussedin Chapter 3. In some populations, nearly all members havc high-speed conncctions. Although somc. problcmsrt,rn,rirr (r,.g., s()mesur_ vcysar('t(x) sophisticitt'rl for tlrc lrardwart,.urd solllv,rrr.,rv,rilal.rL.to Dr.rny

l 'l l i

lr r r 0r . ( r - ) t t , ||( ' N , t,, \ (_ )tt' ,

N N \ tl

,ttr'rl ot r r l t r r . r r l. r l,rl r,rs r' rrrr l l rr.s r.r' rr.rWl r,tl l l ri s t( !ttr\, l rr r.:,' ,r' rrtr., i s l l r,rl \l rt\r'\ ' ( ) r s lr , t \ ' r ' l( )lf ,l l \l .tl (. l l rIi I rl rrt' s l i o rrrra i rt' t1t' si gl rs o tol trl rLtl t' r i rrl cotl t' i o l rl s lonr i ( ) n, r s ( ' r\' (' r.l l rr.y .rl s o h a v t' to c l t' s i Irrr].rl .rb.rst.s tl r.rl w i l l stor.r' \l tt \'( \ ' r ' ( ' s P ( ) ns (ir ()rl l J rri z (' drrre , .r ri rrgfrrl ,.rntl .s r a n ,rtc rrr,rtr., ncccssi bl (' w .l v r)n l l ro s or v c r ' . Another flctor tlrat has contributeti to the growth in web survcying is llrt'witlc vnriety of ways that technical aspects of building and hosting a r\'('L) srrrvey can [rc'managed.Perhaps the simplest and least expensive is by rrsirrgone of scvcral free or inexpensive software applications provided on tlrt' Web. ln this case, the surveyor designs the survey and contacts using I)()intind click software available via the Web through a specific company. llrt' sun,r'yor makcs a series of very sirnple and limited design decisions, w,hit h thc softwarc then translates into the proper computer code that will be nt't'cler.l to form thc web pages and the database. The survey is then hosted on th('company's scrversand, when complete,the surveyor can access the data ol tlata re'ports, depcnding on the level of service initially selected. Although t'.rsy and inexpensive, many such services have a number of limitations that slrotrld be considcred very carefully, such as limited design options, implerrrentation procedures and control over the actual data. Another option is to use off-the-shelf software that allows people to desigrr, host, and implement web surveys without needing to have highJevel programming skills. In recent years there has been rapid improvement in this tvp(' of software, including more options of software to use and more design {)pti()ns within many of the programs themselves. In addition, numerous orli,rrriz.rtions have emerged that will serve as a host for one's survey, allowing orrr. to btrild the questionnaire following the rules of the host organization, rr'lrir lr will collect the data. Both of these options often limit the control that sur'\'('y()rs can have over the design and hosting of the survey as well as the ( r)r l,r(liDg of potential respondents. A lin.rl ()ption is to program and host the survey from scratch. Many l\\ |Pl('hive the programming skills needed for designing and implementing srrr\'('vs As well as an understanding of the methodological issues involved (()r'tlr('v work with people who have the methodological understanding), so rrany organizations are now able to program and host their own surveys. Sorrr.orgrnizations have cven developed their own software and systemsfor rr rllt'ctingarrd rnalyzing data. Some of thesesystemsare quite sophisticated, ,rllow ing onc to innoviltc ncw ways of collectingand processingdata, whereas olirt ls urt'nrorc rtrdimerrtary. Conducting surveys from the ground up in this u',rv plovirlt's tl'rt'rrost dt'sigr.r flc'xibility ancl ability to innovatr', but it is.rlso ()L )ti o n . o l l ctt l lt t ' t ) ) ( ) s( l ' \ t ) ( ' n s i v (' l l rr' lir r ir k ' lir r t ' s l tc rc a n ' .ri n tc d .rl l tr' 11' rn1' :,rl , vr.\,()r\ rvol k l l tr.i r. l) r(,s c n l (.(l rr',tv tl t t ot r l' , lrt lr t lc v c l o P rrc ttl .rrrrlrl r.s i 1 ,,rr o l .r rvr.l ' ,,rr\' .\' trt r\' .tvs l l t,tl rl tl l r.t t r ot lt , t 1i|r r ' s |( ) l l s r,\ l to ttr n r.rrry l r' | r.s rrl tr-,1 ,r' rr,l Irrl ,, l l r,.' ,r' l l rrr,l r.l i rr| s

t, tttt tu t, l ttt i r r ,.,r q r r r r r q Al tu l n u l l \l t

l l u t..,l tt, ttttt..

ttttl

( r ( t ( t _ r ( .ss ||t( tr )\ r r l l l r . l r .r l r r tl l l r l ,,r r.r l r sstl r .so l r r r r r slr .r r t l i n l i r !r .l r sr l |.r r ,\ r , ,r ,.r tr ,l l , t s r l t r isi o r r s,l l ) ( 'l tt I i )| | | r, | | | | | r .r r r r l r l i sp l .tyl l l g I t1 ,",1 ,,,:r 1 i ,,,,,,,,,.,.t1 | r I - ,1 ,.r r r l c | t s . Al tl tr r tr g l t l l r r .r l r ,sr l !r ,ttt( l i l tr p l cn l cn tn t i ,r ,, ,,1 * ,,.1 , sU t.\.( ,\ tl l r ( ,,,tr r ,r l l l n i r ( 's .l r t. r e l a te ti , w t, tl i st r r ss sPt,ci fi cs a b o u t i l r r Pl t,r r r t,r r l i n gu ,r ,l r :,r l r r .1,, r r r the next chapter.

CuiLlt:lint 6.19: Decide uhetlrr an Electronic Altrntltiu,

,lf,':t:,t:1,:

that they c.rn passor t..r r.r.\, .rr|tr rr,l l,, . rll of the appropriate people to answer the qu""ti.,.,, .r, \,v1.,.r.1 th..t, ,,,,,v ,,,,,,,1 r,, keep a paper copy for their records. Software is aiso rrow ur,,,ii,,,.,r,, ,,,.,tt,,,\' rcspondents to enter answers electronically into a pDIr filt, th.rt t,rrr tlr|rr lr| e mailed back to the suryeyor for data entry. Moreovcr, sorlr.soltr,r,,.r|r,tlr.rl is available for designing such fillable pDFs also has the cap.rbililr, to r.r,.r,l rcsponses into a data set, thus eliminating manual clati ( )rrr. t,ntr_v ,rrrr,r rr,rl,r, of fillable.forms is that respondentscan downroacr thc t,nri*,ti.r rrrr.r, ,rrrrl completeitontheir computer; however, they d. nced t. inslaI Acr.lrt,Ar .rrr.rr l{eader on their computer to be able to completc thc srrr!t,1,in tt,,, t,,,.,,,,,t

,r."d to survey establishmenrs whtre ,.rt,,,,,,i,,,,,* n,,r\t,r,,r "1:"" :.;;T:::: oll a paper copy of the questionnaire

Appropriate k.g., Fillable pDF or Enrbeddad E_Mail Strn,ty) Thc advancement of computer and web technologi(,s lras n,srrlltri rrr r.rr rr,r., clectronicalternativesto the traditional web survty. For er.rrnPlt,,lill,rl,L,t ,l rl

to n Wt.lt Sln,\.1/t.. Att,t,.

PUF lorms can be quite handy in sonc sittr,rtiorrs, orrc rir,r\\ lrrr L ^t:no.ugh rs rnat responses are submitted to the server all at ()ne tiltr(,,so r(,sp()n(r(.nts must complete the entire form and submit it for rcsponscs t() bc rcgist(,r.(\l Another alternative to the traditional web survey is to cmbeq qu(,str()ns rl irectly into an c-mail that is then sent to the survey population. This nretlrotl ology was discussed in the previous eaitlo., of tt.ri" t oot as a possibiiity, tru t it was arguc.d that e-mailed surveys are mostly impractical because as answcrs .rre entercd, text in the questions moves, making the final document difficult Io 1.rresg55. In addition, the visual appearance of the e_mail survey may differ lirr rcsponr'lents depending on how they read their e_mail_onl'ine through orrt'of nrany Wcb-based e_mail providers, or through a software program (nr llr(.ir ()wn c()mputer. On the flip side, it is now iossible to send HTML (r'.rllrt'rlhlrr [rl.rin-t(,xt)c-mails that include more adrra.,cedformattrng so llr.rI h(. (lu(.sli()n n,lirc .rpp(,lr 1!irn1y1.. like a traditional web format and images .t r',rrrI't, irrr.lurk'rlirr tlrt, t,-rrail. I krwcvt,r, some e_mail providers will accept orrl y prl ,r irlt r 'r l t ' r r r , r ils or r t t , r ycat r t ion r ( . spot ldent s bef or c im agcs or ot her ,rr,| r' ,rrr r. r I l. r r r , r i. li , r r r .r lisPI , r 't r l ( 1. r .1; 1111, , t his, sot ,<; Lr it llinc Z. ZZin r:tr ( l l .rl rl r' r7) ( ) r 'r , r , r ll, r t is ir r r I . , rl. r l i. ' r , 1 . , 1, , , , , , r, . vlt . , llr t r ,r. nr .ol lh( , sck) r nnt s ttt,ll ' Itr' tIt or r ', t 1'; 'r r) |||t , t lr .loI olr , '! . r . 1r \ . r , \ .rr rr . , ls,, r sllr r . y, r t r ,. ll1, r r1, sst . . st ly l l r.rrrl rrl l r lr l' r , 1, t t \ r . r r r , l, . . r r r . r . ,

tree. andmanvpeopte dtre.rdy tr,rvt.it orrrt,,.,r',,,,,,1,,,,,.,'t,

ll, u

t,l({)M L'l||'.

t',ta.

lil

A L I r tir tlt( r NN AtRt]

tltrirlrlin", hu I )r,rriqrrlrr,l Mtil ttnl Wrh lttrtliruIt tt.t,s.)il ol llrt'irlrrowscr.trr lr.rv nol lr,rvlirrsl,tlltrl llrt.st, ,rtIr.Iitiorr,r | [lug-ins.l.,tstly, lltt'n'is , rlso wirlt'r,,rr.i,ttiotr irrrrsr.r.sr,ltirrgs, arrrl nsp(\)pl(, l)(,(l)ll(, (rlll ltot.(! lht'y,rrt'rlrort, lilt,ly kr r.rrsttrnrizt, Pttt(,r'sdvvy, thcir.st.itings, wltit.lr r.rP,rrrtls lh(.\,,tt'i,lti()n. lhus, it is inlp()rtnrll l(r.lssess thc technologic.rI (,, c,lP,rl.riliti(,s lh('|opulntiollfor cachsurvt,yth.lt is conducted, rccognizingr(,slr\\l .rl which technological changcs arc beingadoptedacnrss many 1.,oprr l,rt i,,ns (irridtlina6.22:TlkcSteps to Ensure that elrcstions DisplnySirtiltrlr1 rtcrost llillrrcnt Platforns,Browsers, and LlserSettings lh,cluse of differences in hardware and softwareconfigurationsand in intli vid ual user settings,the way that web surveysare dispiayed can vary wiclcly ,rcft)ss respondents. As a result, it is necessary to take additional stepsk) try to control the effectsof configurationsand settingson the visual diiplay cif qtrcstions in an effort to try to ensurethat every respondentreceives the same stimulus. One step that can be taken in the web zurvey programmrng is to s('parate the contentof the survey from the stylistic elements (-e.g., fonts, font sii,.es, color, widths, alignment, and other aspectsof visual pres'entation) by tusing cascading style sheets (CSS). Cascading style sheets allow the program_ mer to.specify the order and precedence in which various styles should be ,rpplieddependingon user configurationsand settingssuch asscreenresolu_ tion. In other words, they allow the programmer to tiilor how inlormation is presented to different tlpes of browsers,handheld devices,and other equip_ mcnt used to display web surveysin an effort to minimize variations in how web pagesappear visually. As a simple example, if one respondent has a high screenresolution, which makes items on the."r""., upp"". small, the CSS programming will make the survey pages display witir a larger font (t .g.,20 point). If anotherrespondenthas a very low screen resolution,which t(,ndsto make items on the screenlook big, the CSSprogramrning will make tlr('survey pagesdisplay with a smaller font (e.g.,12 point). Despitethe re_ spondents'different screenresolutions,the size of the text will appear very similariy frx both oftheserespondents because the CSSprogramming adiusts lhc display to make up for differencesdue to resolution."Thus, it is impor_ tnnt f()r web surveyors to use CSSand other newer advancements in web pnrgramming to have greater control over how web questionnaires appear I(, 5111y.'t respondents. Arrothcr programming tool that can be used to help standardize the ap_ of wcb survcy screensacrossvarious user configurations is the lt:'::r.]"." I l l Ml. tatrlt'. lrrcsse nce,tht,survty conte,nt is programmed iito tuble"whe.e llrt, tolurrrrrwitllhs arc s(,1 ,ls pr()porti()Ds of thc screen so that they adiust ,rPPr()l)r'i,rl(,ly il tlrt,br.owscr w jrrriowsizt,is t.h.rngt,rl. Wh.rtthis doe; is help lttittitttizt'tlrt'r.llttts ol r.lr,urtt,s irr lrlowst,r wirrtLrw sizr,()ntl.t(,visual ao_
1 l ( . , l i l l ( ( '( ) l tl l r ' w ( ,l r l l ,tl ,,( .

( /rrrrrrr' l)trulrlin' t'..t1): I /rra,lltr Stn t't,U Will llt I'rovtntutrl tll I ltslnl ('tn tu t 't t r ut t t lru, illtllt ( N rrtl i ,5 Ai //srl , rrl Sl ro rl rrrl s l t rtl i p Lucl tS trru' y ()rg i rri z .r(i ()tr l l i s i n r p( ) r l, r lt t o as s t' s s w ,l rt' l l rt,r' n tr h as tht,caP abi l i ty trf eotttlttr'littg,tnrl ir'npl('nr(.ntinB,l wclr strrvcy tlrlt nlects thc dcsirt'tl golls. Sontt. the training tlrirt nllows tlrcm to design and program all aspects l)t'r)Pl('hJV(' ol lltt'ir own web survt'ys and are krcated in an organization that providcs llrt'rrt't'tlt'tlscrvcr capabilitiesand protection (including backups) to host th(' stttvt'y, rcccive tho data, and send e-mail contacts to potential respondents. lrr adrlitiorr, thcy or others in the organization can troubleshoot and managc strvt,y ,ttrd tcchnical issuesthat may arise in the process.If one's organiz.rliort rkts r1()lhave the capabilities to design, host, and implement a widc r',rrrgcof wcb survcys, there are several other alternatives, as we discussed t..rrlit'r, lor programming and hosting a web survey. Designing and hosting strt vcys lrirs also become an activity in which some people and organizations Itavt, [Tt'comc specialists. It may be necessary to contract out the programrning .rntl hosting of the survey to these people or organizations. Others may irrst(',rclclecide to purchase a software package that can help with designing ,rntl nranaging the implementation of the survey, particularly if they plan to eorrrluct a large number of web surveys. Thus, it is important for surveyors to tlrink about the expectations they have for collecting data and choose from these altcrnatives for designing and hosting web surveys commensurate with thos(. nocds, (-,ttidtlitn 6,2'l: Eualulte the Technological Capabilitiesof the Suraey Population It is irnportant to design the web survey wiih the survey population in mind. Srrrvr.y 1'ropulationsvary in their accessto the necessary technology to compk'tt,a web survey and their understanding of how the technology and the lrr'()e('ssworks (see elaboration in Chapter 3). Some people may not be farrili,rr with completing surveys on the Web and will need more instruction alrrrtrt lrow trr accessand complete the survey (see Guideline 7.24).In addili()n, .rs c()mputer technology expands, there is a wide variation in the types of clt'vict.s available to people for accessing the Intemet, including desktop (\)nrl.tuters, laptops, and handheld or other types of devices (all with varying s('ft'(,I'r siz(.s and with a range of processors, memory, and hard drive sizes). Altlrotrgh mon' people are getting high-speed lnternet connections (T1, DSL, ,tncl cablc) as discussed in Chapter 3, a large number of respondents are still (56k and even 14.4k modems), which are much rrsing rlial-up c()nnecti()ns slowt'r. Morcover, lrigh-speed connections often vary in how fast respondents tittr tkrwnloitcl antl upload information. In addition, tht rvailability of new blrrwst'rslt,rst.xpartdctlrs nrort' peoplt'are onlint', arrtl nrostbrowst'rsarc corrst,tn l l ypr ov ir ling u[ ) d i ]t(' s ,rrr(l n c w p l trg -i n sth ,rtl n ' r' rrl rri rrrl to vi (' w sp(' ci fi c wr'h l r,r B c s llowt ' v c r ' , 5 1 r1 1 1 q ' trrrl h ,tv t' l lr(' nro.{l uP (l ,l h\l v(' rsi ()rl . P1 1 1 rl q ' trta y

? l l 1 l rrr r nt( - ) rr l , , ll N' , tt t

(-)l l ' .tl r rN N ,\tl l l Wdt

t )trullrn\,ltt I )r\{(l|lf t,(Mtil ur,l Wth lltt,;ltrr,lrrri|r'r 2l)ll "gt,tyt,tl tlu' irr,r t1rrt'sliorrs wr,r'r' orrl,"irrt'1lt'r't tlt.t.rrr therl PPlit,tblt' Ph.rsizing attri sigttalingtlrt' rt.sporrt|,rrl lo skip tlrenr(l'otaka,200ft;St.ltistics New Zea lancl,2006). llut olrcc agirirr,suchintcractivc fea tu rcsdcpendon JavaScript or relatt'cl pn)granrmilrg thatcan bc disabled by respondents. In contrast to prcsenting all of the questionson one page, web surveys eln prcscnt eachquestion on its own page and require respondentsto click l button to move from page to page. In this format responses can be saved to the server database "submit" eachtime the button is pressed.One of the main strengthsof this constructionis that surveyorsreceiveresponses to each questionevenifrespondentschoosenot to completethe entiresurvey (which can be helpful in understandingwhere, and sometimeswhy, people are terminating the survey).Because of this continuousinteraction with the server, more interactive capabilitiescan be utilized, such as automatically branching respondentsto subsequentfollow-up questions,modifying questions basedon responses to previous questions,or collectingcertain paradata for cachquestion (seelater discussionunder Guideline 6.32).The page-by-page design format may be best for questionnaireswith complex skip pattems because it gives the surveyor control over which questionscome next rather than having to rely on the respondentto correctly interpret instructions.An additional benefitis that with page-by-page constructionit is often easierfor respondents to know where to focus their attention because every question can appear in the same location (horizontal and vertical alignment) on tht' pages,an elementof consistency that can reduceitem nonresponse. On the negativeside, questionnaires that are constructedwith eachque'stion on its own page often take longer to complete (Couper et aL,2001; Manfreda,Batagelj,& Vehovar,2002). In addition, becausethe questionsare viewed in isolation, respondentsmay have difficulty rememberingthe contcxt established by previous questionsand may needto review their previous responses to remind themselvesof that context.Likewise, respondentswho ttccd to remembertheir answersto a previous question in order to answer a srbsequentquestion or ensure they are not being redundant will face extra difficulty because reviewing previous questionsin the page-by-page format is much more difficult than in a scrolling design.Moreover,respondents who t('mporarily quit the survey and then come back later to complete it may have difficulty rememberingwhere they lefi off or the context of the questions.'l'l.tus,it is important for respondentsto be able to move both forward ,urrl l.rnckw.rrtlin thr: survey and to be able to see their previous answers wltt'tt <luestions itre prescntedon their own pages.However, even when rerl, otrtlt'ntsc.tn nrovt.bockwarclin thc survey,effort needsto be made to make ntrt,t',tcht;rrcslion canstirndolorrc(i.e., cln be understood without needing
lu( 'sli( nr s) . l (l t(,l r,rl o l ) r '( '\ , iolr(s A l i n.rl lonn, r l lor ( r r r slr u( linll r vt 'b r lr r cslionnair t 's, lnt l onc of t he'm ost rrrttttttottl yt t r r r l, ir llr , r l it t wlr it lr r r t r lt iPk' r lr r r , slior r,s u( , I r '( 'scnk\ l on ( ) n( '

l)tt iit I lrtu' Mtrttl gln'slnutt Will llr I'n'sttrl ttt l ul r)trnh'lin' tt,.'.1: Wii/ llt Arruttgttl I lr)Ii,Qt/r'sllr)rts lh(r' r?,lrl

i rl tl t' s i g tti ttgJ w (' l r (II I(':'I i t )|r | |,I i I ( l s ( )rrc o l llt r . nt r r : ilir r r P()rt,trrl < l t.r' i s i trrrs stlrvt'ys Irow ttt,ttryqttt'stionswill [rt'prcst'nted on each screcnor page Web all of tht' or pcr pagc' ,,rrr 1,t,'r.'ttt(nr('qtr('stiorlpcr pag(', rnultiple tlucstions (nl ollc pagc. H0wcvcr, scvcril important imPlicntionsnct'rl r.|lf\'(.y(lt|('sli(n]s on lr, l,.. .,,nsi.k'rr'.i wht'rr cleciding ht.lw many qucstiotls will be presentecl r',t. lt p,tgt'. Wlrr'rr all o[ tlrt' tlttcstions arc presentcd on one page, the web questiotr can rrron'clost'ly approximatcs a paper survcy becausethe respondent rr,ri|r. then scr()ll plr,vir,w thc t'ntire qut'stionnairebefore answering questions and lt has (Dillman,2000a) answering while ,rrr.1[.rockwardwithin it l,,rw,rr',1 about decision informed lrr.r,rrgttcrl th.lt rcspondents can make a more w |x.|| l (. r t ( ) ( \ ) m p|c t eth c s u rv e y w h e n th e y c a n s e c theenti requesti onna| r(, (( r',rwlortl,CouPr:t & Lamias, 2001).However, this format also has several l)epe'nding on the length of the survey, resPondentsmay havc lirrrit,rliotrs. the kr st roll through krts if questions in the questionnaire, thus increasing Branchquestions t lr,rrxcs tlut tht'y will miss questionsor even entire setsof to execute' ing lornrats or skip Patternsmay also be difficult for resPondents to not submitted usually are resPonses sirriilar to papcr surveys ln addition, all of completes resPondent tlrt'server ancl stored in the database until the capare no data llrc rrr.rcstionsand clicks a button to submit the resPonses; of the trrr.ri li,r respondents who break off after completing only a portion srrrv(.y.,,lvirscript or 0tht-r programming technologies can be used to caPturc on ,.r1r,,,,",.'" mort'oftcn in scroll designs, but some people disable scripting at submitting without tlrtlir e.nrr.,rrtt,rs.Their data will be lost if they break off of thc tht't.n.l oi tlrt'survcy Lastly, thi:t format limits the ability to use many based int('rnctiv(' fclturcs of web surveys (e.g, asking follow-up questions the samt' for etc calculations, ) ()n prr'vi()tls uswer!;,PL'rforming automatic prorelated or l,hat is, ntany interactivc features rely on JavaScript rt,,rsorr. options gr.rnrrrrirrg tlr.lt can bc disabled by respondents,so no-script backup nrusl h'providcd for thesc people. srtrveyorsare'usinga hybrid of this scrollableweb survey format Sonrt'wt'[.r and reirr which ntort' intt'rictivc fcaturl:scan be programmed into the survey is comPlctccl sn()ns('scillt s()t'tl('timt'sbt' slvt'cl bt'fi)rc the cntire questionnaire the page' In this lrybrid versittn, rcsPontlcntsstart with scveral questions on or $()metimcsprt'ss a "submit" button' subsehut ,rlt('r tlt('y ('llt('r il r('sPollse will appt'ar (()ll th('slmc plge) or qttestionsalready on tlrt' rqrtt'stiotts tgrrr,ttt [',rst'tlott thcir n'spotrst's Atr t'x'tttt1'lt'o['t {airly long 1ntg.'t,,,tu lrt' rrltxlilit'ti l or [rl rt, t ls ,1 rr,:rt i, , r r r r , r ir . ' llr l t' rll l ri s l ry trri tl s r' ro l l ' l b l t' l o rll l ' l l w ' l s ti ' r' r' l opt' tl | | ' l t(' sl )()l l ( | rl l ' l l l r" l l l i s(l l l | rl l l tr \' tts tts ttt| N Iw Z r' ,1 l ,t | | ls ( 'ill llr r '2ll0( r t(.| l l i ,r i tl ' l P l tl i ri tl rh" l ' tl| ' t tl l ' ttl r' rl |,rrl,tt t qw|. t t r l, l r llt Irl l l l l tl ,l w ,l y l l l ,l t ' l l l r" ' l l rrl l h

ll(l.l

lit( l]\t (-)t!t.,lt

N . , r r | \ ( - ) r r l , , ll( r NNAllt l

\ t r l t t r r l Wt b llt t t r l/ r , Lr lif r 's 2( 15 t , r t r r lr lt t n" / r , , I ) ''r , r , lt , t ,M i trttrtt' tl i ,tt t 'ly st t lr r t r r iit r , r lrt,lr s c r '( 'r r l) ( nt ( l( . t nl o lht 'inlt 'r 't 'st s ol t ht , st r lvt . yorin. r qui t(.tht1', r t ( , nir w. lll r y . lt r l ir r r plir , s llr , ll llt ( 'r espondent ir nt l t lr c sur veyor havc corrflictitrgint(.R,sts. ll tlris tyPq ol information must be included to protect proprictary survcy nrt'lsu rcs (irs was probably the case in this instance) or for any other reason, it sl.rould bc placed after a more welcoming message and pcrhaps even downplayed through the use of properties such as font size or color. Figure 6.15 shows an example of a web page that was designed in a much more welcoming manner. The text on this opening screen describes the survey in a nonthreatening way. As in this example, the opening screen should include the title of the survey, a brief description of its purpose, and instructions for how to proceed. Most opening screensalso include a space for respondents to enterindividual accesscodes, a topic covered in more depth in Chapter 7. This particular screen also includes additional information about participants' rights in the study and contact information that can be used if they have questions. The photo ihat is displayed was selected very carefully because of its appeal to those in the target population (more on this in Chapter 7). Such photos, however, should notbe used without careful considcration of how theymay affect both the decision to respond and the responses people give to individual questions. Surveyors also frequently place sponsorship and other information on the welcome screen to encourage respondents to begin the survey. Howevet information of this sort should be arranged so that it does not interfere with the mator function of the page (i.e., getting respondents into the survey), and it should be easily connected with other implementation features (see Chapter 7). The message in the closing screen should also be written in a friendly, professional manner and should both tell the respondents that they have completed the survey and portray gratitude. In the example shown in Figure 6.15, respondents are inforrned that their questionnaire has been received, a nrcssagethat tells them that they have entirely completed the survey task and can direct their browser to other web pages without losing information. In add ition, consistent with social exchange, the final message respondents receive fnrnr thc surveyor is a reward-gratitude for the effort they have put forth. ()ttiltlirc (t.25: I)r:ttL'lop a ScrtartForfilt that Emphasizes the Respondent Rather lltrl Iltt Spot,sor Wltlrt dtsigning wcb tl uestionnaircs, it is important to approach design from lhr, rr,sporrtlt.rrts' p('r'sp(,cti\,('rrrrltlesign pages that are appealing and in(sin)il.lr t1) lht' wt'lcomc screen discussed previl(,r'r'slin$l() f(.sl)()n(l(,nls , r rrrrri l V ). l ,ol cr , r r r r ph,r , 't r page and choosing ' ; ,rt i rI1 i t lr t . t it lt ' lr ont t lr t 'opt 'r r ir r g ri nrl tl (' l ' ,r , r l) lr i( llr , r l r '( , sl\ ) n( I , nls it |( , r r t ilywillt lo r '( 'l) ( \ r,l l( r oss t 'aclr page 't ( l rr' l I r' r t (|'ur , r lr , lt , ||lir il\ r t i( '|| lr y l r r r sir r li, r llr 'nlionor r n'sP( |r r r lt , nls ( scc ' rfl

il' , , t lt ' l) ( ' lw( ' ( ' lr t lr t r lliI lt ' w c l r 1 r '1 1 1 'sl h i s l o l t t l a t lr,rgc ar rrl n ,slTottrlt'ttls t t , t \ (t"g, st'vt'rtrl is rrlk'D tts(\i wll('tr stll\'('y()rs !v,ltrl t0 BrotlP rt'l.rtt'tl ';ttt'sliotts (lur' sli() rrsnb ()tlt,r ctlrr ( ' lll i( ) l) lr r s iliot l) t t r t lues t it t ns t h l l t t t s e a r ( ) l 'l 1 n l ( n ls c i l c queslor'nr,rl. (;r()uPitlS tlttt'stiotrs ean ht'lp rcspondents as thcy answcr the to navigate' need liorrs,rrrtl clrt rcduce the rltrnr[.rerof pages that respondents ( 1r.rrl ir'rrlarly with longcr questionnaires). Howevet similar to findings on pa(Schwarz, 1996), research on web surveys has shown that when l)(.r'slrv('ys {lu(.sti()ns.rppear on the same page, there is a higher correlation among an-

t*,,r:.; u.ru"..;the items (CouPeret a1.,2001;Tourangeauet al ,2004)' Thus' wlrr,rrclroosing to group multiple questionson the page, it is important to t ouestions that are related, otherwise respondentsmay try to make s(,1(,( doesnot intend' thdt the researcher questions acrrrss r orrntr'tirrns these formats is used deof lrr kday's world of web surveying, which pcnclson the particular needsof eachsurvey.Presentingall of the questions for which only a lirn,,', ,ur" png" ii usually used for shorter questionnaires ik'tl nrrmberof questionsneed to be answered.Presentingeachquestion on its own pageis alsotypically used for somewhatshortersurveys(wherenavor when complexskip doesnot get too cumbersome) igation betweenscreens or other interactivefeaturesneed to be included Constructingweb r].rtterns survcys with multiple questionsPer page is often used for longer surveys arrtl for surveysin which grouping questionson a pagecanhelp respondents pr()ccss them. Screens and Closing Welcome and Informatitte Interesting Cuidtlirc 6.24:Create ItmtlNill HaaeWideAppealto Respondents When web survey respondentsenter the survey URL into their Intemet browser or click on the link provided in the e-mail, they first seethe openby providing a descriPtion ing screen.This page helps orient resPondents lt also servesmuch the proceed to for how nf-ihe sr,..,ey and instructions in that it is the first questionnaire same function as the cover page of a mail and for many experiencethe respondentgets with the actual questionnaire, it is where they either make a commitment to start ambivalent respondents the survey or decide againstdoing so. As a result, the opening screenmust provide immediate and clear confirmation that the respondenthas accessed ihe correctpage and be welcorning and encouragingin ways that have wide surveyrespondents. appealb Potcntial Onc of us recentlyreceiveda survey in which the first iext on the opening the following:"No partsof this surveymay be copiedor used statccl scr('cn shouldbe mailedttr of lcompanynamellletlut'sts w ithoutw ritten pcrmission Icrrntl.r.tnyttantt,.lntl.rdclrt'ss|'Ust'rlft|rissttrvt.ywi||trrttIwrittt'trP('r|ni55i()|t ispttttisll.llrtr.lry|.rw',,St.rr.tirrgrrlfwithlIlisslrrrrr.u'Ir,tIrrrIrIl(||l(.ist]()lwhnt
0lt l, w,tttls lo rLr il lr , y ir r g lo t t r nv it lr t ' t t t t t |r ' , i, l,', 1 l P t l t o l l ( l i 'l l l s w l l o l r a v c lo lir r , r lr r , , r r l, r r r r ll, r r r t r ,l t t t t l r l l t r , r l t t | r l i o t l t l , l i t t " l l lrr,rl llrr,olrr,rri [ s r ' r ' ( . ( . D | ' r,,rt

(-)t t , i r .)ll{ ! li|l( ,N1 'N "

r ( - ) r r l . . r r r r NN' r t t t t Flguro6.16

l)rtirlrltttt:, /,rr Ilr"'{,1i,,{

Mnil ntrl lNt,b llu,tiltl

t t t,r

')07

tlnd Inbrnxllivo wolcorno and closlnq pagos Flguro 6.15 Lrilnlplos i)l llltl'i(trlllll(J l(n ri wob qttoslionnitll{)

Exornpk)ol l| quosttonrxtlro screon lhat omphasizos tho rospondonl

Qudlty ol Ut aury.V
b.lLtito 4,ffi.A r'. bE t',l.'.',tr...#tun ,,rd *&t trtt ,;,iAfr

ithla$ ona ol ih. tcllorlng b.d .La..lh[b i a u i !d fl o sF.v 4 dldd &d.$.' .&mt -Ld.d be B.ote ih.^a -dbrikr" *6 . 1 . | . ! {'f t - o a e 't o . o n p r

For .|nplcrtr|ant .ltr{on?

ri rrmDqrr, . tt-rilrtd,

opbrmr -!nc.-5nr Gr'|orrb* ..a

otn- (c-.cdt,,.

t.!. " l

tL.*,

enbr t@r Ad dE r.'i!e. * *n.

co& lr&d r. b Fr:

t.

,lbr//lr,r "Combining Mail and Internet Methods to Conduct Household Surveys of the (;cncral Public: A New Methodology?"by D. A. Dillman, J. D. Smyth, L. M. Christian,and A. O'Neill, 2008, May, New Orleans,LA. paper presented at the annual conferenceol.the A )crican Associatjonfor public Opinion Research.

prtsented

'lirurangeau, 2007). Similarly, in the previous example, people may rate their rrwn behavior differently depending on what delinquent behaviors are prc_ Iurcd. Overall, it is important to focus on the respondent when designing wc[] clut'sflon12i1g screens,both in the choice of title and any imagery as well ,rs in how information is presented on the page. 6.26: Usco Cotlsistentpage LaVout across Scrcens and Visually l ntylursizL,Qur:stiott Information that Respond.ents Will Needto Comdete the SurrL,yWhilc Dacnrytlnsizing Infon ation thot Is Not Essentiqlto the Tnsk A (l)trsistent page' layout across screens helps respondents easily process lho b,lsic organization of information on the screen and focus on the task ol ,rnswt'ring the questions. This can be especially important as the type of Irrkrlnratiorr retlut'stcd and the question format changesacrossscreens. ln the cx,rrnpk'sirr l;igLrrr'6.6 ilrtrorluced earlier in this chapter,background colors, lilrt,s, .rnti t orrtours lrt'll.r rcsponclents scgment the page into three regions:the lltttrlt't ot b,tttttr.tlt'giorr, lht'nrnirr (l'('sti()r.l .rre., and the footer. l lrr.ltr',rr h.r.'rti l{ \ rl('r f(.lli()'\ ,tt.(. ()ll(.. (1'lsist(,.t.lcr()ss .tll scrccns.For the nttrvt,yi rr l rigr r lr . ( r . t rllt , r .lt t ', r r lt 'r r . r , gion irr r lut [ , s llr t ,r r a r r r t of , lht , r rr . r ivt , r sitancl y . l, his, r r c, r l l tr.l i l l r.ol l lr c sr r llt , y ir t r vlt ilr ,lor r l , r li, r inst , r r l, r lL gr . , r y h, r r . lgr or r r r r l. ()uiltlitt

whcn presented with an image of a person who looks sick and worse when with an image of a person who looks healthy (Couper, Conrad, &

Surveysof the S,,rar'," ( onrbiningMail and lnternetMethodsto ConductHousehold and Christian' L M. D Smlah, J A Dillman. by D. ( icrrcnrl Methodology'l" l)ublic:A New ofthe presented at the annualconference LA Paper A. o'Ncill.2(X)8. May.Ncw Orlcans, lbr PublioOpinionResearch. n llrcricrnAssocialiolr

Fed lrigLrre 6.l6). In contrast, a survey that includes a title such as "Annual (,rnI n ss(,ssmcnt ()f Jtrvcnilt' Dclinqucncy" and imrgt's oI ntttltiplc' delincluent lrt , lr.rrrio rslirt ttst's ttr or t ' ot t t ht ' s 1- x lns or ' s nc et is , r l r t l u 'l r v l l r t 's t r r v t 'y i s b c i n g i t r r 'r g t 's r' onrlttctt'rl lh .lrr ()rr lhc f ( ' s P( ) t l( l( ' lr l lr l . r ddit ior l, l l r l t t r l o l s l l t 't t t 't l ( 'x 'l l l l P l ( " r l l l r " 'l l o l l s l l ) t t ] r, ly ('\'('l] lri,ts lto w t c s Pot t t lt ' t r ls , llls w( ' l ill( liv i( llr 'r l s t t r v t 'y s ' I il r r $r r lr , . . r l l l r l , r 'l l I t i t t w t 'l r il lt , r s lrr't| l()rrrrrl llt , r l 1r , oll11,1, 11rllr

l l {)fl lr t t t r t( - ) t t " t t

N .' rr.\ (-)l l ' ,l l (l N N ,\l t{ l

( ; uhh'l t t . ./,, r , I ) , , r , t , / / , t ! M t il t t t l l\ t lt l) nt : ilt t , ut t t , , , . t ll\ l Irrl r,r 'lr cl lr r . t r wsr . ris s r r sr r , t lll, ;,, Lr r , r llr l olr t lt ( , 1( . lt , t n( l llr r , , , 1o1. 1a, . 11, lr r r llr r r r n , l l rt' r' i glr t .lr r t lr t ' ligr r . r . ,t lr r ., , lr , r t . k, lr ullor r is kxir lt . t i ir r t h, , 1, , *, , r 'r . ililr lr r r rr r r . r ()l th(,( Ju( 'slion, t r ( , , t . t rllr rt1 t , , , r r t , rr illu( , slit ) n, , buuolr is lot . . r k, ril llt ( .lr $, r . r l, , ll corrtt'r. l-his swaprpingof sitit,s front wlt.rt is convetrti()nalin ,r wclr I,r.rrrr.,r.r is vt'ry rlelibt'ratc. ['hc ,,lrt,xt,,. button is placccl tlircctly trrr,Ir:r. I r](. (| rrr. ,| || || | tl rr thc lef t sidc of t he page bot h t o m ake it m or t , casily . r r r t i. lr li, Llv . r , , , - , , , , , , 1 hy rt'spondentsand to encourage thcm t() nrovc f,,r*,i,.f ,rrr,l,,j,r,llrr. srrr vr.y. ' l he " back" but t on is locat ed on t he sam c hor izont al linc lr r r t or r t lr , r . r y, lr l si<lc (i.e., outside of the foveal view) bccausc ti*", r",,.p,,,,,i,:,,rs n,rll rrr.r.rl to ntove backwar d in t he sur vey ( alt hough it gt nt , r ally shor r lt i lr r . l) r ( ) \r t lr \ l bt'causcsome may need it for reasonsde,scribed prt,viourly). Wr, lr,rvr.sr.r,rr no cvidence that swapping the location of the navigati,ut tr,itt,,,,, irr t lris n,,r1. confuscs respondents.Rather, they quickly l"orr, *1,..,,tht.bLrttorrs.rlr.,rrrtl trst'them appropriately.what is most important, howcve f, js th.lt rrrt,jot ,rtiorr of the navigation buttons be consistentacross screcltsils irr lhis t,t,rrrrpk,. Although screen layouts may differ somewhat fr.nr trrt,()r)(,(r('s(.rirt(,(l .

\' (' ry (,l r(' l l tl l y s t' l t' fl trl g r,tIl ri t l o l rr' l I r' r' sP ottrl t' ttl s n r,ry . r ls ( in( ) lu( l( ' , t s ttt,tl l , IrI' | | | (| | Il (rt i tl t'rrt ily wit h lht ' s ttrv t' y l l tt' l rx rl l t r(' ti o l l .()l rl . l i l )s( ()tl l ' l (l IIII()| r,t' trrrIl lilt or w ho l ttr tl l t' sl trvcyol w l to l t.tt' t' (l tt(' s ti ()l rs rrst,b y r es l- r ot r r ir ' t rts .lt thr' is loc,rtt'cl ,rrltlitiorralinfornrltiotr abottt tlrt'survey. l llili informatiot.t hott()nr()f the pnge,simil.lr k) contict or desiSncr infornration for many wtl' sik's. lt is sct on n white birckground, and thc text is in smlllcr black font to rrr.rkt'it lessvisiblc (becausenot all rcspondents will need this information), bLrtit crn still be easily located by those who are looking for it. 'Ihe main qucstion area contains the information respondents need to corrplctc the task of responding to the suryey question: the question stem, any .rtlclitional instructions, and the response options or answer spaces. Becaus(' ihis area is where respondents need to focus their attention, it takes up most ol tlrc page, it is located in the center of the page, and a green background color is used to help draw attention to it. In addition, locating the components of t'.rch individual question in the same region (defined by the contours and the grecn background color) encourages respondents to see the question stem, any additional instructions or definitions, and the answer categories or spaces is a group that belong together. Every question in the survey is designed following the guidelines for question design discussed in Chapter 4 (i.e., darker and larger print for the question stem, sPacing to create subgrouping, visually standardized answer spaces, etc.). Looking across the question screens in Figure 6.6 also reveals, however, that great care was taken to ensure consistency in the visual appearance of the questions as a whole. For example, the question number (i.e , Question 1 of 27) is located in the same location on every screen (both vertically and horizontally), although it is in a smaller font than the question stem so respondents are encouraged to focus more on the question itself. Likewise, the distance that the response oPtions and sPaces are indented compared to the question stem is similar across the Pages. Such consistency eases the response task as it allows the respondent to learn what to expect in the first screen or tlvo and then apply that knowledge to the rest of the survey rather than having to reorient at every screen. In many web surveys, desiSners provide custom navigation buttons for respondents to use when moving through the survey (i e., different buttons from the "back" and "forward" buttons in their Internet browser). The benefit of these navigation buttons is that they allow greater control over how answers are savcd as respondents proceed through the survey. As Figure 6.6 illustratt's, thc navigation buttons, if used, should be located in the central ls thcy m()ve (lu('stion irr(,il t() nrirk('tltcm mort'acccssiblt'to rcsP()ll(l('llts l h r() uUlrt h( ' s r r r v ( ' y . l :ttl l l ttl s l -ri t' l ix P t ' r it , r r c t ' rlt l t k ' rtrt' l tts c rs ttta y trrrl i tt' l l l n l II| (' | | ,| \' III,Iti otI l o l rrrrl ' tr,t' i 11tl i o!l[)tl l l ()l l s Irrtr r l it t lr ilit t t t ' ( ) .(),l r(" ' l \l ( l ..w ,l t.l ' 'trrtttl t,tl r' .1 , lis l' l ,ry rrl i tt l ttl c tttr' t l )f(tw r(' l s l l r.rl r" tl tr" ' l r,trk" l rttl l ott i tt ,tt'r' 11' t t r ' t , t lly

ll:l:_:-ln:O*'t":t In()rmatron on the

rhewebpages shoutd helpthercs;,on.i",,r r,,,,,g.,,,ir,. r1,,,

page. Thus, it is important that surv(ry()rs.s(. (.nsr\l(..1 .r move through intlivitlrral lraA,,.,,. r,.,.,rtt' ,rr

:]yill:I:ij t.lsKcndngesand respondents

the questionnaire.

*.:ss webpages to aid respondcnts asrh,1 (tu,,sti(,1 ry1.,,,,,r (,r.

(.).u iLlr:l ine 6,.27: Do Not Require Responses to erft,strrns Llttltss Alrttlrtlrly Ntctssaryfor the Surztey 'I he design and programming of web survcys ;rllow sllr\,(,yorq to r(\lur.(, r('sPonses to every question. Sometimes requiring r(,spor)\(,\ lo r)[(.()r,t t{.w kt'y tluestions can be essential to the goal ,,f th.; ..urr,"y antl ..,rrrs,rr,,,lrrrr, .tttd cxpense, such as when respondents would have to trr, c.tl1111 111, 1 11r, It'lcphone to collect missing responsesto key questions. I krw(,\,(.r.tor.tr()sl sttrveys,requiring responses before respondentscan move t() th(, tr(,\t s(.t.(,(,1t, P,trtictrlarly f.r evcry question, can have detrimental efftcts .rr r.sP(,.(r(.rl nl()liv,lti()n,on mL-asurement, and on the likelihood that responrlt,nls will r''nrPlck'. thc t'ntirc survey. when respondents do not have an answ(lr r() ,l

survey, or lie and provide an answer that is ntrl Itrrc lor thcrn. lhc first option will increase the likelihood of nu,rr""1,.rt"" lrnrr irr llrt,rlata,.rrrtlthe sccond will introducc, me.asurement error. I'rrllirrg n,sl.rorrrlt,n ls iDt() il situ.lti()n whorc thcy fe,elthey have b lic, kr H(,t p,lsl ,r (lttosti(). nrny als. h,rvt, lht, unr.lcsiral.rlt, si..leeffect ot making l l t(' nr nl o t 1,likr 'ly lo ir r , r t r r r r ar t . , t nsw( , rl( s ) r t lr r cst ior r s Pr . ovit it , lat cr ir r t he r' rrIvr,y. Ar r r . r , r r r r PL. r rtllr is w, r s wilr r t . ssr . t rllr r r it r li, r( r r g|liliv( , inl( , r vicwon n rl tl t' sl t(,n llr . r l r r r lr r ir . r 'r[lr 'oPh. lo i1111i, . ,llr , 1, , r f t 1in, t , r. lll( , y t . ( , ( r , iv( \ lt l . ( , lltll) ( , (rrt| | rl l | l t ilV r olh13. llr r . y , t llr . |( |. \ 1.( ) r r . ' n, s|or r r l|r r l r , \ , , l, t i r \ l llr , t l slr I r , , r lv

req.uircd h' provide oneanyway, theyhave two ()prr(,,rs: :l:::1]:l i::.].: s(,r ||u\tr,||(.(l .tnrl ternrin.tt(,thc

'l l )

l ti( ' f . l ( _) tt . t t , , r ..t' ,

\ (_ )rt .tt,rr\' ..t \tti l

|,ttttl t l t|t

l ttt l tt

t,,ttut,.,A l ,rtl ,ttr,l l \,,1,\t

t.l ttt

uttj tt...

,l l

l rrr'Lrrr r r , t r r r r r ' ss r 'or , o rrl rl rro t .rn .,r\'f( l l rL ' (l rr(' s l ron \V l ri n l l rr' ( r rrrr ttt(ss.tl i (' ,r1 '1 ',' , r |r t lt l ' llir r li lr c r l l r,rl rl rr' l r,rrl l o l )r(' \' rrl ('(r r{' sl ' ,' n:' (,sl rr' l .rrrl l l rttl.trtl I a l rv ,tv sr\' .!nl (\l ,r ,l ' 1i h' t' l tottt tl tt' tt' , '"rrrl ,"W r ' ll it ' s . r r r ir r ' l i l tI rr,l h 1 1 ' ,.rrrtl ()rrc . n( ) \ ! I lr ( r \ ' ( ' Wl ra l s h o u l c lI g c t i t?" (A l th(' i nl (' r& l )i l l rl nn, ',o | 1 ',r r c s y r,.rr s ).1 ). .)()( ln ,rLltliliorrto incrcising nonrclip()nse and measuremcnt crrot requiring ,r,rs!\'r'! s nl(rv be protrlematic for other reasons.For one, many institution.ll rlr icw bo,rrtls requirc surveys kr be designed so that respondents can skip t 1r rlsliors tlrt'y pre fcr not to answer (i.e.,r,oluntary).Also, when web sr-rrveys ,rll rrscrlJs pnrt of a mixed-mode sun,ey with a mail suruey, requiring an\w(,fs nr,r! lend to different stimuli, and ultimately responses, acrossthe two rrro,lt'slrceause .rniiwerscannot be required in mail surveys. Wlrr.rrwt'rclvise against requirin[i responses,we are often asked about .it rr,r lirrrrsin w,hich respondentsneed to provide a responseto an initial quesliorr so thcv cln be routed t() the appropriate follow-up question. Although rr,(loirirr,l.]nswers may be the only solution to this problem for some sur\ {,\'s,ltl(lny surveys can be programmed so the respondent can be branched ,rplrhrpriatelyeven when a responseto the initial question is not provided. It nr,ry f('(luir(' more thouliht on the part of the survcyor to think aboui which ilo(.sli(nrsthe respondent should be asked next, but it will often improve the (lLr,r lity of r('sponses received. some have l{cg,rrdless of the topic, most survcys have a few-and Ilr,r ! (luestions that cannot be honestly answered by respondents.As we Ir.rrc tlisr'rrsscd,requiring responsesoften leads to nonresponse and mea\rrr'rrr'rtl lriasesthat can far outweigh thc benefits of collecting responses l{! r'\ r.fv (luestion. If a responseis going to be required to a question, it is rnl,orl.rrl lo ascertain whether this is necessaryfor the survey and what r'llr.rls it u,ill have on respondent bchavior. ln addition, it may be useful to rrr,lrr,|l.optiorrssuch as "This question does not apply to me," "Prefer not to ,r r:,r{r,r-" or' " l)()n't know" so that rathcr than choosea responsethat does not lrl lor llrt'rrr,respondentswill be able to movc on without having to provide ,!n ir),1(r'rrnl('.rlljweror quit the sun,cy out of frustration.

llr t , issr r t , I . ir r . r . r , r r r r l, lr . 1u, , , , 1, , , , , , ,, r ) i, , r r , . r r , tl l ( sri i rl l (Dr ' JysJ\ ' " I lt is r . r r r r r ot lr t , lot |ncl, , a nt l P, r Ht , git , c sr r I ] t , , 1. , , , , , , ,r, , , . , 1, l (l trv l () l j\ t lt c t 't . r . orllr . t , scnt ( , ss. t A( , sr lr ( , r r sr r allyinar vhilclr . r r lr lr . or r r r , lr r r t lr l ' l .rrk.ol t. t l t t 'xt anr i kr ok I r r y t lif f er entf r onr t hc sur vt , r ,p. , , f , , , .; ; , t lr l lr t r r l l l r.rl l hcse gt 'ncr ic m cssiges ar c r . r nlr elpf r . l t ht , st r e. iit le fr or sr r r r . r , r t r r , r lr , , r . tltt't, rnaV frustrate respondcnts and cause some to ;top trying t() ,r(( r,.,\ tlrl srrr\ L'y_ Iir hclp 1c1;pllnclcnts trcubleshoot thesc types of issues,$,(, r(,(.()nrlr{,n(l tlt'sig.ing and programming survey-specific messagesthat r.acr a rr.* \^,indow and inform respondentsof exactly ' what happene<1 and ho\,1, k) g(,t l).r(k (D track. In our own web srrrveyswe design survey-relatederror 1,agt,s wilh the same look and feel of the regular pugur. We c1othis so that "r.i,"y irrsteadof feeling as if they have left th" ."rpo.ri"nts mentally corlnect ".,r.."y, ttr('error messageto the rest of the survey. The first example in Ftgure 6.12 Figure6.17 Examples of customized errormessaoes.

r,,r,rrs.,',sr,rr.r rr.rrr.,,,r,,,,,;;.';;:,:.,';".,1::ll,',,1 il li:: l rrotrgl r l o lr r '11' r . 1. lr or r lr lt , slr ool

l r r t l r c r r ' r ,tsr ',,, :,l ,r r r r l ,r r ,l r r r r ,,,

t )trirltlirt 6.211: I )rsigrrSrrrlt'r7-Spccific ntrd ltt'nrSyrccifir Erlo/ M.'ss/rqfs i () llc4) lit::ltrutLlotls lioultlrslnolAttrl lssucs TItL'r1 Mtrl EttcttutttL'r s()l l r('l ir ) r ( 'ts ( ' ( lr lr iell i s s rre s m i y ()c c u rth a t re q u i re communi cati ng w i th t' (' sl'orrr['rrls w,lt.tl h,ts h,rPPenecl antl whitt thcv shotrlclrio nt'xt. I:or t'rattrPlt', n 'sPo r r lt ' nlsr r av r x r t h i tv t' t' n tt' rt' r1 th c i r i rt' trti fi c a l i onr' rxk' trrrrt' tl l y ol tl rcv rl rl rrcsrrol l o,rrl ,p.rl rrr,rf rrol I ' r ' , r l' k ' lr r r i t' n ' .t p .rg t'i rr l l rc s u tv r' \' l rr' r, tttsr' l l rl rr" ' 1' r' trrL nl s.rl c orr ,r Irt o l ,rrrrrl ' l tr l r' .tl rttc s .rrr,l trrrrl ,rr lt il I I r c r r . . r r ' , ..r | lo rrr,rt I' r' l rl rrry' l,I rr' l rrrrr r), | 1.,Ir(l | r(| r' || ',1 ,'rr',r ' r r ! . ,lior r I r r . r rro | | rr' r.n r' Ir,| | |!l l l r"'rtt\ r ' \ , r llr ' r . r r l' \ \ ' r ' rrtrl i .rl ' l rl trr' r,rl l o tr' \l ,r' tri rrr rr)i rrri ' l l rl rrrrl trrrrl !l rr,rr, | ' | | I| | | I| | I'rrl l l rr ' ' , r r r r r ' \ r r r , r 1 | r,rI r, | | | | |r,, I , ' I | | rrI,rI rIr1 | .. | r, 1 ,,I .,' rrrrrr,r| ,Ir.| | ,,

:l l :_

\r. i ,. r,.t,n,rr,ort,t. nnd,tr(, rh. rurr.! o. * .b.i t.,oo i r. rry j n, to e((.rr.

r' r' . s i t h . rt rh . rt D , . l , r, rh . s . b . , t { t r. s t ro n r t h . t . rt . r o r t _ n r. , t y . q

,,'r,i.,dtrn,\rsArr, ! 8oo8rr 0867 o, il,l;:iil."i::,i;::,Jllill,rn.

t,ttttl t l utt

l ttt / ),..r.r,rrrrr.\ l tLttl ttrtl l |tl ,

\.tIt,..l j ttj

l t tt,,,

.) l \

r:r".:* a the correct page. If you have Jni qrestio.,", oi**"" .o.,,u", u" at _." Regardless of the cause of the it is important to atways "rror__"irui", provide a link on the error page that will take iesponaents to where they need to go next. In social exchange terms, doing so will reduce the costs to rcspondents by making the process of getting ba"ckinto the survey as easy as P()ssible. fhe second example in Figure 6.17 shows a screenshot of a custom. (r('\rlined error page thdt respondents may encounter as they try to tind the (()rrcct survey in the first place. Rather than a gene.ic error message appear_ this page.appears if respondents muk" u i-ristuk" i., entering a survey,s :rll: ulll. but get the domain correct. It provides immediate notifJaiion to the re_ sporrtlent that he or she is on the right track but may have made a slight error It ,rls. 1)r1rvi6le5 qgntact information in case the respondent cannot resolve the l ,trtl tlt ' r r . In.rtlrlition to these general survey_specific error messages,individual 'lrlr,r()ns nray require specific messages if the answer the respondent enters , h rls rroI rra tch the criteria programmed in a validation step.fi. erumpt", lf u | | *,1,( rs('i:, nt.t'tlecl for a particular question but a respondent leaves 'r it blank, r,rllrcl llr.rr rrsirrga ge,ncricmessagethat says ,,R"sporl"" ,"qrir"a- or -Vn, rrr\t l,r()\'i(lc,t rcsp()nscto this question,,,a survey-specificmessageusing llri, s.rr(' l).tg('l.ryout as the survey pages might explain why the responsers ri.rlrrrrtrl.lirr.t.ranrPle, the surveyor may explain that a.esponse rs required I'r.,,rrrst llrt' rrraingoal is for the federal government to d"t"r'-i* th" rl,.r-b". ('l l\\)l)l(' in tlrc household or that a responseis required so that appropriatf i lrr('sli( )rri.c.ll bc ilrjkedthroughout the rest of the survey.If additional options j1l1'ql, .rh. pr'1r1, ss11.lr .ls ,,prefernot toanswer,, or ,,Thisquestion does not appl1, Ir) r)r(.,"il rna_v be hclpful to remind respondentsthat thcy can provirlc tlrt,st, lvltcs ol rcsponscsso they are able to move on k) th" n"^i 1rng,. ()r'r.rtrll, th('s('survey-specific and itcm_s1.rccifia,...,,, ,r,.,.rug,ts (.(tnl(,tl) ttttIro \ ' ( . r ' ( ' r lr ( ) n( l( ' n n ()ti t \,a ti ()l t w n (,n ts s U (,s ()rrrrl Irr.rl n t.(,(, n( l l r,rl\\,()U l (l ()l l ) l trvi st' t t t , g, t liv r . lyir r rP.l 1 1 t 1 ,s | 1 ;1 1 1 1 q ,1r,rt 11 l rt,l i o r. l l l ., r,.,,,1,r,, i ,rl l t rrrrP ort,rnl l r) l )fr'\' ( ' r r t ,,l l 1 1,rrr l ( ' s l) ( ) n( lr ' l tll\rrrrrrIr,r rrri rr.rI i I r1 , r,.\. 1,,rr l r, rrl .rr lt t(.\..tf|

a,programming errorhasoccurred white answering ifl ,but here and rne survey. ",Y::";-:.:f by clicking reentering your ID number you will be

the message th.rt the survey has timed out in ordcr to conservc server resources but thai the respondent can resume where he left off by reentering the accesscode on the login page. Likewise, if a page cannot load or there is another type of programming issue, a .rstomirei eir.r. page _ay

orr.eturn t<, the rt, ginr"r.,,,";;':;:ll lil'] J,)";l;:l..:iJl,[ :l.:.il;'; explain in

\ rlt,l\\.1 \lrt\r,\ 5 l\ \ r lt r r , r t ( lt t |lr . s s , t ll( , llt , t l . I rr _.| ,( I I I ( |t .| | I | | l l l l l l l t r \ ( . i \ , ( , i l \ l t ( , (' nk.f(\l .lt irtrrrrh ,r l, r r r ls s r . r r r lr , lr r llr is r . r s r , , t lr , . r , . r ; , , , n , l , , r r t i s l o l t l r r l t , r l lras lj()n('wr()rrg (i.t'., l lr t ' t r s t . r ' ll) is inv alic l) . r l] c l is g i r r . r r l r \ , ( ) ( ) l ) l i o t r s tlr,tl itr(, vcrv sp ocific,lrld L r ilor c t l t o t l.

l i kt' l v to l rt . r r r r l. r lr r lr . rrrlr t lr t r . \'('vs (rr.r\'lr(rrr t-('sl)()rrs('( 'lrr('r('qtriredfor s()rr(.{}l rlr(.(1.(,str,,,,.' ,,, ,,,,, .,;;., :,'il ()uirldin o.)!): |.i'tlrutt Ctnfirlly tln,Llst,of Intuadiue L'eatttrcs, Balortcittg Intltoiv ,|utr ttt Mtrtnrrutnt uith thc Impqcton Resptndent Burrlcnanrl tlrc Itttplintiotts for Miyd Mode StffL)eus (-ommon interactive features on the Web include saving responses,branching respondents to appropriate questions based on their iesponses to previous questrons (or previously collected data), and providing error message pages that communicate information about how to troubles-hoot issues that may arise. Most of these types of features are considered essential for web surveys and should be included in all survevs. However, additional interactive features are available as well. C)ne exam_ ple is thai early responses or information already available from previous surveys or from sample records can be used to tailor later questions in the Thi: practice is especially common in establishment surveys and :urv:y. longitudinal surveys for which extensive information is avairable about the respondents ahead of time. Another example is that question wording can be customized, similar to computer_assisted telephone intcrvrcwing an<1 computer-assisted personal interviewing (e.g., When you began vorrr vaca_ tion in "July 2007" to "New Mexico,,,did you drive yo,.,r,,.iirv,,i.r n",rl,ur,, or your "Toyota Rav4" or travel there in some rrther w.ty?). In ,,,,,,..,,rr,,.ur, an enormous amount of information can be brought forwartl to irame clrrt,stions in a more direct manner for respondents.Altlrough tlris abilitv can [rt, incredibly useful, each of these inserts requiresprogramnrirrg that slroultl bt. cxtensively checked to make sure it is working properlv. In addition, web surveys can be designed so that if answers are not within a specified range, error messagesare sent to respondents asking for corrections (i.c.,.validation steps). Although validation steps can improie the quality of thc final data set for the surveyor and save data cleaning and editing time, they are usually designed from the viewpoint of the su"rveyor rather than thc respondent. As such, they can be quite problematic for respondents. First, frcrluent error messagescan be frustrating and burdensome, possibly leading t() p()()r response behavior or even break_offs. Second, it often happens that s.(.nlrios that the surveyor did not think to include as possibilities apply kr sorrre rcsl.rondents. In these cases,the data validation step often creates a silunti()n rrLrclt likc that created by requiring respondents to answer before llrt'y c.rn rrrovr'()n: lt createsa roadblock that requires certain respondents l() (.illl('f Hi\'(.in.recrrrntc clrta or tluit the survey. Including data validation slt'lrs( ,l ) ,tlso lt.l\'(.n(,gJ l i\.(,(,ff(,(.t\ for thc .itrrveytrr First and foremost, these sl r' | s.r| r' rl i llir r r ll t o .tf(.\,(.ry l( P|. 1'1. . 1111, Pt . ( ) lt ( , ) m ist akcst hat can have lar ger l t(Ji .rl r\( rnt l, . t t ls r r r t r l, r l, tr I r r , r I r I t , I . I I ( | , t f {, \ . ( , t - \(,,.\ l) ( , lt \ i\ , ( ,ln . . r t lt ljt i, , n,ihe prr\ r-,.,r' l r h . . r nr n11 ,rrrrI r , , I r I r r r 1i , |. r |. r r , lt lr , \ r , f y ir r I or . r r Lr I irr clillt r csp( \ . tl( )

2l, l

Ijt( rv q )lrl 1'lll'N' , l

A ( J l llillll( lNNAllill

(,

tt'tnh,r lttrI \

gttittg Mtil nul Wr,l()rrr,rlir,llrr',r,s ? | 'i

llow l('r'lx)ll(l('llrs itll( ' r l' lt ' lt \ l lll|. t 1t t t ' s li t t s lr t ' ir r g 't s L t 'r l o l l l l t 't t t l l t l t 'r 't t l i v r ' r, , rlirl,rtio rrst('l)s, itl t' s s t ' t lrt ' , "r ' t , ls r "' llt is I olt ' t r li. t ll y t t s l 'l t t l i t t l r r r t l t a t i o t r ' ( )t lrt,r tyPt,s o l ilrlr . r , r rliv r . l( ! t t i. s lr nv f lt t , r ' n ( a r r d ( ( ) l l l i l l U ( ' t o l r t ') t l t 'r , t 'l orrt'tl lot tht' Web, ltttl lhr'.ttttottttl ol r('s('(lrcllon tlrcse ft'lrtttres vlrics gr("ltly' |t,,rltttt's sttch as drttp-tlowtt trrt'trtts, litlks to clefinitions or t'xitmples, 'tttlrrnrnli( c,rlculirti()n trxrls, progrcss itrdicakrrs, and visual analog scale'shave 'tll lrt,t,rrtt'stcd in web surveys and wcrc discussed in Chapter 4. When deciding wlli( h interactive features to use, it is important to consider which are ncc-

('ssnryand which are more of a novelty or do not directly help respondcnts ,rs they complete the survey This issue was summed uP by a designer ol wll() rrrarrycstablishmentsurveys who commented that some resPondents r'onrplctedthe samesurvey eachyear expectedsuch featuresto be includer-], r.speciallyfor the purpose of avoiding callbacks,but that the same resporrrlt'nts often becamefrustrated when too many such featuresand feedback appearedin the survey.Oftentimes theseinteractivefeaturescost nr(,ssilges they require more advancedprogramminA Inorc and take more time because errors increases'In addition, programming of likelihood and becausethe have to JavaScriptor other client-side nlrny of them require respondents so no-scriptoptions enabledin order to work correctly, scripting technologies (i.e..a different measurement stimulus) must also be Programmedfor thost' important, thesenew featuresshould be Most disabled. who have scripting to seehow they interact with them and whether thc testedwith respondents featureshelp with or hinder survey completion. and Eoaluate of the WebSparingly, Capabilities 6.30:IlseAudtottisual Guideline TheyMay Haaeon Respondents theDiferentialEffect The Web allows for enhancedaudiovisual capabilitiesthat are not Possible in combination with any other survey mode, other than perhaps face-tofaceinterviews.Many web surveyorsare interestedin using audio and video suchasvoiceand visual imagc's it allows them to incorPoratefeatures because modes' For that can make the Web more similar to interviewer-administered example,a survey may have a video of an interviewer who readsthe survt'y questionsor may include an audio recording of the question alongside tht' text version on the screen. Although these featuresare generally included in an effort to assist re spondents,they can be quite problematic for several reasons First, manv c()mFuterhardware capil may not have installed the necessary respondents vidett or audio compont'nts run thc to plug-ins or additional bilities, software, bccatrsc collnccti()ns lllt('rlr('t high-spt't'tl filesoftenrequirc In addition,these Evcn b('tl('rslr(\lrr)ilr8clPnbiliti('s il il downloadtimt's.ttt(l theyhaveshorter
can be guarantecd that resPotlrk'ttts ll,lvt' lltc ttt't'tlt'tl lllt('rll('t illrd e()nlPtl[('l ttttl t'tr ls lr'lvltll', capabilities, mttre sitlrPk' llritllis t ,ttt f!r wlrtllll, stll lt ,ts tt'sl-rt s o l l l , t t l l r t 'y r 'i t t t t r o l l t t " t t ' r lowt t t r t tr l l lt t t t r c r l tht ' vttlttnrt'o tr th t'it t r r lt t I t t llt lhr' ,tttrlirr rr rtttPrrttct t ls .

Molrrvlr', t(,so,tr(,ll ()n llrt,st.rrr,w ()trtlrt,Wt,bis r,rllrlr linrited. l(!tluft.s 'l lrLrs, it is i,'p()rl.lrrt r. fvnlrnt(,rlrt.r'rf.t ts af inclrrtli^q ,rtrtli..r vruc' c.m(nr resP(nr(l(.nl trr()tiv,ltion ilnd ('n tlte concepts l)r,tr('lrts beingmeasured. As tlisr'rrsst'rl inclurlirrg picturcs in surveyscan have detrimental l'n'vioLrsly, ('ll(.('ts (nr mensrrremcnt for particularquestions, so the inclusionof audio 0r vidr,o may also have unknown effectson measurementor nonresDonse. ln .rdditiorr, prt'vious research()n interview featuresmay be imporLnt in r.v,rlLrating irudio,vidco componentsthat include an interviewer readinAthe .lu('stionsto respondents. (ittilrlilc 6.3'l:Allozu Respondents to StoptheSuruey andFinishCompleting It at Att,ll 'r Tine Most wcb survey respondentswill complete the survey at one time rather llr,rncoming back and finishing it later,particularly if the survey is short and only onc'personis neededto answer the questions.However, sometimesre_ rp(nlclents are interrupted (e.g.,because of power outagesor loss of Internet c(nrncctivit, to answer the phone, to respond to an urgent e_mail,to have a Conv(.rsation with anotherperson in the household,etc.) and need to finish lh(.surve'yat anothertime. Irr addition, for more complexand longer surveys ot f()r (.stablishment surveysfor which multiple people often need to be con_ lultcd to answerall of the questions, it is absolutelyessential for respondents hl bc able to easily discontinuethe survey and resumeit later. 'l'he ability to allow respondentsto quit and return to the survey later tttnlr,sstandard with many software packages. When respondentsreturn to lhn nurvcy welcome page and enter their identification code, it is usually hclpftrl kr then route them to the pagewhere they left off so they do not have llr ellck through multiple screensto find where they stopped answenng. r, it is alsoa good idea to allow respondents Howr,ve togo ba&ward from the th('y left off so they can remind themselves of the topic of the prevrous Frlnt hw r;rrr,stions if neededbeforeproceedhg with thosethai remain. Itrr t.sl,rt.rlishment surveysand other complexsurveysthat are likelv to be in multiple sessions, C{rmplctr.d sectionheadingsor oiher types of tjs that of information requestedmay alsohelp orimt respondents l$e rlbt. thc typ.le lvhsn lh(,y resumc the survey or when they are asked to completea section Strlynn. palticularly qualified for by someone elsewho hasalreadycompleted l|tttlltt,ru'ction.In suth c.rses wheredifferentrespondents needto comDlete ()r:i(,(.ti()nsi lllhtr,nt rlrrt'sti0rrs oI the survey, it is very helpful if responjents 3lI r'llr'l on ,r s(\.tion lr('.rdingto move directly to the relevantquestions. For survr.yhe.adings llttttph,, irr,rnt,stahlislrnrcnt may be organized by position r(,sour((,s, r.ontmunicltions, rr.se'arch, etc.),or in an individual [,1,, ltrrrrrarr lllVoy llrr'y rrl,rybt'org,rrrizt,ri bast'tlon thc typ(,of informationrequested lrkrg,r',rphit irI irrkrrnrirliorr, r'nrployrrrr,rrl ltislory,p()litical ll,;1,, attitud(1s, ctc.). ( lvr,rrtll, provitlirrg llx. ,thililylo ft,sllll(, llrl srrvr,y,tt ,t l,lt(,r limc shows
$l l l l t,. ir t r 'r |or r . ivr . l0 t . r 'n|0n( lr . t t lq' lr r . r . r ls,r r l is llr inl, r r r li, r r r or r t " rrIv, , y, r r

.tl t,

lit r \ l( - ) r l, , lt |r.r,. r(, \(_ )r t ,rrtN N \ttrl

|,tt1tIIItt||.

/,,

i 11,,11' 711yl 11 \l ttl

ttnl

ll

,l trn.,,l trtrtttttrrt.,

.tI.i

Irrrrtl lr lr v ill, r r r r r l) ll l r' l l r' :,u r\(' t.n s rv c tttt' ttl i o tttrl .l l ti sl r' .rl tttci s,tl ' sol ttl t' l v ,tttrll ot l r,trl i ttl l ,tt r l i ( u l ,tr l o l ' i t s , o l l o rrg t ' rk rtgl l rs, r-,sl rr li, r lol l s r r llc v s ()r l ,,r (' \' (' n i l strr\' (' vs, l l.o u rr' rt' t, t.t' tt' r' o tttn tt' trtlil l ()r rrl ()st l \'l \.r, ( ) l l) ( ) l) t t l. t lio tts orrlv,r lt'rl, r'r'sPorttlt'ttls rtray nt'ctl t() t.lke .rdv.tntagcof this fcnttrre. FL't'dltttL'k ott 1 I)ortLlttt tlutf PrLntidL' t ,uiltlittr 6. i): Wlttttti,cr /'rrssi{r/r', Co//r'i llIir !lrt li'llltrtultttl Itll(rn(!s illI tltc QUL'stiatnairc ( )rl(.('l llr('r'n()stexciting devekrpments in web surveys is the ability to cttlItr I st.r'r't'r.-sirlt' ,rnclcspecially client-sidc paradata that records the server's (. ,rrrtl lit'nl's irltrr.rctionswith the questionnaire.By collecting paradata, one (.rr r',rPtrrr(' m.lny types ()f intcraction with the survey: how lon8 it takes resl,orrrlt'rrls kr arrswereach question (and the entire survey), whether and how ,rr rsrvcrsto t..rchtluestion are changed,and the sequenceof clicks that respontlt nts rrr.rkcorr cach page (Heerwegh,2003; Heerwegh & Loosveldt, 2002b). llris irrlornration can providc important feedback about how respondents ,rrrsn,r.r' spccific questions and can be collected for all respondents, rather tharr orrly for a st'lect few as is the casewith laboratory studies or cognitive intcrvicws. Thc following are ways in which paradata have recently been us('d t() bcttcr understand questions and questionnaires. . ('hristian et al. (in press) observed that respondents take longer to answer se.rllr questions when they are provided with a number box into which tht.v must type the number corresponding with their resPonse versus w,lrcn they are provided with a labeled scale with circular radio buttons llr.rt correspond to each scalepoint. . srrylh ct al. (2006a) found that, on average, respondents to check-all r;trt'stions answer quicker than respondentsto forced-choicequestions. In lLrrther analyses they were able to determinc that those answering r lrr.r'k all questions the most quickly were also more likely to select rrPlions.rppearin8 in the top of the lists, indicating that they may havc lrttrr satisficing (seeChapter 5, Guideline 5.18). . Stt'rn (irr press) found that students were having considerable trouble in a check-allquestion:"libraries" and tiisccrning betwccn two responses "library instruction." When "library instruction" was presented before "librarics," many students selected that response but later changcd it wll(,n tht'y saw thc "libraries" option. Howevet this was not a problcnr "librarit's" wls presented first. w,lrt.rr ()r'r'r' . r ll, wt ' hav t ' f ot rn rl th n t p a r.rd a toc a n l rc l very vnl l r,rbl crcs(,Ll ti (' l \rr ! rl ol (' ,l r()\' \,(' \' (' is r, tl )Jl r' r .rI rr.rI i rr1 o w c l r s ttrv r' y s .()n (' (,l ttl i ()tr,lv i r ( ' s lx ) lr s ('l!s r ' rtr l t r' r' Lr rr'irrl , rtl rr,tr1r . t r , r r l, r,l, rt r t ' , o l l t' rl rrl , ,r |.t t I Ir' ,rtt trrt rtt I trl i ttl ot ttt,tl rorr \' l rl o" l l l s{ ' l l l l rv l r| r' r' 1 ' .rt.r,l ,rl .t,.lltr srr i l i: , P , r r lir r r l, r r ' lY in rl )o fi ,rttll rr rrl r' rrl i l V .rl ' l , l r' l r rr o t,l r' t l o I' r' r' r' I' r' l t,tr to t ttt rrr rrt rrIr.t. ,| ,t t trI i t t1 11 -,;r ', ' ( l l . .l rl .l l v\t' \

I l on,r.rr'r ', sir r r I r h, iI I I or . r r r , I I ioI rr r . lt st , t s r - t , s1lr r r r str,r r ,is t . , r s\ lo lir . t ( r r lI llr . , . lt t , 1 tt],Itv w( 'l) sur . v( , y( ) t nor . s v r t , gt t lar ly ir t t . lt t t lt lot , . t l t r , sl) ( ) t t \ (lillt . ( , t r , ll{, ( lr ' scri bi ng t ht . it . wcb sLr r vt , y. r s well , r sr lt r eslior , r. , r f r , r , , , .1i, ,,, ,,,,s lr , lr r r r . r , 1, o11r r r 1, .rnalysesfronr ind ivid ual qutstions. Cuidclirtt 6.33:Ti'sltln,Surocy LIsiw t VorittrlLtfpltt.fitrttts, (.ottttttttrrt :i1,r.,.,1 Broutsgy5., on11 LlscrControllttl Scttings, onLli,si tttL, Dutnltursc lo I u5utt, lltttl Items Are Cttllected and CotledCorractly Testing the web survey is one of the most important steps in tlrt,rllsil,n 1,ro cess Although paper surveys rook the same for arl resp(xr(r(,.rs, r,vr.lr s'r r,\ ,, cannot be controlled in the same way and are ()n dcpen.lcnt th(, r(,sl)( rrr(lr,r I lr,. individual setups. All of the hard work in designing lrntl prog1.l111111111,, 11,,, survey does not matter if the survey does not work effc.ctivclv,rrr.r \,,r r.r(,tv r,l

o l o r 1 11 1 .,1 ,1 1 1 '.l l r [,l l

:,rx,,,(r,.,, r \ r,rrr .sr.,,,,,,.;".,1i:l:il,illi,,ll;illill;iill

,.l t r r .t..,. r l r .,l ,,, r ,l ,.,l l o .n .l ( l .r l ( .sl ( ,t.o l tl 1 ,,,r rrr rrI r | , , ,r ., I I

to.be a systematic process whereby pug" in th(, survcy rs vr(,r4,(,(l using these different combinations by"r,".y mjtiple people arrtl 1r.r'rrrr tiillt,n,rrl locations..Oneof the most important things to test is page d()wnt()n(i t jnr(,s to ensure that most pages download very quickly (a few seconds;, tvcn wittr slower connection speeds and processors. Another very important thing to test is what happens if scripting is disabled. It is important to both mate sure rnat the web survey does not crash when accessed by a browser with script_ ing disabled and to make sure that a no_script opiion is available for each Javascript (or other script) function in the survey.'Although it is not ideal to havesomc respondents experiencing the javaScript versioln anJothers expe_ ricncing the no-script version because the stirnulus may differ between the twq JavaScript enables many important web survey functions and therefore is likely b be included to some degree in almost utt *"U..,*"f. e, tcsts nced kr be' conducted with ii both enabled ".r"t, and disabled. Ii the srrrv"y nr ust u tilizt. JavnScript, it should be able to detect if a respondent has scrrpting tiis.rbk.d.. ()nc option is to provide a polite explanatio. u, to *iy scrrpting r(r.(l \ tr) l \ '( . n. ) l'l( r l lr r r I hq, 5ur y. , and how t o enableit . ()ltt'rr u,t'b strrvt,y rlt,signcrs and programmers are on the cutting edge of r)('w {\llrl)lrl(.r ttr hrrokrgy,h.lv(, thc most recent versions of web browiers, ,|tr(| ,r|r' ,r(lr.l]| ,rI t rrstr rrrr q(,ttings. izirrg llrcir ()wn rrs(,r However, it is important l o rl csi l i rr.r r r r llt sl llr r . lvr . l,sr r lr c

of rhepossible combinations of settings r,rtt)()r(,n_ ll,,_1Tl l:i". -ost) ual respondents may use to complete the survey. Morcovt,, k,stirrli rrrtrls

test the survey using various technologicaleonlilirr rations as determined earlier when ui"essiri.-g the capabilitiesl, ,n,,r,,rr,,,r. population. It should be standard web survey procedure to test thc surv(.y usrrg

sotrwdre configurations and across various c{)n }n:t \.(\r\ n{,\ rrr lill*,il::lO lnus, lt ts lmportant to

. )18

lrrr

'f,r

(_)r r',||(

'N

. t(l \ (-)r r',rt

NN \ llr l

l ' tr' l t

l l tt.' i l l

r,l tt,u

tt t

.' l ,r

(lill(' r( 'rtt (()tllPtrl('r'l. r l) s , t l olr r t t r li\ ' ( ' f s it y lt t ' t , t t t s t t t t , r t t \ r r 's l t o l t r l t 't t l s w r r t t l t j be co rlple ting tht'st t r v c y ( ) lr . nr nPt r s . lll t he Pr o e t 's s o l l l s l i r l l i , r l n . r s t l i s covered that two labs trt c.rntlrus lrtrd llot updatcci tht'ir lrrowst'rs itr 5 vc.lrs Although many studcnts were awarc of that and avoided usit'rg those labs,

l'l(lr ttSttN(;

(-)|tS

()NNA

itS

othersdid not know and may have completedthe survey on thosecomputers with the outdated browsers.We believe this situation is not unique and that even more variation exists in types and versions of browsers and in other as individual usershave control over their hardware and connectionspeeds, own machinesand how they run them. Under theseconditions, seemingly small problems in the testing phasewill often becomemuch larger problems oncethe survey goeslive. In addition to testing the survey itself, it is also important to test the This step can be done in combination with testing the survey. As database. and record what testersview the survey pages,they should enter responses can then be checkedin the database.Every they entered so the responses option for every questionshould be testedat leastonce.Testingthe response is collected for every question, databaseinvolves ensuring that a resPonse selected, for the response(s) that the correctnumber is coded in the database (unless desired that open-ended text boxes are not truncating resPonses coding that may be unique to and that any other response by the researcher), the survey is working correctly. Page of theFinal Questionnnire of Each 6.34:Take Screenshots Guideline for iort g and Document at Testin of each Relatedto the testing processis the necessityof taking screenshots page of the final questionnaireso that PeoPlecan seehow the questionnaire can serve as the docupages appeared to respondents.Thesescreenshots similar to copiesof paper mentation of the final version of the questionnaire, questionnaires. Once a survey has been removed from the Web it is often of hardware and softwareupgrades, very difficult, if not impossiblebecause if any amount of time has passed-The especially repost it for viewing, to were taken to document the questionnaire, reality is that if no screenshots there may be no availablerecord of the survey's aPPearance. using should be takenof the final questionnaire Ata minimum, screenshots recent (i.e., the most a PC using on computer settings most common the version of Internet Explorer with ihe default settings, and with common Howevet a betterstandardis to takescreenshots screen sizesand resolutions). usirrg as well as shotsof the final qucstionn,rirc during the testingprocess st'ltings ln .ttitlitiott and uscr-controllt'tl a variety of platforms,browsers,
t o documcntin g th c f in. r l v c r s ion oi t ht ' t 1t t t ' s liot tt l , t i t t ',l l t t 's l s c t t t t r s l r o l s t . t t t hr' l1tsrrrvt'y()rs trrn tlt ar t ' lr ow llt c s r r I t ' r ' V hr ol* s . t r tr r r r l t l l l t r 'tt l r ( r tt l t t l I t - , ltt ( ) l ) s irt llr. lt'slittli sl.t1 1'.

I)tl ri l l ll. l . ( ) t r r it i\ '( .inl( 'tyi( , w,,,t n. sl) ( ) t ) ( l( , t6, ) l lr plr , r r ilr r . r r llt . r ilr . r r , ( lt r , . rl, lr l, atrrlw as t 'r r lt 'r ir r g il nsw( , r sil l( ) , tnc! , , 1y, r lt , r . t , kr pt rwr l . lrvt , r sioI r r r l llr l N. rI r ,, rr . r I 5c(' nct 'lit Lr ndat ionSr r r vcy of ljar . r r t , tl) l r r . t or . r k. slr r r it r r r cr i, ir l, lt lr l, , , i, t , , lr r ' obj ect cdat lant ant lykr t lr e wor t iing of t ht , lollowir r g t llt , st ior r , r l, or r tlr r , rr , r ,r , C14. What is your racial background? ! Am er ican I ndian or Alaska nat ivc ( l'lcast , t lcscr ilr c or r r r r . r ll, , r ll. r I Native Hawaiian or other pacific Islant]er ! Asian ! Black or African Americarr ! White D Decline to Answer Turning to the interviewet shc said, ,,l am of nrixctl r,rr.ial lrL.r.ir,r11,, .rrr,r rr really makes me angry when you forcc mc into ()n(, c,tt(,illr.\.. Wlrt. rl., r,rrr write questions like this?,, Severalminutes of discussionfolkrwcd as sht,t,l.rl,or..r tcrI r ,rr lr.r | , )rr ,.rr,. Finally, the interviewer asked her to takt, itnotltcl lool ,rt tlrr.,lri-,tr(,r ,rr,l see i f ther e was any way she m ight bc ablt , t o r t , liist cr..r r r . r r r . , r lr , r t lr . r t r , r , , acceptable t o her . I t was t hen t hat her or igir r . r l( \ ) n{. ( . f n lu|1r ( , (t l o, , r r r lr . rrr. r , , . , rnent as she r ealizedt he boxes in f r ont of t ht , ansr lr . r( . r t ( , li( ) r r , . , ,u . lr r ,lr r r , . r , , check b oxes and not r adio but t ons, allowct l lr t , r . t o r 1r r , , , . , . . rnr : 1 . r n ir l, \ \ r , ri, as she l iked f or t hat quest ion.The r espor r clt , r \ r\i , . r s( luit r .. r \ v, r t r , ( ) t lr , , \ r r . l, Pageconstruction convention, but becauseshc t..rs so lrr rrsrrl orr rrrr.,Jrrr,,, ti on w o r ding, she m issed com plet cly wlr at t hc ansu, t , r . r . . r lr ) ior ir . s, r llor r r . , l l ' rerto do ( Alt heim er & Dillm an, 2002) .The sinr pk, f ir t or . t lr i. ,l) 11r t jt r . r \ lr \.r., to add the words "You may choose more than orrc resporrst,.,, i | , r. ,.r,,,,, 1,l,, illustrates the importance of evaluating questions .rlr(i qr(,sli()nn.rr(,\ l,nr)l k) the start ()f data collection. No matter how hard orrc has worktrt rrr olllr kr build a grxrd questionnaire,the finai essentialtest is whtrt hapPcrrsr,,r,,rrlrr r('sp()ndents conle faceto facewith the survey itsclf. Truditionally, thc term pr.l.,sti,l.q referred to delivering i.lqucsti(nlr,tir(,tr) irrriivirlLr,tls with spt,citrlknowle.dgeof the topic n. of tht, srrrr,.r, -"^b"..; I)(|[rr| |iI t i(rI] .rntl askirrg tllcnt k) completL,it itnd report any probtcnrs t lr{,\. (,\ p(,rr('n( ('(1.In r('c(.ntycilrs, n \,,lri(,ty of sPccific proceciurcshavc evolvttl l ()r (' \' ,t l u . rling ( l illt 'r t . r r lk ir r <ls of t lt r t , st ions , rnr l t lr r cst ionr rir a cs.M any oI t ht , sr , ar t ' t lt , srr ilr r r i in I 'r . r , sscr . (.,ll. l )r' (\r\l ur '{. s 1 ( 2{10. 1) , . r r t r,t.j r ehpr ( ) ( . ( , du11, c( ) ntl l l rttl | s l( ) llr c r . r '. r lr r , r lir r ' h ol llr t , ollr t , r Pr q1. 1, 11111q, * l) r . ( \ r , s\ir ) t \ , , lvsllt . t l r , . t r r.rtttol All . ll lnr l) of l, lr l l( r ir , lr . slir r li l) f o\ it lr r l, , t ( , ) r t r l, r r , lt cr r . r ol {llt ( , slr ( ) ns ,ttti l rl U r '\ lt o n. ||t r - , , . r r r , l, r , . , . r r r lr r ! r , ( llll'r r . r r ,llr . r l r . r r r Plr . 1sl7q, 1, , r r ir |r . I r r Ilr cL l l rr' " l rcrr 1illt .,,r r r , lt r , , . , r 1r r , , , , ,r.,,l, r . , r lr r | ' rr,Ir,.,I rrrI i r . , . r l. g ll1, ir lr ; , 6r r r l, r l t r . lr

.'.'t)

ltr',Nl (-)r r,.rr,".. r , '

\ ( - ) l I ll( , f r ' ! \ llil N l . r r r y s lr r ,l i r 's l .r r l Io .Ir Ir i cr l

l 't t,lt .l t,t,,'Il |1 ",1 tttu u t tt .

.,.,I

t ll r( \ l i(

) |t t I I ( I I I I I t ( )| I lr r '1 " rrt l . , r r l l | r s r 'r l ' l l t 't 'l t l s t ' (' (l('si1"rl 'r t t t l t l t t r r ' v i I I I I ' ' ' 'tlll'lll ( ( \ '( 'n t l r ( ) l l 8 l ) rll llrr. r Ittl i,rt n ,rlur(' () l llr is ilr l( l- s ( \ ' t iol) , wt ' t lis t t t sr l 'r t l l t l r | t r ' I rI rI I I (' | | I ( ' t I I ,I t i ( )I I is llrt ' t oPit ( ) l t ll( ' t r ( ' x t c hJ pt er ) t ( ) ( 't l l l ) l r 'l s i / t 't l l ( 't 's s e n t i 'r l lirrkagc ltr.rt ttrttsl [Tt',rchit'vetl itl thc crc.]tion of an 'rccePt'lbl(' stlrvcy' t;rttlr'li l'tqlt, ()blnin I:tLLlbockrtrt tha Dr ft QucstionnaireJront a Nunbtr of t'1,..\5: of Sotne Aspect ol Lrttlt LtlWlu)ttt Hos Slx:cittlizctl KtloTuledge Qtxtl i lV

IIr l i I o l r j tr ti r cs l \\,l r sl su r \,( .\,r ) r s l r r r r i t l l r r s o r r l r l o tr r l k'.r g r r r .sr l o r ,r ,r rl l r r . l r ,r l l r ,,.r v r \,l t( ) .l r r ,( ,\p ( .r l s l r l ) h , r s ( 'r ) l l ) r '( 't( 'sl i l l l g s ( ) n l ( 'n s P ( '( t ( ) l \u t\'( 'y c( ) n sl r u cti ( r t o r , a l tl tt.( ) l l t( ,f ( .\l r cn t( ,, t( ) P( .( ) Pl t,\\l l tr a r t 'n r c n r [- r t'r s o f th t.stL r cl y p o p tr l a ti o n . Qu r ,sti o n n a i r t,s l a i l fo r n r a n y r ca so r r s, antl having i'l ijystcm.rtic approach to otrtlinirrll ft,r,clback from a varit,ty ot knowlcdgeable pcople on tr complcte clr.rft of thc cluestionnaire is esscnti.rl for c'vaitrating, in a preliminary w.r\', potcntial surve.y design problems.

l) t t 15lituttlnir

whether Sonrc peol,le art'itblc to look at questions and provide feedback on Others are lll('y lttcrtsure the concepts that the surveyor intends to measure .rl,lt'to kxrk at a list of questions and identify the potential for unintended

Cuideline 6.36: Condu(t Cogtritira I tuoi(aE of th( Complefeeuestionnare u1 OrLlerto ldenttfu Wording, QuestiottOrder, Visual Design, nntl Narigatton Problems Cognitive interviews have now become the dominant mode of testing questions and questionnaires. Some organizations, such as the U.S. Census Bureau, which does many continuing surveys, have even mandated that before question wording changes can be made they must be tested. Cognitive interviews are considered an acceptable procedure for evaluating such changes. Cognitive interviewing emerged in the early 1990s as a means of determining whether respondents comprehend questions as intended by the survey sponsor and whether questions can be answered accurately (Forsyih & Lessler,1991).Potential survey respondents are asked individually to respond to a questionnaire in the presence of an interviewer who asks them to think out loud as they go through the draft questionnaire and tell the interuiewer what is being thought about the questions and also how answers to quesrrons are being formed. The interviewer probes in order to get an understanding of how each question is being interpreted and whether the surveyor,s intent for each question is being realized. When a think-aloud cognitive interview begins, an interviewer explains to the respondent that she will be asked to complete a questionnaire in a special way, which is outlined in Figure 6.18. The respondent is told that this includes telling the interviewer everything she is thinking as she develops and reports her answers. The respondent is then asked to complete a pracrrce tluestion in order to learn the technique. This part of the interview is critical, and we commonly ask two questions: How many residences have you lived in sinceyou were born? Nrrmber of residcnces ll()\'!,nl.llrv wintlows arc in your home? N trrn[-rt'r of winriolvs l l rt' l i r sl r lr r cslior r is olt t 'n t lillit r r lt ior 1. r t r r p11. t o answer ,cspeciallyif t hey l r,rrc l i rr' , 1ir r , r lol ol r lillr . t 'r r l I r t . . r iior r s. l\ , 1'i, ir llys, 'r ncl, ( \ ) pl( , st ar t count ing 1,1' rrl trtt lr , r r 'lr cr r . . rollr : , r . r l\ \ ) l, l( , lt . t lr . . r r r lslir r r . r lr , ll . lr , t slr t t lr ( ) uf ( . \ p( , r ir' rtti rr nr) t ( 'llt , lt ll) 0 lr l' t \ r . \ r , , r r r r r 'lt r , lt lll, {llt r , . , t ir t llr . r sl, r , r . rr r sr . t llr l ,tl

(lu('sti()n ordcr effects. Siill others may look at a questionnaire and identify ,.iuestions that should be asked but are not included Some reviewers are are ablt'to evaluate question structures and identify response categories that inapl.tropriate. For example, a reviewer of a student-designed questionnarre one tlrat used five household income categories for a questionnaire' only ihat designer io the of which was more than $20,000 per year, pointed out most respondents would be clustered in the highest category, thus limiting variation that would be helpful in the analysis Some people have familiariiy able with how demographic data are collected in national surveys and are comParison preclude b ideniify question; that are asked in a way that will with answers to the same general questions from other sources An examPle of such a survey is the U.S Census Bureau's American Community Survey' which is now relied on to supply detailed demographic data for the nation' its states, and most of its counhes. quesOther individuals may be able to identify technical problems with tions that people who are experts on other aspects of question constructron to a may miss. We once had a reviewer comment simply that the Preamble are responsible community question was not true: The phrase "Police in this " was inaccurate' for arresting and fining people who violate traffic laws levies Although ptlice arrest people, it is ihe judicial system that actually to be population the of knowledge with reviewer fines. [n another instance a of a survey mernbers all that surveyed informed us that it was very unlikely popuiation could answer a few specific questions among a series of items and be added' iecommended that "don't know" and "does not apply" categories of It would be unusual for a single person to have the ability to idcntify all questronTesting a questionnaire and the potential problems with questions naires for these items requires getting evaluations from people with tcchnicrl itl knowledge about the survey topic, how dcmographic cl;ttaart'collt'ctcti sttrvt'v nlotlt't'llttls' atttl comparison surv.'vs, statisticll .tn,rlysistt'clrtritlttt's, l " rr' ' I rr l ' ' ch a r,t c lc r i. t ic 't ' I t lt r ' l ro I' trl ' tl i ' tttr' 'I.hr ls ,t |r is s t . |) } ' ( ' l rr' i | | l .t l ' ,trrr.| r' r' |rIrl It' (| l | | | t\' L tl l ts tt| | .tl i rrrl l )| (.1 t,:' l i Il ' i rl tl l Itl trl l r,tl ttl t , r ' r , l' t rr ' llr or " rtr' ,r' , o l ' \l ' r' r l l ' rr" l rl ' l r' lr' 1" ' :' i l | rrl i ' ' rtti l v

222

ttt tt tt t,ttttlt ltuor'aQtttsrtrtNs trt n Qt ttttt

,'ft'trsli,,,{Qla'sliorrrlrrin.s 223 Flgurc0.lE (Con nued\

l'. A ...pondcnt to nlort the torm contrtld In d

Flgura 0,18 Exampleot protocolusod lor tsstlng U.S.consus questlonnairos.

wc ltkcd for yourhclPis thalcvcrylcn vc.6 th. lhnrk you lbr (dnnrt hcr.l(xl y nt hch usoxt. lhc rcoson who is livinSin thc UniicdSlslcsAll resdenccs trcomnlctc connlot cvcryonc tl S. ( cn$rrllurc u condn.ls comPletc andrelumn TodayI am wholilc lhere thd pcoplc lbrmwhh lhc roqucsl rrc tr ilcd r||. ( ctrsus us. in th20| 0 Cmsus Your fof possibl lbrmslhul{rc beitg evalurted toirtt kr llsl you lo kxlk al ( cnsus as to compleie thatwillhelp makethefom aseasy rctrdiors t()thc$ lonnswillprovidcuswnh information ll. llrnd rcrpordcntcontldenllrlltyro.ln fionn. But fiBt ltnexplainwhatit is lhc lirsrihrngI rccd to do is to askyou ro rcadandsjgnthisconsenl in whichis localed Burcau, by usfor lhe U.S Cnsus lt is bingconductd rhnn. I his inicrvicwh voluntary. whocansethe formis confidtrtial Theonlypople writeon theCensus Wr$hiDglofl t).(. l.:vcryihingyou ili)nnstn)nyouprovideareemployesoftheCnsusBuresuandthoseofusatWalhinetonStnieUniversity Burcauto keepindividual uswers who *ill he conducrinsthe inieruiews.We havbenlwom by rhe Census thepcMr poples' dswers in anvwavthatmakes specific dndwe canbc finedifwe rcval eDntidcnrirl, volunleered for thisinieNiw I youto signindicates thatyou have we arcasking i c tiliablc.Thc sraremcnl conductinglhe interviewandI want to assue vou in $dting of bv witl llso sign it as wll sincI an the preon promisc io keep all ofyour infomationconfidential. ( . ]:rpl.ln prccedure when I do' I would likc vou !o h I colple of minutes,I am soins to handyou a censusform in an envelopc. Irlk our loud aboutyour reaciionsto lhe form asyou .eadqu6lions andfill it out. I would like to how so beforc I cvcrythinSyou think aboulit. Talking out loud aboulthesesortsofthings mav sema little unusu.al' tell me mailing. I havea really shon/ra.t.e miling WhenI givc it to vou, please Sivc you lh Cnsus andstartdecidingwhal l,odo with it andthe fom avcrfhing you arc thinkins 3syou look at the envelope. in$idc.I would like lo know any thoughlsyou haveaboutwhetherit strikesvou in a favombleor unfavomble way, whthe.it is clearaboutwbat to do or not do, andso fonh. D, Hrnd..lpodent pr.ctlce m.tl-o|rl you arelhinking aboutwhilc vou fill it ouloul loud andtell me everyahing Oloy, p6aseradthe quslions (Providepositivereinforcmeft,e.g."Good, that s \9hatee heedto kto|' "l ( Encoumge the rspondni to providc other informatiotr,e.8., "Wen tou do the rcal CeB'.r fom i4t he srye ke \|ay the wholething looks' ,)hethe. it s thott'u tell Lt obouttow reactionsto ewry'thina,theeneelope, clear|9hattodootnot.lo.arythinaloudontnddsta,olarythingthatseensstrange") 3:, llind flnt m.il-o to respondent for which vou arconpletitg the Now hereis the dvelope that might arive in th mailbox al lh address ( cnsus you have $al you seein to lervthrtrg takeyourtimeandtell meanyre3crons tormin 2010.Please else.e g . anetderlv fiiend. mark hcr! tr andnake sure for someone tmnt ofyou. (Not: Ifprson is rcsponding rcspondenl udentands ourexPectations.) L Any reaclionsto the miling packag: 2. Did they rcad the coverlette EFtrlly EPanially E Not at all :1.Did rheyractat all to the opponlnily to fill out lh fom ele.tronicallv (from lhe lf so.howl coverlerteo?

doD'rrcad,r h.E. w. d tir,eroryou ro fitr ir ourJU;r hlc you\outd dr honlc, crrcnr r,i ir,* .r"",ir ,*, loudaboutir. andanyrhrns you Ead ro )ourNef;outd hr rcadourl ru..itrcascgo'uhc;i , "iii"
Prob.s thar miShi b uscd: . Whal are you thinki"E ngtu ,to||? . Renenber to rcad atod for ne t,s uD to r'u whaty)u redd, hut N,hatetv.r.t rl.tith.h, htrt out loud so I know what rou ate tookiip u. . Ca, you tell ne nore about thot? . Coutd you desclibe thatfor ne? . Dot I lotget to Ett ne what you dre thinkina as lou do that.

imp,cs*'N o,,r wcworkrr,,c,itr vo!,,' I:Ii:::"ii:^"::.1*::i.-ji.li'r-s1'|1lJ'lk::':irdul rcad wharcvc, you trourd ,Lsr arhonr wh,rc rr,ng,rr,, r,,j*"".,,',i.ir,J" ,_,,,r,ril;:ii;;illil;,1.,,

sh'urvou,

tthtt\.1t,,

G. Reord r.levfir cobmenr!, erro.s, hestrrflotr|,.rd oaner l||dtc.rors of porenrtrt probtcmi dtrrtnt completion (ro be ured to rrrne folow-up qu.!.tors). L Did Lheyrcad rte nole aboul fi ing o rhe fom etectrcnrcajt'?

trNo
2. What readionsdid they volureer, ifany? 3. Did they readth rostrinstructioro? E Fully E Partialy E skimned E Not al all 4. Ary rcactioDshsitariontquesrions to the rosrerhstrucrions? H, Ihbrlefirg qu.dtorr (filst form) L Over"ll how earyor difitcutr *as rheform ro comprelei,

trs.;w#e*"
E sonewhat dimcur E Very dimculr 2. Wis thra''rhing unclea.or connBins abouthow lo fil oul dis Cdsus form? E Yes-t(tf ycs)pt.aseexDtain: ON o 3. Ifthis fo-rmarrivedal you rsidence in rhe mail, how soondo you think you would E The sme day E l If,l -2day s tr In a weekor so tr Two werks or more E Not at atl

2005 andDillman, Mahon-Haff, in more detail byPaNons, reported lJascd onprocedure

somepc'oplewill think of citiesin which they have lived, whereasotherswill Regardless of which way people answer' think about individual residences. that when we haveaskedthis questionin such inwe cxplain to respondents haveanswcrcd cachway.Wealsoexplainthatwhen wc tcrvi('ws somcpeople diffen'ntlyit tcllsus that thequestion intcrprctthc qucstion It'arnthat ;rtxrplc of lhis naturc wt'tio itrtcrvir''ws tlrll in why improvetl, ntrtl lrt't'tls to bl

People's answers to the question on windows usually allow us to begin to probe, for. example asking whether they counted a slijing glass doo4 what they may have done with any multiple pane windows, anjsi on. By the time this practice questionnaireis complete, respondents have usually leamed what the interview is designed to do and have become comfortable with thinking out loud. In addition, the entire interview experience has become far lt'ss thn'atcning to tht m than it may hnv" uppeared at the beginning of thc ink'rvit'w scssion. I'rlcticc qur,stions shift thti tmphasis from providing nllHw(,rs th,lt,rn'right,nntl tlr,rtnr,ryrt,flr,ct orr tht rcspondent,s compctence,

, r, t . l

|t r ,

rn r (-)|l r

" l l (|N ',

li , \ ( - ) l

l ,lll' NN

\llil

l ' trl t

l l ttl

t\. l l i t\J ttt 1,ttttI

(l l l ( sl !(rrl s\\ l l l l (' l ) l ry I I I,| it t I t ' tv t |\ \' I I rrl ' ' trl tl t l ' to l r[' ttts w i l l l l l l (' lo l rr'l 1 r ir r 1' lltr' \l II t r. \ t |t ' Ih(' l l l t(' rvl (' w ' P r()l rt srr t t r ,llt t ' l) l. l( li( ( ' (l tt(' \l i ()trsrl t' l i tl t't' x l rt' tt' tti o l rsl ()r lus\{'erillg ottt-loutl lol Pr(x'('ssthit witlt llrt' rt'.tslrrrs 'n r r,lr' tr'slrrrttrlt'ttts why wc arc undcrstind tht'm to tll'nl, antl help l)l)l),rl)lv s(\'rtls tttrll.lttlrnl k) trai n the ,1 ,,i n 1th ,, , ' ir r l( ' f v it ' w.l h c s ec rl tc i a l b c g i n n i rrg sth u s a re desi gned thcm to be acti\,e particiPants ln n1()tiv.lte irs w('|l .1S .,.'t l.,rr.I r..,sl'r,rr.l('tlts Il rr'1 rr'r x t s s . W|r., r r r lr . . t t t t t ' s t i< l l rl ra i rc t()b e c v a l u a tc d i s h a ndedtol esP ondents(or to continue !rl)l\\rrs ()tr tltt'scrcen for web surveys), they are encouraged Detailed ext,',,1,,,uhat tltt'y have becn doing with the Practice questions that have o[ l,rotocols and questionnaire evaluations we have done ,rrrr1,lt's wwwsesrc site: web this from rrlilizt'rl thirrk-aloud tcchnitlucs arc available .w ,srr.t'Lr r l/ d illnr r n / . whether tinres cognitivc interviews are aimed primarily at evaluating S()rrre wording of questions in the same-w^ay and un1r'oplq ,11gintcrpreting thc In other the questions (Parsons,Mahon-Haft' & Dillman' 2007) ,1,'rltnn.li,,1; to naviable (',rsr's,thcy'may bc aimed at determining whether People-are When 1995) galt' through a qucstionnaire aPproPriately (Dillman & Allen' used a retrospectlve llr('l.ltter is <tur primary concern, we have sometimes silently' questionnaire a to complete int(,rvicwing techniquc, asking peoPle observe simply then We home by themselves itrst.rs they might if ihey were particular, whether mistakes get made' The in and, p.ocess th" nn"*."irlgseems likely to r(.ds()n for this'process is that asking them to think aloud attentron ('ncourage resp;ndents to read questions slower and yith 8Y1"t technique This home at t,, the wording and visual layout than they would of.a questionus to gct a better sense of how the graphical layout <'rr,rl.rlcs has allowed n.rirc is guiding respondents. With web surveys, this technique as a means clicking backward and (e.g., forward rrs kr pick up mannerisms understand to context) ,,t trying k) gct a bctter ctlmprehension of question when not ways that respondents navigate through questionnajres ,.lillt,r'r'rrt reading questions It.tr,itlgttr t]ividc thcir attention between the inteNiewel, providing substantive orrt lotrtl (wlrich is slower than doing it silently), and (Sawycr & Dillman, 2002) .rrtsrvt'rs interviews withWt'cirttttot r('catl hilving completed a set of cognitive question wordings or oul itlt'trtifying at least a fcw potential problems with initial Iiryouts, r.'gardlessof how much cffort went into the tlr rt,stiorrrrain' lntervrewcognitive that rcc()gnizc llowcv|r, it is irlso important to i.l,,sigtr. ally pr,tcticctl,t'xlribits 'r nttmber tlf shortcomings llesPond('nts typi. irrg,'.rs ()r rt'crtlltr,',,,f r,, fr,'r'olrttltttrs obt.tirrt'tlthrorrgh 'rtlvcrtist'mt'nts Pcrsollill o .o l l l (' l () ' l tl trl ti tl l rx ' tl i ott i or tht' P rrxtss trt rrrl n l , lr r r . t t t s col lr t ' i rrg .rs l ' .t'ltl s.trrtl ' | r.rl |.r | .tt.1' ,t.t.stttt| y rt ,ts .t l ..ttttItr ttt |.rkr.t,l. t r r ' , , lt t t It . lllt t rl l l ,r' tl rrl rrl ' ,,|rrl

ir r lcrr r cr . r 'r r s l. r lr or lt l( . nr - i\ . ( , , , llr l , r s. r li( , lr , t , t l nl[ , llr l I ' ()| )t r|,rI i()| | . l ) or r r l', ...t1\' f ( ) l sr r tlt ir r lt . r -i. r r 's r r r 't lr r r . l. r l is r r llr , r r l'il. s'r , t ll {r , . 1g. lr,. ssllr , r r r. , 01 Ittastl l ttclr , t s P|olr lt 'r r r s , lr '( . 011( . ,n t 5, s( x. i. ll(rr!iillr I r , r r . I it r I |. r r . lr r . r r . rlr r , ti: , t r r , , l tht' rt' spot r clt 'nt r,t r r r clLt sions. t r - ( 'olt (lr ,n ( lnl) , ( ) r , l\ . f slr ll. r \ lt r li . t st , t lr r l. r on thnl pnrticul. lrch. lr lct ( 'r ist ic. An cx. r nr Plc. is r r r gt r ilivt ,inl( , t . \ , i( . \ \( ,, s1( r . r , , r , , forms th . t t l'r it vcf ir t r sct l ( ant or r got her issut , s) or r whv n( , n, lr or . ls. ll r ! ( , n omi ttt' tl fr ont t lr c Doct 'nnialCcnsr r r ior nr . An, r r lt . r ; u. t t r . t lgnI lt \r . l( , sllr r (r r , , r , , I ort thi s i s sucshould in pr inciplc, int er viewonly inclivir lr r als lr ( , nr I r(, rrr , t ,I r(, Ir |., having a recentlyborn child (Cork & Vrss, 2{)06). In ou r t,rPcrit,rrt t. it lr,rs lrr.r.r r disconcertingto listen to strong rccorrr mentla tions fol clrarrgt, sorrrr.orrl lr.orrr w ho has conduct edr elat ively f ew int cr views or r a v. r r it . t v, r l i\ \ lr ( , s, , r r ( tlr r , n l earn that t he r ecom m endat ionis sr r m et inr t , s [ r a. t , . 1 Lr p, , n, lr , \ ( ur r ( , n1 r . r l only one or perhaps two interviews or an intcrvicw with ont, r.t.sPrrrrtL.rrl with a particular characteristicof intercst. Under these cir(.unlst.lr((,s rt r, difficuli io know whether any problem that has bcen irlentitirri is .r srrr.rll isolated problem or something ihat will affect a significarrt nrrrrrlrt,r. ot r.r. spondents,and whether changing the question might then cr(!tl(, l)fol)t(,t|r\ for other respondents. Another problem is that the labor-intensivenaturc of cotrrit i\.(, |lrtr,r.\ i(,u,s often leads to their being conducted by entry-lcvel personrrclw h() ,rr.r, ,tt)tr.t{r ask and record responses frorn individuals but who lack lhc skills or. tr.,r rrrrrrl needed for comprehending underlying causcs of tht,prol.rlt,nrs,rrr,|.rrlrrrl,. appropriate follow-up questions.A friend dcscribeclit as tht,(lin(,r(,r)(.(, t)(, tween a medical patient being examined by a physician antl tht. ph_vsir i.rrr,s ()ffice assistant. Although the assistant may have mcdical training iln(l ns ,l result-be able to ask appropriate questions, he probably lacks the in_tleptlr knowledge needed for formulating the needed additional questions ro rure out certain possibiliiies and better isolate the critical problems. We have ob_ served some interviewers who are tremendously skilied at teasing out when interpretations of wording are problems and interacting with the respondent lo identify words and phrases that are easier to understand. However, these same individuals are unable to pick up on the consequences of visual lay_ out probiems and how they might be corrected. The opposite situation also ()ccurs. It is essential that individuals assigned to the cognitive interviewing l,rsk ()f evaluating web and paper surveys be knowledgeable in both of these
. ('i ts.

Wt'll-clone cognitive interviews also require detailed reporting of procetirr|.(.santl cviclt'nce, just as is required for writing quantitative dara reporrs. | ).irrg I hat .rll.ws.th.rs t. rt'rrl the c'vidence and draw their own conclusions ,rl )()l rl i l s s lf ( 'nBllt .li'n, lhir r gs. r r . ls t lisconcer t ingas having t he conduct or (l l f0 ( {i l l nili\ '( . i lr . r 'r 'ir . r vs sr . lr .tr . r Pr obk. r r r f r 0m onc int t , r vicw and sim nlv rh' rl ,ttr' l l t.t l , t ilt ( . 1, lr r lrr t r t r '. I 1'r 'r lr , t r r lgr r l. liyir lt . r r t . r . is r r t , t , r lt , 9r t l r u, h. r t . liclr lr

v
l l r rtl l l r' ti rrtt l rr l i r rl r(l ro l lr . r t ' t r c rr ) llt r ' (' l l tc t l rl l l l l ( r\ l { ' l r' sl o ,tr o trl r tr' tl ' .ttl .r tl t" tl t' s ,t Pto l r[' nr { ' \ lr t t i( ' t r { { ' (ll ,\' r' ttr' l tc ts o tt, o l l l y l o l i l l (l l l t.tl l l l c tcvi si trrr (i (' v(' l ()P th(' tts l () l t.l s l t'tl l l l i s c o l l ((' rl l h' s 1 ' rrrtr| r' rtl s . ollt c t lot P rac Ir1 rl )l ('nr s it in (l i v i (l l l ,rl i trtt' rv i t' w s i n rc P ( )rts,l rr.l ki l l l l stl r(' pro[]es Irrr, o l s r r r r r r r r , r ting n a s m l n y i n tervi cw s l s possi bl catl cl rrrrl l rr's , r r t t t ' l( ) l. ir s , r r(' a tl n ti tti s te rt'itl & d c tai l (D i l l man, [)arst> ns, i n c o n s i d e ra b l c ll r.rl l l rt' t ' r ' ir lt ' nt c is s t tn rn ra ri z e tl I',rrsons& l)il lman, 2001t). M.rlron I l.rlt, 2(10.1; sl/lrly cmerged in the past decade to describe tests of llr. tclrrr rrsrtlri/llt| interviews in which an individual rt't'1, srrrvt'vs.'l ltt'sc tcnd to be one'-ttn-one .rllr.nrl)tsl() (\)rrlplete l web survey. Initially, such usability siudies tended Ir) l(x rrs on tht' challengesindividuals faced in using a comPuter to regisIr.r'r(.\l)ons('s.Ihis practicc has, ovcr time, become somewhat blurred with r r4,,nilivcintt'rvicws irs cvaluators have attemPted to combine evaluation of llrt rvortling antl setlucnct'of questions with the manner in which they are recent years the term l)n,s('nl('darrtl rcspondcd to while on a computer. In questionnaires. evaluating applied to rrrrrlri/i/r/ has also been PaPer ( )rr',rsion.r lly, when wc have proposed conducting cognitive interviews to r.r',rltr,rtt' ir questionnaire,the survey designer has thought we were planning in which a number of people are brought togetherin order to trr rkr lirr'ris.qrottps, Thereis a considerable qucstions in a questionnaire(Morgan, 1997). l,rlk ,rlrout grouPs. The latter tend to and focus rlillt'n'rrce l.lctwct'n cognitive interviews l,t, ,r sor'ill experience in which people not only express their own opinions but list(.n to tlr(. opinions of others, which then may be taken into account as they r,rl,r't'ss atlclitional opinions. The group orientation of such interviews may tr,,rrl to inappropriate conclusionswhen cultural considerationsare related to being studied. t lrt. issrrt. Strch ln cffect appetrred to hapPen in a set of focus groups conducted in of a rcsponst'tate experiment conducted in 1995.In preParation ,r!rIi(iI),rti()11 lor'llrt'2000 l)eccnniil Census an extensive set of exPeriments was conrates rlrrt k'tl witlt tlrt'aim of identifying factors that would imProve resPonse (l)illnr.rn,200(L]). I he influencesof 16 different factorswere testedexPerimenl,rllv lo tlt't'rnritre the likelihood of their improving resPonserates.Only five l,rr'trrls wt,rc found to bc significant, the most powerful of which was the inclusrorrol a box rttt thc outgoing envelope that informed recipients "U.S. Census Vrur l{esponse [s Required by Law" (improved response lirlrrr l,)rr'krsr'<1; r',rtr.sby .tbottt l0 percentage points). Smaller effects were also achieved lry rt'sporrtlt'nt friendly design, Prenotice lettet postcard reminder, and qucstittnnaire.Howevet four focus groups in two different cities f(.1)l,rr('rrr('rrt tll,rl r,oulrl!t to tit'termint' whcther such an announcement on the enveloPe slrorrltl lTt'rtscrl wt're unlrnimous in concluding that ii would not affect restratcgy would be to slronst'r-.rlt's ,tnd that a tlort'efft'ctivt' rt'sptlnse-intlttcitlS " lt ()l l l ' ,l y\ to l l t' (btrntctl ." A r'.r t lillt ' r c t r t. r r t l( ) trrl .(' tl l (' l l t l h t' c n v t' l o P r' : t,l ,rt l r' 1' r' ri tttt' tl l , ' , rl -i (\l ()tl l i tt l l tt' s ,tl n l tr,rtirrtt,tl l l sl o l t lr is t t r ( ' s s , r t 3(,l ' l tow t' tl (,trul tl i tt to. . i7:Wlu,

I' t tI t,l ttr.r'( l ttt.,l trtt tttttI t.

l l r.rl rt lr , r r l . r lr solr r t , , lrr r o r , lr ( \ I or l r r r \ lr ( ) ll\ {'i t 'r lr ' ( l ) t llr r t . r r tlir . r ll r , r , t . r l l (r()(r). llk, St nk'r Lutr* rri o t t s. let r t r , , , , , , , , , , , , , . r - , , , , 1, , ', , 1, 1, , , 1, , 1, ii, h'( 'ot t sit ht l) t ) it t '\ l\ l't t t llt t t t lt t l ()ncc, when anothcr criticai dr:cisionwas about t() be made at tlrt. U.S.( <.trsls llureau t:rlrtht, Dccennial Census qucstionnaire, onc.of us was.tppr().rche(i by a senior statistician who wanted to know the following: ,Can,t you just rftr a few cognitive interviews to find out wheth", lt,, okui t,r .tlunge to thc newJy proposed questionnaire format?" Doing that inrtu;; of u., would have been less costly and ""p"ri^".,t could havJ been ;;;iil much more ctuicklv ' This incident illustrates a common dilemma faced in federal agencies as the practice of testing questions before changing tfl"_ U"."-"" a common expctation. Well-designed experiments can provide quantitative estimates of the effects of proposed changes in questionnaires and survey implemen_

needed. ln the 1990s the U.S. Census Bureau commissionetl an outsitit,firnt to design two new mail-out envelopes and questionnairr,",',rn" li *fli.t., *.,, rolled out publicly as a new Census dcsign t. b. .,;.;'i,,' tl'c Il.,,.cnni.rl Census. Cognitive interviews suggested that neith(.r of the tlcsiglrs wr.rc

;:"ln:::fl :";Hf ::',:H;:il::"T:1,iil::J$::n:*

",,tir"

j.llll-1.:::O::""

that are representJveof the

rrru"], poputation,

llr,rl.w,as rt'r't,alccl tlrrring cognitive testing with volunteers. l l rt' r st '. r t , xlTc. i't 'nhr | . r nt l c. gnit ivt , int er view r esur t s is a .rr,r. r ', r r r . l l ),rl l i (ul .rrl v l) r ) r l'( . r lr l lr , r sis lor . t lr t , , . r , , r lr r , r li91 9[ sr r l, r . y qt r r : st ionsand tl ttt' sl trttttt, r r r r .llillr s. r r , r r r, r lr , l lir . r llir r r , ( l, t ) ( I l)r . ( , 1) t lr l( .llt ( l f ( . ( ,, . , 'r *",r r r r vlr i, . h

*,., s.,ing.r stowrry re.s pon der; *:::t:i*#]:i*:'.H:r"#rj""JH:

the cognitive intervrew report :ll:,l:)ll"ljTve ;1,:.:: ,((1. !r , v.rIraDre. Ftowever, the exDerimental resultsfrom a random sampleof U.S. householdsprovided an uirquestioned quantitative baslsio. reyecting tlrr, tlesigns, and this basiswas

,..,r." designs. q,i,; ;,,rft"ry n u, ll]:ie,:I :::::f $l-"i"nr **had only it seemed been rejected

r,""n l]:]t e\perimentally i:.::li round to improv_e response ratesby as much as 10percentage Pofnts (i.e- "Your Response Is Required by taw;,) was graphica y hidden rn unusual print formats (Dillman, Jenkins,Martin, & ;"M;i;, 1996).An cxperimentaltest that was run at the same time a" thu cog.,itilremterviews wcre conductedshowed that indeed the two new d"rlgr,"io*"."a response ,:a:: by and 5,/o,respectively,both of *t i"f, _"."'"or.,.iJ"-r"a O.u-uti. tlcclines ?1 (Leslie, 1996).Inasmuch as a considerable ."__l _"", had been

*:lt, in partbecarlse themessase tharnoa'fr.*,uu"ry

.r.rll

l r |l ( !N l (-)t

t,,

( )r ., tr | \ ( _) l t,,

L,!

\|lr l

l't r lr ', llt . \ ll r ', lt t 'n t lit . .

.t.ttl

l ',| |,rl l r ' l( ' \ l) ( ' t inl( ' t ll,tl,l tt(i (1 )f| rrl r\{ ' i rl l ( !-\i (' w s l tr (l r( , \\r' l t i l l l l l l )l t' l r\l A l l () l )(ri l l l l ()w .l r(l t' l l o l l s l r' rrtl r' ,1 t hc r c s t t llsol t' a r.l o r l tl rt' s t' rtrtl tP ,ttl i o tt l l rrrrrl i t r (l tr(' s l i o rrs l ti rrI t(' stod,l he J n d p roc(' (l (l r(s tl rc c or r c lus iot t s.tb o ttt ',rrrri l, r r of in, r1,r.rirtlt'ntal .rnrl togttilivt'intcrvi('\,\'ii provided quite dill('r('nt l.'irr.ls what about to be dr.rwn nts i]llowed conclusions li rrr,rtion. Thc t'x1lr'r'irrrt 'r rr,'u actually haplrt'rt il a prtrgdure were tobe adopted but provided only rrrrrrrrrral insight ilrt(i rllry tlrose differencesoccurred. In contrast,the cognitr\r' i|tcrviews provtrl lo lrt'a rich source of hypotheses about the reasons Ncither method of testing can substitute for the tlrru,r'differences occtrr.r'trl. , rlI rr'r'otte. l rPcrimgalt on qu('slir)rr wording and response rate concems are not ,r'rrrt,thingthat shoultl trc trrnsitlt'redonly when conducting national sulveys li,r l)()licy purposes. Wt' h,[ t' fr('quently interacted with survey designers in rrlrvt'rsities and prir'.rtt'srllrrr firms who have produced a questionnaire ,rr rri proposed implcnrt'rtt.rlion prrredure and have wanted to know whether kt questionsworded in a new way and what will give bctt('f .)ns$,('rs lrr,rrple ('\P(\'l with and without incentives. Becauseof the c.rn r{,sponse rate they ()l c()nt('nt,lcngth, and implementation procedures, it rrrriquecombination is usually impossiblc to provicle anything other than a vague estimate of efli'cts. An experinrt'nl irrvolving it few hundred households can often provide insightful infonnalion that will prcvent much larger amounts of money from being spent in prrrsuif of an unachievable obiective. Thus, experimentation can play an esscntial and influential role in evaluating proposed questionnt.rtion procedures. naires and implL'nre of thePopulation n SftulllPilot Study zuitha Subsample Cuideline6.38:CLurdtrtt the among lttft'rcttnnections in Order to F-unluafL' Questionnnire,and Questions, tlleImplamulL Iiott ProcL'dures A pilot study rcfars to a mini-study in \thich the ProPosed questionnaire and all implementation procedures are tested on the survey population in an attempt to identify problems with the questionnaire and related imPlementationprocedures. The goal is to determine whether the proPosed questionnaire and procedures are adequate for the larger study. Although it is possible that experiments can be embedded in a pilot study, we disiinguish between these methods of testing here because a pilot study provides critical information regardless of whether experiments are embedded in the design. It constitutes a final test of the exact procedures to be used in a study. Pilot studies give a good sense of how the study procedures will work in practice.Thcy are particularly useful for making quantititi!t't'sli|n,rtos of re sp ons er at es and t h u s m a v h e .l pi n s e tti n g s a m p l o si zcs l tj r l l rr' l rrl l strrcl v. l , 1" . l l r,tt l l t.tv l rt' Th t'y rr t . t yals o ic ic n ti fy i tt' n t tto trrt' s p o t' ts t' p 1 1 r bl t' tl ts,rrr,l

t.rkcnl o | r. r lr rr , llr r . r r rll . |. I r I i| r 'r r t r r 11iI r , r l slr r r lr r . t s. r t r l, r . r , r ; 'iI r r lr .l, t r lit r I ir r , r r . \ , (l .tt,rs(ts t l) . ll r r r , r l lr t . . r n. r llzr r l lo r I r , |r , I I r r ir rllt c ( . ( '\ l( . nl lo wlt ir lt ir r t iilir lr r . r I r(' sP ()rrs ( '( . ll( 'g( ) f i( . s,( t) rr ( , lr ( , nol t r st t l. I lr t 'v can a lso lr t ' t r sr 't it ( ) g( 't , t t r indi cl ti orr o I whct lr t 'r ir r t lividt r alt lr r t 'slions lnr l sc. r lcs. t ppc. r r k) be wor king as intcnrl('d (i.e.,nte.rsuringc()nccptsin the manner expectcd).Pilot studics are often cleemcd essential when a new survey questionnaire or new implemcntation procedures are to be used for a survey. They are also often necessary when implementation involves many different individuals and divisions within an organization. Pilot studies can provide important feedback on coordination among those individuals and groups and let them "practice" for the actual survey. An example is the "Dress Rehearsal" for the Decennial Census, which is usually conducted 2 years prior to the actual Census in small regions of the United Statesin order to give critical feedback on how well each component of the survey process and its connections to other components have been designed. Pilot studies are also especially valuable for obtaining cost estimates for the actual suruey. However, except for large costly surveys, most surveys now appear to be implemented without the conduct of pilot studies. We live rn an age in which data are increasingly evaluated for timeliness and pressures have increased for doing surveys quickly, withoui taking time to conduct formal pilot studies. Not doing a pilot study can be disastrous for web surveys in particular. As we discuss in the next chapter, a large number of things have to contc togcther in the final preparations for fielding web surveys. The linking of e-mail adrninistration, accesscode assignments,and programming for individual questions provides an enormous opportunity if not likelihood-that something will be done wrong. We have experienced situations in which the programming was not finished, so answers to a particular category were recorded incorrectly in the database. We have also experienced situations in which users of a particular e-mail provider could not get the hyperlink to open the survey, and a poor choice of wording for access codes resulted in respondents entering student ID numbers instead of the access code provided to them. Unlike with mail surveys, respondents may not be able to lust "skip over" a bad question. If respondents cannot get an accesscode to work, providc an answer to a particular question, or advance to the next question, lhe rr thcy are likely to quit, and it is unlikely they can be convinced to go back ,rnrl tlo tl'rr survey after the problem is fixed. Yet the speed with which web sur'\'('vs irc often constructed and implemented means that there is a high likr.lilrtrrtl tlt.lI s()nr('(.rrors will be made. llrcsr.r-orrttrns lr.rrl krl rrs l{) institute the practice of selectinga random r,, | l l )l (' ()l ll) ( '\ Uf \ 'r '\ 1, o1r I r Lr I i''r . rI r r r linvit ing t h( 'm k) com plet c t he sur vey a Ir' rrItrrtrrsot , |. t t , , , r I I , , r , I oI lllt r . r ', I lt r '1, wlt t . r r r r 6pf ( r lr lclt r sar t 't 'r . r cor t r t t cr ed,

, r lO

lrtir

'tt(,)t

tt(

'|

tr r \ ( - ) r t

tt( r tr r - \r r r l

\ ttttt ll..ttr

.t \l

llrr' | (

t I r . I r | | | t | I I t ' : ' I ) ( ) t r ( | r ' | l| (r'l|)l)lr'\

,.tt(

( o r r l' l{ lr ' tl. ltt o lltlt ll( ' L ltl( ' tll

t,l st's, rtl tttl

.l stIf\'('v$rJs i tr l att'r

l', rrlr, r t l . t t l \ , I rr- lio n s , r ' r l c n s i r

( r ' .1 ]., ir r r o lvir r li

u s('ol

t'.trl y atl s$'t'rs wilh

l ) u i l cl i n '6 .1 :

r l si q tr sl i o n s

r r b o r tl tt,tttl s

i tt l l t

orlrr

l h t r ,r ,r tr l ,,

c lr t,tttr lr in ll t o tr slr tr clio tr s,

r lttt'sti ttns

l ()ng('r dow l l

(l u (' s ti ()l rs ), pi l ot tv t' c o n c l u c lcdtr nrort' tracl i ti otral l ,,,rrltir r c s , , lr ( l uni(' :-l (' (l s igr r ilic . r r r t ly o h (' a (l()f tl rt' p l a ttl tc d i mp l c mcntati orl so that rcsponse 'trr.l v rl rstrrlr r r t ior r s anr l al l o th e r ft' trtu rc so f th e s u rv cy coul d be more formal l v ( \.rl u ,r l( \ 1.

ot r' rn,tl ()rrirlrlint 6.5: Avoil uriult,tt ul ryutsliorror tr flli,t.lt Creating a Common Visual Stimulus Guidelinc 6,6: Estqhlish cotsislorcrl iu tha rtisrrrtl l,n,sr,r lttI n,tl of questions (ttcrosspnut,s atrd scrtttrs), rttt,l ,,,t. alignmcnt and acrtictrl spaciuvto Iu,ll tt,sl,t,llt.,,t., organize thc informqtion o thr,yttt Guideline 6-7: Use color and contrast to ht,ht rcsponltttt, rt, ognize tlrc components of thc qrn,stiotrs rrtr tlt natsigational path through tht Llurctio ,|t|in. Guideline 6.8: Visually group related informttiou in rr,!iotrs through the use of contrast and cncrosurt, Guideline 6,9: Consistently identfy the bcginninu ol ctch ryt:, tion andlor section Guideline 6.10: Use aisual elements and propcrtics t.ot,sistt,,ttlt/ across questions nnd pages/scrctns lo t,istrtllV tur phasize or deemphasizecertain tyycs rtl inlitrurrt tion Guiileline 6.11: Aooid aisual clutter Guideline 6.12: Minimize the use of mstrixes qntl tlrcir t ortrylt,.r it tJ

C O N C L U SION t )rr(' oI the significant impacts of the electronicage is that the line between constr Lr( ling .r questionnaireand implementing a survey is increasinglyblurred, .rs is thc line betwe'enwriting a question and constructing a questionnaire Altlrough surve,ydcsign has never been a linear process,the use of technolrlir irr both mail ancl lnternet surveys now pushes surveyors even further rlorvn the path of constantly thinking about all aspects of the survey cre,rlion and implementation process. Our main focus in this chapter has been rrrr wlrat must happen when the decision is made to move from a listing of (lu('stions one wants to ask to the formatted questionnaire that respondents rvill cxperience, whether by mail or through the Web. ln either case the last step of construction is pretesting, which has come a long way since its oriSins irr the 1900s. It is no longer a catch-all, somewhat undefined technique for ide'ntifying potential survey problems. Much more precise methods are now available and need to be used, as discussed here. The next chaPter takes this tested questionnaire toward completion of the survey design process. In it, we bring together sample and questionnaire decisions with the manner in which people are going to be approached with the request to complete the questionnaire.

The Mail Questionnaire

Guideline 6.13: Determi,ne uhether keypunching or optical imaging and scanning zaill be used, and assess the limitqtions that mny impose on designing and. pro cesstng q uestionn.tire s Cuideline 6.14: Construct p aper questionnaires in bo oklet foffi ats, and choosephysical dimensions based upon print_ ing and mailing considerations Guidcline 6.15: Decide question layout and hozu questions u.till be arranged on each page Gnitltlirc 6.16: Use symbols, conttqst, size,proximity, and pagina_ tion efrectiaely zuhen designing branching instruc_ lious to lrclp rcspondents corectlq execute them
(cLtntittucd)

Lrsr or GurnnrNns
Ordering the Questions
Guideline 6.7: Group rclatetl questions that coaer similar topics, and begin zuith questions likely to be salicnt tLt neatlq all respontlents Cuidcline 6.2: Choosc thc first qucstion ctrc.fulhl Cuilclirc 6..i: l'Itcc scttsitittt or yotctrtitIIq tbitIirtturbIt qm'sIi o trs ta tr Ih t tu rl o l l l t 4 r rsl i ttrttrrrt

t l,

lr|(

(-)t 'Nt

t,

i rrr'

tr '

\ (-)l

l,ll0r .lf.l

\llr l

{ r,,tr /t/,/rr,/

,tl \

() i rlitr

l'.t7: ( r ' n! ( tlr,t I s

it t lr t t ' s lit t 1 r n uill

iulir t t t t t i t 't lttpt

{ r,{,rir/,rl{r's lhtl (lui& litr b.lE: At'oil Ihtl

tttl Lt(k ltlml lo r(sql"taltltltrl 11ti111

side ott tt pttgc so lltcitrs rl,rcstir,ts sidc by tulo quesans|ucr to asktd orc rcslttn]|'lents

l inlls tlt ottca'

'I'he Web Questionnaire ( )nilt'liut' 6,'19: L)acitlcQhcther an electronic altematiae to o Tueb sura(v is more appropriate (e'g', fillable PDF ot ) cmltcddcd c-mail suraeY Gnidr'lirc 6.20: Chooschoza the suroey will be programmed and hosttd commensurate Tuith the needs, skills, and sPonsorshiPof each suraeY Cuiddinc 6.2't: Ea uate the technological cqp&bilities of the sur' aeYPoPulation Guidcline 6.22: Take steps to ensurc that questions display similatly actoss dilfcrent platlorms, brousers' and user settrngs Ouideline 6'23: Decide how many questions zttill be presented on each web page and hoTaquestions Toill be arranged Guidcline 6.24: Creqte intetesting dnd informatiue Tuelcome Lnd closing scteens that xoill haae Toide appeal to resoondents Guidcline 6.25: Deaelop a screenformat that emphasizes the resPondcnt tather than the sPonsor Grriclclinc 6,26: LLsca consistent ptge layout across screens and oisually emphasize question infotmation that respondcnts zuill need to complete the suraey zuhile clcttnphasizing infonnation th&t is not essefltial to tlrc task to questiofls unlessabso' Guidt'linc 6,27: l)o not rcquire tesPonses lutelv ncccssary fot the surl'eY and item-specificetlor mesCuilt'liur' 6.28: l)csign suruet1-specific sogast<thclp rcspondentstroubleshoot any issues th.'v mav (ttcornter (lridrlirc 6.29: l't,duttt' corcfulhl thc usc of intcractiac featurts' htlnrci*t inryttottancntsitt tttt'ttcurcmtuttpith tlu' i tPn.I o rtsyondcnl bunltu ttrl tht irultlitrttiotrs lirt ttti'rul-ttnub strn't't1s | | ,'rl ittrt.tl)

()uit lt liur 6. 1( l: I I sr t nliot isr r t I t t 1nlt iI it it s oI I lu, l|Vt hslr r t in, glq, ttttl r ualttrrll tlu' tlilli'n'nlitl tlJi,rt lln,y t,,ttV lttu,t, tttr rtsyottrlttls Cuitfulitc 6..31:Allotrt rcslontlcn!s ltt stolt llk,srn'rq ttrl litislr cornplcting it tt onothtr tinc Guidt'lirc 6.32: Whcncar:rpossiblc, collect paradLrtt lhtl yrtn,ifu, fecdback on hotu the respoudentintcrdcts zoith ll,., questionnqirc Guideline 6.33: Test the surztey using a aariety of platforms, con_ nection speed.s, brotosers, &nd user_controlled set_ tix&s, afld test the database to ensure that items are collected and codeil coftectly Guideline 6.34: Takescreenshots of eachpage oj fhelinal question_ naire for testing and ilocumentation Pretesting Questionnaires

Guideline 6.35: Obtain feedback on the draft questionnaire from a number of people, each of uthom has specialized knowledge of some aspect of questionnairc quality Guideline 6.36: Coruluct cognitiae interoiezus of the complcta qut,stionnaire in order to idcntify tvordittg, rlrrcstitt.u order, ztisual design, and naaigation problcns Guicleline 6.37: When the stqkes qrc high, consitlu doing ctlttn_ ment al ezt aluqtions of quest ionnain: co ntl,o n.,n ts Guideline 6.38: Conduct a small pilot stutly uith a subsamplc of the population in order to eaaluate interconnec_ tions flmong questions, the questionnaire, and the irnp lefi ent atio n pro cedures