Controlling Order-Effect Bias

American Association for Public Opinion Research
Controlling Order-Effect Bias

Author(s): William D. Perreault, Jr.
Source: The Public Opinion Quarterly, Vol. 39, No. 4 (Winter, 1975-1976), pp. 544-551
Published by: Oxford University Press on behalf of the American Association for Public Opinion Research
Stable URL: http://www.jstor.org/stable/2748507 .
Accessed: 12/04/2013 17:15
Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .
http://www.jstor.org/page/info/about/policies/terms.jsp
.
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of
content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms
of scholarship. For more information about JSTOR, please contact support@jstor.org.
.
American Association for Public Opinion Research and Oxford University Press are collaborating with JSTOR
to digitize, preserve and extend access to The Public Opinion Quarterly.
http://www.jstor.org
This content downloaded from 113.210.131.115 on Fri, 12 Apr 2013 17:15:20 PM
All use subject to JSTOR Terms and Conditions
CONTROLLING ORDER-EFFECT BIAS
BY WILLIAM D. PERREAULT, JR.*
Researchers have long been aware of the importance of questionnaire
and interview schedule design on the quality and quantity of response.
Detailed thought is given to the type of questions that should be asked
and to how they should be worded and sequenced. Selecting the
appropriate sequence for the questions is a complex issue in
questionnaire design. The researcher must be careful to place questions
so that he evokes and maintains the respondent's interest, stimulates his
attention, and in some cases even overcomes his resistance to answering
questions. For example, questions which are of a personal nature ("what
is your income?") or those which might prove sensitive to the respondent
("do you approve of interracial marriage?") are frequently placed at the
end of the questionnaire. It is felt that by the time the respondent has
reached the end of the questionnaire he will be more likely to have
adopted a positive response set, and even if he is offended by a question it
will not have influenced his response to the other questions. Thus, in
most situations, the sequence of questions is a positive factor that opin-
ion researchers use as a tool to improve the quality of the research in-
strument. Unfortunately, the position of a question may also exert a
negative bias on response.
THE PROBLEM
The relative position of an item in an inventory of questions or stimuli
may uniquely influence the way in which a respondent reacts to the item.
This phenomenon, referred to as "order effect," may be attributable to
any of a number of factors. Landon suggests that early items in an inven-
tory may tend to act as an "anchor" upon which subsequent responses
are made.1 From a similar perspective, Kornhauser and Sheatsley note
that earlier items of an inventory may create a response set or expectation
that influences response to later items.2 Some bias may result from
dissonance; in the vein of Anderson, Taylor, and Holloway, the re-
*
The author is Assistant Professor of Marketing in the College of Business
Administration, University of Georgia, Athens.
1 E. Laird Landon, Jr., "Order Bias, The Ideal Rating, and the Semantic Differential,"
paper presented at the Fall Conference of the American Marketing Association, Boston,
Massachusetts, August 1970.
2 Arthur Kornhauser and Paul B. Sheatsley, "Questionnaire Construction and Interview
Procedure," in C. Sellitz, M. Jahoda, M. Deutsch, and S. Cook, Research Methods in Social
Relations, New York, Holt, Rinehart and Winston, 1959, pp. 546-574.
ORDER-EFFECT BIAS 545
spondent may begin to alter his true response pattern-as he progresses
through an inventory-in an attempt to be consistent with earlier judg-
ments.3 Sometimes, it is simply the (relative) position of an item which
tends to elicit a particular response, regardless of item content. For
example, Silk discusses an inventory of items which call for the
respondent to check one of two possible alternatives; he noted a
respondent bias to checking the first alternative for each item, although
this led to an inconsistent response pattern when the whole inventory was
considered.4
Although such order effect is commonly recognized as a source of bias
in survey research, it is frequently ignored. This lack of rigor is explained
by two considerations. First, models for statistical estimations of order
effect are complicated and, more important, appropriate only in restric-
tive cases.5 Second, if order bias is not treated statistically, it should be
controlled by randomizing its effect across respondents. This type of
control requires that the researcher produce different questionnaires,
comprised of random orderings of relevant items (while maintaining
proper overall sequence) for each respondent. The initial expense and
clerical difficulties of producing many different questionnaires are ob-
vious and are further compounded by the problems of coding the data
from the completed questionnaires into a consistent (machine-
interpretable) form.
At first thought, it appears that the ordering and printing of items for a
questionnaire could be easily accomplished with the computer. Hughes
and Guerrero have, in fact, used on-line terminals and interactive designs
to control order bias: each subject was presented stimuli in a random
sequence.6 Although this is a creative solution to the order-bias problem,
it is also a highly restrictive one. It is certainly not feasible in most public
opinion research situations. Their application, however, demonstrates
that the problem is not in developing algorithms to (randomly) order
survey items; that can be done with ease in any computer language. The
problem is in printing questionnaires which have an acceptable
appearance-an appearance that will stimulate response. When the order
of the questions is random, programing a set of decision rules to control
the format and appearance of the finished questionnaires is a time-
3 Lee K. Anderson, James R. Taylor, and Robert J. Holloway, "The Consumer and His
Alternatives: An Experimental Approach," Journal of Marketing Research, Vol. 3, Febru-
ary 1966, pp. 62-67.
4 Alvin J. Silk, "Response Set and the Measurement of Self-designated Opinion Lead-
ership," Public Opinion Quarterly, Vol. 35, 1971, pp. 383-397.
5 See, for example, R. Darrell Bock and Lyle V. Jones, "The Measurement and Predic-
tion of Judgment and Choice," San Francisco, Holden Day, 1968, pp. 75-84; Robert B.
Kane, "Minimizing Order Effects in the Semantic Differential," Educational and Psycholog-
ical Measurement, Vol., 31, 1971, pp. 137-144.
6 G. D. Hughes and J. L. Guerrero, "Simultaneous Concept Testing with Computer
Controlled Experiments," Journal of Marketing, Vol. 35, 1971, pp. 28-33.
546 WILLIAM D. PERREAULT, JR.
consuming task, and one which in most cases demands considerable
programing skill. As a simple example, it might be desirable to print two
or more text columns on one page (particularly if standard-size unlined
computer paper is used). With the order of the items varying, this would
be a considerable problem. Similarly, "keeps" (several lines that should
always appear intact in the same text column), upper- and lower-case
letters, page numbers, and many other details add difficulty. In short,
these format problems have discouraged researchers from employing the
computer to control order bias by randomizing its effects.
A SOLUTION
There is, however, a simple way to overcome many of the problems of
different questionnaire formats while stilll maintaining the advantage of
using the computer to order the items to be printed. The researcher (even
one with only novice programing skills) can easily produce
questionnaires by combining the power and flexibility of "canned" text
processing programs with a simple computer program to order the input
to the text processing program. Before proceeding with a specific expla-
nation of this type of application, a brief mention of the general
capabilities of text processors may be useful to unfamiliar readers.
Computer Text Processing
A number of text processing programs are readily available.7 Typically,
input to the programs is a free format "text-string," which consists of
both normal text words and editing-specific control characters. The
computer reads the text-string, and prints the words according to the
control characters. For example, different control characters are used to
indicate what material should be kept together in one text column, when
new lines, paragraphs, or pages should start, when text is to be
capitalized, etc. These control characters, then, determine specific aspects
of the format of the output.
The user also controls more general aspects of the format by specifying
several control cards. These control cards determine such editing param-
eters as print column width, number of print columns per page, spacing
7 Although by no means an exhaustive listing, several of the more popular programs are
discussed in International Business Machine Corporation, IBM TEXT 360: Reference
Manual and Operating Guide, White Plains, N.Y., IBM Corporation, 1969; H. S. Weiner
and P. D. Reilly, TYPIST: A Text Editor Format Program, Ithaca, N.Y., Office of Com-
puter Services, Cornell University, 1973; Gerald M. Berns, The FORMAT Manual, Whea-
ton, Maryland, IBM Washington Scientific Center, 1971; R. Rich, J. Olmer, G. Trotter, D.
Brocklebank, and G. Prophet, Info 360: The JHU/A PL Information Package, Silver Spring,
Maryland, The Applied Physics Laboratory, Johns Hopkins University, 1972.
between lines, whether or not the print column is to be "right justified,"
etc. By simply changing one of these controls, the user may easily alter
the format of the output. Most of the canned text processing programs
are simple to use and require little or no prior computer experience.
These programs are most frequently used for producing documents
where it is useful to be able to update and modify the content or general
format of the text without retyping the whole document.
The advantage of such programs for producing questionnaires rests in
the fact that the specific format-control characters are part of the text-
string input in the program. The total text-string (in this case, the
questionnaire) is broken down into segments (i.e., questions, scales direc-
tions, etc.). Each segment contains both the appropriate text and appro-
priate format controls. These text-string segments are then read into a
computer program and reordered. As with any computer problem, the
logic of the reordering is completely specified by the programer. There-
fore, the researcher may specify broad sequencing specifications that
must be met while randomizing the sequence of other material where
order effect may be a problem. The result of the reordering is a new,
aggregated questionnaire text-string. The controls in that text-string con-
tinue to determine the specific aspects of the format of the material. An
example may further clarify the application.
AN APPLICATION
A recent (small sample) study which focused on industrial purchasing
managers' evaluation of alternative suppliers helps to illustrate the tech-
nique. Members of the sample were mailed questionnaires consisting of
two types of questions. The first set was a series of paired comparisons in
which the purchasing manager indicated the preferred supplier of each
pair. In the second set, the respondent indicated the importance of
different supplier characteristics by completing a constant sum scale. As
in most survey research, there were sequencing requirements for this
questionnaire. It was inappropriate for respondents to answer the con-
stant sum scale questions until after the paired comparisons were com-
pleted. Within this overall sequence, however, there was an order-bias
problem. Pretests (with the series of paired comparisons in a fixed order)
indicated that order effect was a potential source of bias.
To produce questionnaires that controlled this bias with a random
ordering of the paired comparisons, while maintaining the broader se-
quence requirement, a FORTRAN preprocessing algorithm was used in
combination with the FORMAT 4 text processing
program.'
8
Berns, op. cit. This particular algorithm was chosen because it is one of the more
economical text processing programs and is simple to use.
Procedure
The text of the questionnaire was broken into those smaller groupings
of phases which would ultimately be printed together in the finished
questionnaires. For example, each set of directions and each paired
comparison was treated as a distinct segment. Each text segment included
the control characters to determine the format of that particular portion
of the finished questionnaires. These segments were given indentification
numbers and incorporated in the FORTRAN algorithm.
The logic of this algorithm reordered the segments into a total text-
string. Each questionnaire started with a unique code number and a set of
directions. Next, the algorithm generated random numbers and ranked
them. This ranking determined the order in which the paired comparison
segments were to be combined. Next, another set of directions was added
to the text-string, and finally the set of constant sum scales. A simple
index number was used to number the questions sequentially as they
appeared. This procedure was repeated in an iterative fashion until a
questionnaire text-string for each person in the sample had been
specified. Each questionnaire, therefore, met a set of general sequencing
requirements: each began with a code number and a set of directions,
followed by a series of paired comparisons, followed by a second set of
directions, followed by a series of constant sum scales, and concluding
with a "thank-you" message and information concerning return of the
forms. Within this overall sequence, however, the order of the paired
comparisons was randomized.
Thus, the FORTRAN computer program took care of the ordering of the
questionnaire material. Next, the FORMAT 4 program controlled the for-
mat of the questionnaires and printed them, taking care of such
appearance details as the maintenance of consistent margins, page num-
bers, and spacing. The questionnaires, which were printed on 8 1/2-inch
by 11-inch paper with upper- and lower-case letters, appeared to have
been individually typed.
Coding of Data
One problem of such individual questionnaires is the coding of the
data to a usable form. This problem was also overcome. The application
discussed above also helps to illustrate this point.
As was
noted, each
time the program produced a new questionnaire a unique code number
was assigned and printed. For each questionnaire, the program also
produced a corresponding READ statement in computer card form. (In the
FORTRAN computer language, the READ statement is used to control the
input of data to the program.) This READ card was appropriately
punched with a list of variables in the same order they
had
appeared
on
the questionnaire. When the completed questionnaires were returned,
they were coded; first, the questionnaire identification number was
coded, and then on subsequent card images the rest of the data were
coded in sequential columns. The data were punched, and then prior to
the statistical analysis a program was written that performed the
"unshuffling" of the data. The unshuffling procedure is simple. First, the
questionnaire identification code is read in and used by the computer to
identify which questionnaire is being read; then that information is used
in the program logic to direct control to the appropriate READ card
(generated in the earlier step). The data are read into the machine in this
fashion and stored in the computer memory bank, and then may be
written out again, in the order specified by the programer, to tape or
cards for further processing. This approach will prove effective even with
elaborate data format schemes.
Another approach is viable with respect to the coding of the data. It is
conceptually simpler than the preceeding approach, but more difficult
from the standpoint of clerical effort. Even with standard questionnaires,
researchers frequently provide data coders with coding information. This
most frequently takes the form of small numbers placed in parentheses
unobtrusively in the margin. The numbers represent the columns of the
computer coding sheet (computer card) in which the response is to be
recorded. In the traditional questionnaire, these numbers would typically
be sequential. This approach may also be used with questionnaires with
randomized order. The text-string segment for any given question would
simply include the controls to print the appropriate column numbers in
the margin beside the question. With this approach, the human coder
would bear the responsibility for coding the data in the specified
columns. Unlike the case of the traditional questionnaire, the coding
columns would typically not be sequential.
Because the first approach is really a rather simple computer
procedure, it will normally prove more efficient than the second, coder-
oriented approach. Furthermore, the different ordering of the data on
each questionnaire would undoubtedly make the rather dreary task of
data coding even more burdensome and error-prone for the human
coder.
Response Rate
The use of this procedure may have an additional benefit in terms of
response rate. In the purchasing manager study discussed above, it was
pointed out to recipients of the questionnaire that no one else would
receive a questionnaire identical to theirs; of course, the individually typed
appearance of the questionnaire reinforced this statement. About four-
fifths (21 of 26) of the sample completed and returned the long (nine-
page) questionnaire. It cannot be said with total confidence that the
response rate was influenced by the personalized questionnaires; no
control group received questionnaires printed in the traditional fashion.
It does seem, however, that if the respondent feels that his response is
uniquely important, he may be more inclined to respond. The uniqueness
of the individual questionnaire may be emphasized by printing the re-
spondent's name at the beginning of his particular questionnaire. This is
an especially straightforward procedure when respondent's names and
addresses are already compiled for computer mailing labels.
Cost
The costs of the procedure discussed here should be taken into account
in determining the appropriateness of the procedure to a given project.
The costs vary primarily according to (1) how complicated the reordering
stage of the process is (i.e., the price of computer time), (2) which text
processing program is used, and (3) how much actual text there is on a
page of the questionnaire. For the project discussed in this paper, the
total computer cost, including supplies, was about 8 cents a page.
Depending on the questionnaire, cost could vary from about 2 cents per
page to 12 cents per page (or higher, if prices rise).
For surveys involving a long questionnaire, a large sample, or both,
this cost might be prohibitive. In such a case, the randomization
procedure might prove useful during pretest stages to test for order effect.
The pretest sample would be randomly split. Half would receive
questions in a standard order; the remainder would receive
questionnaires processed as discussed here. Both response rate and actual
response distributions could be tested statistically to determine if there
were differences between the two halves. With this additional
information, the researcher could evaluate whether order effect was
present and whether it needed to be controlled.
Alternatively, if order effect was significant, a series of different ques-
tionnaires could be generated, each with its unique ordering. These
questionnaires would then be printed in the standard fashion. Each
individual respondent would not, therefore, receive a unique
questionnaire, but the procedure would be a relatively efficient approach
to reducing order-effect bias.
OTHER APPLICATIONS
Although the focus of this article has been on the applicability of
combining text editing with a preprocessing algorithm to control other
bias in survey instruments, the combination has a number of other useful
applications. Frequently, it is advantageous in survey research to person-
alize advance notice or cover letters.9 Names and addresses may be read
from a mailing list as variables in a preprocessing algorithm and inserted
in the text-string of a letter to be processed. The text is then formatted by
the program to print the personalized letters. This mechanism also pro-
vides a direct procedure to sample randomly from an existing list and
simultaneously generate appropriate letters.
The text processing program may also be used to print mailing labels,
or print names and addresses directly on special envelopes developed for
use on line printers. Envelopes addressed by the text processing program
(with both capital and lower-case letters) are not readily identifiable as
computer output.
CONCLUS ION
Question sequencing is an important aspect of the art and science of
questionnaire construction; question sequence may have both positive
and negative effects. This article has discussed a procedure by which
public opinion researchers may preserve the positive aspects of
sequencing while controlling for problems of other bias. The procedure
has additional benefits in that it may also be used as a speedy and
relatively economical way to personalize survey research materials such
as cover letters, advance notice letters, and even the questionnaires
themselves.
'An excellent review of the impact that personalized survey materials may have on
response has been compiled by Paul L. Erdos, Professional Mail Surveys, New York,
McGraw Hill, 1970.

Controlling Order-Effect Bias

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Controlling Order-Effect Bias

Caricato da

Copyright:

Formati disponibili

American Association for Public Opinion Research

Controlling Order-Effect Bias

Potrebbero piacerti anche