controlling order-effect bias

8/11/2019 Controlling order-effect bias

1/9

American Association for Public Opinion Research

Controlling Order-Effect BiasAuthor(s): William D. Perreault, Jr.Source: The Public Opinion Quarterly, Vol. 39, No. 4 (Winter, 1975-1976), pp. 544-551Published by: Oxford University Presson behalf of the American Association for Public Opinion Research

Stable URL: http://www.jstor.org/stable/2748507.Accessed: 12/04/2013 17:15

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at.http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact [email protected].

.

American Association for Public Opinion Researchand Oxford University Pressare collaborating with JSTOR

to digitize, preserve and extend access to The Public Opinion Quarterly.

http://www.jstor.org

This content downloaded from 113.210.131.115 on Fri, 12 Apr 2013 17:15:20 PMAll use subject to JSTOR Terms and Conditions
http://www.jstor.org/action/showPublisher?publisherCode=ouphttp://www.jstor.org/action/showPublisher?publisherCode=aaporhttp://www.jstor.org/stable/2748507?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/2748507?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=aaporhttp://www.jstor.org/action/showPublisher?publisherCode=oup


2/9

CONTROLLING

ORDER-EFFECT

BIAS

BY WILLIAM D. PERREAULT, JR.*

Researchers ave long been aware of the mportance

f

questionnaire

and

interview cheduledesign

on

the quality nd quantity f response.

Detailed thought

s

given o

the

type

of

questions hat should be asked

and

to how

they should

be worded

and sequenced. Selecting the

appropriate sequence

for the

questions

is

a complex issue in

questionnaire esign.

The

researchermustbe careful o place questions

so thatheevokes and maintains herespondent's nterest,timulates is

attention,

nd

in some

cases

even

overcomes

his

resistance

o

answering

questions.

For

example, uestions

which

re of

a

personalnature what

is

your ncome? )

or

thosewhichmight rove sensitive o therespondent

( do you approve

of

interracial

marriage? )

re

frequentlylaced

at the

end

of

the

questionnaire.

t is felt

hat

by

the

time

the

respondent

as

reached

the end of the

questionnaire

he will

be

more

likely

to have

adopted

a

positive esponse et,

nd even

f

he s offended

y

a

question t

will

not

have

influenced

is

response

to

the

other

questions. Thus,

in

most ituations, he sequence of questions s a positivefactor hatopin-

ion

researchers

se as a tool to

improve

he

quality

of the

research

n-

strument.

nfortunately,

he

position

of a

question may

also exert

a

negative

bias

on response.

THE

PROBLEM

The relative osition f an item

n

an inventoryfquestions rstimuli

mayuniquely nfluenceheway nwhich respondenteacts o the tem.

This phenomenon, eferred o as ordereffect, maybe attributable o

any

of a

number

f

factors.

andon

suggests

hat

arly

tems n

an inven-

torymay

tend

to

act

as an

anchor

upon

which

ubsequent esponses

are made.1

From

a

similar

perspective,

ornhauser

nd

Sheatsley

note

that arlier tems

f

an

inventory ay

reate

response

et

or

expectation

that

influences

esponse

to later

items.2 ome

bias

may

result from

dissonance;

in the vein

of

Anderson,Taylor,

and

Holloway,

the re-

*

The author is Assistant

Professor of Marketing in the College of Business

Administration, niversityfGeorgia, Athens.

1 E. Laird Landon, Jr.,

Order Bias, The Ideal Rating, nd the SemanticDifferential,

paper presented t the

Fall

Conference f the AmericanMarketingAssociation,Boston,

Massachusetts,August 1970.

2 ArthurKornhauser nd

Paul B. Sheatsley, QuestionnaireConstruction nd Interview

Procedure, n C. Sellitz,M. Jahoda,M. Deutsch,

nd S. Cook, ResearchMethods n Social

Relations,New York, Holt, Rinehart nd Winston,1959,

pp. 546-574.

http://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/page/info/about/policies/terms.jsp


3/9

ORDER-EFFECT

BIAS 545

spondentmay beginto alter

his true

responsepattern-as

he

progresses

through

n

inventory-in n attempt o be consistentwith arlier udg-

ments.3 ometimes, t is simply he relative)positionof an itemwhich

tends to elicit a particularresponse,regardless

f item

content.

For

example, Silk discusses an inventory f items which call

for

the

respondentto check one of two possible alternatives;he noted a

respondent ias to checking he first lternative or ach item, lthough

this ed to an inconsistentesponse attern hen hewhole nventory as

considered.4

Although uch ordereffect

s

commonly ecognized

s

a

source

of

bias

insurvey esearch,t s frequentlygnored. his ack

of

rigor

s

explained

bytwo considerations. irst,models for tatistical stimationsforder

effectre complicated nd,

more

mportant,ppropriate nly

n restric-

tive

cases.5Second,

if order

bias

is

not treated

tatistically,

t should be

controlled

by randomizing

ts

effect cross

respondents.

his

type

of

control

requires

that

the

researcher

roduce

different

uestionnaires,

comprised

of

random

orderings

f relevant tems

while maintaining

proper overall sequence) foreach respondent. he initial xpense and

clerical

difficulties

f

producingmany

different

uestionnaires

re

ob-

vious and are further ompoundedby the problems f coding the

data

from the completed questionnaires into a consistent (machine-

interpretable) orm.

At first hought,t appearsthat he ordering nd printingf tems

or

questionnaire ould be easily accomplishedwith

he

computer.Hughes

and Guerrero ave,

n

fact, sed on-line erminalsndinteractiveesigns

to

controlorderbias: each subject was presented timuli n a random

sequence.6Although his s a creative olution o theorder-bias roblem,

it

s also

a

highly estrictivene. It is certainly ot feasible n mostpublic

opinion research ituations.Their application,however,demonstrates

that theproblem s not in developing lgorithms o (randomly)order

survey tems;

hatcan

be

done with

ase

in

any computer anguage.

The

problem is

in

printing questionnaires which have an acceptable

appearance-an appearance hat

will

timulate esponse.Whentheorder

of

thequestions s random, rograming set of decisionrules o control

the format

nd

appearance

of the finished

uestionnaires

s a time-

3 Lee K. Anderson,JamesR. Taylor, nd RobertJ. Holloway, The

Consumer nd His

Alternatives:

n

Experimental pproach, Journal f Marketing esearch,

Vol. 3,

Febru-

ary 1966, pp. 62-67.

4

AlvinJ. Silk, Response Set and the Measurement f Self-designatedpinionLead-

ership, Public Opinion

Quarterly, ol. 35, 1971, pp. 383-397.

5 See,

for

example,R. Darrell Bock and Lyle V. Jones, The Measurement nd Predic-

tion

of Judgmentnd Choice, San Francisco, Holden Day, 1968, pp.

75-84;

Robert

B.

Kane, MinimizingOrderEffectsn theSemanticDifferential, ducational

nd Psycholog-

ical

Measurement, ol., 31, 1971, pp. 137-144.

6

G.

D. Hughes

and

J. L. Guerrero, Simultaneous Concept Testing

with

Computer

ControlledExperiments,

ournal f Marketing, ol. 35, 1971,pp. 28-33.



4/9

546

WILLIAM

D. PERREAULT, JR.

consuming

ask,

and one

which

in most cases demands

considerable

programingkill.

As a

simple xample,

t might e desirable

oprint wo

or more text olumnson one page (particularlyfstandard-size nlined

computer aper

s

used).

With he orderof the tems

arying,

his

would

be a considerable

roblem. imilarly,keeps (several

ines

that

hould

always appear

intact

n

the same

textcolumn), upper- nd

lower-case

letters, age

numbers,

nd

many

otherdetails add difficulty.

n

short,

theseformat roblemshave discouraged

esearchers

rom

mploying

he

computer o

controlorderbias

by

randomizing

tseffects.

A SOLUTION

There s, however,

simpleway to

overcomemany fthe problems f

different

uestionnaire

ormats

while tilllmaintaininghe advantage

of

using

he

computer

o order

he

tems o be printed.

he researchereven

one

with

only

novice programing skills) can

easily produce

questionnaires

y combining he power

and flexibilityf canned text

processing rogramswith simple omputer

rogram o

order he nput

to the textprocessing

rogram.

Beforeproceedingwith specific xpla-

nation of this type of application,a brief mentionof the general

capabilities

f text

processorsmay

be useful o unfamiliar eaders.

Computer

ext

Processing

A

number f

text

rocessing rograms

re readily vailable.7 ypically,

input to the programs

s

a

freeformat text-string,

hich consistsof

both

normal text words and

editing-specific

ontrol characters.

The

computer

reads

the

text-string,

nd

prints

he

words

according

o the

control haracters. or example,differentontrol haracters re used to

indicatewhatmaterial hould be kept

togethern one text olumn,when

new

lines, paragraphs,

or

pages

should

start,

when

text

is to

be

capitalized,

tc. These

control

haracters, hen,

etermine

pecific spects

of the format f

the output.

The user lso

controlsmoregeneral

spectsoftheformat y specifying

several

ontrol ards.These control ards determineuch

editing aram-

eters s

print

olumn

width,

number

f

print olumnsperpage, spacing

7 Althoughby no meansan exhaustive isting, everal fthe morepopular programs re

discussed

n

International

Business Machine Corporation, BM TEXT 360: Reference

Manual and OperatingGuide,

White Plains, N.Y., IBM Corporation,1969; H. S. Weiner

and P. D. Reilly,TYPIST: A Text EditorFormatProgram, thaca, N.Y., Office

f Com-

puter ervices, ornell University, 973; Gerald M. Berns,

The FORMAT Manual, Whea-

ton, Maryland,

BM Washington cientific enter, 971;

R. Rich,J. Olmer,G. Trotter, .

Brocklebank, nd G. Prophet,

nfo 60: TheJHU/APL Informationackage, Silver pring,

Maryland,The Applied Physics

Laboratory,JohnsHopkins University, 972.



5/9

ORDER-EFFECT

BIAS

547

between

ines,whether

r not

the

print

olumn

s to be

right

ustified,

etc. By simply

hanging

ne of thesecontrols,

he user may easily

lter

theformat f the output.Most of the cannedtextprocessing rograms

are simple

to use

and

require

ittleor no prior computer

xperience.

These

programs

are most

frequently

sed for producing

documents

where t s useful

o be

able

to update

and modify he

content

r general

format

f

the text

without

etyping

he whole

document.

The advantage

of suchprograms

orproducing

uestionnaires

ests

n

the

fact

that

the

specific

ormat-control

haracters

re

part

of the

text-

string nput

in the

program.

The total

text-string

in this

case,

the

questionnaire)

s brokendown nto egmentsi.e.,

questions, cales

direc-

tions, tc.). Each segmentontainsboth the appropriate ext nd appro-

priate

format

ontrols.

These text-string

egmentsre

then

read intoa

computer rogram

nd reordered.

As with

any

computer roblem,

he

logic

of the

reordering

s

completely

pecified

y

theprogramer.

here-

fore,

the researcher

may specify

broad

sequencing

specifications

hat

must

be met

while randomizing

he

sequence

of othermaterial

where

order effect

may

be a problem.

The result

of the reordering

s a

new,

aggregated uestionnaire

ext-string.

he

controls

n that

ext-string

on-

tinueto determine

he specific spects

of the

format f the material.

An

examplemayfurtherlarifyhe application.

AN

APPLICATION

A recent

small

sample) study

which

focused n

industrial

urchasing

managers'

valuation

f alternative

uppliers

helps to

illustratehe tech-

nique.

Members

of

the sample

were mailed questionnaires

onsisting

f

two

types

f questions.

The first et

was a series f paired

comparisons

n

which

the purchasing

manager ndicated

he

preferredupplier

f

each

pair.

In

the

second set,

the

respondent

ndicatedthe importance f

different

upplier

haracteristics

y

completing

constant

um

scale.

As

in most survey

research, here

were sequencing

requirements

or this

questionnaire.

t was

inappropriate

or

respondents

o answerthe con-

stant um

scale questions

until

after he paired

comparisons

were

com-

pleted.

Within his

overall sequence,

however, here

was an order-bias

problem.

retestswith

he

series

f

pairedcomparisons

n a fixed rder)

indicated hat order ffect

as

a

potential

ource of bias.

To produce questionnaires hat controlled his bias witha random

ordering f

the pairedcomparisons,

while maintaining

he broader

se-

quence

requirement,

FORTRAN

preprocessing

lgorithm

was used

in

combination

with heFORMAT

4

textprocessing

rogram.'

8

Berns,op. cit.

This particular

lgorithm

was chosen

because it is one

of the more

economical

textprocessing

rograms

nd is simpleto

use.



6/9

548

WILLIAM D.

PERREAULT, JR.

Procedure

The text f

the questionnairewas

broken nto those maller roupings

of

phases

which would ultimately e printed ogether n the

finished

questionnaires. or example, each set of directions nd each

paired

comparison

was

treated s a

distinct egment. ach text egment

ncluded

the control haracters o determine he

format f thatparticular

ortion

ofthefinished

uestionnaires. hese

segmentsweregiven ndentification

numbers nd

incorporated n the

FORTRAN

lgorithm.

The logic of thisalgorithm eordered

he segments nto a total

text-

string. ach questionnaire

tarted

with

uniquecode number nd a

setof

directions.Next,thealgorithm enerated andom numbers nd ranked

them.This

ranking eterminedheorder

n

which hepaired

comparison

segments ere o be

combined.Next,

nother

et of directions as

added

to

the

text-string,nd finally he set of constant um scales. A

simple

index

numberwas

used

to

number he

questions sequentially s they

appeared.

This

procedure

was

repeated

n an

iterative

ashion

until

a

questionnaire

text-string

or each

person

in

the

sample

had been

specified.

ach

questionnaire, herefore,

et

a set

of

general equencing

requirements:

ach

began with

a

code number nd

a

set of

directions,

followedbya seriesofpairedcomparisons, ollowedbya second set of

directions,

ollowed

by

a series

of

constant um

scales,

and

concluding

with a

thank-you message and informationoncerning eturn f the

forms.Withinthis

overall

sequence,however,

he order of the

paired

comparisons

was

randomized.

Thus, the

FORTRAN

computer rogram

ook

care

of the

ordering

f the

questionnaire

material.

Next,

theFORMAT

4

program

ontrolled

he for-

mat of

the

questionnaires nd printedthem, taking care

of

such

appearance

details

s

the maintenance f consistent

margins, age

num-

bers, nd spacing.The questionnaires, hichwereprinted n 8 1/2-inch

by

11-inch

aper

with

upper-

and

lower-case

etters, ppeared

to

have

been

individually yped.

Codingof

Data

One

problem

of

such

individual

questionnaires

s the

coding

of the

data

to a

usable form.

his

problem

was also

overcome.

The

application

discussed

above also

helps

to

illustrate

his

point.

As was

noted,

each

timetheprogram roduceda newquestionnaire uniquecode number

was assigned and

printed.

For

each

questionnaire,

he

program

also

produced

corresponding

EAD

statement

n

computer

ard form.

In

the

FORTRAN

computer anguage,

he

READ

statements used to control he

input

of

data

to the

program.)

This

READ

card

was

appropriately

punched

with listof variables

n

the same order

hey

had

appeared

on



7/9

ORDER-EFFECT BIAS

549

the questionnaire.When the

completed questionnaires

were

returned,

they

were

coded; first, he questionnaire dentification

umber was

coded, and then on subsequent ard imagesthe restof the data were

coded

in

sequential olumns.

The

data

were

punched,

nd then

prior

o

the statistical analysis a programwas written

hat

performed

he

unshuffling

f

the data.

The

unshufflingrocedure

s

simple.First,

he

questionnaire

dentificationode

is read in

and

used

by

the

computer

o

identify hichquestionnaires

being read;

then hat nformation

s

used

in

the program ogic

to

directcontrol to the appropriate

EAD

card

(generated

n

the arlier

tep).

The

data are read

ntothe

machine

n

this

fashion and stored

in

the

computermemorybank,

and

then

may

be

written ut again, in the order specified y theprogramer,o tape or

cards

for

further

rocessing.

his

approach

will

prove

ffective

ven

with

elaboratedata format

chemes.

Another pproach sviablewith espect o thecodingofthedata.

It

is

conceptually impler

han the

preceeding pproach,

but more

difficult

from he tandpoint

f

clerical ffort. ven with tandard

uestionnaires,

researchers requentlyrovidedata coders

with

oding

nformation.his

mostfrequentlyakes the form

f

small

numbers laced

in

parentheses

unobtrusively

n

the

margin.

The numbers

epresent

he

columns

of the

computer odingsheet computer ard) in which theresponse s to be

recorded.

n

thetraditional uestionnaire,hesenumberswould

typically

be

sequential.This approach

may

also be

used

with

questionnaires

ith

randomized rder.

The

text-stringegment or nygivenquestionwould

simplyncludethe controls o

print

he

appropriate olumn

numbers n

the

marginbeside

the

question.

With

this approach,

the

human coder

would bear the

responsibility or coding the data in the specified

columns. Unlike

the

case of the traditional

uestionnaire,

he

coding

columnswould

typically

ot be

sequential.

Because the firstapproach is really a rather simple computer

procedure,

t will

normally rove more efficienthan the second,

coder-

oriented

pproach.

Furthermore,

he

differentrdering f thedata on

each

questionnaire

would

undoubtedlymake the rather reary ask

of

data

coding

even more

burdensome nd error-prone orthe

human

coder.

ResponseRate

The use of thisproceduremay have an additionalbenefit n terms f

responserate. n the

purchasingmanager tudydiscussed bove, it was

pointed

out to

recipients f the questionnaire hat no one else

would

receive questionnaire

dentical o theirs; f course, he ndividuallyyped

appearance

of

thequestionnaire

einforced

his

statement. bout

four-

fifths

21

of

26)

of

the

sample completed nd returned he long (nine-



8/9

550 WILLIAM D. PERREAULT, JR.

page) questionnaire.

t

cannot be said withtotal confidence

hat the

response rate was influenced y the personalized

questionnaires;no

controlgroupreceived uestionnaires rintednthetraditional ashion.

It

does seem, however, hat f the respondent eelsthat his response s

uniquely mportant,

e

may be more nclined

o

respond.The

uniqueness

of the

individualquestionnairemay be emphasizedby printinghe re-

spondent'sname

at the

beginning

f

his particular uestionnaire.

his is

an

especially traightforwardrocedurewhen respondent's amesand

addresses re

alreadycompiled

for

computermailing abels.

Cost

The costs

of

theprocedure iscussedhere houldbe taken nto ccount

in

determining

he

appropriateness f

the

procedure o a given

project.

The

costsvaryprimarilyccording

o

1)

how

complicated hereordering

stage

of

the

process

s

(i.e.,

the

price

of

computer ime), 2) which ext

processing rogram

s

used, and (3)

how

much actual text here s on a

page

of the

questionnaire.

or the

project

discussed

n

this

paper,

the

total

computer cost, including supplies,

was

about

8

cents

a

page.

Dependingon thequestionnaire,ostcould varyfrom bout

2

cents

per

page to 12centsper page (or higher,fpricesrise).

For

surveys nvolving long questionnaire, large sample,

or

both,

this

cost

might

be

prohibitive.

n

such

a

case,

the randomization

proceduremight roveuseful uring retest tages o test or rder

ffect.

The

pretest sample

would be

randomly split.

Half would receive

questions

in

a standard

order;

the

remainder

would receive

questionnaires rocessed

s

discussedhere.

Both

response

ate nd actual

response

distributionsould be tested

tatistically

o determine

f

there

were

differencesbetween the

two

halves.

With

this

additional

information,he researcher ould evaluate whether rder effectwas

present nd

whether

t

needed to be controlled.

Alternatively,

f

ordereffect

as

significant,

series

of

different

ues-

tionnaires ould

be

generated,

ach

with

its

unique ordering.

These

questionnaires

would then be

printed

n

the standard

fashion. Each

individual

respondent

would

not, therefore,

receive

a

unique

questionnaire,

ut

the

procedure

would be a

relatively

fficient

pproach

to

reducing

rder-effectias.

OTHER

APPLICATIONS

Although

the

focus

of this

articlehas been

on the

applicability

f

combining

ext

diting

with

preprocessing lgorithm

o

control

ther

bias

in

survey nstruments,

he

combination

as a

number

f

other seful

applications. requently,

t

s

advantageous

n

survey esearch

o

person-



9/9

ORDER-EFFECT BIAS

551

alize advance noticeor cover

etters.9

ames and

addressesmaybe read

from mailing ist s variables n a preprocessing

lgorithm nd inserted

inthe ext-stringf a lettero beprocessed.The text s thenformatted y

the program o print he personalized etters. his mechanism lso pro-

vides a directprocedure o sample randomly

rom n existing ist and

simultaneously enerate ppropriate etters.

The textprocessing rogrammay also be usedto printmailing abels,

or print ames and addressesdirectlyn specialenvelopesdevelopedfor

use on line

printers. nvelopes

ddressed

by

thetext rocessing rogram

(with both capital and lower-case etters) renot readily dentifiables

computer utput.

CONCLUS

ION

Question sequencing

s

an

important spect

of the art

and science

of

questionnaire onstruction; uestion sequencemay have both positive

and negative ffects. his articlehas discusseda procedureby which

public opinion

researchers

may preserve

the positive aspects

of

sequencing

while

controlling

or

problems

f other

bias.

The

procedure

has additional benefits

n

that

it

may also be

used as a speedy and

relatively conomicalway to personalize urvey esearchmaterials uch

as

cover letters, dvance notice letters, nd

even the questionnaires

themselves.

'An

excellentreview of the impact that

personalized urveymaterials

may

have

on

response has been

compiled by Paul L. Erdos, ProfessionalMail

Surveys,New

York,

McGraw Hill, 1970.

Thi t t d l d d f 113 210 131 115 F i 12 A 2013 17 15 20 PM
http://www.jstor.org/page/info/about/policies/terms.jsp

controlling order-effect bias

Documents