comparing check-all and forced-choice question … · comparing check-all and forced-choice...
Post on 16-May-2018
219 Views
Preview:
TRANSCRIPT
Comparing Check-All and Forced-Choice
Question Formats in Web Surveys: The Role of Satisficing, Depth of Processing,
and Acquiescence in Explaining Differences
Jolene D. Smyth
Don A. Dillman
Leah Melani Christian
Michael J. Stern
Technical Report #05-029
1
ABSTRACT
In an experimental comparison between the check-all and forced-choice formats Rasinski,
Mingay and Bradburn (1994) found that in paper surveys the forced-choice question format
produced significantly more affirmatively marked answers than did the check-all question
format. Their research raises fundamental questions about whether different response processes
are at work in the two question formats and about the implications of those processes for survey
responses. Using results of experiments from two web surveys and a paper survey comparison of
random samples of students at Washington State University, we explore whether or not the use
of the forced-choice format increases the number of options selected affirmatively in web, as
well as paper surveys. In addition we explore three theoretical explanations for these differences
that have been proposed in previous research: satisficing (Krosnick 1991; 1999), depth of
processing (Sudman and Bradburn 1982), and acquiescence (Schuman and Presser 1981).
Finally, we report limited information on item nonresponse differences in the forced-choice
format. Our test uniformly supports the hypothesis that the forced-choice format results in more
options being selected. In addition, the forced-choice format appears to invoke deeper
processing and to eliminate satisficing behavior that occurs among some respondents to the
check-all format. However, virtually no evidence was found to support the presence of
acquiescence in the forced-choice question format. Finally, our exploration of item nonresponse
in forced-choice questions indicated that it is a fairly insubstantial problem.
Jolene D. Smyth, Leah Melani Christian and Michael J. Stern are graduate research assistants in the Department of
Sociology and the Social and Economic Sciences Research Center at Washington State University, Pullman,
Washington. Don A. Dillman is Regents Professor and the Thomas S. Foley Distinguished Professor of
Government and Public Policy in the Departments of Sociology, Department of Community and Rural Sociology
and the SESRC. This research was financed in part by funds provided to the Washington State University SESRC
under a Cooperative Agreement with the USDA-National Agricultural Statistics Service supported by the National
Science Foundation Division of Science Resource Statistics.
2
INTRODUCTION
A common question format in web surveys is the check-all question for which respondents are
asked to mark all that apply from among a list of response options (e.g. Please indicate which of
the following sources of information you have used to find employment in the last month.). One
factor that may encourage its use in web surveys is the fundamental choice between using radio
buttons that allow only one item to be selected or html boxes that allow multiple items to be
marked.
In telephone surveys check-all questions are considered awkward and seldom asked. Instead, a
forced-choice format is typically used where respondents provide an answer (e.g. yes/no) for
each item in the list. However, on self-administered questionnaires there is concern that
respondents will treat forced-choice questions as check-all questions by marking answers only in
the “yes” category and ignoring the “no” category. For this reason some self-administered
questionnaire designers avoid the forced-choice alternative. While the ability to require
responses to each item on web surveys could override that concern, error messages requiring an
answer to be marked may irritate respondents and cause them to terminate their participation in
the survey (Best and Krueger, in press).
In an experimental comparison between the check-all and forced-choice formats Rasinski,
Mingay and Bradburn (1994) found that in paper surveys the forced-choice question format
produced significantly more affirmatively marked answers than did the check-all question
format. Their research raises fundamental questions about whether different response processes
are at work in the two question formats and about the implications of those processes for survey
responses.
Using results of experiments from two web surveys and a paper survey comparison, we aim in
this research to answer the question of whether the use of the forced-choice format increases the
number of options selected affirmatively in web, as well as paper surveys. In addition we
explore three theoretical explanations for these differences that have been proposed in previous
research: satisficing (Krosnick 1991; 1999), depth of processing (Sudman and Bradburn 1982),
and acquiescence (Schuman and Presser 1981). Finally, we report limited information on item
nonresponse differences between check-all and forced-choice formats. The data analyzed here
are from 10 experimental comparisons using eight different questions (one of which was
replicated across all three surveys) collected from random samples of Washington State
University students. The deliberate use of different types of questions allows the examination of
these issues for a wide range of check-all vs. forced-choice situations.
THEORETICAL BACKGROUND
Does the Forced-choice Format Produce More Affirmative Responses?
The only experimental comparison we were able to locate between the check-all that apply
format and the forced-choice format was conducted by Rasinski et al. (1994) in a paper survey
field test for round three of the 1988 National Educational Longitudinal Study. In this
3
experiment half of the respondents were assigned a version of the survey in which three
questions were formatted as check-all that apply questions and the other half were assigned a
version in which these questions were formatted as forced-choice questions with yes/no
categories. For all three items the mean number of options marked per respondent in the forced-
choice version was significantly greater than the mean number marked per respondent in the
check-all version (3.03 vs. 2.86, p = .002; 2.47 vs. 1.53, p = .001, and 1.18 vs. 0.96, p = .001).
Rasinski and colleagues’ findings, although consistent, are limited to behavioral and
factual questions. Specifically, they asked if one’s school provided any of a list of four resources
for students planning to attend college, for which of 12 reasons students decided not to continue
their education right after high school, and which of 20 events had happened in their family in
the last two years. We also include behavioral and factual questions in our surveys (e.g.
resources used at WSU, student group participation, and food vendors used on campus). In
addition to these we seek to extend the work by Rasinski et al. (1994) by including a number of
opinion-based questions (e.g. descriptions of WSU Pullman campus, admittance criteria, and
university budget adjustments) to ascertain whether or not the effects of switching to a forced-
choice format are related to the type of question being asked.
Further, although Rasinski et al. (1994) provided evidence that the forced-choice format
leads to more response options being marked affirmatively on paper surveys, this finding has yet
to be extended to web surveys. The visual nature of both modes suggests that patterns of
responses will be similar across them, and a growing amount of experimental research has found
that other question formats react similarly across paper and web modes (see Christian and
Dillman 2004; Christian 2003). Nevertheless, because the types of specific respondent actions
required in completing a paper survey are somewhat different from those required to complete a
web survey (i.e. the use of the computer, the internet, and the mouse and screen which are
spatially separate as opposed to pencil and paper), the possibility of differences exists. However,
given previous research findings as well as the three theories presented below, we expect to
verify the finding that the forced-choice format results in more response options being marked
affirmatively and to extend that finding to web surveys.
Why Might the Forced-choice Format Produce More Affirmative Responses?
Weak Satisficing. One explanation, proposed by Rasinski et al. (1994), for the increased
affirmative responses is that the check-all that apply format encourages, or at least allows for, a
satisficing response strategy. A satisficing response strategy consists of avoiding expending the
effort required to answer the question optimally (Krosnick 1991; 1999). Satisficing can be
conceptualized as lying on a continuum ranging from strong satisficing to optimizing, with weak
satisficing falling in the middle. One example of a weak satisficing behavior is that respondents
might choose the first response options they can reasonably justify and then move on to another
question without giving adequate attention to the remaining response options (Krosnick 1991;
Krosnick and Alwin 1987). If respondents are satisficing in this way, a pattern of primacy,
where options are selected more often when they appear near the top of the list than when they
appear near the bottom of the list, is likely to occur.
The check-all that apply format may be more prone to this and other satisficing strategies
because it does not communicate the expectation that respondents should look at and come to a
4
judgment about each response option individually to the extent that the forced-choice format
does. Rasinksi et al. (1994) tested this explanation by comparing the endorsement of items in the
check-all format when they appeared in the top half of lists with endorsement of those same
items when they appeared in the bottom half (the response options were reversed in some
treatments) and found no evidence of primacy-related satisficing in their data. Similar to
Rasinski et al. (1994) we test for weak satisficing behavior by looking for primacy effects within
the check-all formatted questions. If primacy-based satisficing is occurring we expect to find
more options selected when they appear in the first half of the list than are selected from the
same response options when the order of presentation is reversed. In addition, comparing the
amount of change across formats for items that appear in the top half of the list to the amount of
change across formats for items that appear in the bottom half can shed light on whether or not
the forced-choice format leads respondents to give more attention to those options appearing
later in the lists than does the check-all format, thus reducing primacy-based satisficing.
Primacy effects may indicate the presence of one type of satisficing, but others may also occur in
check-all questions. For example, respondents may avoid expending sufficient effort for an
optimal answer through other forms of satisficing such as randomly selecting responses or giving
minimal consideration to each response option. These strategies, which most likely do not
produce accurate data, would not necessarily result in primacy effects.
Depth of Processing. Depth of processing may serve as a mechanism underlying all forms of
satisficing. Sudman and Bradburn (1982) proposed that the forced-choice format encourages
deeper processing of the question and the response options. Such deeper processing may reduce
satisficing because with more consideration respondents are likely to recognize that more of the
response options apply to them (Sudman, Bradburn, and Schwarz 1996). Thus, although
Sudman and Bradburn (1982) recognize that the forced-choice format is more “cumbersome”
than the check-all that apply format, they see it as providing “better responses” because it
requires the respondent to consider each option and come to a decision about it (p. 168).
However, they have not empirically tested this proposition.
In this paper we use client side paradata (Heerwegh and Loosveldt 2002; Stern et al. 2004) to
examine depth of processing by looking at the amount of time respondents spend on the check-
all format as opposed to the forced-choice format. If respondents are processing the forced-
choice formatted questions and response options more deeply than the check-all formatted
questions and response options, then they should be taking significantly longer to respond to
them. We recognize that the mechanical aspects of answering forced-choice questions are likely
to require more time as respondents need to mark an answer for every option instead of only
some options, but if the forced-choice format takes substantially more time we can fairly
confidently conclude that there is more going on than simply additional answer marking.
Acquiescence. Another explanation for the finding of more options marked affirmatively in the
forced-choice format compared to the check-all format is that the forced-choice format is prone
to an acquiescence or agreeing response bias (Schuman and Presser 1981) that is absent in the
check-all format. This explanation suggests that respondents have an aversion to rejecting
response options by answering them negatively. In check-all questions respondents can leave
options blank which doesn’t necessarily indicate rejection (additional discussion on this point is
5
below), but when forced to answer “yes” or “no” for each response option, those who are neutral
may be more likely to check “yes” than to check “no” because their tendency is to avoid the
outright rejection that the “no” answer choice implies. Leaving a check box blank on a check-all
question may not carry the negative connotation that is implied by checking “no” on a forced-
choice item. Thus, an acquiescence effect would lead us to expect respondents to select the
affirmative category more often on forced-choice questions.
We test for the existence of an acquiescence effect on two questions by comparing the results of
a forced-choice question with only two categories (i.e. yes/no) to one with three categories (i.e.
yes/no/don’t know or yes/no/neutral). If respondents to the typical forced-choice question (two
categories) are avoiding the “no” category and thus using the “yes” category more frequently, as
would be predicted if acquiescence is occurring, then we would expect to see significantly less
use of the “yes” category in the three-category format as respondents are provided with another
outlet. Respondents who are undecided or neutral should move from “yes” to the third category
that is more fitting for them.
To further test for an acquiescence effect in the forced-choice format, we also included
experimentation testing the effects of using more active category labels (i.e. fan/not a fan versus
yes/no). We hypothesize that the use of more active category labels, intended to remove the
inquiry from an agree/disagree context, would reduce acquiescence by giving the respondent a
more explicit and tangible way to categorize themselves.
The three explanations for the finding of more affirmative responses in the forced-choice
format—satisficing, depth of processing, and acquiescence—will be examined in this paper in an
effort to determine which one(s) explain the findings. This is important given the implications of
each of the explanations. If satisficing lies at the roots of previous findings, then one could argue
that the forced-choice format yields “better answers” than the check-all that apply format. The
same conclusion applies if it is found that respondents are processing the forced-choice format
more deeply. However, if the difference is due to acquiescence, then the forced-choice format
may not yield improved accuracy in responses and the recommendations for using it in the place
of the check-all format may be unfounded. Although we discuss the implications of these three
explanations separately throughout the paper for analytical purposes, we want to stress that these
processes are not necessarily independent of each other and that evidence for or against one
explanation does not automatically confirm or disqualify another.
Item Nonresponse in the Forced-Choice Question Format
In addition to comparing the check-all and forced-choice formats and considering the
explanations for findings, we examine the frequency of item nonresponse in the forced-choice
format. Sudman and Bradburn (1982) point out that there are a number of reasons a respondent
may leave an option unchecked in a check-all question: the option doesn’t apply to them, they
are neutral or undecided about it, or they overlooked it. The forced-choice format, they argue,
eliminates some of this uncertainty by providing a “no” category for respondents to mark if the
option does not apply to them. However, nonresponse in the forced-choice format may occur if
respondents treat it as a check-all question by ignoring the “no” or otherwise negative category
6
and only marking within the “yes” category. If a large proportion of respondents to the forced-
choice questions treat them as check-all questions, thus increasing item nonresponse, the data
may not be more interpretable than the check-all data.
PROCEDURES
We explore these issues using up to four experimental variations of substantively different
questions from two web surveys and one paper survey, all designed to assess the undergraduate
experience at Washington State University (WSU). The first web survey was conducted from
March 11 to May 9, 2003, a year after the paper survey. The sample consisted of 3,004
randomly selected WSU undergraduate students from the Pullman campus. Of these students
1,591 completed the survey for a response rate of 53 percent. Three of the 21 questions from this
survey are analyzed in the current paper. All of the questions appeared on their own page in
black text against a colored background. Answer spaces appeared in white so as to provide
contrast between the answer spaces and the background. All screens were constructed with
HTML tables using proportional widths in order to maintain the visual aspect of the screen
regardless of individual user window sizes. In addition, font size and style were automatically
adjusted using Cascading Style Sheets to accommodate differing user screen resolutions.
The second web survey, conducted in the fall of 2003, was administered to a sample of 3,045
WSU undergraduates, 1,705 of whom completed the entire survey (56 percent response rate).
This survey was designed to assess students’ experiences both on and off campus in Pullman so
as to allow for variation in the substance of the questions. Many design aspects of this survey
were similar to the first including the page-by-page construction; the contrast between the
colored background, black text, and white answer spaces; and the precautions taken to minimize
the extent to which the appearance of the questions would vary across browsers or hardware
configurations. Seven questions from this second web survey are analyzed in the current paper.
Contact letters and follow-ups were sent using both postal mail and electronic mail for each of
the web surveys. All respondents received an initial postal mail contact with a two-dollar
incentive as well as their individual identification number they could use to access the survey.
Those respondents for whom we were able to obtain e-mail addresses also received an e-mail
containing a direct hotlink to the survey. For each web survey four versions were constructed
using the same questions substantively, but varying their formats. A random number generator
was used to assign one of the versions to each respondent when they entered the survey. In the
first web survey 27.3 percent completed version one, 27.5 percent completed version two, 23.1
percent completed version three, and 22.1 percent completed version four. On the second web
survey 23.1 percent completed version one, 26.2 percent completed version two, 24.8 percent
completed version three, and 26 percent completed version four.
The paper survey was administered by mail from March to April 2001 to a random sample of
1,800 WSU (Pullman campus only) undergraduate students. The item from this survey used in
the current analysis was embedded on the last page of a four page questionnaire that was printed
in a two-column format on 8-1/2 x 11 inch pages with a colored background that contrasted with
white answer spaces. Four variations or versions of the questionnaire were mailed randomly to
7
equal subsamples of students in the sample; however, the experimental questions reported here
were identical in two of the four versions. A two-dollar incentive was enclosed with the first
mailing and a follow-up postcard and replacement questionnaire were mailed. The response rate
on this survey was 57.9 percent (1,042).
Statistical tests made in the analyses include chi-square tests for differences between the percent
of respondents using various response behaviors and t-tests for differences in the mean number
of response options checked between formats or in the mean time it took to complete formats.
FINDINGS
Does the Forced-choice Format Produce More Affirmative Responses?
Results in Table 1 from both web experiments and the paper experiment unequivocally support
the expectation that the forced-choice format yields more options marked affirmatively than the
check-all format. Overall in the check-all formatted versions an average of 4.1 options were
marked per question. In the forced-choice version the average number of options marked per
question was significantly higher at 5.0 (t = -18.57, p = .000). Fifteen of the sixteen comparisons
were significantly different in the expected direction and the sixteenth approached significance
(p = .054). Additionally, 91 percent of response options were marked affirmatively more often
when they appeared in the forced-choice format. These results confirm the findings of Rasinski
et al. (1994) that the forced-choice format leads to more options being marked affirmatively than
the check-all format, but more importantly, they extend those findings from paper surveys to web
surveys and across different types of questions ranging from behavioral and factual to opinion
based.
Why Might the Forced-choice Format Produce More Affirmative Responses?
Weak Satisficing. If the observed difference between the check-all and forced-choice formats is
due to weak satisficing we would expect to find a pattern of primacy in the check-all questions;
that is, items would be checked more often when they appear early in the list of response options
than when they appear later in the list. We test for primacy in the check-all format by comparing
the mean number of response options marked in the top and bottom half of the lists when those
lists are presented in original and reverse order.
The top panel of Table 2 shows such comparisons for five questions. For each of these questions
one of the treatments involved presenting the response options in the “original” order and
another involved presenting them in the “reverse” order. Only one question shows a significant
pattern of primacy. For question Q11 the average number of response options checked for
options A - E is 3.89 when they are located in the top half of the table and only 3.75 when they
are located in the bottom half of the table. Similarly, more of options F - J are marked when they
are located in the top as opposed to bottom half of the list (2.07 vs. 1.64). Both of these
differences are significant. In addition to this question seven of the eight remaining comparisons
8
are in the expected direction (more options are endorsed when they appear in the top of the list),
but none of these are significant.
The bottom panel of Table 2 shows similar comparisons, but in this case the analysis is limited to
only the first and last three response options in the list. Limiting the analysis to only the first and
last three options should provide a more refined test for primacy effects as it eliminates any
obscuring effects of the middle categories. Results indicate that four of the ten comparisons
show significant primacy patterns when analyzed in this way. In addition, all but two
comparisons are in the expected direction. Overall then, we are left with mixed evidence of
primacy in the check-all format.
The results in Table 1 tell us that the forced-choice format leads to a higher mean number of
response options marked affirmatively, but do not specify which response options are marked
more often in the forced-choice format. However, if the forced-choice format is eliminating
satisficing (in the form of primacy) from the check-all format, we would expect to see a greater
increase in the bottom half of the list of response options than we see in the top half when we
compare these two question formats. Therefore, an additional test of satisficing consists of
comparing the mean number of options marked in the top and the bottom half of lists of response
options across question formats.
Table 3 shows the mean number of response options marked in the top (and bottom) half of the
forced-choice formatted versions expressed as a proportion of the mean number of response
options marked in the top (and bottom) half of the check-all formatted version for each question.
For example, the number 104 in the first row of Table 3 indicates that for question 11 in the first
web experiment, the top half of the forced-choice version produced a mean number of response
options marked that was 4 percent higher than the mean number in the top half of the check-all
version. The bottom half of the forced-choice version produced a mean number of response
options marked that was 16 percent higher than the mean number in the bottom half of the
check-all version. The difference between the 4 percent increase in the top half and the 16
percent increase in the bottom half when the question is converted to a forced choice format is in
the expected direction, but is not statistically significant. In fact, 10 of the 16 comparisons were
in the expected direction, but only five were significant. In addition, six comparisons were not in
the expected direction, four of which occurred in versions where the response options were
reversed from the original order (which yielded results in the expected direction). These findings
suggest that the mean number of response options marked by location may be more closely
linked to the options themselves than to respondent answer strategies such as satisficing.
Overall, the difference between the mean proportional increase in the top half (22%) and the
mean proportional increase in the bottom half (33%) was only approaching significance (t =
1.58, p = .068). Thus, we are left with minimal evidence that the forced-choice format corrects
for any primacy that may occur in the check-all format.
The lack of primacy and correction for primacy found in these questions is evidence against the
hypothesis that the increased affirmative responses in the forced-choice format are the result of
satisficing in the check-all format. These findings are consistent with those reported by Rasinski
et al. (1994).
9
Depth of Processing. If the explanation for the increased affirmative selection of response
options in the forced-choice format is deeper processing, then we would expect respondents to
spend more time on the forced-choice formatted versions. Paradata results shown in Table 4
indicate that respondents overwhelmingly spent more time on the questions when they were
formatted as forced-choice questions than when they are formatted as check-all questions. This
finding occurs across all 15 comparisons from the two web surveys. Some of this additional
time spent in the forced-choice format is undoubtedly spent marking the “no” category, a step
that is not required on the check-all format; however, the magnitude of the time differences
between the formats suggests that respondents are spending more time on the forced-choice
format independent of this extra response step. At minimum, respondents spent 45 percent more
time on the forced-choice format than on the check-all format and on average they spent two and
a half times as much time on the forced-choice formatted questions as on the check-all formatted
questions.
Further analysis of the time spent on these two question formats can be found in Table 5 which
shows the mean number of response options marked affirmatively by those taking the mean time
or less to complete each version of each question and those taking over the mean time to
complete each version. This table shows that on the check-all formatted versions, respondents
who spent above the mean amount of time on the questions checked more answers on average
than those who spent the mean amount of time or less, suggesting that those spending more time
were processing the response options more deeply and thus finding more response options that
were relevant to them. These differences were significant on all 14 check-all treatments.
Further analysis (not shown) indicates that the mean number of options marked by the
respondents to the check-all version who used greater than the mean amount of time approached
the mean number of options marked affirmatively by all respondents on the forced-choice
version. In fact, for 11 of the 15 web comparisons the mean number of items marked by the
check-all respondents who took above the mean time to answer surpassed the mean for the
forced-choice version, while a 12th
comparison matched. Overall, respondents to the check-all
version who used greater than the mean amount of time marked 5.4 options on average while all
respondents to the forced-choice version marked an average of 5.1 options (two-sided t = 1.91, p
= .077).
The pattern of differences in the mean number of response options marked based on amount of
time spent answering found in the check-all versions (Table 5) did not carry over into the forced-
choice versions of these questions. In 15 of 19 forced-choice treatments there was no significant
difference in the mean number of response options marked affirmatively between those taking up
to the mean amount of time and those taking above the mean amount of time to complete the
question.1 These findings suggest that the additional time spent on the forced-choice format that
we see in Table 4 is sufficient enough for respondents to more deeply process all of the response
options such that spending even more time does not lead to more options being marked.
1 Since this table relies on measures of central tendency which in turn rely on data reduction, OLS regressions were
conducted in which the number of response options marked affirmatively was regressed on the time taken to answer
the question for each treatment of each question. The results (not shown) confirm those in Table 6 such that time
was positively and significantly related to the number of options marked in the check-all format, but not in the
forced-choice format.
10
Taken together, the findings in Tables 4 and 5 as well as the additional analysis of the check-all
responses by time support the explanation that the increase in the mean number of response
options marked in the forced-choice format compared to the check-all format is the result of
deeper processing. In addition they suggest that there is some level of “optimal” processing
which respondents to the forced-choice format and those using over the mean amount of time in
the check-all format seem to be reaching.2 As such they raise concerns about the use of the
check-all format because a substantial proportion of respondents spent at or below the mean
amount of time (range 47% - 76%, mean across questions = 66%) on the questions in this format
and therefore may not have reached that “optimal” processing level.
With this data in mind we returned to the issue of primacy as an indicator for satisficing in the
check-all questions and the question of whether or not the deeper processing in the forced-choice
format is eliminating primacy effects that we see in the check-all format. We tested for primacy
separately for those spending up to the mean amount of time on the check-all questions and those
spending above the mean amount of time. Results in Table 6 indicate that for those spending up
to the mean amount of time, five of the ten comparisons of the mean number of options selected
when they appeared in the top and bottom halves of lists show significant primacy effects.
However, eight of the ten comparisons show significant primacy effects for this same group
when the analysis is limited to the top and bottom three response options in the lists rather than
the top and bottom halves of options (thus eliminating the obscuring effects of the middle
categories which don’t significantly change position in the reversal). For those spending above
the mean amount of time on these same questions there is virtually no evidence of primacy
effects (Results not shown, but available on request). Only one comparison is significant
regardless of whether all the options are considered or only the top and bottom three (Q11,
options A-C in the top three/bottom three analysis).
Acquiescence. The hypothesis that the forced-choice format leads to more affirmative responses
because respondents are apprehensive about outright rejecting response options by marking “no”
and thus are more likely to agree is examined for two questions from the second web experiment
(Fall 2003). The first question presented respondents with descriptors of the WSU Pullman
campus and asked them which they thought applied. Among the experimental treatments was
one in which a third category, “don’t know,” was offered in addition to “yes” and “no”
categories. Similarly, a third category, “neutral,” was offered on one treatment for a question
asking respondents to report which Cougar varsity sports they were a fan of. If acquiescence is
at work we expect to see the “don’t know” or “neutral” category drawing those apprehensive
respondents predominantly from the “yes” category.
As Tables 7 and 8 illustrate, in neither of these questions did the third category draw respondents
predominantly from the “yes” category as expected. Instead, for the campus descriptors question
(Table 7), the “don’t know” category was not used sufficiently enough to draw significant
numbers of respondents from either the “yes” or the “no” categories. If anything, it appears as if
the addition of this third category increased the use of the “no” category, although there were
2 Respondents to the check-all version who used over the mean amount of time still used significantly less time than
respondents to the forced-choice version on 12 of the 15 comparisons. This suggests that the forced-choice format
may be more burdensome than what is “optimal”, but the additional burden may be a small price to pay for a gain in
accuracy.
11
few significant differences among individual response options. However, it did significantly
reduce both the mean number of options marked “yes” and the mean number of options marked
“no.” This finding serves as evidence against acquiescence as an explanation. On the sports fan
questions the findings are a little clearer, but no more in support of an acquiescence explanation.
Table 8 indicates that the addition of the third category drew significantly from the “no” category
for each item except for one, men’s football, which was overwhelmingly popular on all three
surveys. The third category did not result in significant reductions of the use of the “yes”
category for any one response option and it did not significantly reduce the mean number of
options marked yes overall. As a result this question also provides evidence against the
acquiescence explanation.
Table 9 reports the results of two experiments from the first web survey that tested the results of
the use of more active category labels to replace the standard yes/no labels. In the first question
respondents to one version were asked to respond either “yes” or “no” to indicate whether or not
they were fans of each of the cougar varsity sports. Respondents to another version were asked
the same question except the yes/no categories were replaced with “fan” and “not a fan”
categories. In the second question the yes/no categories were replaced with “participated” and
“have not participated” on a question asking respondents if they had participated in various types
of student groups. Consistent with the lack of acquiescence found above, results from both of
these questions indicate that removing items from the agree/disagree context through the use of
more active labels does not produce significantly different response distributions. Across both
questions only one response option was marked by a significantly different percent of
respondents when the active labels were provided. In addition, there was no difference on either
question in the mean number of response options marked by respondents across the treatments.
The finding of no difference between the use of yes/no categories and more active category
labels provides further evidence against acquiescence as an explanation.
Item Nonresponse in the Forced-Choice Question Format
Results shown in Table 10 indicate that between zero and about 11 percent of respondents
treated the forced-choice questions as check-all that apply questions by marking only within the
“yes” category,3 thus introducing item nonresponse. Across all 24 forced-choice treatments
included in all three surveys, the mean percent treating them as check-all questions was 2.68,
suggesting that this response pattern is not detrimental to the forced-choice format in general.4
However, the high percentages on some of the questions do raise alarms. In addition, the
extremely low percentages on other questions lead us to investigate what makes these particular
questions less likely to produce check-all response patterns.
One possible explanation is that respondents treat forced-choice question as check-all questions
more often when they mark the first response option affirmatively because they then continue to
3 Respondents who marked all of the options “yes” were excluded from these percentages as we assume they
sincerely meant “yes” on all options and were not treating the question as a check-all. 4 An additional question, Q24, is included in this analysis that is not included in previous analyses. There is no
check-all treatment for this question which precludes its inclusion in previous analyses, but that limitation is not
relevant for the current analysis.
12
focus on the “yes” category and do not notice other categories. This explanation is suggested by
the two questions in which the percent of respondents treating them as check-all questions are
significantly different in the original and reverse order formats. However, the data did not
support this explanation as only about 62 percent of respondents who treated a forced-choice
formatted question as a check-all question marked the first response option affirmatively (across
all questions). Thus, it is more likely that such response behavior is linked to the particular type
of question being asked. Specifically, questions based on opinions may require more careful
consideration by the respondents and therefore discourage treatment as check-all whereas
behavior- and factual-based questions may not require such careful consideration and as such
may facilitate “quick clicking” which may result in this answering pattern. In this data there are
eight questions that are behavior- or fact-based (designated by “B” in Table 10) and four that are
opinion-based (designated by “O”). The mean percent of respondents treating the forced-choice
questions as check-all questions is 3.47 for the behavior/fact-based questions and only 1.58 for
the opinion-based questions. This difference is approaching significance (one-sided t = -1.55, p
= .067), suggesting support for this explanation. Further support is suggested by the mean time
data in Table 4. On average the behavior/fact-based questions took 23.15 seconds to complete
while the opinion-based questions took 42.16 seconds to complete (one-sided t = 4.79, p =
.002).5 Together, these findings suggest that respondents gave more consideration to (or at least
took longer to process) the opinion-based questions which discouraged their treatment of them as
check-all questions compared to the behavior/fact-based questions.6
DISCUSSION
In this paper we have explored whether posing questions as forced-choice items rather than the
often used check-all format influences the number of options marked affirmatively in self-
administered surveys. We have also explored three theoretical explanations for why these
formats may influence the number of responses chosen as well as whether item nonresponse is a
significant issue/problem in the forced-choice format.
Consistent with experimental results from a mail self-administered survey reported by Rasinski
et al. (1994) our test of 10 items in two web experiments and a mail comparison uniformly
support the hypothesis that the forced-choice format results in more options being selected. Our
results included item order reversals, replication of one item across all three surveys, and opinion
as well as behavioral items of the nature explored by Rasinski et al (1994). Together with
previous findings these data strongly suggest that on self-administered surveys when respondents
are presented with the forced-choice format instead of the check-all format, they will select a
greater number of options.
Three possible explanations for the tendency to choose more items under the forced-choice
condition were evaluated using subsets of items. The most convincing explanation for the
5 This time data should be interpreted with caution because there are significant differences in the length of the
question stems across these two types of categories. Thus, the time differences could be due to increased reading
and comprehending requirements. 6 Use of the more active labels did not significantly influence item nonresponse in the forced-choice format (see
Table 9)
13
increases in options marked affirmatively in the forced-choice format appears to be that the
forced-choice format leads respondents to more deeply process the response options. This was
evident in the differences in the time it took to complete the two question formats as well as in
exploration of the mean number of response options marked by those taking up to and those
taking over the mean amount of time to complete each question. In the check-all format those
respondents who took more time to complete the questions marked more answers, but the same
was not found in the forced-choice format. This finding in conjunction with the overall
differences in the mean time and the mean number of response options marked across the two
formats suggests that the forced-choice format results in deeper processing of response options
among all respondents, not just those who take above the mean amount of time to complete
questions.
Evidence of satisficing in the form of primacy effects in check-all questions was weak when all
respondents were included in the analysis; however, when analyses were conducted separately
for those taking up to the mean amount of time to complete the questions and those taking above
the mean amount of time, fairly strong evidence of primacy was found for those who answered
the questions relatively quickly. Taken together, the evidence suggests that respondents who
take longer to answer questions in the check-all format and respondents to questions in the
forced-choice format reach a more “optimal” level of processing than do the approximately 66
percent of respondents who only take up to the mean amount of time to answer questions in the
check-all format. This deeper processing seems to eliminate satisficing behavior in that no
evidence of primacy was found for respondents to the check-all format who took above the mean
amount of time to complete the questions. Thus, while the evidence suggests that most
respondents in the forced-choice format are adequately processing and responding to each
question and its response options, the same cannot be said for the check-all format.
Virtually no evidence was found to support the presence of acquiescence in the forced-choice
question as the addition of a third category (“don’t know” or “neutral”) did not draw significant
numbers of respondents from the use of the “yes” category. In addition, the use of more active
category labels, thought to reduce acquiescence by eliminating the agree/disagree context of the
question and replacing it with more tangible options, had no significant effects on response
distributions. However, since our analysis was limited to only two items, further testing of this
explanation is needed.
Finally, our exploration of item nonresponse in forced-choice questions indicated that this is not
an enormous impediment to the forced-choice question format. The proportion of respondents
who treated the forced-choice questions as defacto check-all questions tended to be small,
usually less than five percent. However, the occurrence of this response pattern seemed to vary
across different types of questions. With regard to these findings, it is important to note that the
wording of all of the questions tested here avoided a kind of prose that sometimes leads
respondents to mark only “yes” answers (e.g. “Please check which of these sports you are a fan
of”) by including the positive and negative categories as part of the question stem. Question
wording that uses such prose may encourage the occurrence of this response pattern in forced-
choice questions and therefore should be avoided. Nevertheless, in the cases we presented here
we do not see the relatively small number of people who chose not to mark negative answers as a
major barrier to the use of a forced-choice format.
14
CONCLUSION
Using results of experiments from two web surveys and a paper survey comparison we have
confirmed the finding of Rasinski et al. (1994) that the forced-choice format produces more
affirmatively marked response options than the check-all format and we have extended this
finding in three important ways. First, after verifying Rasinski and colleagues’ finding in a paper
survey, we extended it to web surveys. Second, this finding holds across a wide range of
question types (including both behavior/fact-based questions and opinion-based questions) as
well as across a variety of numbers of response options (9 to 14). The third and most important
extension of Rasinski and colleagues’ finding is our finding that the forced-choice format
appears to invoke deeper processing and to eliminate weak satisficing behavior that occurs
among some respondents to the check-all format. It appears as if the increased marking of
options affirmatively is due in large part to deeper processing of the forced-choice format,
confirming the speculation of several researchers (Sudman and Bradburn 1982; Dillman 2000).
We find no evidence in support of acquiescence in the forced-choice format.
It appears that the use of the forced-choice question format is a desirable alternative to the use of
the check-all question format for multiple answer questions in web surveys because it seems to
promote deeper processing of the question and response options and allows for finer
differentiation of meaning for options marked negatively. Although the evidence that the forced-
choice format produces more accurate responses is increasing, like Rasinski et al. (1994) we lack
external validation checks for our data and therefore cannot say with absolute certainty that the
forced-choice format produces more accurate responses. As such, this is an issue in need of
further research.
In addition to external validation checks, an important next step in this research is to compare the
use of forced-choice formats in aural (e.g. telephone) surveys where the check-all format is
rarely used to its use in self-administered surveys. Although we do not yet know whether the
same number of items are likely to be chosen across modes, the forced-choice format, which is
similar in wording to that used in aural surveys, produces selection of significantly more items
than the number chosen in the check-all format on self-administered surveys. This finding gives
ample reason to be concerned about the current common practice of automatically converting
those items from forced-choice to check-all formats when switching from telephone to web or
other self-administered modes.
15
Table 1: Comparison of Mean Number of Response Options Marked Affirmatively
Between the Check-All-That-Apply and Forced-Choice Format
One-Sided
Test of
Significance
Mean:
Check-All
Version
Mean:
Forced-
choice
Version
Difference in
Means T p
WEB EXPERIMENT #1: Spring 2003
Q11: Resources used at WSU (10)
†
Check-All vs. Used/Not Used 5.4 5.7 -0.3 -3.41 .000*
Check-All (R) vs. Used/Not Used (R)††
5.6 6.2 -0.6 -5.43 .000*
Q13: Cougar varsity sports fan (15)
Check-All vs. Yes/No 2.6 3.6 -1.0 -4.43 .000*
Check-All vs. Fan/Not a Fan 2.6 3.9 -1.3 -5.87 .000*
Q16: Student group participation at WSU (11) Check-All vs. Yes/No 1.9 2.6 -0.7 -5.01 .000*
Check-All vs. Part./Not Part. 1.9 2.4 -0.5 -3.67 .000*
WEB EXPERIMENT #2: Fall 2003
Q3: Descriptions of WSU Pullman Campus (12) Check-All vs. Yes/No 4.4 6.6 -2.2 -17.68 .000*
Q6: Admittance Criteria (14) Check-All vs. Yes/No 5.0 6.1 -1.1 -6.27 .000*
Check-All (R) vs. Yes/No (R) 5.2 5.9 -0.7 -4.46 .000*
Q11: University Budget Adjustments (14) Check-All vs. Yes/No 3.5 4.6 -1.1 -7.24 .000*
Q14: Cougar varsity Sports Fan (15) Checked vs. Yes/No 3.1 4.5 -1.4 -6.19 .000*
Q16: Food vendors on campus (9) Checked vs. Yes/No 4.4 5.0 -0.6 -3.56 .000*
Checked (R) vs. Yes/No (R) 4.6 5.1 -0.5 -3.79 .000*
Q20: Possessions in Pullman (13) Check-All vs. Yes/No 6.4 6.7 -0.3 -1.94 .026*
Check-All (R) vs. Yes/No (R) 6.6 6.9 -0.3 -1.61 .054
PAPER EXPERIMENT: Spring 2002
Q5: Cougar varsity sports fan (15)
Check-All vs. Yes/No 2.6 3.8 -1.2 -5.94 .000*
OVERALL MEAN 4.1 5.0 -0.9 -18.57 .000* † The number of response options offered for each question is displayed in parenthesis.
† † “(R)” denotes treatments in which the options were presented in reverse order (inverted).
* p ≤ .05
16
Table 2: Primacy Effects in the Check-All Format: Mean Number of Options Checked
when Options Appeared in (1) Top vs. Bottom Half and (2) Top Three vs.
Bottom Three Positions in List
Mean Number of Options Checked by Location: Top and Bottom Half of List
ORIGINAL
ORDER
REVERSE
ORDER t p
WEB EXPERIMENT #1
Q11: Resources Used at WSU (10)
†
Options A - E 3.89 T 3.75 B 1.92 .027*
Options F - J 1.64 B 2.07 T 5.89 .000*
Q13: Cougar Varsity Sports Fan (15)
Options A - G 1.63 T 1.71 B -0.91 .817
Options I - O 0.96 B 1.03 T 0.73 .233
WEB EXPERIMENT #2
Q6: Admittance Criteria (14)
Options A - G 2.17 T 2.14 B 0.32 .374
Options H - N 2.44 B 2.55 T 0.95 .171
Q16: Food Vendors on Campus (9)
Options A - D 2.41 T 2.32 B 0.89 .184
Options F - I 1.31 B 1.37 T 0.67 .252
Q20: Possessions in Pullman (13)
Options A - F 2.41 T 2.38 B 0.41 .341
Bottom H - M 2.61 B 2.79 T 1.57 .058
Mean Number of Options Checked by Location: Top and Bottom Three Positions of List
WEB EXPERIMENT #1
Q11: Resources Used at WSU (10)
Options A - C 2.32 T 2.16 B 2.89 .002*
Options H - J 0.51 B 0.95 T 8.13 .000*
Q13: Cougar Varsity Sports Fan (15)
Options A - C 0.71 T 0.79 B -1.18 .882
Options M - O 0.55 B 0.61 T 0.91 .183
WEB EXPERIMENT #2
Q6: Admittance Criteria (14)
Options A - C 2.06 T 2.04 B 0.29 .385
Options L - N 0.68 B 0.81 T 2.10 .018*
Q16: Food Vendors on Campus (9)
Options A - C 2.08 T 1.98 B 1.55 .060
Options G - I 0.82 B 0.89 T 1.09 .137
Q20: Possessions in Pullman (13)
Options A - C 0.92 T 0.92 B -0.01 .504
Bottom K - M 1.30 B 1.43 T 1.98 .024*
† The number of response options offered for each question is displayed in parenthesis. * p ≤ .05
17
Table 3: Mean Number of Options Marked by Location in the Forced-Choice Version
Expressed as a Proportion of the Mean Number of Options Checked by
Location in the Check-All Version.1
Proportion
Test of
Significance
Top Bottom X
2 p
WEB EXPERIMENT #1
Q11: Resources used at WSU (10)
†
Check-All vs. Used/Not Used 104 116 3.45 .063
Check-All (R) vs. Used/Not Used (R)††
115 110 0.61 .436
Q13: Cougar varsity sports fan (15)
Check-All vs. Yes/No 115 163 21.81 .000*
Check-All vs. Fan/Not a Fan 125 183 16.86 .000*
Q16: Student group participation at WSU (11) Check-All vs. Yes/No 144 117 4.41 .036
Check-All vs. Participated/Not Participated 139 105 8.13 .004
WEB EXPERIMENT #2
Q3: Descriptions of WSU Pullman Campus (12) Check-All vs. Yes/No 139 164 5.83 .016*
Q6: Admittance Criteria (14) Check-All vs. Yes/No 112 126 3.22 .073
Check-All (R) vs. Yes/No (R) 126 115 0.74 .391
Q11: University Budget Adjustments (14) Check-All vs. Yes/No 128 135 0.58 .448
Q14: Cougar varsity Sports Fan (15) Checked vs. Yes/No 137 185 13.23 .000*
Q16: Food vendors on campus (9) Checked vs. Yes/No 107 115 1.07 .301
Checked (R) vs. Yes/No (R) 127 112 3.25 .072
Q20: Possessions in Pullman (13) Check-All vs. Yes/No 100 108 1.47 .225
Check-All (R) vs. Yes/No (R) 111 105 0.78 .378
PAPER EXPERIMENT
Q5: Cougar varsity sports fan (15)
Check-All vs. Yes/No 125 170 17.19 .000*
MEAN PROPORTIONAL CHANGE 122 133 t = -1.58 .068
1 The proportional change is calculated by: (χfc/χck) *100 Where χfc is the mean number of options marked in
the top (or bottom) half of the forced-choice version and χck is the mean number of options checked in the top
(or bottom) half of the check-all version. * p ≤ .05 and in the expected direction. † The number of response options offered for each question is displayed in parenthesis.
† † “(R)” denotes treatments in which the options were presented in reverse order (inverted).
* p ≤ .05
18
Table 4: Comparisons of the Mean Time (seconds) Spent Answering Questions Between
the Check-All and Forced-Choice Formats
Mean Time One-Sided Test
of Significance
Check-
All
Forced-
choice Difference
Percent
Increase T p
WEB EXPERIMENT #1: Spring 2003
Q11: Resources used at WSU (10)
†
Check-All vs. Used/Not Used 15.88 24.97 -9.09 57 -13.42 .000*
Check-All (R) vs. Used/Not Used
(R)††
19.22 27.89 -8.67 45 -14.45 .000*
Q13: Cougar varsity sports fan (15)
Check-All vs. Yes/No 14.12 30.46 -16.34 116 -13.92 .000*
Check-All vs. Fan/Not a Fan 14.12 27.98 -13.86 98 -17.73 .000*
Q16: Student group participation at
WSU (11)
Check-All vs. Yes/No 16.48 27.08 -10.6 64 -13.03 .000*
Check-All vs. Part./Not Part. 16.48 27.02 -10.54 64 -13.16 .000*
WEB EXPERIMENT #2: Fall 2003
Q3: Descriptions of WSU Pullman
Campus (12)
Check-All vs. Yes/No 11.62 35.47 -23.85 205 -22.26 .000*
Q6: Admittance Criteria (14) Check-All vs. Yes/No 13.04 42.39 -29.35 225 -20.35 .000*
Check-All (R) vs. Yes/No (R) 13.16 44.15 -30.99 235 -20.64 .000*
Q11: University Budget Adjustments
(14)
Check-All vs. Yes/No 14.35 54.54 -40.19 280 -24.10 .000*
Q14: Cougar varsity Sports Fan (15) Checked vs. Yes/No 5.50 27.06 -21.56 392 -28.02 .000*
Q16: Food vendors on campus (9) Checked vs. Yes/No 5.18 12.48 -7.3 141 -19.57 .000*
Checked (R) vs. Yes/No (R) 5.58 12.56 -6.98 125 -22.30 .000*
Q20: Possessions in Pullman (13)
Check-All vs. Yes/No 7.76 20.21 -12.45 160 -19.79 .000*
Check-All (R) vs. Yes/No (R) 7.99 21.72 -13.73 172 -22.79 .000*
OVERALL MEAN 12.03 29.07 -17.04 159 -6.58 .000*
† The number of response options offered for each question is displayed in parenthesis.
† † “(R)” denotes treatments in which the options were presented in reverse order (inverted).
* p ≤ .05
Time outliers were removed at 2 standard deviations above the mean.
19
Table 5: Mean Number of Response Options Marked Affirmatively by Those Taking
Longer Than the Mean Time Per Treatment and Those Taking Less than or
Equal to the Mean Time Per Treatment
One-Sided Test
of Significance
Above
Mean Time
Mean Time
and Below
Difference
in Means t p
WEB EXPERIMENT #1: Spring 2003
Q11: Resources used at WSU (10)†
Check-All 5.8 5.2 0.6 3.84 .000*
Check-All (R)††
5.8 5.5 0.3 1.74 .041*
Used/Not Used 5.8 5.7 0.1 0.40 .347
Used/Not Used (R) 6.2 6.2 0.0 0.05 .479
Q13: Cougar varsity sports fan (15)
Check-All 4.3 1.5 2.8 13.52 .000*
Check-All (R) 4.1 1.7 2.4 10.63 .000*
Fan/Not a Fan 5.1 3.4 1.7 4.26 .000*
Yes/No 4.2 3.6 0.6 0.68 .249
Q16: Student group participation at WSU (11) Check-All 2.4 1.7 0.7 4.18 .000*
Participated/Not Participated 2.9 2.2 0.7 2.98 .002*
Yes/No 3.2 2.4 0.8 3.52 .000*
WEB EXPERIMENT #2: Fall 2003
Q3: Descriptions of WSU Pullman Campus (12) Check-All 5.4 3.7 1.7 9.73 .000*
Yes/No 6.5 6.6 -0.1 -0.76 .775
Yes/No/Don’t Know 5.7 6.2 -0.5 -2.67 .996
No/Yes 6.3 6.7 -0.4 -1.85 .968
Q6: Admittance Criteria (14) Check-All 6.2 4.2 2.0 10.29 .000*
Check-All (R) 6.2 4.0 2.2 12.26 .000*
Yes/No 5.9 6.1 -0.2 -1.06 .854
Yes/No (R) 5.7 6.1 -0.4 -1.61 .946
Q11: University Budget Adjustments (14) Check-All 4.8 2.9 1.9 10.67 .000*
Yes/No 4.8 4.4 0.4 1.56 .059
Q14: Cougar varsity Sports Fan (15)
Check-All 5.2 2.1 3.1 16.49 .000*
Yes/No 5.3 4.1 1.2 2.90 .002*
Yes/Neutral/No 4.0 3.6 0.4 1.05 .148
Yes/No/Neutral 4.3 4.0 0.3 0.70 .241
Q16: Food vendors on campus (9)
Check-All 6.0 3.9 2.1 9.46 .000*
Check-All (R) 5.9 3.9 2.0 10.37 .000*
Yes/No 4.4 5.1 -0.7 -2.43 .992
Yes/No (R) 4.9 5.3 -0.4 -1.58 .942
Q20: Possessions in Pullman (13)
Check-All 7.9 5.8 2.1 7.98 .000*
Check-All (R) 7.8 6.1 1.7 7.66 .000*
Yes/No 6.3 6.8 -0.5 1.53 .936
Yes/No (R) 6.5 7.0 -0.5 1.95 .974
† The number of response options offered for each question is displayed in parenthesis.
† † “(R)” denotes
treatments in which the options were presented in reverse order (inverted). * p ≤ .05
Time outliers were removed at 2 standard deviations above the mean.
20
Table 6: Primacy Effects in Check-All Format For Those Spending The Mean Amount of
Time or Below
Mean Number of Options Checked by Location: Top and Bottom Half of List
ORIGINAL
ORDER
REVERSE
ORDER t p
WEB EXPERIMENT #1
Q11: Resources Used at WSU (10)
†
Options A - E 3.81 T 3.68 B 1.46 .072
Options F - J 1.51 B 2.07 T 6.37 .000*
Q13: Cougar Varsity Sports Fan (15)
Options A - G 1.16 T 1.26 B 0.97 .833
Options I - O 0.38 B 0.60 T 2.49 .007*
WEB EXPERIMENT #2
Q6: Admittance Criteria (14)
Options A - G 2.05 T 1.82 B 2.52 .006*
Options H - N 2.10 B 2.15 T 0.34 .368
Q16: Food Vendors on Campus (9)
Options A - D 2.52 T 2.40 B 1.29 .098
Options F - I 1.15 B 1.32 T 1.85 .032*
Q20: Possessions in Pullman (13)
Options A - F 2.48 T 2.45 B 0.39 .348
Bottom H - M 2.59 B 2.88 T 2.30 .011*
Mean Number of Options Checked by Location: Top and Bottom Three Positions of List
WEB EXPERIMENT #1
Q11: Resources Used at WSU (10)
Options A - C 2.25 T 2.11 B 2.07 .019*
Options H - J 0.39 B 0.96 T 9.15 .000*
Q13: Cougar Varsity Sports Fan (15)
Options A - C 0.40 T 0.50 B -1.30 .903
Options M - O 0.23 B 0.37 T 2.38 .009*
WEB EXPERIMENT #2
Q6: Admittance Criteria (14)
Options A - C 1.91 T 1.68 B 2.72 .003*
Options L - N 0.51 B 0.68 T 2.15 .016*
Q16: Food Vendors on Campus (9)
Options A - C 1.97 T 1.76 B 2.52 .006*
Options G - I 0.61 B 0.72 T 1.67 .048*
Q20: Possessions in Pullman (13)
Options A - C 0.83 T 0.81 B 0.36 .359
Bottom K - M 1.15 B 1.30 T 2.01 .022*
† The number of response options offered for each question is displayed in parenthesis.
* p ≤ .05
21
Tab
le 7
: E
ffec
ts o
f ad
din
g a
“D
on
’t K
now
” C
ate
gory
to a
Fo
rced
-Ch
oic
e Q
ues
tion
Q3
: H
ere
are
so
me
wa
ys
that
the
WS
U P
ull
ma
n C
am
pu
s h
as
bee
n d
escr
ibed
. P
lea
se c
hec
k e
ach
des
crip
tio
n t
ha
t y
ou
th
ink
acc
ura
tely
des
crib
es t
his
cam
pu
s.
V
ER
SIO
N 2
V
ER
SIO
N 3
TE
ST
S O
F S
IGN
IFIC
AN
CE
YE
SE
S
TE
ST
S O
F S
IGN
IFIC
AN
CE
NO
S
Yes
N
o
Yes
N
o
DK
1
Dif
f.
X2
p
Dif
f.
X2
p
Saf
e C
am
pus
85
.7
5
.2
77
.5
11
.3
3.1
8
.2
9.5
5
.00
2**
-6.1
1
1.0
9
.00
1*
Int.
Sti
mu
lati
ng
67
.5
23
.1
66
.0
23
.2
2.6
1
.5
0.2
3
.63
2
-0.1
0
.00
.9
79
Peo
ple
Fri
end
ly
84
.8
6
.1
82
.7
8
.7
0.5
2
.1
0.6
5
.42
2
-2.6
2
.31
.1
29
Fac
ult
y I
nac
cess
ible
3
7.0
5
3.8
3
0.7
5
7.4
3
.5
6.3
3
.80
.0
51
-3
.6
1.1
6
.28
1
Buil
din
gs
Mo
der
n
66
.8
24
.2
65
.0
24
.8
1.9
1
.8
0.3
1
.57
5
-0.6
0
.04
.8
35
Cla
sses
to
o L
arge
32
.7
58
.3
27
.7
62
.6
1.4
5
.0
2.6
5
.10
4
-4.3
1
.72
.1
90
Eq
uip
. U
p t
o D
ate
74
.7
16
.1
68
.8
20
.3
2.8
5
.9
3.6
9
.05
5
-4.2
2
.56
.1
10
Dri
ver
s U
nfr
iend
ly
28
.3
62
.6
24
.8
65
.5
1.7
3
.5
1.3
1
.25
3
-2.9
0
.81
.3
69
Po
liti
call
y A
ctiv
e 4
5.3
4
3.9
3
8.1
4
5.2
8
.5
7.2
4
.67
.0
31
*
-1.3
0
.13
.7
20
Cam
pus
Co
nges
ted
2
6.7
6
3.2
2
4.3
6
5.2
2
.1
2.4
0
.62
.4
31
-2
.0
0.3
9
.53
5
Lac
ks
Div
ersi
ty
28
.3
62
.3
27
.4
63
.4
1.2
0
.9
0.0
7
.78
6
-1.1
0
.10
.7
55
Too
Sp
ort
s O
rien
ted
2
2.0
6
8.4
1
8.9
7
1.4
1
.2
3.1
1
.25
.2
64
-3
.0
0.9
3
.33
4
Mea
n
6.6
5.4
6.0
5.6
0
.3
4.0
-2
.7
N (
1,7
05
) 4
46
42
3
Dif
f. o
f m
ean
s
t =
4.8
6, p
= .
00
0
D
iff.
of
mea
ns
t =
-2
.38,
p =
.0
18
1 D
K s
tand
s fo
r d
on’t
kno
w.
* p
≤ .0
5
22
Tab
le 8
: E
ffec
ts o
f ad
din
g a
“N
eutr
al”
Cate
go
ry t
o a
Fo
rced
-Ch
oic
e Q
ues
tion
Q1
4:
Wh
ich
of
the
foll
ow
ing
Co
ug
ar
va
rsit
y s
po
rts
wo
uld
yo
u c
on
sid
er y
ou
rsel
f to
be
a f
an
of?
P
lea
se C
hec
k-a
ll t
ha
t a
pp
ly.
V
ER
SIO
N 2
V
ER
SIO
N 4
TE
ST
S O
F S
IGN
IFIC
AN
CE
YE
SE
S
TE
ST
S O
F S
IGN
IFIC
AN
CE
NO
S
Y
es
No
Y
es
No
N
eutr
al
Dif
f.
X2
p
Dif
f.
X2
p
Men
’s b
aseb
all
35
.4
52
.2
30
.9
36
.1
26
.0
4
.5
2.0
3
.15
4
16
.1
23
.43
.00
0*
Wo
men
’s b
asket
bal
l 2
3.8
6
2.8
1
9.2
4
4.9
2
8.7
4.6
2
.76
.0
96
1
7.9
2
8.5
3
.0
00
*
Men
’s b
asket
bal
l 4
3.7
4
3.3
4
0.0
2
9.3
2
3.7
3.7
1
.30
.2
55
1
4.0
1
8.6
4
.0
00
*
Wo
men
’s c
ross
co
untr
y
11
.2
74
.4
10
.4
57
.6
24
.4
0
.8
0.1
6
.69
1
16
.8
28
.22
.00
0*
Men
’s c
ross
co
untr
y
10
.1
75
.8
10
.8
56
.4
25
.3
-0.7
0
.13
.7
16
1
9.4
3
7.1
6
.0
00
*
Men
’s f
oo
tbal
l 7
7.4
1
1.7
7
7.0
1
0.4
5.9
0.4
0
.02
.8
93
1.3
0
.37
.54
4
Wo
men
’s g
olf
7.8
7
8.3
6.3
6
2.3
2
4.2
1.5
0
.79
.3
75
1
6.0
2
7.0
8
.0
00
*
Men
’s g
olf
9.2
7
6.7
9.0
5
9.4
2
4.2
0.2
0
.01
.9
33
1
7.3
3
0.6
4
.0
00
*
Wo
men
’s r
ow
ing
18
.2
67
.7
13
.5
54
.9
24
.6
4
.7
3.5
5
.06
0
12
.8
15
.49
.00
0*
Wo
men
’s s
occ
er
34
.3
52
.0
37
.0
35
.7
20
.1
-2.7
0
.71
.3
98
1
6.3
2
4.1
3
.0
00
*
Wo
men
’s s
wim
min
g
15
.7
71
.1
14
.7
53
.3
24
.8
1
.0
0.1
8
.67
1
17
.8
29
.96
.00
0*
Wo
men
’s t
ennis
1
4.1
7
2.0
1
2.6
5
4.9
2
5.3
1.5
0
.42
.5
16
1
7.1
2
8.0
9
.0
00
*
Wo
men
’s t
rack
and
fie
ld
22
.6
63
.0
24
.4
45
.4
23
.0
-1.8
0
.37
.5
42
1
7.6
2
7.8
4
.0
00
*
Men
’s t
rack
and
fie
ld
26
.2
59
.9
27
.3
42
.4
22
.8
-1.1
0
.13
.7
16
1
7.5
2
7.0
2
.0
00
*
Wo
men
’s v
oll
eyb
all
47
.5
40
.1
45
.6
30
.7
16
.5
1
.9
0.3
3
.56
3
9
.4
8.6
5
.0
03
*
Mea
n
4.5
1
0.1
4.1
7.2
3.6
1
.2
15
.2
N (
1,7
05
) 4
46
44
3
Dif
f. o
f M
ean
s
t =
1.5
4, p
= .
12
4
Dif
f. o
f M
ean
s
t =
8.4
1, p
= .
00
0
No
tes:
* p
≤ .
05
23
Table 9: Effects of More Active Category Labels in Forced-Choice Questions
Q13:
Which of the following Cougar varsity sports would you consider yourself to be a fan of?
YES/NO
FORMAT
FAN/NOT A
FAN FORMAT
SIGNIFICANCE TESTS
YES VS. FAN
Yes No Fan Not a Fan X2 p
Men’s baseball 32.8 61.8 37.1 57.5 1.46 .228
Women’s basketball 14.5 79.8 19.9 72.8 3.61 .057
Men’s basketball 35.6 60.1 37.9 55.6 0.40 .530
Women’s cross-country 7.7 86.6 9.2 82.0 0.57 .450
Men’s cross-country 8.3 85.5 10.4 81.5 0.93 .335
Men’s football 78.9 19.4 80.4 16.1 0.24 .626
Women’s golf 10.0 84.0 9.3 81.7 0.10 .748
Men’s golf 13.7 80.3 12.8 79.0 0.12 .731
Women’s rowing 17.1 77.2 16.9 74.4 0.01 .943
Women’s soccer 35.0 60.7 34.6 58.0 0.02 .902
Women’s swimming 11.4 83.2 15.5 75.5 2.63 .105
Women’s tennis 10.5 83.5 15.3 76.3 3.54 .060
Women’s track and field 19.9 74.6 22.9 69.8 0.92 .336
Men’s track and field 22.5 72.4 27.2 65.7 2.16 .142
Women’s volleyball 39.9 55.8 43.6 49.3 1.02 .314
Mean # Marked Affirmatively 3.58 3.93 t = 1.32 .186
N 351 367
Q16: What types of student groups, if any, have you participated in while a student at WSU?
YES/NO
FORMAT
PARTICIPATED/
NOT PART.
FORMAT
SIGNIFICANCE TEST
YES VS. PART.
Yes No Part. Not Part. X2 p
Academic Orgs. 39.3 58.4 39.8 56.7 0.02 .898
Community Service Orgs. 38.5 59.5 36.2 58.9 0.38 .538
Entertainment and Social Orgs. 38.2 59.3 38.4 57.8 0.00 .947
Ethnic and Cultural Orgs. 15.4 81.5 17.4 77.7 0.55 .458
Fraternity or Sorority 25.1 73.2 18.5 76.6 4.52 .034*
Intramural and Rec. Sports Clubs 51.3 47.3 44.4 51.5 3.39 .066
Performing and Fine Arts Groups 11.4 85.5 12.0 82.3 0.06 .805
Religious Orgs. 19.4 77.5 17.2 77.4 0.59 .444
Student Gov. and Political Orgs. 14.2 82.9 12.5 82.3 0.45 .501
Women’s and Sexuality Orgs. 4.8 91.7 4.1 90.2 0.24 .624
Mean 2.58 2.41 t = -1.14 .255
N 351 367
* p ≤ .05
24
Table 10: Treatment of the Forced-Choice Format as a Check-All-That-Apply Format
SIGNIFICANCE TESTS ORIG.
ORDER
REV.
ORDER X2 p
QUESTION
TYPE
WEB EXPERIMENT # 1
Q11: Resources used at WSU (10)
† B
1
Used/Not Used 10.9 3.7 13.59 .000*
Q13: Cougar varsity sports fan (15) B
Fan/Not a Fan 6.5 -- -- --
Yes/No 4.3 -- -- --
Q16: Student Group Participation (11) B
Participated/Not Participated 3.0 -- -- --
Yes/No 2.0 -- -- --
WEB EXPERIMENT # 2
Q3: Describe WSU Pullman Campus (12) O
2
Yes/No 0.0 -- -- --
Yes/No/Don’t Know 0.0 -- -- --
No/Yes 0.0 -- -- --
Q6: Admittance Criteria (14) O
Yes/No 0.0 0.0 -- --
Q11: University Budget Adjustments (14) O
Yes/No 6.1 -- -- --
Q14: Cougar varsity sports fan (15) B
Yes/No 3.4 -- -- --
Yes/Neutral/No 0.5 -- -- --
Yes/No/Neutral 0.5 -- -- --
Q16: Food vendors on campus (9) B
Yes/No 0.7 1.6 1.44 .230
Q20: Possessions in Pullman (13) B
Yes/No 0.2 0.0 1.05 .306
Q24: WSU descriptors (10/5)
Yes/No (Long List) 4.8 1.6 7.42 .006* O
Yes/No (Short List) 2.4 0.9 2.90 .088
PAPER EXPERIMENT
Q5: Cougar varsity sports fan (15) B
Yes/No 11.3 -- -- --
† The number of response options offered for each question is displayed in parenthesis.
* p ≤ .05 1 “B” stands for behavior/fact based question
2 “O” stands for opinion based question
25
WEB EXPERIMENT # 1 (SPRING 2003) Question #11 Question #13 Question #16
Ver
sio
n 1
Ver
sio
n 2
Ver
sio
n 3
Ver
sio
n 4
26
WEB EXPERIMENT # 2 (FALL 2003) Question #3 Question #6 Question #11
Ver
sio
n 1
Ver
sio
n 2
Ver
sio
n 3
Ver
sio
n 4
27
WEB EXPERIMENT # 2 (FALL 2003) Question #14 Question #16 Question #20
Ver
sio
n 1
Ver
sio
n 2
Ver
sio
n 3
Ver
sio
n 4
28
WEB EXPERIMENT # 2 (FALL 2003) PAPER SURVEY (SPRING 2001)
Question #24 Question #5
Ver
sio
n 1
Ver
sio
n 2
Ver
sio
n 1
Ver
sio
n 3
Ver
sio
n 4
Ver
sio
n 2
29
REFERENCES
Best, Samuel J. and Brian Krueger. In press. Internet Data Collection. Sage Publications.
Christian, Leah Melani. 2003. “The Influence of Visual Layout on Scalar Questions in Web
Surveys.” Unpublished Master’s Thesis, Washington State University.
Christian, Leah Melani and Don A. Dillman. 2004. “The Influence of Graphical and Symbolic
Language Manipulations on Responses to Self-Administered Questions.” Public Opinion
Quarterly. 68:58-81.
Dillman, Don A. 2000. Mail and Internet Surveys: The Tailored Design Method. 2nd
ed. New
York, NY: John Wiley and Sons, Inc.
Heerwegh, D. & G. Loosveldt. 2002. “Describing Response Behavior in Websurveys Using
Client Side Paradata.” Paper Presented at the International Workshop on Web Surveys,
Mannheim, Germany, October 2002.
Krosnick, Jon A. 1991. “Response Strategies for Coping with the Cognitive Demands of Attitude
Measures in Surveys.” Applied Cognitive Psychology. 5: 213-236.
Krosnick, Jon A. 1999. “Survey Research.” Annual Review of Psychology. 50: 537-567.
Krosnick, Jon A. and D. F. Alwin. 1987. “An Evaluation of a Cognitive Theory of Response-
Order Effects in Survey Measurement.” Public Opinion Quarterly. 51: 201-219.
Rasinski, Kenneth A., David Mingay, and Norman M. Bradburn. 1994. “Do Respondents Really
‘Mark All That Apply’ on Self-Administered Questions?” Public Opinion Quarterly. 58:
400-408.
Schuman, Howard and Stanley Presser. 1981. Questions and Answers in Attitude Surveys
Experiments on Question Form, Wording, and Context. New York, NY: Academic Press.
Stern, Michael J, Don A. Dillman, Jolene D. Smyth, and Leah Melani Christian. 2004. “The
Uses of Paradata for Evaluating Alternative Versions of Web Survey Questions.” Paper
presented at the 57th
annual conference of the World Association for Public Opinion
Research. Phoenix, Arizona. May 11-13, 2004
Sudman, Seymour and Norman M. Bradburn. 1982. Asking Questions. San Francisco, CA:
Jossey-Bass.
Sudman, Seymour, Norman M. Bradburn, and Norbert Schwarz. 1996. Thinking About Answers:
The Application of Cognitive Processes to Survey Methodology. San Francisco, CA: Jossey-
Bass.
top related