mixed-mode surveyspeople.uncw.edu/maumem/soc500/ch6.pdfe-mail and web surveys gain favor with...

C H A P T E R

Mixed-Mode Surveys

MODES of data collection often produce different results. For example, a self-administered survey of respondents resulted in 15% ranking their health in the top category of "very good" on a four-point scale; a resur- vey of those same respondents by personal interview a short time later resulted in 27% answering "very good," an increase of nearly 12 percentage

(Biemer, 1997). These data are reminiscent of data collected three decades earlier by Hochstim (1967) that found 44% of respondents to a personal interview reported their health as excellent (the top choice in that survey), compared to only 37% on the telephone, and 30% by mail. Respondents consistently offer more positive health assessments to interviewers than to paper questionnaires.

In another survey this one using interactive voice response (IVR) technol- ogy (see Chapter 11) that required people to key in their responses to a series of customer satisfaction questions using the numbers on a touch-tone phone, respondents consistently gave ratings that were significantly higher than those given in a mail questionnaire (Srinivasan and Hanway 1999). A recent survey by Web and telephone also obtained more positive results by phone as well, although the reasons for this were unclear (Kottler, 1998).

Sometimes responses to different survey modes are not compatible and little can be done to change that. It may therefore seem desirable to avoid con- ducting surveys by more than one survey mode. However, in other situations it is impossible to avoid using multiple modes to conduct a particular survey.

TWO quite different requests for survey assistance illustrate why mixed- mode surveys cannot always be avoided. In one instance, I was asked to conduct an e-mail survey of university faculty to obtain quick and inexpensive responses on an important issue facing the university community. However, it was known that some faculty were infrequent or even nonusers of e-mail. The tailored mixed-mode solution was to collect information from most re-

218 TAILORING TO THE SURVEY SITUATION Mixed-Mode Surveys 219

spondents by e-mail and use paper questionnaires to collect infor the remainder (Schaefer and Dillman, 1998).

The second instance involved an urgent request fro agency asked that a survey be done to collect information fro Typical Objective Consequence grass burning, a very controversial issue that had been receiv Reduce cost and Measurement favorable publicity. The survey had to be completed in only three wee nonresponse differences The agency reported that because of the issue's sensitivity some farme had already indicated they would not respond to a telephone survey unless llection of panel data from same Reduce cost and Measurement they could first see the questions in writing. The proposed solution was to pondent at later time nonresponse differences send a questionnaire by U.S. Postal Service priority until a fifth contact) with a request that it be comple ollection of different data from the Improve measurement None apparent ent wait for us to call and collect the information over the short while later, all nonrespondents were telephoned wit complete an interview. A response rate of 74% was achieved, with two- thirds of the responses coming by mail and the remaining on Convenience and reduce Measurement phone (Moore, 1998). Case management software shared by the ma cost differences

telephone data collection units at my university's survey center m easy to keep track of daily responses in order to plan for su Improve coverage None apparent ings and telephone calls. In past years, the lack of such software would have and reduce nonresponse

made the survey center reluctant to contact the same respondents by two methods at once.

Mixed-mode surveys provide an opportunity to compensate for the weak- V E SITUATIONS FOR USE OF MIXED-MODE SURVEYS nesses of each method. For example, one of the potential concerns associated procedures can be mixed with other types of self-administered or inter- with self-administered surveys, the negative effects of which are no methods in different ways. How they are mixed raises quite different documented, is the ability of potential respondents to s deciding whether to respond. Also, the fact that questionna been demonstrated to have a major effect on mail back response rates p

LLECTION OF THE SAME DATA FROM DIFFERENT MEMBERS OF A SAMPLE vides reason to be concerned that nonresponse error may be substantial some mail surveys. In contrast, respondents to telephone surveys often know taining questionnaire responses from some members of a sample by one little about the survey topic until they are in the midst of answering questions (Groves and Couper, 1998). Evidence also exists that people on type of mixed-mode survey. This use of different survey modes is erences, some preferring face-to-face, others preferring tel y justified by a desire to cut costs through use of the least expensive others exhibiting a strong preference for self-administered formats (Gr and Kahn, 1979).

However, mixing modes raises many difficult issues, bility that people may give different answers to each mo edure. An example of such a survey was a large-scale pilot of the National introduce the necessity of linking questionnaire design and imple procedures for self-administered surveys with procedures for methods, and that requires tailoring one set of procedures to fit wi ennial Census for a national sample of people who reportedly have four- Thus, I begin Part I1 of this book with a discussion of the challenges of ing self-administered procedures to the special situation of relying on more survey modes to conduct a survey project.


either had not responded or for whom the 1990 mailing address was no longe An example of such a study is one of medical patients who went to a doc- current. Finally an effort was made to personally contact and interview the r or's office because they were experiencing pain. They completed an initial maining nonrespondents. Use of this procedure in 1992 resulted in mail r -administered questionnaire in the office and were followed up at three- sponse rates ranging from 63 to 79% for various treatment groups. About 30 six-month intervals. Those who did not respond to the mail follow-up of the nonrespondents assigned to the telephone mode were then interviewe re then surveyed by telephone in order to improve response rates. by that method. Follow-up of the remaining nomespondents by face-to-fa The mixing of modes in panel studies is a significant challenge in that the contacts resulted in half of them being interviewed. An overall response ra sual intent is to measure change between time one and time two. Even tiny of 88% was obtained (Mooney Giesbrecht, and Shettle, 1993). hanges in measurement qualities of instruments can influence conclusions

It is likely that the prevalence of mixed-mode surveys of this type will sigruficantly. Thus, this type of mixed-mode survey raises particularly im- crease as people's abilities to screen out telephone interviews, currently t portant questions about the best procedures for formatting questions so the nation's dominant survey mode, increase (Dillman, 1999). Also, as the use sual stimulus of a written questionnaire will be equivalent to the aural stim- e-mail and Web surveys gain favor with surveyors, a formidable barrier lus of an interview. their use is the fact that many people do not have access to the Internet. Mixed mode surveys may be the only alternative for immediately gaining access t

OLLECTION OF DIFFERENT DATA FROM THE SAME RESPONDENTS DURING A all members of many survey populations.

INGLE DATA COLLECTION PERIOD An important concern raised by this type of mixed-mode survey is whethe people answer questions in the same way for each mode. It is possible tha Sometimes respondents who have just completed an interview are asked to introduction of additional modes, while increasing response rates, will complete a self-administered questionnaire that contains additional ques- increase measurement differences. Other issues include how to co tions. Typically this happens when very sensitive questions are being asked, methods in the most cost-effective manner and how to design specific a such as questions regarding sexual behaviors or preferences. Concern exists in the first mode that help the second mode to be successful. that the respondent may be reluctant to divulge that information to the inter-

especially if someone else in the household is present. Other examples en diaries of food consumption must be kept over a period of several

COLLECTION OF PANEL DATA FROM THE SAME RESPONDENTS AT A or when records must be consulted in order to provide correct detail, as LATER TIME y business surveys. Surveys have also been done that required that data Frequently the choice of method for an initial survey is determined by alned from mail questionnaires be reviewed to see if the respondents met sample frame. Sometimes mailing addresses are available but teleph ain criteria for an immediate follow-up interview by telephone. numbers are not, as might be the case for customer address lists. In other ca This type of mixed-mode survey raises different issues than the second adequate samples can be accessed only by area probability samples e outlined above. The self-administered form is being used because of par- households or random telephone digits. However, during the initial data c s it has, such as an ability to decrease the social desirability of lection, contact information (telephone numbers, fax numbers, e-mail a to be offered in an interview, especially if other household dresses, or regular mailing addresses) may be obtained that will allow 0th mbers are Listening, and to avoid item nonresponse. In this case, possible modes to be used for the follow-up survey. Doing so will allow the seco urement differences between what is obtained by one mode versus an- mode to be used in a way that cuts total costs and/or improves response rat er are considered an advantage rather than a major source of concern, as

In addition, the survey situation may change dramatically between scribed earlier for panel surveys. tial and repeat data collections. For example, personal interviews used to survey high school or college students in a baseline study but whe

OLLECTION OF COMPARISON DATA FROM DIFFERENT POPULATIONS is time to conduct follow-up surveys those individuals are geographically persed. Thus, the only economical survey alternative may be to switch to ma recent years, the ease of manipulating large data sets has led to yet another or telephone for the later follow-ups. In panel studies each response is partic e of mixed-mode survey situation, whereby data sets are collected from ularly precious for measuring change. Thus, to the extent that providing nt populations by different modes for purposes of comparison with ternative modes of responding will improve panel response rates, a decisio other. For example, in the social sciences the developing and testing of to switch modes may be made. es across populations is quite common. When belief or opinion scales,


which by their nature tend to rely on vague quantifiers, are used on various modes (Dillman, 1999). We live in a society with high mobility and complex populations and the results compared, it is important that they have the same living arrangements. Some people can be contacted most easily by mail, oth- measurement qualities. Conclusions from such comparisons may be ques- ers by telephone, and still others are most accessible by personal visits. tioned if people respond differently to the various methods. In addition, information age technologies have made it easier for people

Frequently such data are collected by different investigators using differ- to screen themselves from unwanted or unauthorized intrusions. Gated ent modes at different times, and making comparisons is not their objectiv communities, locked apartment buildings, unlisted telephone numbers, In other instances, characteristics of samples dictate that some people canno telephone answering machines, complete reliance on cellular rather than con- be contacted for interviews, while for others mailing addresses are not avail- ventional telephones, and now electronic mail (sometimes with multiple ad- able. Because this type of mixed-mode does not usually involve the mixing ses for one individual) are forming a protective barrier around methods or questionnaire responses for a given population, it may not b viduals. This barrier to direct contact is, in a sense, being pitted against significant design concern for most surveyors. r and better listing services. These services combine information from

hone directories, credit card lists, and mail order requests to facilitate lo- ting people. The time when population access and response could usually

USE ONE MODE TO PROMPT COMPLETION BY ANOTHER MODE e achieved by one mode is past. For increasing numbers of surveys, reliance

In Chapter 4 I discussed the potential value of making a final contact by tele- n only one mode does not provide assurance of reaching or eliciting re- phone to encourage response to a self-administered questionnaire. Introd onses from most of the sampled individuals. ing a new mode at this stage of the data collection may allow information Also in the past, adapting a mail questionnaire to an interview format was be collected that will improve coverage information for a study; for examp time-consuming task which could only be justified for very large studies. allowing ineligible sample units to be dropped from the survey. Its use is s that specialized in one type of surveying, whether face-to-face inter- likely to improve response rates to the other mode sigruficantly. By intro ing, telephone interviewing, or mail surveys, tended to word questions ing a second Survey mode in this way whether fax, e-mail, Web, voice te fferently and use questionnaire instruction procedures that contrasted phone, or touch-tone data entry it may be possible to avoid the poten atly with those most likely to be used for other methods. The ability to measurement differences that must be considered for the other mixed-mo ove word processing files and shape them for other uses now makes mixed- possibilities outlined above, but also benefit the coverage and response qual- ode studies much easier to accomplish. Yet another characteristic of the cur- ities of the survey effort. rent information age is the ability to do better cost modeling so that optimal

dy designs can be developed. Each method is used in a way and to the ex- that it is most cost-effective for purposes of an overall study (Groves,

SOME CAUSES AND CONSEQUENCES OF MIXED-MODE DESIGNS While quality surveying is now being threatened by inadequacies of indi-

The trend towards greater use of mixed-mode surveys has paradoxical co ual survey modes, the ability of survey researchers to manage projects us- sequences. On the one hand, the trend is driven by a desire for better respon g multiple modes has increased. In the recent past, when a large-scale Increasingly many survey sponsors have found it difficult to obtain high s being implemented in large organizations with separate mail and sponse rates by limiting their designs to one mode. On the other hand, data collection units, it was unthinkable that attempts to get re- combining of data collected by different modes raises questions abou onses to more than one mode would be made simultaneously. Assigning a whether people give the same answer to each mode. ple to the mail survey unit and the telephone interviewing unit at the

The summary of mixed-mode surveys shown in Figure 6.1 described e time meant that the only easy way to determine whether a response usual objective for mixing modes and the potential error consequences. Dis dbeen received was for each to seek records from the other, which were of- played in this way it is apparent that three sources of survey error-no n not immediately available. It was therefore difficult to prevent sending response, measurement, and coverageand costs all play a major role placement questionnaires to some people who had just responded by making the decision on whether to conduct such a survey. The remainder one or vice versa. Management software now makes it possible to this chapter provides further elucidation of these error consequences te activities across operations routinely. how to reduce their potential effects. eparture from the one-size-fits-all age of survey design has also re-

A number of factors are substantially influencing this trend of mix ed that simple pronouncements of mode advantages and disadvantages

224 TAILORING TO THE SURVEY SITUATION Mixed-Mode Suweys 225

are not always true. For example, a traditional advantage of the telephone was speed and a disadvantage of mail was slowness. However, as described for the grass-burning survey of farmers at the beginning of the chapter, mixed- mode designs can sometimes incorporate mail as a way of speeding up re-

Mechanism that Potential mode sponse. influences response difference In addition, the ability to quickly complete a telephone survey is depend- behavior

ent upon the number of interviewers and interviewing stations. For large- v v scale surveys, beginning with a mail component may allow one to majority of responses far more quickly than they could be obtained by an ganization's telephone capability. Switching to telephone at the time that m responses begin to slow down can result in bringing the study to a spe

response to social norms conclusion. An example of using mail to achieve speed (as well as lower is the Decennial Census, which in 1990 achieved a response rate o mail, with nearly 90% of those returns being obtained within a wee I sus Day (April 1st) and before sending personal enumerators to the re ing households.

effects: Norm of even-handedness

WHY PEOPLE MAY ANSWER SELF-ADMINISTERED AND INTERVIEW QUESTIONNAIRES DIFFERENTLY f i

A number of studies have noted differences in the answers provided to te phone, face-to-face, and mail questionnaires, making it an issue that must considered in mixed-mode situations (de Leeuw, 1992; Schwarz, Hippler, Noelle-Neumann, 1992; Dillman, Sangster, Tarnai, and Rockwood, 1996).

I only have differences in distributions across categories been observed, different analytical results have been obtained, for example, in the const tion of structural models (de Leeuw, Mellenbergh, and Hox, 1996). The m basic cause of mode differences is the tendency for people to constr tions differently for the two types of questionnaires. For example, phone surveys, surveyors typically do not offer "don't know" option prehended. Third, the interviewer controls delivery of the stimulus, re- such an answer is volunteered the interviewer is instructed to record it. Ins ing, for example, that the respondent listen to every word, whereas a mail administered surveys the option must either be offered explicitly or not at In general, a technique called unimode construction, the topic of the next s tion, can reduce these difficulties. ence of the lack of interview control for self-administered surveys is

There are three reasons for expecting that results from self-administe er itemnonresponse rates (Dillman, 1978; de Leeuw, 1992). The visual de- and interview surveys sometimes differ (Figure 6.2). First, interviews invo principles for questionnaire design discussed in Chapter 3 are aimed in interaction with another person, which leads to respondents taking into ing this differential. count social norms and what they think are culturally acceptable answe ut expanding amount of research has addressed the extent to People tend to be reluctant to tell something to an interviewer that counter to the beliefs or attitudes they believe the interviewer holds. Seco two quite different senses are involved; whereas the self-administered stim lus is visual, the interview stimulus is aural, resulting in different process through which the meaning of a question and the response alternat Its of this research are far from consistent, and several different perspec-

effects: Addition and subtraction

effects: Primacy versus recency

tives exist on the extent to which certain types of differences occur and why Nonetheless, there is sufficient basis for offering certain guidelines on when response differences might present a problem in mixed-mode surveys.

Two main human factors seem to underlie the existence of mode differences, one of which tends to be normative or sociological in nature and the other cognitive or psychological in nature, as shown in Figure 6.2. The normative factor is group-based, that is, when cultural norms are in some way evoked differently in the interview than in the self-administered situation, they lead to culturally-constrained responses. The second factor is cognitive processing by individuals and, in particular, whether people receive the same stimulus from the two modes. Understanding these possible differences provides a basis for understanding the extent to which some differences may be inevitable and others might be overcome by such steps as unimode questionnaire construction. It is particularly important to be aware of the possibility of four potential sources of response differences. They include: social desirability acquiescence, question order effects (including the norm of even-handedness), and primacy/recency effects.

Social Desirability

Several years ago I was asked by a large regional hospital to look at results from two customer satisfaction surveys of similar samples of former patients. One set of data had been obtained by a mail survey and the other by telephone interviews. Results from the mail questionnaires showed much lower satisfaction ratings. The request to examine the results came with the casually offered comment that the hospital had tentatively decided to discontinue the mail survey because of the belief that only unhappy patients seemed to be re sponding, but the sponsor wondered why that was the case. When I asked more about the procedures I was informed that in order to cut costs the tele phone sample had been interviewed by nursing staff on the evening shift, many of whom had cared for the former patients they were asked to call, whereas the other was truly an anonymous survey with no identification number on the retum questionnaire. Thus, I did not find the results to be sur- prising. It seemed likely that patients would offer socially desirable answers by telephone to the nursing staff, that is, answers that reflected what the respondent thought the person who had provided the service hoped to hear. My advice was to at least change who did the interviews, that is, to contract to an outside firm. Even then the elimination of social desirability could not be guaranteed.

Because of the interaction with another person, interview surveys are more likely to produce socially desirable answers for questions about poten- tially embarrassing behavior, such as drug use or cheating on one's spouse.

Mixed-Mode Surveys 227

wever, socially desirable responses are also likely to be offered for drink- a type that is somewhat unacceptable. For example, one

on, "How often do you drive a car after drinking alco- Frequently Occasionally Seldom, Never, or Don't Know."

ereas 52% of the mail survey respondents responded "Never," signifi- e telephone respondents (63%) said "Never" (Dillman and

Even very ordinary questions that seem, on the surface, to have little social desirability consistently exhibit this effect. For example, consider, "How would you describe your current health? Excellent, Good, Fair, or Poor." As

ing of this chapter, this question consistently produces respondents who choose the top category in self-admin-

meets another on the street and offers the conventional erican greeting, "How are you?," people typically respond, "fine." A less

calls for an explanation. Thus, social desirability oper- elow what one thinks of as anti-social behavior, such

ted something from a store." The differential influence interview versus self-administered surveys is the most

ictable mode difference, a finding supported by much research (DeMaio, ; Aquilino, 1994; de Leeuw and van der Zouwen, 1988; Dillman, Sangster, ai, and Rockwood, 1996), and one should routinely expect such differ-

ces to occur in mixed-mode surveys.

iescence is a culturally-based tendency to agree with others. In most cul- e than to disagree with someone when interacting with

ple, a classic interview experiment by Schuman and Presser 1) showed that 60% of one national sample of respondents agreed with tatement, "Individuals are more to blame than social conditions for crime

in this country." However, 57% of a control group agreed with exact reverse of this statement, "Social conditions are more to blame than ividuals . . . " Changing the question structure to a forced choice between

e two items produced an intermediate response. An experiment by Dillman and Tamai (1991) found that in seven compar-

ons assessing opinions on whether proposals for increasing seat belt use would work, telephone respondents were significantly more likely than a comparable sample of mail respondents to agree. Differences ranged from five to 23 percentage points. A similar pattern was observed by Jordan, Mar- cus, and Reeder (1980). However, a meta-analysis by de Leeuw (1992) failed to detect any differences, revealing that the literature is not entirely consistent n this issue. Nonetheless, surveyors should watch carefully for differences etween modes for agree/disagree types of questions.


Norm of Even-Handedness And Other Question Order Effects phone, the reading of response categories by ovide sufficient time between categories, likely re-

A third potential contributor to differences between self-administered and the earlier categories not being remembered. Consequently when terview questionnaires is that people may adjust their answer tot e comes to respond, the last answers are more likely to be remembered two similar questions to take into account their answer to the previous ques 7). It is further argued that in self-administered sur- tion. This may happen because of previewing questions to co s earlier in the list undergo greater cognitive going back and changing answers. For example, Bishop et a o compare and contrast later items. If the initial op- separate surveys of U.S. and German students: "Do you think answer, the prediction is that it is more likely to be ican government should be allowed to set limits on how much Japanese dustry can sell in the United States (Germany)?" It was immediately follow vidence for the existence of pure primacy/recency effects is decidedly by this question, "Do you think that the Japanese government should be ed. Although recency effects have been observed in telephone surveys lowed to set limits on how much American (German) Moore, 1997), these same authors have also Japan?" Not only did asking these questions in this order r y few full mail/telephone comparisons have been agreement than when asked in reverse order by telephone, bu nalysis of 82 experiments in 12 separate surveys eliminated in the self-administered version. The explanation offered an, Brown et al., 1995) found that for both authors is that asking the questions in this order evokes a sense of fai y and recency effects were both as likely to occur when answering the second question, whereas respondents one method or the other. Several complete reversals of expectation, that tered questionnaires can look ahead and thus take into account the seco recency in mail, also occurred. We conclude that question when answering the earlier one, or even change their answer aft sion of these effects in the literature, the occur- wards. This explanation invokes both cognitive and normative consideratio elf-administered surveys and recency effects in as explanations. phone surveys are far from predictable and further experimentation

A replication of this test on a different topic in the United ster (1993) found that although the norm of even-handedness was evoked, effects for mail and telephone were about the different results is that the first evaluation e LICATIONS FOR MIXED-MODE SURVEYS

differences in a classroom setting where students might approach answer fact that interviewer/self-administered survey differences are sometimes the questions more as a test for which it is normal to examine question onsistent does not mean that such differences can safely be ignored. A sequences before answering them and to change answers. In contrast, m s of three studies helps explain why (Figure 6.3). In a general public survey questionnaires for the Sangster (1993) study were sent to people's homes. ason (1984), it was found that a sample of mail question- this setting, respondents seemed more likely to go straight thro respondents was significantly less likely than either telephone or face- without looking back, and not worry too much about consistency e respondents to choose "not a problem," the first offered category in search, some of which was discussed extensively in Chapter 3, has reve ponse to a question about each of nine community issues. ("For each of the existence of question order effects that are not normatively based, ether you believe it is Not a Problem, a Small which mail/telephone comparisons revealed no differences. In these Problem, or you Don't Know on this issue.") Respondents stances people tend to allow the first of two answers to carry over cognitiv d face-to-face surveys were more likely to choose the "not from the first to the second answer. for all 11 items, with individual differences rangmg from

centage points. A similar tendency to choose the most PrimacyJRecmcy Effects ed for 15 other items about community and

A fourth potential source of mode differences is the tendency to choose t first offered answer category (called a primacy effect) more often in mail sur reported similar results for three questions asking

veys, and the last offered category (called a recency effect) in telephone s operty not being kept up, and (3) crime were

veys. The explanation for this potential effect differs from the nor es, or never a problem" for respondents. Responses offered for the preceding effects and is based only on cognitive conside from a comparable sample of interviewed respon-

230 TAILORING TO THE SURVEY SITUATION

Figure 6.3 Differences between percent of self-administered (mail) and interview (telephone or face-to-face) respondents selecting most positive response categories in three surveys.

Average Percent Range of Differences "Not aProblem" ~nterview-self-~dministered

Dillman and Mason, 1984 - - Interview Mail

Nine community issues from two-state general public survey, 47.9% 3 1.9% 6.7-18.8 telephone vs. mail

Nine community issues from same two-state general public survey, face-to-face vs. mail 43.8% 31.9% 4.2-18.7

Tarnai and Dillman, 1992

Five community issues from student survey; self-administered 37.8% 13.0% 12.638.1 vs. telephone (aural only)

Five community issues from student survey; mail vs. telephone while viewing questionnaire 31.3% 13.0% 4.3-20.0 (aural + visual)

Question: "For each issue, please tell me whether you believe it is Not a Problem, a Smll Problem, a Medium Problem, a Serious Problem, or if you Don't Know."

Average Percent Range of Differences "Not a Problem" Interview-Self- Administered

Krysan, Schuman, Scott, and - Interview -- Mail Beattv. 1994

Three issues from metropolitan area general public survey; mail 61.3% 48.0% vs. face-to-face

Question: "Thinking about your own neighborhood, please indicate whether the following are Always a Problem, Ojien a Problem, Sometimes a Problem, or Never a Problem in your neighborhood."

dents was seven to 17 percentage points higher than for respondents to the self-administered questionnaires (Figure 6.3). The fact that the category order for this study was the reverse of that used in the above-mentioned study sug- gests that something other than primacy/recency explanations are necessary for explaining these differences.

In an effort to decipher potential reasons for differences observed in the Dillman and Mason (1984) survey a near replication of five items from this survey was done for a college student sample. The differences found in this

Mixed-Mode Surveys 231

study were even larger, with use of the "not a problem" differences ranging from 13 to 38 percentage points, with telephone averaging 25 points higher

the mail (Tamai and Dillman, 1992). Two other conditions were exam- in an attempt to determine whether these results were due to the

n (aural versus visual), or perhaps other aspects of the differences between telephone and mail. One group was interviewed over the telephone without being able to see the questions. The second group was asked to open an envelope containing the written questionnaire and follow the questionnaire as they responded to the telephone interview. The group interviewed with the questionnaire in front of them averaged seven percentage points higher than the self-administered group for the "not a problemff answer, or 18 percentage points less than the telephone-only group. It was con- cluded that the visual/aural differences in communication were mostly but

ifferences. Concerning all of these tests, it may have exhibited a low threshold of social desir-

sted in their neighborhood or community and ent in a positive light when interacting

r). If so, this might explain the differences be- these findings and those of Krysan et al. (1994), which the cognitive-

cannot explain. Research has not yet ns of the normative and cognitive influences

f potential mode differences leads to the following con- rences between telephone and mail responses have fre-

ently been o b s e ~ e d and are sometimes substantial. Second, the most the literature is that social desirability differences af-

ss than interview respondents. Third, ability, the effects observed in the literature

for items that use vague quantifiers, which require into which category they fit. The differences between never a problem versus a small problem, are some-

difficult for respondents to reliably discern. Fourth, although one d be on the lookout for other types of mode differences, including ac-

en-handedness, and primacy/recency effects, their rrence is far less certain than for social desirability. Finally the data pre- ed by Tarnai and Dillman (1992) raise the possibility that cultural (or

itive considerations may interact to produce re- se differences.

such effects, our ability to pre- s somewhat limited. Nonethe-

ss, enough differences have been observed that it is prudent to tailor the x interview and self-administered methods in ways

at will reduce the chances of responses differing by survey mode.


INTERNET AND INTERACTIVE VOICE RESPONSE SURVEYS Figure 6.4 Changing mail and telephone questions into unimode format.

The newness of Internet (e-mail and Web) and interactive voice response su Traditional mail format:

veys means that virtually no research has addressed the potential mode di ferences that might be associated with these methods. All three method 27. To what extent do you favor or oppose a policy allowing free trade with all

countries of the world, no matter what their human rights record? self-administered, so a priority in research is to determine the extent tow these methods are subject to social desirability influences, and the extent Strongly favor

which they may mirror paper self-administered surveys or even produce le Somewhat favor

social desirability than the latter, as suggested for the completion of compu Somewhat oppose

surveys by Turner et al. (1998). As I shall suggest in Chapter 11, these te Strongly oppose

nologies may lead surveyors to pose questions as stimuli that differ sigru cantly from the way both mail and voice telephone surveys are done, th raising the possibility that a new set of mode considerations will need to b evaluated. 27. To what extent do you favor or oppose a policy allowing free trade with all

countries of the world, no matter what their human rights record? Would you say that you Strongly favor, Somewhat favor, Somewhat oppose or Strongly oppose this policy?

UNIMODE DESIGN AS A SOLUTION FOR CERTAIN MODE DIFFERENCES Strongly favor

Somewhat favor Unimode construction is the writing and presenting of questions to re Somewhat oppose dents in a way that assures receipt by respondents of a common mental Strongly oppose ulus, regardless of survey mode. In Chapters 3 and 4 I described ques No opinion

writing practices, the aim of which was to get the most accurate and camp Refused

answers possible for self-administered questionnaires. My emphasis on us visual principles for questionnaire design was intended to assure that spondents to self-administered questionnaires will receive the same st as that typically received by interview respondents. Visual principl 27. To what extent do you favor or oppose a policy allowing free trade with all

the foundation of unirnode design. Manipulations of font size, brigh countries of the world, no matter what their human rights record? DO YOU

color, location, shape, and figure/ground relationships are coordinated Stron%~ favor, Somewhat favor, Somewhat oppose, Strongly oppose, or have No opinion on this policy?

der to define a common navigational path through the questionnaire items all respondents, further encouraging provision of a common stimulus. Strongly favor

use of visual layout and design also directly attacks the problem of item no Somewhat favor

response, which is associated with paper self-administered questionnaire Somewhat oppose

addition, I advocate the use of cognitive design and more detailed o ended questions (Chapter 2) to address a second common deficie self-administered questionnaires, less complete responses to open questions. The question and questionnaire design principles that follo n the categories in the stem of the question and the listed choices. In ad- ther extend these efforts to obtain equivalent data across survey modes. on, "no opinion" categories are often omitted because of concern that too

people will use them. In contrast, the telephone version is likely to have Principle 6.1: Make all response options the same across modes and incorporate e categories included in the stem of the question in order to facilitate read- them into the stem of the survey question. ategories in the same way to every respondent.

Figure 6.4 presents mail and telephone versions of a survey question r telephone surveys, no opinion categories are seldom mentioned by the typical construction methods for each mode. Mail or self-administere iewer but are listed on the interviewer's screen for use if a respondent vey questions are typically written in a way that minimizes redundan ts on responding in that way. Consequently some people use them.


Therefore, telephone surveys produce "no opinion" responses, while resp ially when they have already read it many times, quickly encourages a sur- dents to the self-administered questionnaire who wish to respond either h sponsor to simplify such questions. Also, from a cognitive standpoint to give a substantive answer or leave the item blank. Their answer is th ere is concern that interviewed respondents cannot remember each of sev-

coded as a refusal or item nonresponse. a1 response choices when the next one is read before the previous one can

The unimode version of this question seeks to provide a common stimul mory. Although self-administered questionnaires can utilize

for the two modes by putting response categories into the stem of the se uch as those used on face-to-face interview show cards, it is

administered question and providing a common set of response categorie d that will work for all methods.

Thus, "no opinion" is explicitly offered in both versions of the questionnair portant as we move towards greater use of -mail and Web surveys in which the horizontal (landscape) orientation of

Principle 6.2: Avoid inadvertently changing the basic question structure across creens and the desire to get all response choices onto one screen encourages modes in ways thaf change the stimulus. fewer response choices. For example, in a recent e-mail sur-

The two most common ways in which question structures are in vey we reduced a scalar item with labels from five to three categories in order

changed is by switching yes/no items to check-all-that-apply items to increase the likelihood that it would appear the same on all respondents'

changing sequences of related questions to row/column matrices screens (Schaefer and Dillman, 1998). The touch-tone telephone

going from interviewer to self-administered questionnaires. Both of use of scales longer than nine points difficult because of the necessity

techniques were discussed in earlier chapters (Principles 2. g two numbers for highest scale values and only one for lower scale

respectively). . It has not been proven that the greater effort required for recording the Changing a sequence of individual items to a matrix creates a structure that t value (10) leads to less use of that category, but an experiment by Mu

is difficult for respondents with less education to understand. It also resu gests that possibility.

in losing control of the navigational process so there is less assurance that ea .4: Use the same descriptive labels for response categories instead of respondent is receiving the same question stimulus. Thus, we seek to upon people's vision to convey the nature of a scale concept. matrices in self-administered questionnaires whenever possible.

Check-all-that-apply self-administered survey questions, for ins n mail respondents are asked to rank their interest in music on a scale of

"Please mark all of the possible goals listed below you have decided to , a line with tick marks may be placed below the question that also

sue during this review period," are inevitably changed for tele in between ends of the scale that can be used to provide an

views, for example, "Please indicate whether each of the poss gure 6.5). The visual representation of the scale offers additional in-

going to read to you is one that you have decided to pursue rovided in the stem of the question. Thus, the self-adrninis-

during this review period." The problem with check-all-that apply qu tionnaire respondent receives a somewhat different stimulus than is the tendency of self-administered respondents to satisfice, that is, sto ne respondent. We should not be surprised if the self-adminis-

reading the list of items after they have checked what they consider an a ed respondent is less likely to use the extreme ends of the scale. The pro-

quate number of responses (Krosnick et al., 1996). The interview for sed revision for use in a mixed-mode survey does not present a visual

better question structure and can be used in self-administered surveys a presentation for either version of the questionnaire.

well. Sometimes visual metaphors are used for survey questions, such as asking spondents to imagine a ladder with ten steps, a thermometer, or some other

Principle 6.3: Reduce the number of response categories to achieve mode similarity. to convey a mental image. Such symbolism can work in both in-

When face-to-face interviewing was the dominant survey method administered surveys if it is uniformly presented to all re- nine-, or even 11-point scales, sometimes with each point labeled, offered to respondents on show cards. The advent of telephone intervi several items must be ranked, precede the ranking question with a encouraged surveyors to reduce the number of response categori cally, especially if similar categories were to be used for many s tions. Listening to interviewers stumble over linguistic sequences such As argued in Chapter 2, all types of surveys should avoid asking respondents "Do you completely agree, mostly agree, somewhat agree, neither agree to rank large numbers of items from top to bottom. Such a task is quite diffi-

disagree, somewhat disagree, mostly disagree, or completely disagree," cult for most respondents to accomplish, especially in interview surveys


items. For example: "If you own your home (asked in Question 4), are cur- sequently order effects remain a concern whatever the underlying reason for

rently employed (asked in Question 30), and had an income of at least $15,000 their occurrence.

last year (as asked in the last question), then you should . . . " If this instruc- The likelihood of such effects occurring seems greater when the questions

tion remains too complicated we then attempt to reorder questions that de- concern abstract attitudes and opinions not firmly possessed by the respon-

termine the skip pattern, and even restructure questions to eliminate the need dent. For example, "To what extent do you agree or disagree that strong lead-

to combine several disparate pieces of information in order to pr ear bright colors than are weak leaders?" Similarly

rectly to the next question. In one questionnaire I developed there was n about ever-present but mundane aspects of their

need for the home ownership question to appear so early in the question- many minutes on average do you spend talking each

naire. Therefore, it was relocated so that home ownership, employment, and er?" have considerable potential to be influenced

income questions were located next to one another. The reconstructed ques- factors other than the substance of the question (Rockwood et al., 1997).

tionnaire contained an instruction similar to the one listed above in the self- A conservative approach to asking such questions in mixed-mode surveys

administered version, but omitted it from the electronic teleph categories in half of the questionnaires.

Consequently although two types of respondents received some mplish in a CAT1 or computer-assisted personal

ent instructional information specific to the mode of response, the substance at more challenging to handle in a mail survey

of each question remained the same. follow-ups, and also poses quality control issues when the data are ems prudent to consider carefully whether the

Principle 6.7: Avoid question structures that unfold. o be influenced by order effects before

When telephone interviewing came of age, some surveyors began the practice le at least two questionnaire versions for each

of breaking questions into parts and allowing them to unfold for the respondent through a series of questions. For example, people might be asked instructions carefully for unintended response whether they tended to agree, disagree, or had no opinion on an issue. If they cts and consider their use for other modes. agreed, the follow-up question of whether they very strongly agreed, somewhat agreed, or slightly agreed was asked. Through these two steps a labeled ours listening to interviews in order to con- seven-point scale of responses could be constructed. Sirmlarly ~ e o p l e might otential differences between the stimuli being received by in-

be asked if their income for the preceding year was above or below $20,000.1f company a self-administered version of they said above, the next question asked would be whether it was more than terviewer answers to respondent questions

$35,000. At least one additional question would be asked depending upon the h as, "Please answer it in terms of what it answer received to the second query in order to select a more precise income t, such as, "Good, these comments are re-

category. helpful," during the interview define what is expected from the respon- Unfolding sequences with their alternative skip patterns are burdensome and may in a subtle way influence answers. As additional interviews are

on self-administered questionnaires, and should therefore be avoided. AS am from interviewers that a question is alternative to unfolding schemes we prefer to offer all categorie to answer questions in certain ways, for creasing the number of categories offered in the event the number becom n about how much money you paid out- too burdensome to offer over the telephone. k, we mean any form of payment-cash,

Principle 6.8: Reverse the order in which categories are listed i n half of age of rushing surveys into the field quickly after very brief pretests, the questionnaires. have often led to providing interviewers with last-

AS noted earlier in this chapter, evidence is far from consistent information to be used in a variety of situations throughout the inter-

effects (the tendency to choose from among the first respon uch efforts are usually justified by the goal of providing the highest

stated) occur in self-administered questionnaires and recency effects ( to the survey sponsor. However, in the mixed-mode

dency to choose from among the last offered response categories) occur tuation it seems important to consider carefully the possible effects of such

terview surveys. Nonetheless, we have observed many instances rovements for interviews which are not also being added to the self-

category order effects have occurred in both mail and telephone su ministered portion of the sample. This concern has also led us to anticipate


interviewer communications, for example, "If you don't kno eyor willingness to develop procedures for imple- swer your best estimate will be fine," and build that phrasing into both th enting a second mode, and the cost of such implementation. terview and self-administered components of the survey. If a token incentive has been sent by mail, there is a possibility that creating

ecrease the likelihood of a refusal when the recip- one. For example, it was observed in an experi-

IMPLEMENTATION ISSUES nal Survey of College Graduates that when called ephone, recipients of a token financial incentive in the first of four pre-

The implementation issues that arise when modes are mixed mailings were less likely to refuse than those who did not receive such

part on how modes are administered in relation to one ano incentive with the first mailing (Mooney et al., 1993).

veyor want people who failed to respond to one mode to respond to anothe act by a second mode take into account the

Or is an auxiliary mode being used to encourage respons response. For example, in a switch from tele-

mode? In either case, the surveyor cannot ignore how one mode influen egin a letter: "Recently we attempted to contact

use of the other. The ways in which modes are mixed affects significantly u by telephone about an important national survey we are doing . . . " A

ways implementation procedures need to be tailored. lar introduction might be developed for switching from mail to tele-

ndpoint, such an introduction communicates t (otherwise why would this extraordinary at-

SWITCHING THE REQUEST FROM COMPLETING A QUESTIONNAIRE BY ONE to telephone provides an opportunity to ask MODE TO ANOTHER uest has questions about the survey and to re- From a social exchange standpoint, switching methods raises new possib itching to mail also allows the surveyor to state ties for communicating greater rewards, lower costs, and increased trust. sons for the survey's importance in different ways than those mentioned large number of failed attempts to reach someone by telephone, many which resulted in written or verbal messages from someone who an the phone and/or voice mail messages, may become a repeti

ICH METHOD SHOULD GO FIRST? is increasingly easy to ignore. Using a different mode offers an opp for the surveyor to provide new information to the respondent in a some instances one may have mailing addresses, telephone numbers, and For example, switching from telephone to mail allows respondents to see en e-mail addresses, so that several possibilities are available. In such cases stationery (encouraging legitimization and trust), a personalized c ay be to go with the least expensive method and collect (encouraging feelings of importance), and the content of a questionnaire (1 h of the data as possible before switching to another method. It might apprehension about questions not yet answered), thus invoking new argued that telephone attempts should begin first because one is bet- change considerations. Evidence exists that people prefer certain mo m nonresponse. Evidence is not compelling from a re- (Groves and Kahn, 1979), and if such preferences are signi sponse rate standpoint to unequivocally argue for always beginning with one reason that people who have not responded to one mode because method or another. it may be receptive to a change in approach. In addition, the ound a number of content and population bases for de- switching modes tends to emphasize the importance of the study, perhap ing to begin with a particular method. In a national survey of manufactur- couraging thoughts along of the lines of "if this weren't important, , which required completion of an 18-page mail questionnaires versus a wouldn't be trying to reach me in a different way." -minute telephone survey, we found it advantageous to begin with attempts

There can also be no doubt that switching modes is effective in improv complete the survey by telephone, and then switch to mail. Whereas a response rates beyond those that can be obtained by relianc ate was obtained in this way a previous attempt to uti- method. Supporting evidence is available from government- and unive efore phoning resulted in a response rate of only 45% sponsored surveys on a variety of topics (e.g., Mooney et al., 199 contrast, for a survey of medical doctors, gatekeepers 1998). The 1990 U.S. Decennial Census switched from mail que ntact extremely difficult. A slightly higher response ter 65% of households responded, and through the use of face-to-face inter e (67% versus 59%) was obtained by sending a U.S. Postal Service priority views increased response rates to over 98%. The major impediments t of which was to get past gatekeepers, and following


it with phone calls to either get the mail form completed or obtain the infor SING ONE MODE TO SUPPORT ANOTHER MODE mation by telephone, depending upon the preference of the responden In a 1976 test it was found that sending preletters to households selected (Moore, Guadino, deHart, Cheadle, and Martin, 1998). More importantly f or response to a telephone survey improved response rates over the tele- this study leading with the mail procedure produced responses more quic hone by nearly 10 percentage points (Dillman, Gallegos, and Frey 1976). On still other surveys we have chosen to send mail questionnaires first s method has been used with dual-frame sample surveys (part listed, part cause of the sensitive nature of the surveys (to help convey the survey's leg andom digit telephone numbers) to improve response to telephone surveys macy) and to decrease the anxiety of respondents over the telephone (Moo (Groves, 1989). This procedure seems particularly desirable if one wishes to 1998). provide a token incentive to individuals about to be surveyed by telephone or

One instance in which it seems particularly desirable to begin with an e-mail. However, one test of sending a preletter by regular mail to encourage terview prior to sending a mail survey to respondents is when it is necess e to an e-mail survey did not improve response rates as much as send- to select a respondent. A telephone call allows the surveyor to ask a sequen quivalent message via the method (e-mail) by which people were be- of questions to determine to whom the questionnaire should be sent. This p asked to respond (Schaefer and Dillman, 1998). cedure is especially important in business surveys because of the finding metimes when multiple contacts have been made and no response is Paxson, Dillman, and Tarnai (1995) that knowing the likely respondent's nam coming, the questionnaire has been greatly shortened so that i t consists resulted in significantly higher response rates than when sending question d y a few key questions. The questions might include those for which it is naires to the "owner/manager." However, even this finding has exceptions. xpected that respondents may differ from non-respondents. A typical ap- Unce, when sending questionnaires to a person in each of the nation's larg peal of this nature is to explain to recipients that they have not responded to 238 universities who could not be identified by name or even position, a pr the longer survey and that tlus behavior is okay but that "filling out responses letter was sent to the president of each university asking them to identify t to these few questions will help us know whether people who did not respond person described in the cover letter and forward a mail questionnaire to the would have answered differently from those who did, which would be enor- Procedures used in this study which involved both mail and telepho ously helpful." In social exchange terms, this kind of appeal can be de- follow-up, resulted in a response rate of 74% (Scott, 1989). ribed simply as reducing the size of the request. In much the same way as

shortening a long questionnaire encourages high response, this kind of re-

PROVIDING RESPONDENTS WITH THE CHOICE OF RESPONSE MODE quest may do so as well. Further, it gives recognition to the respondents that their right to refuse the longer questionnaire is respected and appealing for a

Because some respondents have survey mode preferences, it may seem des small favor would help meet the needs of the study. Most commonly such re- able to offer respondents a choice of which mode to answer. However, quests involve a switch from one survey mode to another. However, if the choice alone may not increase response rates. In an experimental test of ce ode is switched for such a follow-up then it is important to avoid selecting sus questionnaires, a national sample of respondents was given the option o ons that use vague quantifiers. The concern is that response might be returning the mail questionnaire or calling in their answers by telephone (Dill- entially affected by mode so that conclusions about the influence of non- man, Clark, and West, 1995). The result was to stimulate 5% of the survey response would be jeopardized. sample to respond by telephone. However, the overall response rate was no different than when respondents were only given an opportunity to respo by mail.

Nonetheless, it is important to realize that when calls are made to respo CONCLUSION

dents after they have failed to respond to several requests to complete a que Mixing together different survey modes, and particularly data collected by in- tionnaire in another mode, response rates are likely to increase. In a se terview with that collected by self-administered questionnaires, raises many choice of mode has been provided to those who waited, but this situati challenging issues. Not only may the modes be mixed in different ways and quite different because the surveyor called rather than leaving it up to the for different purposes, but some procedures are likely to produce different an- spondent to call in. This possibility did not exist for the census experiment swers. Nonetheless, the possibilities for improving response rates and reduc- cause phone numbers, or even the names of people living at the addresse g nonresponse and coverage errors are such that mixed-mode surveys are included in the sample, were unknown. creasingly likely to be done.

C H A P T E R 7

244 TAILORING TO THE SURVEY SITUATION

Our way of minimizing mode differences, though it is unlikely that can be prevented entirely, is to use a technique I have described as uni construction. This technique involves making a deliberate effort to assure respondents to all modes receive an equivalent stimulus regardless whether it is delivered aurally or visually. This action requires moving aw from the typical construction mode used when each mode is implemente alone, whether by interview or self-administration.

Mixed-mode surveys by themselves represent a tailoring of the proced described in Part I of this book. However, as we will see in later chapters, represent a type of tailoring that becomes an integral feature of surveys geted to particular populations and situations, thus serving as a means to greater end. Alternative Questionnaire

Delivery: In Person, to Groups, and Through Publications

-ADMINISTERED QUESTIONNAIRES are sometimes delivered to people by other than the postal system or computers. For a variety of reasons,

eyors may find it advantageous to deliver paper questionnaires in person dividuals, or to bring people together so that questionnaires can be ad-

d efficiently in a group setting. In addition, surveyors frequently at- o deliver questionnaires through magazines, newsletters, or other

r information sources in efforts to get very inexpensive feedback, usu- limited success. In this chapter I discuss the challenges of tailoring

ures to these delivery situations.

IN-PERSON DELIVERY TO INDIVIDUALS

of the most significant barriers to the use of mail-back questionnaires is ack of an adequate population list from which a sample can be drawn. For

Viewers of an art exhibit Visitors to a national park People who hunt or fish in a particular recreational area Customers of a restaurant Users of an airport Purchasers of day-tickets to a theme park Households in a particular city neighborhood

mixed-mode surveyspeople.uncw.edu/maumem/soc500/ch6.pdfe-mail and web surveys gain favor with...

Documents