Effects of Survey Mode on Responses:

An Experiment of Social Desirability Biases in Tele-phone, Interviewer, and Online Surveys

Indeok SongJoongbu University, Korea

Sung Kyum ChoChungnam National University

Contents

• Background: 3 Surveys on Gambling Addiction

• Research Question & Purpose

• Review of Previous Research

• Methods

• Results

• Discussions & Suggestions

Results of 2010’s Surveys

Online (n=581) Telephone (n=1,797)

Interview (n=1,000)0

0.51

1.52

2.53

3.54

3.83

0.32 0.5

Means on Gambling Addiction Scale (CPGI 0~27)

F (2, 3,375) = 402.72, p < .001

Results of 2010’s Surveys

Online (n=581)

Telephone (n=1,797)

Interview (n=1,000)

0%10%20%30%40%50%60%70%80%90%

100%

35.6

95.2 87.123.1

1.6 6.8

19.1

1.4 4.422.2

1.8 1.7

ProblemModerate RiskLow RiskNon-Problem

χ²= 1,130.35, df = 6, N=3,378, p < .001

Research Question What is the reason for the inconsistencies

among 3 surveys’ results? Which mode does produce the most accurate estimates?

Sampling? Survey Mode?

Previous Research• Survey modes “can be equivalent but are

not always identical” (Riva et al., 2003, p. 79)

–Sensitivity & Social desirability (Booth-Kewley, Lar-son, & Miyoshi, 2007; Couper, 2000; Huang, 2006; Joinson, 1999; Kays, Gather-coal, & Buhrow, 2011; Tourangeau & Yan, 2007, 2011 and so on)

• Mode of admin.: Self- vs. Interviewer-administered– Self-administration increases reporting of socially unde-

sirable behaviors (Corkrey & Parkinson, 2002; Currivan et al., 2004; Hochstim, 1967; Moskowitz, 2004; Mott, 1985; Richman et al., 1999, and so on)

∴ Paper-Pencil, Online > Telephone, Interviewer

Previous Research• Interviewer or 3rd party presence

– Social desirability bias is worse in telephone survey than in face-to-face interviews (Aquilino, 1994; Groves & Kahn, 1979; Holbrook et al., 2003; Johnson, Hougland, & Clayton, 1989; Leeuw & van der Zouwen, 1988, and so on)

– The presence of others affects responses to sensi-tive questions (Aquilino, Wright, & Supple, 2000; Harrison, 2001; Lombard & Ditton, 1997; Sudman, 2001; Tourangeau & Yan, 201, and so on)

– Online survey yields the lowest level of other peo-ple’s presence ∴ Online > Interviewer > Telephone

Research PurposeControlling for variances from sampling and dif-ferent groups of samples,

to examine the effects of survey modes on the responses to sensitive questions.

Methods• Pretest-posttest 3 between-group experi-

mental design

– 93 undergraduate students 66 analyzed• Out of 66, male 38 (57.6%), female 28 (42.4%),

mean age 20.61 (SD=1.97)

Paper-pencil(N=93) 2 weeks Random

Assignment

Online(n=21)

Telephone(n=24)

Interview(n=21)

Methods• Questionnaire Design

– Addiction Scales• Gambling addiction (4pt., 9 items)

– M=9.24, SD=2.54, α=.84• Internet addiction (4pt., 15 items)

– M=29.13, SD=7.24, α=.90• Game addiction (4pt., 20 items)

– M=26.80, SD=8.84, α=.96

– Dissatisfaction with self-appearance (5pt., 6 items)• M=14.43, SD=3.71, α=.66

– Sexual values (5pt., 10 items)• M=27.88, SD=7.14, α=.89

Methods– Sexual experience

• Age of the first experience– n=27, M=18.59, SD=1.67

• Number of sexual partners– n=26, M=10.81, SD=3.68

• Sexual behaviors (physical contact, cuddle, kiss, caress, intercourse) experienced in current rela-tionship

– n=19, M=3.68, SD=1.42

– Demographics• Gender, age, household-income

Results• Homogeneity among groups (pretest: paper-pen-

cil)Source df F p

Gambling Addiction (2, 63) 1.29 .28

Internet Addiction (2, 63) .09 .91

Game Addiction (2, 63) 2.13 .13

Dissatisfaction with Self (2, 63) .31 .73

Sexual Values (2, 63) .28 .76

Age of 1st Experience (2, 24) 1.89 .19

No. of Partners (2, 23) .59 .56

Sexual Behaviors (2, 16) .24 .79

Responses for Game Addiction

Paper-Pencil Posttest20

22

24

26

28

30

32

28.2430.71

24.5

21.92

28

24.91OnlineTelephoneInterview

Time * Mode: F (2, 63) = 3.35, p = .041, eta² = .10

Responses for Dissatisfaction with Self-Appearance

Paper-Pencil Posttest12

12.5

13

13.5

14

14.5

15

13.95

14.33

14.83

12.29

14.38

13.19OnlineTelephoneInterview

Time * Mode: F (2, 63) = 5.58, p = .006, eta² = .15

2 x 3 Factorial ANOVAs• Independent Variables

– 2 groups (‘Low’ & ‘High’) by median split– 3 modes (Online, Telephone, Interview)

• Dependent Variables– Changes in responses = Pretest – Posttest

DVs N M SDGambling Addiction 66 .27 2.30Internet Addiction 66 1.08 6.78Game Addiction 66 1.14 8.05

Dissatisfaction with Self 66 1.18 3.13Sexual Values 66 1.44 4.65

Age of 1st Experience 22 .68 1.49No. of Partners 22 -3.95 16.76

Sexual Behaviors 16 -.13 1.46

Changes in Responses for Gambling Addiction

Online Telephone Interview

-2-1.5

-1-0.5

00.5

11.5

22.5

3

-0.080 -0.07

1.22

2.5

-1.83

Low GroupHigh Group

Mode: F (2, 60) = 5.32, p = .007, eta² = .15G * M: F (2, 60) = 5.06, p = .009, eta² = .14

Changes in Responses for Internet Addiction

Online Telephone Interview

-5

-3

-1

1

3

5

7

-3 -4.17

0.51.82

5.586.44

Low GroupHigh Group

Group: F (1, 60) = 23.15, p < .001, eta² = .28

Changes in Responses forGame Addiction

Online Telephone Interview

-10-8-6-4-202468

10

-7.56

-0.2 -0.441.33

7.225.75

Low GroupHigh Group

Group: F (1, 60) = 18.54, p < .001, eta² = .24Mode: F (2, 60) = 5.65, p = .006, eta² = .16

Changes in Responses for Dissatis-faction with Self-Appearance

Online Telephone Interview

-2

-1

0

1

2

3

4

-1.75

1.36 1.081.44

3.54

1.33

Low GroupHigh Group

Group: F (1, 60) = 7.41, p = .008, eta² = .11Mode: F (2, 60) = 4.89, p = .011, eta² = .14

Changes in Responses for Sexual Val-ues

Online Telephone Interview-101234567

0 0.29 0

1.6

5.8

1.7Low GroupHigh Group

Group: F (1, 60) = 7.32, p = .009, eta² = .11

Changes in Responses for Age of the 1st Intercourse

Online Telephone Interview-101234567

0.330.67

0.29

6

0.5

Low GroupHigh Group

Group: F (1, 17) = 16.90, p = .001, eta² = .50Mode: F (2, 17) = 7.56, p = .004, eta² = .47G * M: F (2, 17) = 19.01, p < .001, eta² = .53

Summary of the Results• Responses for ‘Game Addiction’ & ‘Dissatisfaction

with Self-Appearance’– Increased in the online survey mode– Decreased in the telephone & interview survey modes

• Those who scored higher on the sensitive questions are more likely to change their responses in a so-cially desirable direction– Less addicted (gambling, Internet, game)– Less dissatisfied with self-appearance– Less liberal (more conservative) sexual values– Younger in the age of the 1st sexual intercourse

Discussions & Suggestions• Survey responses to sensitive (privacy-related) questions are

affected by different survey modes.– Online survey in which respondents are less concerned

about their privacies is more likely to elicit franker (more ac-curate) responses.

• Paper-pencil (group administration), telephone, interview modes lead respondents to privacy concern & misreport.

• Suggestion: Dual-mode surveyTelephone Sampling Online Questionnaire

Any idea or sug-gestion

would be wel-

comed.