question or mode effects in mixed-mode surveys: a cross ... · probability based panel, established...
TRANSCRIPT
![Page 1: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/1.jpg)
67th AAPOR Conference
Orlando, May 17-20, 2012
Question or Mode Effects in Mixed-Mode
Surveys: A Cross-cultural Study in the
Netherlands and the UK
Edith de Leeuw & Joop Hox
Department of Methodology & Statistics, Utrecht University
Pamela Campanelli & Gerry Nicolaas
The Survey Coach / NatCen Social Research, UK
Campanelli, De Leeuw, Hox, & Nicolaas
The
Survey
Coach
67th AAPOR Conference
Orlando, May 17-20, 2012 NatCen Social Research
![Page 2: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/2.jpg)
2
Mixing Modes
Mixed-Mode Surveys combine data, but
Are responses between modes comparable?
Modes differ on many mode-inherent factors,
e.g.
Privacy
Interviewer present or not
Computer vs paper
Interactivity
Interviewer help or not
Feedback
Information transmission
Audio vs visual vs both
![Page 3: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/3.jpg)
3
Question Design Different modes have different traditions
Prime example:
Unfolding or two-step aka branching (telephone)
Complete question one-step (web, mail)
Numerous studies have shown that modes differ very
slightly when comparable questions are used, but
Question format influences answers given
Response options, question wording, etc
Early overview Sudman et al 1974
Effects found even within one mode
May data be combined?
Channel
Capacity
![Page 4: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/4.jpg)
4
Mode Experiments in Two Countries
Mode Experiments in the Netherlands*
NL: CATI vs Web (CAWI)
Members Dutch LISS-panel (CenTERdata)
Probability based panel, established in 2007
Random assignment to mode
CATI: assigned 2000, responded 1207 (60%)
Web: assigned 6134, responded 4003 (65%)
* Courtesy of the LISS-panel. The LISS panel data were collected by CentERdata
(Tilburg University, The Netherlands) through its MESS project funded by the
Netherlands Organization for Scientific Research
![Page 5: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/5.jpg)
5
Mode Experiments in Two Countries
Mode Experiments in Great Britain*
GB: CAPI vs CATI vs Web
Respondents to NatCen Omnibus survey
Agreed to be recontacted
Access to internet
Random assignment to mode
CAPI: responded 282 (73%)
CATI: responded 314 (69%)
Web: responded 349 (47%)
* Funded by the Economic and Social Research Council (award RES-175-25-0007).
![Page 6: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/6.jpg)
6
Question Format NL
Eight questions on acceptability of usage of
advanced medical technology
If it will save a live, everything is permitted
It is not desirable to utilize every medical invention, just
because it is technologically possible
Five point scale: Totally Agree….Totally Disagree
Do-not-Know not offered, but accepted when given
Two question format experiments
Unfolding (two step) vs Direct Question (one step)
With vs without explicit ‘neutral’ category
![Page 7: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/7.jpg)
7
Analyses NL
Variables of Interest
Independent variables (dichotomous)
Mode: CATI vs WEB
Format
Unfolding vs Complete question
Dependent variables
Mean score (Concept of interest: what do people think about
the topic)
Extremeness: Proportion extreme answers 1 & 5
Analyses done after appropriate recoding of responses
1: most conservative in acceptance modern technology
5: very prone to accept new medical technology
![Page 8: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/8.jpg)
8
Differences in Means!
Two factors affect mean score (and thereby
conclusions on acceptance of medical
technology)
1. Mode
2. Unfolding
Note: no interactions
What may cause these shifts in mean?
Mean score sensitive for extremes!
More extreme scores?
Is there a primacy (Web) or recency (CATI) effect?
![Page 9: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/9.jpg)
9
Summary of NL Results
Higher Overall Means:
CAWI
Complete Question (one step question)
Same effects for with and without neutral
More Extreme Answers
CATI
Unfolding (two step question)
Without neutral midpoint
More recency in CATI and with Unfolding
More primacy in Web+Unfolding (interaction)
![Page 10: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/10.jpg)
10
Question Format GB
Two questions
Whether standard of living has increased, decreased,
or stayed the same since last election
Seven point scale: Increased by a large amount…. Decreased
by a large amount
Neutral category (i.e. ‘stayed the same’)
Whether respondent considered local shopping
facilities to be poor or good
Six point scale: Extremely poor….Extremely good
No neutral category
Two question format experiments
Unfolding (two step) vs Direct Question (one step)
![Page 11: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/11.jpg)
11
Analyses GB
Variables of Interest
Independent variables (dichotomous) Mode: CAPI vs CATI vs WEB
Format Unfolding (two-step) vs Complete question
Dependent variables Mean score
The higher the mean score, the more respondents chose decrease in standard of living
The higher the mean score, the more respondents chose shopping facilities extremely good
Proportion of extreme answers Standard of living – categories 1 & 7
Shopping facilities – categories 1 & 6
![Page 12: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/12.jpg)
12
Summary of GB results
Standard of living (7 point scale, with midpoint)
Unfolding increases mean score overall and within each mode
Unfolding increases extreme responses overall and just within web
Overall mode effect for extremes but not means
Almost interaction of format and CATI / Web for means
Rating of shopping facilities (6 point scale, no midpoint)
No overall format effect for means or extremes or within modes.
No overall mode effect for means or extremes but some mode
differences within formats
Focusing on last 2 categories, unfolding has consistent but non-
significant higher mean scores across all modes similar to standard
of living question
![Page 13: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/13.jpg)
13
Compare GB & NL results
Consistent results:
Unfolding increases extremeness for both NL analyses of
extremes and 1 GB analysis of extremes
Inconsistent results:
Overall mode effect in NL data but only for 1 GB analysis
of extremes
Only one interaction of format and mode with NL data and
one almost present with GB data
Evidence recency effects in NL data but not in GB data
More extremeness without neutral category for NL data,
less extremeness without neutral for GB
Evidence of response contraction bias in GB results
![Page 14: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/14.jpg)
14
Overall Conclusions
Small, statistically significant, effects of unfolding on extremes in both the Netherlands and Great Britain
Small, statistically significant, effects of mode in the Netherlands and in 1 case in Great Britain
Very few interactions
All effects found point to the same conclusion: Unfolding creates differences within and between modes
Don’t mix unfolding and complete question formats across modes
![Page 15: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/15.jpg)
67th AAPOR Conference
Orlando, May 17-20, 2012
THANK YOU!
Contact information
Edith de Leeuw: [email protected]
Joop Hox: [email protected]
Pamela Campanelli: [email protected]
Gerry Nicolaas: [email protected]
The
Survey
Coach
67th AAPOR Conference
Orlando, May 17-20, 2012 NatCen Social Research
![Page 16: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/16.jpg)
16
Appendix A: Statistical Results NL
Results for the Netherlands
Effects of mode and unfolding on
means and extremeness
Graphs, significance and effects
No covariate adjustment necessary
No selective nonresponse
![Page 17: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/17.jpg)
17
Effect of Mode and Unfolding With Neutral Midpoint
Significant but small effect of mode and unfolding: R2= .008
No Interaction Format and Mode!
Means Overall CATI: 2.93
Web: 3.00
Unfolding: 2.94
Complete Q: 2.99
Mean Score
![Page 18: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/18.jpg)
18
Effect of Mode and Unfolding Without Neutral Midpoint
Significant but small effect of mode and unfolding: R2= .004
No Interaction Format and Mode!
Means Overall CATI: 2.93
Web: 2.97
Unfolding: 2.92
Complete Q: 2.97
Mean Score
![Page 19: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/19.jpg)
19
Effect of Mode and Unfolding With Neutral Midpoint
Significant but small effect of mode and unfolding: R2= .008
No Interaction Format and Mode!
Mean Prop 1 and 5 CATI: 0.26
Web: 0.19
Unfolding: 0.26
Complete Q: 0.19
Extremeness
![Page 20: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/20.jpg)
20
Effect of Mode and Unfolding Without Neutral Midpoint
Significant but small effect of mode and unfolding: R2= .004
No Interaction Format and Mode!
Mean Prop 1 and 5 CATI: 0.31
Web: 0.25
Unfolding: 0.30
Complete Q: 0.25
Extremeness
![Page 21: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/21.jpg)
21
Primacy? Effect Mode and Wording
Significant effect wording and
Interaction, R2=.08
Proportion 1’s Unrecoded
CATI Web Total
Unfolding .07 .10 .08
Complete
Question
.04 .03 .04
Total .06 .06 .06
Neutral Midpoint Without Neutral
Significant effect mode and wording
Interaction , R2=.14
Proportion 1’s Unrecoded
CATI Web Total
Unfolding .08 .15 .11
Complete
Question
.05 .05 .05
Total .06 .10 .09
![Page 22: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/22.jpg)
22
Recency? Effect Mode and Wording
Significant effect mode and wording
R2=.05
Proportion 5’s Unrecoded
CATI Web Total
Unfolding .27 .24 .25
Complete
Question
.21 .15 .18
Total .24 .19 .20
Neutral Midpoint Without Neutral
Significant effect mode, wording,
and interaction , R2=.02
Proportion 5’s Unrecoded
CATI Web Total
Unfolding .27 .26 .27
Complete
Question
.25 .20 .23
Total .26 .23 .24
![Page 23: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/23.jpg)
23
Appendix B: Statistical Results GB
Results for Great Britain
Effects of mode and unfolding on
means and extremeness
Graphs, significance and effects
The GB results are based on analyses
with control variables carefully chosen to
adjust for differential nonresponse
between the 3 modes
![Page 24: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/24.jpg)
24
Effect of Mode and Unfolding With Neutral Midpoint
Significant effect of unfolding, no overall effect of mode: R2= .043
Almost an interaction between format and CATI/Web (p = .101)
Means Overall CAPI: 4.73
CATI: 4.63
Web: 4.81
Unfolding: 4.97
Complete Q: 4.47
Mean Score
X is
unfolding!
![Page 25: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/25.jpg)
25
Effect of Mode and Unfolding With Neutral Midpoint
Significant but small effect of unfolding and mode: R2= .020
No Interaction Format and Mode
Mean Prop 1 and 7 CAPI: 0.23
CATI: 0.16
Web: 0.16
Unfolding: 0.22
Complete Q: 0.15
Extremeness
X is
unfolded
![Page 26: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/26.jpg)
26
Effect of Mode and Unfolding Without Neutral Midpoint
No effect of unfolding, almost effect of mode: R2= .011
No Interaction Format and Mode
Means Overall CAPI: 3.97
CATI: 4.15
Web: 3.98
Unfolding: 4.03
Complete Q: 4.03
Mean Score
Y is
unfolding
![Page 27: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/27.jpg)
27
Effect of Mode and Unfolding Without Neutral Midpoint
No effect of unfolding or mode: R2= .009
No Interaction Format and Mode
Mean Prop 1 and 6 CAPI: 0.10
CATI: 0.07
Web: 0.06
Unfolding: 0.06
Complete Q: 0.09
Extremeness Y is
unfolded
![Page 28: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/28.jpg)
28
Primacy and Recency?
Neutral Midpoint Without Neutral Top category - Primacy - No
Very few cases:
Unfolding 1.3%
Complete question 2.0%
No differences by mode.
Last category - Recency – No Unfolding 21.8%
Complete Question 12.6%
Within complete question, CAPI
(with card) > Web = CATI.
No recency effects in CATI.
Top category - Primacy - No
Very few cases:
Unfolding 2.4%
Complete question 3.3%
Within Unfolding, CAPI (no card) >
CATI = Web .
No primacy effects for Web.
Last category - Recency – No Unfolding 4.8%
Complete question 7.3%
Within complete question, CAPI (no
card) >Web = CATI.
No recency effects in CATI. Not
clear why CAPI, but not also CATI.
![Page 29: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/29.jpg)
29
Response contraction bias?
Tendency to avoid the most extreme response categories (see Tourangeau, et al, 2000)
No sign of this in top categories
But evidence in bottom categories
Combine last two categories
Standard of living – categories 6 & 7
Rating of shopping facilities – categories 5 & 6
Standard of Living Rating of shopping facilities
Unfolding
(two-step)
Complete
Question
Unfolding
(two-step)
Complete
Question
Category 6 34% 21% Category 5 26% 20%
Category 7 22% 13% Category 6 5% 7%
![Page 30: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/30.jpg)
30
Effect of Mode and Unfolding With Neutral Midpoint
Significant effect of unfolding, no overall effect of mode: R2= .043
Almost an interaction between format and CATI/Web (p = .107)
Mean Prop 6 and 7 CAPI: 0.36
CATI: 0.32
Web: 0.38
Unfolding: 0.46
Complete Q: 0.25
Extremeness:
Last 2 categories X is
unfolded.
![Page 31: Question or Mode Effects in Mixed-Mode Surveys: A Cross ... · Probability based panel, established in 2007 Random assignment to mode CATI: assigned 2000, responded 1207 (60%) Web:](https://reader035.vdocuments.mx/reader035/viewer/2022081404/5f044c107e708231d40d4798/html5/thumbnails/31.jpg)
31
Effect of Mode and Unfolding Without Neutral Midpoint
No significant effects of unfolding or mode: R2= .005
No Interaction Format and Mode
Mean Prop 5 and 6 CAPI: 0.32
CATI: 0.34
Web: 0.29
Unfolding: 0.34
Complete Q: 0.29
Extremeness:
Last 2 categories Y is
unfolded