online survey participation via mobile devices - · pdf fileonline survey participation via...
TRANSCRIPT
Online Survey Participation via Mobile Devices
AAPOR 2013, Boston, MAMay 18, 2013
Michael Bosnjak, Teresio Poggio, Kai R. Becker, Frederik Funke, Alexandra Wachenfeld, Beat Fischer
• Definitions of mobile surveys– Interviewer-administered surveys
• Interviews among mobile phone users• Interactive voice response surveys among mobile phone
users– Self-administered surveys
• SMS (text messaging) surveys• Browser-based surveys on mobile devices (e.g., mobile
phones having mobile Internet-access, Smartphones, etc.)
• Our focus: – Self-administered surveys AND– using a mobile device AND– touch-operated browser interface.
Mobile Surveys?Self-Administered Mobile Surveys?
Challenges for Survey Methodology
• Measurement equivalence?– Not treated here, but some experimental
comparisons do exist (e.g., Metzger & Bosnjak, 2009).
• (Non)Response Mobile vs Desktop/Laptop (selection):– (Non)Response rates?– Sample composition differences?– Differences in participation patterns/behavior?– ´Mode-Compliance´: Do participants actually stick
to a designated mode, i.e. the Desktop/Laptop mode for which a survey has been optimized?
Overall Research Questions
• Rate of mobile participants? Is the rate of mobile devices used when conducting online surveys substantial (or negligible)?
• Participation patterns? – Within online study comparisons (mobile versus other):
Amount of data collected, drop-out rates, amount of data available for drop-outs.
– Across online panel studies: Do mobile participants stick to the mobile mode across panel waves?
• Systematic differences between mobile/non-mobile participants?Can the propensity to use a mobile device to participate in online surveys be explained (using demographic variables)?
Studies: OverviewStudy 1:
LINK PanelSeven largely independent samples, market research
access panel
Study 2: GESIS Pilot Panel
Probability-based scientific access panel, German Internet users
Rate of mobile participants?
Weighted average % of mobile participants
(Meta-analytic synthesis)
Number of times a mobile device had been used across eight panel
waves(Descriptive analysis)
Participation patterns?
Mode comparisons for amount of data entered, drop-out
behavior
Participation patterns across 8 waves
Systematic differences
mobile/non-mobile?
Propensity to use mobile devices vs. other
(Logistic regression)
Predictors of (1) the likelihood (not) to use mobile devices at all and (2) incidence rate of using mobile devices in the 8 waves.
(Zero-inflated poisson regression model)
Study 1: Method (Sketch)• Meta-analytic synthesis (RE model) of seven online panel
studies conducted by LINK Germany (March- May, 2012)• Parameters synthesized (selection): Completion rate,
dropout rate, mobile participation rate, length of open-ended entries
• Final sample sizes, topics, and Flash usage (k= 7):– Study 1 (DGE05419), n= 1839, 7 mins, Multitopic-Survey: Finance, Fonds, no
Flash– Study 2 (DGE05426), n= 1711, 58 mins, Political opinions, voting behavior,
Flash– Study 3 (DGE05427 ), n= 906, 54 mins, Political opinions, voting behavior, Flash– Study 4 (DGE05428), n= 867, 53 mins, Political opinions, voting behavior, Flash– Study 5 (DGE05429), n= 1380, 20 mins, Multitopic-Survey: digital liefstyle,
sweets/candy, no Flash– Study 6 (DGE05429_B), n= 274, 3 mins, Replication of study 5 with new sample
(only sweets/candy), no Flash– Study 7 (DGE05433), n= 1838, 19 mins, Advertising effectiveness (sweets),
Flash
Study 1: Results: Mode Non-Compliance
Average Mobile OS participation rate: 5.8% (95 CI: 4.3%/7.3%),
homogeneous effect using RE model
Study 1: Results: Participation Patterns
• No differences Mobile/Desktop OS for– # of pages with entries (20 on average)– # of open-ended question answered (4 on average)– # of characters entered (165 on average)
• Differences for drop-outs Mobile/Desktop OS:– Dropout rate comparison:
Mobile 12% (9/16), Desktop 6% (5/7)– # of pages completed before dropped out:
Mobile 5.9 (5.4/6.3), Desktop 4.9 (4.8/5.0)– # of open-ended questions answered:
Mobile 2.9 (2.5/3.3), Desktop 2.4 (2.2/2.5)– > Higher drop-out rates for Mobile OS, but also more data
for Mobile OS drop-outs compared to Desktop OS!
Study 1: Results: Mobile Survey Propensity
Sketch of results from a micro-data analysis (logistic regression):
• Younger subjects are more likely to answer from Mobile OS devices
• Male subjects more than women• No significant effect of educational level
• Outlier Study 7: Being a participant in this study almost halves the likelihood of using Mobile OS devices (most likely because of the use of video content/stimuli)
Studies: OverviewStudy 1:
LINK PanelSeven largely independent samples, market research
access panel
Study 2: GESIS Pilot Panel
Probability-based scientific access panel, German Internet users
Rate of mobile participants?
Weighted average % of mobile participants
(Meta-analytic synthesis)
Number of times a mobile device had been used across eight panel
waves(Descriptive analysis)
Participation patterns?
Mode comparisons for amount of data entered, drop-out
behavior
Participation patterns across 8 waves
Systematic differences
mobile/non-mobile?
Propensity to use mobile devices vs. other
(Logistic regression)
Predictors of (1) the likelihood (not) to use mobile devices at all and (2) incidence rate of using mobile devices in the 8 waves.
(Zero-inflated poisson regression model)
Study 2: Method (Sketch)
• Post-hoc analysis on the use of mobile devices by members of a probability-based on-line panel
• GESIS Pilot Panel, 8 survey waves (Dec 2010-Feb 2012)
• Analysis focused on the ones who had at least started all the 8 on-line questionnaires (n=587)
Study 2: Results: Mode Non-Compliance
About 8% of respondents used a mobile device at least once. Distribution of mobile device usage across all eight survey waves (# of waves a mobile device was used):
Never: 91.7 %1: 2.9 %2: 1.7 %3: 0.7 %4: 0.3 %5: 0.3 %6: 0.3 %7: 1.4 %8: 0.7 %
Study 2: Results: Participation Patterns
91.7 %: No use at all2.1 %: Almost only mobile users (in at least 7 out of 8 waves)2.2 %: Innovation-led users (used more in most recent waves)1.4 %: Tried mobiles and back (used less in most recent waves)2.6 %: Other (no clear pattern in the use of mobile devices)
Study 2: Results: Mobile Survey Propensity
Two distinct processes likely to be in action here:1) using or not using smartphones (in everyday life)2) if using smartphones: number of participations via mobile
(propensity to use a mobile device)
Our dependent variable: Number of times a mobile device was used in the 8 waves
Type of analysis: Zero-inflated poisson regression model, which basically consists in estimating 2 simultaneous equations:
1) a logistic regression model for the probability of not using mobile devices at all (modeling ‘inflated’ zeros, due to no use of these devices)
2) a poisson regression model for counting the number of times these devices were used in the waves (modeling ‘true’ zeros and 1-8 counts)
Study 2: Results: Mobile Survey Propensity
- For both equations (may have different predictors in principle) we tested the effect of several typical social stratification variables:
o Age and gendero Level of education, occupational class, living in
rural versus urban areaso Marital status and number of members in the
household- A variable ‘positive attitude towards technology was also
considered in the model’- Backward approach: Starting with the same set of
predictors for both equations, only significant variables included in the final model
• Inflate model (logit)(likelihood not to use mobile devices at all)– The elder, the less likely to use mobile devices– Women also less likely (not fully significant)– No clear/significant effects of eduction– No clear/significant effects of urban/rural areas
• Count model (Poisson)(incidence rate of using mobile devices in the 8 waves)– Lower incidence rates for divorced/widowed/single compared to
participants with a partner– The larger the households, the lower the incidence rate– Blue collar workers, but also cadres, have lower incidence rate
than white collar workers– No significant effects of age, education, urban/rural areas
Study 2: Results: Mobile Survey Propensity
Summary of Findings
• Rate of mobile participants? Non ignorable rate of mobile devices used when conducting online surveys (about 6-8% in our two studies)
• Participation patterns? – Study 1: Higher drop-out rates for Mobile OS within
individual studies (but more data for drop-outs).– Study 2: Mainly inconsistent use of mobile devices
(participating /w mobile device does not guarantee later mobile participation).
• Systematic differences between mobile/non-mobile participants?Across both studies, associations with age (younger age groups more likely to use mobile) and gender (men more likely) found. Presumably many more predictors relevant (Study 2, but see limitations).
Discussion
• Major limitation: No data on everyday smartphone / tablet usage in both data sets available (therefore: zero-inflated count model). Replication within the GESIS Panel: Mobile usage assessed during recruitment interview and updated regularly.
• Actionable recommendations? Given these results, how to handle the issue of participants using Mobile OS devices to respond to online surveys?– Exclusion?– Adjustment of survey contents, measurement
options, and implementation procedures? How?
Thank you!
Appendix
Study 1: Identification of ´Mobile´
Mobile OS Desktop/Laptop OS
AndroidBlackberry OS
Apple iOSSymbian
Symbian OSWindows Phone OS
Chrome OSLinux
Apple OS XWindows 2000
Windows 7Windows 98Windows NT
Windows Server 2003Windows VistaWindows XP
unknown
Study 1: Results: Flash & Drop-Out Rates
Mobile OS Desktop/Laptop OS
Without Flash(3 studies)
7.2%(5.3/9.7)
3.2%(1.9/4.4)
With Flash(4 studies)
27.7%(19.1/36.2)
14.1%(12.5/15.6)
Study 2: Zero-inflated Poisson Regression Model