nonresponse bias and measurement error in web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014...
TRANSCRIPT
![Page 1: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/1.jpg)
1Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
Nonresponse Bias and Measurement Error in Web surveys
Presented at TSE 2015Baltimore, MD, USASeptember 21st, 2015
![Page 2: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/2.jpg)
Web survey response rates are generally lower than response rates of other self-
administered surveys such as mail surveys (Couper, 2001).
Web survey researchers have established various strategies to reduce nonresponse
rates: appealing invitation e-mails, e-mail reminders, incentives etc. (Fan & Yan, 2010).
Substantial nonresponse has the potential to induce nonresponse bias.
However, incorporating nonrespondents into Web surveys does not necessarily reduce
nonresponse bias and might cause additional measurement error.
Less motivated respondents might provide less accurate responses.
Background
2Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
![Page 3: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/3.jpg)
Research Questions
3Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
Do nonrespondents have an impact on data quality?
Do initial nonrespondents induce nonresponse bias?
Do survey break-offs induce nonresponse bias?
Should survey break-offs be incorporated into Web surveys?
Does convincing survey break-offs to participate in Web surveys cause additional measurement
error?
![Page 4: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/4.jpg)
Data BasisStudies of University Applicants
4Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
Study 1 Study 2 Study 3 Study 4
Target population: University applicants
Field phase:July/August
2012July/August
2013July/August
2014July/August
2015
Completes*: n=5,977 n=7,395 n=5,996 n=4,034
AAPOR RR6: 32% 40% 35% 24%
No. of Questions: 43 41 47 44
No. of Pages: 38 35 39 36
* Respondents who answered 50 percent or more of all applicable survey questions
![Page 5: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/5.jpg)
3240 34
23
6473 71 67
1814
14
11
3627 29 33
51 46 5266
0
10
20
30
40
50
60
70
80
90
100
Study 1(2012)
Study 2(2013)
Study 3(2014)
Study 4(2015)
Study 1(2012)
Study 2(2013)
Study 3(2014)
Study 4(2015)
in p
erce
nt
(%)
Completes* Questionnaire Breakoff Initial Nonresponse
Gross Sample Net Sample
Response & Nonresponse RatesStudy 1-4
5Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
* Respondents were excluded who completed the questionnaire but answered less than 50 percent of all applicable survey questions. Respondents
who answered 50 percent or more of all applicable survey questions but did not complete the questionnaire were counted as survey break-offs.
![Page 6: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/6.jpg)
Device used to open the surveyStudy 1-4
6Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
84%
75%
69%
63%
16%
25%
31%
37%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
2012 2013 2014 2015
Desktop Mobile
![Page 7: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/7.jpg)
Survey Breakoff RatesStudy 1-4
7Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
36%
27%29%
33%34%
24%26%
30%
45%
37% 36%38%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
2012 2013 2014 2015
percen
tag
eo
fth
en
et
sam
ple
Breakoff (total) Breakoff (desktop) Breakoff (mobile)
![Page 8: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/8.jpg)
64 3 2
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35% o
fth
en
et
sam
ple
last page
Study 2 (2013)
%
cum %
75
2
8
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
% o
fth
en
et
sam
ple
last page
Study 1 (2012)
%
cum %
Breakoff by Page NumberStudy 1 & 2
8Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
open-e
nded
sin
gle
response
sin
gle
response
sin
gle
response
welc
om
epage
list-
sty
le o
pen
-ended
grid
welc
om
epage
![Page 9: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/9.jpg)
Breakoff by Page NumberStudy 3 & 4
9Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
84
25
2
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36% o
fth
en
et
sam
ple
last page
Study 4 (2015)
%
cum %
64
6
0
5
10
15
20
25
30
35
40
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39% o
fth
en
et
sam
ple
last page
Study 3 (2014)
%
cum %
welc
om
epage
we
lco
me
page
sin
gle
response
sin
gle
response
grid
multip
le r
esponse
grid g
rid
![Page 10: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/10.jpg)
Standarad ErrorStudy 1-4: grid question
10Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
Item 1
Item 2
Item3
Item 4
Item 5
Item 6
Item 7
Item 8
Item 9
Item 10
Item11
2012 (15/38)
SEMEAN (c) .006 .016 .010 .008 ,016 .013 .015 .012
SEMEAN (c+b) .006 .015 .010 .008 .015 .012 .014 .012
2013 (16/35)
SEMEAN (c) .010 .011 .011 .013 .011 .010 .009 .010
SEMEAN (c+b) .009 .011 .010 .013 .010 .009 .009 .010
2014 (12/39)
SEMEAN (c) .011 .013 .012 .015 .011 .011 .010 .012
SEMEAN (c+b) .010 .012 .011 .014 .011 .010 .010 .011
2015 (6/36)
SEMEAN (c) .027 .017 .030 .042 .040 .017 .029 .014 .039 .029 .024
SEMEAN (c+b) .025 .016 .028 .039 .037 .016 .027 .013 .036 .027 .023
![Page 11: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/11.jpg)
Relative Nonresponse Bias
11Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
(Groves, 2006)
![Page 12: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/12.jpg)
Relative Nonresponse BiasSociodemographic Variables
12Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
-0,045
-0,011
0,005
0,060
-0,046
-0,011
0,006
0,064
-0,015
-0,0030,000
0,015
Gender(male)
Area of Studies(HSS)
Semester(1st)
Region(ZIP with 6)
2012
-0,033
-0,016
0,0130,017
-0,018-0,012
0,015 0,017
-0,002-0,011
0,001 0,004
Gender(male)
Area of Studies(HSS)
Semester(1st)
Region(ZIP with 6)
2013
-0,031 -0,032
0,010
0,033
-0,032
-0,045
0,011
0,034
-0,008 -0,014
0,001 0,008
Gender(male)
Area of Studies(HSS)
Semester(1st)
Region(ZIP with 6)
2014
-0,002
-0,054
0,0050,014
-0,001
-0,053
0,006 0,014
-0,004-0,020
0,0000,006
Gender(male)
Area of Studies(HSS)
Semester(1st)
Region(ZIP with 6)
2015
![Page 13: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/13.jpg)
Nonresponse Bias
13Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
(Groves, 2006)
![Page 14: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/14.jpg)
Nonresponse BiasSociodemographic Variables
14Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
-0,07
-0,02
0,01
0,09
-0,07
-0,02
0,01
0,10
-0,04
-0,010,00
0,04
Gender(male)
Area of Studies(HSS)
Semester(1st)
Region(ZIP with 6)
2012
-0,05
-0,03
0,02 0,03
-0,03
-0,02
0,03 0,03
-0,01
-0,04
0,00 0,02
Gender(male)
Area of Studies(HSS)
Semester(1st)
Region(ZIP with 6)
2013
-0,05 -0,05
0,02
0,05
-0,03
-0,07
0,02
0,06
-0,05
-0,05
0,000,03
Gender(male)
Area of Studies(HSS)
Semester(1st)
Region(ZIP with 6)
2014
0,00
-0,07
0,010,02
-0,01
-0,07
0,01 0,02
0,00
-0,06
0,000,02
Gender(male)
Area of Studies(HSS)
Semester(1st)
Region(ZIP with 6)
2015
![Page 15: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/15.jpg)
SummaryResults: NRB - Sociodemographic Variables
15Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
Bias estimates for all four variables were at a very low level.
The initial nonresponse bias and the survey breakoff bias add to each other.
In all studies the relative nonresponse bias for all variables is caused by the initial
nonresponse bias to a greater extent and this is not due to the high percentage of initial
nonrespondents.
Therefore, it seems to be more important to reduce initial nonresponse.
However, it is easier and more cost-effective to convince survey breakoffs to participate
in Web surveys by optimizing the questionnaire design.
Does incorporating survey breakoffs cause additional measurement error?
![Page 16: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/16.jpg)
Measurement ErrorIndicators
Degree of differentiation in grid questions(Kaminska et al., 2010; Barge & Gehlbach, 2011)
Number of item missing in grid questions(Barge & Gehlbach, 2011)
Response time of grid questions(Barge & Gehlbach, 2011)
Number of characters of answers to narrative open-ended questions(Smyth et al. 2009)
16Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
![Page 17: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/17.jpg)
Measurement Error(Degree of differentiation)
17Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
0,570,55 0,54
0,37
0,57 0,57 0,56
0,45
2012 2013 2014 2015
Breakoff
Completes
Significance n.s. p<.001 p<.01 p<.05
Question type grid grid grid grid
Question position 15/38 16/35 12/39 6/36
![Page 18: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/18.jpg)
Measurement Error(# of item missing)
18Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
0,06
0,04
0,06
0,19
0,00 0,00 0,00
0,14
2012 2013 2014 2015
Breakoff
Completes
Significance p<.001 p<.001 p<.001 n.s.
Question type grid grid grid grid
Question position 15/38 16/35 12/39 6/36
![Page 19: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/19.jpg)
Measurement Error(Response time in seconds)
19Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
4341
50
66
4649
51
68
2012 2013 2014 2015
Breakoff
Completes
Significance p<.01 p<.001 n.s. n.s.
Question type grid grid grid grid
Question position 15/38 16/35 12/39 6/36
![Page 20: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/20.jpg)
Measurement Error(# of characters)
20Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
128
64 6557
145
107
81
69
2012 2013 2014 2015
Breakoff
Completes
Significance p<.01 p<.001 p<.05 p<.05
Question type open-ended open-ended open-ended open-ended
Question position 4/38 9/35 7/39 8/36
![Page 21: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/21.jpg)
Measurement Error(Degree of differentiation)
21Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
Study 1 Study 2 Study 3 Study 4
Beta Sig Beta Sig. Beta Sig. Beta Sig.
Semester(semester 2+)
Region(area code ≠ 6)
Area of studies(engigneering/science)
Gender(female)
Device(mobile)
-.006
.004
-.054
.054
.007
.775
.359
.000
.000
.198
.024
-.004
-.014
.019
.028
.106
.220
.000
.000
.000
.005
-.002
-.022
.025
.021
.776
.667
.000
.000
.000
--
.057
.049
-.039
.032
--
.062
.092
.178
.271
Breakoff(complete)
.007 .347 -.018 .003 -.019 .003 -.080 .017
Constant
R²
.566
.068
.000 .526
.019
.000 .540
.022
.000 .399
.031
.000
Question type grid grid grid grid
Question position 15/38 16/35 12/39 6/36
Note. Reference categories in brackets. Linear regression model.
![Page 22: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/22.jpg)
Measurement Error(# of item missing)
22Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
Study 1 Study 2 Study 3 Study 4
Beta Sig Beta Sig. Beta Sig. Beta Sig.
Semester(semester 2+)
Region(area code ≠ 6)
Area of studies(engigneering/science)
Gender(female)
Device(mobile)
.00
-.00
-.00
-.01
-.00
.786
.711
.164
.096
.659
.00
.00
-.00
.00
.00
.717
.271
.563
.888
.924
.01
-.00
-.00
.00
-.00
.594
.598
.444
.821
.754
.16
.03
-.03
-.02
-.01
.574
.132
.123
.297
.586
Breakoff(complete)
.06 .000 .04 .000 .06 .000 .05 .049
Constant
R²
.00
.021
.831 -.00
.032
.690 -.00
.043
.758 -.00
.001
.996
Question type grid grid grid grid
Question position 15/38 16/35 12/39 6/36
Note. Reference categories in brackets. Linear regression model.
![Page 23: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/23.jpg)
Measurement Error(Response Time)
23Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
Study 1 Study 2 Study 3 Study 4
Beta Sig Beta Sig. Beta Sig. Beta Sig.
Semester(semester 2+)
Region(area code ≠ 6)
Area of studies(engigneering/science)
Gender(female)
Device(mobile)
-.04
.66
-3.96
1.18
-10.99
.992
.405
.000
.144
.000
2.42
1.71
-6.21
.00
-.04
.638
.109
.000
.999
.975
-5.79
1.91
-4.57
1.37
-2.90
.198
.031
.000
.126
.003
12.24
7.91
-3.60
-2.71
-8.83
.660
.000
.015
.072
.000
Breakoff(complete)
-3.88 .007 -8.73 .000 -.58 .690 -2.35 .233
Constant
R²
57.46
.019
.000 49.40
.006
.000 58.76
.007
.000 60.11
.016
.031
Question type grid grid grid grid
Question position 15/38 16/35 12/39 6/36
Note. Reference categories in brackets. Linear regression model.
![Page 24: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/24.jpg)
Measurement Error(# of characters)
24Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
Study 1 Study 2 Study 3 Study 4
Beta Sig Beta Sig. Beta Sig. Beta Sig.
Semester(semester 2+)
Region(area code ≠ 6)
Area of studies(engigneering/science)
Gender(female)
Device(mobile)
22.4
3.2
3.5
-12.0
5.3
.098
.224
.185
.000
.147
-14.3
-11.0
7.6
-23.6
20.6
.349
.000
.009
.000
.000
-14.8
2.3
-4.2
9.8
13.8
.341
.503
.217
.005
.000
--
.8
-3.6
.9
12.2
--
.790
.211
.759
.000
Breakoff(complete)
-25.0 .000 -39.6 .000 -14.0 .030 -9.5 .072
Constant
R²
121.2
.015
.000 124.4
.050
.000 80.4
.019
.000 61.9
.018
.000
Question type open-ended open-ended open-ended open-ended
Question position 4/38 9/35 7/39 8/36
Note. Reference categories in brackets. Linear regression model.
![Page 25: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/25.jpg)
SummaryResults: Measurement Error
25Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
Breakoffs differentiate significantly less than completes.
Breakoffs show significant higher item missing rates than completes.
Breakoffs take significantly less time to answer questions (2 of 4).
Breakoffs report shorter answers with a significantly lower number of characters than
completes.
Controlling the differences of the sample composition due to nonresponse these results
stay significant.
Therefore, differences can be interpreted as measurement error.
Incorporating breakoffs will increase the measurement error.
![Page 26: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/26.jpg)
Conclusion
26Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
Convincing survey break-offs to participate in Web surveys would increase response
rates, reduce the standard error of the mean and nonresponse bias to a small extent.
However, incorporating break-offs causes additional measurement error.
Further analyses need to examine if the quality of estimates becomes better with our
without including break-offs. Mean Squared Error
Convincing initial nonrespondents to participate in Web surveys would reduce
nonresponse bias to a greater extent but further research is necessary to analyze the
data quality of initial nonrespondents.
![Page 27: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/27.jpg)
Darmstadt University of TechnologyDepartment 02Institute of SociologyResearch MethodsDipl.-Soz. Anke Metzler
Erst-Ludwig-Str. 3/ Im Carree
64283 Darmstadt
Germany
Phone: +49 6151/16-70972Fax: +49 6151/16-72070
Thank you.
![Page 28: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/28.jpg)
intrest of
subject
contribute to
social changes
personal
developmenttalent help others
scientific
interest
contact with
people
problem
solving
Mean(c) 1,18 2,72 1,62 1,41 2,65 1,96 2,43 1,95
Mean(b) 1,22 2,79 1,69 1,47 2,71 2,01 2,43 2,02
Estimate(b) 1,18 2,80 1,64 1,45 2,58 1,89 2,32 1,85
Nonresponse BiasSubstantial Answers - Study 1 (2012)
28Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
NRB -0,04 -0,07 -0,07 -0,06 -0,06 -0,05 0 -0,07
Meas Bias 0 -0,08 -0,02 -0,04 0,07 0,07 0,11 0,10
grid question position 15/38:
![Page 29: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/29.jpg)
Nonresponse BiasSubstantial Answers - Study 2 (2013)
29Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
broad basic
knowledge
foreign
language
organisation
talent
computer
literacywriting skills speaking skills
working
independent
ability to
cooperate
Mean(c) 1,84 2,37 1,96 2,82 1,96 1,76 1,67 1,83
Mean(b) 1,81 2,31 1,98 2,75 1,96 1,71 1,67 1,78
Estimate(b) 1,84 2,24 1,80 2,47 1,90 1,61 1,52 1,62
NRB 0,03 0,06 -0,02 0,07 0 0,05 0 0,05
Meas Bias 0 0,13 0,16 0,35 0,06 0,15 0,15 0,21
grid question position 16/35:
![Page 30: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/30.jpg)
Nonresponse BiasSubstantial Answers - Study 3 (2014)
30Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
broad basic
knowledge
foreign
language
organisation
talent
computer
literacywriting skills speaking skills
working
independent
ability to
cooperate
Mean(c) 1,83 2,27 1,84 2,76 1,84 1,71 1,56 1,84
Mean(b) 1,85 2,21 1,98 2,81 1,89 1,73 1,60 1,86
Estimate(b) 1,92 2,24 2,02 2,75 1,97 1,88 1,62 1,98
NRB -0,02 0,06 -0,14 -0,05 -0,05 -0,02 -0,04 -0,02
Meas Bias -0,09 0,03 -0,18 0,01 -0,13 -0,17 -0,06 -0,14
grid question position 12/39:
cb
![Page 31: Nonresponse Bias and Measurement Error in Web …...0,010 0,033-0,032-0,045 0,011 0,034-0,008 -0,014 0,001 0,008 Gender (male) Area of Studies (HSS) Semester (1st) Region (ZIP with](https://reader030.vdocuments.mx/reader030/viewer/2022040409/5ec7a2f87967e9243207542f/html5/thumbnails/31.jpg)
informationof profs
informationof students
studentcounselling
(phone/email)
studentcounselling(personal)
studentcounselling(faculty)
flyer etc. newspaper homepage TUDayinformation
events of theTUD
online self-assessment
Mean(c) 2,47 2,18 2,40 2,55 2,50 2,34 2,75 2,10 2,59 2,46 2,40
Mean(b) 2,40 2,29 2,54 2,40 2,43 2,39 2,57 2,15 2,56 2,38 2,28
Estimate(b) 2,43 2,40 2,61 2,45 2,43 2,43 2,61 2,12 2,67 2,49 2,27
Nonresponse BiasSubstantial Answers- Study 4 (2015)
31Darmstadt University of Technology | Institute of Sociology | Research Methods | Anke Metzler and Marek Fuchs
NRB 0,07 -0,11 -0,14 0,15 0,07 -0,05 0,18 -0,05 0,03 0,08 0,12
Meas Bias 0,04 -0,22 -0,21 0,10 0,07 -0,09 0,14 -0,02 -0,08 -0,03 0,13
grid question position 6/36: