robert voogt dutch ministery of social affairs and employment (formerly of the university of...
Post on 22-Dec-2015
221 views
TRANSCRIPT
Robert VoogtDutch Ministery Of Social Affairs and
Employment(formerly of the University Of Amsterdam)
Nonresponse in survey research: why is it a
problem?
2
Overview• What is nonresponse, why is it a
problem and why does the traditional way of correcting for nonresponse not solve the problem
• Overview of general correction techniques
• An alternative approach to correct for nonresponse bias
• Real life illustration
3
What is nonresponse, why is it a problem and why does the traditional way of correcting
for nonresponse not solve the problem?
4
Survey research
• Population is sampled• Sample is a good representation of
population when good sample techniques are used
• Not all sample elements will respond
5
Unit vs Item nonresponse
• Some are not reached, others refuse or are not sending back the questionaire: unit nonresponse
• Some who do answer the questionnaire do so incompletely: item nonresponse
6
MCAR, MAR, MNAR
3 general nonresponse mechanisms can be distinguished
• MCAR: Missing Complety At Random• MAR: Missing At Random• MNAR: Missing Not At Random
7
Missing Completely At Random (MCAR)
• Conditional distribution M given the survey outcomes Y and survey design variables Z. Let f(M|Y,) denote the distribution, with the unknown parameters.
• If MCAR: f(M|Y,Z,) = f(M|) for all Y,Z,
• Not a realistic assumption
8
Example MCAR
• Taking a random subsample of a group of nonrespondents
• If random subsample of nonrespondents is analysed (after obtaining answers of all of them), the nonsampled nonrespondents can be said to be MCAR
• So correction methods using the MCAR assumption can be used
9
Missing At Random (MAR)• MAR: f(M|Y,Z,) = f(M|Yobs,Z,) for all Ymis,
• where Yobs denotes all the observed survey data
• This means that missingness depends on the observed variables, the observed values of incomplete variables or on the design variables, but not on the variables or values that are missing
10
Example MAR• For both respondents and
nonrespondents we know their level of education
• Respondents who share the same value of level of education have the same distribution on the unobserved variables
• Most survey nonrespondent adjustment methods assume MAR
11
Not Missing At Random (NMAR)
• NMAR: f(M|Y,Z,) = f(M|Yobs,Ymis,Z,) for all Yobs,Ymis,
• This means that missingness depends on missing values after conditioning on the observed data
• To get an unbiased distribution M, a joint model of the data and the nonresponse mechanism is necessary
12
Example MNAR
• For both respondents and nonrespondents we know their level of education
• Given the level of education nonresponse on the variables of interest is not random
• This means it is not sufficient to use only level of education to correct for nonresponse bias.
13
Nonresponse biasIf nonresponse is not a result of
design, almost always NMAR is the case, with data biased by nonresponse as a result.
The amount of nonresponse bias is dependent on:
1. the correlation between the target variable(s) and the nonresponse mechanism;
2. the level of nonresponse.
14
Nonresponse bias
withYk: the score of element k in the population on
the target variabelek: probability of response of element k in the
population when contacted in the sampleC(,Y): population covariance between response
probabilities and the values of the target variable
1
1( ) ( ( )
N
k kk
C Y Y YN
15
Nonresponse bias
with
• (Yk-Y): the difference between the population score and the score of element k on the variabele of interest
• (k-: the difference between the mean probability to respond and the probability to respond of element k
• It follows from this equation that the response level in itself does not say everything: the amount of bias depends on the relation between the first and second part of the equation
1
1( ) ( ( )
N
k kk
C Y Y YN
16
Traditional correction methods• Use population information to compare to
the respondent group with the population• Use information that is available for both
respondents and nonrespondents• Use information about the difficulty to
obtain data from the respondents
• In fact, the assumption is that the data are MAR, given the values of the variables of which population information or information about the nonrespondents is available
17
Traditional correction methods• No information about the difference
on the variables of interest between the respondents and nonrespondents
• No information about the difference in response probabilities between sample elements that score different on the variables of interest
• So there is no reason why this way of correcting should work
18
Overview of general correction techniques
19
Different correction techniques
• Weighting: assigning each observed element an adjustment weight
• Extrapolation: respondents who are most like the nonrespondents are used for correction
• Imputation: missing values are substituted by estimates
20
Weighting• Weighting: assigning each observed
element with an adjustment weight• Sample elements that belong to
groups that seem underrepresented on the variables used in the weighting will have a high adjustment weight
• Sample elements that belong to groups that seem overrepresented among the respondents will have a low adjustment weight
21
Weighting Example
• Question: Have you ever visited Lugano? (Y/N)
• Population information available about age (18-30 ) (31-64 ) (65-older)
• Comparison of respondents and population
• Weighting
Age Resp Popul
Weight
18-30 20% 30% 30/20=1.5
31-64 70% 50% 50/70=0.7
65+ 10% 20% 20/10=2.0Lug 18-
3031-64
65+
Unw
W*
Yes 20%(4)
50%(35)
10%(1)
40%(40)
33%(33)
No 80%(16)
50%(35)
90%(9)
60%(60)
67%(67)
N 20 70 10 100 100Yes: 4*1.5 + 35*.7 + 1*2.0=6+24.5+2=32.5
No: 16*1.5 + 35*.7 + 9*2.0=24+24.5+18=66.5
22
Extrapolation
• Central idea: some groups of respondents are more like the nonrespondents than others are
• For example, sample elements that first refused, but when contacted for the second time, were persuaded to participate, can be used as proxies for the final refusals
23
Extrapolation Example• Question: Have you
ever visited Lugano? (Y/N)
• Two respondent groups: early respondents and late respondents
• Calculate the distribution among the nonrespondents using the last respondent method
Lug R1 R2 TR NR TS
Yes 48%(29)
28%(11)
40%(40)
20%(10)
33%(50)
No 52%(31)
72%(29)
60%(60)
80%(41)
67%(100
)
N 60 40 100 50 150
Last respondent: L=A2+(A2-A1) (X2-X1/X2), with:
L: theoretical last respondent
A: % response to an item in a wave
X: cumulative % respondents at the end of a wave
L = 50+(50-40) (67-40/67) = 50+*.40=18%
24
Imputation• Imputation: missing values are
substituted by estimatesDifferent methods of imputation:• Single Imputation: for each variable one
value is imputed• Hot Deck Imputation: a missing value is
replaced by an observed value of a comparable respondent
• Multiple Imputation: for each variable several values are imputed; in this way the uncertainty that imputation brings with it is also taken into account
25
Hot Deck Imputation Example• Divide the respondents into homogenous
groups. For exampe, by using CHAID.• CHAID recursively partitions a sample into
groups so that the variance of the dependent variable is minimized within groups and maximized among groups
• Link each nonrespondent to the group it fits in best
• Substitute the values of a random respondent from the same group as the value of the nonrespondent
26
Hot Deck Imputation Example, part 2
CHAID finds groups:
age 18-30,31-64/low
education,
31-64/high education,
65+/maleand65+/female
Grp R HDI NR TS
18-30 20% (4) 25*.20 =5
9
31-64/low 33% (10)
4* .33 =1
11
31-64/high
63% (25)
1* .63 =1
26
65+/male 20% (1) 8* .20 =2
3
65+/female
0% (0) 12* .0 =0
0
% Lug Yes 40% (40)
18%(9) 33% (49)
100 50 150
27
Multiple Imputation Example
• For each case, 5 values for each missing variabele are calculated, using a regression equation and adding a random error term
• These values are combined in one single value, for example, by taking the mean
• The variance will take the uncertainty due to the imputed value into account by combining the within imputation variance (the variance of each estimated data set) and the between imputation variance (in which all 5 data sets are used)
28
Multiple Imputation Example, part 2
Imp1 Imp2 Imp3 Imp4 Imp5 Mean
NR 1 .41 .56 .34 .62 .44 .47
NR 2 .67 .77 .81 .56 .64 .69
NR 3 .28 .11 .07 .15 .22 .17
NR 4 .02 .10 .06 .23 .09 .10
….
NR 50 .21 .32 .46 .16 .20 .27
TNR .33
Percentage that has visited Lugano
29
An alternative approach to correct for nonresponse
30
Key to succes of correction methods• The information used in the
correction method• The correction method must model
the nonresponse mechanism• The variables used in correction
should have a relation with:– the variables of interest– the probability to respond of a
sample element
31
Central Question Method(Betlehem & Kersten, 1984)
• Nonrespondents are asked to answer one (or more) questions central to the subject of the study
• The central questions are believed to have a strong relation with both the nonresponse process and the subject of the study
• Central questions are used in correction
32
Central Question Example
• Central Question: Have you ever visited Switzerland? (Y/N)
• Question of interest: Have you ever visited Lugano? (Y/N)
• Comparison of respondents and non-respondents
• Weighting as correction technique
Lug CQ:Y CQ:N Unw W*
Yes 67%(40)
0%(0)
40%(40)
29%(29)
No 33%(20)
100%(40)
60%(60)
71%(71)
N 60 40 100 100
Yes: 40*.72 + 0*1.43 = 28.8 + 0 = 29
No: 20*.72 + 40*1.43 = 14.4 + 57.2 = 71
CQ Resp Nonr TS Weight
Yes 60% 10% 43% 43/60=0.72
No 40% 90% 57% 57/40=1.43
N 100 50 150
33
Real Life Illustration
34
Illustration
• Election study• High levels of nonresponse• External information available to
test the succes of the correction procedures
35
Our research questions• Does nonresponse causes a problem
in election studies?• Is using background variables
sufficient or do we need central questions?
• Do different correction techniques lead to different results?
• Is it really necessary to recontact nonrespondents?
36
Data Collection
• City of Zaanstad, The Netherlands• N=995; 901 used• Recontacting refusals• Mixed mode data collection• Two central questions:
– Voted in 1998 national elections– Political interest
37
Response rateMethod N %
Telephone Complete question.
452 50.2
Central questions 81 9.0
Mail Complete question.
94 10.4
Central questions 27 3.0
Face-to-face
Complete question.
158 17.5
Central questions 37 4.1
Nonresponse
52 5.8
Total sample 901 100
38
Does nonresponse cause problems?
We distinguish four groups:• Response at first contact (470)• Response after two contacts (76)• Response after three or four contacts
(158)• Nonrespondents (including those who
answered the central questions) (197)
39
Comparison of response groups
R1 R2 R3 NR
Voted nat. elections 86 70 60 62
Voted prov. elections 47 46 25 29
Interested in politics 79 76 55 27
Voting not important 9 17 38 -
Conclusion: nonresponse bias is present
40
How to correct?
Using the Central Question Procedure and compare it with more traditional correction methods
Two central questions:• Voted at national elections (0-1) –
from election lists (so no response bias)
• Political interest (0-1) – from short nonresponse questionnaire
41
Correction methods• Weighting by background variables /
+ central questions• Extrapolation• Hot Deck Imputation by background
variables / + central questions• Multiple Imputation by background
variables / + central questionsfor response levels of 52 % and 78 %
42
Weighting• On background variables: age,
ethnicity, gender, household composition, education, residential value, number of years living in current residence, social cohesion in neighborhood; using an iterative procedure
• As above plus validated voter turnout national elections 1998 and political interest (central questions)
43
Extrapolation
• Last Respondent Method
44
Hot Deck Imputation
• Obtain subgroups by using CHAID• Assign nonrespondents to the groups• Decide exact value to be imputed
using a regression model (multiple imputation)
• For background variables / background variables and central questions
45
Multiple Imputation• Use AMELIA (King et al., 1998) to calculate 10
discrete imputation values for each variable• Calculate the mean distribution by summing
the 10 proportions of each of the categories of the variable and divide it by 10
• Compute variance to take both within- and between-imputation variance into account
• For background variables / background variables and central questions
46
Dependent variables
• Voted at national elections• Voted at provincial elections• Self-reported political interest• Importance of voting
47
Results for weighting52%
78%
Rsp
BG CQ Rsp BG CQ TS
Voted national
85.5
83.3 74.5
78.0
77.5
74.5
74.5
Political Interest
78.8
78.0 65.2
73.0
72.1
65.2
65.2
Voted provincial
47.4
46.1 40.6
42.3
42.0
40.1
39.5
Importance Voting
69.5
68.7 63.4
59.7
59.5
56.5
-
Rsp: Respondents, BG: Background variables
CQ: Central Questions, TS: Total sample
48
Compare different methodsRsp W HD
IMI EX W HDI MI TS
78%
BG BG BG CQ CQ CQ
Voted National
78.0
77.5 77.8
75.7
73.0 74.5 75.4 75.1 74.5
Political Interest
73.0
72.1 72.2
71.7
69.2 65.2 65.6 64.6 65.2
VotedProvincial
42.3
42.0 42.8
42.5
39.0 40.1 41.1 41.2 39.5
Importance Voting
59.7
59.5 59.7
57.8
52.7 56.5 56.2 56.1 -
Rsp: Respondents, W: Weighting, HDI: Hot Deck Imputation, MI: Multiple Imputation. EX: Extrapolation, BG: Background variables, CQ: Central Questions, TS: Total sample
49
Relations: regression turnout provincial elections52 BV CQ 78 BV CQ
Resp W W HDI MI Resp W W HDI MI TS
VtNat
* * * * * * * * * * *
Age * * * * * * * * * * *Urb
Sex *Educ * * * * * * *Ethn
Value
* *
Mobil
* * *
Cohe
50
Conclusions• Using cental questions lead to better
estimates than only using background variables
• Higher response levels lead to better estimates
• All correction techniques perform equally well: the information used in the correction is more important than the technique used
• Correcting bias in regression parameters is less succesful
51
Recommedations• Always reapproach nonrespondents, to try
to reach a response level of 75 %• Always ask (a sample of) nonrespondents
to answer a small number of central questions
• Always try to get as much information as possible from external sources
• The technique used is not so important – simple techniques perform equally well as more complex ones.