1. 1. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski (joint work with Willem Saris) U N I V E R S I T A T P O M P E U F A B R A Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
2. 2. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Measurement Representation Construct Measurement Response Edited data Validity Processing error Measurement error Inferential population Target population Sampling frame Sample Respondents Survey statistic Coverage error Sampling error Nonresponse error (Groves et al. 2004). Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
3. 3. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error ConclConstruct Measurement Response Edited data Validity Processing error Measurement error Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
4. 4. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Assume the step from construct to measurement is already acceptable Assume that the question measures an intended construct: respondent knows the answer, can interpret the question, ... Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
5. 5. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Assume the step from construct to measurement is already acceptable Assume that the question measures an intended construct: respondent knows the answer, can interpret the question, ... reaction of respondent to the question depends on some unobserved value/opinion, which is in turn a measure of construct. Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
6. 6. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Assume the step from construct to measurement is already acceptable Assume that the question measures an intended construct: respondent knows the answer, can interpret the question, ... reaction of respondent to the question depends on some unobserved value/opinion, which is in turn a measure of construct. We focus only on the degree to which the response is a good measure of this unobserved score/opinion, measurement error. Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
7. 7. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Assume the step from construct to measurement is already acceptable Assume that the question measures an intended construct: respondent knows the answer, can interpret the question, ... reaction of respondent to the question depends on some unobserved value/opinion, which is in turn a measure of construct. We focus only on the degree to which the response is a good measure of this unobserved score/opinion, measurement error. (NOT the degree to which the question is interpretable, measures some construct, etc.) Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
8. 8. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Assume the step from construct to measurement is already acceptable Assume that the question measures an intended construct: respondent knows the answer, can interpret the question, ... reaction of respondent to the question depends on some unobserved value/opinion, which is in turn a measure of construct. We focus only on the degree to which the response is a good measure of this unobserved score/opinion, measurement error. (NOT the degree to which the question is interpretable, measures some construct, etc.) Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
9. 9. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Reasons to study measurement error Reliability is an upper bound on validity; responses can never measure underlying construct better than the single indicator. Unreliability increases the variance of estimators: Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
10. 10. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Reasons to study measurement error Reliability is an upper bound on validity; responses can never measure underlying construct better than the single indicator. Unreliability increases the variance of estimators: var() = 1 2 /n, where (0, 1) is reliability Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
11. 11. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Reasons to study measurement error Reliability is an upper bound on validity; responses can never measure underlying construct better than the single indicator. Unreliability increases the variance of estimators: var() = 1 2 /n, where (0, 1) is reliability Unreliability reduces apparent strength of relationships between variables: Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
12. 12. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Reasons to study measurement error Reliability is an upper bound on validity; responses can never measure underlying construct better than the single indicator. Unreliability increases the variance of estimators: var() = 1 2 /n, where (0, 1) is reliability Unreliability reduces apparent strength of relationships between variables: xy = x y XY , where XY is the true correlation and xy the observed correlation. Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
13. 13. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Reasons to study measurement error Reliability is an upper bound on validity; responses can never measure underlying construct better than the single indicator. Unreliability increases the variance of estimators: var() = 1 2 /n, where (0, 1) is reliability Unreliability reduces apparent strength of relationships between variables: xy = x y XY , where XY is the true correlation and xy the observed correlation. Correlated measurement errors will make variables look more related than they really are; e.g. How many minutes does it take to... questions correlate partly because they are all asked in the same way. Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
14. 14. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Reasons to study measurement error Reliability is an upper bound on validity; responses can never measure underlying construct better than the single indicator. Unreliability increases the variance of estimators: var() = 1 2 /n, where (0, 1) is reliability Unreliability reduces apparent strength of relationships between variables: xy = x y XY , where XY is the true correlation and xy the observed correlation. Correlated measurement errors will make variables look more related than they really are; e.g. How many minutes does it take to... questions correlate partly because they are all asked in the same way. Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
15. 15. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Public health ranking: Correction of regression coefcients for Country Educationaldifferentialsinsubjectivehealthwith2s.e.interval -0.4-0.3-0.2-0.10.0 GR CZ PT SI FI HU PL SK LU ES EE DK DE TR IS NO CH BE IE FR UA AT NL SE Uncorrected regression coefficient Measurement error-corrected coefficient 0.82 0.85 0.78 0.73 0.56 0.75 0.71 0.81 0.86 0.85 0.95 0.84 0.91 0.70 0.81 0.87 0.81 0.82 0.92 0.85 0.91 0.81 0.93 0.99 Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
16. 16. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Design characteristics of questions Social Desirability Centrality Reference period Question formulation WH word used Use of gradation Balance of the request Encouragement Showcards present Showcards have pictures ... Emphasis on subjective opinion in request Information about the opinion of other people Use of stimulus or statement in the question Absolute or comparative judgment Response scale: basic choice Number of categories Labels full, partial, or no Labels full sentences Knowledge provided Survey mode ... Order of the labels Correspondence between labels and numbers of the scale Theoretical range of the scale Neutral category Number of xed reference points Dont know option Interviewer instruction Respondent instruction Extra motivation, info or denition available? Agree-disagree scale . . . (Saris & Gallhofer 2007) Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
17. 17. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Question design choices There are a great number of question design characteristics for which it has at some point been found or suggested that they inuence the response; Any question in a questionnaire represents a series of choices (conscious or not) on those characteristics: a method of asking the question; Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
18. 18. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Question design choices There are a great number of question design characteristics for which it has at some point been found or suggested that they inuence the response; Any question in a questionnaire represents a series of choices (conscious or not) on those characteristics: a method of asking the question; It is clear that what is a good method depends strongly on the topic, for example Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
19. 19. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Question design choices There are a great number of question design characteristics for which it has at some point been found or suggested that they inuence the response; Any question in a questionnaire represents a series of choices (conscious or not) on those characteristics: a method of asking the question; It is clear that what is a good method depends strongly on the topic, for example The frequency and importance of an event or series of events asked about determine: reasonable reference periods; reasonable categories - wide or deep; approximately or exactly (Tourangeau et al. 2000). Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
20. 20. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Question design choices There are a great number of question design characteristics for which it has at some point been found or suggested that they inuence the response; Any question in a questionnaire represents a series of choices (conscious or not) on those characteristics: a method of asking the question; It is clear that what is a good method depends strongly on the topic, for example The frequency and importance of an event or series of events asked about determine: reasonable reference periods; reasonable categories - wide or deep; approximately or exactly (Tourangeau et al. 2000). But are some methods generally better than others? Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
21. 21. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Question design choices There are a great number of question design characteristics for which it has at some point been found or suggested that they inuence the response; Any question in a questionnaire represents a series of choices (conscious or not) on those characteristics: a method of asking the question; It is clear that what is a good method depends strongly on the topic, for example The frequency and importance of an event or series of events asked about determine: reasonable reference periods; reasonable categories - wide or deep; approximately or exactly (Tourangeau et al. 2000). But are some methods generally better than others? If so, what about those methods makes them better? Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
22. 22. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Question design choices There are a great number of question design characteristics for which it has at some point been found or suggested that they inuence the response; Any question in a questionnaire represents a series of choices (conscious or not) on those characteristics: a method of asking the question; It is clear that what is a good method depends strongly on the topic, for example The frequency and importance of an event or series of events asked about determine: reasonable reference periods; reasonable categories - wide or deep; approximately or exactly (Tourangeau et al. 2000). But are some methods generally better than others? If so, what about those methods makes them better? Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
23. 23. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Question design choices But are some methods generally better than others? If so, what about those methods makes them better? Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
24. 24. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl Talk outline 1 Question design The inuence of the method Variation in inuence of the method 2 Modeling measurement error Denitions Formal model and assumptions 3 Estimating measurement error Design requirements Estimation of the model 4 Predicting measurement error Description of the data Meta-analysis of the MTMM experiments Program demonstration Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
25. 25. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl The inuence of the method The method inuences the answers Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
26. 26. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl The inuence of the method European Social Survey, 2002 Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
27. 27. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl The inuence of the method European Social Survey, 2002 Method A: ENTER START TIME: 1 TvTot CARD 1 On an average weekday, how much time, in total, do you spend watching television? Please use this card to answer. No time at all Less than hour hour to 1 hour More than 1 hour, up to1 hours More than 1 hours, up to 2 hours More than 2 hours, up to 2 hours More than 2 hours, up to 3 hours More than 3 hours (Dont know) A2 TvPol STILL CARD 1 And again on an average weekday, how much of your time watching television is spent watching news or programmes about politics and current affairs1 ? Still use this card. 00 GO TO A3 01 02 03 04 ASK A2 05 06 07 88 Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
28. 28. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl The inuence of the method European Social Survey, 2002 Method A: ENTER START TIME: 1 TvTot CARD 1 On an average weekday, how much time, in total, do you spend watching television? Please use this card to answer. No time at all Less than hour hour to 1 hour More than 1 hour, up to1 hours More than 1 hours, up to 2 hours More than 2 hours, up to 2 hours More than 2 hours, up to 3 hours More than 3 hours (Dont know) A2 TvPol STILL CARD 1 And again on an average weekday, how much of your time watching television is spent watching news or programmes about politics and current affairs1 ? Still use this card. 00 GO TO A3 01 02 03 04 ASK A2 05 06 07 88 Method B:! !""#$%&'()*%)+!)&,%$# ! -&.# !"#$"#$%&'$()&&*+$,-#./)#012.#340&-#4"#3/3$5-#+/#,/1#67&"+#)$32.4"(# 3&5&%464/"89 :## # # # # ,$/+%#/)#;!0#### ###?@A#BC@0# # # # # # # # -&1# #!"#$"#$%&'$()&&*+$,-#./)#012.#340&-#4"#3/3$5-#+/#,/1#67&"+#5463&"4"(#3/# 3.'$+4/8F :## # # # # ,$/+%#/)#;!G## ?@A#BC@G# # # # # # # # # # # # -&2# !"#$"#$%&'$()&&*+$,-#./)#012.#340&-#4"#3/3$5-#+/#,/1#67&"+#'&$+4"(#3. "&)67$7&'688 :## # # # # ,$/+%#/)#;!G# #?@A#BC@G# # # # #Predicting the quality of a survey question from its design characteristics: SQP Daniel Oberski
29. 29. Introduction Question design Modeling measurement error Estimating measurement error Predicting measurement error Concl The inuence of the method TV watching: method A versus method B 0 h