dr pamela campanelli survey methods consultant chartered statistician chartered scientist
Post on 13-Feb-2016
23 Views
Preview:
DESCRIPTION
TRANSCRIPT
Dr Pamela CampanelliSurvey Methods Consultant
Chartered StatisticianChartered Scientist
The Questionnaire Design Pitfalls of Multiple Modes
AcknowledgementsOther main members of UK “Mixed Modes and Measurement Error” grant team:Gerry Nicolaas Ipsos MORIPeter Lynn University of EssexAnnette Jäckle University of EssexSteven Hope University College London
Grant funding from:UK Economic and Social Research Council (Award RES-175-25-0007)
Larger Project Looked for Evidence of Mode Differences by
• Question Content• Question Format
• Type of task• Characteristics of the task• Implementation of the
task
• Made recommendations
Today a few highlights
Sensitive QuestionsMixed Mode Context
• Very well-known• Sensitive questions prone to social desirability effects
in interviewer modes (see Tourangeau and Yan, 2007; Kreuter, Presser and Tourangeau, 2008)
• But not all questions (Fowler, Roman, and Di, 1998)
• Difference by time frame (Fowler, Roman, and Di, 1998)
Modes to UseSA
(Self-administered)
Recommendations• If mixed mode design includes
• F2F interviews, ask sensitive questions in a paper SA form or use CASI
• TEL interview, pre-test sensitive questions across modes that will be used to see if there are differences
Non-Sensitive: Factual Versus Subjective (1)
Mixed Mode Context
• Subjective questions more prone to mode effects than factual questions (see Lozar Manfreda and Vehovar, 2002; Schonlau et al, 2003)
• But factual questions also susceptible (Campanelli, 2010)
• Subjective scalar questions can be prone to TEL positivity bias
TEL (and F2F) Positivity BiasDillman et al (2009) - aural versus visual effect• TEL Rs giving more extreme positive answers Ye et al (2011) - TEL Rs giving more extreme positive answers• But found that F2F was like TEL• Concluded caused by a MUM effect
Hope et al (2011) – TEL Rs giving more extreme positive answers• But no trace of this in F2F (with a showcard and
without a show card)
Thus, actual cause for the TEL positivity bias is still unclear
Non-Sensitive: Factual Versus Subjective (2)
Modes to Use
F2FTEL?SA
Recommendations
Factual Questions• Use Dillman’s uni-mode principles and test to
see if there are differences across modes Subjective scalar questions• Avoid TEL, if possible, due to TEL positivity bias• Test F2F to see if positivity bias is present
Inherently Difficult Questions (1)
General Questionnaire Design Context
Inherent difficulty: Question is difficult due to conceptual, comprehension and/or recall issues
• Survey satisficing should be greater for inherently difficult questions (Krosnick, 1991)
• But this is not true for all inherently difficult questions
(Hunt et al, 1982; Sangster and Fox, 2000; Nicolaas et al, 2011)
Inherently Difficult Questions (2)
N56y. What are the things that you like about your neighbourhood? Do you like your neighbourhood because of its community spirit? Yes……. 1 No…….. 2
N57y. Do you like your neighbourhood because it feels safe? Yes……. 1 No…….. 2
Etc.
EXAMPLE:
Nicolaas et al (2011)
Inherently Difficult Questions (3)Modes to Use
F2FTEL?SA?
Recommendations
General Questionnaire Design Context • In general, before use, test questions that are
inherently difficult to see how feasible the questions are for Rs • (Testing can be done with cognitive interviewing or
Belson’s (1981) respondent debriefing method)
Mixed Modes Context• In mixed mode design, pre-test questions with
inherent difficulty across modes that will be used to see if there are differences
Mark All That Apply vs. Yes/No for Each (1)
Mark at that apply Yes/No for eachThis card shows a number of different ways for reducing poverty. In your opinion, which of the following would be effective in reducing poverty? MARK ALL THAT APPLY. Increasing pensions 1 Investing in education for children 2 Improving access to childcare 3 Redistribution of wealth 4 Increasing trade union rights 5Reducing discrimination 6 Increasing income support 7Investing in job creation 8None of these 9
Next are a number of questions about different ways for reducing poverty. In your opinion, which of the following would be effective? Would increasing pensions reduce poverty? Yes 1No 2 Would investing in education for children reduce poverty? Yes 1No 2 Etc.
Mark All That Apply vs. Yes/No for Each (2)
General Questionnaire Design Context
‘Mark all that apply’ is problematic
• Sudman and Bradburn (1982)
• Rasinski et al (1994), Smyth et al (2006) and Thomas and Klein (2006)
• Thomas and Klein (2006)
• Smyth et al (2006)
• Nicolaas et al (2011)
Mark All That Apply vs. Yes/No for Each (3)
Mixed Mode Context
• Smyth et al (2008) - student sample
• Nicolaas et al (2011) - probability sample of the adult population
• More research needed
Mark All That Apply vs. Yes/No for Each (4)Mark all that applyModes to Use
F2F?SA?
Recommendations• The ‘mark all that apply’ format is prone to lower
reporting of items, quicker processing time and primacy effects. Therefore probably best to avoid.
• However, it may be less likely to show mode effects in a mixed mode design (F2F with showcard versus SA).
Modes to UseF2FTELSA
Recommendations• The ‘Yes/No for each’ format is strongly supported as
superior to ‘mark all that apply’ by Smyth et al (2006, 2008). But• It can add to the time taken to complete a
questionnaire • Long lists of items should be avoided to reduce
potential R burden• The results from Nicolaas et al (2011) suggest that the
format should be tested across modes before use
Yes/No for each
Ranking versus Rating (1) Ranking Battery of Rating QuestionsWhat would you consider most important in
improving the quality of your neighbourhood? Please rank the following 7 items from 1 (meaning most important) to 7 (meaning least important).
Less traffic □Less crime □More / better shops □Better schools □More / better facilities for leisure activities □Better transport links □More parking spaces □
Next are a number of questions about improving your neighbourhood? How important would less traffic be for improving the quality of your neighbourhood? Very important 1Moderately important 2Somewhat important 3Or not important at all? 4 Etc.
Ranking versus Rating (2)
General Questionnaire Design Context
Ranking • Is difficult (Fowler, 1995) • Primacy effects (see Stern, Dillman & Smyth, 2007) • Better quality (see Alwin and Krosnick, 1985; Krosnick, 1999;
Krosnick, 2000).
Ranking versus Rating (3)
Mixed Modes Context • Rating more susceptible to non-differentiation in Web than
TEL (Fricker et al, 2005)
• Similarly, rating sometimes more susceptible to non-differentiation in Web or TEL than F2F (Hope et al, 2011)
• Ranking more susceptible to non-differentiation in Web than F2F (TEL not tested) (Hope et al, 2011)
Ranking versus Rating (4)
Modes to Use
F2FNOT TEL
SA?
Recommendations
Avoid use of ranking in mixed mode studies• Ranking could be considered for F2F surveys if the list is
short • Ranking is not feasible for TEL surveys (unless 4 categories
or less)• Ranking is often problematic in SA modes• Ranking with programme controls in Web may irritate or
confuse some Rs
Modes to Use
F2FTEL?SA ?
Recommendations
• Avoid long sequences of questions using the same rating scale in mixed mode designs that include Web and possibly TEL
• Could try rating task followed by ranking of the duplicates (except in postal where skip patterns would be too difficult)
Rating
Ranking
Agree/Disagree Questions
• For additional problems see Fowler (1995), Converse and Presser (1986), Saris et al (2010) and recent Holbrook AAPOR Webinar
Mixed Modes Context• Differences across modes were found with more acquiescence bias in
the interview modes and curiously, more middle category selection in SA (Hope et al, 2011)
General Questionnaire Design Context• Agree/Disagree questions are a
problematic format in all modes
• They create a cognitively complex task
• Are susceptible to acquiescence bias
Modes to UseShould not be used
in any mode
Recommendations• Avoid use of agree-disagree scales and use alternative
formats, such as questions with item specific (IS) response options
This neighbourhood is not a bad place to live. Strongly agree 1Agree 2Neither agree nor disagree 3Disagree 4Or strongly disagree? 5
Use of Middle Category (1)
And how satisfied or dissatisfied are you with street cleaning? Very satisfied 1Moderately satisfied 2Slightly satisfied 3Neither satisfied nor dissatisfied 4Slightly dissatisfied 5Moderately dissatisfied 6Very dissatisfied 7
Use of Middle Category (2)General Questionnaire Design Context • Kalton et al (1980)
• Krosnick (1991) and Krosnick and Fabrigar (1997)
• Schuman and Presser (1981)
• Krosnick and Presser (2010)
• Krosnick and Fabrigar (1997)
• O’Muircheartaigh, Krosnick and Helic (1999)
• Hope et al (2011)
Use of Middle Category (3)Mixed modes context
• More use of the middle category in visual (as opposed to aural) mode (Tarnai and Dillman, 1992)
• More selection of middle categories on end-labelled scales than fully labelled scales, but less so for TEL (Hope et al 2011)
• More use of the middle category in Web as opposed to F2F or TEL (Hope et al 2011)
Use of Middle Category (4)
Modes to Use
F2FTEL?SA?
Recommendations
• Probably best not to use middle categories with a mixed modes study with SA
• If mixed mode design includes • TEL interviews be cautious of the
use of end-labelled scales
OverallTypology of Questions
A classification of question characteristics relevant to measurement error
Question content Topic: behaviour, other factual, attitude, satisfaction, other subjective Sensitivity Inherent difficulty: conceptual, comprehension, recall
Question format
OpenClosed
Ratio/interval Ordinal NominalType of task Number
Date Short textual/
verbal
Unconstrained textual/verbal
Visual analogue scale
Agree/disagree Rating-unipolar Rating-bipolar Numeric bands Battery of rating
scales
Yes/no Mark all Ranking
Characteristics of the task
Number of categories Middle categories Full/end labels Branching
Implementation of question
Use of instructions, probes, clarification, etc. Edit checks DK/refused explicit or implicit Formatting of
response boxes Labelling of
response boxes
Size of answer box/text field
Delineation of answer space
Formatting of response lists Showcards
In Summary1) Mode is a characteristic of a question 2) Good questionnaire design is key to minimising
many measurement differences 3) But we are unlikely to eliminate all differences as
there are different types of satisficing in different modes
4) We need to do more to assess any remaining differences and find ways to adjust for these (more on this in the next few slides)
Assessing Mixed Mode Measurement Error (1)
Quality indicatorsFor example:• Mean item nonresponse rate • Mean length of responses to open question • Mean number of responses in mark all that
apply • Psychometric scaling properties • Comparison of survey estimates to a ‘gold’
standard (de Leeuw 2005; Kreuter et al, 2008; Voogt and Saris, 2005)• Although validation data often hard or
impossible to obtain• Etc.
Assessing Mixed Mode Measurement Error (2)
How was the mixed mode data collected? What are the confounding factors or limitations?• Random assignment
• R’s randomly assigned to mode (Nicolaas et al, 2011): But this is not always possible
• Random group changes mode during the interview (Heerwegh, 2009)
• In both cases non-compatibility can occur due to differential nonresponse bias
• R choses mode of data collection • May reduce nonresponse, but selection and
measurement error effects are confounded (Vannieuwenhuyze et al, 2010)
Assessing Mixed Mode Measurement Error (3)Ways to separate sample composition from mode effects
• Compare mixed mode data to that of a comparable single-mode survey (Vannieuwenhuyze et al, 2010)
• Statistical modelling: • Weighting (Lee, 2006) • Multivariate model (Dillman et al, 2009) • Latent variable models (Biemer, 2001)
• Propensity score matching (Lugtig et al, 2011) • Matching Rs from two survey modes which share
the same background characteristics • Identify Rs who are unique to a specific survey mode
and those who are found in both modes• May be a useful technique
Assessing Mixed Mode Measurement Error (4)
The size of effects between modes
Depends on the type of analyses, which
Depends on the type of reporting needed
For example:• Reporting of
• Means• Percentages for extreme categories• Percentages for all categories
We hope that today’s talk has given you. . .
• More understanding of the theoretical and practical differences in how Rs react to different modes of data collection
• More awareness of specific question attributes that make certain questions less portable across modes
• More knowledge and confidence in executing your own mixed modes questionnaires
Thank you all for listening
dr.pamela.campanelli@thesurveycoach.com
Complete table of results and recommendations available upon request
Appendix
Open Questions (1)
General Questionnaire Design Context - SA
Lines in text boxes versus an open box• Christian and Dillman (2004)• But Ciochetto et al (2006)
Slightly larger answer spaces (Christian and Dillman, 2004)
Option 1: Unconstrained textual/verbal open questions (i.e., fully open questions)
Open Questions (2)Option 1: Fully open questions (continued)
Mixed Mode Context• TEL Rs give less detailed answers to open-ended questions than
F2F Rs (Groves and Kahn, 1979; Sykes & Collins, 1988; de Leeuw and van der Zouwen, 1988)
• Paper SA Rs give less complete answers to open-ended questions than F2F or TEL Rs (Dillman, 2007; de Leeuw,1992, Groves and Kahn, 1979)
• Web Rs provide 30 more words on average than paper SA Rs (Schaeffer and Dillman, 1998)
• Positive effects of larger answer spaces may also apply to interview surveys (Smith, 1993; 1995)
Open Questions (3)
Modes to Use
F2FTELSA?
Recommendations
• If mixed mode design includes SA,• Minimise the use of open questions (as
less complete answers are obtained)
• Pre-test SA visual layout 1) To ensure that the question is
understood as intended2) To check if there are differences across
modes
Option 1: Fully open questions (continued)
Open Questions (4)
General Questionnaire Design Context - SA
• Small changes in visual design can have large impact on measurement
• Examples • Couper, Traugott and Lamias (2001)
• Smith (1993; 1995)
• Dillman et al (2004)
• Martin et al (2007)
Option 2: Open question requiring a number, date, or short textual/verbal response
Open Questions (5)Option 2: Short number, date or textual/verbal response (continued)
Mixed Modes Context Modes to Use
F2FTELSA?
Recommendations
• Test SA visual layout 1) To ensure that the question is
understood as intended2) To check if there are differences across
modes
End-labelled versus Fully-labelled (1)
On the whole, how satisfied are you with the present state of the economy in Great Britain, where 1 is very satisfied and 7 is very dissatisfied?
General Questionnaire Design Context
• Krosnick and Fabrigar (1997) suggest that fully-labelled scales are
• Easier to answer • More reliable and valid
• Two formats are not equivalent• Fully-labelled scales produce more positive responses
(Dillman and Christian, 2005; Campanelli et al, 2012)
• End-labelled scales have a higher percent of Rs in the middle category (Campanelli et al, 2012; not discussed in text but in tables of Dillman and Christian, 2005)
End-labelled versus Fully-labelled (2)Mixed Modes Context
• Although higher endorsement of middle categories on end-labelled scales
• Less true for TEL Rs (Campanelli et al, 2012)
Modes to Use
F2FTEL?SA
Recommendations
• Be careful of the use of end-labelled scales as these are more difficult for Rs
• If mixed mode design includes • TEL interviews be cautious of the use
of end-labelled scales
Branching versus No Branching (1)In the last 12 months would you say your health has been good or not good?
Good 1Not good 2
IF GOOD: Would you say your health has been fairly good or very good?
Fairly good 1Very good 2
IF NOT GOOD: Would you say your health has been not very good or not good at all?
Not very good 1Not good at all 2
General Questionnaire Design Context
• In TEL surveys, ordinal scales are often changed into a sequence of two or more branching questions in order to reduce the cognitive burden
• Krosnick and Berent (1993)
• Malhotra et al (2009)
• Hunter (2005)
• Nicolaas et al (2011)
Branching versus No Branching (2)Mixed Modes Context
• Nicolaas et al (2000) found more extreme responses to attitude questions in the branched format in TEL mode (but unclear whether more valid)
• Nicolaas et al (2011) found • Mode differences between F2F, TEL and Web, but with
but with no clear patterns• No mode difference for the non-branching format
• More research needed
Branching versus No Branching (3)Modes to Use
F2FTELSA
Recommendations
• As branching may improve reliability and validity, if used, it should be used across all modes • But testing is recommended to see if mode
differences are present
• Due to R non-compliance with skip patterns in paper SA1, Dillman (2007) recommends • Avoidance of branching questions in mixed mode
surveys that include a postal component• Instead reduce number of categories so that
branching is not required
1 Dillman (2007) shows that the skips after a filter question can be missed by a fifth of postal survey Rs
Implementation of task
Use of instructions, probes, clarifications, etc. (1)
Can I check, is English your first or main language? INTERVIEWER: If ‘yes', probe - 'Is English the only language you speak or do you speak any other languages, apart from languages you may be learning at school as part of your studies?' Yes - English only 1Yes - English first/main and speaks other languages 2No, another language is respondent's first or main language 3Respondent is bilingual 4
Use of instructions, probes, clarifications, etc. (2)
• It is common practice to provide interviewers with additional information that can be used if necessary to improve the quality of information from Rs
• Although not yet studied in mixed modes, it is likely that this may result in differences across modes in a study that uses SA alongside interviewer modes
Use of instructions, probes, clarifications, etc. (3)
Modes to Use
F2FTELSA
Recommendations
• Where possible, all instructions and clarifications should be added to the question for all modes (rather than being left to the discretion of the interviewer) or excluded from all modes
• Dillman (2007) recommends that
interviewer instructions be evaluated for unintended response effects and their use for SA modes considered
Don’t Know (1)What, if any, is your religion? None 1Christian 2Buddhist 3Hindu 4Jewish 5Muslim 6Sikh 7Another religion 8 (Spontaneous only) (Don’t know 98)(Refused 99)
General Questionnaire Design Context
• Offering explicit ‘don’t know’ response greatly increases cases in this category
• Particularly true for R’s with • lower educational attainment
(see Schuman and Presser, 1981; Krosnick et al, 2002)
• Common practice not to provide an explicit ‘don’t know’ in TEL and F2F
• In SA modes, the ‘don’t know’ option
tends to be either an explicit response option or it is omitted altogether
Don’t Know (2)
Mixed Mode Context
• Treating ‘don’t know’ differently in different modes may result in different rates of ‘don’t know’ across the modes
• Fricker et al (2005)
• Dennis and Li (2007) • Bishop et al (1980)
• Vis-Visschers (2009)
Don’t Know (3)Modes to Use
F2FTELSA
Recommendations
• Spontaneous ‘don’t know’ can be offered in mixed mode designs that include only interviewer administered modes (i.e., TEL & F2F).
• For mixed mode designs that include both
interviewer-administered and SA modes, it is generally recommended not to allow ‘don’t know’ as a response option.
• Further research is required to compare
spontaneous ‘don’t know’ in TEL and F2F with alternative methods of dealing with ‘don’t know’ in Web questionnaires (e.g. allowing questions to be skipped without further prompting).
• For questions where it is likely that many Rs may not
know the answer, explicit don’t knows should be used across all modes.
Full References (1)
Alwin, D., & Krosnick, J. (1985). The Measurement of Values in Surveys: A Comparison of Ratings and Rankings. Public Opinion Quarterly, 49(4), 535-552.
Belson, W. (1981). The Design and Understanding of Survey Questions. Aldershot, England: Gower.
Biemer, P. (2001). Non-response Bias and Measurement Bias in a Comparison of Face to Face and Telephone Interviewing. Journal of Official Statistics, 17(2), 295-320.
Bishop, G., Oldendick, R., Tuchfarber, A. & Bennett, S. (1980). Pseudo-Opinions on Public Affairs. Public Opinion Quarterly, 44 (2), 198-209.
Campanelli, P. (2010). Internal analysis documents from ESRC Survey Design and Measurement Initiative grant on Mixed Modes and Measurement Error.
Campanelli, P., Gray, M., Blake, M. and Hope, S. (2012). Suspicious and Non-Suspicious Response Patterns which Are and Are Not Problematic. Unpublished paper.
Full References (2)Christian, L. & Dillman, D. (2004). The Influence of Graphical and
Symbolic Language Manipulations on Responses to Self-Administered Questions. Public Opinion Quarterly, 68(1), 57-80.
Ciochetto, S., Murphy, E & Agarwal, A. (2006). Usability Testing of Alternative, Design Features for the 2005 National Census Test (NCT) Internet Form: Methods, Results, and Recommendations of Round-2 Testing. Human-Computer Interaction Memorandum #85, Washington, DC: U. S. Census Bureau (Usability Laboratory).
Converse, J. & Presser, S. (1986). Survey Questions: Handcrafting the Standardized Questionnaire. Thousand Oaks, California: Sage.
Couper, M., Traugott, M., & Lamias, M. (2001). Web Survey Design and Administration. Public Opinion Quarterly, 65(2), 230.
de Leeuw, E. & van der Zouwen, J. (1988). Data Quality in Telephone and Face-to-Face Surveys: a Comparative Meta-Analysis. In R. Groves et al. (Eds), Telephone survey methodology. Hoboken, New Jersey: Wiley.
Full References (3)de Leeuw, E. (1992). Data Quality in Mail, Telephone and Face to Face
Surveys. Dissertatie. T. T. Publikaties Amsterdam. de Leeuw, E.D. (2005). To Mix or Not to Mix Data Collection Modes in
Surveys. Journal of Official Statistics, 21(2), 233-255.Dennis, M. & Li, R. (2007). More Honest Answers to Web Surveys? A
Study on Data Collection Mode Effects. IMRO’s Journal of Online Research, published on 10/10/2007. http://ijor.mypublicsquare.com/view/more-honest-answers.
Dillman, D. (2007). Mail and Internet Surveys: The Tailored Design Method, 2rd edition. Hoboken, New Jersey: Wiley.
Dillman, D. & Christian, L. (2005). Survey Mode as a Source of Instability in Responses Across Surveys. Field Methods, 17(1), 30-52.
Dillman, D., Parsons, N. & Mahon-Haft, T. (2004). Cognitive Interview Comparisons of the Census 2000 Form and New Alternatives. Technical Report 04-030 of the Social and Economic Sciences Research Center, Washington State University, Pullman, Washington. http://www.sesrc.wsu.edu/dillman/papers/2004/connectionsbetweenopticalfeatures.pdf
Full References (4)Dillman, D., Smyth, J. & Christian, L.M. (2009). Internet, Mail and Mixed-
Mode Surveys: The Tailored Design Method, 3rd edition. Hoboken, New Jersey: Wiley.
Fricker, S., Galesic, M., Tourangeau, R. & Yan, T. (2005). An Experimental Comparison of Web and Telephone Surveys. Public Opinion Quarterly, 69(3), 370-392.
Fowler, F.J. (1995). Improving Survey Questions: Design and Evaluation. Thousand Oaks, California: Sage.
Fowler, F.J., Roman, A. & Di, Z. (1998). Mode Effects in a Survey of Medicare Prostate Surgery Patients. Public Opinion Quarterly, 62(1), 29.
Groves, R.M. & Kahn, R.L. (1979). Surveys by Telephone: A National Comparison with Personal Interview. New York: Academic Press.
Heerwegh, D. (2009). Mode differences between face-to-face and web surveys: An Experimental Investigation of Data Quality and Social Desirability Effects. International Journal of Public Opinion Research, 21(1), 111-121.
Full References (5)Hope, S., Campanelli, P., Nicolaas, G., Lynn, P., Jäckle, A. (2011). The
role of the interviewer in producing mode effects: Results from a mixed modes experiment. ESRA Conference, 21 July 2011.
Hunt, S., Sparkman, R. & Wilcox, J. (1982). The Pretest in Survey Research: Issues and Preliminary Findings. Journal of Marketing Research, 19(2), 269-273.
Hunter, J. (2005). Cognitive Test of the 2006 NRFU: Round 1, Statistical Research Division Study Series, Survey Methodology #2005–07, U.S. Census Bureau. http://www.census.gov/srd/papers/pdf/ssm2005-07.pdf
Jäckle, A., Lynn, P., Campanelli, P., Nicolaas, G., & Hope, S. (2011). How and When Does the Mode of Data Collection Affect Survey Measurement? ESRA Conference, 21 July 2011.
Kalton, G., Roberts, J. & Holt, D. (1980). The Effects of Offering a Middle Response Option with Opinion Questions. Statistician, 29, 65-78.
Full References (6)
Kreuter, F., Presser, S. & Tourangeau, R. (2008). Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity. Public Opinion Quarterly, 72(5), 847-865.
Krosnick, J. (1991). Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys. Applied Cognitive Psychology, 5, 213-236.
Krosnick, J. (1999). Survey Research. Annual Review of Psychology, 50: 537–567.
Krosnick, J. (2000). The Threat of Satisficing in Surveys: The Shortcuts Respondents Take in Answering Questions. Survey Methods Newsletter, 20(1), 2000 (Published by the National Centre for Social Research, London, UK).
Krosnick, J. & Berent, M. (1993). Comparisons of Party Identification and Policy Preferences: The Impact of Survey Question Format. American Journal of Political Science, 37(3), 941-964.
Full References (7)Krosnick, J. & Fabrigar, L. (1997). Designing Rating Scales for Effective
Measurement in Surveys. In L. Lyberg et al. (Eds), Survey Measurement and Process Quality (pp. 141-164). Hoboken, New Jersey: Wiley.
Krosnick, J., Holbrook, A., Berent, M., Carson, R., Hanemann, W., Kopp, R., Mitchell, R., et al. (2002). The Impact of "No Opinion" Response Options on Data Quality: Non-Attitude Reduction or an Invitation to Satisfice? Public Opinion Quarterly, 66(3), 371-403.
Krosnick, J., Narayan, S. & Smith, W. (1996). Satisficing in Surveys: Initial Evidence. In M. T. Braverman & J. K. Slater (Eds.), Advances in Survey Research (Vol. 70, pp. 29-44). San Francisco: Jossey-Bass.
Krosnick, J. and Presser, S. (2010). Question and Questionnaire Design. Handbook of Survey Research, 2nd Edition. In James D. Wright and Peter V. Marsden (Eds). San Diego, CA: Elsevier.
Lee, S. (2006). Propensity Score Adjustment as a Weighting Scheme for Volunteer Panel Web Surveys. Journal of Official Statistics, 22(2), 329-349.
Lozar Manfreda, K. & Vehovar, V. (2002). Mode Effect in Web Surveys. In the proceedings from The American Association for Public Opinion Research (AAPOR) 57th Annual Conference, 2002.
Full References (8)Lugtig, P., Lensvelt-Mulders, G., Frerichs, R., & Greven, A. (2011).
Estimating Nonresponse Bias and Mode Effects in a Mixed-Mode Survey. International Journal of Market Research, 53(5).
Malhotra, N., Krosnick, J. & Thomas, R. (2009). Optimal Design of Branching Questions to Measure Bipolar Constructs. Public Opinion Quarterly, 73(2): 304-324.
Martin, E., Childs, H., Hill, J., Gerber, E., & Styles, K. (2007). Guidelines for Designing Questionnaires for Administration in Different Modes., US Census Bureau, Washington. http://www.census.gov/srd/mode-guidelines.pdf
Nicolaas, G., Campanelli, P., Hope, S., Jäckle, A. & Lynn, P. (2011). Is It a Good Idea to Optimise Question Format for Mode of Data Collection? Results from a Mixed Modes Experiment, ISER Working paper, no. 2011-31, ISER, University of Essex.
Nicolaas, G., Thomson, K. & Lynn, P. (2000). Feasibility of Conducting Electoral Surveys in the UK by Telephone. National Centre for Social Research.
O'Muircheartaigh, C., Krosnick, J. & Helic, A. (2001). Middle Alternatives, Acquiescence, and the Quality of Questionnaire Data, Irving B. Harris Graduate School of Public Policy Studies, University of Chicago, 2001.
Full References (9)Rasinski, K., Mingay, D., & Bradburn, N. (1994). Do Respondents
Really “Mark All That Apply” On Self-Administered Questions? The Public Opinion Quarterly, 58(3), 400-408.
Roy, L., Gilmour, G. & Laroche, D. (2004). The Internet Response Method: Impact on Canadian Census of Population Data. Statistics Canada Internal Report, 2004. http://www.amstat.org/sections/srms/proceedings/y2006/Files/JSM2006-000808.pdf
Sangster, R. & Fox, J. (2000). Housing Rent Stability Bias Study. Washington, DC: U.S. Bureau of Labor Statistics, Statistical Methods Division.
Saris, W., Revilla, M., Krosnick, J. & Shaeffer, E. (2010). Comparing Questions with Agree / Disagree Response Options to Questions with Item-Specific Response Options. Survey Research Methods, 4(1), 61-79.
Schaeffer, D. & Dillman, D. (1998). Development of a Standard E-mail Methodology. Public Opinion Quarterly, 62(3), 378-397 http://www.schonlau.net/publication/03socialsciencecomputerreview_propensity_galley.pdf
Full References (10)Schonlau, M., Zapert, K., Simon, L., Sanstad, K., Marcus, S., Adams, J.,
Spranca, M., et al. (2003). A Comparison Between Responses From a Propensity-Weighted Web Survey and an Identical RDD Survey. Social Science Computer Review, 22(1), 128-138.
Schuman, H. & Presser, S. (1981). Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, & Context. New York: Academic Press.
Smith, T. (1993). Little Things Matter: A Sampler of How Differences in Questionnaire Format Can Affect Survey Responses, GSS Methodological Report no. 78, Chicago: National Opinion Research Center.
Smith, T. (1995). Little Things Matter: A Sample of How Differences in Questionnaire Format Can Affect Survey Responses, paper presented at the annual meeting of the American Association for Public Opinion Research.
Smyth, J., Dillman, D., Christian, L. & Stern, M. (2006). Comparing Check-All and Forced-Choice Question Formats in Web Surveys. Public Opinion Quarterly, 70(1), 66-77.
Full References (11)Smyth, J., Christian, L. & Dillman, D. (2008). Does “Yes or No” on the
Telephone Mean the Same as “Check-All-That-Apply” on the Web? Public Opinion Quarterly, 72(1), 103-113.
Stern, M., Dillman, D. & Smyth, J. (2007). Visual Design, Order Effects, and Respondent Characteristics in a Self-Administered Survey. Survey Research Methods, 1(3), 121-138.
Sudman, S. & Bradburn, N.M. (1982). Asking Questions. San Francisco, California: Jossey-Bass.
Sykes, W. & Collins, M. (1988). Effect of Mode of Interview: Experiments in the UK. In R. Groves et al. (Eds), Telephone survey methodology. Hoboken, New Jersey: Wiley.
Tarnai, J. & Dillman, D. (1992). Questionnaire Context as a Source of Response Differences in Mail vs. Telephone Surveys. In: N. Schwarz, H. J. Hippler, and S. Sudman (Eds.) Order Effects in Social and Psychological Research. New York: Springer-Verlag.
Thomas, R. K. and Klein, J. D. (2006). Merely incidental?: Effects of response format on self-reported behaviour. Journal of Official Statistics, (22) 221-244.
Full References (12)Tourangeau, R. & Yan, T. (2007). Sensitive Questions in Surveys.
Psychological Bulletin, 133(5), 859-883. Vannieuwenhuyze, J., Loosveldt, G., and Molenberghs, G. (2010). A
Method for Evaluating Mode Effects in Mixed-Mode Surveys. Public Opinion Quarterly, 74(5), 1027-1045.
Vis-Visschers, R. (2009). Presenting 'don’t know' in web surveys. Paper presented at 7th Quest Workshop Bergen Norway, 18-20 May 2009.
Ye, C., Fulton, J. & Tourangeau, R. (2011). More Positive or More Extreme? A Meta-Analysis of Mode Differences in Response Choice. Public Opinion Quarterly, 75(2), 349–365.
top related