chapter 4 10/28/2013 12:151geog 3250. thoughts on researcher- administered surveys? keep a list of...

Chapter 4 10/28/2013 12:151Geog 3250

Thoughts on researcher- administered surveys? Keep a list of things not to do in a researcher- administered survey from this tongue-in-cheek video: 10/28/2013 12:15Geog 32502

Structured Interviews 10/28/2013 12:153Geog 3250 MethodInstrumentQuestion OrderContext Structured interview interview schedulestrictusually face-to- face Surveyquestionnairestrictvarious In-depth Interview Interview guide, and the interviewer flexible to keep it conversational usually face-to- face

Structured Interviews: errors and interviewer effects 10/28/2013 12:154Geog 3250 MethodAttitude towards interviewer effects ErrorsOpportunities for Clarification Structured interview minimize high: but clarify in exact same way in all situations Surveyminimize low: due to social distance In-depth Interview use to sustain rapport and keep the conversation meaningful No errors, just opportunities for deeper understanding high: clarification may be the source of meaningful conversation

Structured Interviews and errors Variation due to error can come from: Intra-interviewer variability: an interviewer is not consistent in asking questions or recording answers (with the same respondent or a different one). Inter-interviewer variability: lack of consistency in asking questions or recording answers between different interviewers. 10/28/2013 12:155Geog 3250

Structured Interviews: responses 10/28/2013 12:156Geog 3250 MethodResponse modeData recording Structured interview mostly closed-endedon the interview schedule Surveymostly closed-endedon the questionnaire In-depth Interview open-ended and conversationalField notes or audio recording Example of close-ended question; data recorded on the document

Interview Contexts Work with your neighbour to fill out this table - as many as you can think of per cell 10/28/2013 12:157Geog 3250 Method of Administration StrengthsLimitations Researcher-Administered Telephone e.g. with CATI Face-to-Face Self-Administered Drop-off, mail-back Hand-out, hand-back Email, 0nline

Conducting Structured Interviews: Basic Points Schedule Interviewers should be familiar with the interview schedule. Introduction Respondents should be provided with a credible rationale for the research. Rapport Interviewers should try to develop rapport with interviewees but refrain from being too chatty (unlike semi-structured or unstructured) 10/28/2013 12:158Geog 3250

Conducting Interviews: Basic Points, contd. Asking Questions The question should be asked exactly as stated. Small changes to wording can make a big difference Similarly, the answers should be recorded as exactly as possible. Questions should be asked in the order they are given on the interview schedule. 10/28/2013 12:159Geog 3250

Conducting Interviews: Basic Points, contd. show cards - to provide detailed information, e.g., Likert scale options appropriate if long list of possible answers; same set of answer categories applies to several questions. 10/28/2013 12:1510Geog 3250

Conducting structured interviews: Basic Points Leaving the interview The interviewer must remain professional and have a set response to disengage respondents so that they may do more interviews (i.e., if researcher is motivated to have large sample size) 10/28/2013 12:1511Geog 3250 Sometimes they want to be done and move on, sometimes it is the other way around

Conducting (survey) interviews: Basic Points Training Good interviewing is a skill set to be developed. It usually requires considerable training and supervision to become good at itbut you have to start somewhere! 10/28/2013 12:1512Geog 3250

Questionnaires (self-administered) Questionnaires are essentially structured interviews without an interviewer. They involve filling out a form which is then returned to the researcher, often by mail or in person. Because the respondent has to read the questionnaire without the aid of an interviewer, it has to be very clear and easy to follow layout matters!! 10/28/2013 12:1513Geog 3250 Example of Statistics Canada self- administered survey question

Questionnaires (self-administered) Compared to interviews, questionnaires tend to have fewer open questions, because closed questions are easier to answer; have simple designs, so the respondent can complete all sections without difficulty; be short, to avoid respondent becoming discouraged. 10/28/2013 12:1514Geog 3250

Questionnaires (self-administered) Advantages over structured interviews: Cheaper, quicker, more convenient to administer No interviewer effects, including errors in administering the instrument Social desirability bias seems to be reduced. Respondents are more likely to give genuine answers to sensitive questions, e.g., those regarding sexual practices, drug use, criminal activity, etc. 10/28/2013 12:1515Geog 3250

Questionnaires (self-administered) Disadvantages Researcher cannot explain the question re: validity. Greater risk of missing data, lack supervision. Researcher cannot probe. Difficult to ask a lot of questions. Difficult to ask different kinds of questions, especially open ones and questions requiring a filter. Order effects hard to control may answer out of order Literacy issues Designated respondent may not have completed questionnaire. 10/28/2013 12:1516Geog 3250

Questionnaires (self-administered) Online survey Growing in popularity Finding participant listsrandom? Email surveys Tend to be aimed at smaller, more homogenous groups 10/28/2013 12:1517Geog 3250 This one is free and can be restricted to UWO usernames

10/28/2013 12:1518Geog 3250

Questionnaires (self-administered) Researcher-driven diaries A form of questionnaire Participants record their feelings, perceptions, actions, etc. on a form shortly after they occur. Participants should be given explicit instructions on how to complete the diary, the time periods for recording responses, and the types of experiences to be recorded. Used for quantitative or qualitative research. structured like a regular questionnaire with closed questions, or free text experiences written down without fixed responses, like answers to open questions 10/28/2013 12:1519Geog 3250

Diaries Advantages of diaries over interviews and questionnaires: Fairly accurate data, re: frequency or time spent on behaviours of interest Fairly accurate data about sequencing of behaviours Good for getting data on sensitive matters, e.g., sexual practices Disadvantages More expensive than personal interviews Attrition: people may tire of using them Details may not be recorded quickly enough. Errors, omissions 10/28/2013 12:1520Geog 3250

Respondent Problems - Structured Interviews and Questionnaires Response sets the respondent is not motivated to provide a genuine response answer all the same without reading 10/28/2013 12:1521Geog 3250 I am not paying attention to each question as I respond

Respondent Problems - Structured Interviews and Questionnaires Acquiescence: trying to please the researcher e.g., respondent agrees just to be cooperative partial antidote: mix items so that they have logically opposite positions 10/28/2013 12:1522Geog 3250 Do you like this image?

Respondent Problems - Structured Interviews and Questionnaires Social desirability: avoiding appearance of being bad/unlikeable Answers are not sincere, e.g. questions on racism or the environment 10/28/2013 12:1523Geog 3250 I hope these are the answers she is looking for.

Respondent Problems - Structured Interviews and Questionnaires Laziness or boredom: answers just to get the end of the process E.g., respondent checks all the same answer to get it done 10/28/2013 12:1524Geog 3250 Assume all respondents might be this lazy?

Feminist (and other) Critiques Structured interviews and questionnaires - exploitative. asymmetrical power relationship between the researcher and the respondent. Link to Ethics: Increased attention given to the rights of research respondents in recent years, e.g., privacy rights, the right to end the interview at any time, etc. Is this enough? 10/28/2013 12:1525Geog 3250 Power imbalances are not always this overt

Questionnaire Design Identify at least one problem with each question in the following questionnaire snippet: 10/28/2013 12:15Geog 325026

Open or Closed Questions? Closed Questions Present the respondent with a set of answers from which to choose E.g., see previous slide Open Questions Response decisions are left completely to the respondent Answers must be coded into to numerical data Tend to be used in quantitative research E.g., What do you think about using alternatives to chemical pesticides on your lawn? 10/28/2013 12:1527Geog 3250

Open Questions Advantages Respondents can answer in their own terms Allow for unusual, unanticipated responses No suggested answers, so responses may expose knowledge and be more genuine Good for exploring new or changing areas of research Answers may lead to fixed-choice responses 10/28/2013 12:1528Geog 3250

Open Questions Disadvantages Time-consuming to record answers Answers have to be coded In self-administered questionnaires, respondents may balk at the request to write long answers Recording inaccuracies for verbal answers Why did your degree take longer than the funding period? Not really sure. I worked away on it diligently but the analysis just took longer than expected, I guess. There are worse places to be than grad school. Closer or more effective supervision was the primary reason. Too much procrastination and family issues. What code/categories? Can one response go in multiple categories? 10/28/2013 12:1529Geog 3250

Coding Open Questions Post-coding Data are gathered, then themes or categories of behaviour are discerned. e.g., an open question on the legalization of marijuana might have codes very hostile opposition, indifferent, etc. Each code would then be assigned a number, and attributed to respondent 10/28/2013 12:1530Geog 3250

Coding Open Questions, contd Pre-coding Themes or categories of behaviour are decided upon before the data are gathered. May be done with fixed-response items (i.e. closed- ended) e.g., What is your view on the legalization of marijuana? Are you strongly opposed, opposed, neither opposed nor in favour, in favour, or strongly in favour? Data may be gathered freely through open responses and sorted into pre-set categories afterward. (What if there are responses that seem to require a new code?) 10/28/2013 12:1531Geog 3250

Coding Open Questions, contd Basic principles: The categories must not overlap. The list of categories must be exhaustive. Must cover all possibilities. Other category is usually required. Coders should be provided with a coding frame or coding manual that establishes clear rules about how codes should be applied. 10/28/2013 12:1532Geog 3250

Closed Questions Advantages Easy to process answers Standardization allows comparison of answers Fixed responses may help clarify what the question means Easier and quicker for the respondent to complete Reduced bias in recording answers No interpretation required by researcher 10/28/2013 12:1533Geog 3250

Closed Questions Disadvantages Loss of spontaneity and authenticity - relevant answers may be excluded from choices provided. Antidote: Other category with open area to elaborate (but you may want to code it afterwards!) Defining categories that do not overlap. Antidote: Pre-test as open-ended to establish appropriate and distinct categories. Difficult to make forced-choice answers exhaustive 24. Why do you oppose the construction of more wind turbines in your community? Bird/bat collisions Economic costs outweigh benefits Health threats Impacts on farm and domestic animals Unfair siting process Other _____________________ 10/28/2013 12:1534Geog 3250

Closed Questions, contd Disadvantages (contd) Respondents may differ in their interpretation of the wording of fixed responses. e.g., the meaning of strongly in strongly agree Respondents may not find a fixed response that they feel applies to them. Large numbers of closed questions can reduce rapport in interviews Antidote: include open-ended responses even if you may not code them? 10/28/2013 12:1535Geog 3250

Types of Questions Personal factual questions e.g., age, occupation, number of cars owned Factual questions about others, events etc? Avoid these: makes respondent feel they are being tested and look stupid Antidote: Questions about perceptions are better puts them in position of being the expert. This survey tests, but it is for education, not research purposes 10/28/2013 12:1536Geog 3250

Types of Questions, contd. Beliefs e.g., Do you believe climate change has happened globally over the last 50 years? Attitudes Very common, subjective, taste, or perception use of Likert scales E.g., Do you feel we should do more to mitigate the effects of global climate change. Gabe Kotter tests attitudes where participant is the expert regardless of how they choose 10/28/2013 12:1537Geog 3250 Likert agree scale, often used to create an index from multiple items

General Rules for Designing Questions Keep the research questions/hypotheses in mind. Focus on exactly what you want to know. Be specific. e.g., How many children 18 and under are living at home? vs. How many children do you have? Put yourself in the position of the respondent How would you answer the question? 10/28/2013 12:1538Geog 3250

Specific Rules for Designing Questions AVOID: ambiguous terms Even commonly used terms may be vague, e.g., does income include tips? Investments income? Does it mean after tax income? long questions: respondents may lose track of what the question is asking multiple phenomena in the same question E.g., Do you agree your network of closest friends and family is helpful and supportive? 10/28/2013 12:1539Geog 3250

Specific Rules for Designing Questions AVOID: several very open-ended questions (do unstructured interviews instead?) e.g., What do you think of the prime minister? leading questions e.g., Do you feel the monarchy, which costs taxpayers over $5 million each year, be abolished? 10/28/2013 12:1540Geog 3250

Specific Rules for Designing Questions AVOID: questions that include negatives, especially double negatives. e.g., Do you oppose the rule stating that laptops are not allowed in classrooms? jargon/technical terms. And be sure that respondents have the knowledge needed to answer the question. E.g., How would you rate the level of social capital in your neighbourhood? 10/28/2013 12:1541Geog 3250

Specific Rules for Designing Questions Ensure symmetry between a closed question and its answers. e.g., avoid: Responses should be very satisfied, somewhat satisfied, Ensure the answers are balanced. e.g., avoid: 10/28/2013 12:1542Geog 3250 Dont let instructors design their own course evaluations?

Specific Rules for Designing Questions Memory Beware of expecting respondents to recall what theyve done or observed in detail. e.g., number of hours spent online per week Dont know option Consider avoiding this(e.g., use neither agree/disagree instead if you do not want to force them to choose) Thought to prevent forcing the respondent to choose, but it actually serves to offer an easy way out. 10/28/2013 12:1543Geog 3250

Exercise Design a survey question that violates one rule of questionnaire design identified above 10/8/2013 10:15Baxter Geog 215344 Text 335633 and your question to 37607 Or Respond at this linkthis link poll results here YES peek back up the notes NB You need to be in slideshow mode (full screen) for any of the links to work

Specific Rules for Designing Questions Question order All respondents should receive questions in the same order (unless testing for order effects). Asking a particular question may affect the responses given to subsequent questions. Early questions should be directly related to the announced research topic. Questions likely to be of interest to respondents should be asked early in the interview. Sensitive questions (e.g., Have you ever been a victim of sexual assault?) should be asked well into the schedule, but not at the very end. 10/28/2013 12:1545Geog 3250

Specific Rules for Designing Questions Question order, contd Questions should be grouped logically and into related sections. Within a group of questions, general questions should precede specific ones. Opinion and attitude questions should precede behaviour questions. Even if a respondent provides an answer to a question before it is asked, the question should be repeated at the appropriate time. i.e., there could be question order issues. 10/28/2013 12:1546Geog 3250

Designing the Questionnaire Provide a clear layout and presentation Choose vertical or horizontal answers, as appropriate Vertical tends to be more pleasing to the eye 10/28/2013 12:1547Geog 3250 Vertical layout Horizontal layout

Designing the Questionnaire Provide clear instructions on how to respond. Is more than one response allowed? Should the choice of answer be circled? Underlined? Keep a question and its answer together on the same page. 10/28/2013 12:1548Geog 3250

Vignette Questions Presenting people with one or more scenarios and asking them how they would respond Anchor the choices in a realistic situation. Creates distance between question and respondent. Hopefully more candid response. Weakness: how people say they would act in a particular situation may be very different from how they would really act. Can provide useful information, or at least a starting point for further research. 10/28/2013 12:1549Geog 3250 Michelle and Philip bought their home 5 years ago and 2 years ago wind turbines were installed nearby.Please provide your thoughts about their situation in the following 5 questions.

Pilot Studies Used to test whether individual items or the instrument as a whole operate well. Used with open questions to generate closed questions for subsequent studies. Provide interviewers with experience in administering the instrument. Can be used to ensure that there is variability in the answers given. 10/28/2013 12:1550Geog 3250

Pilot Studies Can identify questions that are embarrassing, uninteresting, etc. Can identify questions that are difficult to understand. Can be used to determine the adequacy of the instructions. Can be used to determine whether the instrument has satisfactory flow. 10/28/2013 12:1551Geog 3250

Using Existing Questions If certain questions have worked well for other researchers, they may be appropriate for your study. The original source must be cited. Allows results from different studies to be compared. Questions may be modified to suit ones purposes better. 10/28/2013 12:1552Geog 3250

chapter 4 10/28/2013 12:151geog 3250. thoughts on researcher- administered surveys? keep a list of...

Documents

structured interviews

face slide

document slide

nline slide

conducting interviews

interview contexts work

interviewer flexible

interinterviewer variability