Questionnaire development Devesh Roy (September 22, 2015)

IFPRI-ICAR training

Some definitions

• Semi-structured Interview:

• A semi-structured interview is an interview with an individual or individuals that follows a pre-defined set of questions. It is flexible, allowing new questions to be brought up during the interview as a result of what the interviewee says.

questionnaire

• The survey instrument for every evaluation should be carefully designed to gather all the information needed to test the hypotheses underscoring the project logic as well as to construct other control variables for impact estimation.

• A questionnaire to collect agricultural data should usually be based on the agricultural household model. This model suggests that agricultural production units behave as both a firm and a household and, in the presence of market failure, production decisions cannot be separated from household consumption decisions. Hence, collecting household as well as farm information should be a priority in every questionnaire that aims to analyze agricultural household behavior.

• In general, the modules to be included should obtain a full picture of the household livelihood strategies, including crop and livestock activities but also non-agricultural activities, migration and remittances among others.

• Many of these variables may not be measured directly so a series of questions may need to be asked to obtain a single key variable. For example, questions on crop profits or gross margins require having information on revenues and costs of crop production. Questions that directly ask this information are unlikely to be reliable so it requires asking a series of questions related to revenue and costs that help to obtain the information.

Questionnaire: essential elements (World Bank 2008)• What do you want to know?

• Tailor survey to capture outcomes of interest

• Use reliable and valid instruments

• Gender disaggregation!!

• Be careful: what‟s reliable and valid in one context may not be so in another

• test your questionnaire!• In house

• In the field

Mistakes in questionnaire development (medal level from the point of correction potential)

• Gold medal mistakes• Start work on the questionnaire before finalizing the evaluation design

• Start work on the questionnaire before setting up the methods• Start work on the questionnaire before fixing the sampling design • Finalizing questionnaire before pre-testing• No validation checks built in the instrument

• Silver medal mistakes • Not incorporating the environment of the household • High density of open ended questions• Data prone to noise (like income)sought in questionnaire• Objective of the survey (the starting point not fixed in a team but by an individual)• Household roster not specified – who is to be included in household

• Bronze medal mistakes • Enumeration load, sensitive questions, loaded words

Questionnaire guidelines (world bank 2008)• Define topics and concepts to avoid confusion

• Define your identity properly – are you government, are you NGO etc. affects the responses

• Question order matters

• Keep it short and make it user-friendly

• Every time you write a question, ask yourself “Why do I want to know this?” Answer it in terms of the way it will help you to answer your research question. “It would be interesting to know” is not an acceptable answer.

• Phrase questions clearly

• Use established techniques to minimize respondent mistakes (e.g. calendars for event histories). Asking monthly or weekly rather than asking regularly

• If continuous markers cannot be obtained get categorical information like income in ranges.

• Operate in modules – treat modules as you treat paragraphs in writing

Questionnaire guidelines

• Time period and saliency –purchase of house more salient should be timed in the questionnaire in terms of say one year period while time period for purchase of clothing could be say one month

• If asking questions about others focus more on strong ties (son daughter) and less on weak ties (such as co-worker) unless it is by design such as in social network analysis.

• Role of multiple options possible

• Role of other as an option in responses – use but use sparingly studies show this is underreported

Example of a good question (trace test- get 3-4 people and check the interpretation –this is part of in house test)

• Aside from any work you do here at home or at a job, do you do anything regularly—that is, on a daily basis—that helps you keep physically fit?

• Yes

• No

Best practice

• Search for questions on the same topic by other researchers

• Precautions:• Mark the change in the wording even if it is small

• Take account for the differences in timing of the earlier and current surveys

Ask exactly what you want to know (Bradburnet al)• The precise wording of questions plays a vital role in determining the

answers given by respondents. This fact is not appreciated as fully as it should be, even in ordinary conversation. For example, someone mentioned that he needed to buy something. Only day he could come was Saturday before labor day weekend. Although he called on Friday to make certain the store was open, he arrived at the store on Saturday only to find a sign on the door that said “Closed Labor Day Weekend.” When asked if he remembered what question he had asked the clerk at the store, he said, “I asked him what hours he was open on Saturday, and he replied ‘Nine to five.”

Some points to note (can get in the revision after pretesting)• Use of aided recall –Instead of asking what did you eat in the last

week you would produce a list of items to choose from

• Long lists anchor on last or first ones

• Use of proxy means (dwellings, assets, capital etc)

Bias and variability

• Bias refers to an estimate that is either more or less than the true value. Variability is measured by the susceptibility of measurements to differences in question wording. This variability is sometimes called the reliability of a measure, since random errors may arise from the form of the measurement itself (rather than from systematic error due to a sample bias or some other aspect of the measurement instrument).

• In order to clarify the sources of response effects, let us look at a particular behavioral question. A common question in surveys is “What was your total family income from all sources last year?” There is a true answer to this question, even though we may never know what it is. However, even though there is a true answer to this question, we may get an erroneous answer because the respondent simply forgot about certain amounts of income, particularly those from less obvious sources.

Bias and variability: continued

• Pure recall errors could be classical in the sense that on average they might cancel out some overestimate and some underestimate

• But there could be misreporting because of motivation to hide or blow up income (to look better)

Bradburn et al (Loaded words in questions produce loaded results) (Bradburn et al)• Karl Marx in a survey of workers asked Does your employer or his

representative resort to trickery in order to defraud you of part of your earnings

• Could have been asked as are you fairly compensated?

Best practices

• A good question is understood consistently by all respondents

• A good question is administered consistently to all respondents

• A good question elicits the kind of answers the researcher wants:

• • BadQ: When did you move to Seoul, Korea?

• • A: In 1964

• • A: When I was 20 years old

• • A: After I finished college

• • BetterQ: In what YEAR did you move to Seoul, Korea?

Best practices

• A good question is one where the respondents have the necessary knowledge to answer

• A good question is one where the respondent is willing to provide the „true‟ answer

• difficult for sensitive questions

Compatibility of questionnaire with methods

• If cross-sectional variation relied on what questions should be there

• If double difference methods employed what should be included

• What in case of most significant change

Other elements – accounting for environment

• Account for environment – what other projects • Make a distinction between time varying and time invariant

• Compatibility with methods- if there is time invariant participation in some project given the method may not need to include

• Intensity should always be asked

• For the project and others timing of engagement should always be asked

• A community questionnaire should be employed for accounting for environment (how to define a community?)

Some leads from LSMS surveys

• Have metadata such as dates of interviews, household roster information on any children identities of respondents, and time required for each interview.

• Household roster well defined

• Consumption measures should be used measure- The measurement of consumption is highly sensitive to differences in methods, and that consumption measurement must be standardized over time.

Lessons: continued

• First, detailed information on household members' work for household to collect in agriculture and enterprise modules not in employment modules

• Generically pre test with short and long modules in specific cases

• Household enterprises are very important in developing countries-have a separate module on them