Designing Surveys

the art and science

For any type and style…

• Soil biota and effects of land use change• Effectiveness of extension organisations• Market chains for farm produced timber• Local knowledge of water management• Genetic variation in medicinal trees• Livelihood – environment interactions• Mechanisms for science influencing policy• Participatory assessment of NRM problems

Principles and practice

• Principles are very widely applicable– you better know what they are!

• Application depends on possibilities and constraints of the context– know your situation well– be familiar with methods used by others– refer to guides and methods sections of papers

• BUT…– traditions and common practice can be inefficient or

flawed– some of the most innovative and informative research

comes from transfer of methods between disciplines

Objectives

• It all starts here!

• Infinite variety, but often either:– current state (and change over time)– patterns, associations, differences…

• But they need to be very clear.

• Expect to iterate between objectives and study details

Two examples

1. Increased elephant damage reported in some villages. Are elephants moving along usual migration routes?

2. Is striga infestation worse when fields are suffering from soil erosion?

What are the challenges?1. Elephants 2. Striga

Population Farmland in areas reporting increased animal damage

Maize-growing areas of W Kenya

Unit Village 2m x 2m quadrat

Sampling scheme 30 villages selected at random

10 villages (random)

10 fields per village

2 quadrats per field (1/3 and 2/3 of the way across)

Measurement tool Questionnaire for village meeting

Visual assessment of damage

Striga counts

Soil erosion score

Units

• The items being studied

• May be several– person, household, village– tree, field, watershed– gene, individual, family, population– farmer, extension officer, project, organisation

Problems

• Don’t confuse: unit(s) which are determined by objective and those you measure– may need to move information to new levels

• ‘Ecological fallacy’ – trying to make inferences about individuals from measurements on groups

• The opposite also fails!• ‘Modifiable areal unit problem’ : the relationship

between X and Y depends on the scale at which they are measured.

Population

• The complete set of units you wish to study

• Determined by the domain of the study – the extent of the problem you want the information to apply to

• Common problem– poorly defined population– not stated but implied by sampling

Sampling

4 ideas you have to understand• Simple random sampling• Stratification• Hierarchical (multistage) sampling• Systematic sampling

• Note: good practical sampling schemes will probably use a combination of these

Common problems

• No explicit sampling scheme– bias– subjective results

• Weak use of stratification– can greatly improve studies to identify

relationships

Stratification to estimate relationships

• Example: survey to look at ‘market integration’ and relation with poverty.

• Two variables associated with market integration (mi); distance from main road (d), length of time settled (t)

mi

dt

Random sampling

d

1. Most observations clustered around the mean d, hence poor estimate of response

d

t

2. d and t negatively correlated – can not estimate the joint response surface well

Solution• Divide into strata• Response

surface ideas to choose sample size in each stratum

• May require large sample fraction in some strata

• NB: often felt by researchers to be ‘biased’ – explain carefully!

d

t

Sample size

• Should be based on rational analysis– How much variation is there?– How precise do you want the answer to be?

• Too large – waste of effort• Too small – can not meet objectives• Software available to help• Expect to iterate

– make objectives more modest if required sample size too large

Common problems

• Arbitrary selection of sample size– illegal in other disciplines!

• ‘Rules’ which have no basis– ‘at least 30 farms for farmer survey’– ‘10% sample’– NB. Sample fraction (almost) never relevant

• 50 from 5000 (=1%) gives same precision as 50 from 500000 (=0.01%)

• Too few ‘higher level’ units. – Eg 1:

• Objective - factors influencing effectiveness of extension.

• Sample: 200 farmers from 2 different extension projects.

– Eg 2:• Objective - SOM change after forest conversion• Sample: 200 plots around 1 forest

• Too few units for many x – variables– Eg: Determine how adoption depends on

gender, education, farm size, group membership, income, and land quality.

– Don’t expect to learn all that from observing 30 farms!

• Forgetting about – effect of sampling scheme (clustering larger

overall size needed)– non-sampling errors

Sampling and non-sampling errors

• Sampling error:– Due to not measuring whole population– Described by statistical measures (eg standard error, confidence

interval)– Control by statistical methods

• Non-sampling error• Non-response• measurement errors• inaccurate sampling frame• Coding or data entry errors• Operator differences

– Manage by good survey practice– Make allowance in sample size calculation– NB: larger not always better!

Measurement

• Focus on objectives, use conceptual frame

• Only measure what you know how to use

• There are always alternatives– How can you measure maize yield in a

farmer’s field?

• Check how others have done it– many guides and manuals available

• Pilot EVERY measurement tool

Factors affecting impact of extension organizations

Farmers learning

Farmersplanting

Farmer to farmer dissem.

Impact on incomes/

livelihoods

Org. charact-eristics

Org. resources

External Environ-

ment

Org. engages

Farmers sensi-tized

Org. strategy

Farmers trained

Other benefits

Impactindicators - quantity and quality

Factors affecting impact

And then…

• Logistics, transport

• Workflows and timing

• Materials and equipment

• Data handling

• Quality control

• …

Ethical issues

• Prior informed consent

• Sensitive topics

• Confidentiality

• IPR

• Legal requirements

• Feedback to data providers