1 surveys: collecting policy relevant data rachel govoni-smith kinnon scott, decrg january 17, 2006
DESCRIPTION
3 Household Surveys and the Impact of Economic Policies on Poverty and Income Distribution Estimating Incidence of Indirect Taxes Analyzing the Incidence of Public Spending Behavioral Incidence Analysis of Public Spending Estimating Geographically Disaggregated Welfare Levels and Changes Assessing the Poverty Impact of an Assigned Program Ex Ante Evaluation of Policy Reforms Micro LevelTRANSCRIPT
1
Surveys: Collecting Policy Relevant Data
Rachel Govoni-SmithKinnon Scott, DECRGJanuary 17, 2006
2
Sources-
• The Impact of Economic Policies on Poverty and Income Distribution: Evaluation Techniques and Tools, eds. Francois Bourguignon and Luiz Al Pereira da Silva, World Bank, Washington, D.C., 2003.– Scott, Kinnon (2003) “Generating Relevant Household
Level Data: Multi-topic Household Surveys”
• Muñoz, Juan and Kinnon Scott (2005) “Household Surveys and the Millennium Development Goals”, report for Paris21 Task Force on Improved Statistical Support for Monitoring Development Goals
3
Household Surveys and the Impact of Economic Policies on Poverty and Income Distribution
• Estimating Incidence of Indirect Taxes• Analyzing the Incidence of Public Spending• Behavioral Incidence Analysis of Public Spending• Estimating Geographically Disaggregated
Welfare Levels and Changes• Assessing the Poverty Impact of an Assigned
Program• Ex Ante Evaluation of Policy Reforms
Micro Level
4
Household Surveys and the Impact of Economic Policies on Poverty and Income Distribution
• The Effect of Aggregate Growth on Poverty• Linking Macro-consistency Models to
Household Surveys• Partial Equilibrium; Multi-market Analysis• The 123PRSP Model• Social Accounting Matrices• Poverty and Inequality Analysis and CGE
models
Macro Level
5
Goals and Needs
Goals:
• Measure the poverty impact of economic policy
• Measure the distributional impact of economic policy
Needs:
• Rely heavily on household survey data
6
Household Surveys
• Single Topic
• In-between
• Multi-topic
7
Household Surveys
• Single Topic
• Labor Force Surveys( LFS) (ILO)
• Housing Surveys
• Census – national, UNFPA, 10 years
• In-between
• Multi-topic
8
Household Surveys
• Single Topic
• In-between
• Agricultural Surveys (FAO)
• Demographic and Health (DHS)
• Household Budget Surveys (HBS)• Multi-topic
9
Household Surveys• Single Topic• In-between
• Multi-topic• Multiple Indicator Cluster Surveys (MICS,
UNICEF)• Survey on Income and Living Conditions (SILC,
EU)• Core Welfare Indicator Surveys (CWIQ, WB)• Living Standards Measurement Study Surveys
(LSMS) and Integrated Surveys (IS) (WB)• Family Life Surveys (FLS, RAND)
10
What type of household data?
• Poverty measure: per capita or per adult equivalent consumption
• Government programs receipt, format, costs (formal and informal), use level
• Consumption of taxed goods
• Labor market participation (sector, hours, earnings)
• Income by sources
11
Census
• Accurate measure of the population of a country
• Geographic distribution of the population
• Basic demographic information
Purpose
12
Census
• Not a sample
• Universal coverage
• No sampling errors in estimates
• Some corrections for non-response may be needed
Sample
13
Census
• Short
• Trade-off between coverage and content
• Two types of errors: sampling and non-sampling
Content
DECRG: May 7 2004 Sample size
Sampling errorNon-sampling error
Sampling vs. non-sampling errors
Total error
15
Census
• Short
• Trade-off between coverage and content
• Two types of errors: sampling and non-sampling
Content
• Cost •Time•Non-response• Training
16
Census
• Demographic information: age, sex, race/ethnicity, family and household composition
• Housing information
• Others: basic education, labor, disability
Content
17
Census
• Basic needs– Subjective
– Limited monitoring use
Poverty Measurement
• Income: Panama example
•Albania: 2001 (1989)•BiH 1991 (1981)•Montenegro 2003 (1991)
– Limited use if looking at impact of policies affecting taxes, tariffs or pricing
18
Census
• Sample frame
• Link with household surveys for small area estimation
Uses
19
Poverty Indicator by Commun, Albania, 200
20
Labor Force Survey
• Direct measurement of unemployment
• General characteristics of the labor force
Purpose
21
Labor Force Surveys
• Relatively large samples
Need for precise estimates (change)
Desire to disaggregate to different geographic areas
• Individuals of working age
Sample
22
Labor Force Survey
• Characteristics of the labor force– Demographics
– Education
• Sectoral distribution of employment
• Degree of formality
• Seasonal
• Income
Content
23
Labor Force Survey
Three problems:
• LFS typically capture partial, not total, income– Under-estimate welfare (vs. NA)
– Mis-ranking of households by welfare level
Poverty Measurement
24
Venezuela: Income and Expend Survey
25
Venezuela: Social Survey
26
Labor Force Surveys, cont.
Three problems:
• LFS typically capture partial, not total, income
• Measurement Error– Labor income measurement error
– At both ends of the distribution
Poverty Measurement
27
LFS in Latin AmericaItem non-response
Salaried Self-employed
Employer All Indep-endent
Mean non- response rate
3.9% 10.2% 12.0 10.6%
Source: Feres, 1998
28
Labor Force Surveys, cont.
Three problems:• Partial vs total, income• Measurement error
• Income vs consumption measure– Potential vs actual welfare
– Smoothing
– Measurement Error
Poverty Measurement
29
Household Budget Surveys
• Inputs to national accounts on consumer expenditures
• Track changes in expenditures over time
• Track changes in the relative share of different expenditures
• Weights for the consumer price index
Purpose
30
Household Budget Surveys
• Medium size sample
• Sampling errors high at disaggregated level
• High non-response rates
• In some parts: only urban (capital city or group of large cities)
Sample
•Non response rates (Eurostat, 2003)•Bulgaria: 39.7%•Estonia, 44%•Hungary, 58.8% before replacement•Romania, 21.6 %
31
Household Budget Surveys
• Total Income
• Total Consumption
• Short Demographics
• In FSU and Central Europe: agriculture
Content
32
Household Budget Surveys
• Possible to construct both total consumption and total income
• Income may suffer from same measurement errors as LFS
Poverty Measurement
33
Household Budget Surveys
• Consumption based welfare measure
• Purpose of an HBS survey is NOT to measure welfare but to precisely measure mean expenditures on specific goods and services
• These are conflicting goals
Poverty Measurement
34
Household Budget Surveys
• Shortest possible reference periods
• Minimize number of omitted expenditures
• Good for precise measurement of regional or national means
• Because of lumpy nature of purchases, not good for comparisons among households
Need to adjust (lengthen) the reference periods used in HBS
Poverty Measurement
35
Household Budget Surveys
• Focus on expenditures– Not all expenditures are consumption
– Only purchases of durable goods and housing
Durable goods: list of items owned by household, age of items, current value
Housing: housing characteristics affecting value
Poverty Measurement
36
Household Budget Surveys
• Good for taxation issues
• Good for public (and private) transfers
• Sometimes has basic labor
• FSU and Central European countries: agriculture
• No health, education data
• Limited for other areas
Uses
37
Multi-topic Household Surveys
Those with a focus on measuring poverty
• National Socio-Economic Survey of Indonesia, SUSENAS
• Survey on Income and Living Conditions (SILC)
• Rand Family Life Surveys (FLS)
• Living Standards Measurement Study Surveys (LSMS)
38
Multi-topic Household Surveys
• Analysis of welfare levels and distribution
• Study links between welfare levels and individual and household characteristics, economic, human and social capital
• Social exclusion
• Causes of observed social outcomes
• Levels of access to, and use of, social services, government programs and spending
Purpose
39
Multi-topic Household Surveys
• Small sample sizes
• Trade-off issue: Quality and cost considerations
• Limits ability to assess programs or policies that affect small groups or small areas (over-sample)
• Infrequent in many countries (exceptions, inter alia, Indonesia, Panama, Jamaica, Peru, Ghana)
Sample
40
Multi-topic Household Surveys
ContentHousehold Demographics* Agricultural Activities*Housing* Non-farm household businesses*Education* Food consumption (purchase, produced, gift)*Health* Non-food consumption and durables*Labor* Other income (incl. public &private transfers)*Migration* Social capitalFertility* Shocks, vulnerabilityPrivatization Time UseCredit Subjective measures of welfareAnthropometrics
Note: Starred modules are those most often used.
41
Multi-topic Household Surveys
• Total consumption– Longer reference periods
– Able to calculate use value of durables and housing
• Total income– Suffers from standard measurement errors
Poverty Measurement
42
Multi-topic Household Surveys
• Poverty levels and distribution
• Social exclusion
• Public and private transfers
• Incidence analysis
• Tax policy
• Labor markets
• Education, health, social protections
• Changes in relative prices
• Monitoring (PRSP, MDGs), impact evaluation
Uses
43
Cross Section or Panel Cross Section or Panel Surveys?Surveys?
• Substantive applications
• Methodological issues
44
PPanelsanels1. Why do we need longitudinal data?
2. Designs for surveys across time
3. Advantages and uses of panels
4. Methodological issues
45
Understanding changeUnderstanding change
Longitudinal data are needed to understand the process of change, transitions between states, and the factors or events that are associated with those transitions
‘Longitudinal’ data is a catch-all phrase for a wide range of different types of studies
46
Designs for surveys across time Designs for surveys across time
Repeated cross sectional surveys (e.g. Household Budget Survey, Labour Force Survey)
• Common design for large government surveys
• New sample drawn for each survey• Carry similar questions each year• Used for trend analysis at aggregate level
47
Designs for surveys across time Designs for surveys across time Cohort Studies • Sample often based on an age group • Follow up same sample members at fairly long
intervals • Developmental data as well as social and
economic data• Data from parents, teachers associated with
cohort member
48
Designs for surveys across timeDesigns for surveys across timeRotating Panel Survey Survey of Income and Programme
Participation, USA (SIPP)
• Respondents stay in the panel for a set period of time and are rotated out systematically and replaced by new sample members.
• Used where the interviews are fairly close together (every 3 to 6 months) and respondent burden is high.
• Used where the collection of short spells e.g. a few weeks unemployed or in receipt of a particular benefit, is critical.
49
Designs for surveys across timeDesigns for surveys across time
Indefinite Life Panel Surveys e.g. Panel Study of Income Dynamics, USA – since 1968!
Living in BiH, LSMS Albania, LSMS Serbia
• Draw a sample at one point in time and follow those sample members indefinitely (or as long as the funding continues)
• Collect individual level data in household context• Repeated measures at fixed intervals (annual data
collection)
50
Panels from conference attendee Panels from conference attendee countriescountries
• Albania – 4 waves 2002 - 2005 • BiH – 4 waves 2001 - 2004• Serbia – 2 waves 2002 - 2003
51
Advantages of Panel DataAdvantages of Panel Data• Comparison of same individual over time - outcomes• Track of aspects of social change• Facilitates study of change and causal inference • Minimise the problem of inaccurate recall• Compare a person’s expectations with real change• Look at how changes in individuals’ behaviour affects their households
Identifies the co-variates of change and the relative risks of particular events for different types of people
52
Changes in Employment Changes in Employment StatusStatus
A: CROSS-SECTIONAL INFORMATION
Unemployed
Employed
2001 2007
Net change - 0.1% unemployed
53
Changes in Employment Changes in Employment StatusStatus
B: PANEL INFORMATION
Still Unemployed
Still Employed
Unemployed
Employed
2001 2007
Net change - 0.1% unemployed Actual change is 10.1
continuouslyemployed
86.7%
employed 2001but unemployed 2007
5%
continuouslyunemployed
3.2%
unemployed 2001 but employed 2007
5.1%
54
Balkan Examples
Albania - 15% of the unemployed in 2002 had made the transition to formal sector employment by 2004
BiH - About half who were poor in 2001 remained poor in 2004. Many individuals moved out of poverty.
(Cross section headcount 18% for both years)
55
Employment and the labour market Unemployment duration and exit rates Do the unemployed find stable employment? The effect of non-standard employment on
mental health Temporary jobs: who gets them, what are they
worth, and do they lead anywhere?
Family and Household Patterns of household formation and dissolution Breaking up - finances and well-being following
divorce or split The effect of parents’ employment on children's
educational attainment
56
Panel analysis
Mobility, poverty and well-being among the informally employed – Peter Sanfey European Bank for Reconstruction and Development
The origins of self employment, Leora Klapper et al, WB (soon to use Albania Panel also)
The impact of health shocks on employment, earnings and household consumption, Kinnon Scott et al
57
A SampleA Sample• Concept of ‘longitudinal household’
problematic for a panel - households change in composition over time or disappear altogether
• Individual level sample
58
Following rulesFollowing rules• All members of households interviewed at
Wave One• Children born to these original sample
members • Original members are followed as they
move house, and any new individuals who join with them are eligible to be interviewed
• New sample members are followed if they split from the original member
59
Questionnaire designQuestionnaire design
• Core content carried every wave• Rotating core questions • One-off variable components
– lifetime job history– marital and fertility history
• Variable questions to respond to new research and policy agendas
60
Attrition in panel surveysAttrition in panel surveys• Inevitable to some extent but can be
minimised• Multiple sources of attrition in a panel
– refusal to take part– respondents move and cannot be traced– non-contacts
• Worry is potential bias if people who drop out differ significantly from those who stay in
61
70
87.7 90.394.9 94.8 97.5 97.2 97
010
20304050
607080
90100
Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 Wave 7 Wave 8
UK Panel Wave 1 RespondentsUK Panel Wave 1 RespondentsWave-on wave re-interview ratesWave-on wave re-interview rates
62
FieldworkFieldwork• respondent incentives as a ‘thank-you’• extended fieldwork period for ‘tail-enders’• refusal conversion programme• tracking procedures during fieldwork• panel maintenance between waves
– Change of Address cards to update addresses– mailing of Respondent Report– details of contacts with respondents between waves
63
Post-field checking and cleaningPost-field checking and cleaning
• Within wave consistency
• Cross wave consistency and longitudinal integrity
• Sample management– individuals within households correctly
identified across time
– issuing of sample for each wave
64
The user databaseThe user database
• Longitudinal data is complex• Provide users with database structure
which enhances usability• Consistent record structure over time• Key variables for matching and linking data
cross wave• Consistent variable naming conventions
65
Added valueAdded value
• ‘Added value’ to data set• Extensive set of derived variables • Production of weights
– household and individual levels – cross sectional and longitudinal
• Imputation of missing data• Flags to indicate imputed values
66
ConclusionsConclusions• Longitudinal panel data allows us to answer
research questions that cannot be answered with with cross-sectional data
• Provides a different view of the world - see process through the life-course not just a static picture
• Is complex (but so is the real world) - so needs to be well designed and conducted with sufficient resources to be successful
67
System of Household Surveys
• GOAL: System able to respond to evolving needs: not produce data X or survey Y– Determine data needs before they are
URGENT
– Identify appropriate instruments,
– Implement them properly, timely fashion,
– Analyze the resulting data
68
Improving the SHS• Linking Users and Producers• Providing adequate resources• Continuous Survey Program
– Not necessarily permanent survey– Benefits
• Avoid loss of capacity• Create greater levels of capacity (building on existing)• Economies of scale• Policy makers know when data will be available• Protects NSO from pressures for ad hoc surveys• Ongoing system actually allows more flexibility and
responsiveness
69
Final points
• Welfare: household surveys- always missing the homeless, street children, institutionalized population
• No one survey can meet all needs, review its purpose, coverage, content and quality before using
• Need a system of surveys that meets the needs of data users