young lives closer
TRANSCRIPT
An extensive longitudinal study into children & youth around the world
Methodological challenges, strengths and weaknesses
Marta Favara, Young Lives
CLOSER Longitudinal Methodology Series seminar
July 28th , 2016
Outline of this presentation1. Overview of Young Lives data
₋ Young Lives in pills₋ Sampling design and survey components₋ Survey questionnaire₋ Processes in place for designing and implementing the survey questionnaire
2. Nice features of Young Lives data
3. Challenges and risk-mitigation strategies
Young Lives in pills₋ Multi-disciplinary study that aims to:
- improve understanding of childhood poverty and inequalities- provide evidence to improve policies & practice
⁻ Young Lives components: Household survey (child, caregiver, younger siblings, children of the YL children,
community representatives); Longitudinal qualitative research; School survey: parallel to round 2 and 5 of the household survey.
₋ Following nearly 12,000 children in 4 countries: Ethiopia; India (Andhra Pradesh & Telangana); Peru and Vietnam
₋ Over a 15-year period: first data collected in 2002, with 5 survey rounds
₋ Two age cohorts in each country:- 2,000 children born in 2000-01- 1,000 children born in 1994-95
₋ Collaboration: - Partners in each study country- Publicly archived survey data (UK Data Archive and listed on the World Bank Micro Data
website) and core-funded by DFID, DGIS, IrishAid
12,000 children in 4 countries over 15 years
– Sentinel site sampling; four stages sampling process (region, district/provinces, sentinel sites, random sampling of children within sites);
– Purposively over-sampled poor areas (40% urban / 60% rural) using different poverty indicators in each countries
Ethiopia India Peru Vietnam
Sampling design
– Demographic information (hh roster), socio economic indicators (wealth index, food consumption)
– Health information and anthropometrics (YL child, parents, siblings and child of YL children)
– Education history (all hh members) and cognitive skills (YL child, siblings)– Subjective wellbeing and psychosocial competencies (YL child, siblings)– Employment status/history and time use (all hh members) – Job related skills – Job and Educational Aspirations/expectations (YL child, parents)– Expectations about marriage and parenthood (YL child, parents)– Fertility history– Marriage/cohabitation history– Control over assets (intra-household decision making)– Social norms indicators– Knowledge on SRH and access to contraceptives– Sexual behaviours, risky behaviours and criminal activities (Peru)
Information collected
Step 1: Design The Survey Questionnaire
Step 2: Tracking and preparing CAPI programme
Step 3: Training and piloting
Step 4: Fieldwork
Step 5: Data cleaning, validations
Step 6: Preliminary analysis and Research
Six steps from design to the field
1. Comprehensive set of information collected at community and household level (caregiver, YL child, a subsample of (younger) siblings and the children of the YL children)
2. Longitudinal data covering a period of 15 years from early childhood to adulthood
3. A life-course approach, very relevant for policy design (early childhood, middle childhood and adolescence)
4. Cross-cohorts, cross-countries comparisons
Nice features
–Allows to identify links between earlier circumstances and later outcomes– Identifying when differences emerge: how persistent particular circumstances are; what shapes later
well-being
– Testing the ‘dynamics’ of well-being (Controlling for time-constant unobservable characteristics)
– Compare two cohorts at the same age (trends, exposure to different policy context)
– Use (panel) siblings data to investigate how household or community circumstances affect child outcomes at the same age; explore intra-household dynamics; controlling for the influence of past events and circumstances
– A new generation (Children of YL’s children)
Multi cohorts longitudinal data: Main benefits
– Challenge 1. Cohort maintenance– Challenge 2. Getting comparable measures over time– Challenge 3. Across countries coordination/comparability– Challenge 4. Ensure high quality data– Challenge 5. Data collection methods: switch to CAPI
Multi cohorts longitudinal data: Main Challenges
Challenges :– Some attrition is inevitable– Cohort is relatively small for a longitudinal study – Study period is relatively long (three years gap between waves)
Risk mitigating strategies:– Collecting detailed contact information– Importance of tracking
₋ Reduces time looking for children when we start the fieldwork₋ Maintains continuity of social contact and trust between researchers and
families– Reduce refusal rates as much as possible:
₋ Importance of explaining what we’re doing ₋ Importance of maintaining field teams₋ Give photos back to families (part of ethics/reciprocity)₋ Ensure no respondents are over-loaded (by different elements/sub-studies)₋ Compensations (Losing a day of work has big impact on income)
Challenges: 1. Cohort maintenance & attrition
…and we have been quite successful!YC OC Overall
Ethiopia 2.2% 8.4% 4.3%
India 2.6% 4.3% 3.2%
Peru 6.3% 10.3% 7.3%
Vietnam 2.9% 9.9% 5.3%
Total 3.6% 8.1% 5.0%
ETHIOPIA INDIA
PERU VIETNAM
Source: Outes and Dercon, 2008
Non-random attrition
₋ Attriting households (R1-R2) tend to have fewer assets, poorer access to services and utilities and are less educated (more in Ethiopia and India than Peru and Vietnam) (Panel A)
₋ These averages hide substantial variation between different types of attriting households (Panel B)
₋ The presence of non-random attrition does not necessarily imply attrition bias: no attrition bias found when looking (Ethiopia is an exception)
Challenges:– The questions need to change as the children grow up– Change in primary respondent/hh head– Keep as many questions as possible the same across rounds (panel variables)– Asking the same questions of the YC as we did the OC in earlier rounds (core base
variables)– Ensure comparability over time (e.g. cognitive tests-- Item Response Theory)– Keep the order of the survey modules the same over the time
Limitations for comparability:- Switch from PAPI to CAPI; - Some changes in the structure of the questionnaire are inevitable- `Getting stuck’ with the errors of the past to the seek of maintain comparability across
rounds
Challenges: 2. Getting comparable measures over time
Benefits: – How patterns of relationships are similar/different across countries.– Understanding why and how specific policies or programmes are effective in one
country.– Comparative analysis can give greater confidence that evidence found in one country is
applicable to others.– Learning in relation to methods: trying to develop measures that can be used across
cultures.Challenges:
– Constructing a questionnaire that suits different national contexts.– Ethical committee approval and country specific sensitivities.– Deal with different fieldwork processes.
Risk mitigating strategies :– Define research priorities and relevant survey questions in each country– There are also some country variations – Translation and back translation is key to ensure consistency – Continuity of country team leaders and fieldworker coordinators.
Challenges: 3. Across countries coordination and comparability
Challenges:– Maintaining increasingly complex survey instruments– Maintaining strong coordination and liaison between Quant/Qual/ School survey teams– Participant recall – Panel conditioning
Risk mitigating strategies:– Piloting and training are crucial!
₋ Ensure research questions work in the field and are consistent with local situations and children’s ages₋ Ensure questionnaire are not too long / burdensome₋ Train teams and learn from practical experience of field work to improve instrument design₋ Produce accurate instrument manuals and protocols₋ Uncover ethics issues and give safe space for discussion₋ Initiate, build and maintain positive team dynamics₋ Ensure that good data collection systems are in place
– Consistency checks are embedded in CAPI, some information are prefilled, ultimately some inconsistencies can be solved ex-post
Challenges: 4. Quality of the data
• CAPI introduced in R4 – is a different way of doing surveys (e.g. changes dynamic of interview)
Benefits: – Eliminate data entry error. – Know how work is progressing– Avoid mistakes before they happen– Ask the right questions (embedded skip pattern)– Quality improvement (?)– Reduction in the length of the interview (?)
Challenges:– Requires more time at the front end (building the programme)– Fieldworkers to get familiar with a new instruments– Put in place a data management and transfer systems – Devolve responsibilities to the in-country data managers (in Peru and Vietnam)
Risk mitigating strategies:– Extra effort at the front end in programming– Piloting and testing the application is crucial!– Training country data managers and fieldworkers on data management and transfer systems.
Challenges: 5. Introducing CAPI
Annex
Ethiopia
Sampling design (1)Four stages sampling process:1. Regions (Amhara, Oromia, SNNPR, Tigray
and Addis Ababa, accounting for 96% of national population)
2. Woredas (districts) (3-5 districts in each regions, 20 in total)
3. Kebele (at least 1 for each woredas)4. 100 young children (born in 2001-02)
and 50 older children (born in 1994-5) were selected within those sites.
Criteria to select districts:5. Districts with food deficit profile6. Districts which capture diversity across
regions and ethnicities in both urban and rural areas
7. Manageable costs in term of tracking for the future rounds
Comparing with DHS and WMS 2000: 2000:Poor hh are over-sampled, but YL covers the diversity of children in the country including up to 75% percentile of the Ethiopian population.
India
Sampling design (2)Four stages sampling process:1. Regions (Coastal Andhra, Rayalaseema,
and Telangana)2. Districts 3. 20 sentinel sites (mandal)4. 100 young children (born in 2001-02)
and 50 older children (born in 1994-5) were randomly selected within those sites.
Criteria followed:5. Uniform distribution across regions6. One poor and one non-poor district in
each region (based on economic, human development and infrastructure indicators)
Comparison to the DHS 1998/9: YLs hh seem to be slightly wealthier than the average household in Andhra Pradesh. Despite these biases YL sample covers the diversity of children in poor households in Andhra Pradesh
Peru
Sampling design (3)
Sampling process:1. Sample frame at district level excluding
the top 5% richest district based on poverty map 2001
2. Districts divided in population groups ordered by poverty index and randomly selected to cover rural, urban, peri-urban coastal, mountain and amazon areas (random selection proportional to district population)
3. Within the selected districts a village was randomly chosen
4. Within each village the street blocks were counted and randomly numbered to select the starting point.
Comparison to the DHS 2000: YL cover the diversity of children and hh in Peru
Vietnam
Sampling design (4)Four stages sampling process:1. Regions (5/8 regions, North-East region, Red River
Delta, City, South Central Coast, Mekong Delta.2. Provinces (5 in total ,1 per region, Lao Cai, Hung
Yen, Da Nang Phu Yen, Ben Tre).3. Sentinel sites (4 commune per province, 2 poor, 1
average and 1 above-average commune )4. 100 young children (born in 2001-02) and 50 older
children (born in 1994-5) were selected within those sites.
Criteria followed (to rank communes):5. Development of infrastructure, 6. Percentage of poor households in the commune7. Child malnutrition status.Comparison to the DHS and VHLSS 2002: The urban sector is under-represented (in terms of population and the level of development). YL includes hh with on average less access to basic services and slightly poorer than the average in Viet Nam. YL sample covers the diversity of children in the country.
Cognitive skills
Cohort Round 1 (2002) Round 2 (2007) Round 3 (2010) Round 4 (2013) Round 5 (2016)OC 8 years old 12 years old 15 years old 19 years old 22 years old
Raven's test PPVT PPVT -Math* Math Math Math
Cloze test Reading comprehension
YC 1 year old 5 years old 8 years old 12 years old 15 years oldPPVT PPVT PPVT PPVT
CDA quantitative Math Math MathReading
comprehension
Note: *One Item; CDA=Cognitive Development Assessment ; PPVT=Peabody Picture Vocabulary Test; Cloze test=Cloze test on reading comprehension
Soft skillsCohort Round 1 (2002) Round 2 (2007) Round 3 (2010) Round 4 (2013) Round 5 (2016)
OC 8 years old 12 years old 15 years old 19 years old 22 years oldAgency Agency Agency AgencyPride Pride Pride PrideTrust TrustInclusion InclusionSubjective wellbeing
Subjective wellbeing
Subjective wellbeing
Subjective wellbeing
Self-esteem Self-esteemSelf-effi cacy Self-effi cacy
Parent relations Parent relationsPeer relations
GritNeuroticism, ConscientiousnessJob skills
YC 1 year old 5 years old 8 years old 12 years old 15 years oldAgency Agency AgencyPride Pride PrideSubjective wellbeing
Subjective wellbeing
Subjective wellbeing
Parent relations Parent relationsPeer relations Peer relations
Aspirations and expectationsCohort Round 1 (2002) Round 2 (2007) Round 3 (2010) Round 4 (2013) Round 5 (2016)
OC 8 years old 12 years old 15 years old 19 years old 22 years oldAspirations about Marriage and Fertility
Aspirations about Marriage and Fertility
Educational aspirations/expectations
Educational aspirations/expectations
Job Aspirations/Expectations
Job Aspirations/Expectations
Job Aspirations/Expectations
YC 1 year old 5 years old 8 years old 12 years old 15 years oldAspirations about Marriage and Fertility
Educational aspirations/expectations
Educational aspirations/expectations
Job AspirationsJob Aspirations/Expectations
Job Aspirations/ExpectationsSubjective earnings expectations