measuring coverage: post enumeration surveys owen abbott office for national statistics, uk
TRANSCRIPT
![Page 1: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/1.jpg)
Measuring Coverage:Post Enumeration Surveys
Owen AbbottOffice for National Statistics, UK
![Page 2: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/2.jpg)
Agenda
• Introduction• Why have a PES?• Essential features of a PES
– Survey Design– Fieldwork
• Analysing the data– Matching– Estimation
• Results from 2001 UK Census• Discussion
![Page 3: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/3.jpg)
Why do we need a PES?
• Census won’t count every household or person• Undercount causes bias in estimates• In the UK in 2001, we estimated that 3 million
persons (6%) did not fill in the form• Increasing problem from 1981 to 1991 to 2001• The undercount is not evenly spread
– Inner Cities – Deprived areas – Young persons
![Page 4: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/4.jpg)
Why do we need a PES?
• Census counts alone not good enough• UK Users demand robust census population
estimates– Central Government resource allocation– Yearly demographic population estimates– Government Policy
• So we need to measure how many households and persons the census misses, and work out:
– where they are missed from– their characteristics
![Page 5: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/5.jpg)
Basic Methodology
• PES - Census Coverage Survey (CCS) in UK– In the UK approx 1% population
• Match the PES to the Census• Use the people the PES sees that the census didn’t
to estimate how many missed– where and characteristics
• Add to the Census counts (either at aggregate level or impute (UK))
![Page 6: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/6.jpg)
2001 UK ‘One Number Census’ frameworkCENSUS
CENSUS +CCS
DESIGN GROUPESTIMATE BYAGE AND SEX
LADESTIMATES
ADJUSTEDINDIVIDUAL AND
HOUSEHOLD DATAAND TABLES
NATIONALPOPULATIONESTIMATE
MATCHING
QualityAssurance
Dual System andregression estimation
CCS
Synthetic estimation
Imputationcontrolled to LAD
estimates
Sum
![Page 7: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/7.jpg)
Post Enumeration Survey
Key features:
A - Design– Sample survey– Sample size dependent on accuracy (and geographic
level) requirements
B - Fieldwork– Conducted after the census has finished– Independent re-enumeration– Area based– Door to door interview– Focused on measuring coverage
![Page 8: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/8.jpg)
Post Enumeration Survey - Design
• Multi-stage Stratified sample • Select a sample of (small) geographical areas that
can be re-enumerated– UK uses Postcodes (about 20 hhs)– US uses blocks (about ????100 hhs)
• Sample stratified by:– Geography– Area type– Demography
![Page 9: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/9.jpg)
2001 UK PES Design
Geographical Strata:• Local Authorities (mean pop 120k) grouped into
contiguous groups called Estimation areas (EAs), each having 500k pop
Area Type and Demographic strata:• Within every EA a sample of 1991 Enumeration
Districts was selected, stratified using a hard-to-count index and the 1991 age-sex structure
– (1991 EDs have about 200 households)
![Page 10: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/10.jpg)
2001 UK PES Design
• Hard to count index was a national stratification using a combination of variables associated with undercount e.g:
– Unemployed– Multi-occupied– Private rented– Language difficulty
• 3 level index, split into 40%, 40%, 20% nationally
• Within each selected ED a sample of 3, 4 or 5 postcodes was selected
![Page 11: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/11.jpg)
Post Enumeration Survey - Field
• Aim: enumerate all the people and households in the sampled areas
• Carry out the survey after the Census– Census fieldwork finished
• Independence critical (see later)– Interview based– Independent re-enumeration– Separate fieldforce and management– No address list (UK have address list for Census)– Difficult if doing quality at same time, as not independent
![Page 12: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/12.jpg)
Post Enumeration Survey - Field
• In UK, focused on measuring coverage– Previously measured quality as well– Found that separate surveys more effective– Can focus on getting maximal response in sampled areas
• UK 2001 PES used very short interview– key household and demographic questions only
• Accommodation type
• Tenure
• Name
• Gender
• Date of Birth (or Age)
• Student
• Ethnicity
• Activity last week
![Page 13: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/13.jpg)
Post Enumeration Survey - Field
• Other initiatives to maximise response:– Pairwork and teamwork– Refusal avoidance training – Calling strategy– Up to 10 attempts to interview– Last attempt deliver form to return in post
![Page 14: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/14.jpg)
Post Enumeration Survey
• Interviewer Duties:– Establish the postcode boundaries
– Conduct independent listing of all residential and non-residential addresses
– Seek out obscure accommodation
– Deliver advance notification cards
– Identify/probe for all households at an address
– Make contact with householders
– Conduct doorstep interviews
– Persuade potential refusals
– Report Progress
![Page 15: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/15.jpg)
Post Enumeration Survey
• Map
![Page 16: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/16.jpg)
Post Enumeration Survey
• Property Listing Listing Sheet for Postcode …………………….. Sheet ….…. of ……….
Interviewer No. Address/Household Advance Leaflet Delivered
Form No.
Date 1 Date 2 Date 3 Date 4 Date 5 Date 6 Date 7 Date 8 Date 9 Date 10 Time 1 Time 2 Time 3 Time 4 Time 5 Time 6 Time 7 Time 8 Time 9 Time 10 Interview Refusal Non-residential Vacant Visitor only Communal
Notes:
![Page 17: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/17.jpg)
Analysing the data - Matching
• Match Census returns to CCS returns• Require very high quality
– Minimise false negative matches (missed matches, see later)
• In 2001, we used hierarchical nature of data to help match
– Match within sampled areas (geographical blocking)– First match household– Then match persons within households
![Page 18: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/18.jpg)
Analysing the data - Matching
• Used a five stage strategy, designed to minimise false negative matches:
– Exact matching– High probability matching– Clerical assisted probability matching– Clerical matching– Final expert review of non-matches
• Developed our own in-house system• Allowed access to scanned form images (this was
crucial)
![Page 19: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/19.jpg)
PO155RR
ERIC
SMITH
13
MALE
SINGLE
ERIC
SMITH
13
MALE
SINGLE
PO155RR
29 29
![Page 20: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/20.jpg)
Analysing the data - Matching
• Output:– Match between Census and CCS– Census only– CCS only
![Page 21: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/21.jpg)
Analysing the data – Estimation
• Dual System Estimation (DSE)– Capture-recapture as used for wildlife
• Simple example: How many fish in a lake?– Catch as many as possible on day 1
• Count them (N1)
• Mark with a red dot• Return them to the lake
– Catch as many as possible on day 2• Count them (N2)
• Count how many have red dots (N12)
– Number of fish in lake= (N1 * N2)/N12
![Page 22: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/22.jpg)
Analysing the data - Estimation
• Use matched Census+CCS data• DSE estimates adjustment for those missed in both
Census and CCSCounted By CCSYes No
Counted Yes n11 n10 n1+
By Census No n01 n00 n0+
n+1 n+0 n++
DSE count (for a postcode):
n++ = n1+ x n+1 n11
![Page 23: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/23.jpg)
Analysing the data - Estimation
• DSE assumptions– Independence– Homogeneity of capture probabilities– Perfect matching– Closure– No list inflation
• Violation of these assumptions leads to bias (in both directions)
• Lots of literature on DSE
![Page 24: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/24.jpg)
Analysing the data – Estimation
• DSE can only be used within the sample• Need additional step to get to population totals• In 2001, we used DSE at postcode level• Then used a ratio estimator to predict for non-
sampled postcodes (again lots of literature)
Census
DSE
![Page 25: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/25.jpg)
Analysis – Getting to small areas
• Ratio estimator produced estimates for 500k population blocks
• Needed estimates for Local Authorities (about 120k population)
• Sample size not sufficient to do directly• So used small area estimation techniques
– these borrow strength across areas– We used a fixed effect to model LA differences
• LA population estimates from the model then constrained to EA totals
![Page 26: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/26.jpg)
Quick summary of 2001 UK method
• In 2001, One Number Census methodology was developed
– Large CCS (320,000 households)– Matching– Capture Recapture– Modified ratio estimator– Small area estimation to get LA totals– Imputation
• Estimated 1.5 million households missed• 3 million persons missed (most from the missing
households but some from counted households)
![Page 27: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/27.jpg)
Results
• England and Wales population about 50m individuals in 20m households
• Estimated 1.5 million households missed• 3 million persons missed (most from the missing
households but some from counted households)
![Page 28: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/28.jpg)
Underenumeration in 2001Underenumeration of Census by agegroup
0.0%
2.0%
4.0%
6.0%
8.0%
10.0%
12.0%
14.0%
16.0%
0 1-4 5-9 10-14 15-19 20-24 25-29 30-34 35-39 40-44 45-49 50-54 55-59 60-64 65-69 70-74 75-79 80-84 85+
Agegroup
ON
C/C
en
su
s
Males Females
![Page 29: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/29.jpg)
Response Rates in 2001
![Page 30: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/30.jpg)
Summary
• Fundamental that the census is good– This does not make a bad census good, it makes a
good census better!
• US, Australia, NZ, Canada, UK all measure coverage (and most use a PES)
– All aim at measuring coverage for assessing census quality, most do not fully adjust the outputs
– Coverage for most is around 96-98%– Increasing problems of overcoverage
• The design and fieldwork of the PES are important to get right
![Page 31: Measuring Coverage: Post Enumeration Surveys Owen Abbott Office for National Statistics, UK](https://reader036.vdocuments.mx/reader036/viewer/2022062511/5514b533550346b2598b64d6/html5/thumbnails/31.jpg)
More info
• Brown, J.J., Diamond, I.D., Chambers, R.L., Buckner, L.J., and Teague, A.D. (1999), “A methodological strategy for a one-number census in the UK,” Journal of the Royal Statistical Society A, 162, 247-267.
• www.statistics.gov.uk/census2001/onc.asp