measuring the quality of regional estimates from the abs jennie davies and daniel ayoubkhani
TRANSCRIPT
Measuring the quality of regional estimates from the ABSJennie Davies and Daniel Ayoubkhani
Overview
• ABS• User needs• ABS estimation• Methods• Results• Recommendations• Outcomes
Annual Business Survey (ABS)
• ONS’s largest business survey• Sample of ~73,000 UK businesses• Covers agriculture (part), production,
construction, distribution and service (part) sectors
• Variables (collected and derived) include:TurnoverPurchases of goods and servicesApproximate Gross Value AddedNet capital expenditure
Annual Business Survey (ABS)
• National publication (November and June)4-digit SIC breakdown
• Regional publication (July)12 UK regions, 2-digit SIC breakdown
• Special Analysis systemAd-hoc requests for lower level estimates (eg local
authority, 3-digit SIC by employment sizeband etc)
• Standard errors and CVs provided with national estimates but not regional
User needs
• Large range of ABS usersCentral government, local government and devolved
administrations, Eurostat, National Accounts, academia, consultancy firms, media, public
• Quality measures therefore important for sound decision making
• Lack of standard errors for regional estimates makes it difficult for users to assess accuracy
User needs
• Recent UKSA Assessment of ABS:“there is insufficient information about methods and
quality”
“there is no information about the resulting quality of the statistics and no caveats around their use”
• QIF project therefore undertaken to develop methodology for calculating standard errors of published regional ABS estimates
• Aim to provide users with information about the quality of regional estimates
Reporting unit data
National estimates
Modelled local unit data
Regional estimates
ABS estimation
Regional apportionment model
Ratio estimation
Ratio estimation
National standard errors
GES
Regional standard errors
???
Small area estimation for small domains
7
Standard error estimation (1)
• Assume that the apportioned values are “true” returns
• Single stage cluster sampling• Use GES to calculate standard errors
Standard error estimation (2)
• The regional apportionment model parameters depend on the sample data so are variable
• Use bootstrapping to capture the use of the regional apportionment model
Bootstrapping
• Standard errors capture sampling variability ie how much estimates vary under different possible
samples
• Bootstrapping re-samples from the original sample to create a new sample
• Carry out estimation on new sample• Repeat lots of times• Calculate standard error of the resulting
estimates
Bootstrapping
• Fix the model parameters based on the full sample dataquality assure method against GES
• Re-fit the regional apportionment model in each iterationinclude (possible) additional variance from the model
Results
• Compared the methods in terms of:Differences in standard errors
Practical considerations
• Results for turnover presentedOther variables produced similar results
GES vs bootstrap without re-fitting the model
• Bootstrapping without re-fitting the regional apportionment model in each iteration should be comparable with estimates from GES
Bootstrapping with and without re-fitting the model
• Comparing these to see if the regional apportionment model increases variances
• Compared to bootstrap rather than GES to remove additional differences seen before
Practical implications
• Bootstrapping took ~30 hours• Relied on exact replication of the regional
apportionment model Problem for derived variables such as aGVA
Recommendations
• Use GES to produce standard errors• With caveats that the regional apportionment
model is assumed to be fixed
Outcomes
• Report published on ONS website (Feb 2014)• Methodology approved by ABS Survey
Management Board• Standard errors of regional estimates to be
published for first time in July 2014• Another QIF bid submitted to investigate
standard errors of small area ABS estimatesWould allow for quality measures to be published
alongside majority of Special Analysis requests
Summary
• ABS• User needs• ABS estimation• Methods• Results• Recommendations• Outcomes