emilio di meglio and emanuela di falco (eurostat)
DESCRIPTION
Continuous improvement of EU-SILC quality: standard error estimation and new quality reporting system. Emilio Di Meglio and Emanuela Di Falco (EUROSTAT). Why variance estimation?. Requested by regulation Quality report Compliance Requested by users Policy relevance of indicators - PowerPoint PPT PresentationTRANSCRIPT
Continuous improvement of EU-SILC quality: standard error estimation and
new quality reporting system
Emilio Di Meglio and Emanuela Di Falco (EUROSTAT)
Q2014 Conference Vienna 1
Why variance estimation? Requested by regulation
Quality report Compliance
Requested by users Policy relevance of indicators
Requested by researchers
Q2014 Conference Vienna 2
Current legal requirements According to Reg.1982/2003, the X and L (initial sample) data
are to be based on a nationally representative probability sample of the population residing in private households.
Representative probability samples shall be achieved both for households and for individual persons in the target population.
The sampling frame and methods of sample selection should ensure that every individual and household in the target population is assigned a known and non-zero probability of selection.
Reg. 1177/2003 defines the minimum effective sample sizes to be achieved.
Q2014 Conference Vienna 3
Main challenges for EU SILC Difficulty to find the « best » possible method for
variance estimation at Eurostat level– Different designs (flexibility)– Missing information– Debate on methods ongoing
Differentiate the needs: accuracy estimates for policy usage and accuracy estimates for researchers.
Q2014 Conference Vienna 4
Sampling design by country (2012)
Q2014 Conference Vienna 5
Sampling design Country Without stratification
Simple random sampling MT ,DK, IS Systematic sampling SE,NO
With stratification Stratified sampling according to different design by rotational group HU
Stratified simple random sampling LU, CY, SK, CH, LT, DE*,AT Stratified and systematic sampling EE Stratified multi-stage sampling CZ, ES, PL,RO,IE Stratified two-stage clustered sampling PT Stratified two-stage systematic sampling SI, NL, HR Stratified multi-stage systematic sampling FR, LV, UK, BE, BG, EL, IT Stratified two-phase sampling FI * from former participants of micro census
Our objective Resampling taking into account all the possible elements
coming from 32 countries would be extremely computationally and resource intensive
Variance estimation methods balancing between scientific accuracy and administrative considerations (time, cost, simplicity) are the only viable solution
Aim: to quickly provide to users and policy makers standard errors for the SILC-based indicators, particularly the AROPE (At-Risk-Of-Poverty or social Exclusion), its components and its main breakdowns.
Q2014 Conference Vienna 6
The method (synthesis) Linearization is a technique based on the use of linear approximation to
reduce non-linear statistics to a linear form, justified by asymptotic properties of the estimator (Särndal et al, 1992 ; Deville, 1999 ; Wolter, 2006 ; Osier, 2009)
The "ultimate cluster" approach (Särndal et al, 1992) is a simplification consisting in calculating the variance taking into account only variation among Primary Sampling Unit (PSU) totals
This method requires first stage sampling fractions to be small which is nearly always the case.
This method allows a great flexibility and simplifies the calculations of variances.
It can also be generalized to calculate variance of the differences of one year to another (Berger, 2004 , 2010 ).
Applicable with the main statistical packages (SAS, R, STATA)
Q2014 Conference Vienna 7
Results on AROPE For 6 countries 95% Confidence Interval for AROPE equal or
smaller that ±1.0% (CZ, IT, SI, DE, FI, NO)
For 9 countries 95% Confidence Interval for AROPE between ± 1% and ±1.5% (ES, PL, UK, EE, AT, SK, CH, SE, IS)
For 8 countries 95% Confidence Interval for AROPE between ±1.5% and ±2% (BE, DK, HR, HU, NL, PT, CY, MT)
For 6 countries 95% Confidence Interval for AROPE larger than ±2% (BG, EL, IE, RO, LT, LV)
Complete results in EU-SILC quality reportQ2014 Conference Vienna 8
Measurement of net changesTo measure the significance of the evolution of social indicators
Example: When the At-risk-of-poverty or social exclusion rate for Poland goes from 27.2% in 2011 to 26.7% in 2012, are we able to say that this change is significant?
Exercise already done for AROPE and other main EU-SILC indicators
Q2014 Conference Vienna 9
Output
Q2014 Conference Vienna 10
CountryAROPE (2011)
%
AROPE (2012)
%
Difference 2012 – 2011
(% points)
Standard error
(% points)
Significance of change
HU 31.0 32.4 1.5 0.7 YMT 21.4 22.2 0.8 0.4 YNL 15.7 15.0 -0.8 0.2 YPL 27.2 26.7 -0.5 0.3 N
EU-SILC Quality reports(Reg. No 1777/2003)
At national level, Member States have to produce: • An Intermediate QR (by the end of the year N+1)Based on cross-sectional data of year N• A Final QR (by the end of the year N+2)Based on longitudinal and cross-sectional data year N
At European level, EUROSTAT has to produce :• EU Comparative Intermediate QR (by June of the year N+2)Based on the national Intermediate QRs• EU Comparative Final QR (by June of the year N+3)Based on the national Final QRs
Q2014 Conference Vienna 11
Quality reporting Revision process• New template (ESQRS)
• Revision of the Contents• Introduction of annexes and questionnaire
• ESS Metadata Handler (old NRME)
Q2014 Conference Vienna 12
EU SILC key quality dimensions• Accuracy• Comparability• Coherence National ESQRS• Cost and burden• Statistical processing
• Timeliness and punctuality• Relevance EU ESQRS
Q2014 Conference Vienna 13
Availability of quality metadata• Quality reports• Questionnaires• Methodological papers
Further action: integrate more information in a wiki platform
Q2014 Conference Vienna 14
Conclusion and future plans The variance estimation methodology is of relatively simple
application It can be considered as a good compromise between scientific
soundness and feasibility under current constraints. The next steps consist in still improving these calculations by asking
Member States to provide the necessary information where missing. Dissemination of further information to users. Better disseminate quality reports
Q2014 Conference Vienna 15