volume 1 quality in statistics - home - · pdf filevolume 1 quality in statistics ... 3.3...

Standards and Guidelines

Volume 1

Quality in Statistics

Quality Assurance & Audit Section Version 1.0 January 2006

Volume 1: Quality in Statistics 3

TABLE OF CONTENTS

1 INTRODUCTION ................................................................................................. 4

2 OFFICE POLICY ................................................................................................. 4

3 QUALITY IN STATISTICS ................................................................................... 5

3.1 Defining Quality ....................................................................................................................................... 5

3.2 The ESS Quality Framework ................................................................................................................... 6

3.3 Quality Control and Auditing ................................................................................................................... 6

4 STATISTICAL QUALITY STANDARDS AND GUIDELINES ............................... 7

4.1 Planning for Statistics Production ............................................................................................................ 7

4.2 Classifications and Standards ................................................................................................................... 7

4.3 Survey Frame ........................................................................................................................................... 8

4.4 Sampling Procedures ................................................................................................................................ 9

4.5 Questionnaire Design ............................................................................................................................... 9

4.6 Measures to Reduce Non-Response ....................................................................................................... 10

4.7 Data Editing ........................................................................................................................................... 11

4.8 Data Quality Evaluation (including Macro-Editing) .............................................................................. 11

4.9 Seasonal Adjustment .............................................................................................................................. 12

4.10 Statistical Confidentiality ....................................................................................................................... 13

4.11 Presentation and Dissemination ............................................................................................................. 14

4.12 Use of Administrative Data .................................................................................................................... 14

4.13 Documentation and Metadata ................................................................................................................. 15

4.14 Revisions to Published Data ................................................................................................................... 15

REFERENCES ......................................................................................................... 17

APPENDICES .......................................................................................................... 18

A. European Statistics Code of Practice (2005) .......................................................................................... 18

B. UN Fundamental Principles of Official Statistics .................................................................................. 23

C. Recommendations of LEG on Quality ................................................................................................... 24


1 Introduction Good quality information is essential for informed public debate and decision-making. As the

provider of official national statistics it is vital that the output of the Office is of the highest quality.

Several National Statistical Institutes (NSIs) have advanced their reputations through the

implementation of effective quality assurance procedures. The issue of quality assurance was

addressed in the Deloitte & Touche Report to the National Statistics Board – Review of

Organisational Performance and Capability of the Central Statistics Office – May 1997. The report

recommended the establishment of quality assurance and internal audit functions. This

recommendation was a High Level Goal in the Statement of Strategy 2001-2003 and the Quality

Assurance & Audit section was duly set up in 2002.

This document, Quality in Statistics, mainly draws on guidelines developed by other NSIs1,2

and

adapted to fit the Office’s statistical system. The document has a number of objectives and uses:

To introduce and define what is meant by quality in the realm of official statistics (Section 3)

To further awareness of the importance of quality in the work of the Office

To help ensure adherence to principles to which the Office subscribes (see Section 2)

To form the basis of the Office’s quality assurance system

To identify approaches to improve the overall quality of statistical outputs

To set out in a clear and unambiguous way the high level rules under which the Office

carries out, or aims to carry out, its work. These can be seen as goals to which all engaged in

the work of the Office must aspire.

Quality in Statistics represents a visible symbol of the Office’s commitment to quality.

Overall the purpose of the document is to be of assistance to those working in the Office and

particularly those who are directly responsible for the data we produce.

2 Office Policy The Office subscribes to the European Statistics Code of Practice (see Appendix A) and the UN

Fundamental Principles of Official Statistics (see Appendix B). Eurostat’s increasing quality

requirements are an ongoing consideration for the Office.

Customer Service has been set as a key competency for all staff in the Office. Quality is at the core of

customer service and is a challenge for all.

This document is to be regarded as Office policy. It contains an agreed set of ways and means of

ensuring that standards are met. It provides information on good practice and lists guidelines which

should be followed in the work of the Office. Implementation of these guidelines, where relevant, is a

requirement for each business area. However, it is accepted that the professional judgement by a

relevant staff member in particular situations and circumstances may result in the guidelines not being

fully implemented.

For a summary of the Deloitte & Touche report’s recommendations see: Corporate Documents\CSO\General

information\1997 Deloitte & Touche Report on the CSO - Summary of Recommendations Made and

Management Response


3 Quality in Statistics

3.1 Defining Quality

Until fairly recently the quality of statistical output has traditionally been viewed in terms of accuracy.

However, quality as employed in other activities has generally included broader interpretations. These

tend to take in concepts such as ‘fitness for use’, ‘meeting user need’ and ‘customer satisfaction’.

Such concepts are clearly appropriate for official statistics and highlight the shortcomings of accuracy

alone as a measure of overall quality. Quality initiatives in the European Statistical System (ESS)

have reflected this need for a broader interpretation. The International Standards Office (ISO)

approach is considered particularly appropriate, to both statistical products and statistical services.

They define quality as ‘the totality of features or characteristics of a product or service that bear on its

ability to satisfy stated or implied needs of customers’.

Arising from the broader interpretations of quality is a need to define the elements that impact on

quality and that can be used to characterise quality. There is no universally agreed list of the

characteristics which define quality, however there is considerable overlap in the approaches adopted

by various NSIs. A lot of effort has gone into standardising quality concepts within the ESS3.

In February 2005, the ESS Statistical Programme Committee (SPC) unanimously endorsed the

European Statistics Code of Practice. The Code of Practice has the dual purpose of:

- Improving trust and confidence in the independence, integrity and accountability of both National

Statistical Authorities and Eurostat, and in the credibility and quality of the statistics they

produce and disseminate

- Promoting the application of best international statistical principles, methods and practices by all

producers of European Statistics to enhance their quality.

The code is a self-regulatory instrument consisting of 15 principles grouped into three sections

addressing respectively the institutional environment, statistical processes and statistical outputs as

follows: -

European Statistics Code of Practice: 15 Principles

Institutional Environment Statistical Processes Statistical Output

1. Professional Independence 7. Sound Methodology 11. Relevance

2. Mandate for Data Collection 8. Appropriate Statistical Procedures 12. Accuracy and Reliability

3. Adequacy of Resources 9. Non-Excessive Burden on

Respondents

13. Timeliness and Punctuality

4. Quality Commitment 10. Cost Effectiveness 14. Coherence and Comparability

5. Statistical Confidentiality 15. Accessibility and Clarity

6. Impartiality and Objectivity

A full explanation of the Code of Practice and indicators of good practice for each of the 15 Principles

are given in Appendix A.

The ESS comprises Eurostat and the statistical offices, ministries, agencies and central banks that

collect official statistics in EU Member States, Iceland, Norway and Liechtenstein.


3.2 The ESS Quality Framework

The principle risk for any NSI is that it should face a loss of credibility by users. Consequently, the

last decade has seen considerable effort spent on assuring and evaluating the quality of the statistics

produced. Much of the work has been undertaken co-operatively between NSIs in the ESS. In 1998

Quality of Statistics was the theme of the annual conference of Presidents and Directors-General of

the European NSIs. This led in 1999 to the establishment of a Leadership Group (LEG) on Quality.

The LEG was given a broad remit to:

Establish a framework for considering quality issues

Identify key elements to be considered

Obtain information on the status of these elements in the ESS

Demonstrate with examples how improvements in NSIs and in the ESS could be made

Propose future actions for the ESS.

The output of the LEG consisted of a summary report (see SMD\Quality\Summary Report from the

Leadership Group (LEG) on Quality) containing 22 recommendations (see Appendix C).

Several of the recommendations have been acted on and quality related requirements specified in

Council Regulations now apply to certain statistics required by Eurostat. Such statistics include the

Labour Force Survey, Short-term Business Statistics, Structural Business Statistics and Labour Costs

Statistics

All NSIs within the ESS have signed up to the LEG report, therefore the Office is committed to its

recommendations.

3.3 Quality Control and Auditing

Internal audit forms part of any quality control system. Quality Assurance & Audit Section will

conduct audits to monitor the implementation of quality standards and to evaluate systems throughout

the Office. Such audits might result in reports with recommendations to management with

responsibility and authority for the matter in question.

It is intended to issue, separately, standards and guidelines in relation to non-statistical (including

financial) aspects of the Office’s work. Auditing of internal financial systems will be undertaken.

Requirements arising from the Report of the Working Group on the Accountability of Secretaries

General and Accounting Officers (Mullarkey Report) will continue to be addressed.


4 Statistical Quality Standards and Guidelines

4.1 Planning for Statistics Production

The planning process for a new statistical activity or the redesign of an ongoing activity should

include the definition of broad objectives, a targeted user population and the key questions or issues to

which analysis will be directed. In order to translate this initial planning into actual production,

objectives and uses should be stated precisely to help ensure that the new or redesigned activity will

meet specific user requirements.

The Office has set up a Project Office and has adopted project management as a core standard for the

way the Office conducts its business. The principles of project management apply to all projects

regardless of size. The document Project Management Guidance for the CSO (see Corporate

Documents\Strategic Planning\Project Management\Documents\Project Management Guidance for

the CSO) explains how project management is effectively applied in the CSO.

Guidelines

4.1.1 Focus analysis of user needs on finding the most cost-effective solution for both the short and

long term.

4.1.2 Develop survey objectives in partnership with important users and stakeholders through, for

example, Liaison Group meetings. Establish and maintain relationships with users in order to

enhance the relevance of the information produced and as part of marketing products and

services.

4.1.3 In determining the extent to which a survey will meet user needs, seek a reasonable trade-off

between these needs and the budget, response burden and confidentiality considerations.

Although the Office may have little discretion where a legal requirement is in place, in other

cases look at alternative methodological approaches, frequencies, geographical details, etc.

with a view to arriving at an optimum solution.

4.1.4 Review ongoing statistical activities at regular intervals. Statistical activities need to evolve,

adapt and innovate to keep pace with the demands of the users they serve.

4.1.5 Apply project management to the statistical activity. State clearly the scope of the project and

agree a detailed project plan including budgets and resource allocations.

4.2 Classifications and Standards

The Classifications and Related Standards system (CARS) (see Corporate Documents\CSO\Home

Pages\Classifications & Standards Homepage) is the Office's system for storing and accessing

classifications. The CARS database is the Office's central repository for all classifications,

concordances and coding indexes.

The purpose of CARS is to:

Use database technology to provide centralised classification storage, maintenance and access

facilities for classification data that are used both in the development and processing of

surveys and in the subsequent analysis and evaluation of the data

Help reduce the time and resources required when developing surveys and to contribute to

improved data quality by supporting the use of standard classifications

Facilitate the comparison and analysis of data by storing concordances.


Guidelines

4.2.1 All classifications used in the Office must reside on CARS. The responsibility for ensuring

this rests with the Office's business areas.

4.2.2 The official Office policy for CARS (see Corporate Documents\IT\Systems\Policy for Using

CARS) must be followed in all matters relating to classifications.

4.3 Survey Frame

A survey frame is any list or register that delimits, identifies, and allows access to the elements of the

target population. The target population is the set of elements about which information is wanted and

estimates are required. The extent to which a survey frame includes all the elements of the target

population is referred to as coverage. Practical considerations may dictate that some units be excluded

(e.g. companies with less than five employees, institutionalised individuals) from some frames.

The survey frame should conform to the target population and contain minimal undercoverage and

overcoverage (e.g. duplication). Frame creation, use, maintenance and monitoring should be

implemented within operational and cost constraints.

Characteristics of the frame units (e.g. classification, contact, address, size) should be of high quality

because of their use in stratification, collection, follow-up, estimation, record linkage, quality

assessment and analysis.

Guidelines

4.3.1 In designing business surveys, or in the redesign of existing ones, the Office’s Business

Register should be used to construct the appropriate survey frame.

4.3.2 Where possible, use the same frame for surveys with the same target population, to avoid

inconsistencies and to reduce costs of frame maintenance and evaluation.

4.3.3 Incorporate procedures to eliminate duplication and to update for births, deaths, out-of-scope

units and changes in characteristics.

4.3.4 Monitor the frame quality by periodically assessing its coverage.

4.3.5 For area frames, implement map checks to ensure clear and non-overlapping delineation of

the geographic areas used in the sampling design (e.g. through field checks or the use of other

map sources).

4.3.6 For statistics production from administrative sources, determine and monitor coverage

through contact with the source manager. Where influence on the frame is possible, negotiate

required changes with the source manager.

4.3.7 Whenever necessary, adjust the statistical results or use supplementary data to offset coverage

differences between the frame and the target population.

4.3.8 Include descriptions of the target population, frame and coverage in the survey

documentation.


4.4 Sampling Procedures

Sampling is the selection of a set of units from a survey frame. This set of units is referred to as the

sample. The choice of sampling method has a direct impact on the data quality. It is influenced by

many factors, including the desired level of precision of the information to be produced, the

availability of appropriate frames, the availability of suitable stratification variables, the estimation

methods that will be used and the available budgets.

The intention is to gather useful information from the sampled units to allow inferences about the

target population.

Guidelines

4.4.1 When determining sample size, take into account the required levels of precision needed for

the survey estimates, the type of design and estimator to be used, the availability of auxiliary

information, as well as both sampling factors (e.g. stratification) and non-sampling factors

(e.g. non-response).

4.4.2 For highly skewed populations, include in the survey a stratum of large units that will be

sampled with certainty.

4.4.3 In determining sample allocation for stratified samples, account for expected rates of

misclassification of units in the frame.

4.4.4 For periodic surveys that use designs in which the sample size grows as the population

increases, develop a method to keep the sample size stable.

4.4.5 For periodic surveys, if efficient estimates of change are required or if response burden is a

concern, use a rotation sampling scheme that replaces part of the sample in each period.

4.4.6 For periodic surveys develop procedures to monitor the quality of the sample design over

time. Set up an update strategy for selective redesign of strata that have suffered serious

deterioration.

4.5 Questionnaire Design

A questionnaire is a set of questions designed to collect information from a respondent. A

questionnaire may be interviewer-administered or respondent-completed, using paper methods of data

collection or electronic modes of completion. Questionnaires play a central role in the data collection

process. They have a major impact on data quality, respondent behaviour, interviewer performance

and respondent relations.

The design of questionnaires takes into account the statistical requirements of data users,

administrative requirements of the survey organisation, and the requirements for data processing, as

well as the nature and characteristics of the respondent population.

Guidelines

4.5.1 Questionnaires in periodic surveys should be evaluated regularly.

4.5.2 Use words and concepts that have the same meanings for both respondents and the

questionnaire designers. In the case of business surveys, choose questions, time reference

periods and response categories that are compatible with the respondent's record-keeping

practices.


4.5.3 In the introduction to all questionnaires:

Provide the title or subject of the survey

Explain the purpose of the survey

Request the respondent’s co-operation

Indicate the authority under which the survey is taken, and what confidentiality protection

arrangements are in place.

4.5.4 Ensure that the value of providing information is made very clear to respondents. In addition,

the importance of completing the questionnaire and how the survey data will be used must be

highlighted.

4.5.5 The opening questions should be applicable to all respondents, be easy and interesting to

complete, and establish that the respondent is a member of the target population.

4.5.6 Questionnaires that are to be administered in person or over the telephone must be made

interviewer-friendly as well as respondent-friendly.

4.5.7 Ensure that the instructions to respondents and or interviewers are short, clear, and easy to

find. Provide definitions at the beginning of the questionnaire or in specific questions, as

required.

4.5.8 Ensure that time reference periods and units of response are clear to the respondent, specify

“include” or “exclude” in the questions themselves and not in separate instructions.

4.5.9 Ensure that response categories are mutually exclusive and exhaustive.

4.5.10 Provide titles or headings for each section of the questionnaire, and include instructions and

answer spaces that facilitate accurate answering of the questions.

4.6 Measures to Reduce Non-Response

Non-response has two effects on results: one contributing to bias of estimates when non-respondents

differ from respondents in the characteristics measured; the other contributing to a decrease in the

accuracy of the survey estimates resulting from the smaller effective sample size.

The degree to which response is pursued is subject to budget and time constraints and the risk of non-

response bias. Adjustments are subsequently made to data to compensate for non-response (e.g.

weighting adjustments or imputation).

An effective respondent relations programme and a well-designed questionnaire are critical elements

in maximising response.

Guidelines

4.6.1 Establish and maintain good relationships with respondents.

4.6.2 Ensure interviewers are fully trained in interviewing techniques etc.

4.6.3 When operational constraints permit, follow-up the non-respondents either as a complete

enumeration or on a sub-sample basis.

4.6.4 Prioritise follow-up activities. For example, in business surveys, follow-up large or influential

units first.


4.6.5 Record and monitor reasons for non-response (e.g. refusal, non-contact, temporarily absent,

technical problem).

4.6.6 Use an appropriate method of imputation to compensate for non-response. Only as a last

resort should grossing factor adjustment be used.

4.6.7 Where applicable, ensure that those being surveyed are aware that the survey is statutory.

4.7 Data Editing

The goals of editing are to:

Provide the basis for future improvement of the survey vehicle

Provide information about the quality of the survey data

Tidy up the data.

It may be that a disproportionate amount of resources is concentrated on the third objective of

‘cleaning up the data’. A danger is that learning from editing processes may play an undeserved,

secondary role.

While it is recognised that fatal errors (e.g. invalid or inconsistent entries) should be removed from

the data sets in order to maintain credibility and to facilitate further automated data processing and

analysis, caution should be exercised against the overuse of query edits (those pointing to

questionable records that may potentially be in error).

Guidelines

4.7.1 Ensure that all edits are internally consistent (i.e. not self-contradictory).

4.7.2 Reapply edits to units to which corrections were made to ensure that no further errors were

introduced directly or indirectly.

4.7.3 Perform edit checks for missing values, invalid values, etc. as quickly and as expediently as

possible in the processing cycle.

4.7.4 Rationalise 'query' editing (i.e. checks for apparent errors or inconsistencies), and find an

appropriate balance between error detection and cost.

4.7.5 Consider editing to be an integral part of the data collection process in its role of gathering

intelligence about the process. Use editing to:

Sharpen definitions

Evaluate the quality of the data

Identify non-sampling error sources

Serve as a basis of future improvement of the whole survey process.

4.8 Data Quality Evaluation (including Macro-Editing)

Data quality evaluation refers to the process of evaluating the final statistical output in the light of the

original objective of the statistical activity, in terms of the data’s accuracy or reliability. Such

information allows users to make more informed interpretations of the survey results, and can be used

by the Office to improve surveys.


Two general types of data quality evaluation can be distinguished:

Macro editing or quality validation is the process of reviewing the data before official release

to ensure that grossly erroneous data are not released, or to identify data of marginal quality.

Sources of error studies generally provide quantitative information on specific sources of

error in the data. While timeliness is still important, the results of these studies often are only

available after the official release of the data.

Guidelines

4.8.1 Make planning of data quality evaluations part of the overall statistical process, as the

information needed to conduct such evaluations often must be collected during the process

itself.

4.8.2 Involve users of the results, whether they are external or internal, in setting the objectives for

the data quality evaluation programme. Where circumstances permit, also involve them in the

evaluation process itself.

4.8.3 The following macro editing and quality validation methods should be used:

Checks of consistency with external sources of data, for example from other surveys or

from previous instances of the same survey

Internal consistency checks, for example calculation of ratios that are known to lie within

certain bounds (sex ratios, average value of commodities, etc.)

Unit-by-unit reviews of the largest contributors to aggregate estimates, typically the case

in business surveys

Debriefings with staff involved in the collection and processing of the data.

4.8.4 The following sources of error should be evaluated:

Coverage errors, which consist of omissions, erroneous inclusions, and duplications in

the frame used to conduct the survey

Non-response errors, which occur when the survey fails to get a full response

Measurement errors, which occur when the response received differs from the ‘true’

value, and can be caused by the respondent, the interviewer, the questionnaire, the mode

of collection, or the respondent’s record-keeping system

Processing errors, which can occur at the subsequent steps of data editing, coding,

capture, imputation and tabulation

Sampling errors, which occur when the results of the survey are based on a sample rather

than the entire population.

4.9 Seasonal Adjustment

Seasonal adjustment consists of estimating seasonal factors and applying them to a time series to

remove the seasonal variations. These variations represent the composite effect of climatic and

institutional factors that repeat with a certain regularity within the year.

Many series are published in seasonally adjusted form to reveal the underlying trend movements and

to help data analysis. Seasonally adjusted series comprise not only the trend but also the irregular

component; consequently, they only give an approximate idea of the underlying trend movements. To

eliminate the irregular component the seasonally adjusted series may be further smoothed and the


trend derived. In some cases the trend estimates may be considered for publication if the seasonally

adjusted series is considered unsuitable.

The Statistical Methods and Development (SMD) Seasonal Adjustment Policy document (see

SMD\Quality\Corporate Seasonal Adjustment Policy) sets out the procedures to be followed for the

seasonal adjustment of data in the Office.

Guidelines

4.9.1 All time series which exhibit evidence of seasonality and for which the underlying seasonality

can be identified reliably should be seasonally adjusted.

4.9.2 Before publishing a seasonally adjusted time series for the first time, conduct a thorough

seasonal adjustment analysis to assess if the seasonality is identifiable.

4.9.3 Follow the procedures set out in the SMD Seasonal Adjustment Policy document.

4.9.4 For seasonal adjustment, use concurrent seasonal factors. These are the factors obtained using

all recent values of the series. This is will give rise to frequent, mostly small, revisions to

seasonally adjusted series. A definite policy on the publication of the revisions should be used

and clearly explained to users.

4.9.5 For aggregate series resulting from the addition or subtraction of component series, seasonally

adjust only those component series that contain identifiable seasonality, and leave the others

unadjusted. Seasonally adjust the aggregate series directly (i.e. do not use the aggregate of the

component seasonally adjusted series). Inform users that small discrepancies will arise in the

seasonally adjusted estimates of aggregate series.

4.9.6 Wherever seasonally adjusted figures pertaining to the same economic activity are published,

co-ordinate the seasonal adjustment options applied by the areas involved.

4.10 Statistical Confidentiality

The Statistics Act, 1993 (see Corporate Documents\CSO\Home Pages\CSO Policies Home Page)

provides that all information collected by the Office can be used only for statistical purposes and that

(subject to stated exceptions) any information which can be related to an identifiable person or

undertaking cannot be disseminated, shown or communicated to any person or body.

The requirements of the Statistics Act, 1993 are additional to:

The obligations on the staff of the CSO under the Official Secrets Act, 1963 not to make

unauthorised communications, directly or indirectly, about matters which come to their

knowledge in the course of their official duties

The obligations on the CSO under the Data Protection Act, 1988 regarding the handling of

personal data.

Guidelines

4.10.1 The Office's Code of Practice on Confidentiality (see Corporate Documents\CSO\Home

Pages\CSO Policies Home Page|CSO Statistical Confidentiality Code of Practice) should be

followed at all times.


4.11 Presentation and Dissemination

Presentation and dissemination are among the final stages in the statistical production process. While

presentation and dissemination procedures may not affect the accuracy of the statistical data, the

perception of quality by the users may be strongly influenced.

The forms of presentation and dissemination should facilitate access and interpretation by users.

Tables should have clear headings, identifiable units and logical layout. Charts should convey a clear

and accurate representation of the phenomenon being studied.

Guidelines

4.11.1 In official statistical releases and reports a suitable analysis highlighting the more important

and interesting features of the published statistics should be provided. This should be based

on the statistical results available and should never contain any political or similar

judgements. In general the larger and more infrequent the release or report the more detailed

the analysis should be.

4.11.2 Guidelines for tables:

Every table should be labelled to clearly identify the content

The table layout must be clear and easy to follow

All units of measurement should be displayed clearly

Where a comparison between numbers is required, these should be listed in columns to

facilitate reading

Ensure all rounding to significant digits is mathematically correct

Ensure that footnotes are clearly marked and that the text is clear and readable.

4.11.3 Guidelines for charts:

The chart title must explain what phenomenon is represented and the time periods

covered.

All axes must be clearly labelled and include the units of measurement.

Legends, labels and tick marks should all be clear and readable.

All elements on the chart should be identified.

Any apparent discrepancy should be highlighted and explained.

4.11.4 The document Write Well, Write Clearly (see Corporate Documents\Dissemination

\Information Section\Write Well, Write Clearly - CSO Usage and House Style (1996)) is the

official policy on written content and must be followed for all releases and publications.

4.12 Use of Administrative Data

The term administrative records refers to data collected for the purpose of carrying out various

programmes, for example, income tax collection.

Administrative records present a number of advantages. Since they already exist, costs of direct data

collection and further burden on respondents are avoided. They are usually available for the complete

universe, and hence, they are most of the time not constrained by sampling error limitations. Most

importantly, they can be used in numerous ways in the production of statistical outputs. Examples of

their uses include:

The creation and maintenance of frames


The complete or partial (via record linkage) replacement of statistical collection

Editing, imputation and weighting of data from statistical collection

The evaluation of statistical outputs.

Guidelines

4.12.1 Actively investigate and assess all potential sources of administrative data. Bear in mind that

even partially complete and/or partially inaccurate administrative data may still prove useful

in areas such as the reduction of the response burden in surveys and the improvement of

survey results.

4.12.2 Set up an edit and imputation procedure or a weight adjustment procedure to deal with non-

response.

4.12.3 The confidentiality implications of the publication of information from administrative records

must always be borne in mind. Although the Statistics Act, 1993 provides the CSO with the

authority to access administrative records for statistical purposes, the use may not have been

foreseen by the original suppliers of information.

4.12.4 Maintain continuing liaison with the provider of administrative records.

4.12.5 Understand and document concepts, definitions and procedures underlying the collection of

the administrative data.

4.12.6 Implement continuous or periodic assessment of incoming data quality.

4.13 Documentation and Metadata

Documentation refers to the collection of material that provides a description of the activity. It should

include the concepts, definitions, metadata, methodology, and an outline of the production processes

used.

The Business Process Improvement Inquiry (BPI) (see Corporate Discussion\ITSIP\BPI\ Business

Process Improvement Project Report) was a means of documenting, at a high level and in a standard

way, the inputs, methodologies, processes and outputs for every survey in the Office.

Guidelines

4.13.1 Business areas must ensure that a Business Process Improvement (BPI) methodology

questionnaire has been completed and is kept up to date for every survey or statistical process.

4.13.2 The metadata held on the Databank and, in future on the new Data Management System, must

be reviewed and updated whenever a change is made to any part of the statistical activity.

4.14 Revisions to Published Data

Revisions to published data can, and do, occur for many reasons. It is important that all users of

statistics are at all times made fully aware of the revision policy relating to these statistics.

Common sources of revisions to statistics are:

The availability of additional information (e.g. late survey responses; a new period’s data

available for the calculation of seasonal factors)

The receipt of amended information (e.g. as a result of the response to a query)

Changes resulting from additional and more detailed data editing.


Occasionally revisions can arise from a change in methodology when it is deemed best to recompile

previous period’s data using the new methodology.

Guidelines

4.14.1 Produce, for users, a clear and concise data revisions policy for all published statistics.

4.14.2 When publishing data that is likely to be subsequently revised (e.g. preliminary estimates)

always indicate this to the user.

4.14.3 Describe the main reasons why particular statistics are subject to revision.

4.14.4 For established statistical data series inform users of the frequency and size of past revisions.


References 1Statistics Canada, October 1998, Statistics Canada Quality Guidelines, 3

rd edition

2Statistics Finland, 2002, Quality Guidelines for Official Statistics

3Eurostat, 2003, Assessment of quality in statistics, Methodological documents – Definition of Quality in

Statistics, Doc. Eurostat/A4/Quality/03/General/Definition


Appendices

A. European Statistics Code of Practice (2005)

Institutional Environment Institutional and organisational factors have a significant influence on the effectiveness and credibility

of a statistical authority producing and disseminating European Statistics. The relevant issues are

professional independence, mandate for data collection, adequacy of resources, quality commitment,

statistical confidentiality, impartiality and objectivity.

Principle 1: Professional Independence - The professional independence of statistical authorities

from other policy, regulatory or administrative departments and bodies, as well as from private sector

operators, ensures the credibility of European Statistics.

Indicators

– The independence of the statistical authority from political and other external interference

in producing and disseminating official statistics is specified in law.

– The head of the statistical authority has sufficiently high hierarchical standing to ensure

senior level access to policy authorities and administrative public bodies. He/she should be

of the highest professional calibre.

– The head of the statistical authority and, where appropriate, the heads of its statistical

bodies have responsibility for ensuring that European Statistics are produced and

disseminated in an independent manner.

– The head of the statistical authority and, where appropriate, the heads of its statistical

bodies have the sole responsibility for deciding on statistical methods, standards and

procedures, and on the content and timing of statistical releases.

– The statistical work programmes are published and periodic reports describe progress

made.

– Statistical releases are clearly distinguished and issued separately from political/policy

statements.

– The statistical authority, when appropriate, comments publicly on statistical issues,

including criticisms and misuses of official statistics.

Principle 2: Mandate for Data Collection - Statistical authorities must have a clear legal mandate

to collect information for European statistical purposes. Administrations, enterprises and households,

and the public at large may be compelled by law to allow access to or deliver data for European

statistical purposes at the request of statistical authorities.

Indicators

– The mandate to collect information for the production and dissemination of official

statistics is specified in law.

– The statistical authority is allowed by national legislation to use administrative records for

statistical purposes.

– On the basis of a legal act, the statistical authority may compel response to statistical

surveys.

Principle 3: Adequacy of Resources - The resources available to statistical authorities must be

sufficient to meet European statistics requirements.

Indicators


– Staff, financial, and computing resources, adequate both in magnitude and in quality, are

available to meet current European statistics needs.

– The scope, detail and cost of European statistics are commensurate with needs.

– Procedures exist to assess and justify demands for new European statistics against their

cost.

– Procedures exist to assess the continuing need for all European statistics, to see if any can

be discontinued or curtailed to free up resources.

Principle 4: Quality Commitment - All ESS members commit themselves to work and co-operate

according to the principles fixed in the Quality Declaration of the European Statistical System.

Indicators

– Product quality is regularly monitored according to the ESS quality components.

– Processes are in place to monitor the quality of the collection, processing and

dissemination of statistics.

– Processes are in place to deal with quality considerations, including trade-offs within

quality, and to guide planning for existing and emerging surveys.

– Quality guidelines are documented and staff are well trained. These guidelines are spelled

out in writing and made known to the public.

– There is a regular and thorough review of the key statistical outputs using external experts

where appropriate.

Principle 5: Statistical Confidentiality - The privacy of data providers (households, enterprises,

administrations and other respondents), the confidentiality of the information they provide and its use

only for statistical purposes must be absolutely guaranteed.

Indicators

– Statistical confidentiality is guaranteed in law.

– Statistical authority staff sign legal confidentiality commitments on appointment.

– Substantial penalties are prescribed for any wilful breaches of statistical confidentiality.

– Instructions and guidelines are provided on the protection of statistical confidentiality in

the production and dissemination processes. These guidelines are spelled out in writing

and made known to the public.

– Physical and technological provisions are in place to protect the security and integrity of

statistical databases.

– Strict protocols apply to external users accessing statistical microdata for research

purposes.

Principle 6: Impartiality and Objectivity - Statistical authorities must produce and disseminate

European statistics respecting scientific independence and in an objective, professional and

transparent manner in which all users are treated equitably.

Indicators

– Statistics are compiled on an objective basis determined by statistical considerations.

– Choices of sources and statistical techniques are informed by statistical considerations.

– Errors discovered in published statistics are corrected at the earliest possible date and

publicised.

– Information on the methods and procedures used by the statistical authority are publicly

available.

– Statistical release dates and times are pre-announced.


– All users have equal access to statistical releases at the same time and any privileged pre-

release access to any outside user is limited, controlled and publicised. In the event that

leaks occur, pre-release arrangements should be revised so as to ensure impartiality.

– Statistical releases and statements made in Press Conferences are objective and non-

partisan.

Statistical Processes European and other international standards, guidelines and good practices must be fully observed in

the processes used by the statistical authorities to organise, collect, process and disseminate official

statistics. The credibility of the statistics is enhanced by a reputation for good management and

efficiency. The relevant aspects are sound methodology, appropriate statistical procedures, non-

excessive burden on respondents and cost effectiveness.

Principle 7: Sound Methodology - Sound methodology must underpin quality statistics. This

requires adequate tools, procedures and expertise.

Indicators

– The overall methodological framework of the statistical authority follows European and

other international standards, guidelines, and good practices.

– Procedures are in place to ensure that standard concepts, definitions and classifications are

consistently applied throughout the statistical authority.

– The business register and the frame for population surveys are regularly evaluated and

adjusted if necessary in order to ensure high quality.

– Detailed concordance exists between national classifications and sectorisation systems and

the corresponding European systems.

– Graduates in the relevant academic disciplines are recruited.

– Staff attend international relevant training courses and conferences, and liaise with

statistician colleagues at international level in order to learn from the best and to improve

their expertise.

– Co-operation with the scientific community to improve methodology is organised and

external reviews assess the quality and effectiveness of the methods implemented and

promote better tools, when feasible.

Principle 8: Appropriate Statistical Procedures – Appropriate statistical procedures, implemented

from data collection to data validation, must underpin quality statistics.

Indicators

– Where European statistics are based on administrative data, the definitions and concepts

used for the administrative purpose must be a good approximation to those required for

statistical purposes.

– In case of statistical surveys, questionnaires are systematically tested prior to the data

collection.

– Survey designs, sample selections, and sample weights are well based and regularly

reviewed, revised or updated as required.

– Field operations, data entry, and coding are routinely monitored and revised as required.

– Appropriate editing and imputation computer systems are used and regularly reviewed,

revised or updated as required.

– Revisions follow standard, well-established and transparent procedures.


Principle 9: Non-Excessive Burden on Respondents - The reporting burden should be

proportionate to the needs of the users and should not be excessive for respondents. The statistical

authority monitors the response burden and sets targets for its reduction over time.

Indicators

– The range and detail of European statistics demands is limited to what is absolutely

necessary.

– The reporting burden is spread as widely as possible over survey populations through

appropriate sampling techniques.

– The information sought from businesses is, as far as possible, readily available from their

accounts and electronic means are used where possible to facilitate its return.

– Best estimates and approximations are accepted when exact details are not readily

available.

– Administrative sources are used whenever possible to avoid duplicating requests for

information.

– Data sharing within statistical authorities is generalised in order to avoid multiplication of

surveys.

Principle 10: Cost Effectiveness - Resources must be effectively used.

Indicators

– Internal and independent external measures monitor the statistical authority’s use of

resources.

– Routine clerical operations (e.g. data capture, coding, validation) are automated to the

extent possible.

– The productivity potential of information and communications technology is being

optimised for data collection, processing and dissemination.

– Proactive efforts are being made to improve the statistical potential of administrative

records and avoid costly direct surveys.

Statistical Output Available statistics must meet users’ needs. Statistics comply with the European quality standards and

serve the needs of European institutions, governments, research institutions, business concerns and the

public generally. The important issues concern the extent to which the statistics are relevant, accurate

and reliable, timely, coherent, comparable across regions and countries, and readily accessible by

users.

Principle 11: Relevance - European statistics must meet the needs of users.

Indicators

– Processes are in place to consult users, monitor the relevance and practical utility of

existing statistics in meeting their needs, and advise on their emerging needs and priorities.

– Priority needs are being met and reflected in the work programme.

– User satisfaction surveys are undertaken periodically.

Principle 12: Accuracy and Reliability - European statistics must accurately and reliably portray

reality.

Indicators

– Source data, intermediate results and statistical outputs are assessed and validated.

– Sampling errors and non-sampling errors are measured and systematically documented

according to the framework of the ESS quality components.


– Studies and analyses of revisions are carried out routinely and used internally to inform

statistical processes.

Principle 13: Timeliness and Punctuality - European statistics must be disseminated in a timely and

punctual manner.

Indicators

– Timeliness meets the highest European and international dissemination standards.

– A standard daily time is set for the release of European statistics.

– Periodicity of European statistics takes into account user requirements as much as

possible.

– Any divergence from the dissemination time schedule is publicised in advance, explained

and a new release date set.

– Preliminary results of acceptable aggregate quality can be disseminated when considered

useful.

Principle 14: Coherence and Comparability - European statistics should be consistent internally,

over time and comparable between regions and countries; it should be possible to combine and make

joint use of related data from different sources.

Indicators

– Statistics are internally coherent and consistent (e.g. arithmetic and accounting identities

observed).

– Statistics are coherent or reconcilable over a reasonable period of time.

– Statistics are compiled on the basis of common standards with respect to scope,

definitions, units and classifications in the different surveys and sources.

– Statistics from the different surveys and sources are compared and reconciled.

– Cross-national comparability of the data is ensured through periodical exchanges between

the European Statistical System and other statistical systems; methodological studies are

carried out in close co-operation between the Member States and Eurostat.

Principle 15: Accessibility and Clarity – European statistics should be presented in a clear and

understandable form, disseminated in a suitable and convenient manner, available and accessible on

an impartial basis with supporting metadata and guidance.

Indicators

– Statistics are presented in a form that facilitates proper interpretation and meaningful

comparisons.

– Dissemination services use modern information and communication technology and, if

appropriate, traditional hard copy.

– Custom-designed analyses are provided when feasible and are made public.

– Access to microdata can be allowed for research purposes. This access is subject to strict

protocols.

– Metadata are documented according to standardised metadata systems.

– Users are kept informed on the methodology of statistical processes and the quality of

statistical outputs with respect to the ESS quality criteria.


B. UN Fundamental Principles of Official Statistics

Adopted by The United Nations Statistical Commission, in its Special Session of 11-15 April 1994.

Principle 1. Official statistics provide an indispensable element in the information system of a

society, serving the government, the economy and the public with data about the economic,

demographic, social and environmental situation. To this end, official statistics that meet the test of

practical utility are to be compiled and made available on an impartial basis by official statistical

agencies to honour citizens’ entitlement to public information.

Principle 2. To retain trust in official statistics, the statistical agencies need to decide according to

strictly professional considerations, including scientific principles and professional ethics, on the

methods and procedures for the collection, processing, storage and presentation of statistical data.

Principle 3. To facilitate a correct interpretation of the data, the statistical agencies are to present

information according to scientific standards on the sources, methods and procedures of the statistics.

Principle 4. The statistical agencies are entitled to comment on erroneous interpretation and misuse

of statistics.

Principle 5. Data for statistical purposes may be drawn from all types of sources, be they statistical

surveys or administrative records. Statistical agencies are to choose the source with regard to quality,

timeliness, costs and the burden on respondents.

Principle 6. Individual data collected by statistical agencies for statistical compilation, whether they

refer to natural or legal persons, are to be strictly confidential and used exclusively for statistical

purposes.

Principle 7. The laws, regulations and measures under which the statistical systems operate are to be

made public.

Principle 8. Coordination among statistical agencies within countries is essential to achieve

consistency and efficiency in the statistical system.

Principle 9. The use by statistical agencies in each country of international concepts, classifications

and methods promotes the consistency and efficiency of statistical systems at all official levels.

Principle 10. Bilateral and multilateral co-operation in statistics contributes to the improvement of

systems of official statistics in all countries.


C. Recommendations of LEG on Quality

Recommendation no. 1: Each NSI should report product quality according to the ESS quality

dimensions and sub-dimensions.

Recommendation no. 2: The measurability of each ESS quality dimension and sub-dimension should

be improved.

Recommendation no. 3: Process measurements are vital for all improvement work. A handbook on

the identification of key process variables, their measurement, and measurement analysis should be

developed.

Recommendation no. 4: All organisations in the ESS should adopt a systematic approach to quality

improvement. ESS members should use the EFQM excellence model as a basis for their improvement

work except for those already using a similar model.

Recommendation no.5: NSIs should strive to improve their relationships with data suppliers, and

research should be conducted on how data suppliers perceive their task. A special emphasis should be

placed on issues that involve a decrease of the respondent burden and enhance suppliers’ awareness of

the role of statistics in society.

Recommendation no. 6: ESS members should develop service level agreements for their main

programmes.

Recommendation no. 7: A development project regarding the design, implementation and analysis of

customer satisfaction surveys should be initiated.

Recommendation no. 8: Each ESS member should provide a report regarding the present status of its

user – producer dialogue including descriptions of any user involvement in the planning process.

Good practices in promoting user awareness of quality problems should be collected and made

available to ESS members.

Recommendation no. 9: An in-depth analysis of the most important ESS strengths and weaknesses

should be conducted. An action programme should be developed based on the findings of this

analysis.

Recommendation no. 10: NSIs should develop CBMs for their most common processes. A handbook

for developing CBMs covering construction, dissemination, implementation and revision of CBMs

should be developed. Existing and relevant CBMs should be collected and distributed in the ESS.

Recommendation no. 11: A set of recommended practices for statistics production should be

developed. The work should start by developing recommended practices for a few areas followed by a

test of their feasibility in the ESS.

Recommendation no. 12: ESS members should use the list of current good information management

and dissemination practices compiled by the LEG and consider actions for internal use.

Recommendation no. 13: The user needs of the current ESS information system should be reviewed

and Eurostat’s current database expanded accordingly. Guidelines regarding the future management

of the information system should be developed.

Recommendation no. 14: A biennial conference covering any methodological and quality-related

topics of relevance to the ESS should be organised.

Recommendation no. 15: A generic checklist should be developed for a simple self-assessment

programme for survey managers in the ESS.

Recommendation no. 16: The methods for auditing on different levels and for different purposes

such as internal, external, one point in time, continuing or rolling, rapid, and more extensive (such as

EFQM assessment) should be reviewed and recommendations should be provided to the ESS.


Recommendation no. 17: ESS members should study staff perception. One way to do this is to

conduct staff perception surveys.

Recommendation no. 18: ESS members should analyse their documentation status in a report. The

report should include an action plan with clear priorities for improvement and a timetable.

Recommendation no. 19: Each ESS member should make publicly available documents describing

its mission statement, dissemination policy and quality policy.

Recommendation no. 20: All staff should be trained in quality work with different types of training

programmes for different types of staff. Each ESS member should develop a training programme.

Training on a European level should be enhanced.

Recommendation no. 21: A biennial quality award in official statistics should be established. The

award could be given to a single improvement project team, for an innovative idea, to a well-

performing ESS organisation or to a statistical programme team.

Recommendation no. 22: There is a need to establish a LEG Implementation Group that coordinates

the activities generated by recommendations approved by the SPC.