principles, criteria and methods part 2

Copyright 2010, The World Bank Group. All Rights Reserved.

Principles, criteria and methods

Part 2

Quality management

Produced in Collaboration between World Bank Institute and the Development Data Group (DECDG)


15 principles:

• PROFESSIONAL INDEPENDENCE

• MANDATE FOR DATA COLLECTION

• ADEQUACY OF RESOURCES

• QUALITY COMMITMENT

• STATISTICAL CONFIDENTIALITY

• IMPARTIALITY AND OBJECTIVITY

• SOUND METHODOLOGY

Continued:

• APPROPRIATE STATISTICAL PROCEDURES

• NON-EXCESSIVE BURDEN ON RESPONDENTS

• COST EFFECTIVENESS• RELEVANCE• ACCURACY AND RELIABILITY• TIMELINESS AND PUNCTUALITY• COHERENCE AND

COMPARABILITY• ACCESSIBILITY AND CLARITY

European Statistics Code of Practice

2


ESCP questionnaire

• Provides a tool for self-assessment of quality aspects

• Designed in a way that most parts can be filled in centrally, e.g. by the quality manager of the organization

• Follows the structure of the European Statistics Code of Practice - subdivided into 15 principles and 3-7 indicators for each principle

• For each principle the questionnaire concludes with a follow-up part in which statistical authorities are requested to reflect upon improvement actions and a time frame

3


Quality management in practice

• Quality management in the practice of statistical offices has two main aspects:

• Assessing the quality of statistical outputs – users judge the quality of outputs on aspects relating to their use of the statistics. To meet user quality requirements, a framework for assessing the quality of outputs should be used

• Managing quality in the ongoing production of statistics - a standard set of indicators for different parts of the production process should be used

4


Assessing the quality of outputs

• The key assessment aspects for the users are:

Relevance Accuracy Coherence Interpretability Timeliness

• Accessibility of data and metadata

• To measure output quality for each of those, indicators can be used

5


Output quality indicators – accuracy/1

• Quality measures (e.g. sampling errors) and indicators (e.g. non-response rates) regularly produced and monitored for source data

• Definitions of data consistent with both user and provider understanding

• A revisions policy which balances the need to inform users of improved estimates with possible confusion from insignificant changes

6


Output quality indicators – accuracy/2

• Insignificant and consistent (size and direction) differences between preliminary and final estimates

• Data rebased regularly

• Data source samples redesigned or reselected regularly to maintain sample errors within quality standards

7


Output quality indicators – coherence/1

• All source data measured in accordance with standards (frame, statistical units, definitions, classifications, processes)

• Data presented in a framework, along with other relevant data

• Long term time series are available, with explanations or adjustments for breaks in the series

• Input data from different sources are confronted and reconciled

• Output data is consistent or reconcilable with other sources

8


Output quality indicators – coherence/2

• Concordances available to allow data to be related to previous classification versions or related classifications

• Consistency between aggregates and components

• Key classifications are maintained so that comparability over time is maintained

9


Output quality indicators – interpretability/1

• Seasonal and trend analysis and other adjustment techniques are undertaken to enhance the usefulness of the data and reduce problems of interpretation and comparisons

• Preliminary or early estimates are clearly indicated as such and information provided on expected level of revisions

• Major changes to the data from revisions, rebasing, etc. are published separately from, and where appropriate ahead of, the release of new information

10


Output quality indicators – interpretability/2

• All revisions are clearly marked and explained

• Presentation standards used for tables and graphs

• Information on methods, concepts, data sources, etc. Is up-to-date and readily available to users

• Information on quality achieved is readily available

11


Output quality indicators - accessibility

• Main findings are made widely available (e.g. through press releases, website, public libraries)

• Information on data availability (published and unpublished) is readily known

• Catalogues/directories are available for all fields of statistics• Information is available in formats and media required by users• First release of key data is made available in all media at the same

time • Release dates are announced in advance• Information is affordable• New information and communication technology is being used to

improve the presentation and accessibility of data

12


Managing quality of processes

• The following is necessary:

• Quality standards specified and known to all process managers and staff

• Processes and rules documented and the information readily accessible

• Owners of the outcomes of each stage identified along with their responsibilities

• Outputs at each stage defined along with standards and tolerances

• Production and monitoring of indicators to ensure the standards are met and users can be informed of the quality achieved

• A system for registering process problems and managing action taken

• A system for change management13


Monitoring the quality of surveys/1

• Examples of indicators to be monitored are:

• Sample and non-sample errors• Response rates • Proportion of proxy interviews• Impact of births and deaths in business surveys• Impact of rotation

14


Monitoring the quality of surveys/2

• Outliers and imputation

• Population change

• Respondent load

15


Monitoring respondent management/1

• Key respondent management issues are:

• Forms in source data collections tested with respondents

• Good, up-to-date understanding of respondent information sources

• Respondents able to provide requested information via preferred method

• Help/support system available

• Measures of load produced regularly

16


Monitoring respondent management/2

• Administrative data used wherever possible, for small firms tax data is normally a useful source.

• Sound security of information

• Confidentiality checking of releases

17


Quality support systems

• Quality management is impossible without adequate support systems

• Key tools are:

• Expert service units for standards and techniques for reducing error (e.g. sampling methodologists, questionnaire designers, time series analysts)

• Project management methodology to manage collections • Protocols with associated standards and guidelines for graphs, tables,

time series presentation, revisions, releasing data with error, form design, etc

• Standard frames, frameworks, definitions, questions, classifications, code files

• Easy access to documentation• Peer reviews of the design of various aspects of a collection• Registers for logging problems and tracking their resolution

18


Quality management and leadership

• High level management support and leadership is needed

• Important principles are:

Output managers actively taking responsibility for the quality for their products

Information on quality regularly collected and used Customer orientation Documentation is encouraged Good practices sought out and promoted Extension of solutions/innovations to other processes Regular reviews of performance of systems Staff development and support

19


Managing staff quality/1

• The key to success in quality management is staff with the right skills

• Important staff skills in this respect are:

It starts with a good understanding of the topic they report about. That is subject matter knowledge. Good understanding of key users and emerging new

stakeholders Good understanding of public policy directions and issues Ensuring that information is produced at the right frequency to

allow timely monitoring of changes. Knowledge of the relevant processes and the methods they

need to apply.

20


Managing staff quality/2

• Ensuring that information provides a useful estimate of what key users want to measure

• Ensuring that questionnaires, definitions and classifications reflect contemporary needs and situations

21

principles, criteria and methods part 2

Documents