data collection and processing, data quality management seminar on global assessment yalta, 23 - 25...
TRANSCRIPT
Data collection and Data collection and processing, data qualityprocessing, data quality
Management seminar on global assessment
Yalta, 23 - 25 September 2009Session 10
1
Heinrich Brüngger
10
Part I: Data collection and processing
Part II: Confidentiality (with emphasis on data collection and processing)
Part III: Data quality (quality management)
2
Nature of decisions on Nature of decisions on methods of data collection methods of data collection and processingand processingUnder professional independence (FP
2), therefore within the statistical system without outside interference once the decision about the “what” to measure in broad terms have been made
Decisions about what official statistics should measure may involve the political level, and have to involve discussions with a broad range of users
3
Professional Professional independenceindependence
The “how” part of statistical activities has to focus exclusively at obtaining the most reliable representation of the phenomenon of the real world with the given resources
Scientific methods, international standards, and empirically established good practices are the best guide for professional independence
The statistical law may require the adoption by government in the case of statistical surveys because of explicit legitimacy in the interaction with respondents and the burden caused to respondents, but governments should not interfere in methodological issues
4
Main data sources (FP 5)Main data sources (FP 5)Official statistics has to have the
right to use three types of sources:◦Statistical surveys: data collection
from individuals, households, corporate and unicorporated businesses, and public entities outside the own government structure, for the exclusive purpose of statistics (most of which official statistics)
◦Administrative records: secondary use of data collected primarily for administrative purposes about individuals or private businesses by other parts of the same government structure
5
Main data sources (ctd.)Main data sources (ctd.)
◦Secondary use of environment and territorial observation and monitoring data collected by specialised government agencies
For statistical surveys, the statistical law applies to all phases of the statistical activity; for the other two sources, primary data collection has to be based on other laws, and the statistical legislation becomes only applicable once data are handed over to a producer of official statistics
6
Statistical programmesStatistical programmesIf the law requires input-oriented
programmes, a clear distinction has to be made between statistical surveys, where the statistical law and the statistical programmes are the legal basis for data collection, and the second and third data source, where the primary data collection is for another purpose and has therefore to have a different legal basis
7
Minor data sources for Minor data sources for national official statisticsnational official statisticsOfficial statistics from a partner
country or from an international organisation (especially for international flows)
Registers in the hand of specialised private companies (as one of the sources used to keep statistical registers updated)
8
Statistical surveysStatistical surveys
Exhaustive vs. sample surveys (mixed forms possible)
Mail, face to face interview, telephone interviews, internet (mixed forms possible)
Compulsory participation of the respondent vs. right to decline participation (mixed forms possible)
Recall and follow-up policy for non-response, depending on type of survey
9
Response burdenResponse burdenResponse burden has to be
assessed in advance and taken into account:◦all new surveys have to be tested first
(pilots)◦all respondents have to be informed
about the purpose and legal basis of the survey, and especially about the confidentiality measures
◦forms and questions have to be understandable, non-intrusive, and make answers possible for him/her from memory or from existing material (businesses) 10
Response burden (ctd.)Response burden (ctd.)◦first reminders have to be
proportionate ◦response rates have to be closely
monitored
The NSI has also to act as custodian for the respondents; users will tend to minimise the issue of response burden
11
Surveys (ctd.)Surveys (ctd.)Information given by a respondent in
statistical surveys have no legal implications for him; they can (and should) therefore be checked and when necessary altered (edited) by the NSI
In many cases, the characteristics of interest to users cannot be taken over directly into questionnaires, but have to be compiled as derived characteristics using several questions
12
Non-response (valid for all Non-response (valid for all data sources)data sources)The NSI has to develop a policy for
treating general and item non-response, which has to be subsequently fine-tuned and documented for each survey
Imputations for missing (or unusable) information is legitimate, but a careful balance has to be found between “not enough” and “too much” (risk of introducing biases)
13
Administrative sourcesAdministrative sourcesThe statistical office must have the
legal and de facto right to receive, for its tasks, regularly and on an ad hoc basis, microdata sets from other ministries and public entities taken from their administrative sources
This does not mean that direct identifiers have to be included in all cases, but the possibility should not be entirely excluded
14
Administrative sources Administrative sources (ctd.)(ctd.)Statistical producers have the right to
alter (edit an impute) the administrative data received from other ministries to improve compatibility with statistical definitions and classifications
Data received in this way should never be given back to the data owner or transferred by the statistical producer to a third party for administrative purposes
15
Relationship between data Relationship between data sourcessourcesBefore considering introducing a new
survey, or expanding an existing one, for new information needs, other possible data sources (especially administrative) have to be examined for their potential
The possibility of combining questions about different areas in the same survey is limited: the survey system has to be structured carefully
16
Relationship between data Relationship between data sources (ctd.)sources (ctd.)Administrative sources are in constant
change, and some risk of disappearing completely
The final decision-making about administrative sources is outside the statistical system. Therefore, the NSI has the responsibility to constantly monitor (if possible anticipate) and react to planned changes that may affect the use of administrative sources for official statistics positively or negatively
17
Relationship between data Relationship between data sources (ctd.)sources (ctd.)
NSIs have to avoid considering data sources, and especially surveys, as stove-pipe type of parallel operations, but rather as a system of interrelated operations with n:m relationship between data sources and results
Very often, the best estimates are based on a judicious combination of sources, combining the individual strengths and reducing the individual weaknesses
Statistical registers of good quality are a cornerstone for a good survey system, and for matching different sources
18
Relationship between data Relationship between data sources (ctd.)sources (ctd.)When results on the same issue are
published from different sources separately (i.e. without prior integration), an accounting relationship between the results should be established, with the components quantified if possible
Otherwise, only the most complete aggregate should be released as results of official statistics in absolute terms
19
Confidentiality (FP 6)Confidentiality (FP 6)
Statistical confidentiality aims at protecting the privacy of individual units about which data are collected and processed. It has two components:◦Producers of official statistics do not disclose, either directly or indirectly, characteristics about protected units to any third party in such a way that any user might derive additional information about a protected unit 20
FP 6 (ctd.)FP 6 (ctd.)Producers of official statistics use data about
protected individual units only for statistical purposes (official statistics in the first place)
The statistical system, and in the first instance the NSI, have to make sure in an effective a credible way that the two elements of the confidentiality principle are strictly followed in all parts of the production and dissemination processes, and that confidential data are stored in a safe way
21
Confidentiality in data Confidentiality in data collectioncollection
◦Legal situation in countries may differ with respect to: Protected units: physical persons, private
households, private businesses (whether legal persons or unincorporated enterprises) should be fully protected; government units as institutions cannot invoke statistical confidentiality; public enterprises in a competitive market are a borderline case
All characteristics treated alike, or some considered to be free (e.g. for legal persons), or particularly sensitive (higher degree of protection, e.g. about health, crime, ethnicity)
22
Confidentiality in data Confidentiality in data collection (ctd.)collection (ctd.)
◦Data collection in statistical surveys: replies of respondents should go back as directly as possible to the statistical producer. No other government agency should be involved as intermediary (regional level!)
◦Data processing: except in the context of statistical registers, identifiers of units should be separated from the context variables by the statistical producer at an as early stage as possible, and either stored separately or destroyed
23
Confidentiality in data Confidentiality in data collection (ctd.)collection (ctd.)
◦Data security: final (edited) microdata sets have to be stored safely, and access limited to those within the NSO who use the data regularly. No direct access from outside the NSO should be possible. For files with identifiers (other than statistical registers), each access has to be documented.
◦Data security, update and access regulations for statistical registers have to be worked out carefully in a separate book of rules.
24
Quality management: Quality management: transparency as prerequisitetransparency as prerequisiteAll decision on methods
(collection/processing/dissemination) have to be documented and made accessible for the public (FP 3)
Quality management is addressed by several principles of the EU Code of Practice
Quality management refers to products, processes and staff
25
Quality management: Quality management: productsproductsProducts to be regularly assessed
with pre-defined quality criteria (and rejected if minimal quality level for fitness of use not met):◦Final micro-data sets from a data
collection instance◦Results of official statistics prior to
releaseProducts to be assessed occasionally
◦Questionnaires for surveys◦Dissemination products
26
Products to be releasedProducts to be releasedAn important element of
professional independence is the right of the NSI not to release official results if pre-defined minimal quality criteria are not met; this has to be a decision by the chief statistician
This does not preclude the use and release of such results as one input among others into a broader (e.g. geographical) aggregate
27
Quality management: Quality management: processesprocessesProcesses should be defined in
terms of who decides on what and when within the statistical system, and who has to be consulted and informed; they have to be documented for mainly internal use
Processes should be defined for each phase in a generic statistical business model (from design to archiving)
28
Processes (ctd.)Processes (ctd.)It is important to define the system of
processes and responsibilities in such a way that the tendency for creating “black boxes” by responsible organisational units is offset. Stove-pipes (same organisational unit responsible for all phases of the production process for a given activity such as survey) are problematic also from this point of view
29
Processes (ctd.)Processes (ctd.)The process definitions will be
different between on-going activities (with room for small improvements), and major redesigns/innovations, and between production activities ending in the release of results of official statistics and infrastructure activities such as dissemination or training
30
Cross-cutting evaluationsCross-cutting evaluationsAn overall evaluation of the portfolio
of activities and the extent to which the needs of users are met is necessary when the preparation for the next multi-annual programme starts
Extensive user consultations are necessary, and external expertise (such as in a GA) may help in identifying and prioritising gaps to be addressed in the next programme
31
Tendency in organisational Tendency in organisational structures of NSIstructures of NSIFirst-level division by broad subject
areas used to be predominantRecent shift to functional break-
down at first level (data collection; processing; dissemination) for various reasons, among others improved quality management
Senior management has to provide incentives for quality management leading to measurable improvements
32
Staff managementStaff managementQuality management has to become
an almost automatic part of the activity for each staff member
Regular training on management issues should include a substantial part on quality management
Regular assessment of the individual performance of each staff member can be very instrumental, if not implemented as a bureaucratic routine, but as open dialogue towards improvements
33
Thank you for your attention
34