ESS-net DWH
ESSnet on microdata linking and data warehousing
in statistical production
Harry Goossens – Statistics NetherlandsHead Data Service Centre / ESSnet [email protected]
ESS-net DWH 2
MEETS action 3.1:
“make better use of data that already exist in the statistical system, including the possibility of estimates”
Ultimate aim:
“create fully integrated data sets for enterprise and trade statistics at micro level”
a 'data warehouse' approach to statistics
General context
ESS-net DWH 3
ESS-net coordinator:
Statistics Netherlands (CBS)
Co-partners:
Estonia, Italy, Lituania, Portugal, Sweden, UK\
Starting date:
4 October 2010
SGA 1: first year, till 3 October 2011
SGA 2: last 2 years, till 3 October 2013
Partnership (FPA)
ESS-net DWH 4
Broad definition for this ESSnet:
A common conceptual model for all available data of interest, enabling the NSI: to (re)use this data to create new data/new outputs, to produce the necessary statistical information andto perform reporting and analysis, regardless of the data’s source.
DWH Definition
ESS-net DWH 5
General objective ESS-net:
Provide assistance in the development and implementation of a maximum efficient statistical process for business and trade statistics, independent of any (technical) specific architecture
Results in daily statistical practice: increase the efficiency of dataprocessing
in statistical production systems maximize the reuse of already collected data
Objectives
ESS-net DWH 6
Review of current practices in integrated business data systems
- realised solutions, ongoing developments- problems, desired solutions, opportunities
Examination of ways in which data can be combinedto support new outputs
- Role of Business Register, metadata
Provision of recommendations on how the ESScan improve data warehousing
Dissemination of results to all ESS countries (and broader)
Main goals (at start)
ESS-net DWH 7
1. Stocktaking / Inventory of current situation in MS- several MS already introduced data warehousing;- important to make it possible to compile and compare
current practices.
2. Analysis and Development- development of a functional model of the S-DWH;- model as basis to assess all issues, raised within this
ESSnet, in a generic and standardized way.
3. Dissemination- sharing knowledge and expertise.
Actions around 3 axes
ESS-net DWH 8
Activities mainly aroundaxis 1 (stock taking)
What is already been done, experienced by MS ?
DWH is a very broad field and scope: First strictly determine specific detailed
subjects/topics to explore and study in depth.
and axis 3 (disssemination)
Knowing and sharing what is going on
Finding out what the ESS needs / wants
First SGA
ESS-net DWH 9
The Statistical Data Warehouse
AggregateStatistics
AggregateStatistics
Microdata
Dataextracts
Dataextracts
Dataextracts
Dataset
Dataset
Dataset
Businessregister
Selectedsamples
Admin datasource
BRsnapshots
Storage, combination OutputsInput / Micro dataInput reference frameS
tagi
ng a
rea
Wor
king
dat
a
Rules for generating samples etc
Rules for updating BR
ESS-net DWH 10
Conclusions Data Warehousing in statistics is ‘hot’
Metadata is found important…..but also often neglected !
S-DWH is very difficult to compare with common commercial DWH
Visiting NSIs has proven very effective for gathering information AND for sharing knowledge and expertise
Great need for knowledge & expertise
First SGA
ESS-net DWH 11
24 months (till 10/2013 – end of MEETS)
3 Workshops planned
4 WP (+ management & coordination)
Interconnecting with other ESSnets
Keep overviews BP up-to-date
Continuation of visiting NSIs to support actively
Strong focus on dissemination
Second SGA
ESS-net DWH 12
Building a framework which defines metadata requirements and roles in the S-DWH context
Study on the use of metadata models and standards:define the various functionalities of a metadata system to facilitate and support the operation of the S-DWH
Provide recommendations and guidelines on the governance of metadata management in the S-DWH
WP 1 - Metadata
ESS-net DWH 13
WP 2 - Methodological topics
Building the S-DWH: Methodological topics
Study and recommendations on the position and role of the Business Register (BR) in the context of a S-DWH
Examine selective editing options for the S-DWH and provide guidelines
Study and recommendations on the various ways of ‘linking data’, considering aspects of combining data (from surveys and administrative sources etc.)
Study on the impact of data confidentiality in micro data linking and data warehousing, considering confidentiality aspects of combining data from various sources etc.
ESS-net DWH 14
WP 3 - Technical aspects
Building the S-DWH: Technical aspects
Provide a Business Architecture of the S-DWH Produce a functional architecture of the S-DWH Produce a modular workflow of the S-DWH Relate the 'ideal' architectural schemes (frameworks) into
an actual implementation and development strategy
ESS-net DWH 15
WP 4 - Dissemination
Keep the ‘stocktaking of best practices, problems encountered and desired solutions’ up-to-date
Create a glossary and keep it up-to-date Create an overall report, containing all deliverables of this
ESSnet (guidelines, recommendations, models etc.) Create an overall handbook to set up the S-DWH Establishing and implementing a ‘knowledge and expertise
centre’ to manage the expertise and knowledge acquired in an active/dynamic manner
ensure sustainability after this ESSnet is completed
ESS-net DWH 16
Specific attention:
overlap, compatibility and interfacing with other ESSnets
- sharing and using results
scoping, prioritising
- keep the ESSnet manageable
Project management