ESS-net DWH

ESSnet on microdata linking and data warehousing

in statistical production

Harry Goossens – Statistics NetherlandsHead Data Service Centre / ESSnet Coordinatorhct.goossens@cbs.nl

ESS-net DWH 2

MEETS action 3.1:

“make better use of data that already exist in the statistical system, including the possibility of estimates”

Ultimate aim:

“create fully integrated data sets for enterprise and trade statistics at micro level”

a 'data warehouse' approach to statistics

General context

ESS-net DWH 3

ESS-net coordinator:

Statistics Netherlands (CBS)

Co-partners:

Estonia, Italy, Lituania, Portugal, Sweden, UK\

Starting date:

4 October 2010

SGA 1: first year, till 3 October 2011

SGA 2: last 2 years, till 3 October 2013

Partnership (FPA)

ESS-net DWH 4

Broad definition for this ESSnet:

A common conceptual model for all available data of interest, enabling the NSI: to (re)use this data to create new data/new outputs, to produce the necessary statistical information andto perform reporting and analysis, regardless of the data’s source.

DWH Definition

ESS-net DWH 5

General objective ESS-net:

Provide assistance in the development and implementation of a maximum efficient statistical process for business and trade statistics, independent of any (technical) specific architecture

Results in daily statistical practice: increase the efficiency of dataprocessing

in statistical production systems maximize the reuse of already collected data

Objectives

ESS-net DWH 6

Review of current practices in integrated business data systems

- realised solutions, ongoing developments- problems, desired solutions, opportunities

Examination of ways in which data can be combinedto support new outputs

- Role of Business Register, metadata

Provision of recommendations on how the ESScan improve data warehousing

Dissemination of results to all ESS countries (and broader)

Main goals (at start)

ESS-net DWH 7

1. Stocktaking / Inventory of current situation in MS- several MS already introduced data warehousing;- important to make it possible to compile and compare

current practices.

2. Analysis and Development- development of a functional model of the S-DWH;- model as basis to assess all issues, raised within this

ESSnet, in a generic and standardized way.

3. Dissemination- sharing knowledge and expertise.

Actions around 3 axes

ESS-net DWH 8

Activities mainly aroundaxis 1 (stock taking)

What is already been done, experienced by MS ?

DWH is a very broad field and scope: First strictly determine specific detailed

subjects/topics to explore and study in depth.

and axis 3 (disssemination)

Knowing and sharing what is going on

Finding out what the ESS needs / wants

First SGA

ESS-net DWH 9

The Statistical Data Warehouse

AggregateStatistics

AggregateStatistics

Microdata

Dataextracts

Dataextracts

Dataextracts

Dataset

Dataset

Dataset

Businessregister

Selectedsamples

Admin datasource

BRsnapshots

Storage, combination OutputsInput / Micro dataInput reference frameS

tagi

ng a

rea

Wor

king

dat

a

Rules for generating samples etc

Rules for updating BR

ESS-net DWH 10

Conclusions Data Warehousing in statistics is ‘hot’

Metadata is found important…..but also often neglected !

S-DWH is very difficult to compare with common commercial DWH

Visiting NSIs has proven very effective for gathering information AND for sharing knowledge and expertise

Great need for knowledge & expertise

First SGA

ESS-net DWH 11

24 months (till 10/2013 – end of MEETS)

3 Workshops planned

4 WP (+ management & coordination)

Interconnecting with other ESSnets

Keep overviews BP up-to-date

Continuation of visiting NSIs to support actively

Strong focus on dissemination

Second SGA

ESS-net DWH 12

Building a framework which defines metadata requirements and roles in the S-DWH context

Study on the use of metadata models and standards:define the various functionalities of a metadata system to facilitate and support the operation of the S-DWH

Provide recommendations and guidelines on the governance of metadata management in the S-DWH

WP 1 - Metadata

ESS-net DWH 13

WP 2 - Methodological topics

Building the S-DWH: Methodological topics

Study and recommendations on the position and role of the Business Register (BR) in the context of a S-DWH

Examine selective editing options for the S-DWH and provide guidelines

Study and recommendations on the various ways of ‘linking data’, considering aspects of combining data (from surveys and administrative sources etc.)

Study on the impact of data confidentiality in micro data linking and data warehousing, considering confidentiality aspects of combining data from various sources etc.

ESS-net DWH 14

WP 3 - Technical aspects

Building the S-DWH: Technical aspects

Provide a Business Architecture of the S-DWH Produce a functional architecture of the S-DWH Produce a modular workflow of the S-DWH Relate the 'ideal' architectural schemes (frameworks) into

an actual implementation and development strategy

ESS-net DWH 15

WP 4 - Dissemination

Keep the ‘stocktaking of best practices, problems encountered and desired solutions’ up-to-date

Create a glossary and keep it up-to-date Create an overall report, containing all deliverables of this

ESSnet (guidelines, recommendations, models etc.) Create an overall handbook to set up the S-DWH Establishing and implementing a ‘knowledge and expertise

centre’ to manage the expertise and knowledge acquired in an active/dynamic manner

ensure sustainability after this ESSnet is completed

ESS-net DWH 16

Specific attention:

overlap, compatibility and interfacing with other ESSnets

- sharing and using results

scoping, prioritising

- keep the ESSnet manageable

Project management