eurostat unit b3 – it and standards for data and metadata exchange sdmx basics training – 2012...
TRANSCRIPT
Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
IT architectures for data exchangeSDMX-RI and the Hub approach
Nadezhda VlahovaMarco Pellegrino
2Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
Data Repository (Warehousing) Architecture
NSI
EurostatPull Requestor
eDAMIS
Data Input
SDMX Registry
Intermediatestorage
Verification /ConversionTo SDMX
Receiveddata in
SDMX-MLLoader
register
Warehousestorage
Eurobase
query
Dissemination
XSL forSDMX-ML
PULL
PUSH
3Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
4Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
The European Census Hub: key issues
Dissemination of the data from the 2011 population and housing censuses in the European Union
Data that are methodologically comparable and structured according to “hypercubes” agreed with Member States (Census Regulation)
Providing users with an easy access to detailed census data (advanced functionalities)
Management of massive amounts of data produced and controlled by Member States
High accessibility to data and metadata
Harmonised concepts and definitions
Maximum flexibility to cross-tabulate data from different sources
Easy to use
4
5Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
EU Census: Implementing measures
Regulation (EC) 763/2008 on population and housing censuses authorises the European Commission to adopt implementing measures on:
– technical specifications of the topics and their breakdown (Regulation (EC) 1201/2009)
– programme of the statistical data and metadata to be transmitted to Eurostat (Regulation (EU) 519/2010)
– quality reporting and technical format of data transmission (Regulation (EU) 1151/2010)
6Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
Article 6 of Regulation (EU) 1151/2010
– Member States shall transmit the required data conforming to the data structure definitions and related technical specifications provided by the Commission (Eurostat)
– The technical format to be used for the transmission of data and metadata for the reference year 2011 shall be the Statistical Data and Metadata eXchange (SDMX) format
– Member States shall store until 1 January 2025 the required data and metadata for any later transmission requested by the Commission (Eurostat)
Technical format for data transmission
7Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
Based on Census Regulation - Data Hub is:
System of DSDs built on SDMX 2.0 (standard concepts and codes) and in use for 31 countries of the European Statistical System
data dissemination portal based on SDMX data model– communicating with data providers via SDMX Web Service
no data processing (no editing or aggregation) additional reusable modules for
– LAU / NUTS management – Tool for handling SDMX structural metadata
innovative user interface allowing user to extract data by starting from the statistical concept
Tests on-going: started with dummy data and sample hypercube, now to continue with real hypercubes
MSs status: 22 up and running
8Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
Population topics
Sex SEXAge AGELegal marital status LMSCountry/place of birth POBCountry of citizenship COCPlace of usual residence - one year prior to the census ROY(Size of the) Locality LOCHousehold status HSTType of private household TPHSize of private household SPHFamily status FSTType of family nucleus TFNSize of family nucleus SFN
Topics required for all geographical levels down to Local Administrative Units (LAU = municipalities)
Housing topics
Occupancy status of conventional dwellings OCSNumber of occupants NOCUseful floor space and/or Number of rooms UFS/NORDensity standard DFS/DRMDwellings by type of building TOBDwellings by period of constructionPOCType of living quarters TLQ
Which data for which geographical area?
9Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
Topics required for aggregated geographical levels NUTS 2, NUTS 1 and nation
Population topics
Year of arrival in the country YAE / YATEducational attainmentEDULocation of place of work LPWCurrent activity status CASOccupation OCCIndustry INDStatus in employment SIETenure status of households TSH
Housing topics
Housing arrangements HARType of ownership (of dwellings) OWSWater supply system WSSToilet facilities TOIBathing facilities BATType of heating TOH
Which data for which geographical area?
10Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
Example: DSD for Table 6 (Marital Status)
ID CONCEPT CODELIST
TIME Time period or range CL_TIME
GEO Geographical area CL_GEO
SEX Sex CL_SEX
FST Family status CL_FST
LMS Legal marital status CL_LMS
CAS Current activity status CL_CAS
POB Country/place of birth CL_POB
COC Country of citizenship CL_COC
AGE Age CL_AGE
FREQ Frequency CL_FREQ
ID ATTACHMENT LEVEL
CODELIST
OBS_STATUS Observation CL_OBS_STATUS
OBS_LEVEL Observation CL_OBS_LEVEL
OBS_NOTE Observation
HC_NOTE Series
ID NAME
OBS_VALUE Observation value
Dimensions
Measures
Attributes
11Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
The components
Data warehouse
Data warehouse
Data warehouse
SDMX-RI
(web service)
SDMX-RI
(web service)
SDMX-RI
(web service)
Data Hub
Data Providing Organizations Data collector Organizations Users
messagesSDMX
Data warehouse
Data warehouse
Data warehouse
SDMX-RI
(web service)
SDMX-RI
(web service)
SDMX-RI
(web service)
SDMX-RI
(web service)
SDMX-RI
(web service)
SDMX-RI
(web service)
Data Hub
Data Providing Organizations Data collector Organizations Users
messagesSDMX
12Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
Through the SDMX Hub, a data user can…
Browse the Hub to define a dataset of interest, navigating via structural metadata:- Select hypercube or search by topic (filters)- Select data (level of detail, breakdowns)- Select layout (axes)
View a table
Save a query
Export a file (CSV, Excel, SDMX-ML)
13Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
How the Hub works
Eurostat CensusHub
National Statistical Institute
National Statistical Institute
14Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
15Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
Lesson learnt and benefits in participating
Statistical needs first. Then, technological aspects.
Capacity-building is a must– Participating organisations are gaining a good in-house experience in
SDMX and its implementation
A system of distributed databases is harmonised through the use of SDMX standards and content guidelines
SDMX-RI can be reused for sharing data in other domains
– Limited cost for installations, development costs can be reduced– Step forward towards generic solutions for statistical domains
16Eurostat Unit B3 – IT and standards for data and metadata exchangeSDMX Basics Training – 2012
For more information
CIRCA: http://circa.europa.eu/Public/irc/dsis/x-dis-xensus-hub/library