sdmx it tools sdmx use in practice in na
DESCRIPTION
Table of contents Objectives National Accounts (NA) NA business needs NA Domains NA Technical Integration NA Data Validation Process NA Project CycleTRANSCRIPT
Eurostat
November 2015Eurostat Unit B3 – IT and standards for data and metadata exchange
SDMX IT ToolsSDMX use in practice in NA
Jean-Francois LEBLANCChristian SEBASTIAN
National Account use case
Eurostat
Table of contents1. Objectives2. National Accounts (NA)
1. NA business needs2. NA Domains3. NA Technical Integration4. NA Data Validation Process5. NA Project Cycle
2
Eurostat
Table of contents2. SDMX Tools in practice
1. Data Structure Wizard2. SDMX Converter3. SDMX Reference Infrastructure4. Tools relationship5. SDMX Tools – Summary
3. SDMX Tools for Data Providers1. Possibilities for Data Providers2. IT Architectures for Data Providers3. How to use the tools4. Conclusion
3
Eurostat
4
1 Objectives• To reflect how National Accounts work• To present the SMDX IT tools• To expose how to work with the data and SDMX in
real situations• To give a base understanding of the tools and
their relationship that will be of help for further explanations
Eurostat
5
2 National Accounts (NA)
BOP NationalAccounts
MainAggregates
SectorAccounts
Supply & Use
Input / Output
GFS
FDI
Excel questionnairesGESMES files
Eurostat
6
2 Why SDMX for NA?
Within Eurostat
Within the ESS
Internationally
Eurostat
2.1 NA business needs• Main trigger:
new methodology• SNA 93 SNA 2008• ESA 95 ESA 2010
• Business process reengineering• Corporate production systems• Integration of production
chains• Harmonised coding
7
Within Eurostat
Within the ESS
Internationally
Eurostat
8
2.1 NA business needs
Cross-domain integration• Balance of Payments• Foreign Direct Investment
ESS Business process reengineering
• Standardisation• Improved validation• Reuse of tools
Within Eurostat
Within the ESS
Internationally
Eurostat
9
2.1 NA business needs
Data quality• 1 country 1 year 1 GDP• Shared validation rules• Across all organisations
Lowering reporting burden• 1 data flow from national to
internationalInternational cooperation
• Inter-Agency Group on Economic and Financial Statistics = SDMX Sponsors
• Example: PGI Websitehttp://www.principalglobalindicators.org
Within Eurostat
Within the ESS
Internationally
Eurostat
10
2.2 NA Domains
National Accounts
SNA 2008ESA 210
Other related domains
FDI statistics
Other domains
Balance of
Payment
Statistics
Cross Domain Concepts and
Code Lists
Shared Concepts and
Code Lists
Eurostat
11
2.3 NA Technical Integration
Metadata driven process based on SDMX standards and tools Full automation of the data exchange possible
National statistical organizations
International organizations
National DBsMappingSourceSourceSource
SDMX DSDs
& Data Flows
National data sources
Eurostat
12
2.4 NA Data Validation ProcessBefore/during transmission“First Level”
Currently covered by SDMX• Format Check (SDMX-ML)• Code Check (SDMX DSD)• Code used correctly (SDMX Dataflow & Constraint)
After Transmission“Second Level”
Not covered by SDMX• Detailed value check• Validation expressions• …
Eurostat
13
2.5 NA Project Cycle
Eurostat
3. SDMX Tools in practice
• Find DSDs
• Edit/Create DSDs
• Data files -> SDMX files
• Database -> SDMX WS/files
14
Registry
DSW
Converter
SDMX-RI
Eurostat
15
3.1 Data Structure Wizard• Java tool (open source and platform independent)• Supports
• Maintenance of SDMX v2.0 and v2.1 structures• Import and export of artefacts from/to SDMX-ML v2.0,
v2.1 and GESMES format• Interaction with Euro SDMX Registry (retrieval,
modification and search of artefacts)• Sample dataset creation
Eurostat
16
3.2 SDMX Converter
• Java tool (open source and platform independent)• File based conversion• Different ways of using it:
• Graphical user interface• Command Line interface• Web Service interface• Common API interface
• Formats*: SDMX-ML 2.0, SDMX-ML 2.1, SDMX-EDI (Gesmes), FLR, CSV, Google DSPL, predefined Excel templates
*Exceptions may apply
Eurostat
17
3.3 SDMX Reference InfrastructureData Provider Data Collector
Non-SDMXlocal database
SDMX data set
SDMX Registry
DSD
SDMX-RI
Mapping Assistant
Web Svc
Test Client
Web Client
Eurostat
18
3.4 Tools relationship
DSW
SDMX Converter
SDMX Registry
SDMX-RI Mapping Assistant
DSD
National DBCSV or other datafile
DSD
SDMX Dataset
Mapping
produces
produces uses
produces
produces
uses uses
ESS-MHMSD
stores
MSD
uses
produces
SDMX MetaDataset
exposes
Census Hub
Eurostat
SDMX-ML Dataset
3.4.1 Tools relationship – Example
19
DSW
SDMXConverter
SDMX Registry
SDMX-RI
Mapping Assistant
DSD
National DB
Eurostat produces NA DSDs
NSI usesNA DSDs
Eurostat stores NA DSDs
NSI produces
NSI mapsNA DSDs
Mapping
NSI exports NA data (CSV, XLS)
NSI usesNA DSDs
DSD
Eurostat
20
3.5 SDMX Tools – Summary • Eurostat tools Eurostat SDMX Info Space
• SDMX Converter• SDMX Reference Infrastructure & Mapping Assistant• SDMX Registry & Data Structure Wizard
Eurostat
21
4. SDMX Tools for Data ProvidersTools offered Use Action
Web Forms (EU) Excel-like templatesTransmission of low volumes of data
No costs for EU organisationsManual work for senders (type, copy/paste)
SDMX Converter Converts data files from different file formats to SDMX format
Local installation on sender’s PC
Installation on server
SDMX Reference Infrastructure
A set of tools that allows to transform directly your data from a database to SDMX
Installation on serverMapping of database to DSDs
Common API Supports the development of local SDMX systems
Local development in organisations
Eurostat
22
4.1 Possibilities for Data Providers
WebformsEDAMIS
Single Entry Point
SDMX-ML file
Database export SDMX Converter SDMX-ML file
Database SDMX Reference Infrastructure
SDMX-ML fileSDMX Web Service
Same DSDSame Format
Excel sheets SDMX Converter SDMX-ML file
Eurostat
23
4.1.1 Possibilities for Data Providers. NA.
WebformsEDAMIS
Single Entry Point
SDMX-ML file
Database export SDMX Converter SDMX-ML file
Database SDMX Reference Infrastructure
SDMX-ML fileSDMX Web Service
Same DSDSame Format
Excel sheets SDMX Converter SDMX-ML file
Not applicable in National AccountsWill be phased
out in National Accounts
Eurostat
24
4.2 IT Architectures for Data Providers
Webforms Web-basedPush
mode (EDAMIS)
Database export
SDMX ConverterLocal conversion
Push mode
(EDAMIS)
Database SDMX Reference Infrastructure
Pull mode
Excel sheets
SDMX ConverterLocal conversion
Push mode
(EDAMIS)
Eurostat
25
4.3 How to use the tools
Inform your data providers for the available DSD
If you are a Data Collector
Eurostat
26
If you are a Data Provider4.3 How to use the tools
Download/pull the DSD from the SDMX Registry
If you have a DDB DDB
Data stores as files
Install and use the SDMX-RI
Install and use the Converter
Push via EDAMIS
Expose the data to be pulled (WS, HUB)
Eurostat
27
4.4 Conclusion SDMX statistical and technical standards & SDMX IT tools and IT infrastructureprovide the "toolkit" for the harmonisation and integration of data
structures the rationalisation of national/international data
flows finally improving the data quality of official
statistics.
Eurostat
28
SDMX in practice in NA