EurostatEurostat

EDAMIS AND VALIDATION SERVICES USER GROUP 2013

Summary and conclusions

DRAFT

August Götzfried and Håkan Linden

Unit B5

EurostatEurostat

1. Introduction• Eurostat outlined the context of the work of the EVUG covering:

• The Edamis and Validation services User Group deals with the ESS Single Entry Point Services and ESS validation services;

• New challenges for modernisation of European statistics: sharing of standards and services for data processing and sharing of data to reduce burden and improve efficiency;

• Two main ESS modernisation projects covered by EVUG: ESS VIP on Validation and the ESDEN cross-cutting project.

• The importance of new versions of EDAMIS and the work on SDMX statistical and technical standards implementation was highlighted regarding in particular future data exchange and validation;

• Data exchange; data sharing and data validation are of strategic importance for a more integrated ESS;

• This type of User Group should be enhanced to share experiences and work together on key issues related to the use of EDAMIS and validation services contributing to the modernisation work.

EurostatEurostat

2. Adoption of the minutes of the EDUG meeting 2012

• The minutes from the EDAMIS User Group in 2012 were approved.

EurostatEurostat

3.1 ESS vision and the ESS VIP program • Eurostat gave an overview of the ESS Vision, the ESS VIP program and the link to the need

for standards for integration, shared services and for sharing data; also the legislation needs to be integrated (like the framework regulation for business statistics);

• The ESS planning is also reflecting the business process integration (e.g. European Statistical Program 2013-2017);

• The ESS VIP program consists of common infrastructure and standards, sharing of information and services through business related projects and the governance framework;

• The EVUG is in particular concerned by two major actions under the ESS VIP program: ESDEN and the ESS Common Validation Policy;

• ESS business process integration and the implementation of the ESS Vision is a medium and long term process;

• The modernization of European Statistics and the ESS VIP program are also discussed regularly by the ESSC (ref. item 4 of the ESSC 27.9.2013).

EurostatEurostat

3.2 ESDEN - The future of EDAMIS under the ESS.VIP program

• Eurostat presented the ESS VIP cross cutting project on the European Statistical Data Exchange Network (ESDEN);

• The objective of ESDEN is to respond to business needs by providing value added services for the exchange of confidential and non-confidential statistical data among ESS partners;

• ESDEN is a possible network solution to replace the current EDAMIS infrastructure assuring continuity of the services offered by the SEP;

• The project will run in different stages building on feasibility studies and their implementation covering the period 2013-2018;

• The EVUG asked for clarifications in relation to SDMX-RI and EDAMIS and as well for their consistency; the SDMX-RI is seen as complementary to ESDEN;

• New business needs require different ways of data exchange (HUB, pull or transmission of confidential data) and the ESS IT infrastructure have to adapt to meet these needs;

• In the medium term ESDEN could lead to replacement of EDAMIS.

EurostatEurostat

3.3 The ESS group structure for corporate ESS statistical and IT services

• Eurostat presented the current ESS Group Structure related to Corporate statistical and IT services (covering Director Groups, Working Groups and User Groups);

• More integration of the current structure is needed and currently discussed at Eurostat; the main principles being that the work structure should be more organized along business layers as outlined in enterprise architecture;

• The proliferation of technical working groups needs to be limited; therefore the EDAMIS and the Validation User Group are merged;

• The EVUG supported the further integration and rationalization of the different Groups and a proposal was put forward to also integrate EVUG and SDMX-RI User Group;

• The EVUG also stressed the need for ongoing cooperation in-between meetings and between the Groups in order to progress in a coherent way on technical integration.

EurostatEurostat

3.4 The Mandate of the EVUG and legislation for standards

• Eurostat presented EVUG mandate and the draft terms of reference and these were accepted by the group;

• If the ESS group structure will change affecting the EVUG, the mandate needs to be updated accordingly;

• Eurostat gave an update on the latest state when it comes to the work on legislation on processes, standards and metadata;

• Currently, there are two options: (1) including a paragraph on standards in the amendment of 223/2009; or (2) to use the process of the Sponsorship on Standardization for defining the standards mentioned above.

EurostatEurostat

4.1 Use of EDAMIS - progress report

Eurostat presented the recent progress on EDAMIS covering:• EDAMIS 3.1 currently in production (including ECAS etc.);• EDAMIS 3.3 in pre-production phase including many new functionalities and

features (like integration of STADIUM engine into EDAMIS core; validation feed-back channel; multiple receivers etc.);

• EDAMIS 3.4 in testing soon and expected to be in production 2014 (incl. the pull requestor for pulling SDMX data via web services; and security improvements etc.);

• EDAMIS 3.5 is in progress to cover encrypted outgoing data sets and implementation of validation workflow (ESS VIP Validation requirement).

EurostatEurostat

4.1 Use of EDAMIS - progress report (cont'd)

• Currently 7 different version of EWA are in production. Countries was asked to upgrade to the latest EWA version in order to reduce maintenance costs;

• When the portal 3.3 is in production, EWA 3.3 will be the recommended version to use by countries;

• EDAMIS will continue to be developed and maintained until ESDEN solutions are in place;

• Eurostat answered to many technical questions from the participants regarding EDAMIS versioning, migration, functionalities, reports etc.;

• The discussions will be continued in sub-groups.

EurostatEurostat

4.2 EDAMIS - future developments

• Eurostat presented three major upcoming enhancements in EDAMIS covering: the Pull mode; the encryption for outgoing data; and the data validation process as defined by the ESS VIP on Validation;

• The Pull Requestor is integrated into EDAMIS 3.4 and once configured it can automatically pull available dataset instances;

• New business needs require datasets and validation reports containing confidential information to be sent also from Eurostat and therefore the necessary encryption procedures need to be set up and built into EDAMIS;

• The validation workflow also including the EDIT tool needs to be further specified and the necessary changes it implies to EDAMIS assessed and implemented;

• The EVUG was of the opinion that the business needs for the enhancements should be at in detail with some improvements to be done (like the possibility to test validations before transmission; the management of encryption keys, etc.).

EurostatEurostat

4.3 EDAMIS - architecture • Eurostat gave an overview of the EDAMIS infrastructure covering the components visible

to countries (EWA, EWP) and internal components (EDAMIS/Stadium server, Statel);• The security level of data exchange in EDAMIS was also briefly explained;• The EVUG took note of the EDAMIS architecture and was alerted on a number of

upcoming issues, such as:• 30.9.2013 is the deadline for several data collections that could imply a delay in

sending and receiving data. Countries were asked to deliver, if possible, before this date or in the morning on 30.9.2013;

• To use ECAS for the login since CIRCABC is phased out;• The EDAMIS web application is often part of the intranet of the NSIs and this causes

often problems for the communication when changes are done in the security infrastructure of the NSIs.

EurostatEurostat

4.4 EDAMIS: communication and training• Eurostat gave an overview of the Data Exchange and Validation Courier (DEVC) that is

prepared and distributed every two months;• Around 120 persons receives the DEVC;• In relation to EDAMIS it covers statistics and compliance for the preceding 3 months.• Eurostat provides several training services on EDAMIS: in-house courses to Eurostat

production units; ad hoc visits to countries, and also webinars are being set up to train national data users;

• The first webinar on EDAMIS will take place on 27.11.2013;• The EVUG recognized the value of the DEVC and showed interest in the web training

services; • The EVUG discussed the accessibility of information about planned data transmissions in

different statistical domains and as well the use of the SDMX-RI for the SDMX implementation projects.

EurostatEurostat

Breakout group discussions• A break out in four different groups was organized to discuss the following topics in more detail:

The ESS data transmission services; EDAMIS and its functionalities; EDAMIS user support; EDAMIS training services;

• The subgroup discussions concentrated on EDAMIS/CIRCABC access rights, detailed EDAMIS functionalities, the business processes for future version of EDAMIS and the integration with validation;

• In general EDAMIS is considered to work quite well; the possibility to validate files before transmission was mentioned several times;

• The transmission deadlines are not always correct or precise enough; a general high satisfaction of the EDAMIS support was expressed;

• The DEVC is considered useful and also redistributed inside NSIs; regarding training different needs require different types of courses also including some hands on trainings with exercises on EDAMIS etc.;

• The reporters of each subgroup were asked to send a summary of all the issues discussed to the EDAMIS support functional mailbox.

EurostatEurostat

5.1 The VIP "Data validation" – final deliverables

• Eurostat gave an overview of the ESS VIP on Validation covering the deliverables of the first phase (2012-2012) and the expected work packages of the second phase running as from 2013;

• The overall objective of the first phase was to develop validation standards and solutions to be used by different production processes within the ESS;

• Several of the deliverables from the first phase are of direct relevance for all the members of the ESS, such as: template and guidelines for a standard documentation of the validation process; standard definition of validation levels; a common typology of validation rules; and guidelines for distribution of responsibilities in the whole production chain;

• Work in the first phase was also started on IT/IS validation solutions and architecture providing the platform for a more integrated validation system for the ESS;

EurostatEurostat

5.1 The VIP "Data validation" – final deliverables (cont'd)

• The second phase focus on implementation of methodological developments done in first phase; functional specifications and integrated solutions for micro data validation; adaptation of existing IT tools to the functional specifications from the first phase; and deployment of validation applications/solutions to Member States;

• Eurostat is also undertaking an assessment of all statistical domains in the field of validation;

• A Task Force with Member States will be set up to identify best practices in the ESS and advice on implementation aspects. The terms of reference for the Task Force will be circulated to the EVUG within one month;

• The EVUG took note of the efforts to standardize validation in the ESS by working on technical and statistical standards and as well generic IT solutions for validation. This answers to the general ESS validation strategy.

EurostatEurostat

5.2 ESS validation services in Structural Business Statistics

• Eurostat gave an overview of the work on how to improve the data validation process for Structural Business Statistics (SBS) and the experiences in using EDIT;

• The old data validation process was considered time consuming and cumbersome with a lot of "ping pong" with countries regarding data quality;

• The redesigned validation process aims to give data providers as quick as possible feed-back on the data quality and avoid multiple versions to be treated. The basic idea being that data validation should be done by the data provider and a common IT application should be made available;

• EDIT is used as the IT tool for checking the quality of the data. The experiences seem in general positive from countries and the quality of the data has de facto improved;

• The SBS domain manager agreed on sharing of responsibilities for data validation for the reference year 2013;

• EVUG took note of the work on providing standard validation services in SBS also noting that the process on validation in SBS has improved and as well the data quality.

EurostatEurostat

5.3 The ESS validation services and their implementation

• Eurostat informed about the business needs for more efficient validation processes and the response to these needs by developing the generic data editing/ validation software EDIT;

• EDIT is a data validation/editing tool allowing users to import data, run validation programs and get reports;

• EDIT exist as standalone, server, web-based User Interface versions;• The future ESS Data Validation Service stemming from the ESS VIP on Validation

requirements will be built around EDIT as the validation tool and EDAMIS;• The EVUG noted the progress of the work on the ESS EDIT services; the need for ensuring

consistency with the ESS VIP on Validation work packages was emphasized; EDIT gest increasing used in statistical domains; there is a need to work on the new business requirements concerning the security/confidentiality of transmissions;

• The EVUG will also have an increasing role regarding the ESS shared validation services in the future.

EurostatEurostat

5.4 SDMX and data validation standards

• Eurostat gave an overview of the global SDMX work related to validation standards;

• The current initiative in the SDMX Technical Working Group on a standard validation language supporting the statistical process was highlighted;

• The work on a standard Validation and Transformation Language (VTL) was launched in 2012; the VTL implementation plan was also outlined covering the period until December 2014 when the full documentation should be available;

• The EVUG noted the effort of adding data validation to SDMX and stressed the importance to ensure good coordination of this SDMX work with the ESS VIP on Validation related activities in order to avoid that several validation languages are developed; in addition the necessary compliance with the GSIM information model was stressed.

EurostatEurostat

5.5 ESS validation services: training and communication

• Eurostat presented the new Wiki on the ESS Validation services and the training courses on validation as provided by Eurostat;

• The Wiki is accessible through the Eurostat website and is the single portal for information on the ESS validation services covering validation standards available; the status of implementation projects; information about ESS tools for validation; and dates of upcoming events;

• The training cover mainly training on the EDIT application; basic training on data validation as such; and implementation aspects for statistical domains;

• Webinars on the use of EDIT for countries are being set up and the first webinar will take place on 9 October 2013;

• EVUG took note of the ESS Validation wiki and the training services regularly provided (including e-training to the ESS users of validation).

EurostatEurostat

6.1 SDMX implementation: Progress report • Eurostat gave a debriefing from the SDMX Global Conference held 11-13

September 2013 in Paris and provided an overview of the ongoing SDMX implementation actions;

• Highlighted from the Global Conference were the open source components for the community; the work on the VTL; and the SDMX-RI usage also beyond the Census;

• There are now quite a number of statistical domains implementing SDMX involving many times also partners outside the ESS;

• The National Accounts project was used to push for a model of better international data sharing (with the principles of single flow from national to international, and the use of single value for a variable across all organizations involved);

• The EVUG took note of the accelerated implementation of SDMX in statistical domains and that SDMX has grown beyond being a pure data exchange standard.

EurostatEurostat

6.2 The Euro SDMX Registry (version 2) • Eurostat gave an overview of the Euro SDMX Registry as the repository for storing SDMX

artefacts used for SDMX based data/ metadata exchange by Eurostat, Member States and others;

• The Euro SDMX Registry version 2013 is expected to contain many enhancements and new functionalities (plus an improved performance, redesigned GUI, better access control mechanism and the support of the SDMX 2.1 technical specification);

• The work of the SDMX Sponsors on the Global SDMX Registry was also highlighted; this Registry will store the SDMX artefacts for global use. Other registries will be linked and SDMX artefacts will only be stored once; this registry is expected to be ready by end of 2013;

• The EVUG took note of the developments of the Euro SDMX Registry and the Global Registry; both should be released by the end of the 2013; these two applications will be linked to each other and contain all SDMX artefacts used for data and metadata exchange;

EurostatEurostat

All other business

• Redesign of the ESS work structure still to come in 2014• UNECE related events: possible sprint session on the revision of GSBPM and GSIM

in November 2013; possible workshop/seminar on standards for statistical modernization in autumn 2014.

• The SDMX RI User Group will take place 24-25 Sept. 2013.• The Classification WG will take place 28-29 Nov. 2013.• The next ESS MH User Group + Metadata WG + Classification WG is scheduled for

18-19 June 2014.• The next EVUG meeting is scheduled for 20-21 Oct. 2014.