essnet scfe deliverable d3-1 - ec.europa.eu · essnet scfe report on the implement ation of three...

Report on the implementation of three statistical services

This document is licensed under a Creative Commons License: Attribution-ShareAlike 4.0 International

DELIVERABLE


under a Creative Commons License: ShareAlike 4.0 International

ESSnet SCFEDELIVERABLE


Project acronym:

SCFE

Project title:

“Sharing common functionalities in the ESS”

Name(s), title(s) and organization or the author(s):

Romain Thailhurat

Franck Cotton

Mélanie Martin

Insee

Tel: +33 1 41 17 50 50

[email protected]

[email protected]

ESSnet SCFE D3-1

ation of three statistical services

“Sharing common functionalities in the ESS”

(s) and organization or the

33 1 41 17 50 50

,


TABLE OF CONTENT

1 Introduction................................

1.1 Work package objectives

1.2 The services ................................

1.3 Re-using organizations

1.4 Report contents ................................

2 Description of the services

2.1 The Time Series Service

2.1.1 Description and specification

2.2 The Questionnaire Generation Service


2.2.2 Original implementation

2.3 The Metadata dissemination service

2.3.1 General context ................................



3 The re-use of the Time Series Service

3.1 Preparation for re-use

3.1.1 Liaison with the National Bank of Belgium

3.2 Service re-use ................................

3.2.1 Re-use by Destatis ................................

3.2.2 Re-use by the ONS ................................

3.2.3 Re-use by Eurostat

4 The re-use of the Questionnaire Generation Service


4.1.1 Service re-design ................................

4.1.2 Documentation................................

4.1.3 Open-sourcing ................................

4.1.4 Development of the related tools

4.2 Service re-use by the SURS

5 The re-use of the Metadata Dissemination Service


5.2 Service re-use by the CASD


................................................................................................................................

Work package objectives ................................................................................................

................................................................................................................................

using organizations ................................................................................................

................................................................................................................................

Description of the services ................................................................................................

The Time Series Service ................................................................................................

Description and specification ................................................................................................

The Questionnaire Generation Service ................................................................


Original implementation ................................................................................................

mination service ................................................................

................................................................................................................................


Original implementation ................................................................................................

me Series Service ................................................................................................

use ................................................................................................

Liaison with the National Bank of Belgium ................................................................

................................................................................................................................

................................................................................................

................................................................................................

................................................................................................

use of the Questionnaire Generation Service ................................................................

use ................................................................................................

................................................................................................................................

................................................................................................................................

................................................................................................................................

Development of the related tools ................................................................................................

use by the SURS ................................................................................................

use of the Metadata Dissemination Service ................................................................

use ................................................................................................

use by the CASD ................................................................................................

2

.......................................................................... 4

...................................................................... 4

............................................................... 4

.......................................................................... 6

........................................................ 6

............................................................................. 6

........................................................................ 6

........................................................ 6

........................................................................... 9

........................................................ 9

............................................................... 10

............................................................................ 11

................................................. 11

..................................................... 11

............................................................... 12

..................................................... 13

........................................................................ 13

........................................................... 13

......................................................... 14

........................................................................... 14

............................................................................ 17

.......................................................................... 17

........................................................ 18

........................................................................ 18

.............................................. 18

.................................................. 19

................................................... 19

.............................................. 19

............................................................... 20

........................................................... 22

........................................................................ 22

............................................................... 23


6 Conclusion and lessons learned

6.1 Aim at reusable services from the start

6.2 Re-use is a collaboration

6.3 Re-use improves the quality of the services

6.4 Re-use is never just a technical problem

6.5 Standards are key for re

6.6 Good packaging facilitates re

6.7 Open source is the way to go

7 Bibliography ................................


Conclusion and lessons learned ................................................................................................

e services from the start................................................................

use is a collaboration ................................................................................................

use improves the quality of the services ................................................................

use is never just a technical problem ................................................................

andards are key for re-use, but sometimes not sufficient ................................

Good packaging facilitates re-implementation ................................................................

Open source is the way to go ................................................................................................

................................................................................................................................

3

................................................................ 25

....................................................................... 25

.................................................................... 26

.............................................................. 26

.................................................................... 26

............................................................... 27

........................................................ 27

........................................................... 28

...................................................................... 29


This report is the product of a collaborative work between theinvolved in the project, especially in the developing and reusing organizations.

1 Introduction

1.1 Work package objectives

The main goal of Work Package 3 is to demonstrat(developing organization, or DO in the(re-using organization, RO) and be integrated to its

This is of course in line with the owhich aims at setting up conditions for sharing statisticalStatistical Production Architecture (integration in statistical production processes at national, ESS and Commission level.

Work Package 3 is indeed the core of the Sharing illustrated by the important part of the estimated budget which is dedicated to it (29%) and the large number of participants involved.

1.2 The services

The grant technical specification gave the following details:

• the implementations in the ROs should preferably go beyond the Preach real business implementation or a level close to

• at least one of the applied services should be reenvironment, and at least one other in a

• the work package should demonstrate the remember,

• it should also demonstrate the renecessarily developed by an ESS member.

Regarding this last point, the consortiusolutions would be privileged. This aspect is detailed in the following paragraphs.

Insee France was leading WP3, and selected the services that were proposedEurostat's requirements, three services were proposed,production process.

• The Time Series Service (TSS in the rest of the document) deals with timeproposes various functions, notably in the fields of seasonal adjustThis service has a functional coverage which is quite focused and implements a widely established and shared methodology. It is based on JDemetra+, which is the reference time series software for the ESS. A preliminary version of context of the first CSPA pilots organized by the Uconsisted of the wrapping of JDemetra+


s report is the product of a collaborative work between the main redactors and theinvolved in the project, especially in the developing and reusing organizations.

Work package objectives

The main goal of Work Package 3 is to demonstrate that a service developed by an ESS Member “A” (developing organization, or DO in the rest of this document) can be implemented in ESS Member “B”

, RO) and be integrated to its business processes.

This is of course in line with the orientations of the SERV Vision Implementation Project as a whole, which aims at setting up conditions for sharing statistical services building on UNECE's Statistical Production Architecture (CSPA) and participating in their selection, implementation and integration in statistical production processes at national, ESS and Commission level.

Work Package 3 is indeed the core of the Sharing Common Functionalities ESSnet, important part of the estimated budget which is dedicated to it (29%) and the large

number of participants involved.

The grant technical specification gave the following details:

the implementations in the ROs should preferably go beyond the Proof on Concept level and reach real business implementation or a level close to that,

at least one of the applied services should be re-implemented by replication in the RO's local environment, and at least one other in a shared platform approach,

k package should demonstrate the re-use of up to 3 services developed by an ESS

it should also demonstrate the re-use of a service based on an open-sourcenecessarily developed by an ESS member.

Regarding this last point, the consortium submission went further and stated thatsolutions would be privileged. This aspect is detailed in the following paragraphs.

Insee France was leading WP3, and selected the services that were proposed ments, three services were proposed, all of which were used in Insee's statistical

The Time Series Service (TSS in the rest of the document) deals with timeproposes various functions, notably in the fields of seasonal adjustment and benchmarking. This service has a functional coverage which is quite focused and implements a widely established and shared methodology. It is based on JDemetra+, which is the reference time

ESS. A preliminary version of the TSS was developed by Insee in the context of the first CSPA pilots organized by the UNECE in 2014. This first version

wrapping of JDemetra+ in a REST web service, accompanied by

4

redactors and the people involved in the project, especially in the developing and reusing organizations.

e that a service developed by an ESS Member “A” implemented in ESS Member “B”

rientations of the SERV Vision Implementation Project as a whole, services building on UNECE's Common

and participating in their selection, implementation and integration in statistical production processes at national, ESS and Commission level.

Common Functionalities ESSnet, which is important part of the estimated budget which is dedicated to it (29%) and the large

roof on Concept level and

replication in the RO's local

developed by an ESS

source solution not

m submission went further and stated that open-source solutions would be privileged. This aspect is detailed in the following paragraphs.

for re-use. In line with all of which were used in Insee's statistical

The Time Series Service (TSS in the rest of the document) deals with time series and ment and benchmarking.

This service has a functional coverage which is quite focused and implements a widely established and shared methodology. It is based on JDemetra+, which is the reference time

TSS was developed by Insee in the in 2014. This first version basically

accompanied by a substantial


redesign of the input schema;dependencies.

The TSS is fully open-source: both JDemetra+ and the additional developments made by Insee are available on GitHub under

• The Questionnaire Generation Service (QGS) was devesurvey data collection platform (description of a questionnaire expressed in DDI (Data Documentation Initiativeproduces a web collection instrument ready tocollection platform (an XForms serverQGS relies on methods, standards and technologies that are innovative in the ESS, and thus re-use can have an important

All the developments made for the QGS have been published on GitHub by Insee under an MIT license. The only component of the complete deployment within Insee which is not open source is the XForms server (Orbeonproduct which is published on GitHubQGS does not require any

• The Metadata Dissemination Service (MDS) is a part of Insmétadonnées statistiques). component of RMéS, which is based on RDFmodel, the MDS is a kind of metadata model in order faire amount of work at the semantic level.

All the developments made for the MDS have been published on GitHub by InseeMIT license. The only component of the complete deployment within Insee which is not open source is the RDF triple store (GraphDBcan easily be replaced by any compliant triple store like which are open-source. The MDS does not require any featuresolutions.

We see that, in order to diversify the experiences and maximize the lessons learned, the chosen services were of quite different types, with varying levels of impact on the different interoperability axes.

1 http://www.ddialliance.org/ 2 See http://www.orbeon.com/ 3 The repository is available at https://github.com/orbeon/orbeon4 See https://www.w3.org/standards/techs/rdf#w3c_all5 See code repositories under https://github.com/InseeFr/6 Available at http://graphdb.ontotext.com/7 See http://rdf4j.org/ and http://jena.apache.org/


redesign of the input schema; this means that the service has virtually no technical

source: both JDemetra+ and the additional developments made by Insee are available on GitHub under permissive licenses.

The Questionnaire Generation Service (QGS) was developed by Insee as a survey data collection platform (named Coltrane). The service operates on the formal description of a questionnaire expressed in DDI (Data Documentation Initiativeproduces a web collection instrument ready to be personalized and deployed on a

n XForms server). Although its functional coverage is also very defined, QGS relies on methods, standards and technologies that are innovative in the ESS, and thus

an important organizational impact.

All the developments made for the QGS have been published on GitHub by Insee under an MIT license. The only component of the complete deployment within Insee which is not open source is the XForms server (Orbeon Forms2), but there is also a Community Edition of this product which is published on GitHub3 under a GNU Lesser General Public License v2.1. The QGS does not require any features not available in the Community Edition.

The Metadata Dissemination Service (MDS) is a part of Insee's RMéS tool (Référentiel de métadonnées statistiques). More precisely, it is the linked open metadata repository

, which is based on RDF-related standards4. Since RDF is a MDS is a kind of “meta-service”: the DO and RO have to agree on a

in order to re-use the service. We can thus expect that the refaire amount of work at the semantic level.

All the developments made for the MDS have been published on GitHub by InseeMIT license. The only component of the complete deployment within Insee which is not open source is the RDF triple store (GraphDB6), but there is also a free version of this product, or it can easily be replaced by any compliant triple store like Eclipse RDF4J or Apache Fuseki

. The MDS does not require any features not available in any of these

We see that, in order to diversify the experiences and maximize the lessons learned, the chosen t types, with varying levels of impact on the different interoperability

https://github.com/orbeon/orbeon-forms https://www.w3.org/standards/techs/rdf#w3c_all and https://www.w3.org/standards/techs/owl#w3c_all

https://github.com/InseeFr/ http://graphdb.ontotext.com/

http://jena.apache.org/

5

that the service has virtually no technical

source: both JDemetra+ and the additional developments made by Insee

loped by Insee as a component of its operates on the formal

description of a questionnaire expressed in DDI (Data Documentation Initiative1) and personalized and deployed on a survey

. Although its functional coverage is also very defined, QGS relies on methods, standards and technologies that are innovative in the ESS, and thus its

All the developments made for the QGS have been published on GitHub by Insee under an MIT license. The only component of the complete deployment within Insee which is not open

s also a Community Edition of this under a GNU Lesser General Public License v2.1. The

not available in the Community Edition.

ee's RMéS tool (Référentiel de More precisely, it is the linked open metadata repository

. Since RDF is a very basic d RO have to agree on a specific

We can thus expect that the re-use will imply a

All the developments made for the MDS have been published on GitHub by Insee5 under an MIT license. The only component of the complete deployment within Insee which is not open

), but there is also a free version of this product, or it RDF4J or Apache Fuseki7, not available in any of these

We see that, in order to diversify the experiences and maximize the lessons learned, the chosen t types, with varying levels of impact on the different interoperability

https://www.w3.org/standards/techs/owl#w3c_all


1.3 Re-using organizations

The re-using organizations were decided at the kick

• Destatis volunteered for the reEurostat, the OECD or the ABS showed some interest for the service during the timethe ESSnet, but no precise reactually re-use the service for its own needs. It should be Belgium offered a precious support to the recontributed. Though the installation of the service on a shared platform was a possible solution, Destatis chose to reconfidentiality and organizational reasons.

• SURS proposed to re-use the Questionnaire Generation Service. At the beginning of the ESSnet, the QGS was not available as a web service and due to some technical dependencies iwas not immediately possible to package it as a web service, so SURS decided to reimplement the core service within its own IT environment.

• Both the ONS and the CASD expressed interest in reService. After a few monthsthe only one described in this report. In order to maximize the benefits of the redecided to adopt a “share

1.4 Report contents

In the next sections, this report describes in more detail the services selected for recontext they were developed within their and what was the original implementation of the service.

A second part details the operations that were realized in order to prepare the service to be reused as a CSPA service, and what operations were made by the reimplement the service.

The report concludes with an evocation of the lessons conducted in the future.

2 Description of the services

This section gives a more detailed description of the services selected for reuse. For each service, we give a brief functional description, followed by some service.

2.1 The Time Series Service


2.1.1.1 Time series treatment

Time series treatment in generalofficial statistical process. Indeed, trends computing and seasonal adjustment are explicitly part of a GSBPM sub-process (6.1).


using organizations

using organizations were decided at the kick-off ESSnet meeting in Paris:

Destatis volunteered for the re-use of the Time Series Service. Other orgEurostat, the OECD or the ABS showed some interest for the service during the timethe ESSnet, but no precise re-use case could be defined with them, although Eurostat did

use the service for its own needs. It should be noted that the National Bank of Belgium offered a precious support to the re-use operation, and the Bundesbank also contributed. Though the installation of the service on a shared platform was a possible solution, Destatis chose to re-implement the TSS in its environment, in particular for confidentiality and organizational reasons.

use the Questionnaire Generation Service. At the beginning of the QGS was not available as a web service and due to some technical dependencies i

was not immediately possible to package it as a web service, so SURS decided to reimplement the core service within its own IT environment.

Both the ONS and the CASD expressed interest in re-using the Metadata Dissemination After a few months, the CASD re-use case was prioritized for various reasons

the only one described in this report. In order to maximize the benefits of the reshare” pattern for the re-use of this service.

ctions, this report describes in more detail the services selected for recontext they were developed within their developing organizations, what are their functionalities,

what was the original implementation of the service.

t details the operations that were realized in order to prepare the service to be reused as a CSPA service, and what operations were made by the re-using organization in order to

The report concludes with an evocation of the lessons learned and of additional work that could be

Description of the services

This section gives a more detailed description of the services selected for reuse. For each service, we give a brief functional description, followed by some details on the original implementation of the

Time Series Service

Description and specification

Time series treatment in general, and seasonal adjustment in particular, are core subjects in the s. Indeed, trends computing and seasonal adjustment are explicitly part of a

6

off ESSnet meeting in Paris:

use of the Time Series Service. Other organizations like Eurostat, the OECD or the ABS showed some interest for the service during the time span of

use case could be defined with them, although Eurostat did noted that the National Bank of

use operation, and the Bundesbank also contributed. Though the installation of the service on a shared platform was a possible

its environment, in particular for

use the Questionnaire Generation Service. At the beginning of the QGS was not available as a web service and due to some technical dependencies it

was not immediately possible to package it as a web service, so SURS decided to re-

using the Metadata Dissemination or various reasons and is

the only one described in this report. In order to maximize the benefits of the re-use, it was

ctions, this report describes in more detail the services selected for re-use: in which organizations, what are their functionalities,

t details the operations that were realized in order to prepare the service to be reused using organization in order to

learned and of additional work that could be

This section gives a more detailed description of the services selected for reuse. For each service, we details on the original implementation of the

core subjects in the s. Indeed, trends computing and seasonal adjustment are explicitly part of a


We proposed early in the ESSnet to select a Time Series Service as one ofreasons:

• as already mentioned, the domain of time serbuilding a service around that capability seems obviously a good idea,

• the statistical methodology of seasonal adjustment is particularly wellsupported,

• the existence of the Center of Excellencfor guidance, training material and tooling related to this domain.

2.1.1.2 Specification

As that could be the case with most shareable services, the goal is to provide a package as close as possible to the state of the art. Regarding the methodology of seasonal adjustment, this state of the art is described by the ESS Guidelines on Seasonal Adjustment.

The TSS is compliant with the ESS Guidelines on several points:

• it provides pre-treatment methods for outliers and

• it implements the recommended seasonal adjustment methods like TramoSeats (based on signal extraction) and X13 ARIMA (semi

We can also mention that the service, as already indicated, is provided as an opackage, and that is also a recommendation of the Guidelines on Seasonal Adjustment.

Furthermore, the service includes more

• automodelling: TRAMO and X13 provide automatic model identification identifying RegARIMA models (regression model with ARIMA noises) that fit given time series,

• benchmarking: find the set of parameters that preserves as much as possible the movements of the original series,

• other kinds of time series rel

8 See http://ec.europa.eu/eurostat/sa


et to select a Time Series Service as one of the reuse case for several

, the domain of time series treatment is a core process in statistics, so building a service around that capability seems obviously a good idea,

the statistical methodology of seasonal adjustment is particularly well

the existence of the Center of Excellence on seasonal adjustment, which, training material and tooling related to this domain.

As that could be the case with most shareable services, the goal is to provide a package as close as f the art. Regarding the methodology of seasonal adjustment, this state of the

art is described by the ESS Guidelines on Seasonal Adjustment.

The TSS is compliant with the ESS Guidelines on several points:

treatment methods for outliers and calendar effects using RegARIMA

it implements the recommended seasonal adjustment methods like TramoSeats (based on signal extraction) and X13 ARIMA (semi-parametric methods)

We can also mention that the service, as already indicated, is provided as an opackage, and that is also a recommendation of the Guidelines on Seasonal Adjustment.

Furthermore, the service includes more than just seasonal adjustment and provides:

automodelling: TRAMO and X13 provide automatic model identification identifying RegARIMA models (regression model with ARIMA noises) that fit given time

benchmarking: find the set of parameters that preserves as much as possible the movements

of time series related functions.

http://ec.europa.eu/eurostat/sa-elearning/regarima-models

7

the reuse case for several

ies treatment is a core process in statistics, so building a service around that capability seems obviously a good idea,

the statistical methodology of seasonal adjustment is particularly well-defined and

e on seasonal adjustment, which maintains support

As that could be the case with most shareable services, the goal is to provide a package as close as f the art. Regarding the methodology of seasonal adjustment, this state of the

calendar effects using RegARIMA8

it implements the recommended seasonal adjustment methods like TramoSeats (based on

We can also mention that the service, as already indicated, is provided as an open-source software package, and that is also a recommendation of the Guidelines on Seasonal Adjustment.

and provides:

automodelling: TRAMO and X13 provide automatic model identification (AMI) routines for identifying RegARIMA models (regression model with ARIMA noises) that fit given time

benchmarking: find the set of parameters that preserves as much as possible the movements


At a high level, inputs and outputs of the service can be described as

2.1.1.3 Original implementation

A version of the service may existand JDemetra are largely in use in

For example, Insee started a few years agolike the Producer Price Index. One of the main goalacross multiple applications.

In this context, seasonal adjustmentJDemetra+.

This first attempt at a cross-domain seasonal adjustment service was followed by the and release of a CSPA Seasonal Adjustment Service that was part of theproject: Insee worked closely with domain experts, the National Bank of Belgium CSPA actors to propose a fully-working service along with the proper documentation.

In parallel, the NBB started to deployJDemetra+. This implementation was

During this period, Insee worked closely with domain experts, the National Bank of Belgium and various CSPA actors to propose a fully working Beside the packaging as a service, the main development consisted in the restructuration of the schema of the input data to be sent to the service in order to service security.


At a high level, inputs and outputs of the service can be described as follows:

Original implementation

exist in a pre-SOA form in several NSIs: indeed, we know that Demetra JDemetra are largely in use in the ESS.

few years ago a renovation program (Premice) for its business indexes like the Producer Price Index. One of the main goals of the project was the sharing of functionalities

, seasonal adjustment calculations were obtained through a SAS macro

domain seasonal adjustment service was followed by the release of a CSPA Seasonal Adjustment Service that was part of the UNECE 2014

Insee worked closely with domain experts, the National Bank of Belgium working service along with the proper documentation.

deploy internally their own implementation of a web This implementation was fitted with the Bank's internal IT environment.

During this period, Insee worked closely with domain experts, the National Bank of Belgium and opose a fully working CSPA service along with the proper documentation.

Beside the packaging as a service, the main development consisted in the restructuration of the schema of the input data to be sent to the service in order to better align on GSIM and

8

we know that Demetra

a renovation program (Premice) for its business indexes of the project was the sharing of functionalities

through a SAS macro wrapping

domain seasonal adjustment service was followed by the development UNECE 2014 “CSPA Pilots”

Insee worked closely with domain experts, the National Bank of Belgium (NBB) and various working service along with the proper documentation.

own implementation of a web service based on fitted with the Bank's internal IT environment.

During this period, Insee worked closely with domain experts, the National Bank of Belgium and service along with the proper documentation.

Beside the packaging as a service, the main development consisted in the restructuration of the better align on GSIM and to improve


Those CSPA developments wereweb service encapsulating the JDemetra+ core package into a RESTful CSPA service.

2.2 The Questionnaire Generation Service


The Questionnaire Generation Service is Insee in order to generate survey collection instruments in a fully automated way from the formal description of the questionnaires. The main dand industrialize the production of questionnaires across the Institute, to facilitate the creation of instruments for different survey modes, as well as to showcase the concept of active metadata.

The metadata-driven approach reduces development costs, permits to have the documentation of the survey always in sync with the questionnaire in production. And beyond building collection instrument, this principle could be implemented on other subInsee has recently started a project aiming at generating from questionnaire specifications the data editing applications used in short

At a very high level, the principle is thus:

The choice of DDI as a modeling format for questionnaire

• A strong need for a precise modeling content and logical flow in different modes (i.e. a semantically rich model);

• DDI being an open standar

• DDI being serialized in XML, and thus easily processable.

A more detailed description of the design principles behind Eno can beCotton, & van der Vlist, 2013).

Eno was developed in several succhelper tools were added to it during the time span of the ESSnetwas created in order to help the survey designers to visually specify their questioPogues creates the DDI description of the questionnaire that is then fed into Eno. not initially in the perimeter of the very useful tool, and even indis


e CSPA developments were later integrated by the NBB into the JDemetra+ code base to form a web service encapsulating the JDemetra+ core package into a RESTful CSPA service.

Questionnaire Generation Service

and specification

The Questionnaire Generation Service is based on a software component (named Eno)Insee in order to generate survey collection instruments in a fully automated way from the formal description of the questionnaires. The main drivers behind this development were to standardize and industrialize the production of questionnaires across the Institute, to facilitate the creation of

for different survey modes, as well as to showcase the concept of active metadata.

driven approach reduces development costs, permits to have the documentation of the survey always in sync with the questionnaire in production. And beyond building collection instrument, this principle could be implemented on other sub-processes in the GSBPM.Insee has recently started a project aiming at generating from questionnaire specifications the data editing applications used in short-term business surveys.

At a very high level, the principle is thus:

ling format for questionnaire descriptions was motivated by:

trong need for a precise modeling of the questionnaire in order to be able to render its content and logical flow in different modes (i.e. a semantically rich model);

DDI being an open standard;

DDI being serialized in XML, and thus easily processable.

A more detailed description of the design principles behind Eno can be found in

Eno was developed in several successive steps (see next section), and various during the time span of the ESSnet. In particular, the Pogues design tool

was created in order to help the survey designers to visually specify their questioPogues creates the DDI description of the questionnaire that is then fed into Eno. not initially in the perimeter of the service proposed for re-use, it became quickly clear that it was a

, and even indispensable for an efficient use of Eno.

9

the JDemetra+ code base to form a web service encapsulating the JDemetra+ core package into a RESTful CSPA service.

(named Eno) developed by Insee in order to generate survey collection instruments in a fully automated way from the formal

rivers behind this development were to standardize and industrialize the production of questionnaires across the Institute, to facilitate the creation of

for different survey modes, as well as to showcase the concept of active metadata.

driven approach reduces development costs, permits to have the documentation of the survey always in sync with the questionnaire in production. And beyond building collection

he GSBPM. For example, Insee has recently started a project aiming at generating from questionnaire specifications the data

was motivated by:

of the questionnaire in order to be able to render its content and logical flow in different modes (i.e. a semantically rich model);

found in (Sigaud, Tailhurat,

essive steps (see next section), and various new functionalities and . In particular, the Pogues design tool

was created in order to help the survey designers to visually specify their questionnaires. Actually, Pogues creates the DDI description of the questionnaire that is then fed into Eno. While Pogues was

use, it became quickly clear that it was a



The original implementation of Eno takes place iinformation system. The figure below gives a very high

In this system, the DDI representationvariable schemes, concepts, etc. are stored in Insee's central instances are either edited by hand produced via the Pogues visual design toolthe Eno generator which producesplatform.

Eno and the collection platform are part of the Coltrane information system, together with several other modules for authentication, contact management, personalized web portal, survey management, etc. Of course, this very simplified view omits a lotpossibility to pre-fill the questionnaires with individual data or the process that extracts the response data to transfer them in the downstream data editing applications.

The first release of Eno aimed at creating difBusiness Survey (SBS). The innovative part of this operation was to produce the form in Open Document Format (then replaced by“generator”. The main goal was tometadata description was successfulwhole chain (from DDI to ODF) was achieved in February 2013.

The second release of Eno was able to generate web questionnaires from their description in DDI 3.2; those questionnaires were online in March 2015 for another business survey, and (mainly business) surveys wereextend the use of Eno and its associated tools to its household surveys.



of Eno takes place in the general framework of the . The figure below gives a very high-level view of this system

The RMéS/Coltrane information system

representations of the questionnaires, as well as the associated code lists, oncepts, etc. are stored in Insee's central metadata repository

instances are either edited by hand (that was mostly the case at the beginning of the system) or visual design tool. Once finalized, the DDI metadata are exported and fed to

the Eno generator which produces the XForms ready to be uploaded on the survey collection

Eno and the collection platform are part of the Coltrane information system, together with several other modules for authentication, contact management, personalized web portal, survey

Of course, this very simplified view omits a lot of functionalities, for example the fill the questionnaires with individual data or the process that extracts the

response data to transfer them in the downstream data editing applications.

The first release of Eno aimed at creating different types of questionnaires for the Structural Business Survey (SBS). The innovative part of this operation was to produce the form in Open

replaced by fillable PDF) from its metadata description in DDI 3.1, using a e main goal was to prove that this way of generating questionnaires from their

metadata description was successful, and thus to promote the paradigm of active metadatawhole chain (from DDI to ODF) was achieved in February 2013.

Eno was able to generate web questionnaires from their description in DDI 3.2; those questionnaires were online in March 2015 for another business survey, and

were integrated over the two following years. In 2017, Inextend the use of Eno and its associated tools to its household surveys.

10

n the general framework of the RMéS/Coltrane level view of this system:

the questionnaires, as well as the associated code lists, metadata repository (RMéS). The DDI

(that was mostly the case at the beginning of the system) or Once finalized, the DDI metadata are exported and fed to

ready to be uploaded on the survey collection

Eno and the collection platform are part of the Coltrane information system, together with several other modules for authentication, contact management, personalized web portal, survey

of functionalities, for example the fill the questionnaires with individual data or the process that extracts the

ferent types of questionnaires for the Structural Business Survey (SBS). The innovative part of this operation was to produce the form in Open

fillable PDF) from its metadata description in DDI 3.1, using a prove that this way of generating questionnaires from their

, and thus to promote the paradigm of active metadata. The

Eno was able to generate web questionnaires from their description in DDI 3.2; those questionnaires were online in March 2015 for another business survey, and tens of other

In 2017, Insee decided to


Future work will provide new output formats for Eno (paperfor Blaise questionnaires should follow soonDDI 3.1 to DDI 3.2. Regarding the Poguesthe possibility to define complex controls process but during the data editing process too.

2.3 The Metadata dissemination service

2.3.1 General context

Metadata management is recognized as a central piece of an industrialGSBPM, it is part of an overarching capability on top of the business processes alongsManagement.

Several initiatives exist or have existed to foster the adoption of standards and common usage of metadata in the official statistical industry.

One of the most recent was the 2016 UStandards” project, whose main

• conceive and build a database of globally harmonized core statistical metadata (concepts and codes) and make it available in both user

• create a semantically enhanced the main business models published by the HLGdocumenting services,

• develop a Modernizationlevels, and a Modernizationpossible.

The project clearly demonstrated leveraged to help achieve these objectives; of the major outcomes is “the proof that Linked Data is an excellent paradigm for representing statistical classifications and models sosoftware systems”. Following this (developed as one of the CSPA 2014

In the same idea, and for several yearsprojects, like the Australian Bureau of Statistics Metadata RegistryCanada Picasso project.

Insee, for its part, has been conducting for Statistiques, for Statistical Metadata Repository).subsystem of RMéS.

2.3.2 Description and specifica

As already mentioned, the MDS can e used to disseminate a large variety of metadata, for example concepts, classifications, provenance or quality metadata, product descriptions, etc. ecosystem provides numerous ontologies that can metadata.


Future work will provide new output formats for Eno (paper via PDF is nearly finishedquestionnaires should follow soon) and will upgrade the fillable PDF

. Regarding the Pogues questionnaire design tool, new developments will include the possibility to define complex controls that could be used not only during the data collection

editing process too.

Metadata dissemination service

Metadata management is recognized as a central piece of an industrial statistical systemGSBPM, it is part of an overarching capability on top of the business processes alongs

or have existed to foster the adoption of standards and common usage of metadata in the official statistical industry.

was the 2016 UNECE High Level Group “Implementing Modernmain objectives were to:

conceive and build a database of globally harmonized core statistical metadata (concepts and codes) and make it available in both user-friendly and machine-actionable formats,

create a semantically enhanced information system containing semantic representations of the main business models published by the HLG-MOS and of the CSPA standards for

Modernization Maturity Model to help statistical organizationsModernization Roadmap to help them progress to the next levels as efficiently as

clearly demonstrated how the powerful features of linked open metadata these objectives; the project's outputs are availablethe proof that Linked Data is an excellent paradigm for representing

statistical classifications and models so that they are formally correct and complete and usable by Following this idea, a new implementation of the Classification Retrieval

the CSPA 2014 pilot projects) was proposed.

and for several years now, several NSIs have started metadataprojects, like the Australian Bureau of Statistics Metadata Registry and Repository or the Statistics

has been conducting for some time the RMéS project (Référentiel de tatistiques, for Statistical Metadata Repository). The Metadata Dissemination Service

Description and specification

As already mentioned, the MDS can e used to disseminate a large variety of metadata, for example concepts, classifications, provenance or quality metadata, product descriptions, etc. ecosystem provides numerous ontologies that can support the description of such statistical

11

via PDF is nearly finished, specifications fillable PDF functionality from , new developments will include

that could be used not only during the data collection

statistical system. In the GSBPM, it is part of an overarching capability on top of the business processes alongside Quality

or have existed to foster the adoption of standards and common usage of

Implementing ModernStats

conceive and build a database of globally harmonized core statistical metadata (concepts and actionable formats,

information system containing semantic representations of MOS and of the CSPA standards for

organizations assess their current next levels as efficiently as

the powerful features of linked open metadata can be ts are available on the UNECE wiki. One

the proof that Linked Data is an excellent paradigm for representing omplete and usable by

Classification Retrieval Service

started metadata management and Repository or the Statistics

éférentiel de Métadonnées Metadata Dissemination Service is actually a

As already mentioned, the MDS can e used to disseminate a large variety of metadata, for example concepts, classifications, provenance or quality metadata, product descriptions, etc. The vast LOD

the description of such statistical


The original thinking regarding the redissemination and re-use of statistical concepts and classifications standards and open APIs.

This thinking was based on the fact that tthe semantic web regarding the Organization System (SKOS) and its extension (X

• SKOS provides classes and properties needed when describing controlled vocabularies (like classification schemes, thesauri, etc.), and relations between them,

• XKOS extends SKOS for theproviding more specific relationsconcordance tables.

It turned out that the most promising reoperations and products. This shows how the flexibility of RDF can be leveraged in order to adapt to different contexts.

In all cases, the MDS offers:

• Full access to the metadata through free SQARQLresults can be returned in different open formats (by standard content negotiation mechanisms.

• Simple access to the basic information about a resourceURI dereferencing.

• At an intermediate levequeries and thus allow the retrieval of rich information without the need to use SPARQL.

Since the metadata models used are themselves expressed in RDFsame mechanisms, which greatly improves (meta)data discoverability.


Insee has been disseminated RDF data and metadatacodes, for over 10 years on its RDF web siteweb community; in particular, it took part in the Datalift

This open source solution provides both the aforementioned capabilities, support for web semantic standards and open API. Its core principle is to "lift" data: raw data sources are provided in various format as inputs, then transformed into Linked Data using relevant vocabularies, and finally published via Datalift dissemination features (which includes querying t

9 See https://www.w3.org/TR/sparql1110 The “concise bounded description”, see 11 Using RDFS (https://www.w3.org/TR/rdfoverview/ and https://www.w3.org/TR/owl212 See https://datalift.org/


original thinking regarding the re-use of the MDS was that the ESSnet could of statistical concepts and classifications using Linked Open Data (LOD)

This thinking was based on the fact that there have been some important developments in the field of web regarding the support for statistical classifications, notably

KOS) and its extension (XKOS):

SKOS provides classes and properties needed when describing controlled vocabularies (like classification schemes, thesauri, etc.), and in particular gives the possibilityand relations between them,

XKOS extends SKOS for the specific needs of statistical classifications, for example in providing more specific relations between concepts, or representing classification levels or

It turned out that the most promising re-use case was about quality metadataoperations and products. This shows how the flexibility of RDF can be leveraged in order to adapt to

Full access to the metadata through free SQARQL9 queries from any HTTP client; query returned in different open formats (JSON, XML or CSV), the choice been made

by standard content negotiation mechanisms.

Simple access to the basic information about a resource10, in the same formats, through direct

At an intermediate level, access through REST APIs which encapsulate the most common queries and thus allow the retrieval of rich information without the need to use SPARQL.

Since the metadata models used are themselves expressed in RDF11, they can be obtained by the sms, which greatly improves (meta)data discoverability.


RDF data and metadata, in particular classifications and geographic RDF web site. The Institute has also been quite active in the semantic

n particular, it took part in the Datalift12 industrial research

This open source solution provides both the aforementioned capabilities, support for web semantic d open API. Its core principle is to "lift" data: raw data sources are provided in various

format as inputs, then transformed into Linked Data using relevant vocabularies, and finally published via Datalift dissemination features (which includes querying the data via SPARQL).

https://www.w3.org/TR/sparql11-overview/

The “concise bounded description”, see https://www.w3.org/Submission/CBD/ https://www.w3.org/TR/rdf-schema/) or the OWL2 mapping to RDF(https://www.w3.org/TR/owl2https://www.w3.org/TR/owl2-mapping-to-rdf/)

12

could focus on the Linked Open Data (LOD)

here have been some important developments in the field of , notably the Simple Knowledge

SKOS provides classes and properties needed when describing controlled vocabularies (like possibility to define concepts

, for example in , or representing classification levels or

use case was about quality metadata on statistical operations and products. This shows how the flexibility of RDF can be leveraged in order to adapt to

queries from any HTTP client; query ), the choice been made

, in the same formats, through direct

l, access through REST APIs which encapsulate the most common queries and thus allow the retrieval of rich information without the need to use SPARQL.

, they can be obtained by the

, in particular classifications and geographic The Institute has also been quite active in the semantic

research project.

This open source solution provides both the aforementioned capabilities, support for web semantic d open API. Its core principle is to "lift" data: raw data sources are provided in various

format as inputs, then transformed into Linked Data using relevant vocabularies, and finally he data via SPARQL).

https://www.w3.org/TR/owl2-


Insee later chose Datalift and RDF technologies as one of the main pillars metadata repository. The following figure gives a broad overview of the organization of the resulting RMéS information system.

Organization of t

In this system, client applications access via the the Colectica and Datalift repositories, whereas, of course, SPARQL applies only to RDF data. However, the above figure describes the full system which is internal to Insee, but only the repository is published on the web, and thus the external client applications can only access the of the API which is built on the RDF data. Strictly speaking, it isand of the SPARQL access that is called the Metadata Dissemination Service.

3 The re-use of the Time Series

This section describes chronologically the collaboration between the National Bank of Belgium (NBB) - TSS Developing Organization (DO), the Reusing Organizations, mainly Destatis, and Insee as a coordinator.


3.1.1 Liaison with the National Bank of Belgium

The first contacts with the NBB were made a few wemeeting. As indicated previouslyalready a history of collaboration between TSS: indeed, building on the pilotthe NBB set up an internal prototype of a time series service, tig


Insee later chose Datalift and RDF technologies as one of the main pillars of the redesign of its central metadata repository. The following figure gives a broad overview of the organization of the resulting

Organization of the RMéS information system

client applications access via the API and SPARQL entry pointsColectica and Datalift repositories, whereas, of course, SPARQL applies only to RDF data.

e figure describes the full system which is internal to Insee, but only the repository is published on the web, and thus the external client applications can only access the

built on the RDF data. Strictly speaking, it is the combination of this partial APIand of the SPARQL access that is called the Metadata Dissemination Service.

Time Series Service

This section describes chronologically the collaboration between the National Bank of Belgium (NBB) Developing Organization (DO), the Reusing Organizations, mainly Destatis, and Insee as a

use

Liaison with the National Bank of Belgium

NBB were made a few weeks after the inaugural Paris indicated previously when giving some context on the service inception, there had been

already a history of collaboration between the NBB and Insee. This was a strongpilot CSPA service for seasonal adjustment and on its own developments

NBB set up an internal prototype of a time series service, tightly integrated with JDemetra+.

13

the redesign of its central metadata repository. The following figure gives a broad overview of the organization of the resulting

entry points. The API is common to Colectica and Datalift repositories, whereas, of course, SPARQL applies only to RDF data.

e figure describes the full system which is internal to Insee, but only the Datalift repository is published on the web, and thus the external client applications can only access the part

the combination of this partial API

This section describes chronologically the collaboration between the National Bank of Belgium (NBB) Developing Organization (DO), the Reusing Organizations, mainly Destatis, and Insee as a

eks after the inaugural Paris coordination when giving some context on the service inception, there had been

strong starting point for asonal adjustment and on its own developments,

htly integrated with JDemetra+.


The first meetings were a good occasion to agree on the perimeter for

• Seasonal adjustment algorithms, the well

• Benchmarking methods:

• Automatic modeling, using Forecast and Checklast methods.

We also rapidly agreed that the serviceand outputs, whereas the first versions of thesimple dictionary-oriented payloads.GSIM objects and a stricter validation of the requests made to th

Starting from the work done for the CSPA pilot, t2016, with help from Insee on naming rules and namespaces organization. Those schemas were from the start integrated in the JDemetra+ core schemas

According to the agreed planning, of October that included the schemas and the features defined previously.was made available to Destatis,

Quickly following the October release, the design of the schemas sparked some discussions and call for a light redesign which was done on the following weeks. Insee also contproviding documentation, code and

At the very beginning of 2017, Insee proposedefine the use case and design the target solution. Actuallydescribed for the meeting with the appropriate test data (more on that in the following section).

Mid-March 2017, the NBB announced a new version of the service to be released by themonth, which indeed was the case: updated web service as well as arequest payloads for the service.

In May, the NBB confirmed to Destatis that future release of JDemetra+.

3.2 Service re-use

During the inception of the ESSNet, three

• The ONS was the first reLisbon meeting in July 2016

• Destatis was subsequently

• Eurostat also showed interest f

3.2.1 Re-use by Destatis

We described in a previous section the preparatory work needed to provide athe ROs. This work led to a first public version first steps in integrating the service during


The first meetings were a good occasion to agree on the perimeter for the candidate service for SCFE:

adjustment algorithms, the well-knows X13 and Tramo-Seats,

Benchmarking methods: Cholette and Denton,

Automatic modeling, using Forecast and Checklast methods.

the service should adopt a schema-based approach for handling inputsthe first versions of the NBB JDemetra+ web service consumed and produced

oriented payloads. The schema-oriented approach allowed an easier mapping to GSIM objects and a stricter validation of the requests made to the service.

Starting from the work done for the CSPA pilot, the NBB produced the schemas by the end of 2016, with help from Insee on naming rules and namespaces organization. Those schemas were from the start integrated in the JDemetra+ core schemas to benefit from a coherent development.

planning, the NBB delivered the first official release of TSS on the first days that included the schemas and the features defined previously. This release of the service

, which started to assess it and to test its functionalities.

release, the design of the schemas sparked some discussions and call for a light redesign which was done on the following weeks. Insee also contributed to the project, by

ntation, code and general feedback.

At the very beginning of 2017, Insee proposed a joint meeting with the NBB and Destatis to precisely case and design the target solution. Actually, Destatis announced two use

described for the meeting with the appropriate test data (more on that in the following section).

NBB announced a new version of the service to be released by themonth, which indeed was the case: on March 31, JDemetra+ 2.2.0 was published, which included the updated web service as well as a new tool for transforming JDemetra+ workspaces into proper

for the service.

B confirmed to Destatis that their intention was to update the web service

of the ESSNet, three potential re-using organizations were identified:

ONS was the first re-using candidate organization, however it was decided after the y 2016 that the priority was on their work on the g

subsequently designed as the main re-using organization

interest for the service, but no specific re-use case was provided.

We described in a previous section the preparatory work needed to provide athe ROs. This work led to a first public version of the TSS in October 2016. Hencefirst steps in integrating the service during this same month.

14

the candidate service for SCFE:

Seats,

based approach for handling inputs consumed and produced

oriented approach allowed an easier mapping to

NBB produced the schemas by the end of summer 2016, with help from Insee on naming rules and namespaces organization. Those schemas were from

to benefit from a coherent development.

NBB delivered the first official release of TSS on the first days This release of the service

assess it and to test its functionalities.

release, the design of the schemas sparked some discussions and call ributed to the project, by

NBB and Destatis to precisely nced two use cases, to be

described for the meeting with the appropriate test data (more on that in the following section).

NBB announced a new version of the service to be released by the end of the 31, JDemetra+ 2.2.0 was published, which included the

workspaces into proper

to update the web service with every

s were identified:

didate organization, however it was decided after the the guidelines (WP1).

using organization for the TSS.

use case was provided.

We described in a previous section the preparatory work needed to provide a first running service to ence, Destatis took its


This exploration by Destatis rapidly produced

• Mid-October, the question of the possibility of custom specificationadjustment algorithm was r

• During the following months, documentation.

As a result of those feedbacks, it was suggested that the best way to report would be to use the issue tracker on the designated source repository on Gitsuccessfully used to report two

Destatis presented their two use caseone about the future development of the car market and the other about the German GDP. In both cases, the “as-is” and “to-be” architectures were quite similar, and the discussion during the meetinghelped grounding the re-use on

The previous figures show the work flow and technical architecture of the recurrent setup, Destatis uses the official X12 binaries from the US Bureau of CensusSAS macro which is executed on a Solarcorresponding departments responsible for seasonal adjustment of the timeexported to Excel for verificationparameters of the seasonal adjustment procedure.

In the target situation (below), the call to the X12 binaries is replaces by a call to the TSS web service:

13 More detail is provided in the presentation made by Destatis at the Wiesbaden workshop, which is available at https://ec.europa.eu/eurostat/cros/conten


This exploration by Destatis rapidly produced results:

October, the question of the possibility of custom specifications for eacwas raised, with a quick answer from NBB, backed by code examples,

following months, various comments were made on minor

As a result of those feedbacks, it was suggested that the best way to report questionsthe issue tracker on the designated source repository on GitH

two bugs that were quickly corrected by the NBB.

use cases for the TSS during the Ljubljana meetingone about the future development of the car market and the other about the German GDP. In both

architectures were quite similar, and the discussion during the meetinguse on a firm technical base.

s show the work flow and technical architecture of the re-

current setup, Destatis uses the official X12 binaries from the US Bureau of Censuswhich is executed on a Solaris server. This macro is invoked by the analysts in the

corresponding departments responsible for seasonal adjustment of the time-exported to Excel for verification and further processing. Other file exchanges are made for the

eters of the seasonal adjustment procedure.


More detail is provided in the presentation made by Destatis at the Wiesbaden workshop, which is available at

https://ec.europa.eu/eurostat/cros/content/wiesbaden-workshop-presentations-session-2_en

15

for each seasonal aised, with a quick answer from NBB, backed by code examples,

minor discrepancies in the

questions or comments Hub. This method was

NBB.

ing (December 2016): one about the future development of the car market and the other about the German GDP. In both

architectures were quite similar, and the discussion during the meeting

-use cases13. In the current setup, Destatis uses the official X12 binaries from the US Bureau of Census, wrapped into a

This macro is invoked by the analysts in the -series. The results are

and further processing. Other file exchanges are made for the


More detail is provided in the presentation made by Destatis at the Wiesbaden workshop, which is available at 2_en


This target situation is in fact an intermediary step towards a solution where the service would be called directly from the analyst's workstation.

During the Ljubljana meeting, DestatisNBB, which, as previously mentionedagreed on a schedule running from January to June, with a first integrated version of the service in Destatis infrastructure at the end of June.

The February meeting in Brussels was pinpointed the different subjects that still needed to behandle them.

The first important information gathered was the for Germany official statistics: the would come in different formatadjustment parameters were to be provided by the Bundesbankform corporate servers, different in the Lessons Learned section.

The Brussels meeting was also important for identifying solution. For example, transforming JDemetraneed to use dedicated software, to be shows the different components:

Taking these elements into considerationuse operation.


is in fact an intermediary step towards a solution where the service would be analyst's workstation.

During the Ljubljana meeting, Destatis also expressed the need for a coordination meeting withmentioned, was set up for the start of February of the next year. We also

schedule running from January to June, with a first integrated version of the service in Destatis infrastructure at the end of June.

The February meeting in Brussels was an important step in the re-use case for ects that still needed to be dealt with, and produced a roadmap to

The first important information gathered was the precise organization around seasonal adjustment for Germany official statistics: the SCFE implementers were not the only pilotswould come in different formats from the various methodology departments,adjustment parameters were to be provided by the Bundesbank Also, as the treatments were called form corporate servers, different IT actors had also to be involved. This subject is in the Lessons Learned section.

The Brussels meeting was also important for identifying the components needed for the target solution. For example, transforming JDemetra+ workspaces into proper web need to use dedicated software, to be developed by our Belgian colleagues. The following diagram shows the different components:

Taking these elements into consideration, the meeting produced an integrated schedule for the re

16

is in fact an intermediary step towards a solution where the service would be

the need for a coordination meeting with the , was set up for the start of February of the next year. We also

schedule running from January to June, with a first integrated version of the service in

use case for the TSS, as it dealt with, and produced a roadmap to

organization around seasonal adjustment only pilots in the ship. The data

from the various methodology departments, and the seasonal Also, as the treatments were called

. This subject is further discussed

components needed for the target proper web services requests would

The following diagram

the meeting produced an integrated schedule for the re-


On the following weeks, the first steps to implementing the solution werethe current SAS environment and adding glue code for part of the time was spent waiting for other actorquestions to the methodology department or the Bundesbank.

To closely monitor the agreed-upon and Insee set up a weekly report every were the input format of the seasonal adjustmby NBB on the JWSrequests tool

At the end of the summer, a prototypeDestatis. Then, some discussions started with the prototype into the already existing lscenario, considering the desired

During September 2017, Destatis was implementing and testing this first businessthe web service. A lot of work was done to define de especially since some changes were planned in 2018 on the input data

It was necessary to specify precisely department, and how to shape those data into a format data are converted from SAS to ExcelSeries Service. The result of the service is returned in XML, and ha

The whole process had to be fit into a singprocess of seasonal adjustment and data analysis. component (a “CSPA Adaptor”)

In October, the results associated to the the methodology department. The following extended scenario (multiple time

Using other and more complex use reported and corrected by the NBB.

As of January 2018, additional tests are scheduled and the production at the beginning of March.

3.2.2 Re-use by the ONS

As stated before, the ONS couldproof-of-concept integration of the service not followed into production since prioritized elsewhere.

3.2.3 Re-use by Eurostat

In October 2016, contact was made with Nikhilthe TSS, which he successfully installed days later. Some issues, both on the technicalbusiness sides, were raised and transmitted to


On the following weeks, the first steps to implementing the solution were made, including adapting nment and adding glue code for the process orchestrati

waiting for other actors to answer different questions, methodology department or the Bundesbank.

upon schedule and to act proactively on the pending and Insee set up a weekly report every Thursday starting mid-April. Some important developments were the input format of the seasonal adjustment specifications on one side and the correction made by NBB on the JWSrequests tool on the other.

At the end of the summer, a prototype web service was implemented within the IT infrastructure of Then, some discussions started with the methodology departments on how to implement type into the already existing legacy production flows. Destatis settled for a specific

desired outputs and the requirements for given input

, Destatis was implementing and testing this first businessthe web service. A lot of work was done to define de input/output structure for

were planned in 2018 on the input data.

precisely how the input data would be provided by the methodology how to shape those data into a format that the web service could

data are converted from SAS to Excel, and then into XML. They are subsequently sentSeries Service. The result of the service is returned in XML, and has to be parsed back into SAS.

The whole process had to be fit into a single SAS macro, in order to make it usable within the existing ment and data analysis. The National Bank of Belgium developed a Java

) for this.

results associated to the first re-use case scenario was calculatedThe following step was to extend the prototype in order to cover an

extended scenario (multiple time-series per workspace, custom calendar data

Using other and more complex use cases was the occasion to discover a bug in JDemetra+NBB.

additional tests are scheduled and the service is scheduledproduction at the beginning of March.

could not participate as a RO of the TSS. Nonetheless, integration of the service into an existing system. The PoC was

since the target system was phased out and the

, contact was made with Nikhil-Vijay Kale from Eurostat who was very interested by TSS, which he successfully installed days later. Some issues, both on the technical

business sides, were raised and transmitted to the NBB using the GitHub tracker.

17

made, including adapting orchestration. However, a large

s to answer different questions, in particular

the pending issues, Destatis . Some important developments one side and the correction made

within the IT infrastructure of methodology departments on how to implement

flows. Destatis settled for a specific input formats.

, Destatis was implementing and testing this first business re-use case using for this business case,

data would be provided by the methodology could use. Typically, the

. They are subsequently sent to the Time to be parsed back into SAS.

it usable within the existing he National Bank of Belgium developed a Java

calculated and under review by step was to extend the prototype in order to cover an

calendar data…) to be executed.

in JDemetra+ that was

is scheduled to be put in

TSS. Nonetheless, the ONS realized a into an existing system. The PoC was successful, but it was

the resources was

Vijay Kale from Eurostat who was very interested by TSS, which he successfully installed days later. Some issues, both on the technical and the

ub tracker.


It was agreed with Eurostat that theeveryone.

It should be noted though that in the context of this reEurostat's SOA platform. Consequently, the service is now available on a production platform of the ESS.

4 The re-use of the Questionnaire


The service was initially developedcompliance or reusability by other NSIs. As a consequence, it was quite dependant on its technical environment and not at all internationalized. Furthermore, there was no particular license ato the code, which was not published as open

Thus, the following points had to be improved be

• Service re-design in order to make it more flexible, and independent

• Documentation (functional and technical)

• Release as open sourceESSnet submission)

Moreover, it became quickly evident that in order to demonstrate the service it was indispensable to show the whole process, from the design of the questionnaire to the generation of the survey instrument and its publication on a collection platform.internationalize, document and open source all the tools that support this process, Pogues questionnaire designer.

4.1.1 Service re-design

The service re-design was guided

• A clear separation between the functionalities that are of general interestmore specifically used by Inseeorganization (for example a specific welcome page in the generated questionnaires).

• Similarly, a clear separation questionnaire structure and those rinstrument.

• Finally, an encapsulation of technical dependencies (security prrepositories…) so that the corresponding functions could easily be retarget environment.

The redesign was conducted internally Insee engaged much more resources in this operation that what was reserved for the ESSnet, but the results were also clearly beneficial for the Institute


that their re-use of the TSS was to be developed on a

that in the context of this re-use case, the TSS was successfully installed on tform. Consequently, the service is now available on a production platform of the

Questionnaire Generation Service

use

loped within Insee, without specific considerations for its CSPA compliance or reusability by other NSIs. As a consequence, it was quite dependant on its technical environment and not at all internationalized. Furthermore, there was no particular license a

hich was not published as open source.

Thus, the following points had to be improved before offering the service for

design in order to make it more flexible, and more environment

tation (functional and technical)

(though not a CSPA requirement, it was an orientation laid out in the

Moreover, it became quickly evident that in order to demonstrate the service it was indispensable to whole process, from the design of the questionnaire to the generation of the survey

instrument and its publication on a collection platform. Consequently, Insee decided to internationalize, document and open source all the tools that support this process, Pogues questionnaire designer.

guided by different objectives:

lear separation between the functionalities that are of general interestmore specifically used by Insee, and those that could be personalized by each re

(for example a specific welcome page in the generated questionnaires).

clear separation between the features of the generator dealing solely with the questionnaire structure and those related to the presentation or layout

Finally, an encapsulation of technical dependencies (security providers, access to …) so that the corresponding functions could easily be re-

The redesign was conducted internally as part of the maintenance activities of the different tools. Insee engaged much more resources in this operation that what was reserved for the ESSnet, but the results were also clearly beneficial for the Institute itself.

18

developed on a best-effort for

se case, the TSS was successfully installed on tform. Consequently, the service is now available on a production platform of the

without specific considerations for its CSPA compliance or reusability by other NSIs. As a consequence, it was quite dependant on its technical environment and not at all internationalized. Furthermore, there was no particular license associated

fore offering the service for re-use:

environment- and language-

, it was an orientation laid out in the

Moreover, it became quickly evident that in order to demonstrate the service it was indispensable to whole process, from the design of the questionnaire to the generation of the survey

Consequently, Insee decided to internationalize, document and open source all the tools that support this process, including the

lear separation between the functionalities that are of general interest, those that are that could be personalized by each re-using

(for example a specific welcome page in the generated questionnaires).

the features of the generator dealing solely with the elated to the presentation or layout of the resulting

oviders, access to -implemented in the

as part of the maintenance activities of the different tools. Insee engaged much more resources in this operation that what was reserved for the ESSnet, but the


Another redesign operation was conducted on Eno in order to properly separate the core logic with its invocation interfaces, leading to splitting the codelogic, the REST API and a commandunder three packagings: command line tool, web service and Docker container.

4.1.2 Documentation

Besides this architectural refactoring, it was crucial to translate comments…) and the documentation of the service before openwas accomplished with the help of

In parallel, a first English version of the guide on Eurostat and SURS. The guide had been previously communicated to the DDI Alliance as illustration of the metadata-driven building of collection

This example of the “active metadatacontacts with Statistics Netherland (CBS) questionnaire generation. These contacts resulted in the organization of a meeting, which was scheduled at in Lille, whereas a parallel meetioutcome of these meeting was an agreement on the fact that the guide should be regarded as the specification of a common level of interoperability between the different solutions.

4.1.3 Open-sourcing

Although there was a strong commitment new to the institute. After the release in September of the first intGitHub, we hesitated for some time between the open code and the developments that were still made internally. After a few months building experience, it was decidinternal developments?" with the aid of a specialized company.and 28th of March 2017, and brought answers to most of our interrogationsworkshop were transmitted to our Portuguese colleagues in order to feed their own work on opensource (WP5).

4.1.4 Development of the related tools

As indicated above, the development of Pogues was an QGS and to help its re-use. In particular, it allowed the creation of a demonstration platform where users could experiment the whole process of designing and generating

Based on the initial software createthe start), the developments were realized in two phases:

• A first tender at the end of 2016 produced a version which Ljubljana meeting.

• A second call for tender was published to further develop meet the needs expressed by SURS. Indeed, a Slovenian colleagues at the beginning of Q4

14 Available at https://inseefr.github.io/Eno/ddi


was conducted on Eno in order to properly separate the core logic with , leading to splitting the code base into three component

logic, the REST API and a command-line application. This opened the possibility to release the QGS under three packagings: command line tool, web service and Docker container.

Besides this architectural refactoring, it was crucial to translate the code (variable names, umentation of the service before open-sourcing it. This interna

th the help of a subcontractor, starting by the end of July 2016.

version of the guide on representing questionnaires in Eurostat and SURS. The guide had been previously communicated to the DDI Alliance as

driven building of collection instrument using the standard.

active metadata” paradigm sparked interest in other NSIs. with Statistics Netherland (CBS) and exchanged views on our respective work

These contacts resulted in the organization of a joint led at the end of January 2017. The technical part of the meeting was held

meeting at senior management level took place in Parisoutcome of these meeting was an agreement on the fact that the guide should be regarded as the

ation of a common level of interoperability between the different solutions.

a strong commitment by Insee on the question, the open-he release in September of the first internationalized version of Eno

GitHub, we hesitated for some time o the best organization, in particular for the coordination between the open code and the developments that were still made internally. After a few months

t was decided to organize an internal workshop on “How to openwith the aid of a specialized company. This workshop took place

2017, and brought answers to most of our interrogations. The deliverables to our Portuguese colleagues in order to feed their own work on open

Development of the related tools

As indicated above, the development of Pogues was an important factor in order to demonstrate the In particular, it allowed the creation of a demonstration platform where

users could experiment the whole process of designing and generating a questionnaire.

software create internally by Insee (but open sourced and intethe start), the developments were realized in two phases:

tender at the end of 2016 produced a version which was showcased during the

second call for tender was published to further develop Pogues featuresmeet the needs expressed by SURS. Indeed, a preview version was produced for our Slovenian colleagues at the beginning of Q42017.

https://inseefr.github.io/Eno/ddi-qg-guide.html

19

was conducted on Eno in order to properly separate the core logic with components: the generation

This opened the possibility to release the QGS under three packagings: command line tool, web service and Docker container.

(variable names, This internationalization

the end of July 2016.

representing questionnaires in DDI14 was sent to Eurostat and SURS. The guide had been previously communicated to the DDI Alliance as an

instrument using the standard.

r NSIs. In particular, we had views on our respective works on

joint CBS-SURS-Insee The technical part of the meeting was held

at senior management level took place in Paris. The main outcome of these meeting was an agreement on the fact that the guide should be regarded as the

ation of a common level of interoperability between the different solutions.

-sourcing was somewhat ernationalized version of Eno on

o the best organization, in particular for the coordination between the open code and the developments that were still made internally. After a few months

How to open-source shop took place on the 27th

. The deliverables of the to our Portuguese colleagues in order to feed their own work on open-

important factor in order to demonstrate the In particular, it allowed the creation of a demonstration platform where

a questionnaire.

(but open sourced and internationalized from

was showcased during the

features, in particular to preview version was produced for our


As already mentioned, all the code of the UI issource every other component of the system, particularly the gateway between Pogues and the generation service.

4.2 Service re-use by the SURS

The original re-using organizations for Statistics Lithuania. However, during the Lisbon meeting, the resources. Also, the ONS expressed interest in theagreed that their potential re-use

It must be also noted that the QGS was installed by Eurostat on its SOA platform shortly before the end of the ESSnet.

The first step in SURS re-use of which the service and its associated questpresented.

After the first open-source release of the servcenter in September 2016. It was good news to see had at the same time the first discussions about generating questionnaires, which led Insee to translate the aforementioned modeling guide and communicate it to our Slovenian colleaguesand difficulties for modeling their existing questionnaires, most specifically using the right parts of DDI, according to the previously shared guide. Durprovided a more detailed report on the reuse case and the design. It was quickly agreed that a dedicated rapidly on the pending questions

A mission of Insee DDI experts to multiple transportation problems, March. SURS took advantage of this delay to into their production platform, and concluded that toperation. This option implied a change in the ESSnet spending plan, and was thus validated with the project coordinator in Eurostat before it w

At the beginning of March, a coordination meeting between SURS and Insee prepared the DDI mission, and indeed, two weeks later, the mission was a success: both the issues on DDI modeling and the questions of technical integration were emerged:

• the need to have an explicit DDI proforder to verify that a given questionnaire conformed to the modeling guidelines

• a growing interest of SURSand the publication of the

Also, some questions were still pending regarding


, all the code of the UI is open-source, and we decided in early 2017 to openother component of the system, particularly the gateway between Pogues and the

by the SURS

using organizations for the QGS were the Statistical Office of Sloveniaduring the Lisbon meeting, the latter retracted because of a lack of

the ONS expressed interest in the service during a meeting in February 2017, use would be addressed outside of the ESSnet frame.

must be also noted that the QGS was installed by Eurostat on its SOA platform shortly before the

use of the QGS was a video-conference in March 2016 with Inseewhich the service and its associated questionnaire design tool (at that time quite rudimentary)

source release of the service, SURS was able to install it successfully in its data center in September 2016. It was good news to see that the technical integration had at the same time the first discussions about the details of the DDI modeling required for

, which led Insee to translate the aforementioned modeling guide and communicate it to our Slovenian colleagues. SURS reported in the following weeks some

difficulties for modeling their existing questionnaires, most specifically using the right parts of DDI, according to the previously shared guide. During the Ljubljana meeting in

ided a more detailed report on the reuse case and the issues encountered with the that a dedicated working meeting was needed in order to progress

rapidly on the pending questions.

to Ljubljana was thus planned for February 2017to multiple transportation problems, the mission had to be postponed and was

took advantage of this delay to perform a system analysis for the integration o, and concluded that the most efficient scenario

implied a change in the ESSnet spending plan, and was thus validated with the in Eurostat before it was launched.

, a coordination meeting between SURS and Insee prepared the DDI mission, and indeed, two weeks later, the mission was a success: both the issues on DDI modeling

integration were addressed. Regarding the modeling, two subjects

the need to have an explicit DDI profile (an XML schema) and associatedorder to verify that a given questionnaire conformed to the modeling guidelines

of SURS in the questionnaire design tool as a front-the questionnaires.

Also, some questions were still pending regarding the pipeline of the publication of

20

source, and we decided in early 2017 to open-other component of the system, particularly the gateway between Pogues and the

Statistical Office of Slovenia (SURS) and retracted because of a lack of

service during a meeting in February 2017, but we et frame.

must be also noted that the QGS was installed by Eurostat on its SOA platform shortly before the

rence in March 2016 with Insee, during (at that time quite rudimentary) were

it successfully in its data hat the technical integration was rather fluid. We

DDI modeling required for , which led Insee to translate the aforementioned modeling guide and

S reported in the following weeks some questions difficulties for modeling their existing questionnaires, most specifically using the right parts of

ing the Ljubljana meeting in December 2016, SURS encountered with the questionnaire

working meeting was needed in order to progress

to Ljubljana was thus planned for February 2017. Unfortunately, due had to be postponed and was rescheduled for mid-

system analysis for the integration of the QGS scenario was to subcontract the

implied a change in the ESSnet spending plan, and was thus validated with the

, a coordination meeting between SURS and Insee prepared the DDI experts mission, and indeed, two weeks later, the mission was a success: both the issues on DDI modeling

modeling, two subjects

associated validation tools in order to verify that a given questionnaire conformed to the modeling guidelines,

-end to the generation

publication of web forms.


Following the Ljubljana workshop, weekly webdo list” defined.

At the beginning of June 2017, SURS demoed the integration ofin the pre-production collection platform, which was a major milestone in this reuse case.following figure sums up the integrated architecture put in place.

After this first achievement, SURS started the analysis of requirements for generating questionnaires using DDI. The June demo was a first step, the goal in a production context.

In the first phases of the re-use help of two tools developed by

• a DDI mapping Excel macro

• a ESTAT SDM to ENO DDI gerepository used by SURS.

It was doubtful that these tools could cope with mind, so it was decided during a web conference would give SURS access to a preto use a Docker-based distribution, which involved rescheduling order to create this Docker based release as soon as

If the Pogues tool could help in some technical challenges, especiallyset up between Insee and SURS

The basis for the re-use requirements analysis was the questionnaire activities of institutes and institutions. The questionnaire consistquestions, using hierarchical row and header codes. After consultation with Ithat the questionnaire had to be redesigned according to the DDI specifications afeatures. The redesign would also improve the questionnaire logic and legibility.


Following the Ljubljana workshop, weekly web conferences were set up to track progress on the

SURS demoed the integration of questionnaires generated by production collection platform, which was a major milestone in this reuse case.

following figure sums up the integrated architecture put in place.

SURS started the analysis of requirements for generating questionnaires using DDI. The June demo was a first step, the goal being to achieve

use project, SURS created the DDI code mainly manually SURS early in the project ENO:

Excel macro,

to ENO DDI generator, ESTAT SDM being the ISO/IEC 11179 metadata repository used by SURS.

It was doubtful that these tools could cope with the more complex questionnairesduring a web conference in the beginning of September

would give SURS access to a pre-release version of the Pogues DDI generator. based distribution, which involved rescheduling the developments

ocker based release as soon as possible.

If the Pogues tool could help in the production of the DD, it was still in beta versionchallenges, especially for the back-office components. Several web conferences were

and SURS in order to solve the problems encountered.

requirements analysis was the questionnaire on Research and development activities of institutes and institutions. The questionnaire consisted in several interlinked table

chical row and header codes. After consultation with Ibe redesigned according to the DDI specifications a

The redesign would also improve the questionnaire logic and legibility.

21

rences were set up to track progress on the “to-

questionnaires generated by the QGS production collection platform, which was a major milestone in this reuse case. The

SURS started the analysis of requirements for generating more complex to achieve the reuse of QGS

mainly manually, then with the

SDM being the ISO/IEC 11179 metadata

questionnaires that SURS had in September 2017 that Insee

DDI generator. The choice was made the developments on Pogues in

still in beta version and came with . Several web conferences were

Research and development several interlinked table-based

nsee, SURS concluded be redesigned according to the DDI specifications and Eno supported

The redesign would also improve the questionnaire logic and legibility. The use of Pogues


proved very easy and straightforward requiring any DDI background.

The analysis also showed that most of the metadata questionnaires with DDI already existedBlaise questionnaires were generated. redesigned in order to support tool for designing and managing questionnaires metadata. Based on the experience with the R&D questionnaire, it was also decided thatreview of the internal questionnaireredesign is presented in the following figure.

At the time of writing, it was expected that the redesign operation would take several months.then, SURS can use Eno for generatiSDM or use Pogues without reusing metadata from SDM module

5 The re-use of the Metadata


As previously indicated, the first ideas for classifications. This was based on service definition for classification dissemination. in early 2016, leading to the invitation of Trygwere compared. There was an agreement on the higher level CSPA service definition but differences on the service specification, essentially because of the underlying model chosen by each organization: Statistics Norway definevocabularies of the semantic web

The MDS also benefited from the U2016. In particular, a couple of web client applications were September 2016, one of them being a the interest of the Linked Data model for the representation of a concordance tables. The IMS project


very easy and straightforward for the redesign of this complex questionnaire,any DDI background.

showed that most of the metadata necessary for describing ready existed in the ESTAT SDM metadata repository frgenerated. SURS concluded that the SDM metadata module should be

a complete DDI specification and the integration of Pogues as a visaging questionnaires metadata. Based on the experience with the R&D

was also decided that this redesign of the SDM module wouldinternal questionnaire design standard. The future architecture

redesign is presented in the following figure.

At the time of writing, it was expected that the redesign operation would take several months.no for generating questionnaires using available questionnaire metadata from

SDM or use Pogues without reusing metadata from SDM module.

the Metadata Dissemination Service

use

As previously indicated, the first ideas for the re-use of the MDS were about concepts and This was based on a first attempt made in 2015 with Statistics Norway

classification dissemination. Insee thus renewed contact with in early 2016, leading to the invitation of Trygve Falch to the Lisbon meeting,were compared. There was an agreement on the higher level CSPA service definition but differences on the service specification, essentially because of the underlying model chosen by each

Norway defined a custom model whereas Insee basedweb.

benefited from the UNECE's Implementing ModernStats Standards , a couple of web client applications were developed during a sprint in Rome in

September 2016, one of them being a statistical classification explorer that gives a good illustration of the interest of the Linked Data model for the representation of a set of related classifications and

project was also the occasion to develop a REST API on top of the

22

omplex questionnaire, without

describing such types of in the ESTAT SDM metadata repository from which the

SDM metadata module should be integration of Pogues as a visual

aging questionnaires metadata. Based on the experience with the R&D would be started after the

architecture envisaged after the

At the time of writing, it was expected that the redesign operation would take several months. Until e questionnaire metadata from

were about concepts and made in 2015 with Statistics Norway at a common

renewed contact with Statistics Norway , where both approaches

were compared. There was an agreement on the higher level CSPA service definition but differences on the service specification, essentially because of the underlying model chosen by each

a custom model whereas Insee based it solution on

tats Standards (IMS) project in during a sprint in Rome in

gives a good illustration of set of related classifications and

was also the occasion to develop a REST API on top of the


standard SPARQL interface, with a first RAML specification.

Links were also made with the Package 3 workshop in Luxembourg

All these actions were very beneficialCSPA service for the dissemination of classifications that is now a good candidate for returned out that they would not be organization role shifted from the ONS to the CASDrefocused accordingly on a different type of metadata: those about the statistical operations and quality reporting.

Insee publishes these metadata about the operations on the legacy metadata repository that predates RMéS, and their migrationthe beginning of 2018. This is a complex and timetranslation from a simple text-oriented model to a much more detailed model based on the SIMS European standard16. Moreover, the base format is also changed, since the metadata on statistical operations is stored in the RDF part of RMéS (see an OWL ontology (or more precisely the SDMX metadata model, which is at the base of SIMS, was converted to an OWL, see (Cotton, 2017)

In order to accommodate the needs of the CASD, Imodel and the procedures for the conversion from the legacy system to the new metadata repository. A first version of the metadata set was made available at the beginning of 2017, and enriched after that by continuous improvement during the whole year.

5.2 Service re-use by the CASD

The French Center for Secure Data Access (CASD) researchers to remotely and securely work with highlythe interface between the Research community and the Official Statistics organization. It manages a web site where are listed all the statistical products availablestatistical sources and broader statistical programs.not connected to the RMéS metadata on statistical operations, although it is essentially identical, at least for the statistical sources coming from Insee

The CASD saw a great interest in reput in place by Insee. Firstly, it would allow an important reduction of the internal resources dedicated to the management of tcoherence of the meta-information produced by Insee and the CASD, to the benefit of the users. This kind of re-use was not possible with the Insee's legacy system, since the data was not available through direct query or web service, and it was very difficult to use whereas RDF is natively machine-actionable.

15 See https://www.insee.fr/en/metadonnees/sources16 Available at http://ec.europa.eu/eurostat/web/quality/quality17 The CASD also gives access to datasets from other producers, but Insee is the m


SPARQL interface, with a first implementation release in November 2016, supported by a

DIGICOM project. A presentation of the service was ackage 3 workshop in Luxembourg in November 2016.

beneficial for the progress of the RMéS system, and they resulted in a mination of classifications that is now a good candidate for re

turned out that they would not be useful in the SCFE re-use case, since the main reed from the ONS to the CASD as explained in the introduction. The reon a different type of metadata: those about the statistical operations and

Insee publishes these metadata about the operations on its web site15. Internally, they were stored in the legacy metadata repository that predates RMéS, and their migration to RMéS was scheduled at the beginning of 2018. This is a complex and time-consuming operation, because it implies a

oriented model to a much more detailed model based on the SIMS Moreover, the base format is also changed, since the metadata on statistical

operations is stored in the RDF part of RMéS (see 2.3.3). To this purpose, the SIMS was (or more precisely the SDMX metadata model, which is at the base of SIMS, was

(Cotton, 2017)).

In order to accommodate the needs of the CASD, Insee prioritized the development of the previous model and the procedures for the conversion from the legacy system to the new metadata repository. A first version of the metadata set was made available at the beginning of 2017, and enriched after

continuous improvement during the whole year.

by the CASD

Data Access (CASD) manages an equipment designed to allow researchers to remotely and securely work with highly-detailed microdata. In this role, the CASDthe interface between the Research community and the Official Statistics organization. It manages a

where are listed all the statistical products available for the researches, organized by statistical sources and broader statistical programs. This information was managed internally, and not connected to the RMéS metadata on statistical operations, although it is essentially identical, at

ources coming from Insee17.

The CASD saw a great interest in re-using the metadata on statistical operations provided by the QGS put in place by Insee. Firstly, it would allow an important reduction of the internal resources dedicated to the management of the information. Secondly, it would ensure a much greater

information produced by Insee and the CASD, to the benefit of the users. This use was not possible with the Insee's legacy system, since the data was not available

rough direct query or web service, and it was very difficult to use whereas RDF is natively

.fr/en/metadonnees/sources

http://ec.europa.eu/eurostat/web/quality/quality-reporting The CASD also gives access to datasets from other producers, but Insee is the main provider

23

implementation release in November 2016, supported by a

entation of the service was made in a Work

the RMéS system, and they resulted in a mination of classifications that is now a good candidate for re-use, but it

use case, since the main re-using as explained in the introduction. The re-use case

on a different type of metadata: those about the statistical operations and their

. Internally, they were stored in to RMéS was scheduled at

consuming operation, because it implies a oriented model to a much more detailed model based on the SIMS

Moreover, the base format is also changed, since the metadata on statistical To this purpose, the SIMS was converted to

(or more precisely the SDMX metadata model, which is at the base of SIMS, was

nsee prioritized the development of the previous model and the procedures for the conversion from the legacy system to the new metadata repository. A first version of the metadata set was made available at the beginning of 2017, and enriched after

manages an equipment designed to allow In this role, the CASD is at

the interface between the Research community and the Official Statistics organization. It manages a e researches, organized by

This information was managed internally, and not connected to the RMéS metadata on statistical operations, although it is essentially identical, at

ations provided by the QGS put in place by Insee. Firstly, it would allow an important reduction of the internal resources

he information. Secondly, it would ensure a much greater information produced by Insee and the CASD, to the benefit of the users. This

use was not possible with the Insee's legacy system, since the data was not available rough direct query or web service, and it was very difficult to use whereas RDF is natively

ain provider


In order to launch the DO/RO collaboration, abetween Insee and the CASD, where the latter presentinformation model, named CIMES (for Centralizing and Integrating Metadata from European Statistics). It became quickly clear that the most important task for the reCIMES model and the RMéS model. A second meeting took place in technical architecture, the process designdifficulties were expected at the technical integration level.

An important amount of work was carried out models alignment. A first agreement was reached by the end of February presentation at the Vilnius meeting

• A mapping between the cl

• An agreement on the attribute lists of each object class

• An agreement on the code lists

• An agreement on a commoncategorization for each object.

In the meantime, progress was

• Insee started the development of the software dedicated to the integration of the existing metadata in an instance of MDS,

• a version of the service was deployed on ththe features of the services and its

• the re-use architecture was finalized (see figure


In order to launch the DO/RO collaboration, a first joint meeting was held in December 2016 between Insee and the CASD, where the latter presented their re-use case and thei

CIMES (for Centralizing and Integrating Metadata from European . It became quickly clear that the most important task for the re-use would be to align the

model. A second meeting took place in January 2017, the process design and the data flow. That meeting confirmed that no great

difficulties were expected at the technical integration level.

work was carried out in the following months of collaboration agreement was reached by the end of February and improved until its

presentation at the Vilnius meeting. The alignment included:

A mapping between the classes of the two models (see figure below)

An agreement on the attribute lists of each object class

code lists for the coded attributes

An agreement on a common thematic category list for the high level objects and on the actual ization for each object.

In the meantime, progress was also made on the technical side:

Insee started the development of the software dedicated to the integration of the existing metadata in an instance of MDS,

a version of the service was deployed on the SCFE testing environment in order to present atures of the services and its API.

use architecture was finalized (see figure below)

24

first joint meeting was held in December 2016 use case and their existing

CIMES (for Centralizing and Integrating Metadata from European use would be to align the

January 2017, with a focus on the . That meeting confirmed that no great

in the following months of collaboration regarding the and improved until its

for the high level objects and on the actual

Insee started the development of the software dedicated to the integration of the existing

testing environment in order to present


Once the common model was fixed, mprecise list of statistical operations and the refinement of the use case, including the final user interface. A first version of thislater included in the new version of the CASD web site launched during the

At the time of writing, Insee and the CASD are ironing out the last differences in the operation listexpect to put the service in production during the spring of 2018. constitutes a very strong base to take on (ministerial services, archivistsprocess redesign that will run on a much larger scale in the next months.

6 Conclusion and lessons lea

After two years of work, all the consortium partners are happy that the goals assigned to Work Package 3 were reached. The three services are reand all are durably maintained by their developing organiz

In this last section, we proposeproject.

6.1 Aim at reusable services from the start

It was anticipated that the selected services would need some extra effort to be properly reusaThe two main concerns were the internationalization of both code and documentations and the publication to open-source platformsthe re-using organizations. What was less anticipated wasQGS, translating code and documentation took more time and consumed more resources than anticipated.

These costs can be drastically reduced if the services are directly developed with international rein mind. This is what Insee did with the Pogues questionnaire design tool, which was from the very start open-sourced, internationalized, and developed and documented in English. When it became clear that SURS needed Pogues for the creation of more complex queto provide the tool.


Once the common model was fixed, more meetings and exchanges were needed to iterate on thetatistical operations and the refinement of the use case, including the final user

UI was shown at the Wiesbaden workshop in Julylater included in the new version of the CASD web site launched during the fall of 2017.

At the time of writing, Insee and the CASD are ironing out the last differences in the operation listexpect to put the service in production during the spring of 2018. The common model and list constitutes a very strong base to take on board other actors in the French statistical system (ministerial services, archivists, statistical authority), so we believe that we have in fact started a process redesign that will run on a much larger scale in the next months.

and lessons learned

After two years of work, all the consortium partners are happy that the goals assigned to Work Package 3 were reached. The three services are re-used in or near production, all are open source and all are durably maintained by their developing organization.

In this last section, we propose, in no particular order, some of the lessons that we

Aim at reusable services from the start

It was anticipated that the selected services would need some extra effort to be properly reusaThe two main concerns were the internationalization of both code and documentations and the

platforms, that we saw as a sine qua non condition to establish trust with What was less anticipated was the cost of those tasks: particularly for

QGS, translating code and documentation took more time and consumed more resources than

These costs can be drastically reduced if the services are directly developed with international reis what Insee did with the Pogues questionnaire design tool, which was from the very

internationalized, and developed and documented in English. When it became clear that SURS needed Pogues for the creation of more complex questionnaires, it was very simple

25

ore meetings and exchanges were needed to iterate on the tatistical operations and the refinement of the use case, including the final user

at the Wiesbaden workshop in July 2017, and it was fall of 2017.

At the time of writing, Insee and the CASD are ironing out the last differences in the operation list. We The common model and list

board other actors in the French statistical system , statistical authority), so we believe that we have in fact started a

After two years of work, all the consortium partners are happy that the goals assigned to Work used in or near production, all are open source

that we learned during the

It was anticipated that the selected services would need some extra effort to be properly reusable. The two main concerns were the internationalization of both code and documentations and the

condition to establish trust with the cost of those tasks: particularly for the

QGS, translating code and documentation took more time and consumed more resources than

These costs can be drastically reduced if the services are directly developed with international re-use is what Insee did with the Pogues questionnaire design tool, which was from the very

internationalized, and developed and documented in English. When it became stionnaires, it was very simple


6.2 Re-use is a collaboration

The naive idea that re-using a service wrong. In the three cases that were implemented for this project, the operatioengagement from the developing organization, in particular to describe the service, explain how it works, give advice on the architecture, support the reevolutions to fulfill a specific needbetween the developing and the reimplies shared project chartering and monitoring.

6.3 Re-use improves the quality of the services

Making services re-usable, especially in an international context, requires that the code and documentation are exposed widely and requirement than for internal services, which can be less well documented or

A second point is that services which are made for re(or any specific) IT infrastructure. This encouragesspecificities are abstracted, which make services more flexiblor evolutions. This of course is also beneficial to the developing organization itself.

Thirdly, when a service is re-used by a different organization, in a different context and maybe for different use cases, it generates additional feedback, suggestions or bug discovery. Recontribute, for example translate the service in a new language or develop new features. The animation of this community-like collaboration is probably an important factor for the succshared services vision.

6.4 Re-use is never just a technical problem

Initially, we anticipated that there would be a technical cost of integrating Organization's IT system. That cost of course existsthat it is actually a small part of the total cost of re

If we consider the levels of interoperability defined in 2017), we found out that re-using servicesalso semantic, organizational and

The impact at the semantic level is the most obvious. It was particularly strong in the case of the MDS, where the whole business model had to be agconsumed actually most of the time and resources spent on the renoticeable resources at the semantic level for the requite precise model: the DDI data collection module (see below).

The organizational impact has been observed in the TSS case. It appeared during the project that the Bundesbank was an important actor in the process, since it was providing methodological parameters for the seasonal adjustment procedures.This complicated a bit the re-implementation, but led to a clarification of the processesgenerally, business services are not only code plus runtime process 18 It also validated a posteriori the reorganization of the input schema for better alignment to GSIM, with a clear distinction between process inputs and process input parameters (see


use is a collaboration

using a service involves essentially work from the re-using organization is were implemented for this project, the operatio

m the developing organization, in particular to describe the service, explain how it works, give advice on the architecture, support the re-implementers, even occasionally make some evolutions to fulfill a specific need. Re-using a service should be seen as a collaborative project

and the re-using organizations, and should be managed as suchimplies shared project chartering and monitoring.

use improves the quality of the services

usable, especially in an international context, requires that the code and are exposed widely and can be understood by anyone. This is a much stronger

requirement than for internal services, which can be less well documented or

A second point is that services which are made for re-use must have less adherences with the local (or any specific) IT infrastructure. This encourages development patterns where technical specificities are abstracted, which make services more flexible and adaptable to new environments

This of course is also beneficial to the developing organization itself.

used by a different organization, in a different context and maybe for es additional feedback, suggestions or bug discovery. Re

contribute, for example translate the service in a new language or develop new features. The like collaboration is probably an important factor for the succ

use is never just a technical problem

Initially, we anticipated that there would be a technical cost of integrating a service in . That cost of course exists, although variable across

it is actually a small part of the total cost of re-use.

If we consider the levels of interoperability defined in the European Interoperability framework using services can have impacts at all the levels: technical

semantic, organizational and even legal.

antic level is the most obvious. It was particularly strong in the case of the MDS, where the whole business model had to be agreed-upon between the DO and the RO, and that consumed actually most of the time and resources spent on the re-use case. But we alsonoticeable resources at the semantic level for the re-use of the QGS, even if there was in this case a

model: the DDI data collection module (see below).

has been observed to various extents in the three rein the TSS case. It appeared during the project that the Bundesbank was an important actor in the

s, since it was providing methodological parameters for the seasonal adjustment procedures.implementation, but led to a clarification of the processes

, business services are not only code plus runtime processes, they also come with

the reorganization of the input schema for better alignment to GSIM, with a clear distinction between process inputs and process input parameters (see 2.1.1.3)

26

using organization is were implemented for this project, the operation required also strong

m the developing organization, in particular to describe the service, explain how it implementers, even occasionally make some

using a service should be seen as a collaborative project using organizations, and should be managed as such, which

usable, especially in an international context, requires that the code and is a much stronger

requirement than for internal services, which can be less well documented or coded.

use must have less adherences with the local development patterns where technical

e and adaptable to new environments This of course is also beneficial to the developing organization itself.

used by a different organization, in a different context and maybe for es additional feedback, suggestions or bug discovery. Re-users can also

contribute, for example translate the service in a new language or develop new features. The like collaboration is probably an important factor for the success of the

service in a Re-using ROs, but we learned

the European Interoperability framework (EIF, can have impacts at all the levels: technical of course, but

antic level is the most obvious. It was particularly strong in the case of the upon between the DO and the RO, and that

use case. But we also spent use of the QGS, even if there was in this case a

in the three re-use cases, especially in the TSS case. It appeared during the project that the Bundesbank was an important actor in the

s, since it was providing methodological parameters for the seasonal adjustment procedures. implementation, but led to a clarification of the processes18. More

es, they also come with, or

the reorganization of the input schema for better alignment to GSIM, with a clear distinction


modify, existing organizational processethat are impacted by the capabilities of the service, but also supporting services that should adapt to the re-using project.

Finally, the legal level also impacted the TSS rewithin the Destatis infrastructure, rather than using a shared instance, was motivated by legal (actually: constitutional) provisions that request that conf

6.5 Standards are key for

There have been a lot of activities those last years on the definition of standards and their importance in the Official Statistics business. We refer to instardardisation and more generally has been produced in various U

Those activities have fostered the use of standards in business services. As described earlier in this document, the Questionnaire Generation Service is an example of a service driven by the power of a standard like DDI.

We discovered during this ESSncondition: indeed, we often need to go further than knowing a particular standard and must define precise application profiles. This is for example our major conclusSURS: the business domains and applications covered by DDI are quite standard is quite detailed, in particular for questionnaire modeling, it is still necessary to make implementation choices on which object or attribute to use when and how.

This is especially important when the reIn this case, it becomes necessary to develop or use an adaptator.questionnaire metadata repository that could be used as an input for Eno, but this repository was simply ISO/IEC 11179 when EnoSURS ISO/IEC 11179 metadata into DDI.was presented at the Wiesbaden workshop and drew a lot of attention.

Ideally, the implementation choices schema languages19. At the minimum, they should be documented extensively.profiles should be considered as companion standards and agreed

As a counterpoint, we should mention that the TSS reuse case was very fluid on this matter because the seasonal adjustment field is today wto the coherent framework provided by the ESS.

6.6 Good packaging facilitates re

Even if the technical cost of integrating a service in Reusing Organization IT systems isn't the costlpart of the process, it still exists, and a good packaging helps to reduce this cost.

19 For example, XML Schema or Schematron for X


existing organizational processes. Those processes can both be business domain processes that are impacted by the capabilities of the service, but also supporting services that should adapt to

the legal level also impacted the TSS re-use, since the choice of a re-implementing the service within the Destatis infrastructure, rather than using a shared instance, was motivated by legal (actually: constitutional) provisions that request that confidential data stay under German custody.

re-use, but sometimes not sufficient

There have been a lot of activities those last years on the definition of standards and their tatistics business. We refer to initiatives like the ESSNet on

d more generally to the ESS Standardization bodies and processeshas been produced in various UNECE activities (GSBPM, GSIM, CSPA-LIM, etc.

Those activities have fostered the use of standards in business services. As described earlier in this document, the Questionnaire Generation Service is an example of a service driven by the power of a

net that the use of such a standard is a necessary but not sufficient condition: indeed, we often need to go further than knowing a particular standard and must define

. This is for example our major conclusion from the reuseSURS: the business domains and applications covered by DDI are quite wide, and though the standard is quite detailed, in particular for questionnaire modeling, it is still necessary to make

ich object or attribute to use when and how.

This is especially important when the re-using organization cannot immediately adopt the standard.it becomes necessary to develop or use an adaptator. For example, SURS already had a

metadata repository that could be used as an input for Eno, but this repository was no needs DDI. A first step was to develop an adap

ISO/IEC 11179 metadata into DDI. That was an illustration of the concept was presented at the Wiesbaden workshop and drew a lot of attention.

choices for standards would be formally described in appropriate At the minimum, they should be documented extensively.

profiles should be considered as companion standards and agreed-upon at the ESS or Unece level.

As a counterpoint, we should mention that the TSS reuse case was very fluid on this matter because the seasonal adjustment field is today well defined and its methodology less subject to debate thanks to the coherent framework provided by the ESS.

Good packaging facilitates re-implementation

Even if the technical cost of integrating a service in Reusing Organization IT systems isn't the costlpart of the process, it still exists, and a good packaging helps to reduce this cost.

example, XML Schema or Schematron for XML standards, SHACL or ShEx for RDF models

27

business domain processes that are impacted by the capabilities of the service, but also supporting services that should adapt to

implementing the service within the Destatis infrastructure, rather than using a shared instance, was motivated by legal

idential data stay under German custody.

There have been a lot of activities those last years on the definition of standards and their ESSNet on

bodies and processes, but also what , etc.).

Those activities have fostered the use of standards in business services. As described earlier in this document, the Questionnaire Generation Service is an example of a service driven by the power of a

et that the use of such a standard is a necessary but not sufficient condition: indeed, we often need to go further than knowing a particular standard and must define

ion from the reuse case of QGS by , and though the DDI

standard is quite detailed, in particular for questionnaire modeling, it is still necessary to make

using organization cannot immediately adopt the standard. For example, SURS already had a

metadata repository that could be used as an input for Eno, but this repository was an adaptor to transform

That was an illustration of the concept of CSPA Adaptor that

would be formally described in appropriate At the minimum, they should be documented extensively. In all cases, these

upon at the ESS or Unece level.

As a counterpoint, we should mention that the TSS reuse case was very fluid on this matter because ell defined and its methodology less subject to debate thanks

Even if the technical cost of integrating a service in Reusing Organization IT systems isn't the costly part of the process, it still exists, and a good packaging helps to reduce this cost.

ML standards, SHACL or ShEx for RDF models


Deployment standards differ from one NSI to another, so it is way that is easily deployable everycontainerization technologies like Docker

As an illustration, when SURS decided to use the Pogues questionnaire metadata, they chose to use thhas greatly facilitated the deployment.

Docker images can easily be made available on GitHub and directly accessed from there. For examplethe whole chain of services that constitute the QGS demo platform (Pogues, Eno and associated support services) is available as a set of installed in one command line f

6.7 Open source is the way to go

“Be open” was one of the three principles found that it really gave us important guidance in the fulfillment of our objectivesnot always the easiest path to take, tsolutions, and a special effort was made and wildocumented software.

WP5 has in the meantime helped finding ESS, and provided guidance and recommendations for producing good openextensive discussion on this subject within the consortium, it is very clear in our views that open source is an excellent solution for improving the quality of the services and for strengthening trust between developing and re-using organization

The European Commission has source, but we think ESS should find the right development and support.

20 See https://www.docker.com/ 21 See https://github.com/InseeFr/Pogues


from one NSI to another, so it is not easy to packy that is easily deployable everywhere. A good option, though, is to leverage recent

containerization technologies like Docker20.

hen SURS decided to use the Pogues application in order questionnaire metadata, they chose to use the dockerized version of Pogues, a

deployment.

Docker images can easily be made available on GitHub and directly accessed from there. For examplethe whole chain of services that constitute the QGS demo platform (Pogues, Eno and associated

is available as a set of eight Docker images and a composition scriptinstalled in one command line from any remote location.

Open source is the way to go

one of the three principles put forward by the consortium in its applicatifound that it really gave us important guidance in the fulfillment of our objectivesnot always the easiest path to take, the three candidate services are now based on opensolutions, and a special effort was made and will continue to be made for producing openly

in the meantime helped finding interesting open-source solutions that can be usedguidance and recommendations for producing good open-source softwa

extensive discussion on this subject within the consortium, it is very clear in our views that open is an excellent solution for improving the quality of the services and for strengthening trust

using organizations.

has produced strategic plans mostly addressing the e, but we think ESS should find the right open-source economics model to foster OSS

m/InseeFr/Pogues-Back-Office/tree/master/docker

28

easy to package an application in a s to leverage recent

to generate DDI e dockerized version of Pogues, and indeed, this solution

Docker images can easily be made available on GitHub and directly accessed from there. For example, the whole chain of services that constitute the QGS demo platform (Pogues, Eno and associated

eight Docker images and a composition script21. All can be

put forward by the consortium in its application, and we found that it really gave us important guidance in the fulfillment of our objectives. Although it was

based on open-source l continue to be made for producing openly available,

that can be used in the source software. After

extensive discussion on this subject within the consortium, it is very clear in our views that open is an excellent solution for improving the quality of the services and for strengthening trust

mostly addressing the demand side of open-to foster OSS


7 Bibliography

Cotton, F. (2017). SDMX Metadata

Récupéré sur https://linked-statistics.github.io/SDMX

EIF. (2017). European Interoperability Framework.

Sigaud, É., Tailhurat, R., Cotton, F., & van der Vlist, É. (2013). example. Retrieved from Balisage Series on Markup Technologies, vol. 11: https://www.balisage.net/Proceedings/vol11/html/Cotton01/BalisageVol11


SDMX Metadata - An RDF vocabulary for representing the SDMX metadata model.

statistics.github.io/SDMX-Metadata/sdmx-metadata.html

European Interoperability Framework. Récupéré sur ISA2: https://ec.europa.eu/isa2/eif

rat, R., Cotton, F., & van der Vlist, É. (2013). XForms generation: a real

Retrieved from Balisage Series on Markup Technologies, vol. 11: https://www.balisage.net/Proceedings/vol11/html/Cotton01/BalisageVol11

29

RDF vocabulary for representing the SDMX metadata model. metadata.html

Récupéré sur ISA2: https://ec.europa.eu/isa2/eif

XForms generation: a real-world

https://www.balisage.net/Proceedings/vol11/html/Cotton01/BalisageVol11-Cotton01.html

essnet scfe deliverable d3-1 - ec.europa.eu · essnet scfe report on the implement ation of three...

Documents