EMODnet Chemistry 2

Service Contract MARE/2012/10 S12.656742

Technical developments of the EMODNet Chemistry portal

ByDick M.A. Schaap – Technical Coordinator

Split – Croatia, 17 – 18 June 2014, 1st Plenary Progress Meeting

Contract requirements for the portal

gather all measurements of a particular chemical species with their appropriate metadata within a given space and time windowvisualise the measurement density in a given time and space windowvisualise a time evolution of a selected group of measurementsshow concentration plots for a given time and space window and also along the coastcalculate spatially distributed data products specifically relevant for selected Marine Strategy Framework Directive Descriptorsinclude the physical conditions under which the measurements were made (from EMODnet physical parameters portal or the GMES marine core service)show inflows from rivers of nutrients. The user should be able to select a section of coast, a country or a region (NUTS3) and obtain time series of inflows of parameters expressed as mass or moles per unit time per river (or section of coast)

EMODNet Chemistry portal

The EMODNet Chemistry portal sits on top of the SeaDataNet infrastructure and makes use of its services that have been adapted and are further developed for specific EMODNet Chemistry needs. It provides various services and functionalities to users for browsing and viewing the chemistry data products and for identifying and requesting access to the gathered chemistry data sets for the European waters.The primary services are:

CDI Data Discovery and Access Service giving facilities for searching and retrieving chemistry data sets; OceanBrowser Viewing Service giving facilities for viewing, browsing and downloading Chemistry data products; Sextant Products metadata catalogue giving facilities for searching Chemistry data products and linking to the viewing service.

CDI service for discovery and unified access of data

Already 96 data centres connected and more underway

CDI Data Discovery and Access Service

Based up ISO19115 content standard and ISO 19139 XML Schema, fully INSPIRE compliantDedicated tools and facilities for generating CDI entries, format associated data sets and populating the CDI serviceShopping basket mechanism for discovery, access request, and downloading of data sets from distributed data centres Downloading in harmonised SeaDataNet formats: SDN ODV ASCII, and soonSDN NetCDF (CF) Adopted in many projects and ongoing improvements Operational governance scheme

Chemistry CDI Data Discovery and Access Service

Dedicated CDI service for EMODNet Chemistry scope Three search interfaces for human users:

Quick Search with dynamic drilling down of search results; Extended Search with more flexibility for combining search options, including free search;Variables Vs Marine Regions with an interactive Matrix of variables in specific marine regions.

Chemistry CDI Service upgrading

ISO 19115 content standard and XML Schema have been migrated to INSPIRE compliant ISO 19139

Extra fields included:EDMED referencesCSR referencesPublication referencesQuality Info referencesVocabularies NVS 2.0 upgradingSea regions (C19) references via geo-tagging

User interface also upgraded to support new elements and extra search options

Shopping basket extended from 500 to 10.000 CDI’s requests

Geo-tagging CDI’s with Sea regions (C19)

C19 – polygons for marking up all CDI entries with C19 terms using spatial queries + extra tags via C19 BT relations

Sea regions vocabulary (C19 as hierarchical service)

C19

Chemistry CDI Service upgrading – search interface

Extra P02

Extra Duration

Extra Sea Region

Extra Multiple

Chemistry CDI Service upgrading – details

EDMED references

Sea regionreferences

Chemistry CDI Service upgrading – details

CSR references

EDMOreferences

EDMERPreferences

Chemistry CDI Service upgrading – details

Publicationreferences

Quality inforeferences

Chemistry CDI Service – filter

EMODNet Chemistry 2 project has an extended scope of chemical substances:

Pesticidides and biocides Antifoulants PharmaceuticalsHeavy metalsHydrocarbonsRadionuclidesFertilisers Organic matter(e.g. from sewers or mariculture)ChlorophyllPartial pressures of dissolved gasesAcidity (from pH, pCO2, Total Inorganic Carbon, alkalinity)Others

Mapping between new EMODNet Chemistry 2 groups and P02 Vocabulary doneSets filter for virtual subset of SeaDataNet CDI to EMODNet Chemistry CDI collection

Chemistry CDI Service – present coverage (16 June 2014)

661095 CDI records and data sets

European waters (N80 W-30; N20 E45)587538 CDI records and data sets 62 Data Centres31 Countries248 Originators1868 – 2014 years82% unrestricted18% to be negotiated

Chemistry CDI Service – WMS / WFS services

The CDI locations and related metadata can be shared with other portals by means of OGC WMS (Web Mapping Services) and WFS (Web Feature Services)

http://geoservice.maris2.nl/wms/seadatanet/cdi_v2/emodnet/chemistryGetcapabilities

http://geoservice.maris2.nl/wms/seadatanet/cdi_v2/emodnet/chemistry?service=WMS&request=GetCapabilitiesExample WMS request:

http://geoservice.maris2.nl/wms/seadatanet/cdi_v2/emodnet/chemistry?styles=&format=image/png&transparent=true&request=getmap&version=1.1.1&srs=EPSG:4326&layers=lines&width=580&height=290&bbox=-180,-90,180,90Getcapabilites indicates what is available. Implementing WFS is depending on the client and needs programming. Example of WFS request:

http://geoservice.maris2.nl/wfs/seadatanet/cdi_v2/emodnet/chemistry?service=WFS&version=1.0.0&request=getfeature&outputformat=gml3&typename=points&maxfeatures=10&bbox=-0.2793103448275872,57.4448275862069,1.5206896551724127,59.244827586206895 

Further developing controlled vocabularies for EMODNet Chemistry

As part of SeaDataNet controlled vocabularies (NVS 2.0) are maintained and served by NERC BODC as web services for marking up all metadata and data entries. At present more than 160 vocabulary lists are served with more than 150.000 concepts and with established and active gouvernance

The parameter usage vocabulary list P01 is used for data sets while the parameter discovery vocabulary P02 is used for the CDI metadata. P01 are narrower terms of P02. At present P01 already contains more than 30.000 concepts.

Therefore activities are undertaken in EMODNet Chemistry for:New entries because of the extended scope of substancesMaking mapping to P01 easier and more efficient by exposing the semantic model behind P01 and making it retrievable by componentsGrouping various P01 terms under an aggegrated term in a new vocabulary P35 that will facilitate data aggegration and product labelling

Further developing controlled vocabularies for EMODNet Chemistry

Example of P01 term: ‘Concentration of tributyltin cation {tributylstannyl TBT+ CAS 36643-28-4} per unit dry weight of biota {Mytilus galloprovincialis (ITIS: 79456: WoRMS 140481) [Subcomponent: flesh]}’

Semantic model:

Discovery and harvesting of data sets for regional product groups

EMODNet Chemistry 2 project is generating chemistry products per sea region (as defined by MSFD (draft))

This requires discovery and harvesting of data sets per MSFD region for specific chemical parameters

In EMODNet Chemistry 1 this was done on a manual basis using the CDI Discovery and Shopping mechanism

In EMODNet Chemistry 2 this was done initially in a semi-automatic way and progress is made towards an almost full automatic method:

using a Robot harvester via the shopping mechanism to discover and retrieve specific data sets from distributed data centres to compile and maintain specific aggregate data sets as internal central data buffers that can be transferred to regional groups for further processing and products generation.

Discovery and harvesting of data sets for regional product groups – 1st year

Filter set to discover nutrients data and to MSFD regions (approx.)

Boundaries of regions approximated by VLIZ and then schematised by MARISwith extra margins as GEO-filter

Discovery and harvesting of data sets for regional product groups – 1st year

Robot harvester has gathered circa 440.000 CDI records and data sets for nutrients in the given regions

These sets were transferred to the EMODNet regional groups per region

An online Buffer Content Management System (Buffer CMS) has been developed and tested for configuring specific data buffer profiles in agreement (SLA’s) with data providers AND for specific data user communities (such as EMODNet Chemistry regional product groups, MyOcean, SeaDataNet regional dataproduct groups, …)

Configuration settings concern discovery filter, buffer group, motivation, users (by means of SeaDataNet AAA services)

Robot harvester can be activated to perform retrieval for each buffer profile and also to maintain the central metadata and data buffers automatically for new entries and updates of existing entries

Progress of the robot harvesters is administered in the existing online Request Status Manager (RSM) system which is part of the CDI Shopping mechanism for tracking and tracing requests by users, data providers and overall

Discovery and harvesting of data sets for regional product groups – now almost automatic

Buffer Content Management System (Buffer CMS)

logon

Overview of buffersConfiguring profile of specific buffer

Buffer CMS + Central User Interfacing

CDI User Interfaces

CDI Robot harvester

Specific data buffers

CDI User Interface + Shopper:

Access regulated via AAA

Agreed Settings

Dynamic Maintenance

Robot CMS to configure Robot harvesting profiles

MARIS master

RSM systemextended with

administering robot transactions and via

central interfaces

API

Central User Interface with logon (AAA service) following authorisation in buffer CMS profiles

logon

Overview of authorized buffersCentral buffer UI incl direct shopping

The new Buffer CMS and Central buffer UI and API (under development for full machine-to-machine interaction) together with the new central shopping mechanism and upgraded RSM will greatly facilitate the maintenance of central buffers and regular delivery of data sets incl metadata to the Chemistry regional groups

The Central shopping mechanism works on the data Buffers and can deliver (in delayed mode) large data sets which are divided over downloadable zip files with maximum 10.000 data sets each; all shopping transactions are administered in new section of the RSM

REMARK: The central buffers are exclusive for specific applications and acces is secured via AAA service only for authorised users. These buffers do not replace the distributed CDI infrastructure and its shopping process for regular users.

Discovery and harvesting of data sets for regional product groups

Request Status Manager (RSM) service extended with administering of Central buffer interfaces

Logon as user/provider/master

New functions for central buffer shopping

However the central buffers will contain and deliver ‘raw’ data sets for specific parameters and as harvested from the distributed data centres => further action is needed for making the collection more homogeneous and validated => aggregated data sets

Aggregation and validation for generating homogeneous data collections can be done by using ODV software and specific expertise per region and chemical substance

Use will also be made of the new P35 Vocabulary for aggregating P01 terms. The P35 population is making progress:http://seadatanet.maris2.nl/v_bodc_vocab_v2/welcome.asp

The final buffers of aggregated and validated data sets will provide the input for data products and advanced visualisation services

Converting buffer data sets to validated aggregated data sets

EMODnet Chemistry extension

Specific data buffers

Oceanotron &OpenEarth services

ODV QC + aggregation

QCd data buffers

Regional experts

DIVA + OceanBrowser service

CDI RobotHarvester

Advanced services

The validated and aggregated buffers of data sets will be input for the products and advanced services:

Interpolated maps as produced with DIVA software (Ulg)

Time series graphics for selected stations via WPS (Deltares)

OceanBrowser viewer (Ulg) as common service for viewing the DIVA maps and giving access to the time series graphics

In a later stage also using Oceantron server (IFREMER) in between ODV binary files and OceanBrowser for additional visualisation services of all data sets

OceanBrowser Viewing Service (Ulg)

The viewer provides access to two kinds of products:DIVA interpolated maps. Output images available as horizontal sections and vertical sections. The latter can be selected by drawing an appropriate transect; (via OpenDAP, NetCDF, ncWMS); inclusion of predefined coastal sections Time series plots of selected stations via WMS of station locations and WPS (Deltares) for graphics on-the-flyIntegration with CDI service via WMS – WFSIntegration with SEXTANT Product Catalogue .

Sextant Products metadata catalogue (IFREMER)

Used to describe the Chemistry data products such as DIVA maps. This facilitates searching for specific data products and the exchange and use of the Chemistry data products in other services, such as the Chemistry OceanBrowser, and other portals with OGC WMS support. Metadata format: ISO19115 - ISO19139 with SDN Controlled Vocabs; CSW service based upon GeoNetWork

Planned developments and extensions

Single-Sign-On (SSO): only for requesting data using CDI service; uses CAS; work ongoing for supporting also OpenID and ShibbolethSearch:

the CDI search facility with be expanded with MSFD sea regions (once officially established) and possibly user drawn geographical polygon OceanBrowser menu will integrate the Sextant catalogue to facilitate searching of specific products for viewing

Visualisation:Further developing the time series visualisation services and their structured integration in system architecture Further analysis of use of Oceantron service on top of ODV binary files towards OceanBrowser for visualisation of data fields (first in frame of SeaDataNet 2)

Exploring cloud hosting and computing

Exploring options for hosting of central aggregrated buffer data sets and applications in a cloud hosting and computing environment

Prepared a background paper and Terms of Reference for cloud hosting of the validated buffer data sets and the advanced services for visualisation

Cloud hosting as neutral environment and highly performing

Dialogue with Cineca about options and possible way forward. Cineca is a non profit Consortium, made up of 69 Italian universities, and 3 Institutions, including OGS and CNR. It is the largest Italian computing centre.

Cineca is also partner in EUDAT, an FP7 project towards a pan-European Collaborative Data Infrastructure which will allow researchers to share data within and between communities and enable them to carry out their research effectively.

We will keep partners informed about progress.

www.emodnet-chemistry.eu