observations & prÉvisions cÔtiÈres synthesis of the data management activities (na) s. iona...

OBSERVATIONS

& PRÉVISIONS CÔTIÈRES

www.seadatanet.org

Synthesis of the data management activities (NA)

S. Iona (HCMR), R. Lowry (BODC), F. Nast (BSH),

D. Schaap (MARIS)

OBSERVATIONS


23rd Plenary Annual Meeting, Madrid, 25-27 March 2009

Overview• Training and Capacity building

• Metadata Services in V1

• System status and data integration in V1- EDMED

- CSR

- EDIOS

- EDMERP

- EDMO

- CDI

• Summary

OBSERVATIONS



Training and Capacity buildingAims:

• to ensure that the data and metadata to be integrated in the system will be formatted, checked for quality and disseminated according to the common protocols developed during the project

• to transfer expertise and to train IT experts of the SeaDataNet data centers in the basics, installation and operation of the SeaDataNet technical components

OBSERVATIONS



Training and Capacity building

Training Workshops, at IOC Rroject Office for IODE,

Oostende, (Belgium):(Training material and presentations available on SDN Extranet)

1. February 12-17 2007:

• focused on data re-formatting, quality control, and processing using ODV software.

• training on project metadata and related web services.

OBSERVATIONS



Training and Capacity building

Training Workshops, IOC Rroject Office for IODE,


2. June 4-5 2007:

• dedicated on generating XML records with the use of MIKADO tool

OBSERVATIONS



Training and Capacity buildingTraining Workshops, IOC Rroject Office for IODE,


3. June 16-19 2008:

• use of the new V1 formats, interfaces and maintenance tools (MIKADO, online CMS, Web services, Validation services, Vocabularies)

• data quality control and assessments, using ODV software

• analysis and data presentations, using ODV – DIVA software

OBSERVATIONS



Metadata Services

The objectives are to:

• Maintain and expand the national metadata-bases.

• Standardize the information using common vocabularies (as Web Services)

• Interconnect the national inventories in a common Pan-European directory.

One of the SDN Online Services to facilitate the marine data

searching, location and retrieval.

OBSERVATIONS



Metadata Services StructureIt is composed of several thematic inventories of different levels:

• High Level Directories – EDMED, CSR, EDIOS (for describing observational datasets collected by ships, by laboratories, by continuous observing systems)

• Common References - EDMERP, EDMO and Common Vocabularies (available as Web Services) that interconnect the directories

• The Common Data Index (CDI) that gives access to the data distributed by the TA platforms (NODCs, Satellite Centers)

OBSERVATIONS



System Maintenance – V0

Version 0: 2006-2007

• Continuation and maintenance of past Sea-Search system :

the data access needed several different requests to each data centre

and the data sets were delivered in different formats

OBSERVATIONS



System Upgrade to V1

Version 1: 2008-2010

• Setup of the integrated online data service to users :

networking the distributed data centres

unique request to the interconnected data centres

and the data sets are delivered with a unique format

OBSERVATIONS



Maintenance modalities and governance • Maintenance and content upgrade from Version 0 to Version 1 by using on-line

(like CMS forms) and off-line tools (MIKADO) that produce XML ISO 19115 compliant exchanges and developed by the Technical Task Team in the joint research activities JRA1, JRA2:

Directory + pan-European coordinator

Format info Online CMS

XML exchange from national to pan-European level

XML validation service (Managed by RNODC and BODC http:/ / www.seadatanet.org/ validator/ )

EDMED (coordinator BODC)

Check EDMED format

No Mikado Online

EDMERP (coordinator MARI S)

Check EDMERP format

Online Mikado Online

CSR (coordinator BSH)

Check CSR format

Online Mikado Online

EDI OS (coordinator BODC)

Under development

No Under development

Under development

EDMO (coordinator MARI S

Online No Not applicable

CDI (coordinator MARI S)

Check CDI format

No Mikado Online

Reformatting Tools now available:

- Med2MedSDN- NEMO

Generation of coupling table for DM, mapping download:

- NEMO- Mikado V1.6

Training hour tomorrow

OBSERVATIONS



System status and data integration in V1

European Directory of Marine Environmental Datasets (EDMED)

Cruise Summary Reports (CSR)

European Directory of the initial Ocean-observing Systems (EDIOS)

European Directory of Marine Environmental Research Projects (EDMERP)

European Directory of Marine Organizations (EDMO)

Common Data Index (CDI)

OBSERVATIONS




• V0 to V1 content changes

- Content developed to be ISO19115-compliant through a profiling exercise

• Major change was that fields such as parameters were converted from plaintext to vocabulary term lists that could be implemented as ISO19115 keywords

• Some fields needed for mandatory ISO19115 core were added

• A small number of fields rendered redundant by plaintext replacement were dropped

OBSERVATIONS




• Content conversion plan

- A CMS has been provided by Maris

(http://www.seadatanet.maris2.nl/vu_edmed/welcome.asp)

- Partners upload V0 records into the CMS

- CMS template guides conversion to V1

- V1 records exported as XML and ingested into BODC database

OBSERVATIONS



V1 Infrastructure Status

• Infrastructure Work Completed

- V1 Oracle storage schema

- V1 XML ingestion software

- V1 Web Service export

• Infrastructure Work To Do

- Upgrade presentation schema to V1

- Storage to presentation schema translator

- V1 User Interface

OBSERVATIONS



V1 Content Conversion Status

• BODC

- 703 (out of 3357!) V0 records are the responsibility of BODC

• Initial conversion completed by junior member of staff

• List of issues to be resolved by Lesley

- Status of other partners’ progress checked on 2009-03-23

OBSERVATIONS




0

50

100

150

200

250

300

350

400

Gre

ece

ICE

S

Sp

ain

Be

lgiu

m

Ice

lan

d

No

rwa

y

Sw

ed

en

Po

lan

d

Lith

ua

nia

Ru

ssia

Ge

org

ia

Tu

rke

y

Mo

rocc

o

La

tvia

Cro

atia

Cyp

rus

Est

on

ia

Ukr

ain

e

Isra

el

Slo

ven

ia

Not OK

OK

Countries with entries completed or nearing completion

OBSERVATIONS



0

50

100

150

200

250

Por

tuga

l

Fin

land

JRC

Rus

sia

Bul

garia

Rom

ania

Mal

ta

Alb

ania

Alg

eria

Leba

non

Not OK


Countries with entries apparently not converted

OBSERVATIONS




SUMMARY• Records upgraded (as of 23 March): 1448 (45%)

• Countries supplying V1 records in XML (i.e. will not be converted in CMS): France, Germany, Ireland, Italy, Netherlands

• CLS (France) will contribute with V1 records in XML

• Romania, Portugal: large collections with no upgrade or submission of XML files

• Remaining records to be upgraded: ~430 (excluding BODC’s 700)

• Upgrade needs to be done very soon, assistance is available

OBSERVATIONS










OBSERVATIONS



EDMERP content upgrade

• Using CMS for upgrading V0 to V1 and also for adding new records

• Using MIKADO and XML import

[available as webservice for XML V1 files (export and import)]

OBSERVATIONS



EDMERP Web Interface - Retrieval

http://seadatanet.maris2.nl/v_edmerp/search.asp

Browse list

Additional details

OBSERVATIONS



EDMERP CMShttp://seadatanet.maris2.nl/vu_edmerp/welcome.asp

OBSERVATIONS



STATUS OF EDMERP UPGRADING FROM V0 TO V1 LEVEL, 17-Mar-2009

- 95% of the upgrade is done- the rest should finish the next months

collate_id country total EDMERP records # ok # not_ok # not_active

43 United Kingdom 247 222 25 2196 Germany 40 40 0 0

120 Italy (OGS) 66 49 17 4193 Poland 35 35 0 0269 Greece 94 94 0 2353 Spain 98 94 4 15396 Ireland 118 117 1 6431 Belgium 134 134 0 6486 France 118 115 3 8545 Sweden 21 21 0 4575 Netherlands 88 79 9 4583 Iceland 6 6 0 13590 Portugal 28 3 25 4612 Norway 57 54 3 3628 Finland 24 24 0 3642 Italy (JRC) 2 0 2 1680 Lithuania 13 11 2 0681 Russian Federation (RNODC) 91 91 0 0685 Russian Federation (IORAS) 12 11 1 1691 Morocco 6 6 0 7692 Bulgaria 23 23 0 0693 Georgia 106 106 0 3696 Turkey 58 56 2 1697 Romania 23 22 1 4698 Latvia 28 28 0 1700 Croatia 16 16 0 3708 Malta 29 29 0 2710 Israel 4 4 0 0711 Cyprus 5 5 0 8713 Estonia 7 7 0 2727 Ukraine 81 81 0 0729 Denmark (NERI) 49 49 0 3730 Denmark (ICES) 3 3 0 0

1149 Belgium 3 0 3 01228 Albania 2 0 2 01229 Slovenia 10 8 2 01230 Lebanon 13 13 0 01231 Algeria 4 4 0 01232 Tunisia 5 3 2 0

TOTAL 1767 1663 104 129

OBSERVATIONS










OBSERVATIONS



CSR Status

~ 40 000 CSRs (1873 -2009)

17 partners submitting new entries in V1

in an Oracle DB

16 entries from 2009 (Skandinavia)

OBSERVATIONS



V1 Maintenance

• New entries (and updates)

- Online for individual entries (CMS)

- XML files for bulk submission with the use of MIKADO tool

OBSERVATIONS



CSR Content upgrade to V1

- Online

• contact BSH-DOD requesting CSRs to be updated

• BSH-DOD loads requested CSRs to entry database

• modify CSRs in entry database

- Offline

• contact BSH-DOD requesting CSRs to be updated

• BSH-DOD sends requested CSRs as XML V0 files (free text entries)

• Use of MIKADO tool and submit modified CSRs in XML V1 format to BSH-DOD

OBSERVATIONS



Status of Upgrading to V1• 7 Partners upgrading systematically using online application

• 17 partners submit new online entries in V1

• 5 partners submit new entries and upgrades in XML V1 • Slovenia (NIB), Italy (OGS), UK (BODC), Turkey (IMS-METU) and

France (IFREMER)

• altogether around 4800 completed CSRs in V1 format

• only 10% is in V1 !!!

• Remark: many project information which were originally in the V0 format get

lost due to missing EDMERP entries

OBSERVATIONS



big new entries and upgrades of XML V1 submissions by 5 partners: France, UK, Italy, Turkey, Slovenia

operational new online entries in V1 by 17 partners

Countrynew entries since

last annual meetingCSRs in V1 /

upgrade in process comments

AlbaniaAlgeriaBelgium Bulgaria 2 2 ONLINE (new)Croatia 21Cyprus 8 8 ONLINE (new)Denmark 5 4 ONLINE (new)Denmark (Faroe Islands) 20 15 ONLINE (new)Estonia Finland 2 1 ONLINE (new)France 1377 1377 XML V1 (new and upgrade)Georgia 14 14 ONLINE (new)Germany 158 1668 ONLINE (new and upgrade)Greece 23 ONLINE (new)Iceland 2Ireland 59 103 ONLINE(new and upgrade)Israel Italy 107 107 XML V1 (new and upgrade)Latvia 48 ONLINE (upgrade)LebanonLithuania 5 5 ONLINE (new)Malta Morocco Netherlands 3 entry by BSHNorway 68 166 ONLINE (new and upgrade)Poland 1 entry by BSHPortugal Romania Russian Federation 47 ONLINE (new and upgrade)Slovenia 64 64 XML V1 (new)Spain 34 337 ONLINE (new and upgrade)Sweden 31 104 ONLINE (new and upgrade)Tunisia 1 ONLINE (new)Turkey 17 82 XML V1 (new and upgrade)Ukraine 16 ONLINE + XML V0 (new)United Kingdom 625 625 XML V1 (new and upgrade)

Total 2663 4777

total ~40800 CSRs

OBSERVATIONS



ISSUE only part (10%) of the total

CSRs are now upgraded, mainly from the last decade

not all partners started upgrading

not all partners started using new online entries in V1

ACTION

Suggestion Begin with the latest data and

go backwards

Partners get export to update those CSRs either online or by Mikado

Partners get a reminder, may need help?

OBSERVATIONS



New features in CSR User Interface

• Platform link to EurOcean RV Infobase

• CSR Online Entry - uploading of track charts, station lists and additional reports directly into database

• Organisation link to EDMO record

• XML download in V1 format for upgraded CSRs (being tested)

OBSERVATIONS



CSR V1 Retrieval

http://seadata.bsh.de/csr/retrieve/V1_index.html

EurOcean RV Infobase link

EDMO link

Track chart

OBSERVATIONS



CSR V1 CMS

http://seadata.bsh.de/csr/online/V1_index.html

Upload station list

Upload reports

Upload track charts

OBSERVATIONS



CSR V1 Retrieval

http://seadata.bsh.de/csr/retrieve/V1_index.html

OBSERVATIONS



Planned Actions Interconnecting Meta-catalogues:

Link from CSR to EDMERP

Link from EDMERP to CSR

to CDI ?

Printable form for CSR

Planning CSR Web Service for use together with MIKADO

Use of POGO ship schedule information for preparing initial CSRs

OBSERVATIONS










OBSERVATIONS



Status• No new input has been requested over the last year

• Improved Oracle (database) schema

• Supported by common vocabularies, EDMO and EDMERP

• ISO 19115 content model:

• Description of the format and XML tags available

• XML example file available

• Programme and Series XML Schema available

• Upgrade is underway

• MIKADO V1.6: for new entries and updates

• New user Interface available

OBSERVATIONS



New User Interface: http://seadatanet.maris2.nl/v_edios/search.asp

OBSERVATIONS










OBSERVATIONS



Maintenance

• Update of existing entries via CMS forms

• Access for NODC‘s

• New records only to be created via MARIS (because EDMO = vocabulary)

• Available as webservice (export only)

OBSERVATIONS



European Directory of Marine Organizations

• Currently, EDMO lists and describes more than 1.500 organizations.

• There is direct crosslink with EDMED - data sets, EDMERP - research projects, CSR - research cruises, EDIOS - observing stations/ systems, and CDI V0 and V1 - index to data.

• Content upgrade from V0 to V1 is finished

OBSERVATIONS



EDMO V1 search and retrievalhttp://seadatanet.maris2.nl/edmo

OBSERVATIONS



EDMO CMShttp://seadatanet.maris2.nl/vu_organisations/welcome.asp

EDMO CMS geo-locator via Google maps

OBSERVATIONS










OBSERVATIONS



Common Data Index: gate to data access

• CDI V0 System:

- More than 340.000 CDI entries

- Data requests through different requests at data centres

- Data delivered at different formats

• CDI V1 System:

- Unique request

- order and access for downloading via a shopping basket mechanism and a communication between the SeaDataNet portal and the local systems

- Data delivery in unique format

OBSERVATIONS



CDI V1 – Quick Search http://seadatanet.maris2.nl/v_cdi_v1/browse_step.asp

OBSERVATIONS



CDI V1 – Extended Search

http://seadatanet.maris2.nl/v_cdi_v1/search.asp

OBSERVATIONS



Integration of data centres in CDI V1CDI partnername Country

XML files ready in import system (awaiting DM testing)

CDI V1 XML testfiles checked and imported in test system DM/Interim DM version Test DM/Interim ok? # CDI files in production Date last files/contact remarks

In progress:NERI DK ok DM V1.1c ok 126046 20090217OGS IT ok DM V1.1c ok 77138 20090227 duplicates with enea removedRNODC RU ok DM V1.1d ok 56256 20090310RWS NL ok interim ok 14109 20090224 interim procedure explained and activeMSI EE ok DM V1.1c ok 11306 20090303 metadata updatedKNMI NL ok DM V1.1c ok 5090 20081106NIOZ NL ok DM V1.1c ok 4605 20081124ENEA IT ok DM V1.1c ok 7184 20090320MHI UA ok DM V1.1c ok 85009 20090320 awaiting final production files, small metadata problem

PUT AL tested, not ok ? 20080916 only testfiles received and checkedCYODC CY ok DM V1.1c no ok 20090317 Order process seemed ok, Database connection in progressIEO ES 15471 ok DM V1.1c no ok 20090218 DM in test, CDI XML ready, latest version in progressCLS FR 818 ok DM V1.1c no ok 20090323 Working on DM installation, metadata already in import.IFREMER FR ok DM V1.1c no ok 20090316 Waiting for latest version DM, being installed now. Awaiting final test.BSH GE ok DM V1.1c no ok 20090310 testfiles loaded, DM in developmentHNODC GR 39841 ok DM V1.1d no ok 20090323 Metadata loaded in import, DM installation latest version in progressIOLR IL ok DM V1.1c no ok 20090311 DM installed, few problemsMRI IS testfiles checked, not ok DM V1.1c no ok 20081218 Testfiles checked, no news about DM progressTNO NL 10436 ok DM (own version) no ok 20090305 order process ok, small problems to be solvedNIOO NL 10554 ok DM V1.1c ok 20090320 awaiting production filesIHPT PT ok ? 20090227 multiple times testfiles received, but no answers about what the actual datasets are, or about DM installationIORAS RU ok DM V1.1c no ok 20081014 testfiles loaded, no progress reported sinceNIB SL ok ? 20090223 Testfiles loaded, no progress reported about DM installationBODC UK 53259 testfiles checked, not in test system interim no ok 20090310 large batch of files received, loaded in import database. Small problem to be solved regarding coordinates of lines and areas

Total 130379 386743

Still to come:MUMM BEIOBAS BGISMAL DZFIMR FITSU GEIOF HRMI IENCMS LBCMR LTLIAE LVINRM MAIOI MTNMD NOIMGW PLNIMRD ROSMHI SEINSTM TNIMS-METU TR

• 9 partners fully connected, 386.743 CDI files loaded in the production environment

• 15 partners waiting the DM/interim tests, 130.379 CDI files loaded in the import system

• 18 partners still to come, assistance is available

OBSERVATIONS



SummaryThe system upgrade from V0 to V1 is making good progress, but needs to be completed very soon and especially CDI (considering that the EMODNET chemical pilot is starting) :

DirectoryStatus of upgrade

% ACTIONS

EDMED 1448/3357 45% Acceleration of the work is needed

CSR 4777/40800 10% Acceleration of the work is needed

EDIOS

EDMERP 1663/1767 95% Very good progress

EDMO 1607/1607 100% Completed

CDI386743 in the production system

Good progress

observations & prÉvisions cÔtiÈres synthesis of the data management activities (na) s. iona...

Documents