workshop on research metadata in context – sept 7/8, nijmegen, netherlands

EUFAR - EUropean Facility for Airborne Research

Workshop on Research Metadata in Context –

Sept 7/8, Nijmegen, Netherlands

Metadata Frameworks in EUFARMatt Freer, EUFAR

Overview

EUFAR Intro.

Airborne Metadata• Challenges

• Schemes

EUFAR Conv.• External

• Embedded

Summary

Overview

EUFAR Introduction

Airborne Science Metadata Overview

EUFAR Implementation

Summary

EUFAR - EUropean Facility for Airborne Research

EUFARIntegrating Activity of the EC FP7

Budget 8 M€ Duration 4 years (2008-2012) 33 Partners

7 instruments and 22 aircraft open to Trans-national Access

www.eufar.net

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

EUFAR Introduction

• Jülich (DE)

• ULEI (DE)

• STFC (UK)

• USZ (HU)

• UCAM-DCHEM (UK)

• UWAR (PL)

• COSINE (NL)

• IRSN (FR)

• COMAT (FR)

• VKI (BE)

• UZH (CH)

• WU (NL)

• USBE (CZ)

• TAU (IL)

• GFZ (DE)

• PML (UK)

• PTB (DE)

• ONERA (FR)

• Météo-France (FR) Coordinator

• MetOffice (UK)

• DLR (DE)

• NLR (NL)

• Enviscope (DE)

• CNRS (FR)

• NERC (UK)

• INTA (ES)

• FUB (DE)

• KIT (DE)

• AWI (DE)

• CNR (IT)

• UNIMAN (UK)

• VITO (BE)

14 aircraft or instruments operatorsand

18 experts in airborne measurements

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

EUFAR Introduction

EUFAR objectives:

Provide scientists with access at equal terms to the most complete range of research infrastructuresDevelop trans-national access to national infrastructuresReduce redundancy and fill the gapsPromote the use of research infrastructures, particularly among young scientists from countries where such infrastructures are lackingStrengthen expertise through exchange of knowledge, development of standards and protocols, construction of databases and joint instrumental research activities

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

Airborne Metadata Overview

Aircraft in EUFAR measure large range of atmospheric data:• Aircraft parameters

– Altitude/Lat/Long– Roll/Pitch/Yaw

• State parameters– Temperature– Pressure– Humidity

• Chemistry– CO2– O3– NOx

• Microphysics– Water content– Ice crystal habits – Particle concentration– Particle size distributions

• Radiation– Long wave– Short wave

• Winds• Hyperspectral imaging

– Land use– Reflectivity

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary


Challenges:Heterogeneity:

• Instruments• Measurements• Data systems• Processing techniques• Data streams

Collaborations on projects, research and data production between varied countries and institutions (Europe, USA, etc)Many different levels of data (raw, intermediate, processed, final, etc)

All of these necessitate clearly defined, robust protocols for describing data

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary


External metadataGives users broad overview of data

• Location• Project• Responsible party• Quality

Aids in data classification, search, retrieval, archival

Embedded metadataMakes files self-describingCan define metadata globally or per-variableReduces errors in usageHelps automate data processing tasksStores logistical information (calibration, maintenance and other historical sensor information)

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

EUFAR Metadata Conventions

Draft version of proposed conventions has been created and circulated to members for commentAt the International Conference on Airborne Research in the Environment (ICARE) in Toulouse, Oct 2010, version 1.0 of these standards will be finalizedEUFAR proposes:

External• INSPIRE

Embedded • NetCDF• CF Conventions• IWGADTS

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

External Metadata – INSPIRE Directive

Infrastructure for spatial information in Europe (INSPIRE)Directive entered into force May 2007 by the European Commission

Ensures compatibility and usability of spatial data infrastructures with the adoption of common implementing rules (e.g. Metadata)Implementation rules available at http://inspire.jrc.ec.europa.eu

INSPIRE PrinciplesData should be collected only once and kept where it can be maintained most effectivelySpatial information should be combined into seamless productData should be scalable to multiple levels – detailed enough for investigation, general enough for strategic purposesGeographic information should be readily available at all levelsAvailable data, metadata and usage conditions should be clear and easy to find

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

INSPIRE Metadata elements

INSPIRE Metadata regulation mainly based on ISO 19155-2Includes Information on

Data set or seriesClassification (scientific field)Geographic locationTemporal referenceQuality and validityConformityConstraints related to access and useThe responsible organisation

Descriptions of different image and instrument processing steps

List has been expanded within EUFAR by additionalquality indicators

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

INSPIRE Metadata rules implementation into EUFAR (1)

Identification1. Resource title: EUFAR project acronym2. Resource abstract: EUFAR abstract3. Resource type: dataset or series4. Resource locator: www.eufar.net5. Unique resource identifier: EUFAR file name specification6. Coupled resource: not applicable7. Resource language: eng

Additional:List of related available filesPlatform: aircraft all signInstrument: list of instruments incl. basic sensor characteristics (e.g. spectral coverage)

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary


1. Classification• Topic category: EUFAR main scientific field• Spatial data service type: not applicable

2. Keyword• Keyword value: EUFAR scientific theme• Originating controlled vocabulary: GEMET - INSPIRE themes,

version 1.0 2008-06-01

3. Geographic Location• Geographic bounding boxAdditional:

Area information: Country, Province/Region, Local name

4. Temporal Reference• Temporal extent: date of data acquisition • Date of publication: date of entry into EUFAR-DB• Date of last revision: date of last revision• Date of creation: date of data processing

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary


Quality and Validity1. Lineage2. Spatial resolution

Additional:Processing information (e.g. processing level)Calibration information (e.g. date of spectral calibration)Acquisition information (e.g. overall heading)Data quality layers for hyperspectral imagery

Sensor calibration and system correction Image data artefacts and processing errors GPS / IMU related errors, geometric correction Atmospheric correction and atmospheric conditions

quality control flags

hyperspectral image

attitude and position

black body information, …

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary


1. Conformity• Specification: INSPIRE implementing rules• Degree: degree of conformity (true or false)

2. Constraints related to access and use• Conditions applying to access and use: EUFAR consortium

agreement• Limitations on public access: EUFAR consortium agreement

3. Organisation responsible for the establishment, management, maintenance and distribution• Responsible party: [email protected]• Responsible party role: pointOfContact

4. Metadata on metadata• Metadata point of contact: e.g. DLR, [email protected], PAF• Metadata date: date when the metadata record was created

or updated• Metadata language: eng

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

Embedded metadata

Tangent: EUFAR recommended file format NetCDF (network Common Data Form)

Wide use in atmospheric science communityAPIs for most common programming languages (C, C++, Fortran, Java, Python, Matlab, IDL)NetCDF is:

• Binary• Portable• Scalable:

– Multi-dimensional array oriented– Subsets of large datasets easily accessible

AppendableSelf describing

Data model:DimensionsVariablesAttributes

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

Embedded metadata

Tangent: NetCDF (network Common Data Form)NetCDF is:

• Binary• Portable, i.e. machine independent• Scalable, I.e. multi-dimensional array oriented• Appendable• Self describing

Wide use in atmospheric science communityData model:

• Dimensions• Variables• Attributes

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

EUFAR NetCDF Metadata Conventions

EUFAR conventions for NetCDF extend existing, common standards:1. NetCDF Climate and Forecast (CF) Conventions

• Broad conventions, applicable across atmospheric and earth science

1. Interagency Working Group for Airborne Data and Telemetry Systems (IWGADTS) Conventions• US-based group• Tailored for aircraft

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary


Specifies file structureNames for time dimensions

• Time for 1Hz signal• Time25 for 25Hz signal• etc

Order of dimensionsDefinition of time coordinate – follow CF conventions

Defines standard file attributesConventionsTitleInstitutionSourceHistoryReferencesComment

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary


Defines standard variable attributesunitsLong nameStandard nameValid rangeSampled rateCategoryCalibration coefficientsDependenciesProcessor

Provides standard vocabularies based on CF Conventions

standard namesunits

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary


EUFAR NetCDF Metadata Conventions do not:Define or standardize variable names

Why?Unique and varied instruments and measurements between aircraft and institutionsCan be multiple measurements of the same quantity

Example: NCAR C130 Pressure measurements

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

NetCDF metadata rules for EUFAR

Example: NCAR C130 Pressure measurements PSFD

PSFDC

PSFRD

PSFRDC

PCAB

Solution: Data DiscoveryDefine global attribute ‘air_pressure’The value of this attribute indicates recommended fieldExample: reference:air_pressure = PSFDC

– Data discovery helps users and software choose correct measurement

All pressure: Which one to use?}

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

NetCDF metadata rules for EUFAR

• EUFAR standard data discovery fields– Highly recommended

• Latitude• Longitude• Altitude

– Recommended• Wind speed• Upward air velocity• Air temperature• Platform speed• Dew point temperature• Air pressure• Platform roll angle• Platform pitch angle• Platform orientation• Ground speed E/W• Ground speed N/S

Overview

EUFAR Intro.


• Schemes


• Embedded

Summary

Summary

• EUFAR conventions:– Proposal has been circulated– Version 1.0 being released in Oct 2010– External - INSPIRE

• General description• Archival • Search

– Embedded - NetCDF• Data usage• Processing• Accuracy

workshop on research metadata in context – sept 7/8, nijmegen, netherlands

Documents