the nerc datagrid

NOCS, PML, STFC, BODC, BADC

The NERC DataGridThe NERC DataGrid

+ ++ + =

Bryan Lawrence

Director of the STFC Centre for Environmental Data Archival

(BADC, NEODC, IPCC-DDC etc)

Representing the NDG and a dose of the “Portals Project”

Royal Society, April, 2008

http://ndg.nerc.ac.uk

British Atmospheric Data Centre

British Oceanographic Data Centre

Simulations

Assimilation

Complexity + Volume + Remote Access = Grid Challenge


Data Sets

• Ground based observation networks Met Office surface stations

• Model output NWP, ECMWF reanalyses & Climate models

• Satellite data TOMS, Envisat & MSG

• NERC programmes data UTLS, CWVC & URGENT

“A collection of files with a common theme and administration”

Datasets statistics• 147 datasets • ~130TB (unique)• ~80 Million files


The NDG Use Cases

1. Find data2. Find out about data3. Subset and difference data4. Visualise data5. All using SECURE interdisciplinary technology.Sound easy doesn’t it?

Want interdisciplinary semantic access to information, not abstract data– getData(potential temperature from ERA-40 dataset in North

Atlantic from 1990 to 2000)– not: getData(“era40.nc”, ‘PTMP’, 20:50, 300:340, 190:200)– or even worse:

for j=1990:2000getData(“era40_”+j+“.nc”, ‘PTMP’, 20:50, 300:340)


The big: Climate in 2010 – A graphic Illustration

Figures from Gary Strand, NCAR, ESG website


… and the little … land use!


ISO19109 in a nutshell


Dataset Metadata


MOVIE(s)


NDG Deployment

Vocab Server:

120,000+ concepts100 lists75,000 Relationships (SKOS)


Dataset Metadata


MOLES: Metadata Objects for Linking Environmental Science

A

data production tool

deployed

at an

observation station

on behalf of an

activity

produces a

dataset.

Datasets have

provenance


Getting down to the data itself

A-Metdata• We choose to deploy an “Application

Schema” of the “Geographic Markup Language”.

• This allows us to define our data to support RE-USE by other communities – interdisciplinarity, and to be INSPIRE compliant!


CSML Dataset

Dictionaries: •coordinate reference systems,• phenomena (parameters)• units of measure

Feature Collections• CSML Features are SPECIALISATIONS of GML Features!

Storage Descriptor• Re-usable description of storage

(NB: CSML Software Stack, WMS, WCS etc)


CSML Feature Types

Abstract

Specific: point data, time-series of point data, trajectory data, point collections, profiles, profile series, Ragged profile series, section, ragged section, scanning radar, grid, gridseries and swath


Beyond measuring things at points

CF concepts:

Cell-Methods

(statistical e.g means etc)

Cell-Bounds

(intensive quantities,

Extensive quantities,

Relationship to coverage domain)

New work:

Exploiting Sensor Web

Enablement


Storage Descriptor


Data Modelling Workshop

• April 2007, at The Cosenors House, Abingdon• Brought NDG, Unidata, OGC, Italian NRC

together to get a shared vision for scientific data types.

• Roadmap for bringing our activities together over next few years.

• Harmonising:– NDG CSML– OGC Observations and Measurements (with CSML

and MOLES)– UNIDATA Community Data Model– INRC NCML-G


Data Model Relationships

Route to harmonising GeoSciML and CSML


Wider InternetNERC Grid

taperobot

XML data-base

XML data-base

BADC NDG Wrapper

OnlineData

OnlineData

BODC NDGWrapper

OnlineData

XML data-base

Group NDGWrapper

Software Agent

Grid User

Satellite Supercomputer

Research Group DataSources

Internet Link

Internet User

Internet LinkESG (&other)Applications

Wider Internet

NDGWeb

Portal

XML data-base

The Original Vision


What we’ve done (are doing)

Wider InternetNERC Grid

taperobot

XML data-base

XML data-base

BADC NDG Wrapper

OnlineData

OnlineData

BODC NDGWrapper

OnlineData

XML data-base

Group NDGWrapper

Software Agent

Grid User

Satellite Supercomputer

Research Group DataSources

Internet Link

Internet User

Internet LinkESG (&other)Applications

Wider Internet

NDGWeb

Portal

XML data-base

Schema: MOLESCSMLDIF+Lots of Population Issues

Protocols: OGC +OAI

Pylons based portalPython

WCS/WMS ClientsCSML Toolbox

Vocab Server

PylonsInterface & portals


Now and the Future

More protocol improvements

(= wider community of providers and users)

More deployment.

More population.

Evolution WITH the web.

the nerc datagrid

Documents