brokering ig session - earthobservations.org€¦ · sba 1 sba 2 sba 8 enterpris e system k ... (f...
TRANSCRIPT
GEO DAB (Discovery and Access Broker)
Stefano Nativi on behalf of the ESSI-lab team
(Dept. of Florence of CNR-IIA)
GEOSS Applications
GEOSS Providers
GEOSS Application Developers
(intermediate Users)
GEOSSend-Users
GEOSS ApplicationsGEOSS ApplicationsGEOSS Applications
Enterprise System j
… .
Enterprise System 1
System 4Enterprise
System 3
Enterprise System 2
… .
… .
… .
SBA 1 SBA 2
SBA 8
Enterprise System
K
Enterprise System 3
System 4
Enterprise System
1
Enterprise System 2 Enterprise
System Z
Enterprise System 1
System 4Enterprise
System 2
Enterprise System 3
GEOSS Portal
DOWNSTREAM
UPSTREAM
MIDSTREAMGEOSS Common Infrastructure
APIs
Mediation modules
Yellow
Page
GEOSS Common Platform (GCI)
Some Key Numbers:155 Brokered data providers;
Monitoring and reporting• Periodical reporting
Some Key Numbers:About 45 million datasets;
More than 200 million granules
From Client-Server to Client-Broker-Server
• Appliance of the Brokering pattern and introduction of a
Brokering services tier
• Provide discoverability, access and transformation capabilities
GEO DAB capabilities
• A (service) Interoperability Broker
• Connect, mediate and harmonize hundreds of heterogeneous data/information systems/services
• Transform data and metadata content where useful
• Maps the diverse data and metadata models onto its owninternal model
– general enough to comprise the diverse concepts
• The internal data and metadata model is flexible and extensible to allow new concepts and related attributesaddition
GEO DAB realizes an abstract and
harmonized view of the diverse
data/metadata implementations
GEO DAB TECHNOLOGY
GEO DAB
• Main Services
– Discovery Broker service
– Access Broker service
– Semantic extension Broker service
– Transformation services
• Main Artifacts
– Common Data Model
– GEO APIs (including conceptual and implementation models)
– Accessors (upstream mediators)
– Profilers (downstream facades)
– Community and User defined Views
Common Data model: Context
Resource
Result set
Query&View
Service
O&M
Dataset
BPDocument
Metadata
Semantics
OGCCSW2.0.2APISO1.0 INPE
OGCCSW2.0.2ebRIMEO CKAN
OGCCSW2.0.2ebRIMCIM DCAT
ESRIGEOPORTAL10 GI-cat
OAI-PMH2.0 ESRIGEOPORTAL10
OpenSearch1.1 NCML-OD
OpenSearch1.1ESIP BCODMO
OpenSearchGENESIDR NCML-CF
CKAN NetCDF-CF1.4
CUAHSIHIS-Central FTPpopulatedwithsupportedmetadatatypesESRIRESTAPI10.3 WAFWebAccessibleFolders
OGCWCS GeoNetwork (2.2.0orgreater)
OGCWMS EcologicalMarkupLanguage2.1.1
OGCWFS1.0.0,1.1.0,2.0.0 NERRS(NationalEstuarineResearchReserveSystem)
OGCWMTS HMACSW2.0.2ebRIM/CIM
OGCSOS1.0.0,2.0.0,2.0.0HydroProfile HDF
OGCWPS1.0.0 IADCDB(MySQL)
OGCCSW2.0.0Core GrADS-DS
OGCCSW2.0.2APISO1.0 FedEO
OGCCSW2.0.2ebRIM/EOAP ARPADB(basedonMicrosoftSQL)
OGCCSW2.0.2ebRIM/CIMAP ESRIMapServer
IRISStation SHAPEfiles(FTP)
IRISEvent KISTERSWeb- EnvironmentofCanada
HYRAXTHREDDSSERVER1.9 EnvironmentCanadaHydrometricdata(FTP)
OAI-PMH2.0- Harvesting OpenSearch1.1
GBIF EarthEngine
DIF RASAQM
HYDRO EGASKRO
UNAVCO SITAD(SistemaInformativoTerritorialeAmbientaleDiffuso)
CDI1.04,1.3,1.4 FileSystem
ISO19115-2 GDACS
THREDDS1.0.1,1.0.2 GeoRSS 2.0
THREDDS-NCISO1.0.1,1.0.2 Degreecatalogservice2.2
THREDDS-NCISO-PLUS1.0.1,1.0.2 OpenSearchGENESIDR
Accessors and Profilers
Cloud-based SaaS
(IaaS and PaaS)
Computing
Storage
Monitoring
Auto Scaling
Load Balancing
Routing
NoSQL Database Clustering
MOST RECENT ADVANCEMENTS
Improving Discovery results
• GEO DAB and GEOSS Portal implement
ViewsRanking and Paging
Result set Filtering
Ranking and Paging
No-SQL DB
Good performances on
large stores
No preliminary constraint
on data structure
Need to preliminarily
index queryable elements GEO DAB Internal
Metadata Model
Pre-calculated in batch,
based on:
Metadata Quality
Accessibility
Etc.
Calculated on-the-fly,
based on:
Query Constraints
Applied to scores
(configurable)
Present Filters used by GEOSS
– Organization
– Keywords
– Format
– Source
– Protocol
• Any other queryable element can be used to filter
result set
GEOSS View
• Definition:
– Subset of the whole GEOSS resources defined by applying
(via the DAB) a set of clauses
• Discovery clauses (e.g. spatial envelope, keywords,
sources, etc.)
• Access clauses (e.g. data format, access protocol, CRS,
etc.)
• Defined “View” exposed on the GEOSS Portal
Consumer-defined View – i.e. Client-side available only for the client application that defined the
view.
Provider-defined View –i.e. Server-side available for all client applications (e.g. a Community)
GEO DAB ACCESS AND USE
GEO DAB APIs
• A set of high-level APIs (Application Program
Interfaces) have been designed and developed
• Documentation and usage examples are available
A set of standard Web service interfaces:
• e.g. OGC service interfaces, CKAN, OAI-PMH, FTP, etc.
A set of APIs for software developers:
• Client side APIs:
– (high-level) JavaScript library
– … . (Python)
• Server side APIs:
– REST/JSON APIs
– OpenSearch APIs
– … .
API conceptual model (ontology)
JavaScript
implementation
(Client-side APIs)
REST implementation
(Server-side APIs)
OGC WxS
(Web Service
Interfaces)
Implementation Schemas
Yearly requests managed by GEO DAB
over the past 3 years
410.967
2.556.138
4.480582
0
500,000
1,000,000
1,500,000
2,000,000
2,500,000
3,000,000
3,500,000
4,000,000
4,500,000
5,000,000
GEOSS and the GCI Evolution: Software Ecosystems
(GEOSS Evolve Initiative)
• GEO-related
– GEO Regional Data Hubs scope
– GCI
– GEO Flagships,
– GEO Community Portals
• External
– Data Systems
– Thematic/International federation systems
– Datacube & Satellite Data Analytics systems
– Other Applications
Publications
Mattia Santoro, Stefano Nativi, Paolo Mazzetti, Contributing to the GEO Model Web implementation: A brokering service for business processes, Environmental Modelling & Software, Volume 84, October 2016, Pages 18-34
Stefano Nativi, John Caron, Ben Domenico, Lorenzo Bigagli, 2008, Unidata’s Common Data Model mapping to the ISO 19123 Data Model, Earth Science Informatics, September 2008, Volume 1, Issue 2, pp 59-78
Nativi, S., Domenico, B., Caron, J., Davis, E., and Bigagli, L.: Extending THREDDS middleware to serve OGC community, Adv. Geosci., 8, 57-62, doi:10.5194/adgeo-8-57-2006, 2006.
Stefano Nativi, Paolo Mazzetti, Gary N. Geller, Environmental model access and interoperability: The GEO Model Web initiative, Environmental Modelling & Software, Volume 39, January 2013, Pages 214-228.
Lucy Bastin, Dan Cornford, Richard Jones, Gerard B.M. Heuvelink, Edzer Pebesma, Christoph Stasch, Stefano Nativi, Paolo Mazzetti, Matthew Williams, Managing uncertainty in integrated environmentalmodelling: The UncertWeb framework, Environmental Modelling & Software, Volume 39, January 2013, Pages 116-134.
B. Domenico, S. Nativi, CF-netCDF3 Data Model Extension Standard, Open Geospatial Consortium Inc.. document, 2012
Stefano Nativi, Paolo Mazzetti, Mattia Santoro, Fabrizio Papeschi, Max Craglia, Osamu Ochiai, Big Data challenges in building the Global Earth Observation System of Systems, Environmental Modelling & Software, Volume 68, June 2015, Pages 1-26.
Giuliani, G., Nativi, S., Lehmann, A., Ray, N., WPS mediation: An approach to process geospatial data on different computing backends. Computers & Geosciences, vol. 47, Oct 2012, pages 20-33.
Stefano nativi and Andrew woolf, 2009, An holistic view of coverage model and services for SISE-SEIS, proceedings of European conference of the Czech Presidency of the Council of the EU TOWARDS eENVIRONMENT Opportunities of SEIS and SISE: Integrating Environmental Knowledge in Europe
