enes climate analytics service (ecas) - thematic …...2centro euro-mediterraneo sui cambiementi...
TRANSCRIPT
![Page 1: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/1.jpg)
ENES Climate Analytics Service (ECAS)Thematic Service, EOSC-HUB
Sofiane Bendoukha1, Tobias Weigel1, Sandro Fiore2,Alessandro D’Anca2
1Deutsches Klimarechenzentrum (DKRZ), Hamburg, Germany2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy
11. April 2018
EOSC-hub is EU’s Horizon 2020 project under grant agreement 777536
![Page 2: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/2.jpg)
![Page 3: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/3.jpg)
ENES Climate Analytics Service (ECAS)
ENES Climate Analytics Service (ECAS)
I ECAS is a contribution to the EOSC-hub project.I It enables scientific end-users from climate data community
and other disciplines to perform data analysis experiments onlarge volumes of (climate) data.
I Server-side, PID1-enabled and parallel approachI uplift existing computing capabilities of Ophidia framework to
be available as an EOSC serviceI integrate with EOSChub services for ease of useI provide hands-on training to overcome adoption barriers
1Persistent IdentifierBendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 4: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/4.jpg)
ENES Climate Analytics Service (ECAS)
ENES Climate Analytics Service (ECAS)
source: Tobias WeigelBendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 5: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/5.jpg)
ENES Climate Analytics Service (ECAS)
Ophidia: Big Data Analytics FrameworkOphidia is a research effort carried out at the Euro MediterraneanCentre on Climate Change (CMCC) to address big data challenges,issues and requirements for climate change data analytics.
I Server-side, parallel and distributedI ExtensibleI Client application available as user interfaceI Workflow support
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 6: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/6.jpg)
ENES Climate Analytics Service (ECAS)
Ophidia: Server Architecture
Figure: Sandro Fiore
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 7: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/7.jpg)
ENES Climate Analytics Service (ECAS)
Ophidia: Array based Primitives
I Ophidia provides a wide set of array-based primitives (about100) to perform summarization, sub-setting, predicatesevaluation, statistical analysis, etc
I Primitives comes as plugins and are applied on a singledatacube chunk (fragment)
I Primitives can be nested to get more complex functionalitiesI New primitives can be easily integrated as additional plugins
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 8: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/8.jpg)
ENES Climate Analytics Service (ECAS)
Ophidia: Data Cube Operators
source: Sandro Fiore
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 9: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/9.jpg)
ENES Climate Analytics Service (ECAS)
Ophidia client: The Ophidia Terminal
oph_term is a terminal-like commands interpreter serving as aclient for the Ophidia framework.Through the oph_term the user runs (send) commands(operators) to the Ophidia framework to manipulate datasets(datacubes).
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 10: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/10.jpg)
ENES Climate Analytics Service (ECAS)
Ophidia client: The Ophidia Terminal
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 11: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/11.jpg)
ENES Climate Analytics Service (ECAS)
Ophidia client: PyOphidia
Python package to interact with the Ophidia frameworkI Alternative to Oph TermI Runs on Python 2.7, 3.3, 3.4 and 3.5I Requires a running Ophidia instanceI It provides two main modules:
I client.py: generic low level class to submit any type ofrequests (simple tasks and workflows), using SSL and SOAPwith the client ophsubmit.py
I cube.py: high level cube-oriented class to interact directlywith cubes, with several methods wrapping the operators.
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 12: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/12.jpg)
ENES Climate Analytics Service (ECAS)
Ophidia client: PyOphidia
I Instantiate a client and submit a requestfrom PyOphidia import client
ophclient = client.Client("oph-user", "oph-password", "127.0.0.1", "11732")
ophclient.submit("oph_list level=2", display=True)
I Set a Client for the Cube classfrom PyOphidia import cube
cube.Cube.setclient(’oph-user’,’oph-password’,’127.0.0.1’,’11732’)
#Export to NetCDF filecube.Cube.createcontainer(container=’test’,dim=’lat|lon|time’,dim_type=’double|double|double’)
#Show a Cube structure and infomycube1.info()
#Subset a Cubemycube=mycube1.subset(subset_dims=’lat|lon’,subset_filter=’1:10|20:30’,subset_type=’coord’)
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 13: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/13.jpg)
ENES Climate Analytics Service (ECAS)
Workflow Support{
"name": "Example1","author": "Foo","abstract": "Simple workflow with three tasks","exec_mode": "sync","ncores": "1","cube": "http://hostname/1/1","tasks":[
{"name": "Extract maximum value","operator": "oph_reduce","arguments": [ "operation=max" ]
},{
"name": "Extract minimum value","operator": "oph_reduce","arguments": [ "operation=min" ]
},{
"name": "Evaluate max-min range","operator": "oph_intercube","arguments": [ "operation=sub" ],"dependencies":[
{ "task": "Extract maximum value", "type": "single", "argument":"cube" },
{ "task": "Extract minimum value", "type": "single", "argument":"cube2" }
]}
]}
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 14: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/14.jpg)
ENES Climate Analytics Service (ECAS)
Ophidia Use Cases
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 15: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/15.jpg)
ENES Climate Analytics Service (ECAS)
Ophidia Use Cases
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 16: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/16.jpg)
ENES Climate Analytics Service (ECAS)
Ophidia Use Cases
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 17: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/17.jpg)
ENES Climate Analytics Service (ECAS)
ECASLab
A user-friendly scientific data analysis environment.ECASLab consists of:
I Ophidia clusterI JupyterHub instance
I Jupyter notebookI Ophidia client (PyOphidia, Ophidia terminal)
I Set of pre-installed Python libraries for running datamanipulation, analysis and visualization
I Data publication service
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 18: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/18.jpg)
ENES Climate Analytics Service (ECAS)
How to use it?
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 19: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/19.jpg)
ENES Climate Analytics Service (ECAS)
Integration with EUDAT Services
source: https: // eudat. eu/
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 20: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/20.jpg)
ENES Climate Analytics Service (ECAS)
Integration with EUDAT Services
I B2DROP: Through B2DROP, ECAS provides a secure andtrusted data exchange service to store and exchange processeddata outputs
I B2SHARE: ECAS offer users to receive, store and publishprocessed data outputs
I B2HANDLE: Through the B2HANDLE service, ECASassigns Persistent Identifiers (PID) to Ophidia output andrecords kernel information according to a community profile
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549
![Page 21: ENES Climate Analytics Service (ECAS) - Thematic …...2Centro Euro-Mediterraneo sui Cambiementi Climatici (CMCC), Lecce, Italy 11.April2018 EOSC-hub is EU’s Horizon 2020 project](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f789741eda0846f002af436/html5/thumbnails/21.jpg)
ENES Climate Analytics Service (ECAS)
For more informationsI Thursday 16:45, oral presentation on ECASLab.Room M2.
I EOSC-HUB project http://eosc-hub.eu/I EUDAT services www.eudat.eu/servicesI Ophidia http://ophidia.cmcc.it/I Ophidia use cases https://github.com/OphidiaBigData/
ophidia-workflow-catalogueI Get in touch [email protected]
Bendoukha, Weigel, Fiore, D’Anca 11. April 2018 EGU2018-12549