Download - Bde euro proworkshop
BIG DATA EUROPE'S INTEGRATOR PLATFORM
A ONE-STOP SOLUTION FOR BIG AND SMART DATA MANAGEMENT
EuroPro Workshop @ EDBT/ICDT Conference21.03.2017
Supporting the Societal Domains with Big Data Technology
BigDataEurope Project
2 mai 2023www.big-data-europe.eu
BigDataEurope Action EC Horizon 2020 Coordination & Support
Action o ~5mio €, 2015-2017
Show societal value of Big Datao Across all societal challenges addressed by H2020
Lower barrier for using big data technologieso Effort to setup and deploy use-case workflowso Lack of skills & expertise
Help establish data value chains across domains & orgs. 2 mai 2023www.big-data-europe.eu
Consortium
NCSRDEMOKRITOS
Data Value Chain Evolution
2 mai 2023
Extraction, Curation Quality, Linking, Integration
Publication, Visualization, Analysis
Extraction, Curation, Quality, Linking, Integration, Publication,
Visualization, Analysis
HealthTransport
Security
Extraction Curation Quality Linking Integration Publication Visualization Analysis
Data Repositori
es
Linked Open Data
TIME
Food SocietiesClimate EnergyProprietary, ‘locked-in’solutions
OS Solutions,Big Data Stacks
www.big-data-europe.eu
A flexible, generic platform for (Big) Data Value Chain Deployment2. Architecture
Big Data Integrator
2 mai 2023www.big-data-europe.eu
Big Data Integrator: Architecture
Stacks Open Source solutions (Free)
Dockerization Facilitates integration and
deployment Plug-and-play BD Platform Key BDE additions
o Support layer: integrated UIo Semantification layer
Final BDI Release:o (3rd) May 2017] Final
2 mai 2023www.big-data-europe.eu
Big Data Integrator: In-Use
Big Data Integrator:https://github.com/big-data-europe
WIKI : extensive documentation, information on supported components, instructions, etc. 2 mai 2023www.big-data-europe.eu
BDE vs Hadoop distributions
Hortonworks Cloudera MapR Bigtop BDE
File System HDFS HDFS NFS HDFS HDFSInstallation Native Native Native Native lightweight
virtualizationPlug & play components (no rigid schema)
no no no no yes
High Availability Single failure recovery (yarn)
Single failure recovery (yarn)
Self healing, mult. failure rec.
Single failure recovery (yarn)
Multiple Failure recovery
Cost Commercial Commercial Commercial Free Free
Scaling Freemium Freemium Freemium Free FreeAddition of custom components
Not easy No No No Yes
Integration testing yes yes yes yes --Operating systems Linux Linux Linux Linux AllManagement tool Ambari Cloudera
managerMapR Control system
- Docker swarm UI+ Custom
9
Data Acquisition: Apache Flume Data Storage: Hue, Apache Cassandra, ScyllaDB,
Apache Hive, Postgis Search/Indexing: Apache Solr Message Passing: Apache Kafka Data Processing: Spark, Flink Semantic Components: Sansa, Silk, Strabon,
Sextant, GeoTriples, Semagrow, Limes, 4Store, Openlink Virtuoso
BDI Docker Containers (..and counting)
2 mai 2023www.big-data-europe.eu
Semantic Layer
www.big-data-europe.eu
Semantic Data Lakes o Minimal
ingestion pre-processing
o Semantic layer maintains metadata
o Add meaning when retrieving/processing
Data Lake: scalable unstructured data store
Relationship definitions and metadata
JSON-LD CSVW R2RMLXML2RDF
Ongoing Research for Semantic Big Data & Analytics Knowledge
Graphs
Semantic Layer tools
2 mai 2023www.big-data-europe.eu
BDE tooling for Semantic Data Lake:o Swagger: Semantics of RESTful
APIso Semantic Analytics Stack
(SANSA): Distributed data processing over large-scale Knowledge Graphs
o Semagrow: SPARQL over Big Data stores
o Ontario: Querying over Semantic Data Lakes
Demonstrating the Societal Value through 7 Pilot ‘Real-world’ use-cases
1. Overview
BigDataEurope Pilots
2 mai 2023www.big-data-europe.eu
7 Pilots◎ BDI Platform Instantiations
o Allow end-users to easily deploy functionality in own system environment
o Modularized Docker approach - easier to replace components
o Reduces effort to keep 3rd party software updated & integrated
◎ 7 Societal Challenge Pilots o Aligned with 7 European Commision H2020 Societal
Challengeso Real-world use-cases (Data, Objectives, Solutions)o Some pilots have different data & objectives but a similar
solution◎More info:
2 mai 2023www.big-data-europe.eu
www.big-data-europe.eu/pilot/
7 BDI Instances
2 mai 2023www.big-data-europe.eu
Free Workshops, Hangouts & WebinarsBigDataEurope Activities
2 mai 2023www.big-data-europe.eu
3rd round of Societal Workshops
2 mai 2023www.big-data-europe.eu
Health 11/12 May 2017 Malta Collocated with EC eHealth Week
Food&Agri
31 March 2017 Brussels
Co-organized with e-ROSA H2020 project
Energy Autumn 2017 Brussels
(Details to Follow)
Transport
12/13 September 2017
Collocated with Big Data for Transport, Tisa workshop, organised by Ertico
Climate 21/22 September 2017
Brussels
Collocated with iScape Project Workshop – Improving the Smart Control of Air Pollution in Europe by EC JRC Ispra.
Societies
Autumn 2017 T.B.C. (Details to Follow)
Security Autumn 2017 Brussels
Standalone Event
PLUS! Online
Hangouts & Technical Webinar
BIG DATA INTEGRATOR www.github.com/big-data-europe
SEVEN PILOT DESCRIPTIONSwww.big-data-europe.eu/pilot/
PROJECT COORDINATION (Fraunhofer IAIS)Prof. Sören Auer, auer © cs.uni-bonn · de > Dr. Simon Scerri, scerri © cs.uni-bonn · deEIS Department/Group,Fraunhofer IAIS & CS Department Uni-Bonn, Bonn, Germany
Questions & Contactswww.big-data-europe.eu
2 mai 2023#BigDataEurope
leads the Fraunhofer Big Data Alliance