scalable graph data analytics with gradoop · scalable graph data analytics with gradoop erhard...
TRANSCRIPT
![Page 1: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/1.jpg)
www.scads.de
SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP
ERHARD RAHM,MARTIN JUNGHANNS, ANDRÉ PETERMANN, ERIC PEUKERT
![Page 2: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/2.jpg)
,
“GRAPHS ARE EVERYWHERE”
2
![Page 3: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/3.jpg)
,
“GRAPHS ARE EVERYWHERE”
Alice
Bob
Eve
Dave
Carol
Mallory
Peggy
Trent
3
![Page 4: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/4.jpg)
,
“GRAPHS ARE EVERYWHERE”
Alice
Bob
Eve
Dave
Carol
Mallory
Peggy
Trent
4
![Page 5: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/5.jpg)
“GRAPHS ARE HETEROGENEOUS”
Alice
Bob
AC/DC
Dave
Carol
Mallory
Peggy
Metallica
∪ , ∪
5
![Page 6: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/6.jpg)
0.2
0.28
0.26
0.33
0.25
0.26
“GRAPHS CAN BE ANALYZED”
Alice
Bob
AC/DC
Dave
Carol
Mallory
Peggy
Metallica
3.62.82
∪ , ∪
6
![Page 7: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/7.jpg)
“GRAPHS CAN BE ANALYZED“
Assuming a social network1. Determine subgraph2. Find communities3. Filter communities4. Find common subgraph
7
![Page 8: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/8.jpg)
all V challenges (volume, variety, velocity, veracity) ease-of-use high cost-effectiveness powerful but easy to use graph data model support for heterogeneous, schema-flexible vertices and edges support for collections of graphs (not only 1 graph) powerful graph operators
graph-based integration of many data sources
versioning and evolution (dynamic /temporal graphs)
interactive, declarative graph queries
scalable graph mining
comprehensive visualization support
GRAPH DATA ANALYTICS: REQUIREMENTS
8
![Page 9: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/9.jpg)
COMPARISON
9
Graph Database SystemsNeo4j, OrientDB
data model rich graph models(PGM)
focus queries
query language yes
graph analytics no
scalability vertical
Workflows no
persistency yes
dynamic graphs / versioning
no
data integration no
visualization (yes)
![Page 10: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/10.jpg)
COMPARISON (2)
10
Graph Database SystemsNeo4j, OrientDB
Graph Processing Systems(Pregel, Giraph)
data model rich graph models(PGM)
genericgraph models
focus queries analytic
query language yes no
graph analytics no yes
scalability vertical horizontal
Workflows no no
persistency yes no
dynamic graphs / versioning
no no
data integration no no
visualization (yes) no
![Page 11: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/11.jpg)
COMPARISON (3)
11
Graph Database SystemsNeo4j, OrientDB
Graph Processing Systems(Pregel, Giraph)
Distributed DataflowSystems (Flink Gelly, Spark GraphX)
data model rich graph models(PGM)
genericgraph models
genericgraph models
focus queries analytic analytic
query language yes no no
graph analytics no yes yes
scalability vertical horizontal horizontal
Workflows no no yes
persistency yes no no
dynamic graphs / versioning
no no no
data integration no no no
visualization (yes) no no
![Page 12: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/12.jpg)
An end-to-end framework and research platform forefficient, distributed and domain independent graph
data management and analytics.
WHAT‘S MISSING?
12
![Page 13: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/13.jpg)
Data Volume and Problem Complexity
Ease
-of-u
se
Graph Processing Systems
Graph Databases
Graph Dataflow Systems Gelly
13
![Page 14: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/14.jpg)
Intro Graph Analytics Graph data Requirements Graph database vs graph processing systems
Gradoop Architecture Extended Property Graph Model (EPGM) Implementation Evaluation
Summary/Outlook
AGENDA
14
![Page 15: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/15.jpg)
Hadoop-based framework for graph data management and analysis persistent graph storage in scalable distributed store (Hbase) utilization of powerful dataflow system (Apache Flink) for parallel,
in-memory processing
Extended property graph data model (EPGM) operators on graphs and sets of (sub) graphs support for semantic graph queries and mining
Declarative specification of graph analysis workflows Graph Analytical Language - GrALa
End-to-end functionality graph-based data integration, data analysis and visualization
Open-source implementation: www.gradoop.org
GRADOOP CHARACTERISTICS
15
![Page 16: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/16.jpg)
integrate data from one or more sources into a dedicated graph storewith common graph data model
definition of analytical workflows from operator algebra
result representation in meaningful way
END-TO-END GRAPH ANALYTICS
Data Integration Graph Analytics Visualization
16
![Page 17: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/17.jpg)
HIGH LEVEL ARCHITECTURE
HDFS/YARNCluster
HBase Distributed Graph Store
Extended Property Graph Model
Flink Operator Implementations
Data Integration
Flink Operator Execution
Workflow Declaration
Visual
GrALa DSLRepresentation
Data flow
Control flow
Graph Analytics Representation
17
![Page 18: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/18.jpg)
Graph Operators
Extended Property Graph Model
Graph Store
Streaming Dataflow Runtime
DataSet DataStream
HadoopM
R
Table
Gelly
Flin
kML
Table
Cluster (e.g. YARN)Local Cloud (e.g. EC2)
Data Storage (e.g. Files, HDFS, HBase, S3, JDBC, Kafka, …)
CEP
……
GRADOOP AS A FLINK EXTENSION
18
![Page 19: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/19.jpg)
includes PGM as special case
support for collections of logical graphs / subgraphs can be defined explicitly can be result of graph algorithms / operators
support for graph properties
powerful operators on both graphs and graph collections
Graph Analytical Language – GrALa domain-specific language (DSL) for EPGM flexible use of operators with application-specific UDFs plugin concept for graph mining algorithms
EXTENDED PROPERTY GRAPH MODEL (EPGM)
19
![Page 20: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/20.jpg)
• Vertices and directed Edges
20
![Page 21: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/21.jpg)
• Vertices and directed Edges• Logical Graphs
21
![Page 22: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/22.jpg)
• Vertices and directed Edges• Logical Graphs• Identifiers
1 3
4
5
21 2
3
4
5
1
2
22
![Page 23: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/23.jpg)
• Vertices and directed Edges• Logical Graphs• Identifiers• Type Labels
1 3
4
5
21 2
3
4
5Person Band
Person
Person
Band
likes likes
likes
knows
likes
1|Community
2|Community
23
![Page 24: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/24.jpg)
• Vertices and directed Edges• Logical Graphs• Identifiers• Type Labels• Properties
1 3
4
5
21 2
3
4
5Personname : Aliceborn : 1984
Bandname : Metallicafounded : 1981
Personname : Bob
Personname : Eve
Bandname : AC/DCfounded : 1973
likessince : 2014
likessince : 2013
likessince : 2015
knows
likessince : 2014
1|Community|interest:Heavy Metal
2|Community|interest:Hard Rock
24
![Page 25: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/25.jpg)
Operators
25
![Page 26: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/26.jpg)
Operators
Unary BinaryAlgorithms
* auxiliary
Graph
Collection
LogicalG
raph
Aggregation
Pattern Matching
Transformation
Grouping Equality
Call *
Combination
Overlap
Exclusion
Equality
Union
IntersectionDifference
Gelly Library
BTG Extraction
Frequent Subgraphs
Limit
Selection
DistinctSort
Apply *Reduce *Call *
Adaptive Partitioning
Subgraph
26
![Page 27: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/27.jpg)
Combination
Overlap
Exclusion
LogicalGraph graph3 = graph1.combine(graph2);LogicalGraph graph4 = graph1.overlap(graph2);LogicalGraph graph5 = graph1.exclude(graph2);
BASIC BINARY OPERATORS
1 34
52
3
1 2
1 34
52
12 4
5
3
27
![Page 28: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/28.jpg)
udf = (graph => graph[‘vertexCount’] = graph.vertices.size())graph3 = graph3.aggregate(udf)
AGGREGATION
1 34
52
3
1 34
52
3 | vertexCount: 5
UDF
28
![Page 29: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/29.jpg)
LogicalGraph graph4 = graph3.subgraph((vertex => vertex[:label] == ‘green’))LogicalGraph graph5 = graph3.subgraph((edge => edge[:label] == ‘blue’))LogicalGraph graph6 = graph3.subgraph(
(vertex => vertex[:label] == ‘green’),(edge => edge[:label] == ‘orange’))
SUBGRAPH
3
1 34
52
3
4
1 2
5
35
2UDF
UDF
UDF 3
6
1 2
29
![Page 30: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/30.jpg)
GraphCollection collection = graph3.match(“(:Green)‐[:orange]‐>(:Orange)”);
PATTERN MATCHING
3
1 34
52 Pattern
4 5
1 34
2
Graph Collection
30
![Page 31: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/31.jpg)
LogicalGraph grouped = graph3.groupBy([:label], // vertex keys[:label]) // edge keys
LogicalGraph grouped = graph3.groupBy([:label], [COUNT()], [:label], [MAX(‘a’)])
GROUPING
Keys
3
1 34
52
+Aggregate
3
a:23 a:84
a:42
a:12
1 34
52
a:13
a:21
4
count:2 count:3
max(a):42
max(a):84
max(a):13 max(a):21
6 7
4
6 7
31
![Page 32: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/32.jpg)
[2] Community | interest : Graphs | vertexCount : 4
[1] Community | interest : Hadoop| vertexCount : 3[0] Community | interest : Databases | vertexCount : 3
SAMPLE GRAPH [0] Tag
name : Databases[1] Tag
name : Graphs[2] Tag
name : Hadoop
[3] Forumtitle : Graph Databases
[4] Forumtitle : Graph Processing
[5] Personname : Alicegender : fcity : Leipzigage : 23
[6] Personname : Bobgender : mcity : Leipzigage : 30
[7] Personname : Carolgender : fcity : Dresdenage : 30
[8] Personname : Davegender : mcity : Dresdenage : 42
[9] Personname : Evegender : fcity : Dresdenage : 35speaks : en
[10] Personname : Frankgender : mcity : Berlinage : 23IP: 169.32.1.3
0
1
2
3
4
5
6 7 8 9
10
11 12 13 14
15
16
17
18 19 20 21
22
23
knowssince : 2014
knowssince : 2014
knowssince : 2013
hasInterest
hasInterest hasInterest
hasInterest
hasModeratorhasModeratorhasMember hasMember
hasMember hasMember
hasTag hasTaghasTag hasTag
knowssince : 2013
knowssince : 2014
knowssince : 2014
knowssince : 2015
knowssince : 2015
knowssince : 2015
knowssince : 2013
32
![Page 33: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/33.jpg)
GROUPING: TYPE LEVEL (SCHEMA GRAPH)
vertexGrKeys = [:label]edgeGrKeys = [:label]sumGraph = databaseGraph.groupBy(vertexGrKeys, [COUNT()], edgeGrKeys, [COUNT()])
[11] Person
count : 6
[12] Forum
count : 3
[13] Tag
count : 3
hasMembercount : 4
knowscount : 10
hasInterestcount : 4
hasTagcount : 4
hasModeratorcount : 2
24
26
28
27
25
33
![Page 34: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/34.jpg)
personGraph = databaseGraph.subgraph((vertex => vertex[:label] == ‘Person’),(edge => edge[:label] == ‘knows’))
vertexGrKeys = [:label, “city”]edgeGrKeys = [:label]sumGraph = personGraph.groupBy(vertexGrKeys, [COUNT()], edgeGrKeys, [COUNT()])
GROUPING: PROPERTY-SPECIFIC
1 3
[11] Person
city : Leipzigcount : 2
[12] Person
city : Dresdencount : 3
[13] Person
city : Berlincount : 1
24
25
26
27
28
knowscount : 3
knowscount : 1 knows
count : 2
knowscount : 2
knowscount : 2
34
![Page 35: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/35.jpg)
GraphCollection filtered = collection.select((graph => graph[‘vertexCount’] > 4));
SELECTION
UDF
vertexCount > 4
1 | vertexCount: 5
2 | vertexCount: 4
0 23
41
5 7 86
1 | vertexCount: 5
0 23
41
35
![Page 36: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/36.jpg)
GraphCollection frequentPatterns = collection.callForCollection(new TransactionalFSM(0.5))
CALL (E.G. FREQUENT SUBGRAPHS)
FSM
Threshold: 50%
1
0 1 23
4
5 6 78
9
1013
14
2
3
11 12
15 16
17 18
19 20
4
5
6
21 2322
25 2624
7
8
36
![Page 37: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/37.jpg)
Implementation
37
![Page 38: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/38.jpg)
GRAPH REPRESENTATION
Id Label Properties Graphs
Id Label Properties SourceId TargetId Graphs
EPGMGraphHead
EPGMVertex
EPGMEdge
Id Label Properties POJO
POJO
POJO
DataSet<EPGMGraphHead>
DataSet<EPGMVertex>
DataSet<EPGMEdge>
Id Label Properties Graphs
EPGMVertex
GradoopId := UUID128‐bit
String PropertyList := List<Property>Property := (String, PropertyValue)PropertyValue := byte[]
GradoopIdSet := Set<GradoopId>
38
![Page 39: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/39.jpg)
Id Label Properties
1 Community {interest:Heavy Metal}
2 Community {interest:Hard Rock}
Id Label Properties Graphs
1 Person {name:Alice, born:1984} {1}
2 Band {name:Metallica,founded:1981} {1}
3 Person {name:Bob} {1,2}
4 Band {name:AC/DC,founded:1973} {2}
5 Person {name:Eve} {2}
Id Label Source Target Properties Graphs
1 likes 1 2 {since:2014} {1}
2 likes 3 2 {since:2013} {1}
3 likes 3 4 {since:2015} {2}
4 knows 3 5 {} {2}
5 likes 5 4 {since:2014} {2}
likessince : 2014
likessince : 20131 3
4
5
2
1|Community|interest:Heavy Metal
2|Community|interest:Hard Rock
Personname : Aliceborn : 1984
Bandname : Metallicafounded : 1981
Personname : Bob
Personname : Eve
Bandname : AC/DCfounded : 1973likes
since : 2015
knows
likessince : 20141 2
3
4
5
DataSet<EPGMGraphHead>
DataSet<EPGMVertex> DataSet<EPGMEdge>
GRAPH REPRESENTATION: EXAMPLE
39
![Page 40: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/40.jpg)
// input: firstGraph (G[1]), secondGraph (G[2])
1: DataSet<GradoopId> graphId = secondGraph.getGraphHead()2: .map(new Id<G>());3: 4: DataSet<V> newVertices = firstGraph.getVertices()5: .filter(new NotInGraphBroadCast<V>())6: .withBroadcastSet(graphId, GRAPH_ID);7:8: DataSet<E> newEdges = firstGraph.getEdges()9: .filter(new NotInGraphBroadCast<E>())
10: .withBroadcastSet(graphId, GRAPH_ID)11: .join(newVertices)12: .where(new SourceId<E>().equalTo(new Id<V>())13: .with(new LeftSide<E, V>())14: .join(newVertices)15: .where(new TargetId<E>().equalTo(new Id<V>())16: .with(new LeftSide<E, V>());
Exclusion
OPERATOR IMPLEMENTATION
likessince : 2013
likessince : 20141 3
4
5
2
1|Community|interest:Heavy Metal
2|Community|interest:Hard Rock
Personname : Aliceborn : 1984
Bandname : Metallicafounded : 1981
Personname : Bob
Personname : Eve
Bandname : AC/DCfounded : 1973likes
since : 2015
knows
likessince : 20141 2
3
4
5
40
![Page 41: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/41.jpg)
IMPLEMENTATION OF GRAPH GROUPING
GroupBy(1,2,3) +GC + GR* + MapAssign edges to groupsCompute aggregatesBuild super edges
Filter + MapExtract super vertex tuplesBuild super vertices
GroupBy(1) + GroupReduce*Assign vertices to groupsCompute aggregatesCreate super vertex tuplesForward updated group members
V
E
MapExtractattributes
Filter + Map Extract group membersReduce memory footprint
Join*Replace Source/TargetIdwith corresponding super vertex id
MapExtractattributes
*requires worker communication
V1 V2
V3
V‘
E1 E2 E‘
41
![Page 42: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/42.jpg)
ITERATIVE COMPUTATION OF FREQUENT SUBGRAPHS
42
n‐edge3‐edge2‐edge1‐edge
result
collecting intermediate iteration results
searchspace
1‐edgeR C F
2‐edgeG R C F
3‐edgeG R C F
n‐edgeG R C F
G : grow frequent patternsR : report supported patternsC : count global frequencyF : filter by min frequency
![Page 43: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/43.jpg)
Evaluation
![Page 44: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/44.jpg)
TEST WORKFLOW: SUMMARIZED COMMUNITIES
http://ldbcouncil.org/
1. Extract subgraph containing only Persons and knows relations
2. Transform Persons to necessary information
3. Find communities using Label Propagation
4. Aggregate vertex count for each community
5. Select communities with more than 50K users
6. Combine large communities to a single graph
7. Group graph by Persons location and gender
8. Aggregate vertex and edge count of grouped graph
44
![Page 45: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/45.jpg)
TEST WORKFLOW: SUMMARIZED COMMUNITIES
https://git.io/vgozj
1. Extract subgraph containing only Persons
and knows relations
2. Transform Persons to necessary information
3. Find communities using Label Propagation
4. Aggregate vertex count for each community
5. Select communities with more than 50K users
6. Combine large communities to a single graph
7. Group graph by Persons location and gender
8. Aggregate vertex and edge count of grouped graph
45
![Page 46: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/46.jpg)
BENCHMARK RESULTS
Dataset # Vertices # Edges
Graphalytics.1 61,613 2,026,082
Graphalytics.10 260,613 16,600,778
Graphalytics.100 1,695,613 147,437,275
Graphalytics.1000 12,775,613 1,363,747,260
Graphalytics.10000 90,025,613 10,872,109,028
16x Intel(R) Xeon(R) 2.50GHz (6 Cores) 16x 48 GB RAM 1 Gigabit Ethernet Hadoop 2.6.0 Flink 1.0-SNAPSHOT
0
200
400
600
800
1000
1200
1 2 4 8 16
Runtim
e [s]
Number of workers
RuntimeGraphalytics.100
1
2
4
8
16
1 2 4 8 16
Speedu
p
Number of workers
SpeedupGraphalytics.100 Linear
46
![Page 47: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/47.jpg)
BENCHMARK RESULTS 2
Dataset # Vertices # Edges
Graphalytics.1 61,613 2,026,082
Graphalytics.10 260,613 16,600,778
Graphalytics.100 1,695,613 147,437,275
Graphalytics.1000 12,775,613 1,363,747,260
Graphalytics.10000 90,025,613 10,872,109,028
1
10
100
1000
10000
Runtim
e [s]
Datasets
16x Intel(R) Xeon(R) 2.50GHz (6 Cores) 16x 48 GB RAM 1 Gigabit Ethernet Hadoop 2.6.0 Flink 1.0-SNAPSHOT
47
![Page 48: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/48.jpg)
EVALUATION OF GROUPING: SCALABILITY
Speedup for grouping on type Runtime for grouping on type
48
![Page 49: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/49.jpg)
Intro Graph Analytics Graph data Graph databases vs graph processing systems
Gradoop Architecture Extended Property Graph Model (EPGM) Use cases Evaluation
Summary/Outlook
AGENDA
49
![Page 50: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/50.jpg)
Big Graph Analytics Hadoop-based graph processing frameworks based on generic graphs Spark/Flink: batch/streaming-oriented workflows (rather than interactive OLAP) graph collections not generally supported generally missing: graph-based data integration, built-in support for dynamic graph data
GraDoop (www.gradoop.org) open-source infrastructure for entire processing pipeline: graph acquisition, storage,
integration, transformation, analysis (queries + graph mining), visualization extended property graph model (EPGM) with powerful operators (e.g., grouping, pattern
matching) and support for graph collections leverages Hadoop ecosystem Apache HBase for permanent graph storage Apache Flink to implement operators
ongoing implementation
SUMMARY
50
![Page 51: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/51.jpg)
COMPARISON
51
Graph Database SystemsNeo4j, OrientDB
Graph Processing Systems(Pregel, Giraph)
DistributedDataflow Systems (Flink Gelly, Spark GraphX)
data model rich graphmodels (PGM)
genericgraph models
genericgraph models
Extended PGM
focus queries analytic analytic analytic
query language yes no no no
graph analytics no yes yes yes
scalability vertical horizontal horizontal horizontal
Workflows no no yes yes
persistency yes no no yes
dynamic graphs / versioning
no no no no
data integration no no no (yes)
visualization (yes) no no limited
![Page 52: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/52.jpg)
instrumental to develop Gradoop in relatively short time
elegant and intuitive DataSet API
very good out-of-the-box performance for non-custom types
stumbling blocks collecting intermediate results during iterations requires non-intuitive
workarounds missing possibility to reuse datasets in data flow programs missing multicast operator with multiple outputs of possibly different types (to
replace filter hierarchies causing duplication of previous ouputs) missing support for theta-joins (e.g., via user-defined join predicates) missing adaptive configuration of parallelism (e.g., to keep data local as long
as possible)
LESSONS LEARNED ABOUT FLINK
52
![Page 53: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/53.jpg)
Graph-based data integration unified approach for knowledge graphs and regular data graphs holistic data integration for many sources
Graph analytics automatic optimization of analysis workflows optimized graph partitioning approaches visualization of graphs and analysis results interactive graph analytics dynamic graph data
OUTLOOK / CHALLENGES
53
![Page 54: SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP · SCALABLE GRAPH DATA ANALYTICS WITH GRADOOP ERHARD RAHM, MARTIN JUNGHANNS, ANDRÉ PETERMANN, ... Graph Processing Systems Graph Databases](https://reader035.vdocuments.mx/reader035/viewer/2022070811/5f0a8e857e708231d42c3752/html5/thumbnails/54.jpg)
M. Junghanns, A. Petermann, K. Gomez, E. Rahm: GRADOOP - Scalable Graph Data Management and Analytics with Hadoop. Tech. report (Arxiv), Univ. of Leipzig, 2015
M. Junghanns, A. Petermann, N. Teichmann, K. Gomez, E. Rahm: Analyzing Extended Property Graphs with Apache Flink. Proc. ACM SIGMOD workshop on Network Data Analytics (NDA), 2016
M. Junghanns, A. Petermann, N. Neumann, E. Rahm: Management and Analysis of Big Graph Data: Current Systems and Open Challenges. In: Big Data Handbook (eds.: S. Sakr, A. Zomaya) , Springer, 2017 (to appear)
A. Petermann, M. Junghanns, S. Kemper, K. Gomez, N.Teichmann, E. Rahm: Graph Mining for Complex Data Analytics. Proc. ICDM 2016 (Demo paper)
A. Petermann, M. Junghanns, R. Müller, E. Rahm: BIIIG : Enabling Business Intelligence with Integrated Instance Graphs. Proc. 5th Int. Workshop on Graph Data Management (GDM 2014)
A. Petermann, M. Junghanns, R. Müller, E. Rahm: Graph-based Data Integration and Business Intelligence with BIIIG. Proc. VLDB Conf., 2014
A. Petermann, M. Junghanns, R. Müller, E. Rahm: FoodBroker - Generating Synthetic Datasets for Graph-BasedBusiness Analytics. Proc. 5th Int. Workshop on Big Data Benchmarking (WBDB), 2014
A. Petermann; M. Junghanns: Scalable Business Intelligence with Graph Collections. it - Information Technology Special Issue: Big Data Analytics, 2016
REFERENCES
54