implementation of a jni wrapper & mapreduce compatibility layer for streammine3g … ·...
TRANSCRIPT
Implementation of a JNI Wrapper & MapReduce Compatibility Layer for StreamMine3G
by Benjamin Schmidt
Faculty of Computer Science Institute of Systems Architecture, Chair of Systems Engineering
Dresden, 25.11.13
Dresden, 25.11.13 Folie 2 von 25
Outline
• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks
Dresden, 25.11.13 Folie 3 von 25
Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks
Dresden, 25.11.13 Folie 4 von 25
Introduction to StreamMine3G
Dresden, 25.11.13 Folie 5 von 25
Introduction to StreamMine3G
Dresden, 25.11.13 Folie 6 von 25
Introduction to StreamMine3G
...
...
Dresden, 25.11.13 Folie 7 von 25
Introduction to StreamMine3G
• Event processing engine• Written in C++• Provides fault tolerance at low cost• Supports stateful operators
Dresden, 25.11.13 Folie 8 von 25
Introduction to StreamMine3G
Dresden, 25.11.13 Folie 9 von 25
Introduction to StreamMine3G
• Streams – Sequence of tuples/events• Topology – Pattern of communication
between Operators
• Slices/Partitions – Can be deployed on different nodes
Dresden, 25.11.13 Folie 10 von 25
Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks
Dresden, 25.11.13 Folie 11 von 25
JNI Wrapper : Introduction
Java
Dresden, 25.11.13 Folie 12 von 25
JNI Wrapper : Introduction
Dresden, 25.11.13 Folie 13 von 25
Dresden, 25.11.13 Folie 14 von 25
JNI Wrapper : Bottlenecks
• C++ more efficient execution• Additional JNI Redirections• Datatype conversion• Generation of Java Objects during
De-/Serialization C++
Dresden, 25.11.13 Folie 15 von 25
JNI Wrapper : Reasons for Java
• C++ rather difficult to implement• Most Pupils and Students learn Java from
the Beginning• Already implemented solutions• Possibly easier to debug
Dresden, 25.11.13 Folie 16 von 25
JNI Wrapper: Problems
• Datatype conversion (Pointer & String)• Thread handling• Optimization (Bytearrays)
Dresden, 25.11.13 Folie 17 von 25
Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks
Dresden, 25.11.13 Folie 18 von 25
Hadoop MapReduce
• Written in Java• Cluster of Nodes• Needs Mapper & Reducer Operator• Before Reducer phase Mapper phase must →
be finished• Batch Processing
Dresden, 25.11.13 Folie 19 von 25
MapReduce Compatibility Layer
JNI Wrapper
MapReduce CL
Dresden, 25.11.13 Folie 20 von 25
Outline• Introduction to StreamMine3G• JNI Wrapper• MapReduce Compatibility Layer• Future Tasks
Dresden, 25.11.13 Folie 21 von 25
Future Tasks : Apache Pig
• Abstraction build on top of Hadoop• consists of a domain-specific dataflow
language, Pig Latin• Generates and compiles MapReduce
programs on the fly
Dresden, 25.11.13 Folie 22 von 25
Future Tasks : Apache Pig
Apache Pig Compatability Layer
Hadoop MapReduceCompatability Layer
Dresden, 25.11.13 Folie 23 von 25
Future Task : Storm
• Real-time processing• Topologies run forever• Scalable and fault-tolerant• Consists of spout, bolts, event streams and
Nimbus
Dresden, 25.11.13 Folie 24 von 25
Future Tasks : Storm
STORM OperatorCompatability Layer
STORM
PerformanceEvaluation
?
STORMOperator
STORMOperator
Dresden, 25.11.13 Folie 25 von 25
Thank you for your attention!