getting started with opensplice and esper
DESCRIPTION
This slides, co-produced by Angelo Corsaro and Thomas Bernhardt introduce stream processing with OpenSplice and Esper. The slides also provide a basic introduction to OpenSplice DDS and the Esper CEP engine.TRANSCRIPT
Getting Started with
OpenSplice and Esper
Angelo Corsaro, Ph.D.Chief Technology Officer OMG DDS Sig [email protected]
Thomas BernhardtChief Technology Officer / [email protected]
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Agenda
! Welcome & Intro
! DDS Overview
! CEP Overview
! DDS + CEP in Action
! Q&A
DDS Overview...from a Stream Processing Perspective
Angelo CORSARO, Ph.D.Chief Technology Officer OMG DDS Sig [email protected]
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Stream Processing ! Stream Processing is an architectural style for building
systems that operate over continuous (theoretically infinite) streams of data
! Stream Processing is often reified under one of its many declinations, such as:! Reactive Systems! Signal Processing Systems! Functional Stream Programming! Data Flow Systems
[1/3]
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Stream Processing! The Stream Processing Architecture very
naturally models systems reacting to streams of data and events produced by the external world, such as the data produced by sensors, a camera or even the data produced by the stock exchange.
! Stream Processing Systems usually operate in real-time over streams and generate in turns other streams of data providing information on what is happening or suggesting actions to perform, such as by stock X, raise alarm Y, or detected spatial violation, etc.
[2/3]
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Stream Processing! Stream Processing Systems are typically
modeled as collection of modules communicating via typed data channels called usually streams
! Modules usually play one of the following roles:! Sources: Injecting data into the System! Filters/Actors: Performing some
computation over sources! Sinks: Consuming the data produced by
the system
[3/3]
Filter
Filter
Filter
source
Sink
Stream
More on Streams
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Defining Streams! In abstract terms, a stream is an infinite sequence of
data samples of a given type T
! Streams can be further classified in continuous and discrete streams. Sometimes referred as Behaviors/Signals and Events
! In this presentation we’ll refer to Continuous Streams as Data Streams and to Discrete Streams as Event Streams
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Data StreamsData Streams! The value of a Data Stream
is always defined, i.e. continuous.
! Good examples of a Data Stream are the value assumed by a real-world entity, such as temperature, pressure, a price, etc.
time
Temp
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Event StreamsEvent Streams! The value of the stream is
defined at precise point in time, i.e. it is discrete
! Good examples of Event Streams are events in the real world, such a violation of a regulatory compliance, the temperature higher than a given value, etc.
OverheatAlarm
time
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
What is DDS?! DDS is a high-performance, real-time, highly-
available, fully-distributed, messaging technology that allows you to define data/event streams and make them dynamically discoverable
! DDS is equipped with a rich set of QoS providing control on the key temporal and availability properties of data
[A Stream Perspective]
Defining Streams in DDS
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
DDS Topics! A Topic defines a stream class/
category! A Topic has associated a user
defined type and QoS! The Topic name, type and QoS
defines the key functional and non-functional invariants
! Topics can be discovered or locally defined DURABILITY,
DEADLINE,PRIORITY,…
[1/2]“org.opensplice.demo.TTempSensor”
struct TempSensor { long Id; float temp; float hum;}#pragma keylist TempSensor id
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
DDS Topics
! DDS Topic types can have associated keys
! Each unique key-value identifies a unique sub-stream of values -- called Topic Instance
[2/2]“org.opensplice.demo.TTempSensor”
DURABILITY,DEADLINE,PRIORITY,…
struct TempSensor { long id; float temp; float hum;}#pragma keylist TempSensor id
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
“Seeing” Streams
Topic
InstancesInstances
struct TempSensor { @key long id; float temp; float hum;};
id =701
id =809
id =977
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
QoS Policies
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Anatomy of a DDS Application
Domain
Reader/WritersUser Defined for Types
Session
val dp = DomainParticipant(0)
// Create a Publisher / SubscriberPublisher p = dp.create_publisher();Subscriber s = dp.create_subscriber();// Create a TopicTopic<TempSensor> t = dp.create_topic<TempSensor>(“com.myco.TSTopic”)
DomainParticipant
Publisher
DataWrter
Topic Subscriber
DataReader
// Create a DataWriter/DataWriterDataWriter<TempSensor> dw = pub.create_datawriter(t);DataReader<TempSensor> dr = sub.create_datareader(t);
Gives access to a DDS Domain
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Anatomy of a DDS Application
Domain
Reader/WritersUser Defined for Types
Session
val dp = DomainParticipant(0)
// Create a Publisher / Subscriberval pub = Publisher(dp)val sub = Subscriber(dp)// Create a Topicval topic = Topic[TempSensor](dp, “org.opensplice.demo.TTempSensor”)
DomainParticipant
Publisher
DataWrter
Topic Subscriber
DataReader
// Create a DataWriter/DataWriterDataWriter<TempSensor> dw = pub.create_datawriter(t);DataReader<TempSensor> dr = sub.create_datareader(t);
Pub/Sub Abstractions
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Anatomy of a DDS Application
Domain
Reader/Writers for User Defined for Types
Session
DomainParticipant
Publisher
DataWrter
Topic Subscriber
DataReader
// Create a DataWriter/DataWriterval writer = DataWriter[TempSensor](pub, topic)val reader = DataReader[TempSensor](sub, topic)
// Write dataval t = new TempSensor ts(101, 25, 40)writer write ts;
Reader/Writer for application defined Topic Types
// Create a Publisher / Subscriberval pub = Publisher(dp)val sub = Subscriber(dp)// Create a Topicval topic = Topic[TempSensor](dp, “org.opensplice.demo.TTempSensor”)
val dp = DomainParticipant(0)
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Data & Event Streams! DDS does not provide different types for Data/Event Streams.
The difference between the two can be made through the DataReader API by using properly using read/take operations
! DataReader::read! Reads the value of the stream w/o removing it from the stream. As a result
multiple read can see the last known value of the stream
! DataReader::take! Takes the value available on the stream (if any yet) and removes it from the
stream
Stream Processing
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Content Filtered Topics! Content Filtered Topics
provide a way of defining a filters over an incoming stream associated with a given topic
! Filters are expressed as the “WHERE” clause of an SQL statement
! Filters can operate on any attribute of the type associated with the topic
Example:
// Create a Topic (on default domain)val topic = Topic[TempSensor](“TTempSensor”)val ftopic = ContentFilteredTopic[TempSensor](“CFTempSensor”, topic, filter, params)
// - filter is a WHERE-like clause, such as: // “temp > 20 AND hum > 50”// “temp > %0”// “temp > hum”// “temp BETWEEN (%0 AND %1)//// - params is the list of parameters to pass to the // filter expression – if any
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Filter Expression Syntax
! DDS Filters are condition over a topic type attributes
! Temporal properties or causality cannot be captured via DDS filter expression
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
History
! DDS provides a way of controlling data windows through the History QoS
The window keeps the last n data samples
Data older than “n samples ago” get’s out the window
pastfuture
now
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
[Putting it All Together]
TempSensor Moving Averageobject MovingAverageFilter { def main(args: Array[String]) { if (args.length < 2) { println("USAGE:\n\tMovingAverageFilter <window> <filter-expression>") }
val topic = Topic[TempSensor]("TTempSensor") val ftopic = ContentFilteredTopic[TempSensor]("CFTempSensor",topic, args(1))
val rqos = DataReaderQos() <= KeepLastHistory(args(0).toInt) val reader = DataReader[TempSensor](ftopic, rqos)
reader.reactions += { case e: DataAvailable[_] => { var average: Float = 0 val window = e[TempSensor].reader.history window foreach (average += _.temp) average = average / window.length println("+--------------------------------------------------------") println("Moving Average: " + average) } } }}
Product Organization
!"#$%&'
()*+,--.*/%&01234(*5**6
77*8&'()0*8303%93:
Product OrganizationCommercial Edition! No Cost Runtime Licenses! Your choice of licensing
! LGPL or Commercial! Subscription or Perpetual
! Complete DDS Implementation! Comprehensive Developer and
Deployment Support Options with a range of Service Level Agreements
Commercial Add-Ons! Individually licensable technologies! Rich ecosystem covering tools,
integration, testing, etc.
CommercialEdition
Commercial Add-ons
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Key Points So Far
! DDS key abstractions for building stream processing architectures
! DDS provides some event processing capabilities that facilitate the development of Stream Processing Filters
! What else can you use for “Stream Processing” in combination with DDS? ! Let’s have Tom introduce us to the world of CEP and Esper!
���������� ��� ���������������
���������
���������������������������������������������
�
���� !
���� ������������!� �������������
"�!������#�$��������������
%!��#������������������&!���!��'��&(
)
�����!� �������������
�����
�����*!+!�!�����#������������
&���,����!� ��-� !-#����!��+!�!�����#����������
.���������/�&#������
0���#������!�� ��� 12��#�!���� ��+���� �2��,!�������1��
3���������3��
��455�������� ��!������
�������������
�����#���2�!�����6������' ���#�������!27$82�1$9(
��������!##����������� ����-������ ������/�&
��������!##��������������������!����
:�����!�#!-�#��2�������!�!��!-�#��
0���#�����#�����#�2����!#�;!���
1
���4����#�$��������������
������!����!�������� !!
<�#���,����� !���!����/���#����=������#!�'������() ���
����!�������!���������������#!�'������(����
0����!����!���������
��/���#�<"�������!�� �����!�8>,�����
��##�,� -�?!���@<"�� ����!�� �����!�� >���A
��8�������� ��!� B��8�#!�����!�-�8��#�����A
����#�$��������������
8
�������!�� ��0CD%
�0CD%
%��� !!'!#�(
:!� #�E������
%F&'��!-��!�� (
��E���5��������
������������
�������
���
%�����#��
:!� #� !!
G ��&4��������������&!���!��
%�-����-�53����
����6�!��!#��
H IJ���������E����JK
��������
������ ��
���������
������ ��
���
�
���<!#��
���#��!����!#��4���������������������������� �����
��� ���������#�$�����2�� ��� #!����2!� ����-������L
�� �������
�������!#��������#!���������������������������
�������!#��������#!���������������� ���!����!���5��!���
.���������!#�����-�������� ������!� �$�#���������#�$�����
������#����!#��4�����!���������!� ���#�����!+���5
0C5C������������',��!����#��-!�=�������(
�� ��!#���������;���%F&��!����!� +!�!��!���������� ��#!�!���
��#��'��������-���������!���� �(
�����!���-������������ ����=+����0C!� C����������!���
-�������#����
&�,������������� ���M!�#��#������5:�����!�#!-�#��,��!����#�
����-�� ��!#!-#��#!�����!���#� ���-N�����!���
9
��&%�#���#!���
%�#���#!���
%!��#�
[insert into insert_into_def]select select_listfrom stream_def [as name] [, stream_def [as name]] [,...][where search_conditions][group by grouping_expression_list][having grouping_search_conditions][output output_specification][order by order_by_expression_list]
select acctId, sum(amount) from Withdrawal.win:time(1 minute)group by acctIdhaving sum(amount) > 1000order by acctId asc
7
�!����$��������O����P.���!���
�!����$������������������!���!���!� �!���
����!���
����4C!���-��# ���-#��=�
M�#���$���������
.-��������������-!�� �����
������#������-�����������$���!#�����2����!�.%���������,��=
�����
.���!���
&����!#.���!���4���������������� ��
.���!����!�����##������#�4� ���������������� ����������
/�!� �4����������
������#�������!� ���������!����$���������-!�� ���$���!#
�����QM�##�,�R�!����!��4�!��S���!$�
������!#����!��4��
T
&����!#.���!���
.���!���$!��#��
M�##�,� -�
�!������������������$���������
&����!#���
�����##�����������,��� ���
select * from pattern [w=Withdrawal -> Deposit(acctId = w.acctId)]
w=Withdrawal -> (pin=PinChange(acctId = w.acctId) and pwd=PasswordChange(acctId = w.acctId) )
every ( (w=Withdrawal -> Deposit(acctId = w.acctId)) or (d=Deposit -> Withdrawal(acctId = d.acctId)) )
�
&����!#.���!�������
.���!���$!��#��
������!�����!��������!#��
.-��������������� �������!���!����������!#
/�!� ������������ ���-�$��������!���!����������!#
������������!�������
������������!�����������������
every login=LoginEvent -> (timer:interval(20 sec) and not LogoutEvent(userId=login.userId))
every [2] login=LoginEvent
every [2..] login=LoginEvent until logout=LogoutEvent
every login=LoginEvent -> logout=LogOutEvent where timer:within(2 sec)
��
��� ���.�������
�������� ������ ��� � ��
H.�������!����!��2����#������#������ ��=2��#��!#����������!�����
H�#�������!� ��-!�=���!�!-�#�����������!� ���!���!�
H0������43��� ���!���Q%!����#���#F�%��������!���
H%���#�����!���!#��E��������Q�$����-#�2�������
��������������������������� ��� ��!��"�����#$��%
H���,���$����������������������&!���!��
H���������E������2!�����!���2N����2�!��!#��!� ������������2N������������!# !!2����
�#�,�����#2����
H�!�� ���#�����!� ��#�����
H:�����������2#�,#!����
H%!� !#�������-� � 2�������2+!�!�!� !� �������
������� ���# � ��������� ��& � � ��
H�������4�������!�!�����2��!#�����������!�����!#�;!���!� ! ����E������
H��������4D�#��#!��� !!�����!����Q�����������!� ��-���������!�!�������������
H�����"�'�4+0C�����#�!��#���!� �������� ����������������!-�#��
��
�������
����#����!�����2#��!#�!���2
��������!����!��2�#�������!� ��-!�=���
��� ����������!#����������
����%��!�
���������6
! !����
:������!# !!
!�����#!���
����F����6�!��!#���!���&!���!��
����
����������
%!�����
�.+.
.���
! !����
�.+.
%�-����-���
������������!����
:�������� �������#���
��!#���� !!���!��
:������!# !!
3!��
"�� �,�
��������� �����
��� ��� ��
�����"�'�
'%������� ����(
�����"�'�
'�#��� �����(
��������
(��������� ��)*���+��
�������,�#���*���-".��/
0 ���� 1�� ��)2���
���������� ��� ���������������
��!�=���
��455,,,�������������
����V������������
DDS + CEP in Action
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
OpenSplice + Esper
! Esper Provides an EsperIO framework for plugging-in new stream transports
! Plugging OpenSplice into Esper is trivial even w/o relying on the EsperIO framework
! Let’s have a look…
Demo Application
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
iShapes Application! To explore play with OpenSplice and
Esper, we’ll use the simd-cxx ishapes application
! Three Topics! Circle, Square, Triangle
! One Type:
struct ShapeType { string color; long x; long y; long shapesize;};#pragma keylist Shapetype color
Spotted shapes represent subscriptions
Pierced shapes represent publications
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Esper Setup
Step 1: Register Topic Typesval config = new Configurationval ddsConf = new ConfigurationEventTypeLegacy
ddsConf.setAccessorStyle(ConfigurationEventTypeLegacy.AccessorStyle.PUBLIC)
config.addEventType("ShapeType", classOf[org.opensplice.demo.ShapeType].getName, ddsConf)
val cep: EPServiceProvider = EPServiceProviderManager.getDefaultProvider(config)
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Esper Setup
Step 2: Register a Listener for receiving Esper Events
val listener = new UpdateListener { def update(ne: Array[EventBean], oe: Array[EventBean]) { ne foreach(e => {! // Handle the event }) } }
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
Esper Setup
Step 3: Hook-up DDS to Esper
reader.reactions += { case e: DataAvailable[ShapeType] => { (e.reader read) foreach(runtime sendEvent _)
}}
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
iShapes FrameRate
! Let’s suppose that we wanted to keep under control the iShapes Frame rate for ech given color
! In Esper this can be achieved with the following expression:
insert into ShapesxSec select color, count(*) as cnt from ShapeType.win:time_batch(1 second) group by color
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
iShapes Center of Mass
! Suppose that we wanted to compute the center of mass of all the shapes currently displayed over the last second
! The Esper expression for this would be:
select ShapeFactory.createShape(color, cast(avg(x),int), cast(avg(y),int), shapesize) as NewShape from ShapeType.win:time(10 sec)
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
" Fastest growing JVM Language" Open Source" www.scala-lang.org
References
" #1 OMG DDS Implementation" Open Source" www.opensplice.org
" Scala API for OpenSplice DDS" Open Source" code.google.com/p/escalier
" #1 Java-Based CEP Engine" Open Source" www.espertech.com
!"#$%&'
()*+,-,.*/%&01234(*5**6
77*8&'()0*8303%93:;
:: Connect with Us ::
! @prismtech
! @acorsaro
! youtube.com/opensplicetube ! slideshare.net/angelo.corsaro
! [email protected]! [email protected]
! opensplice.com ! forums.opensplice.org
! opensplice.org ! [email protected]