Modeling Performance of a Parallel Streaming Engine:Bridging Theory and Costs

Ivan Bedini Sherif Sakr Bart Theeten Alessandra Sala Peter CoganAlcatel-Lucent Bell Labs, Ireland and Belgium

{firstname.lastname}@alcatel-lucent.com

ABSTRACTWhile data are growing at a speed never seen before, par-allel computing is becoming more and more essential toprocess this massive volume of data in a timely manner.Therefore, recently, concurrent computations have been re-ceiving increasing attention due to the widespread adoptionof multi-core processors and the emerging advancements ofcloud computing technology. The ubiquity of mobile de-vices, location services, and sensor pervasiveness are exam-ples of new scenarios that have created the crucial need forbuilding scalable computing platforms and parallel architec-tures to process vast amounts of generated streaming data.In practice, efficiently operating these systems is hard dueto the intrinsic complexity of these architectures and thelack of a formal and in-depth knowledge of the performancemodels and the consequent system costs. The Actor Modeltheory has been presented as a mathematical model of con-current computation that had enormous success in practiceand inspired a number of contemporary work in this area.Recently, the Storm system has been presented as a real-ization of the principles of the Actor Model theory in thecontext of the large scale processing of streaming data. Inthis paper, we present, to the best of our knowledge, thefirst set of models that formalize the performance character-istics of a practical distributed, parallel and fault-tolerantstream processing system that follows the Actor Model the-ory. In particular, we model the characteristics of the dataflow, the data processing and the system management costsat a fine granularity within the different steps of executing adistributed stream processing job. Finally, we present an ex-perimental validation of the described performance modelsusing the Storm system.

Categories and Subject DescriptorsC.1.2 [Computer Systems Organization ]: MultipleData Stream Architectures (Multiprocessors)— Parallel pro-cessors; C.1.4 [Computer Systems Organization ]: Par-allel Architectures—Distributed Architecture

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICPE’13, April 21–24, 2013, Prague, Czech Republic.Copyright 2013 ACM 978-1-4503-1636-1/13/04 ...$15.00.

KeywordsDistributed Stream Processing Engine, Performance Model,Measurement

1. INTRODUCTIONToday’s era of Big Data is witnessing a continuous in-

crease of user and machine connectivity that produces anoverwhelming flow of data which demands a paradigm shiftin the computing architecture requirements and large-scaledata processing mechanisms. Therefore, concurrent com-putations have been receiving increased attention due towidespread adoption of multi-core processors and the emerg-ing advancements of cloud computing technology. For exam-ple, the MapReduce framework [?] has been introduced as ascalable and fault-tolerant data processing framework thatenables the processing of a massive volume of data in paral-lel on clusters of horizontally scalable commodity machines.By virtue of its simplicity, scalability, and fault-tolerance,MapReduce is becoming ubiquitous, gaining significant mo-mentum within both industry and academia. However, theMapReduce framework, open sourced by the Hadoop1 Imple-mentation, and its related large-scale data processing tech-nologies (e.g. Pig2 and Hive3) have been mainly designed forsupporting batch processing tasks but they are not adequatefor supporting real-time stream processing tasks [?]. Theubiquity of mobile devices, location services, sensor perva-siveness and real-time network monitoring have created thecrucial need for building scalable and parallel architecturesto process vast amounts of streamed data.

In general, distributed stream processing systems supporta large class of applications in which data are generated frommultiple sources and are pushed asynchronously to serverswhich are responsible for processing. Therefore, stream pro-cessing applications are usually deployed as continuous jobsthat run from the time of their submission until their can-cellation. The Actor Model [?] is a mathematical model ofconcurrent computation which has enjoyed enormous suc-cess and has inspired a number of systems in this area. To-day, this field is mature and provides the proper foundationto operate a horizontally scalable system for massive paral-lel computations of big data [?, ?]. Recently, Twitter hasopen-sourced Storm4 as a parallel, distributed and fault-tolerant system which is designed to fill the gap of providinga platform that supports real-time processing of large scale

1http://hadoop.apache.org/2http://pig.apache.org/3http://hive.apache.org/4https://github.com/nathanmarz/storm/wiki

173

streaming data on clusters of horizontally scalable commod-ity machines. In principle, the Storm system represents arealization of all the principles of the actor model in the con-text of concurrent processing on streaming data. In particu-lar, it relies on the idea of creating and interconnecting inter-active universal entities, called Actors, which are responsiblefor executing tasks concurrently and asynchronously whilecommunicating with each other through a message-passingprotocol [?, ?].

Contributions. In practice, many researchers and en-gineers are currently increasingly shifting to parallel anddistributed systems for large-scale data precessing. How-ever, the lack of expertise and in-depth understanding ofthe theoretical foundation of this area interferes with the ef-ficient analytic design or with the proper resource allocationstrategies to achieve expected performance [?, ?]. In orderto tackle this challenge, we introduce, to the best of ourknowledge, the first set of performance models that formal-ize the performance characteristics of an Actor Model baseddistributed stream processing engine, i.e. Storm. This per-formance model can play a fundamental role in guiding theuser of these systems with the understanding of the systemcosts which are independent of the executed jobs; the levelof parallelization necessary to efficiently execute a particularjob; and to better predict the latency, throughput and phys-ical resource costs. In particular, we formalize three mainperformance cost components as follows:• Data Flow Costs: This component captures informa-

tion about the amount of data flowing (transmitted)through the different processing steps of a distributedstream processing job.• Data Processing Costs: This component captures in-

formation about the execution time of the differentprocessing steps within a distributed stream process-ing job.• System Management Costs: This component captures

information about the system communication overheadbetween the different components of a distributed streamprocessing job for ensuring the reliability and fault tol-erance during the execution steps.

We use these components to model the general performancecharacteristics of an Actor Model based implementation of adistributed stream processing system. Finally, we leveragethese models to drive our experimental evaluations withinthe Storm framework.

RoadMap. The remainder of this paper is organized asfollows. Section ?? defines the preliminary concepts usedin this paper about the Actor Model and the Storm sys-tem. We present the modelling of the data flow costs inSection ??. The modelling of the data processing costs ispresented in Section ?? while the modelling of the systemmanagement costs is presented in Section ??. The resultsof our experimental evaluation are presented in Section ??.We discuss the related work in Section ?? before we finallyconclude the paper in Section ??.

2. BACKGROUND AND MOTIVATIONSThis section introduces the fundamental concepts of the

Actor Model which are used in this paper and provides anoverview of the Storm system which is used as our case studyfor implementing distributed stream processing jobs usingthe concurrent computations principles of the Actor Model.

2.1 Actor Model TheoryThe Actor Model is one of the first computational models

that proposes substantial differences from the Turing Ma-chine Model [?]. In particular, the fundamental differencewith the Turing Model is the absence of global states in theActor Model. Indeed, the Turing Model makes use of globalstates that have also been described by McCarthy and Hayesin 1969, as “the complete state of the universe at an instantof time”. The Actor Model is defined under a fundamen-tally different concept of “reacting to input events”. Theseevents are messages that flow through Actors and trigger re-sponses/computations when computed in some Actor, there-fore Actors are based on the concept of Partial ordering [?].Since the arrival orderings are indeterminate, they cannot bededuced from prior information based on mathematical logicalone. This has been referred to as indeterminacy in con-current computation by Hewitt [?]. Specifically, this logicevolved into the concept of unbounded nondeterminism inwhich the possible delay in serving a request can becomeunbounded given the arbitration of contention for shared re-sources. However, an important property is still guaranteedthat “the request will eventually be serviced”. In principle,the fundamental concepts of the Actor Model focuses on in-terpreting everything as an Actor entity. The concurrencymodel introduced with the Actor Model is based on threefundamental concepts:

1. The Actors are the computational units whereby theyexecute a computation on some input and generate anoutput.

2. Actors’ input and output are realized through the con-cept of messages that are defined as the communicationunit for exchanging data among Actors.

3. Actors’ communication channels are realized throughbuffers or queues from where Actors read and send thedata to process. In particular, Actors communicatewith each others via simple primitives such as: receiptof messages from other Actors, execution of some com-putation on incoming messages, generation of outputmessages, and finally dispatch of output messages toother Actors.

In practice, jobs usually involve multiple Actors. These Ac-tors communicate among each other only through the con-cept of addresses and can be visualized as a graph topologywhere each node is an Actor and a direct edge from Actor Ato Actor B means that there is a flow of messages from A toB. However this graph may be highly dynamic where Actorshave the possibility to reconfigure their topology on the flybased on the input messages, i.e. Actors can be dynamicallyadded and removed from the system. Therefore, any con-current system implemented through the Actor concurrencymodel is theoretically perfectly parallel and scalable.

2.2 From Theory to a Real Streaming SystemThe Storm system has been presented as a distributed

and fault-tolerant stream processing system that instanti-ates the fundamental principles of Actor theory. The keydesign principles of Storm are:• Horizontally scalable: Computations and data process-

ing are performed in parallel using multiple threads,processes and machines.• Guaranteed message processing : The system guaran-

tees that each message will be fully processed at least

174

�����

����

����

����

�����

����

Figure 1: Sample Storm Topology

once. The system takes care of replaying messagesfrom the source when a task fails.• Fault-tolerant : If there are faults during execution of

the computation, the system will reassign tasks as nec-essary.• Programming language agnostic: Storm tasks and pro-

cessing components can be defined in any language,making Storm accessible to nearly anyone. Clojure,Java, Ruby, Python are supported by default. Supportfor other languages can be added by implementing asimple Storm communication protocol.

2.2.1 Design Specifics in StormThe core abstraction in Storm is the stream. A stream

is an unbounded sequence of tuples. Storm provides theprimitives for transforming a stream into a new stream in adistributed and reliable way.

The basic primitives Storm provides for performing streamtransformations are spouts and bolts. A spout is a source ofstreams. A bolt consumes any number of input streams, car-ries out some processing, and possibly emits new streams.Complex stream transformations, such as the computationof a stream of trending topics from a stream of tweets, re-quire multiple steps and thus multiple bolts.

A topology is a graph of stream transformations whereeach node is a spout or bolt. Edges in the graph indicatewhich bolts are subscribing to which streams. When a spoutor bolt emits a tuple to a stream, it sends the tuple to everybolt that subscribed to that stream. Links between nodesin a topology indicate how tuples should be passed around.

Each node in a Storm topology executes in parallel. In anytopology, we can specify how much parallelism is requiredfor each node, and then Storm will spawn that number ofthreads across the cluster to perform the execution. Fig-ure ?? depicts a sample Storm topology.

Communication Semantics. The Storm system re-lies on the notion of stream grouping to specify how tuplesare sent between processing components. In other words, itdefines how that stream should be partitioned among thebolt’s tasks. In particular, Storm supports different types ofstream groupings such as:

1. Shuffle grouping where stream tuples are randomly dis-tributed such that each bolt is guaranteed to get anequal number of tuples.

2. Fields grouping where the tuples are partitioned by thefields specified in the grouping.

3. All grouping where the stream tuples are replicatedacross all the bolts.

4. Global grouping where the entire stream goes to a sin-gle bolt.

In addition to the supported built-in stream grouping mech-anisms, the Storm system allows its users to define their owncustom grouping mechanisms.

WORKER (node A)

SPOUT (task 1)

Spout Implementation

OutputCollector

emit() Transfer Queue

b: partition, serialize(remote tasks)

BOLT (task 2)

Receive Queue OutputCollector

Bolt Implementation

nextTuple()

Read Loop

TQ Loop

1: take()

a: p

artit

ion

(loca

l tas

ks)

RQ Loop

1: take()

2: execute()

emit()

a: partition(local tasks)

WORKER (node B)

BOLT (task 3)

Receive Queue OutputCollector

Bolt ImplementationRQ Loop

1: take(),deserialize

2: execute()

emit()

a: partition(local tasks)

Transfer Queue

b: partition, serialize(remote tasks)2:

tran

sfer

ser

ializ

ed tu

ple

b: partition, serialize(remote tasks)

put tuple

SocketRead Loop

Figure 2: Execution of a Storm Topology

Storm Cluster. In general, a Storm cluster is super-ficially similar to a Hadoop cluster. One key difference isthat a MapReduce job eventually finishes while a Storm jobprocesses messages forever (or until the user kills it). Inprinciple, there are two kinds of nodes on a Storm cluster:• The Master node runs a daemon called Nimbus (sim-

ilar to Hadoop’s JobTracker) which is responsible fordistributing code around the cluster, assigning tasksto machines, and handling failures.• The worker nodes run a daemon called the Supervi-

sor . The supervisor listens for work assigned to itsmachine and starts and stops worker processes as nec-essary based on what Nimbus has assigned to it.

In a Storm cluster all the interactions between Nimbus andthe Supervisors are done through a ZooKeeper5 cluster, anopen source configuration and synchronization service forlarge distributed systems. Both the Nimbus daemon and Su-pervisor daemons are fail-fast and stateless, where all stateis kept in ZooKeeper or on local disk.

Communication between workers living on the same hostor on different machines is based on ZeroMQ sockets overwhich serialized java objects (representing tuples) are beingpassed.

2.2.2 The Execution Steps of Storm TopologiesIn practice, a storm job is called a topology which continu-

ously processes stream(s) of incoming tuples (events) withina Java Virtual Machine (JVM). Figure ?? illustrates the var-ious phases in processing a tuple in a sample topology con-sisting of a single spout (task 1) and two bolts (task 2 andtask 3) distributed over a 2 node cluster (Worker Node A andWorker Node B). The figure should be read starting at theRead Loop part of the spout. Here, tuples are continuouslybeing read from an unspecified tuple source, which could forexample be a text file or a queue. The spout’s nextTuple()method is called continuously. The user-defined implemen-tation of this method can for example read the next lineof text or read the next tuple of the queue and transformthe read data into a new tuple (i.e. a sequence of key-valuepairs). Then, the implementation calls the emit() functionon the spout’s OutputCollector object to actually inject thetuple into the storm cluster. The OutputCollector appliesthe stream groupings onto the tuple which associates one ormore task ids for each tuple. A task id uniquely refers to aparticular task process/thread in the cluster where the tuple

5http://zookeeper.apache.org/

175

should be sent for processing. If it turns out that the taskid refers to a local task, it will simply add the tuple onto thereceive queue of the local task (task in the same worker).Otherwise, it needs to send the tuple to another worker (i.e.JVM), either on the same node or a different node in thecluster and therefore it is first serialized and then added tothe transfer queue. The continuous loop, (TQ Loop), readsthe serialized tuples off the transfer queue and sends themto the destination worker(s).

As shown in Figure ??, the tuples emitted by the spoutare sent directly to the subsequent bolt which is executingtask 2. As this is a local task which is running in the sameworker as the spout, the tuple is directly placed onto thereceive queue of the local bolt. The RQ Loop continuouslyreads tuples off the receive queue and calls the user-definedexecute() method on the bolt referred to by the task id as-sociated with that tuple. When the implementation decidesthat a new tuple should be triggered, it should construct anew tuple and emit it to the bolt’s OutputCollector. Inthe case of task 3, which is a remote task, the new tupleis first serialized and then placed on the worker’s transferqueue. The TQ Loop will read the tuple and send it to nodeB where the tuple will be placed on the receive queue of task3. The RQ Loop at node B will read the tuple, deserializeit and the same process as described above continues.

3. DATA FLOW COST MODELThe data flow cost model is designed to capture informa-

tion about the size of data flowing through an active stormtopology and the cost of transferring these data between theprocessing actors.

3.1 Data SizeThe size of data flowing in a Storm job is expressed per

time unit (e.g. per second) and is composed of the followingtwo components:• The input data size: It represents the total amount

of tuples which are injected into the storm cluster bythe various spouts in the topology.• The output data size: It represents the total amount

of tuples which are generated and emitted by the vari-ous processing bolts in the topology and are consumedduring further processing by other processing bolts.

Input Data Size In general, an input spout can generateinput tuples of different types. The data size of an inputspout per time unit can therefore be modeled as:

S.dataSize =n∑

t=1

S.numTuplest ∗ tupleSizet

where S refers to a particular input spout, n refers to thenumber of different types of input tuples produced by thatspout, S.numTuplest refers to the average number of tuples(of type t) emitted by spout S per time unit and tupleSizetrepresents the average byte size of tuples of type t. Thetotal size of data input per time unit of a particular stormtopology is then defined as follows:

T.inputSize =s∑

i=1

Si.dataSize

where T refers to a particular topology and s refers to thetotal number of input spouts for that topology.

Output Data Size Each processing bolt of a topologyconsumes at least one stream of tuples and possibly gen-

erates and emits one or more new streams of tuples. Thenumber of output tuples of a processing bolt for a specificinput tuple type t per time unit is defined as follows:

B.numTuplest =m∑

d=1

Ed.numTuplest ∗B.selectivityt

where Ed refers to a source executor (i.e. spout or bolt) thatemits tuples for which B is a destination, m is equivalentto the number of direct edges in the topology graph froman executor E to bolt B, and B.selectivityt represents theratio between the number of input tuples of type t and thenumber of new output tuples being emitted as the resultof processing those input tuples by a particular bolt B. Inprinciple, each processing bolt can have different selectivityvalues for the different types of tuples it can process and itcan produce tuples of multiple types as output. ThereforeB.selectivityt can be expanded to:

B.selectivyt =n′∑

t′=1

B.selectivityt,t′

where n′ refers to the number of different types of outputtuples produced by bolt B in response to receiving tuples oftype t. As a consequence, the output data size of a process-ing bolt per time unit is defined as follows:

B.dataSize =n∑

t=1

B.numTuplest ∗ tupleSize

where tupleSize represents the average size of output tuples,which can be further expanded as:

n∑t=1

m∑d=1

(Ed.numTuplest ∗

n′∑t′=1

(B.selectivityt,t′ ∗ tupleSizet′)

)where n refers to the number of different input tuple typesreceived by bolt B and n′ refers to the number of differentoutput tuple types produced and emitted by B.

Topology Data Flow Size. The total data flow size fora storm topology can then be represented as the summationof the total data input size injected by all spouts into thetopology and the total data output size emitted by all theprocessing bolts in the topology. This can be expressed as:

T.dataSize =s∑

i=1

Si.dataSize +b∑

j=1

Bj .dataSize

where s refers to the total number of spouts and b refersto the number of bolts which are defined in the executedtopology.

3.2 Data Transfer CostThe data transfer cost is the cost of actually delivering the

tuple to a destination task (i.e. bolt instance). A distinctionmust be made between various allocations of communicat-ing spouts and bolts across the cluster because running tasksremotely or locally produces substantially different commu-nication costs in terms of bandwidth consumption and com-munication time, i.e. latency. Therefore, we distinguishbetween three categories of data transfer cost (α) as follows:

1. Local Java virtual machine (JVM) allocation(αlocal): where communicating actors (i.e. spout→boltor bolt→bolt) are allocated within the same workerand thus within the same JVM. In this case, the tu-ple serialization/ deserialization step via the transferqueue is by-passed and tuples are immediately placedon the consuming bolt’s receive queue.

176

2. Local node allocation (αnode): where communicat-ing actors are allocated on different workers that arehosted on the same cluster node. In this case, a seri-alization/ deserialization cost to send the tuple fromone JVM to another is incurred.

3. Remote allocation (αremote): where communicat-ing actors are allocated on different workers hosted ondifferent cluster nodes. This is the most expensive caseas both a serialization/deserialization cost and a net-work transfer cost are incurred.

In practice, Storm’s allocation strategy sequentially and it-eratively allocates all instances of each component (spoutor bolt) up to the configured parallelization factor, wherebyan outer loop iterates across all workers and an inner loopiterates across all nodes in the cluster. In the following sub-sections, we describe how the fraction of tuples that can beprocessed according to each category can be calculated.

Calculating the Local JVM Allocation. Let P bethe collection of worker processes running across the clus-ter and let D be the collection of communicating actors(spout→bolt or bolt→bolt) in the topology. For each pro-cess p ∈ P and for each communicating actor d ∈ D in thesender role (i.e. emitting tuples), let Aemitterp,d be thenumber of instances of that actor that are allocated in theprocess p. Similarly, for each process p ∈ P and for eachcommunicating actor d ∈ D in the receiver role (i.e. receiv-ing tuples), let Areceiverp,d be the number of instances ofthat actor that are allocated within the same process p. Fi-nally, for each communicating actor d ∈ D, let us assumethat Aemitterd and Areceiverd represent respectively thetotal number of instances of the sending and receiving ac-tors across the entire cluster. It follows that the fractionof tuples which are processed within a single JVM can becomputed as:

αlocal =

|D|∑d=1

|P|∑p=1

(Aemitterp,d∗Areceiverp,d

)|D|∑d=1

Aemitterd∗Areceiverd

Calculating the Local Node Allocation. Let n ∈ Nbe a node in the cluster (where |N | is the cluster size), andAemittern,d and Areceivern,d represent the number of in-stances of that actor d, respectively in the sender and re-ceiver role, that are allocated on the same node n. Thefraction of tuples that are processed among tasks within asingle node αnode can then be derived by computing the dis-tribution of the tasks among the nodes of the cluster, whichwe call αintra node, given by the following formula:

αintra node =

|D|∑d=1

|N|∑n=1

(Aemittern,d∗Areceivern,d

)|D|∑d=1

Aemitterd∗Areceiverd

Thus, the fraction of tuples that are processed on the samenode but on a different processes:

αnode = αintra node − αlocal

Calculating the Remote Allocation. Given theabove formulas, the fraction of tuples that are transferredacross the network to another node, contributing to the over-all data transfer cost can be expressed as follows:

αremote = 1− αnode

Overall Data Transfer cost. Given the informationof the fraction of tuples that are processed in each node andin each JVM, we can now compute the total IO cost pertime unit as follows:

Let us assume that the stream comprises of a single tupletype and let Γlocal, Γnode and Γremote be respectively theCPU cost of sending a tuple to the local queue (i.e. withinthe same JVM), to the same node (but on a different JVM),and to different nodes in the cluster, the total IO cost pertime unit can be represented as follows:

IOCost = numTuples ∗ IOTranfer

where

IOTransfer = αlocal∗Γlocal+αnode∗Γnode+αremote∗Γremote

In the case where the stream comprises different types oftuples, the IOCost would be computed for each tuple typet and summed together which leads to:

IOCost =n∑

t=1

numTuplest ∗ IOTransfert.

In the above formulas, numTuples and numTuplest referrespectively to the total amount of tuples and those tuplesof type t, flowing through the system.

4. DATA PROCESSING COST MODELThe data processing cost model describes the execution

time of the different processing steps within an active stormtopology as well as the IO cost of exchanging tuples betweenspouts and bolts. The model is again expressed per time unitand has the following two main components:• Spout processing cost : This represents the cost of read-

ing a raw event of an unspecified source and injectinga storm tuple into the topology.• Bolt processing cost : This represents the cost of pro-

cessing a tuple in a bolt and possibly emitting newtuples into the topology for further processing.

In the following, we provide an in-depth analysis of theCPU processing costs for spouts and bolts, by identifyingall the logical operations that are a factor of these costs.

4.1 Spout Processing CostFigure ?? illustrates the data processing steps in the pro-

cessing component (e.g. spout or bolt) which can be split ina number of sequentially executed phases:

1. Read : reads a raw data item from an event source (e.g.a queue or a file).

2. Transform (γ): formats the raw data item into a stormtuple sequence.

3. Emit : adds the storm tuple sequence to the spout’soutput collector. There will be one destination perdirect edge from the spout to a bolt in the topology.

4. Partition: defines where (i.e. which task) to send thetuple sequence to for each destination defined in theemit step. Depending on the chosen stream grouping,this could be one or more tasks (i.e. bolt instancesidentified by a node and port pair).

5. Serialize: transforms the tuple sequence in a formatthat can be transmitted efficiently across the networkor across JVMs. This phase will only be executed whenthere is at least one remote destination, i.e. a bolt taskthat is not running within the emitting spout’s JVM.

177

Figure 3: Data Processing Steps in a ProcessingComponent

The Read, Transform and Emit phases are part of the user-defined implementation logic of the spout, while the remain-ing phases are part of the Storm platform.

The CPU processing cost for an input spout (S) per timeunit can therefore be modeled as:

S.CPUCost =n∑

t=1

S.numTuplest ∗ S.CPUCostt

with

S.CPUCostt = S.inputCostt + S.outputCostt

where

S.inputCostt = S.CPUReadt + S.CPUTransformt

and where

S.outputCostt =

S.numDestt∑i=1

Bi.groupingCostt

+ S.hasExternalTaskt ∗ CPUSerializetwhere• n is the number of the different types of tuples which

are emitted by the spout S.• S.numTuplest is the number of tuples of type t, emit-

ted by spout S per time unit.• CPUReadt is the CPU cost of reading a raw input

tuple of type t.• CPUTransformt is the CPU cost of transforming a

raw input tuple of type t into a Storm tuple of type t.• S.numDestt is the number of destination bolts that

consume tuples of type t emitted by spout S.• Bi.groupingCostt represent the cost of partitioning

the input tuples of type t which varies according tothe stream grouping algorithm (Shuffle, Fields, All,Global) chosen by the processing bolt Bi.• S.hasExternalTaskt is a boolean variable which stores

the value 1 if the tuples of type t emitted by spout S areconsumed by at least one external task. An externaltask is an instance of a bolt running on a differentworker than the one which is hosting the instance ofspout S that is emitting the tuple. Note that thiscould still be on the same host, but on a different JVM.Otherwise it stores the value 0.• CPUSerializet is the CPU cost of serializing a tuple

of type t.

4.2 Bolt Processing CostIn practice, bolt processing involves running through the

same sequence of phases as spout processing with the onlydifference being that of instead of having a transform phase,there is a more general execute phase in which the actualbolt’s processing logic is executed. It is also in this executephase that the decision will be made as to which and howmany new tuples (of type t′) will be emitted by the bolt.Therefore, the CPU processing cost for a processing bolt(B) per time unit is modelled as the sum of the cost to pro-cess all input tuples of type t received per time unit and thecost of processing all resulting output tuples (of type t′), forall possible tuple types this bolt can receive. This formulais shown as:

B.CPUCost =n∑

t=1

B.numInputTuplest ∗B.inputCostt

+n′∑

t′=1

(B.numOutputTuplest,t′ ∗B.outputCostt′)

with

B.inputCostt = B.CPUReadt +B.CPUExecutet +B.isExternalTask ∗ CPUDeserializet

and

B.outputCostt′ =B.numDestt′∑

i=1

Bi.groupingCostt′

+ B.hasExternalTaskt′ ∗ CPUSerializet′

where• B.numInputTuplest is the number of tuples of typet received by the bolt B, which, as was shown before,can be represented as:

B.numInputTuplest =

m∑d=1

Ed.numTuplest

where Ed refers to the source executor (spout or bolt)that emits tuples for which B is a destination, m isthe amount of executors that emit to this bolt andEd.numTuplest refers to the number of tuples of typet emitted by executor Ed

• B.numOutputTuplest,t′ is the number of new tuplesof type t′ generated and emitted by the bolt B in re-sponse to receiving input tuples of type t, which canbe expressed as:B.numOutputTuplest,t′ =B.numInputTuplest ∗B.selectivityt,t′

• B.CPUReadt is the cost of reading an input tuple oftype t by bolt B, excluding potential deserializationcost.• B.CPUExecutet is the cost of the algorithm imple-

mented by bolt B.• B.isExternalTask is a boolean variable that stores

the value 1 if the tuple being read was sent by an exter-nal task and is therefore in serialized form. Otherwise,it stores the value 0.• CPUDeserializet is the CPU cost of deserializing a

tuple of type t.

4.3 Topology costThe total processing cost for a storm topology represents

the summation of the total CPU processing cost for allof its spouts and bolts. Therefore, the topology cost, i.e.

178

ZooKeeper

Nimbus

Supervisor Worker

1. Synchronize Topology

(check current assignments

and reassign if necessary)

2. Synchronize Supervisors

(read assignments and reassign

if necessary)

4. Supervisor Heartbeat

(update run-time information

about supervisor)

3. Synchronize Worker

(check active storms, check

assignments and connections +

re-establish if necessary)

5. Worker Heartbeat

(update run-time information

about each executor/task)

Supervisor WorkerWorkerWorker

Figure 4: ZooKeeper interactions

T.CPUCost, for a storm topology is defined as:

T.CPUCost =s∑

i=1

Si.CPUCost +b∑

j=1

Bj .CPUCost

where s represents the total number of spouts in the topol-ogy and b represents the total number of bolts in the topol-ogy.

5. SYSTEM MANAGEMENT COST MODELThe System Management costs model the impact of the

set of tasks, abstracted away from the user, which are re-quired to run a Storm cluster. These tasks include provisionof support for node failure/addition/removal, JVM failures,network issues etc. Storm’s system management tasks arecoordinated through ZooKeeper and are mainly related tothe interaction between ZooKeeper and Nimbus, Supervisorand Worker as shown in Figure ??. There are 5 recurringSystem Management tasks that interact with ZooKeeper,and the frequency of these interactions can be configuredmanually. These parameters are described in the followinglist:

1. Synchronize Topology: Every configurable St sec-onds, Nimbus checks the active assignments and com-pares them to the required assignments according tothe topology specification. If a difference is detected,e.g. because of node failure, Nimbus will reassign theunassigned tasks over the available worker processesin the cluster. (period: St, number of requests: Rt,bandwidth: Bt)

2. Synchronize Supervisors: Every configurable Ssseconds, each supervisor reads its assignments fromZooKeeper and re-assigns them if it detects a differ-ence between what it has currently assigned across itsworkers. Re-assignment takes the form of updates toZooKeeper’s assignments for the workers to query dur-ing their next poll cycle.(period: Ss, number of requests: Rs, bandwidth: Bs)

3. Synchronize Workers: Every configurable Sw sec-onds, each worker reads its assignments from ZooKeeper.If there is a mismatch, the missing connections are es-tablished. In addition to this, each worker also checks

the active storm topologies. If the storm topology forwhich it is running tasks would no longer be active (be-cause it was explicitly killed), the worker would needto stop processing.(period: Sw, number of requests: Rw, bandwidth: Bw)

4. Supervisor Heartbeat: Every configurable S ′s sec-onds, each supervisor will send a heartbeat to ZooKeeper.A heartbeat takes the form of storing some run-timeinformation about the supervisor in ZooKeeper.(period: S ′s, number of requests: R′s, bandwidth: B′s)

5. Worker Heartbeats: Similarly, every configurableS ′w seconds, each worker sends heartbeats to ZooKeeper.A worker heartbeat includes statistics information abouteach task running in that worker.(period: S ′w, number of requests: R′w, bandwidth: B′w)

In principle, the System Management Cost has two compo-nents: a component that reflects the load on ZooKeeper,expressed in number of requests per time unit (R) and anetwork load component which represents the network band-width consumption (B).

5.1 Number of requests to ZooKeeperThe number of ZooKeeper requests per time unit is a

function of the amount of nodes (i.e. supervisors) and theamount of workers in the cluster as reflected by the followingequation:

R =Rt

St+ numNodes ∗

(Rs

Ss+R′sS ′s

)

+numWorkers ∗(Rw

Sw+R′wS ′w

)whereRt,Rs,Rw,R′s andR′w are all fixed (implementation-related) constants and Ss, Sw, S ′s and S ′w are configurableconstants. It should be noted that the number of requestsper time unit is independent of the amount of tasks/componentsinstantiated around the cluster6 and is linear with the sizeof the cluster.

5.2 Network Bandwidth ConsumptionThe network bandwidth consumption depends both on the

number of supervisors and workers in the cluster, and alsoon the number of tasks running on those workers. Specifi-cally, an increase in the number of workers leads to moreworker heartbeat messages per time unit as each workersends the heartbeats separately; an increase in number ofsupervisors leads to more assignment info messages beingexchanged with ZooKeeper as each supervisor is in chargeof managing its own assignments and finally, an increase inthe number of tasks increases the size of both the workerheartbeat and assignment info messages as they contain in-formation that are specified to each task.

In order to present a formula to calculate the bandwidthconsumption of system management tasks, we will first lookat two of the most important data structures used to com-municate with ZooKeeper, i.e. Worker Heartbeat (WHB)and Assignment Info (AI).

Worker Heartbeats A Worker Heartbeat data struc-ture contains information about the status of the various

6with the caveat as explained above that numWorkers is adynamic thing in case numTasks is smaller than the sum ofthe configured maximum number of workers on each node.

179

tasks assigned to a worker. The byte size of one round ofworker heartbeats can be defined as:

numWorkers∑i=1

|WHBi| = a ∗ numTasks+ b ∗ numWorkers

where a and b are constants related to the specific imple-mentation of this data structure in Storm, which will not befurther detailed to improve readability.

Assignment Information Assignment information con-tains a mapping from task ID to host and port and thetimestamp of when the component was started.

The byte size of the assignment information data structureis given by

|AI| = c ∗ numNodes+ d ∗ numTasks+ e

where again c, d and e are constants related to the specificdesign of the AI data structure, which will not be furtherdetailed. It should be noted that the overhead of the AI isindependent of the amount of workers per node.

Bandwidth As was mentioned previously, an approxi-mation of system management bandwidth consumption canbe written as the sum of the bandwidth consumed by eachof the recurring tasks as follows:

B =|Rt|St

+ numNodes ∗(|Rs|Ss

+|R′s|S ′s

)+numWorkers ∗

(|Rw|Sw

+|R′w|S ′w

)where |X | is used to denote the size (in bytes) of X ; numNodesis the number of nodes in the Storm cluster running a su-pervisor; numWorkers is the sum of all active workers onall nodes in the Storm cluster. A worker becomes active ifit has at least one task assigned to it.

By leveraging an in depth analysis that correlates Rt, Rs,Sw, S ′s and S ′w with numNodes, numWorkers, |AI| and|WHB| (which we do not report here for brevity), we canexpress the System Management Bandwidth as:

B = k1 + k2 ∗ |AI|+ numNodes ∗ (k3 + k4 ∗ |AI|)

+ numWorkers ∗ (k5 + k6 ∗ |AI|) + k7 ∗∑i

|WHBi|

where all of k1 through k7 are constants.In conclusion, with this final formulation we can see that

ZooKeeper traffic bandwidth increases proportionally withthe number of tasks and nodes running in the cluster, whichis also validated by the experiment reported in Figure ??.In the figure we plot the measured BW (in KB) attributedto ZooKeeper communication as a function of the numberof tasks running around the cluster, for various cluster sizes.Note also that ZooKeeper communication is independent ofthe specific algorithms implemented in spouts or bolts.

6. EXPERIMENTAL EVALUATIONThis section provides a detailed experimental evaluation

of the performance characteristics of the Storm system us-ing real industrial datasets. This section aims to experimen-tally quantify the performance variation in running different

0  

20  

40  

60  

80  

100  

120  

140  

0   20   40   60   80   100   120   140   160   180  

BW  (K

B/s)  

Number  of  tasks  

1  node  

2  nodes  

3  nodes  

4  nodes  

5  nodes  

Linear  (1  node)  

Linear  (2  nodes)  

Linear  (3  nodes)  

Linear  (4  nodes)  

Linear  (5  nodes)  

Figure 5: System management traffic as a functionof the number of components

Storm configurations by leveraging insights from the perfor-mance model. Furthermore, the intent of this experimentalanalysis is to detect the causes of this variability in the sys-tem performance and shed light on the precautions requiredin order to run the system under optimal configurations.

6.1 Dataset and Environment SetupThe experiments have been run with a real Telco dataset

consisting of Performance Management (PM) observationsof a large Femtocell7 network. Femtocells were designed foruse in a home or small business to improve localized cellularservice and offload bandwidth usage from macrocells (i.e.,traditional cell towers). During operation, Femtocells pro-duce many low-level performance logs (e.g. number of suc-cessful handovers, number of call initiation attempts) acrossa range of performance categories (e.g. packet data perfor-mance, handover performance). The considered dataset iscomposed of hourly PM logs collected over 15 days for a net-work of 70k Femtocells, totalling approximately 11 millionXML files. These files contain a list of 128 key-value pairs ofoperational and statistical counters with a mixture of integerand floating-point values. These data are usually analyzedto monitor network performance characteristics and predicttraffic peaks or failures. The experiments presented in thispaper deploy a simple topology composed of two actors (onespout and one bolt) whose task it is to identify the Femto-cells which require the most bandwidth. The spout readsinput messages from an external queue (this corresponds tothe XML data) and produces a stream of tuples. The boltreceives the stream of tuples with a shuffle grouping andemits the Femtocell list.

Cluster Configuration. Each test runs on a clustercomprising 5-nodes of identical configuration. Additionalmachines were used to generate load into this cluster and tohost the external message queue(s). Each machine is a dual4-core Intel Xeon 3Ghz 32bit 16 GB Memory, 1 Gb/s net-work interface. Nodes are interconnected through a 10GbOmniSwitch 6850 Ethernet Switch. Each node runs Linuxversion 2.6.32-220.4.1.el6.i686. The following software com-ponents were used: Java 1.6 OpenJDK Runtime Environ-ment (IcedTea6 1.10.4), ZeroMQ 2.1.7, Zookeeper 3.4.2 andKestrel 2.3.4. We used a modified version of Storm, Storm-0.8.0-SNAPSHOT (version of July 2012), to include timers

7Alcatel-Lucent 9360 Small Cell. http://www.alcatel-lucent.com/small-cells/

180

Spout Parallelisation0 5 10 15 20 25 30 35

Late

ncy

[mse

c/tu

ple]

0

100

200

300

400

500

Thr

ough

put [

tupl

es/s

ec]

0

2000

4000

6000

8000

10000

(a) Effect of Spout Parallelisation

Bolt Parallelisation0 5 10 15 20 25 30 35

Late

ncy

[mse

c/tu

ple]

1.5

2

2.5

3

3.5

4

4.5

5

Thr

ough

put [

tupl

es/s

ec]

020040060080010001200140016001800200022002400

(b) Effect of Bolt Parallelisation

Figure 6: Measuring latency and throughput on the different system configurations.

and related statistical tools for the purpose of the experi-ments.

6.2 Experiment ConfigurationsA comprehensive test automation suite was developed which

is composed of various shell scripts and Java programs toautomatically execute the Storm topologies and collect run-time statistics on the system performance. Storm configura-tions were tested while varying the number of nodes, spouts,bolts and workers. Each specific configuration was tested 10times in order to accumulate statistical information. Care-ful consideration was taken to ensure that each configurationwas tested under the same operational conditions.

Each individual test case is executed on a clean cluster,meaning that all processes from a previous run were killed onall cluster nodes before starting the new ones. In addition,all data generated by the previous run is erased from thefilesystem. Each test consists of an initial warm-up phaseof 1 minute (in which we do not collect statistics becausethe system may not be in its computational steady stateyet) followed by a 5 minute measuring phase during whichstatistics are being gathered, including:

1. Throughput (µ) which represents the total number ofevents processed per second.

2. Latency (L) which represents time to process a tupleboth within a single actor and within the entire system.

3. External and internal queue sizes and their growthrate.

4. Network bandwidth in terms of the number of mes-sages exchanged among workers on different nodes.

5. Various statistics on system management overhead (e.g.communication with ZooKeeper).

6. Approximate memory usage and CPU usage per thread.Each test is run with a different Storm configuration which

is determined by the following parameters:• Parallelization factor : represents the number of tasks,

from {1, 2, 4, 8, 16, 24, 32}, instantiated per actor.• Cluster size: represents the number of nodes partici-

pating in the Storm cluster, i.e. from 1 to 4.• Worker pool size: represents the number of workers

(JVMs) per node, which host the tasks. In our experi-ment this number is selected from {1, 2, 4, 8, 16, 24, 32}.• Event injection rate: represents the number of events

injected per second into the queue which provides datato the spouts. In our experiments the event injectionrate varies among three different rates, 5K, 10K and50K tuples per second, to observe how Storm adjuststo different conditions.

6.3 Experimental AnalysisIn order to optimize the system performance while run-

ning Storm jobs there are several variables and parameterswhich need manual configuration. The configuration is thusa complex task that requires a precise knowledge of the mostrelevant parameters and how they impact the system per-formance. In particular, we focus on the following metrics:the throughput (µ), the latency (L) and the system resourceutilization (CPU, memory, network bandwidth).

In the following subsections we present a set of experi-ments that aim to shed light on how to configure this sys-tem based on the theoretical understanding derived from theperformance model presented in the previous sections.

6.3.1 Performance Variability for Different SystemConfigurations

The experiments presented in this section comprise obser-vations of the impact on latency and throughput producedby different configurations in terms of parallelization, work-ers and cluster size.A configuration is expressed as the specification of the sys-tem parameters, i.e. number of nodes, spouts, bolts, andworkers (hereafter simply < n, s, b, w >). For brevity, weonly report experiments run on a single node cluster withexternal queue fed with 50k messages per second. However,the same experiments run on a 1 through 4 node clustershow similar results, with only a few small deviations at-tributed to the increased system management costs to runmore nodes which will be discussed in detail in the nextsections.

Spout Parallelisation In order to study the impact ofthe number of spouts on the overall system performance, wefixed all other parameters at 1 while varying the numberof spouts in {1, 2, 4, 8, 16, 24, 32}. Figure ?? illustrates theeffect upon throughput (in terms of tuples per second) andthe latency (in terms of milliseconds to process a tuple) asthe number of spouts is adjusted. The system throughputcan be increased by increasing the number of spouts - how-ever as the number of spouts continues to increase beyondsome threshold, the throughput declines. This can be under-stood by observing the latency which exhibits exponentialgrowth. Beyond some threshold (determined by the systemhardware), the system is overstressed with many processesand the context switch among them kills the system perfor-mance.

Bolt Parallelisation In order to study the impact ofthe number of bolts on the overall system performance, we

181

fix all other parameters at 1 while varying the number ofbolts in {1, 2, 4, 8, 16, 24, 32}. Figure ?? illustrates the ef-fect upon throughput (in terms of tuples per second) andthe latency (in terms of milliseconds to process a tuple) asthe number of bolts is adjusted. An increase in bolt par-allelisation reduces throughput due to the extra CPU loadassociated with scheduling. However, a reduction in latencytowards a lower limit is also observed when the number ofbolt is within [8, 24] range. This limit represents the fastestpossible bolt execution time, which is the cost of the systemfrom the emission of the tuple by the spout up to the com-pletion of the algorithm implemented by bolt B, defined inthe performance model in Section ??.

6.3.2 Costs of Distributing ComputationThis section aims to quantify the variation in performance

when varying the number of workers, and the way communi-cating tasks are distributed across these workers. Section ??showed that the dataflow cost depends on whether two com-municating tasks are hosted (a) within the same JVM, (b)within the same node but on different JVMs, or (c) on dif-ferent nodes. Specifically, cases (b) and (c) incur a serializa-tion/deserialization cost, while case (c) additionally incursnetwork transfer cost. Here we demonstrate its practicalimpact.

Cost of Communicating JVMs Figure ?? displaysthe system latency L when some tasks are assigned to thesame physical node, but to different JVMs running on thatnode. Each point in the plot represents a different systemconfiguration. On the x-axis we report the specific configu-ration setting, i.e. the actual instantiation of < n, s, b, w >and on the y-axes we plot two different results: the left-handside axis displays the latency, in milliseconds, of the systemfor each configuration while the right-hand side y-axis es-timates αnode defined in Section ??, i.e. the percentage oftuple exchanges across different JVMs on the same physi-cal node. In this experiment, an increment of the latencycorresponds to the cost of transferring the associated tuplesfrom one JVM to another, which mainly comes from theserialization/ deserialization costs. We show that when thetheoretical approximation of αnode increases the correspond-ing latency for that configuration goes up, which means thatαnode can be used to predict the system latency of a par-ticular configuration. Within the yellow box, starting withconfiguration < 1, 8, 8, 1 >, we observe that the system fallsin an unstable configuration in the sense that the serializa-tion/deserialization costs are not the dominant costs any-more and the latency degrades from 5ms to 28ms in the lastconfiguration. We have increased the number of tasks andwe can observe that the system is affected by extra costs suchas: management overhead, Java garbage collection, schedul-ing and context switching among JVMs.

Impact of Increasing Workers. One of the bene-fits of a distributed system is its capacity to increase theamount of parallel threads of execution and reliably dis-tribute them over the cluster nodes so as to improve pro-cessing efficiency in time and capacity. However, the num-ber of allocable JVMs per single node has a direct impact onperformance. This number is highly related to the numberof CPUs/cores per node, therefore, the number of allocablethreads per worker per CPU needs to be carefully calibratedto achieve optimal system utilization before undesirable ef-

Figure 7: The latency for different system configu-rations.

fects on the performance show up. Figure ?? and Figure ??show the throughput and the latency for different systemconfigurations. In this experiment, we highlight the effect ofinstantiating from 1 worker with 16 threads to 16 workerseach with 1 thread. The direct proportional effect of the tu-ple serialization/deserialization cost on the latency, which isclearly visible in Figure ??, is no longer apparent for higherthread counts. In fact, for higher thread counts, Figures ??and ?? show that we can improve throughput and latencyby instantiating multiple workers, each with a lower threadcount to achieve the same overall level of parallelism.

This experiment shows that when there are more threadsthan cores the system performance can have unexpected be-havior. For instance, if the number of threads per JVM istoo high, as in configuration < 1, 8, 8, 1 > with 16 threadsper JVM, the throughput is lower than when there is onlya single thread per JVM as in configuration < 1, 8, 8, 16 >.One of the factors explaining this observed behavior is theincreased JVM garbage collection activity for higher threadcounts. Figure ?? and ?? confirm that this trend is the samefor all cluster sizes.

Bandwidth Consumption. As we have previouslyshown, the more cluster nodes we leverage, the better through-put we may expect to achieve. However, the participatingnodes need to communicate by transferring tuples amongeach other, which comes at a cost that is constrained by theavailable network bandwidth.

In the simple topology we have used for our experiments,we can estimate total bandwidth consumption as follows:BWtotal = BWexternal +BWsystem +BWdataflow

where• BWexternal is the bandwidth required by the spout to

read raw events from the event source (i.e. a Kestrelqueue in our experiments).• BWsystem is the system management bandwidth we

calculated in Section ??. For simplicity, we can over-estimate this portion of the overall bandwidth by tak-ing an upper bound for at least 100 active tasks asreported in Figure ??, i.e. 56K bytes per second forthe 2-node configuration.• BWdataflow is the bandwidth related to exchanging

tuples between spout and bolt instances. For this por-tion, we can use the formulas presented in Section ??.More specifically, BWdataflow = numTuples∗αremote∗tupleSize where numTuples represents the amount oftuples emitted by the spout per time unit.

From measurements performed on our dataset, we havedetermined that the size of an external event is 360 bytes

182

(a) Effects of the parallelism on the Throughput (µ) (b) Effects of the parallelism on the Latency (L)

Figure 8: Measuring latency and throughput with 1 to 16 JVMs respectively with 1 to 16 threads per node.

and the size of a transformed event, i.e. a Storm tuple,is 44 bytes. Therefore, in this experiment, the bandwidthconsumption can be computed using the following formula:

BWtotal = numTuples ∗ (360 + 44 ∗ αremote) + 56000

Figure ?? shows the calculated bandwidth consumption asa function of system configuration for clusters consisting of2,3 and 4 nodes. These bandwidth calculations were verifiedusing the Unix dstat tool.

Figure 9: System Communication Cost

7. RELATED WORKThe growing demand for large-scale data processing and

data analysis applications has spurred the development ofnovel solutions from both the industry (e.g., web-data anal-ysis, click-stream analysis, network-monitoring log analysis)and the sciences (e.g., analysis of data produced by massive-scale simulations, sensor deployments, high-throughput labequipment). MapReduce [?] is a framework which was in-troduced by Google for programming commodity computerclusters to perform large-scale data processing. The frame-work is designed such that a MapReduce cluster can scaleto thousands of nodes in a fault-tolerant manner. However,the basic architecture of the MapReduce framework requiresthat the entire output of each map and reduce task be mate-rialized into a local file before it can be consumed by the nextstage. Therefore, it is not adequate for supporting real timeprocessing of streaming data. Several approaches have beenproposed to tackle this challenge. For example, the MapRe-duce Online approach [?] has been proposed as a modifiedarchitecture of the MapReduce framework in which inter-mediate data is pipelined between operators while preserv-ing the programming interfaces and fault tolerance modelsof previous MapReduce frameworks. The Incoop system [?]has been introduced as a MapReduce implementation thathas been adapted for incremental computations which detect

the changes on the input datasets and enables the automaticupdate of the outputs of the MapReduce jobs by employinga fine-grained result reuse mechanism. In particular, it al-lows MapReduce programs which have not been designed forincremental processing to be executed transparently in anincremental manner. The M3 system [?] has been proposedto support the answering of continuous queries over streamsof data bypassing the HDFS so that data are processed onlythrough a main-memory-only data-path and totally avoidsdisk access. In this approach, Mappers and Reducers neverterminate where there is only one MapReduce job per queryoperator that is continuously executing. While these ap-proaches could improve the performance of the MapReduceframework for incremental and streamed data processing,they are still unable to fully support the distributed realtime processing requirements of data intensive applications.

Several distributed stream processing systems have beenpresented in the literature such as Aurora [?], Borealis [?],SPADE [?], Stormy [?] and Apache S4 8. For example, inBorealis, the collection of continuously running queries aretreated as one giant network of operators. The processingof these operators is distributed to multiple sites where eachsite runs an instance of the Borealis server. The query pro-cessing is controlled by an Admin component which takescare of moving query diagram fragments to and from re-mote Borealis nodes when instructed to do so by other com-ponents. SPADE is a declarative stream processing enginewhich supports a set of basic stream-relational operatorswith powerful windowing and punctuation semantics. Thesystem is designed to execute a large number of long-runningjobs that take the form of data flow graphs where each graphconsists of a set of processing elements connected by streamsand each stream carries a series of Stream Data Objects.The processing elements can communicate with each othervia their input and output ports. The concepts and ideasof our proposed models can be easily adopted to be usedwithin the context of these systems. Some studies havebeen presented for analyzing and modeling the service la-tency of event-based systems. They have been mainly fo-cusing on different classes of implementations such as thePublish/Subscribe systems [?, ?] and message-oriented mid-dlewares [?, ?].

When designing parallel and distributed systems, a crit-ical problem is the one of how to tune the system configu-ration (e.g., in terms of number of nodes), so as to achievecertain performance goals (e.g., maximize throughput, min-imize latency or meet a prefixed latency constraint). To this

8http://incubator.apache.org/s4/

183

end, various authors tried to build abstract models for theperformance of such systems, starting from the well-knownAmdahl law [?] and its various enhancements, some of whichare summarised in [?].

Giurgiu [?] has presented an approach for estimation ofperformance for mobile-cloud applications. The approachtried to identify the factors which impact interaction re-sponse times, such as the application distribution schemes,workload sizes and intensities, or the resource variations ofthe mobile-cloud application setup. It also attempted to findcorrelations between these factors in order to better under-stand how to build a unified and generic performance estima-tion model. The Starfish system [?, ?] is the most relevantsystem for our work. It represents a cost-based optimizerfor MapReduce programs which focuses on the optimizationof configuration parameters for executing these programs onthe Hadoop platform. It relies on a Profiler component thatcollects detailed statistical information from executing theprograms and a What-if Engine for fine-grained cost esti-mation processing. For a given MapReduce program, therole of the cost-based optimizer component is to enumerateand search efficiently through the high dimensional space ofconfiguration parameter settings, making appropriate callsto the What-if Engine, in order to find the optimal configu-ration setting. It clusters parameters into lower-dimensionalsubspaces such that the globally-optimal parameter settingin the high-dimensional space can be generated by compos-ing the optimal settings found for the subspaces. Our cur-rent study represents the first step in the implementationof a similar what-if analyzer component in the more com-plex environment of real time distributed stream-processingengines.

8. CONCLUSIONSThe lack of expertise and in-depth understanding of the

theoretical foundation for the performance modeling of largescale distributed processing system is a crucial problem.Many researchers, engineers and organizations are currentlyshifting their focus to exploit the advantages of the new dataprocessing paradigm shift. In this paper, we presented a de-tailed set of models that formalize the performance charac-teristics of the Storm system as a representative of a practi-cal distributed, parallel and fault-tolerant stream processingengine that follows the Actor Model theory. Our approachis applicable to estimate the performance characteristics ofdistributed stream processing jobs which follow the sameprinciples in addition to optimizing the configuration set-tings of the underlying system and hardware cluster. Sev-eral directions represent good candidates for future work tofurther understand and optimise the complex environmentof real time distributed streaming processing engines. Forexample, we are planning to build a What-if Engine that canhelp users to find the right configuration setting and param-eters for their jobs. In addition, we plan to implement amore intelligent allocation strategy for the processing com-ponents across the running workers which can optimize thedata transfer cost.

9. REFERENCES[1] Abadi, D. J., et al. Aurora: A Data Stream Management

System. In SIGMOD Conference (2003), p. 666.

[2] Abadi, D. J., et al. The Design of the Borealis StreamProcessing Engine. In CIDR (2005), pp. 277–289.

[3] Agha, G. Actors: a model of concurrent computation indistributed systems. MIT Press, Cambridge, MA, USA,1986.

[4] Aly, A. M., et al. M3: Stream Processing onMain-Memory MapReduce. In ICDE (2012).

[5] Amdahl, G. M. Validity of the single processor approachto achieving large scale computing capabilities. InProceedings of AFIPS ’67, Spring Joint ComputerConference (1967), pp. 483–485.

[6] Bhatotia, P., et al. Incoop: MapReduce for IncrementalComputations. In SOCC (2011).

[7] Cherkasova, L. Performance modeling in mapreduceenvironments: challenges and opportunities. In ICPE(2011), pp. 5–6.

[8] Clinger, W. D. Foundations of Actor Semantics. Tech.rep., Cambridge, MA, USA, 1981.

[9] Condie, T., et al. MapReduce Online. In NSDI (2010),pp. 313–328.

[10] Cucinotta, T. Optimum scalability point for parallelisablereal-time components. In Proceedings of SOMRES (2011).

[11] de Gooijer, T., et al. An industrial case study ofperformance and cost design space exploration. In ICPE(2012), pp. 205–216.

[12] Dean, J., and Ghemawat, S. MapReduce: Simplified DataProcessing on Large Clusters. In OSDI (2004), pp. 137–150.

[13] Gedik, B., et al. Spade: the system s declarative streamprocessing engine. In SIGMOD Conference (2008),pp. 1123–1134.

[14] Giurgiu, I. Understanding performance modeling formodular mobile-cloud applications. In ICPE (2012),pp. 259–262.

[15] Herodotou, H., et al. MapReduce Programming andCost-based Optimization? Crossing this Chasm withStarfish. PVLDB 4, 12 (2011), 1446–1449.

[16] Herodotou, H., et al. Starfish: A Self-tuning System forBig Data Analytics. In CIDR (2011), pp. 261–272.

[17] Hewitt, C. ORGs for Scalable, Robust, Privacy-FriendlyClient Cloud Computing. IEEE Internet Computing 12, 5(2008), 96–99.

[18] Hewitt, C. ActorScript(TM): Industrial strengthintegration of local and nonlocal concurrency forClient-cloud Computing. CoRR abs/0907.3330 (2009).

[19] Hewitt, C. Actor Model for Discretionary, AdaptiveConcurrency. CoRR abs/1008.1459 (2010).

[20] Hewitt, C., et al. A Universal Modular ACTORFormalism for Artificial Intelligence. In IJCAI (1973),pp. 235–245.

[21] Loesing, S., et al. Stormy: an elastic and highly availablestreaming service in the cloud. In EDBT/ICDT Workshops(2012), pp. 55–60.

[22] Meseguer, J., and Talcott, C. L. A partial order eventmodel for concurrent objects. In Proceedings of CONCUR(1999).

[23] Muhl, G., et al. Stochastic Analysis of HierarchicalPublish/Subscribe Systems. In Euro-Par (2009).

[24] Sachs, K., et al. Benchmarking of message-orientedmiddleware. In DEBS (2009).

[25] Sachs, K., et al. Performance evaluation ofmessage-oriented middleware using the SPECjms2007benchmark. Perform. Eval. 66, 8 (2009).

[26] Sakr, S., et al. A Survey of Large Scale DataManagement Approaches in Cloud Environments. IEEECommunications Surveys and Tutorials 13, 3 (2011),311–336.

[27] Schroter, A., et al. Stochastic performance analysis andcapacity planning of publish/subscribe systems. In DEBS(2010).

184