[IEEE 22nd International Conference on Data Engineering (ICDE'06) - Atlanta, GA, USA (2006.04.3-2006.04.7)] 22nd International Conference on Data Engineering (ICDE'06) - Better Burst Detection

Download [IEEE 22nd International Conference on Data Engineering (ICDE'06) - Atlanta, GA, USA (2006.04.3-2006.04.7)] 22nd International Conference on Data Engineering (ICDE'06) - Better Burst Detection

Post on 30-Mar-2017




0 download

Embed Size (px)


  • Better Burst Detection

    Xin Zhang Dennis ShashaDepartment of Computer Science

    Courant Institute of Mathematical SciencesNew York University



    A burst is a large number of events occurring within acertain time window. Many data stream applications re-quire the detection of bursts across a variety of windowsizes. For example, stock traders may be interested in burstshaving to do with institutional purchases or sales that arespread out over minutes or hours.

    In this paper, we present a new algorithmic frameworkfor elastic burst detection [1]: a family of data structuresthat generalizes the Shifted Binary Tree, and a heuristicsearch algorithm to find an efficient structure given the in-put. We study how different inputs affect the desired struc-tures and the probability to trigger a detailed search. Exper-iments on both synthetic and real world data show a factorof up to 35 times improvement compared with the Shifted Bi-nary Tree over a wide variety of inputs, depending on the in-puts.

    1. Introduction

    A burst is a large number of events occurring withina certain time window. Its a noteworthy phenomenonin many natural and social processes. For example,stock traders are interested in bursts of trading vol-ume, which may reflect hidden news. Astrophysicists areinterested in gamma ray bursts, which may reflect the oc-currence of a supernova. Furthurmore, many data applica-tions require detection of bursts across a variety of win-dow sizes. For example, interesting gamma ray burstscould last several seconds, several minutes or even sev-eral days.

    The elastic burst detection problem [1] is to detect burstsacross multiple window sizes. When the aggregate in a win-dow exceeds the threshold for that window size, a burst oc-curs. A naive algorithm is to check each window size of in-terest one at a time. To detect bursts over k window sizesin a sequence of length N naively requires O(kN) time.

    In [1], the authors show that a simple data structure calledthe Shifted Binary Tree can beat the naive algorithm by sev-eral orders of magnitude when the probability of bursts isvery low.

    A Shifted Binary Tree includes a binary tree as the basestructure. It also includes shifted sublevels to each base sub-level above level 0. The shifted sublevel i is still of length 2i,but the correspondending window is shifted by 2i1 fromthe base sublevel. Figure 1.a shows an example.

    The overlap between the base sublevels and the shiftedsublevels guarantees that all the windows of length w, w 1 + 2i, are included in one of the windows at level i + 1.Let f(w) be the threshold for size w. Because the aggrega-tion function is monotonically increasing, whenever morethan f(2+2i1) events are found in a window of size 2i+1,then a detailed search must be performed to check if somesubwindow of size w, 2 + 2i1 w 1 + 2i, has f(w)events. All bursts are guaranteed to be reported and manynon-burst windows are filtered away without requiring a de-tailed check. Unfortunately, when bursts are rare but notvery rare, the number of fruitless detailed searches grows,suggesting we may want more levels than Shifted BinaryTree provides; conversely, when bursts are exceedingly rarewe may need fewer levels. In other words, we want a struc-ture that adapts to the input.

    This paper presents a family of multiresolution over-lapping data structures, called Shifted Aggregation Trees,which generalizes the Shifted Binary Tree and includesmany other structures. We then present a heuristic search al-gorithm to find an efficient data structure given the sampleinput series and window thresholds. We theoretically ana-lyze and empirically study how different inputs affect thedesired structures and the probability to trigger a detailedsearch. Experiments on both synthetic data and real worlddata show that the Shifted Aggregation Tree outperformsthe Shifted Binary Tree over a variety of inputs, yieldingup to a factor of 35 times speedup in some cases. Due tospace limitations, details and references missing in this pa-per can be found in [2].

    Proceedings of the 22nd International Conference on Data Engineering (ICDE06) 8-7695-2570-9/06 $20.00 2006 IEEE

  • Level 0

    Level 1

    Level 2

    Level 3

    Level 4

    (a) Shifted Binary Tree

    Level 0Level 1

    Level 2

    Level 3

    Level 4

    (b) Embed a Shifted Binary Tree (SBT) in an Ag-gregation Pyramid (AP). Each shaded cell in theAP corresponds to a node in the SBT. The differ-ent shadings in level 2 show the one-to-one cor-respondence.

    (c) An example of Shifted AggregationTree embedded in the Aggregation Pyra-mid

    Figure 1. Shifted Binary Tree, AggregationPyramid and Shifted Aggregation Tree.

    2. Shifted Aggregation Tree

    2.1. Embed a Shifted Binary Tree in an Aggrega-tion Pyramid

    Aggregation Pyramid is an N -level isosceles triangular-shaped data structure built over a time window of size Nshown in Figure 1.b and 1.c: level 0 has N cells and isin one-to-one correspondence with the original time series;level 1 has N 1 cells, the first cell stores the aggregateof the first two data items (say, data items 1 and 2) in theoriginal time series, the second cell stores the aggregate ofthe second two data items (data items 2 and 3), etc; and soon. In all, it stores the original time series and all the ag-

    SBT SATNumber of children 2 2

    Levels of children for i ilevel i+ 1

    Shift at level i+ 1: Si+1 2 Si k Si, k 1Overlapping window window size wi

    size at level i+ 1: Oi+1 at level i: wi

    Table 1. Comparing the Shifted AggregationTree (SAT) with the Shifted Binary Tree (SBT)

    gregates for every window size starting at every time pointwithin this time window.

    Recall that in a Shifted Binary Tree (SBT), level 0 storesthe original time series, and level i stores the aggregates ofwindow size 2i. So, each node in a SBT has a correspond-ing cell in the aggregation pyramid. Thus the SBT can beembedded in the aggregation pyramid as shown in Figure1.b.

    Obviously, there are many other possible embeddingsinto the aggregation pyramid. By using different structureson different data inputs, we can achieve optimal perfor-mance by trading off structure maintenance against filter-ing selectivity.

    2.2. Shifted Aggregation Tree Generalizes ShiftedBinary Tree

    Like a Shifted Binary Tree, a Shifted Aggregation Treeis a hierarchical tree structure defined on a subset of thecells of an aggregation pyramid. It has several levels, eachof which contains several nodes. The nodes at level 0 arein one-to-one correspondence with the original time series.Any node at level i is computed by aggregating some nodesbelow level i. Two consecutive nodes at the same level over-lap in time.

    A Shifted Aggregation Tree (SAT) is different from aShifted Binary Tree (SBT) in two ways:

    The parent-child structure defining the topological re-lationship between a node and its children, i.e. howmany children it has and their placements.

    The pattern defining how many time points apart are(called the shift) two neighboring nodes at the samelevel.

    Table 2.2 gives a side-by-side comparison of the differ-ence between a SAT and a SBT. Clearly, a SBT is a specialcase of a SAT. Figure 1.c shows one example of Shifted Ag-gregation Trees.

    The Shifted Aggregation Tree shares an important prop-erty with the Shifted Binary Tree: any window of size w,w hi si + 1, is included by a node at level i, where

    Proceedings of the 22nd International Conference on Data Engineering (ICDE06) 8-7695-2570-9/06 $20.00 2006 IEEE

  • hi is the corresponding window size of level i, and si is theshift of level i. This shared property yields a similar detec-tion algorithm to that of a Shifted Binary Tree. When newdata points come, the nodes in a Shifted Aggregation Treeare updated from bottom up. After a node at level i+1 is up-dated, if the new aggregate is more than f(hi si + 2), adetailed search is performed on the subwindows of size w,hisi+2 w hi+1si+1+1. An efficient Shifted Ag-gregation Tree should balance between the update time andthe detailed search time.

    3. Heuristic state-space algorithm to searchan efficient Shifted Aggregation Tree

    To find an efficient Shifted Aggregation Tree (SAT), onecan use a state-space search algorithm, given the sample in-put series and window thresholds. Each SAT is seen as astate. By adding a level onto the top of SAT B, if we canget another SAT A, we say state B can be transformed tostate A. The algorithm starts from a SAT having level 0only, then keeps transforming the candidate set of SATs, un-til a set of final SATs (i.e. those which can detect bursts inall windows of interst) are reached. In order to find the bestSAT, the best-first strategy is used to explore the state space.Each state is associated with a cost. The state with the min-imum cost is picked as the next state to be explored, andthe final SAT with the minimum cost is picked as the de-sired structure.

    The cost associated with each state is used to indicatewhich structure to choose in term of running time. We usea theoretical cost model the expected number of opera-tions to model the CPU running time. Our model is a sim-ple RAM model: all operations (updates and comparisons)take constant time. The total cost is the sum of the numberof updating operations and the expected number of compar-ison operations, given the sample input series, the windowthresholds and the tree structure. Our experiment shows thatthe theoretical cost model models the actual CPU runningtime well [2].

    4. Empirical Results

    4.1. Synthetic Data

    Two classes of probabilistic distributions widely used tomodel many real world applications were chosen to gener-ate the synthetic data: the Poisson distribution and the expo-nential distribution. For each distribution, we synthesized aset of data with different parameters in a broad range. Wewant to see how different distributions and thresholds af-fect the desired Shifted Aggregation Trees.

    Assume that each point in the input time series has amean and a standard deviation . Assume that for each

    window size, the probability to exceed the threshold besome value p. We can infer [2] that the alarm probability Pafor an aggregate of window size W to exceed the thresoldfor window size w is

    ((T 1





    where T = W/w, denotes the bounding ratio, and (x) isthe normal cumulative distribution function.

    So Pa is determined by the distribution parameters and, the threshold parameter p, the bounding ratio T and thelevel w in the underlying aggregation pyramid. The experi-ments on both Poisson data and exponential data also showthe following:

    The larger the ratio

    is, the larger the alarm probabil-ity Pa. To mitigate this, the Shifted Aggregation Treebecomes denser in order to bring down Pa. When

    becomes very large, Pa is close to 1. So the structureturns sparser again to reduce the updating time, but isessentially useless.

    The smaller the burst probability p, the larger thethreshold, the smaller Pa, so the structure becomessparser since there are less bursts to worry about.

    As the bounding ratio T decreases, so does Pa. In aShifted Aggregation Tree, T could be very close to 1,e.g. W = w + 1, whereas T in a Shifted Binary Treeis designed to be about 4. Thus a SAT is able to ad-just its structure to reduce the alarm probability.

    As the size w increases, so does Pa, while the bound-ing ratio T becomes smaller in the Shifted Aggrega-tion Tree to try to reduce Pa.

    In summary, because the Shifted Aggregation Tree canadjust its structure to reduce the alarm probability, it canachieve far better running time than the Shifted Binary Tree.

    4.2. Real World Data

    We have used two real world data sets to test the pro-posed framework: the Sloan Digital Sky Survey (SDSS)SkyServer traffic data which records all the access traffic tothe SDSS SkyServer in 2003, and the NYSE TAQ stock datawhich includes tick-by-tick trading activities of the IBMstock between Jan. 1st, 2001 to May 31st, 2004. All theevents within the same second are aggregated as a singledatum. The statistics show that the SDSS data follows thePoisson distribution while the IBM data is close to the ex-ponential distribution.

    We are interested in comparing the Shifted AggregationTree (SAT) with the Shifted Binary Tree (SBT) under dif-ferent settings.

    Proceedings of the 22nd International Conference on Data Engineering (ICDE06) 8-7695-2570-9/06 $20.00 2006 IEEE

  • CPU Time vs. Threshold








    2 3 4 5 6 7 8 9

    Burst Probability p=10-k


    U T

    ime (m






    (a) Threshold

    CPU Time vs. Max Window Size of Interest







    10 30 60 120 300 600 1800

    Max Window Size of Interest


    U T

    ime (m

    s) SDSS_SAT




    (b) Maximum window size of interest

    CPU Time vs. Set of Window Sizes







    1 5 10 30 60 120

    Window Size Step


    U T

    ime (m






    (c) Different set of window sizes of interest

    Figure 2. Performance test: CPU time comparison on the Sloan Digital Sky Survey (SDSS) SkyServertraffic data and the IBM stock data

    Different thresholds are set to reflect a burst probabil-ity ranging from 102 to 109.

    Different maximum window sizes of interest are setfrom 10 seconds up to 1800 seconds.

    Different sets of window sizes of interest are set to bea set of sizes n, 2 n, 3 n, ...., where n is set to be 1,5, 10, 30, 60, 120 respectively.

    Figure 2 shows the results for both data sets. The ShiftedAggregation Tree performs better than the Shifted BinaryTree by a factor of from 2 to 5 times.

    We also studied how sensitive the desired Shifted Ag-gregation Tree is to the training data and how the searchparameters in the state-space algorithm affect the desiredstructure. The experiments show that the training sets withsimilar statistics produce similar structure, while the train-ing set with largely different characteristics from the test-ing set could cause the desired structure to perform poor.Its also shown that small search parameters work well inthe best-first search algorithm.

    5. Related Work

    As an interesting and important phenomenon, monitor-ing and mining bursty behaviors is attracting increasing in-terest recently. Due to the space limitations, the referencebibliographies can be found in [2].

    Wang et al. model the bursty behavior in self-similar timeseries, such as Ethernet, disk traffic, etc. Kleinberg stud-ies the bursty and hierarchical structure in temporal textstreams to discover how high frequency words change overtime. His work focus on modeling the bursty behaviors,while our focus is a high-performance algorithm to detectbursts, thus complementing his work. Vlachos et al. minethe bursty behavior in the query logs of the MSN search en-

    gine. We share the view that burst detection should be a pre-liminary primitive for further knowledge mining process.

    Neill et al. study the problem of detecting signifi-cant spatial clusters in multidimensional space. They con-sider a general non-monotonic density function a...


View more >