Aurora: a new model and Aurora: a new model and architecture for data stream architecture for data stream

managementmanagementDaniel J. AbadiDaniel J. Abadi11, Don Carney, Don Carney22, Ugur , Ugur

CetintemelCetintemel22, Mitch Cherniack, Mitch Cherniack11, Christian , Christian ConveyConvey22, Sangdon Lee, Sangdon Lee22, Michael , Michael

StonebrakerStonebraker33, Nesime Tatbul, Nesime Tatbul22, Stan Zdonik, Stan Zdonik22

1 Department of Computer Science, Brandeis University2 Department of Computer Science, Brown University3 Department of EECS and Laboratory of Computer Science, M.I.T.

Presenter: Saurin KadakiaPresenter: Saurin Kadakia

ABOUT MEABOUT ME MS CS STUDENTMS CS STUDENT GRADUATING IN DEC 08GRADUATING IN DEC 08 INTERESTED IN DATABASES AND INTERESTED IN DATABASES AND

WEB TECHNOLOGYWEB TECHNOLOGY

WHAT ARE MONITORING WHAT ARE MONITORING APPLICATIONS??APPLICATIONS??

MONITORING APPLICATIONS ARE MONITORING APPLICATIONS ARE APPLICATIONS THAT MONITOR APPLICATIONS THAT MONITOR CONTINUOUS STREAMS OF DATA.CONTINUOUS STREAMS OF DATA.

EXAMPLES??EXAMPLES?? MILITARY APPLICATIONSMILITARY APPLICATIONS FINANCIAL ANALYSIS APPLICATIONSFINANCIAL ANALYSIS APPLICATIONS TRACKING APPLICATIONSTRACKING APPLICATIONS

TRADITIONAL DBMS ASSUMPTIONSTRADITIONAL DBMS ASSUMPTIONS

HUMAN ACTIVE, DBMS PASSIVE MODELHUMAN ACTIVE, DBMS PASSIVE MODEL

ONLY CURRENT VALUE IMPORTANTONLY CURRENT VALUE IMPORTANT

TRIGGERS/ASSERTIONS ARE SECONDARYTRIGGERS/ASSERTIONS ARE SECONDARY

QUERIES MUST HAVE EXACT ANSWERSQUERIES MUST HAVE EXACT ANSWERS

NO REAL TIME SERVICE REQUIREMENTSNO REAL TIME SERVICE REQUIREMENTS

REALITY FOR MONITORING REALITY FOR MONITORING APPLICATIONSAPPLICATIONS

DBMS ACTIVE, HUMAN PASSIVE MODELDBMS ACTIVE, HUMAN PASSIVE MODEL

HISTORY OF VALUES REQUIREDHISTORY OF VALUES REQUIRED

TRIGGER ORIENTED APPLICATIONSTRIGGER ORIENTED APPLICATIONS

APPROXIMATE ANSWERS TO QUERIESAPPROXIMATE ANSWERS TO QUERIES

REAL TIME REQUIREMENTSREAL TIME REQUIREMENTS

SYSTEM MODELSYSTEM MODEL

Externaldata source

User application

Operatorboxes data flow Continuous

& ad hoc queries

HistoricalStorage

AuroraSystem

QoS spec

Query spec

Applicationadministrator

QUERY MODELQUERY MODEL TraditionalTraditional

SStructured tructured QQuery uery LLanguageanguage Declarative query on static dataDeclarative query on static data

AuroraAurora Data flow model for data streamData flow model for data stream

Application manager will construct queries using GUIApplication manager will construct queries using GUI SStream tream QuQuery ery AlAlgebragebra

Queries are processed by SQuAl operators on the data Queries are processed by SQuAl operators on the data stream stream

Some of the operators are filter, map, union, Some of the operators are filter, map, union, aggregate, join bsort, resample.aggregate, join bsort, resample.

AURORA QUERY MODELAURORA QUERY MODELb1 b2 b3

b4

b5 b6

b7 b8 b9 app

app

QoS spec

QoS spec

QoS spec

continuous query

view

ad-hoc query

Connectionpoint

data input

AURORA QoS GRAPH TYPESAURORA QoS GRAPH TYPES

OPTIMIZATIONOPTIMIZATION

Filter BSort

Filter

Union

AggregateJoin

Aggregate

Map

Hold

Hold

pull data

Continuous query

FilterMap

Ad hoc query

JoinStatic storage

OPTIMIZATIONOPTIMIZATIONDynamic continuous query Dynamic continuous query optimizationoptimization

Inserting projectionsInserting projectionsCombining boxesCombining boxesReordering boxesReordering boxes

AURORA RUNTIME ARCHITECTUREAURORA RUNTIME ARCHITECTURE

Buffer manager

StorageManager

Persistent Store

Q1

Q2

Qm

Q1

Q2

Qn

Scheduler

LoadShedder

QoSMonitor

CatalogBox Processors

σμ

Router

inputs outputs

SUMMARYSUMMARYSolution approach itselfSolution approach itself

Rethink about everything for the Rethink about everything for the requirementsrequirements

Query modelQuery modelData flow style query specificationData flow style query specification

OptimizationOptimizationDynamic runtime optimizationDynamic runtime optimizationQoS specification based resource QoS specification based resource managementmanagement

QUESTIONS???QUESTIONS???