niagaracq : a scalable continuous query system for internet databases jianjun chen et al computer...
DESCRIPTION
NiagaraCQ A CQ system for the Internet Continuous Queries on XML data sets Scalable CQ processing Incremental group optimization Handles both change based and timer based queries in a uniform wayTRANSCRIPT
![Page 1: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/1.jpg)
NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al
Computer Sciences Dept. University of Wisconsin-Madison
SIGMOD 2000 Talk by S. Sudarshan
![Page 2: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/2.jpg)
Continuous QueriesExample
Inform me when there is a new publication related to multi-query optimization
A broad classification Change based Timer based
![Page 3: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/3.jpg)
NiagaraCQA CQ system for the InternetContinuous Queries on XML data setsScalable CQ processingIncremental group optimizationHandles both change based and timer based queries in a uniform way
![Page 4: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/4.jpg)
NiagaraCQ command language
Creating a CQCreate CQ_name XML-QL queryDo action { START start_time} { EVERY
time_interval} { EXPIRE expiration_time}
Delete CQ_name
![Page 5: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/5.jpg)
Expression SignatureQuery examples Where <Quotes><Quote><Symbol>INTC</></></> element_as $g in “http://www.stock.com/quotes.xml”
construct $g
Where <Quotes><Quote><Symbol>MSFT</></></> element_as $g in “http://www.stock.com/quotes.xml”
construct $g
Expression signatures
Quotes.Quote.Symbol in quotes.xml
constant
=
![Page 6: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/6.jpg)
Query plansTrigger Action I Trigger Action J
File Scan
Select Symbol = “MSFT”
Select Symbol = “INTC”
File Scan
quotes.xml quotes.xml
![Page 7: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/7.jpg)
GroupGroup Signature
Common signature of all queries in the group
Group constant table Constant_value
Dest_buffer
INTC Dest. I MSFT Dest. J
![Page 8: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/8.jpg)
The group plan
![Page 9: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/9.jpg)
Incremental Grouping AlgoWhen a new query is submitted
If the expression signature of the new query matches that of existing groups
Break the query plan into two partsRemove the lower partAdd the upper part onto the group
planelse create a new group
![Page 10: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/10.jpg)
![Page 11: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/11.jpg)
Query split with materialized intermediate files
Why not use a pipeline scheme ? Split operator may block simple queries Gives a single complicated execution plan A large portion of query plan may not need to be
executed at each invocation Does not work for grouping timer based queries
Using intermediate files Cut query plan into 2 parts at split operator Add a file scan operator to upper part to read
intermediate file Intermediate files are monitored just like other
data sources
![Page 12: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/12.jpg)
The query split scheme
![Page 13: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/13.jpg)
Trade-offsOther advantages of materialized intermediate files
Only the necessary queries are executed Uniform handling of intermediate files and
original data source files
Disadvantages Split operator becomes a blocking
operator Extra disk I/Os
![Page 14: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/14.jpg)
Range PredicatesE.g. R.a < val or val1 < R.a < val2Multiple such rangesProblem
Intermediate files may contain duplicate tuples
Idea: Virtual intermediate files Use an index to implement this
![Page 15: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/15.jpg)
Incremental grouping of selection predicates
Multiple selection predicates in a query CNF for predicates on same data source Incremental grouping
Choose the most selective conjunct and implement virtual file on this conjunct
Evaluation of other predicates Upper levels of continuous query
Example queryWhere <Quotes><Quote><Symbol>”INTC”</> <Current_Price>$p</></> element_as $g </> in “quotes.xml”, $p < 100Construct $g
![Page 16: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/16.jpg)
Incremental grouping of join operators
A join queryQuotes.Quote.Change_Ratio constant in “quotes.xml”Where <Quotes><Quote><Symbol>$s</></>
element_as $g </> in “quotes.xml”,<Companies><Company><Symbol>$s</></>
element_as $t</> in “companies.xml”construct $g, $t
![Page 17: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/17.jpg)
Queries that contain both join and selection
Example query :Where <Quotes><Quote><Symbol>$s</><Industry>”Computer Service”</></>element_as $g </> in “quotes.xml”,<Companies><Company><Symbol>$s</></>element_as $t</> in “companies.xml”construct $g, $t
Where to place the selection operator ? Below the join
Eliminates irrelevant tuples Above the join
Allows sharing Pick based on cost model
![Page 18: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/18.jpg)
Grouping timer-based queries
Challenge Sharing common computation
Event List Stores time events sorted in time order
![Page 19: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/19.jpg)
Incremental evaluationInvoke queries only on changed data
For each source file, NiagaraCQ keeps a delta file Also for the intermediate files Time stamp store the each tuple
Incremental evaluation of join operators requires complete data files
![Page 20: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/20.jpg)
Memory CachingThousands of continuous queries can’t fit in memoryWhat should we cache ?
Grouped query plans What about non-grouped queries ?
Favor small delta files Front part of the event list
![Page 21: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/21.jpg)
System Architecture
![Page 22: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/22.jpg)
CQ processing
![Page 23: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/23.jpg)
Experimental ResultsExample query :
Where <Quotes><Quote><Symbol>”INTC”</></>element_as $g </> in “quotes.xml”, construct $g
N = number of installed queriesF= number of fired queriesC = number of tuples modified
![Page 24: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/24.jpg)
Performance ResultsCase 1: F=N, C=1000Case 2: F=100, C=1000
![Page 25: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/25.jpg)
Performance ResultsF=N=2000, vary data size
![Page 26: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/26.jpg)
Thank You
![Page 27: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/27.jpg)
OutlineGeneral strategy of incremental group optimizationQuery split with materialized intermediate filesIncremental grouping of selection and join operatorsSystem architectureExperimental results
![Page 28: NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al Computer Sciences Dept. University of Wisconsin-Madison SIGMOD](https://reader036.vdocuments.mx/reader036/viewer/2022081605/5a4d1bab7f8b9ab0599cac89/html5/thumbnails/28.jpg)
Incremental group optimization
General Strategy
Why can’t we regroup all queries when a new query is added ?
Use of expression signatures for grouping
Same syntax structure Different constant values