chapter 13: query processing - cse, iit bombaysudarsha/db-book/slide-dir/ch13.pdf · chapter 13:...

DatabaseSystemConcepts,5thEd.Silberschatz,KorthandSudarshan

Seewww.dbbook.comforconditionsonreuse

Chapter13:QueryProcessingChapter13:QueryProcessing

Aug10,2006

http://www.db-book.com/

Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

Chapter13:QueryProcessingChapter13:QueryProcessing

Overview MeasuresofQueryCost SelectionOperation Sorting JoinOperation OtherOperations EvaluationofExpressions


BasicStepsinQueryProcessingBasicStepsinQueryProcessing

1. Parsingandtranslation2. Optimization3. Evaluation


BasicStepsinQueryProcessingBasicStepsinQueryProcessing(Cont.)(Cont.)

Parsingandtranslation translatethequeryintoitsinternalform.Thisisthentranslatedinto

relationalalgebra. Parsercheckssyntax,verifiesrelations

Evaluation Thequeryexecutionenginetakesaqueryevaluationplan,executes

thatplan,andreturnstheanswerstothequery.


BasicStepsinQueryProcessing:BasicStepsinQueryProcessing:OptimizationOptimization

Arelationalalgebraexpressionmayhavemanyequivalentexpressions E.g.,balance


BasicSteps:Optimization(Cont.)BasicSteps:Optimization(Cont.)

QueryOptimization:Amongstallequivalentevaluationplanschoosetheonewithlowestcost.

Costisestimatedusingstatisticalinformationfromthedatabasecatalog e.g.numberoftuplesineachrelation,sizeoftuples,etc.

Inthischapterwestudy Howtomeasurequerycosts Algorithmsforevaluatingrelationalalgebraoperations Howtocombinealgorithmsforindividualoperationsinorderto

evaluateacompleteexpression InChapter14

Westudyhowtooptimizequeries,thatis,howtofindanevaluationplanwithlowestestimatedcost


MeasuresofQueryCostMeasuresofQueryCost

Costisgenerallymeasuredastotalelapsedtimeforansweringquery

Manyfactorscontributetotimecost diskaccesses,CPU,orevennetworkcommunication

Typicallydiskaccessisthepredominantcost,andisalsorelativelyeasytoestimate.Measuredbytakingintoaccount

Numberofseeks*averageseekcost+Numberofblocksread*averageblockreadcost+Numberofblockswritten*averageblockwritecost

Costtowriteablockisgreaterthancosttoreadablock dataisreadbackafterbeingwrittentoensurethatthe

writewassuccessful Assumption:singledisk

Canmodifyformulaeformultipledisks/RAIDarrays Orjustusesinglediskformulae,butinterpretthemas

measuringresourceconsumptioninsteadoftime


MeasuresofQueryCost(Cont.)MeasuresofQueryCost(Cont.)

Forsimplicitywejustusethenumberofblocktransfersfromdiskandthenumberofseeksasthecostmeasures

tTtimetotransferoneblock tStimeforoneseek CostforbblocktransfersplusSseeks

b*tT+S*tS WeignoreCPUcostsforsimplicity

RealsystemsdotakeCPUcostintoaccount Wedonotincludecosttowritingoutputtodiskinourcostformulae SeveralalgorithmscanreducediskIObyusingextrabufferspace

AmountofrealmemoryavailabletobufferdependsonotherconcurrentqueriesandOSprocesses,knownonlyduringexecution Weoftenuseworstcaseestimates,assumingonlytheminimum

amountofmemoryneededfortheoperationisavailable Requireddatamaybebufferresidentalready,avoidingdiskI/O

Buthardtotakeintoaccountforcostestimation


SelectionOperationSelectionOperation

Filescansearchalgorithmsthatlocateandretrieverecordsthatfulfillaselectioncondition.

AlgorithmA1(linearsearch).Scaneachfileblockandtestallrecordstoseewhethertheysatisfytheselectioncondition.

Costestimate=brblocktransfers+1seek brdenotesnumberofblockscontainingrecordsfromrelationr

Ifselectionisonakeyattribute,canstoponfindingrecord cost=(br/2)blocktransfers+1seek

Linearsearchcanbeappliedregardlessof selectionconditionor orderingofrecordsinthefile,or availabilityofindices


SelectionOperation(Cont.)SelectionOperation(Cont.)

A2(binarysearch).Applicableifselectionisanequalitycomparisonontheattributeonwhichfileisordered.

Assumethattheblocksofarelationarestoredcontiguously Costestimate(numberofdiskblockstobescanned):

costoflocatingthefirsttuplebyabinarysearchontheblocks log2(br)*(tT+tS)

Iftherearemultiplerecordssatisfyingselection Addtransfercostofthenumberofblockscontaining

recordsthatsatisfyselectioncondition WillseehowtoestimatethiscostinChapter14


SelectionsUsingIndicesSelectionsUsingIndices

Indexscansearchalgorithmsthatuseanindex selectionconditionmustbeonsearchkeyofindex.

A3(primaryindexoncandidatekey,equality).Retrieveasinglerecordthatsatisfiesthecorrespondingequalitycondition

Cost=(hi+1)*(tT+tS) A4(primaryindexonnonkey,equality)Retrievemultiplerecords.

Recordswillbeonconsecutiveblocks Letb=numberofblockscontainingmatchingrecords

Cost=hi*(tT+tS)+tS+tT*b A5(equalityonsearchkeyofsecondaryindex).

Retrieveasinglerecordifthesearchkeyisacandidatekey Cost=(hi+1)*(tT+tS)

Retrievemultiplerecordsifsearchkeyisnotacandidatekey eachofnmatchingrecordsmaybeonadifferentblock Cost=(hi+n)*(tT+tS)

Canbeveryexpensive!


SelectionsInvolvingComparisonsSelectionsInvolvingComparisons

CanimplementselectionsoftheformAV(r)orAV(r)byusing alinearfilescanorbinarysearch, orbyusingindicesinthefollowingways:

A6(primaryindex,comparison).(RelationissortedonA) ForAV(r)useindextofindfirsttuplevandscanrelation

sequentiallyfromthere ForAV(r)justscanrelationsequentiallytillfirsttuple>v;donotuse

index A7(secondaryindex,comparison).

ForAV(r)useindextofindfirstindexentryvandscanindexsequentiallyfromthere,tofindpointerstorecords.

ForAV(r)justscanleafpagesofindexfindingpointerstorecords,tillfirstentry>v

Ineithercase,retrieverecordsthatarepointedto requiresanI/Oforeachrecord Linearfilescanmaybecheaper


ImplementationofComplexSelectionsImplementationofComplexSelections

Conjunction:12...n(r) A8(conjunctiveselectionusingoneindex).

SelectacombinationofiandalgorithmsA1throughA7thatresultsintheleastcostfori(r).

Testotherconditionsontupleafterfetchingitintomemorybuffer. A9(conjunctiveselectionusingmultiplekeyindex).

Useappropriatecomposite(multiplekey)indexifavailable. A10(conjunctiveselectionbyintersectionofidentifiers).

Requiresindiceswithrecordpointers. Usecorrespondingindexforeachcondition,andtakeintersection

ofalltheobtainedsetsofrecordpointers. Thenfetchrecordsfromfile Ifsomeconditionsdonothaveappropriateindices,applytestin

memory.


AlgorithmsforComplexSelectionsAlgorithmsforComplexSelections

Disjunction:12...n(r). A11(disjunctiveselectionbyunionofidentifiers).

Applicableifallconditionshaveavailableindices. Otherwiseuselinearscan.

Usecorrespondingindexforeachcondition,andtakeunionofalltheobtainedsetsofrecordpointers.

Thenfetchrecordsfromfile Negation:(r)

Uselinearscanonfile Ifveryfewrecordssatisfy,andanindexisapplicableto

Findsatisfyingrecordsusingindexandfetchfromfile


SortingSorting

Wemaybuildanindexontherelation,andthenusetheindextoreadtherelationinsortedorder.Mayleadtoonediskblockaccessforeachtuple.

Forrelationsthatfitinmemory,techniqueslikequicksortcanbeused.Forrelationsthatdontfitinmemory,externalsortmergeisagoodchoice.


ExternalSortMergeExternalSortMerge

1. Createsortedruns.Letibe0initially.Repeatedlydothefollowingtilltheendoftherelation:(a)ReadMblocksofrelationintomemory(b)Sorttheinmemoryblocks(c)WritesorteddatatorunRi;incrementi.LetthefinalvalueofibeN

2. Mergetheruns(nextslide)..

LetMdenotememorysize(inpages).


ExternalSortMerge(Cont.)ExternalSortMerge(Cont.)

1. Mergetheruns(Nwaymerge).Weassume(fornow)thatN


ExternalSortMerge(Cont.)ExternalSortMerge(Cont.)

IfNM,severalmergepassesarerequired. Ineachpass,contiguousgroupsofM1runsaremerged. ApassreducesthenumberofrunsbyafactorofM1,and

createsrunslongerbythesamefactor. E.g.IfM=11,andthereare90runs,onepassreduces

thenumberofrunsto9,each10timesthesizeoftheinitialruns

Repeatedpassesareperformedtillallrunshavebeenmergedintoone.


Example:ExternalSortingUsingSortMergeExample:ExternalSortingUsingSortMerge


ExternalMergeSort(Cont.)ExternalMergeSort(Cont.)

Costanalysis: Totalnumberofmergepassesrequired:logM1(br/M). Blocktransfersforinitialruncreationaswellasineach

passis2br forfinalpass,wedontcountwritecost

weignorefinalwritecostforalloperationssincetheoutputofanoperationmaybesenttotheparentoperationwithoutbeingwrittentodisk

Thustotalnumberofblocktransfersforexternalsorting:br(2logM1(br/M)+1)

Seeks:nextslide


ExternalMergeSort(Cont.)ExternalMergeSort(Cont.)

Costofseeks Duringrungeneration:oneseektoreadeachrunandoneseekto

writeeachrun 2br/M

Duringthemergephase Buffersize:bb(read/writebbblocksatatime) Need2br/bbseeksforeachmergepass

exceptthefinalonewhichdoesnotrequireawrite Totalnumberofseeks:

2br/M+br/bb(2logM1(br/M)1)


JoinOperationJoinOperation

Severaldifferentalgorithmstoimplementjoins Nestedloopjoin Blocknestedloopjoin Indexednestedloopjoin Mergejoin Hashjoin

Choicebasedoncostestimate Examplesusethefollowinginformation

Numberofrecordsofcustomer:10,000depositor:5000 Numberofblocksofcustomer:400depositor:100


NestedLoopJoinNestedLoopJoin

Tocomputethethetajoinrsforeachtupletrinrdobeginforeachtupletsinsdobegin

testpair(tr,ts)toseeiftheysatisfythejoinconditioniftheydo,addtrtstotheresult.

endend

riscalledtheouterrelationandstheinnerrelationofthejoin. Requiresnoindicesandcanbeusedwithanykindofjoincondition. Expensivesinceitexamineseverypairoftuplesinthetworelations.


NestedLoopJoin(Cont.)NestedLoopJoin(Cont.)

Intheworstcase,ifthereisenoughmemoryonlytoholdoneblockofeachrelation,theestimatedcostisnrbs+brblocktransfers,plusnr+brseeks

Ifthesmallerrelationfitsentirelyinmemory,usethatastheinnerrelation. Reducescosttobr+bsblocktransfersand2seeks

Assumingworstcasememoryavailabilitycostestimateis withdepositorasouterrelation:

5000400+100=2,000,100blocktransfers, 5000+100=5100seeks

withcustomerastheouterrelation 10000100+400=1,000,400blocktransfersand10,400seeks

Ifsmallerrelation(depositor)fitsentirelyinmemory,thecostestimatewillbe500blocktransfers.

Blocknestedloopsalgorithm(nextslide)ispreferable.


BlockNestedLoopJoinBlockNestedLoopJoin

Variantofnestedloopjoininwhicheveryblockofinnerrelationispairedwitheveryblockofouterrelation.foreachblockBrofrdobegin

foreachblockBsofsdobeginforeachtupletrinBrdobegin

foreachtupletsinBsdobeginCheckif(tr,ts)satisfythejoinconditioniftheydo,addtrtstotheresult.

endend

endend


BlockNestedLoopJoin(Cont.)BlockNestedLoopJoin(Cont.)

Worstcaseestimate:brbs+brblocktransfers+2*brseeks Eachblockintheinnerrelationsisreadonceforeachblockinthe

outerrelation(insteadofonceforeachtupleintheouterrelation Bestcase:br+bsblocktransfers+2seeks. Improvementstonestedloopandblocknestedloopalgorithms:

Inblocknestedloop,useM2diskblocksasblockingunitforouterrelations,whereM=memorysizeinblocks;useremainingtwoblockstobufferinnerrelationandoutput Cost=br/(M2)bs+brblocktransfers+

2br/(M2)seeks Ifequijoinattributeformsakeyoninnerrelation,stopinnerloop

onfirstmatch Scaninnerloopforwardandbackwardalternately,tomakeuseof

theblocksremaininginbuffer(withLRUreplacement) Useindexoninnerrelationifavailable(nextslide)


IndexedNestedLoopJoinIndexedNestedLoopJoin

Indexlookupscanreplacefilescansif joinisanequijoinornaturaljoinand anindexisavailableontheinnerrelationsjoinattribute

Canconstructanindexjusttocomputeajoin. Foreachtupletrintheouterrelationr,usetheindextolookuptuplesins

thatsatisfythejoinconditionwithtupletr. Worstcase:bufferhasspaceforonlyonepageofr,and,foreachtuple

inr,weperformanindexlookupons. Costofthejoin:br(tT+tS)+nrc

Wherecisthecostoftraversingindexandfetchingallmatchingstuplesforonetupleorr

ccanbeestimatedascostofasingleselectiononsusingthejoincondition.

Ifindicesareavailableonjoinattributesofbothrands,usetherelationwithfewertuplesastheouterrelation.


ExampleofNestedLoopJoinCostsExampleofNestedLoopJoinCosts

Computedepositorcustomer,withdepositorastheouterrelation. LetcustomerhaveaprimaryB+treeindexonthejoinattribute

customername,whichcontains20entriesineachindexnode. Sincecustomerhas10,000tuples,theheightofthetreeis4,andone

moreaccessisneededtofindtheactualdata depositorhas5000tuples Costofblocknestedloopsjoin

400*100+100=40,100blocktransfers+2*100=200seeks assumingworstcasememory maybesignificantlylesswithmorememory

Costofindexednestedloopsjoin 100+5000*5=25,100blocktransfersandseeks. CPUcostlikelytobelessthanthatforblocknestedloopsjoin


MergeJoinMergeJoin

1. Sortbothrelationsontheirjoinattribute(ifnotalreadysortedonthejoinattributes).

2. Mergethesortedrelationstojointhem1. Joinstepissimilartothemergestageofthesortmergealgorithm.2. Maindifferenceishandlingofduplicatevaluesinjoinattributeevery

pairwithsamevalueonjoinattributemustbematched3. Detailedalgorithminbook


MergeJoin(Cont.)MergeJoin(Cont.)

Canbeusedonlyforequijoinsandnaturaljoins Eachblockneedstobereadonlyonce(assumingalltuplesforanygiven

valueofthejoinattributesfitinmemory Thusthecostofmergejoinis:

br+bsblocktransfers+br/bb+bs/bbseeks +thecostofsortingifrelationsareunsorted.

hybridmergejoin:Ifonerelationissorted,andtheotherhasasecondaryB+treeindexonthejoinattribute

MergethesortedrelationwiththeleafentriesoftheB+tree. Sorttheresultontheaddressesoftheunsortedrelationstuples Scantheunsortedrelationinphysicaladdressorderandmergewith

previousresult,toreplaceaddressesbytheactualtuples Sequentialscanmoreefficientthanrandomlookup


HashJoinHashJoin

Applicableforequijoinsandnaturaljoins. Ahashfunctionhisusedtopartitiontuplesofbothrelations

Intuition:partitionsfitinmemory hmapsJoinAttrsvaluesto{0,1,...,n},whereJoinAttrsdenotesthe

commonattributesofrandsusedinthenaturaljoin. r0,r1,...,rndenotepartitionsofrtuples

Eachtupletrrisputinpartitionriwherei=h(tr[JoinAttrs]).

r0,,r1...,rndenotespartitionsofstuples

Eachtupletssisputinpartitionsi,wherei=h(ts[JoinAttrs]).

Note:Inbook,riisdenotedasHri,siisdenotedasHsiandnisdenotedasnh.


HashJoin(Cont.)HashJoin(Cont.)



rtuplesinrineedonlytobecomparedwithstuplesinsiNeednotbecomparedwithstuplesinanyotherpartition,since:

anrtupleandanstuplethatsatisfythejoinconditionwillhavethesamevalueforthejoinattributes.

Ifthatvalueishashedtosomevaluei,thertuplehastobeinriandthestupleinsi.


HashJoinAlgorithmHashJoinAlgorithm

1. Partitiontherelationsusinghashingfunctionh.1. Whenpartitioningarelation,oneblockofmemoryisreservedasthe

outputbufferforeachpartition,andoneblockforinput2. Ifextramemoryisavailable,allocatebbblocksasbufferforinputand

eachoutput2. Partitionrsimilarly.3.nextslide..

Thehashjoinofrandsiscomputedasfollows.



1. Foreachpartitioni:(a) Loadsiintomemoryandbuildaninmemoryhashindexonit

usingthejoinattribute. Thishashindexusesadifferenthashfunctionthantheearlier

oneh.(b) Readthetuplesinrifromthediskonebyone.

Foreachtupletrprobetheinmemoryhashindextofindallmatchingtuplestsinsi Foreachmatchingtupletsinsi

outputtheconcatenationoftheattributesoftrandts

Relationsiscalledthebuildinputandriscalledtheprobeinput.

HashJoinAlgorithm(cont)


HashJoinalgorithm(Cont.)HashJoinalgorithm(Cont.)

Thevaluenandthehashfunctionhischosensuchthateachsishouldfitinmemory.

Typicallynischosenasbs/M*fwherefisafudgefactor,typicallyaround1.2

Theproberelationpartitionssineednotfitinmemory Recursivepartitioningrequiredifnumberofpartitionsnisgreater

thannumberofpagesMofmemory. insteadofpartitioningnways,useM1partitionsfors FurtherpartitiontheM1partitionsusingadifferenthash

function Usesamepartitioningmethodonr Rarelyrequired:e.g.,recursivepartitioningnotneededfor

relationsof1GBorlesswithmemorysizeof2MB,withblocksizeof4KB.


HandlingofOverflowsHandlingofOverflows

Partitioningissaidtobeskewedifsomepartitionshavesignificantlymoretuplesthansomeothers

Hashtableoverflowoccursinpartitionsiifsidoesnotfitinmemory.Reasonscouldbe

Manytuplesinswithsamevalueforjoinattributes Badhashfunction

Overflowresolutioncanbedoneinbuildphase Partitionsiisfurtherpartitionedusingdifferenthashfunction.

Partitionrimustbesimilarlypartitioned. Overflowavoidanceperformspartitioningcarefullytoavoidoverflows

duringbuildphase E.g.partitionbuildrelationintomanypartitions,thencombinethem

Bothapproachesfailwithlargenumbersofduplicates Fallbackoption:useblocknestedloopsjoinonoverflowedpartitions


CostofHashJoinCostofHashJoin

Ifrecursivepartitioningisnotrequired:costofhashjoinis3(br+bs)+4nhblocktransfers+2(br/bb+bs/bb)seeks

Ifrecursivepartitioningrequired: numberofpassesrequiredforpartitioningbuildrelation

sislogM1(bs)1 besttochoosethesmallerrelationasthebuildrelation. Totalcostestimateis:

2(br+bslogM1(bs)1+br+bsblocktransfers+2(br/bb+bs/bb)logM1(bs)1seeks

Iftheentirebuildinputcanbekeptinmainmemorynopartitioningisrequired

Costestimategoesdowntobr+bs.


ExampleofCostofHashJoinExampleofCostofHashJoin

Assumethatmemorysizeis20blocks bdepositor=100andbcustomer=400. depositoristobeusedasbuildinput.Partitionitintofivepartitions,each

ofsize20blocks.Thispartitioningcanbedoneinonepass. Similarly,partitioncustomerintofivepartitions,eachofsize80.Thisisalso

doneinonepass. Thereforetotalcost,ignoringcostofwritingpartiallyfilledblocks:

3(100+400)=1500blocktransfers+2(100/3+400/3)=336seeks

customerdepositor


HybridHashJoinHybridHashJoin

Usefulwhenmemorysizedarerelativelylarge,andthebuildinputisbiggerthanmemory.

Mainfeatureofhybridhashjoin:Keepthefirstpartitionofthebuildrelationinmemory. E.g.Withmemorysizeof25blocks,depositorcanbepartitionedintofive

partitions,eachofsize20blocks. Divisionofmemory:

Thefirstpartitionoccupies20blocksofmemory 1blockisusedforinput,and1blockeachforbufferingtheother4

partitions. customerissimilarlypartitionedintofivepartitionseachofsize80

thefirstisusedrightawayforprobing,insteadofbeingwrittenout Costof3(80+320)+20+80=1300blocktransfersfor

hybridhashjoin,insteadof1500withplainhashjoin. HybridhashjoinmostusefulifM>> sb


ComplexJoinsComplexJoins

Joinwithaconjunctivecondition:r12...ns

Eitherusenestedloops/blocknestedloops,or Computetheresultofoneofthesimplerjoinsris

finalresultcomprisesthosetuplesintheintermediateresultthatsatisfytheremainingconditions

1...i1i+1...n Joinwithadisjunctivecondition

r12...ns Eitherusenestedloops/blocknestedloops,or Computeastheunionoftherecordsinindividualjoinsris:

(r1s)(r2s)...(rns)

(appliesonlytothesetversionofunion!)


OtherOperationsOtherOperations

Duplicateeliminationcanbeimplementedviahashingorsorting. Onsortingduplicateswillcomeadjacenttoeachother,andallbut

onesetofduplicatescanbedeleted. Optimization:duplicatescanbedeletedduringrungenerationaswell

asatintermediatemergestepsinexternalsortmerge. Hashingissimilarduplicateswillcomeintothesamebucket.

Projection: performprojectiononeachtuple followedbyduplicateelimination.


OtherOperations:AggregationOtherOperations:Aggregation

Aggregationcanbeimplementedinamannersimilartoduplicateelimination.

Sortingorhashingcanbeusedtobringtuplesinthesamegrouptogether,andthentheaggregatefunctionscanbeappliedoneachgroup.

Optimization:combinetuplesinthesamegroupduringrungenerationandintermediatemerges,bycomputingpartialaggregatevalues Forcount,min,max,sum:keepaggregatevaluesontuples

foundsofarinthegroup. Whencombiningpartialaggregateforcount,addupthe

aggregates Foravg,keepsumandcount,anddividesumbycountatthe

end


OtherOperations:SetOperationsOtherOperations:SetOperations Setoperations(,and):caneitherusevariantofmergejoinafter

sorting,orvariantofhashjoin. E.g.,Setoperationsusinghashing:

1. Partitionbothrelationsusingthesamehashfunction2. Processeachpartitioniasfollows.

1. Usingadifferenthashingfunction,buildaninmemoryhashindexonri.

2. Processsiasfollows rs:

1. Addtuplesinsitothehashindexiftheyarenotalreadyinit.2. Atendofsiaddthetuplesinthehashindextotheresult.

rs:1. outputtuplesinsitotheresultiftheyarealreadythereinthe

hashindex rs:

1. foreachtupleinsi,ifitisthereinthehashindex,deleteitfromtheindex.

2. Atendofsiaddremainingtuplesinthehashindextotheresult.


OtherOperations:OuterJoinOtherOperations:OuterJoin

Outerjoincanbecomputedeitheras Ajoinfollowedbyadditionofnullpaddednonparticipatingtuples. bymodifyingthejoinalgorithms.

Modifyingmergejointocomputers Inrs,nonparticipatingtuplesarethoseinrR(rs) Modifymergejointocomputers:Duringmerging,forevery

tupletrfromrthatdonotmatchanytupleins,outputtrpaddedwithnulls.

Rightouterjoinandfullouterjoincanbecomputedsimilarly. Modifyinghashjointocomputers

Ifrisproberelation,outputnonmatchingrtuplespaddedwithnulls Ifrisbuildrelation,whenprobingkeeptrackofwhich

rtuplesmatchedstuples. Atendofsioutputnonmatchedrtuplespaddedwithnulls


EvaluationofExpressionsEvaluationofExpressions

Sofar:wehaveseenalgorithmsforindividualoperations Alternativesforevaluatinganentireexpressiontree

Materialization:generateresultsofanexpressionwhoseinputsarerelationsorarealreadycomputed,materialize(store)itondisk.Repeat.

Pipelining:passontuplestoparentoperationsevenasanoperationisbeingexecuted

Westudyabovealternativesinmoredetail


MaterializationMaterialization

Materializedevaluation:evaluateoneoperationatatime,startingatthelowestlevel.Useintermediateresultsmaterializedintotemporaryrelationstoevaluatenextleveloperations.

E.g.,infigurebelow,computeandstore

thencomputethestoreitsjoinwithcustomer,andfinallycomputetheprojectionsoncustomername.

)(2500 accountbalance


Materialization(Cont.)Materialization(Cont.)

Materializedevaluationisalwaysapplicable Costofwritingresultstodiskandreadingthembackcanbequitehigh

Ourcostformulasforoperationsignorecostofwritingresultstodisk,so Overallcost=Sumofcostsofindividualoperations+

costofwritingintermediateresultstodisk Doublebuffering:usetwooutputbuffersforeachoperation,whenone

isfullwriteittodiskwhiletheotherisgettingfilled Allowsoverlapofdiskwriteswithcomputationandreduces

executiontime


PipeliningPipelining

Pipelinedevaluation:evaluateseveraloperationssimultaneously,passingtheresultsofoneoperationontothenext.

E.g.,inpreviousexpressiontree,dontstoreresultof

instead,passtuplesdirectlytothejoin..Similarly,dontstoreresultof

join,passtuplesdirectlytoprojection. Muchcheaperthanmaterialization:noneedtostoreatemporaryrelation

todisk. Pipeliningmaynotalwaysbepossiblee.g.,sort,hashjoin. Forpipeliningtobeeffective,useevaluationalgorithmsthatgenerate

outputtuplesevenastuplesarereceivedforinputstotheoperation. Pipelinescanbeexecutedintwoways:demanddrivenandproducer

driven

)(2500 accountbalance


Pipelining(Cont.)Pipelining(Cont.)

Indemanddrivenorlazyevaluation systemrepeatedlyrequestsnexttuplefromtopleveloperation Eachoperationrequestsnexttuplefromchildrenoperationsas

required,inordertooutputitsnexttuple Inbetweencalls,operationhastomaintainstatesoitknowswhat

toreturnnext Inproducerdrivenoreagerpipelining

Operatorsproducetupleseagerlyandpassthemuptotheirparents Buffermaintainedbetweenoperators,childputstuplesinbuffer,

parentremovestuplesfrombuffer ifbufferisfull,childwaitstillthereisspaceinthebuffer,andthen

generatesmoretuples Systemschedulesoperationsthathavespaceinoutputbufferand

canprocessmoreinputtuples Alternativename:pullandpushmodelsofpipelining


Pipelining(Cont.)Pipelining(Cont.)

Implementationofdemanddrivenpipelining Eachoperationisimplementedasaniteratorimplementingthe

followingoperations open()

E.g.filescan:initializefilescan state:pointertobeginningoffile

E.g.mergejoin:sortrelations; state:pointerstobeginningofsortedrelations

next() E.g.forfilescan:Outputnexttuple,andadvanceandstore

filepointer E.g.formergejoin:continuewithmergefromearlierstate

tillnextoutputtupleisfound.Savepointersasiteratorstate.

close()


EvaluationAlgorithmsforPipeliningEvaluationAlgorithmsforPipelining

Somealgorithmsarenotabletooutputresultsevenastheygetinputtuples

E.g.mergejoin,orhashjoin intermediateresultswrittentodiskandthenreadback

Algorithmvariantstogenerate(atleastsome)resultsonthefly,asinputtuplesarereadin

E.g.hybridhashjoingeneratesoutputtuplesevenasproberelationtuplesintheinmemorypartition(partition0)arereadin

Pipelinedjointechnique:Hybridhashjoin,modifiedtobufferpartition0tuplesofbothrelationsinmemory,readingthemastheybecomeavailable,andoutputresultsofanymatchesbetweenpartition0tuples Whenanewr0tupleisfound,matchitwithexistings0tuples,

outputmatches,andsaveitinr0 Symmetricallyfors0tuples

DatabaseSystemConcepts,5thEd.Silberschatz,KorthandSudarshan

Seewww.dbbook.comforconditionsonreuse

EndofChapterEndofChapter

http://www.db-book.com/


Figure13.2Figure13.2


ComplexJoinsComplexJoins

Joininvolvingthreerelations:loandepositorcustomer Strategy1.Computedepositorcustomer;useresulttocompute

loan(depositorcustomer) Strategy2.Computerloandepositorfirst,andthenjointheresult

withcustomer. Strategy3.Performthepairofjoinsatonce.Buildandindexon

loanforloannumber,andoncustomerforcustomername. Foreachtupletindepositor,lookupthecorrespondingtuples

incustomerandthecorrespondingtuplesinloan. Eachtupleofdepositisexaminedexactlyonce.

Strategy3combinestwooperationsintoonespecialpurposeoperationthatismoreefficientthanimplementingtwojoinsoftworelations.

chapter 13: query processing - cse, iit bombaysudarsha/db-book/slide-dir/ch13.pdf · chapter 13:...

Documents

H.Lu/HKUST L05: Query Processing & Tuning Queries Query Processing -- Principles Tuning Queries

MADRID! Query Processing

Query Processing

Chapter 13: Query Processing Basic Steps in Query Processing

Query Processing and Optimization. Query processing Overview The activities involved in retrieving data from the database. Query processing is the procedure

13 query processing

Query Processing Optimization

Adaptive Query Processing

8. Query Processing

QUERY OPTIMIZATION AND QUERY PROCESSING

Query Processing Overview Iterators iteratorinst.eecs.berkeley.edu/~cs186/sp06/lecs/lecture12QPIntro.pdf · 2006-02-26 · • Exposure to query processing architecture – Query

Query Processing: Joins

Query Processing and Query Optimization

Query processing and optimization. Advanced DatabasesQuery processing and optimization2 Definitions Query processing –translation of query into low-level

Query Processing Chapter 21 in Textbook. Overview What is Query Processing? What is Query Optimization? Example. Phases of Query Processing. 1

File Processing : Query Processing

Query Processing and Optimization. Query Processing Efficient Query Processing crucial for good or even effective operations of a database Query Processing

Query Processing: Query Formulation

Distributed Query Processing

Query Processing and Query Optmization

QUERY PROCESSING QUERY PROCESSING 1. AN OVERVIEW OF QUERY PROCESSING 2. FAST ACCESS PATHS 3. TRANFORMATION RULES 4. ALGEBRA-BASED OPTIMIZATION

Query Processing Chapter 21 in Textbook. Overview What is Query Processing? What is Query Optimization? Example. Phases of Query Processing. 1. Decomposition

Basic Steps in Query Processing Chapter 13: Query ... - SLU · Chapter 13: Query Processing Chapter 13: Query Processing!Overiw!M ea surofQyCt!S elc tionOpra!Sorting!J oinOperat!

Structure Query Processing – Data models – Query models – Approaches – Challenges Keyword query processing on RDF Structured query processing on RDF Structured

Query Processing + Optimization: Outlinepeople.cs.aau.dk/~luhua/courses/itev-db08/2.3.pdf · 1 Query Processing + Optimization: Outline • Operator Evaluation Strategies Query processing

Query Processing Notes - University of California, San Diegodb.ucsd.edu/static/CSE232F15/handouts/QueryProcessing.pdf · Query Processing Notes CSE232 Query Processing • The query

1 5. Distributed Query Processing Chapter 7 Overview of Query Processing Chapter 8 Query Decomposition and Data Localization

9. Query Processing

Introduction :- Query Processing :