chapter 13: query processing - cse, iit bombaysudarsha/db-book/slide-dir/ch13.pdf · chapter 13:...

of 55 /55
Database System Concepts, 5th Ed. ©Silberschatz, Korth and Sudarshan See www.db-book.com for conditions on re-use Chapter 13: Query Processing Chapter 13: Query Processing Aug 10, 2006

Author: hathu

Post on 30-Jan-2018

222 views

Category:

Documents


4 download

Embed Size (px)

TRANSCRIPT

  • DatabaseSystemConcepts,5thEd.Silberschatz,KorthandSudarshan

    Seewww.dbbook.comforconditionsonreuse

    Chapter13:QueryProcessingChapter13:QueryProcessing

    Aug10,2006

    http://www.db-book.com/

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    Chapter13:QueryProcessingChapter13:QueryProcessing

    Overview MeasuresofQueryCost SelectionOperation Sorting JoinOperation OtherOperations EvaluationofExpressions

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    BasicStepsinQueryProcessingBasicStepsinQueryProcessing

    1. Parsingandtranslation2. Optimization3. Evaluation

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    BasicStepsinQueryProcessingBasicStepsinQueryProcessing(Cont.)(Cont.)

    Parsingandtranslation translatethequeryintoitsinternalform.Thisisthentranslatedinto

    relationalalgebra. Parsercheckssyntax,verifiesrelations

    Evaluation Thequeryexecutionenginetakesaqueryevaluationplan,executes

    thatplan,andreturnstheanswerstothequery.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    BasicStepsinQueryProcessing:BasicStepsinQueryProcessing:OptimizationOptimization

    Arelationalalgebraexpressionmayhavemanyequivalentexpressions E.g.,balance

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    BasicSteps:Optimization(Cont.)BasicSteps:Optimization(Cont.)

    QueryOptimization:Amongstallequivalentevaluationplanschoosetheonewithlowestcost.

    Costisestimatedusingstatisticalinformationfromthedatabasecatalog e.g.numberoftuplesineachrelation,sizeoftuples,etc.

    Inthischapterwestudy Howtomeasurequerycosts Algorithmsforevaluatingrelationalalgebraoperations Howtocombinealgorithmsforindividualoperationsinorderto

    evaluateacompleteexpression InChapter14

    Westudyhowtooptimizequeries,thatis,howtofindanevaluationplanwithlowestestimatedcost

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    MeasuresofQueryCostMeasuresofQueryCost

    Costisgenerallymeasuredastotalelapsedtimeforansweringquery

    Manyfactorscontributetotimecost diskaccesses,CPU,orevennetworkcommunication

    Typicallydiskaccessisthepredominantcost,andisalsorelativelyeasytoestimate.Measuredbytakingintoaccount

    Numberofseeks*averageseekcost+Numberofblocksread*averageblockreadcost+Numberofblockswritten*averageblockwritecost

    Costtowriteablockisgreaterthancosttoreadablock dataisreadbackafterbeingwrittentoensurethatthe

    writewassuccessful Assumption:singledisk

    Canmodifyformulaeformultipledisks/RAIDarrays Orjustusesinglediskformulae,butinterpretthemas

    measuringresourceconsumptioninsteadoftime

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    MeasuresofQueryCost(Cont.)MeasuresofQueryCost(Cont.)

    Forsimplicitywejustusethenumberofblocktransfersfromdiskandthenumberofseeksasthecostmeasures

    tTtimetotransferoneblock tStimeforoneseek CostforbblocktransfersplusSseeks

    b*tT+S*tS WeignoreCPUcostsforsimplicity

    RealsystemsdotakeCPUcostintoaccount Wedonotincludecosttowritingoutputtodiskinourcostformulae SeveralalgorithmscanreducediskIObyusingextrabufferspace

    AmountofrealmemoryavailabletobufferdependsonotherconcurrentqueriesandOSprocesses,knownonlyduringexecution Weoftenuseworstcaseestimates,assumingonlytheminimum

    amountofmemoryneededfortheoperationisavailable Requireddatamaybebufferresidentalready,avoidingdiskI/O

    Buthardtotakeintoaccountforcostestimation

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    SelectionOperationSelectionOperation

    Filescansearchalgorithmsthatlocateandretrieverecordsthatfulfillaselectioncondition.

    AlgorithmA1(linearsearch).Scaneachfileblockandtestallrecordstoseewhethertheysatisfytheselectioncondition.

    Costestimate=brblocktransfers+1seek brdenotesnumberofblockscontainingrecordsfromrelationr

    Ifselectionisonakeyattribute,canstoponfindingrecord cost=(br/2)blocktransfers+1seek

    Linearsearchcanbeappliedregardlessof selectionconditionor orderingofrecordsinthefile,or availabilityofindices

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    SelectionOperation(Cont.)SelectionOperation(Cont.)

    A2(binarysearch).Applicableifselectionisanequalitycomparisonontheattributeonwhichfileisordered.

    Assumethattheblocksofarelationarestoredcontiguously Costestimate(numberofdiskblockstobescanned):

    costoflocatingthefirsttuplebyabinarysearchontheblocks log2(br)*(tT+tS)

    Iftherearemultiplerecordssatisfyingselection Addtransfercostofthenumberofblockscontaining

    recordsthatsatisfyselectioncondition WillseehowtoestimatethiscostinChapter14

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    SelectionsUsingIndicesSelectionsUsingIndices

    Indexscansearchalgorithmsthatuseanindex selectionconditionmustbeonsearchkeyofindex.

    A3(primaryindexoncandidatekey,equality).Retrieveasinglerecordthatsatisfiesthecorrespondingequalitycondition

    Cost=(hi+1)*(tT+tS) A4(primaryindexonnonkey,equality)Retrievemultiplerecords.

    Recordswillbeonconsecutiveblocks Letb=numberofblockscontainingmatchingrecords

    Cost=hi*(tT+tS)+tS+tT*b A5(equalityonsearchkeyofsecondaryindex).

    Retrieveasinglerecordifthesearchkeyisacandidatekey Cost=(hi+1)*(tT+tS)

    Retrievemultiplerecordsifsearchkeyisnotacandidatekey eachofnmatchingrecordsmaybeonadifferentblock Cost=(hi+n)*(tT+tS)

    Canbeveryexpensive!

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    SelectionsInvolvingComparisonsSelectionsInvolvingComparisons

    CanimplementselectionsoftheformAV(r)orAV(r)byusing alinearfilescanorbinarysearch, orbyusingindicesinthefollowingways:

    A6(primaryindex,comparison).(RelationissortedonA) ForAV(r)useindextofindfirsttuplevandscanrelation

    sequentiallyfromthere ForAV(r)justscanrelationsequentiallytillfirsttuple>v;donotuse

    index A7(secondaryindex,comparison).

    ForAV(r)useindextofindfirstindexentryvandscanindexsequentiallyfromthere,tofindpointerstorecords.

    ForAV(r)justscanleafpagesofindexfindingpointerstorecords,tillfirstentry>v

    Ineithercase,retrieverecordsthatarepointedto requiresanI/Oforeachrecord Linearfilescanmaybecheaper

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    ImplementationofComplexSelectionsImplementationofComplexSelections

    Conjunction:12...n(r) A8(conjunctiveselectionusingoneindex).

    SelectacombinationofiandalgorithmsA1throughA7thatresultsintheleastcostfori(r).

    Testotherconditionsontupleafterfetchingitintomemorybuffer. A9(conjunctiveselectionusingmultiplekeyindex).

    Useappropriatecomposite(multiplekey)indexifavailable. A10(conjunctiveselectionbyintersectionofidentifiers).

    Requiresindiceswithrecordpointers. Usecorrespondingindexforeachcondition,andtakeintersection

    ofalltheobtainedsetsofrecordpointers. Thenfetchrecordsfromfile Ifsomeconditionsdonothaveappropriateindices,applytestin

    memory.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    AlgorithmsforComplexSelectionsAlgorithmsforComplexSelections

    Disjunction:12...n(r). A11(disjunctiveselectionbyunionofidentifiers).

    Applicableifallconditionshaveavailableindices. Otherwiseuselinearscan.

    Usecorrespondingindexforeachcondition,andtakeunionofalltheobtainedsetsofrecordpointers.

    Thenfetchrecordsfromfile Negation:(r)

    Uselinearscanonfile Ifveryfewrecordssatisfy,andanindexisapplicableto

    Findsatisfyingrecordsusingindexandfetchfromfile

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    SortingSorting

    Wemaybuildanindexontherelation,andthenusetheindextoreadtherelationinsortedorder.Mayleadtoonediskblockaccessforeachtuple.

    Forrelationsthatfitinmemory,techniqueslikequicksortcanbeused.Forrelationsthatdontfitinmemory,externalsortmergeisagoodchoice.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    ExternalSortMergeExternalSortMerge

    1. Createsortedruns.Letibe0initially.Repeatedlydothefollowingtilltheendoftherelation:(a)ReadMblocksofrelationintomemory(b)Sorttheinmemoryblocks(c)WritesorteddatatorunRi;incrementi.LetthefinalvalueofibeN

    2. Mergetheruns(nextslide)..

    LetMdenotememorysize(inpages).

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    ExternalSortMerge(Cont.)ExternalSortMerge(Cont.)

    1. Mergetheruns(Nwaymerge).Weassume(fornow)thatN

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    ExternalSortMerge(Cont.)ExternalSortMerge(Cont.)

    IfNM,severalmergepassesarerequired. Ineachpass,contiguousgroupsofM1runsaremerged. ApassreducesthenumberofrunsbyafactorofM1,and

    createsrunslongerbythesamefactor. E.g.IfM=11,andthereare90runs,onepassreduces

    thenumberofrunsto9,each10timesthesizeoftheinitialruns

    Repeatedpassesareperformedtillallrunshavebeenmergedintoone.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    Example:ExternalSortingUsingSortMergeExample:ExternalSortingUsingSortMerge

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    ExternalMergeSort(Cont.)ExternalMergeSort(Cont.)

    Costanalysis: Totalnumberofmergepassesrequired:logM1(br/M). Blocktransfersforinitialruncreationaswellasineach

    passis2br forfinalpass,wedontcountwritecost

    weignorefinalwritecostforalloperationssincetheoutputofanoperationmaybesenttotheparentoperationwithoutbeingwrittentodisk

    Thustotalnumberofblocktransfersforexternalsorting:br(2logM1(br/M)+1)

    Seeks:nextslide

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    ExternalMergeSort(Cont.)ExternalMergeSort(Cont.)

    Costofseeks Duringrungeneration:oneseektoreadeachrunandoneseekto

    writeeachrun 2br/M

    Duringthemergephase Buffersize:bb(read/writebbblocksatatime) Need2br/bbseeksforeachmergepass

    exceptthefinalonewhichdoesnotrequireawrite Totalnumberofseeks:

    2br/M+br/bb(2logM1(br/M)1)

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    JoinOperationJoinOperation

    Severaldifferentalgorithmstoimplementjoins Nestedloopjoin Blocknestedloopjoin Indexednestedloopjoin Mergejoin Hashjoin

    Choicebasedoncostestimate Examplesusethefollowinginformation

    Numberofrecordsofcustomer:10,000depositor:5000 Numberofblocksofcustomer:400depositor:100

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    NestedLoopJoinNestedLoopJoin

    Tocomputethethetajoinrsforeachtupletrinrdobeginforeachtupletsinsdobegin

    testpair(tr,ts)toseeiftheysatisfythejoinconditioniftheydo,addtrtstotheresult.

    endend

    riscalledtheouterrelationandstheinnerrelationofthejoin. Requiresnoindicesandcanbeusedwithanykindofjoincondition. Expensivesinceitexamineseverypairoftuplesinthetworelations.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    NestedLoopJoin(Cont.)NestedLoopJoin(Cont.)

    Intheworstcase,ifthereisenoughmemoryonlytoholdoneblockofeachrelation,theestimatedcostisnrbs+brblocktransfers,plusnr+brseeks

    Ifthesmallerrelationfitsentirelyinmemory,usethatastheinnerrelation. Reducescosttobr+bsblocktransfersand2seeks

    Assumingworstcasememoryavailabilitycostestimateis withdepositorasouterrelation:

    5000400+100=2,000,100blocktransfers, 5000+100=5100seeks

    withcustomerastheouterrelation 10000100+400=1,000,400blocktransfersand10,400seeks

    Ifsmallerrelation(depositor)fitsentirelyinmemory,thecostestimatewillbe500blocktransfers.

    Blocknestedloopsalgorithm(nextslide)ispreferable.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    BlockNestedLoopJoinBlockNestedLoopJoin

    Variantofnestedloopjoininwhicheveryblockofinnerrelationispairedwitheveryblockofouterrelation.foreachblockBrofrdobegin

    foreachblockBsofsdobeginforeachtupletrinBrdobegin

    foreachtupletsinBsdobeginCheckif(tr,ts)satisfythejoinconditioniftheydo,addtrtstotheresult.

    endend

    endend

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    BlockNestedLoopJoin(Cont.)BlockNestedLoopJoin(Cont.)

    Worstcaseestimate:brbs+brblocktransfers+2*brseeks Eachblockintheinnerrelationsisreadonceforeachblockinthe

    outerrelation(insteadofonceforeachtupleintheouterrelation Bestcase:br+bsblocktransfers+2seeks. Improvementstonestedloopandblocknestedloopalgorithms:

    Inblocknestedloop,useM2diskblocksasblockingunitforouterrelations,whereM=memorysizeinblocks;useremainingtwoblockstobufferinnerrelationandoutput Cost=br/(M2)bs+brblocktransfers+

    2br/(M2)seeks Ifequijoinattributeformsakeyoninnerrelation,stopinnerloop

    onfirstmatch Scaninnerloopforwardandbackwardalternately,tomakeuseof

    theblocksremaininginbuffer(withLRUreplacement) Useindexoninnerrelationifavailable(nextslide)

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    IndexedNestedLoopJoinIndexedNestedLoopJoin

    Indexlookupscanreplacefilescansif joinisanequijoinornaturaljoinand anindexisavailableontheinnerrelationsjoinattribute

    Canconstructanindexjusttocomputeajoin. Foreachtupletrintheouterrelationr,usetheindextolookuptuplesins

    thatsatisfythejoinconditionwithtupletr. Worstcase:bufferhasspaceforonlyonepageofr,and,foreachtuple

    inr,weperformanindexlookupons. Costofthejoin:br(tT+tS)+nrc

    Wherecisthecostoftraversingindexandfetchingallmatchingstuplesforonetupleorr

    ccanbeestimatedascostofasingleselectiononsusingthejoincondition.

    Ifindicesareavailableonjoinattributesofbothrands,usetherelationwithfewertuplesastheouterrelation.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    ExampleofNestedLoopJoinCostsExampleofNestedLoopJoinCosts

    Computedepositorcustomer,withdepositorastheouterrelation. LetcustomerhaveaprimaryB+treeindexonthejoinattribute

    customername,whichcontains20entriesineachindexnode. Sincecustomerhas10,000tuples,theheightofthetreeis4,andone

    moreaccessisneededtofindtheactualdata depositorhas5000tuples Costofblocknestedloopsjoin

    400*100+100=40,100blocktransfers+2*100=200seeks assumingworstcasememory maybesignificantlylesswithmorememory

    Costofindexednestedloopsjoin 100+5000*5=25,100blocktransfersandseeks. CPUcostlikelytobelessthanthatforblocknestedloopsjoin

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    MergeJoinMergeJoin

    1. Sortbothrelationsontheirjoinattribute(ifnotalreadysortedonthejoinattributes).

    2. Mergethesortedrelationstojointhem1. Joinstepissimilartothemergestageofthesortmergealgorithm.2. Maindifferenceishandlingofduplicatevaluesinjoinattributeevery

    pairwithsamevalueonjoinattributemustbematched3. Detailedalgorithminbook

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    MergeJoin(Cont.)MergeJoin(Cont.)

    Canbeusedonlyforequijoinsandnaturaljoins Eachblockneedstobereadonlyonce(assumingalltuplesforanygiven

    valueofthejoinattributesfitinmemory Thusthecostofmergejoinis:

    br+bsblocktransfers+br/bb+bs/bbseeks +thecostofsortingifrelationsareunsorted.

    hybridmergejoin:Ifonerelationissorted,andtheotherhasasecondaryB+treeindexonthejoinattribute

    MergethesortedrelationwiththeleafentriesoftheB+tree. Sorttheresultontheaddressesoftheunsortedrelationstuples Scantheunsortedrelationinphysicaladdressorderandmergewith

    previousresult,toreplaceaddressesbytheactualtuples Sequentialscanmoreefficientthanrandomlookup

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    HashJoinHashJoin

    Applicableforequijoinsandnaturaljoins. Ahashfunctionhisusedtopartitiontuplesofbothrelations

    Intuition:partitionsfitinmemory hmapsJoinAttrsvaluesto{0,1,...,n},whereJoinAttrsdenotesthe

    commonattributesofrandsusedinthenaturaljoin. r0,r1,...,rndenotepartitionsofrtuples

    Eachtupletrrisputinpartitionriwherei=h(tr[JoinAttrs]).

    r0,,r1...,rndenotespartitionsofstuples

    Eachtupletssisputinpartitionsi,wherei=h(ts[JoinAttrs]).

    Note:Inbook,riisdenotedasHri,siisdenotedasHsiandnisdenotedasnh.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    HashJoin(Cont.)HashJoin(Cont.)

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    HashJoin(Cont.)HashJoin(Cont.)

    rtuplesinrineedonlytobecomparedwithstuplesinsiNeednotbecomparedwithstuplesinanyotherpartition,since:

    anrtupleandanstuplethatsatisfythejoinconditionwillhavethesamevalueforthejoinattributes.

    Ifthatvalueishashedtosomevaluei,thertuplehastobeinriandthestupleinsi.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    HashJoinAlgorithmHashJoinAlgorithm

    1. Partitiontherelationsusinghashingfunctionh.1. Whenpartitioningarelation,oneblockofmemoryisreservedasthe

    outputbufferforeachpartition,andoneblockforinput2. Ifextramemoryisavailable,allocatebbblocksasbufferforinputand

    eachoutput2. Partitionrsimilarly.3.nextslide..

    Thehashjoinofrandsiscomputedasfollows.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    HashJoin(Cont.)HashJoin(Cont.)

    1. Foreachpartitioni:(a) Loadsiintomemoryandbuildaninmemoryhashindexonit

    usingthejoinattribute. Thishashindexusesadifferenthashfunctionthantheearlier

    oneh.(b) Readthetuplesinrifromthediskonebyone.

    Foreachtupletrprobetheinmemoryhashindextofindallmatchingtuplestsinsi Foreachmatchingtupletsinsi

    outputtheconcatenationoftheattributesoftrandts

    Relationsiscalledthebuildinputandriscalledtheprobeinput.

    HashJoinAlgorithm(cont)

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    HashJoinalgorithm(Cont.)HashJoinalgorithm(Cont.)

    Thevaluenandthehashfunctionhischosensuchthateachsishouldfitinmemory.

    Typicallynischosenasbs/M*fwherefisafudgefactor,typicallyaround1.2

    Theproberelationpartitionssineednotfitinmemory Recursivepartitioningrequiredifnumberofpartitionsnisgreater

    thannumberofpagesMofmemory. insteadofpartitioningnways,useM1partitionsfors FurtherpartitiontheM1partitionsusingadifferenthash

    function Usesamepartitioningmethodonr Rarelyrequired:e.g.,recursivepartitioningnotneededfor

    relationsof1GBorlesswithmemorysizeof2MB,withblocksizeof4KB.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    HandlingofOverflowsHandlingofOverflows

    Partitioningissaidtobeskewedifsomepartitionshavesignificantlymoretuplesthansomeothers

    Hashtableoverflowoccursinpartitionsiifsidoesnotfitinmemory.Reasonscouldbe

    Manytuplesinswithsamevalueforjoinattributes Badhashfunction

    Overflowresolutioncanbedoneinbuildphase Partitionsiisfurtherpartitionedusingdifferenthashfunction.

    Partitionrimustbesimilarlypartitioned. Overflowavoidanceperformspartitioningcarefullytoavoidoverflows

    duringbuildphase E.g.partitionbuildrelationintomanypartitions,thencombinethem

    Bothapproachesfailwithlargenumbersofduplicates Fallbackoption:useblocknestedloopsjoinonoverflowedpartitions

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    CostofHashJoinCostofHashJoin

    Ifrecursivepartitioningisnotrequired:costofhashjoinis3(br+bs)+4nhblocktransfers+2(br/bb+bs/bb)seeks

    Ifrecursivepartitioningrequired: numberofpassesrequiredforpartitioningbuildrelation

    sislogM1(bs)1 besttochoosethesmallerrelationasthebuildrelation. Totalcostestimateis:

    2(br+bslogM1(bs)1+br+bsblocktransfers+2(br/bb+bs/bb)logM1(bs)1seeks

    Iftheentirebuildinputcanbekeptinmainmemorynopartitioningisrequired

    Costestimategoesdowntobr+bs.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    ExampleofCostofHashJoinExampleofCostofHashJoin

    Assumethatmemorysizeis20blocks bdepositor=100andbcustomer=400. depositoristobeusedasbuildinput.Partitionitintofivepartitions,each

    ofsize20blocks.Thispartitioningcanbedoneinonepass. Similarly,partitioncustomerintofivepartitions,eachofsize80.Thisisalso

    doneinonepass. Thereforetotalcost,ignoringcostofwritingpartiallyfilledblocks:

    3(100+400)=1500blocktransfers+2(100/3+400/3)=336seeks

    customerdepositor

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    HybridHashJoinHybridHashJoin

    Usefulwhenmemorysizedarerelativelylarge,andthebuildinputisbiggerthanmemory.

    Mainfeatureofhybridhashjoin:Keepthefirstpartitionofthebuildrelationinmemory. E.g.Withmemorysizeof25blocks,depositorcanbepartitionedintofive

    partitions,eachofsize20blocks. Divisionofmemory:

    Thefirstpartitionoccupies20blocksofmemory 1blockisusedforinput,and1blockeachforbufferingtheother4

    partitions. customerissimilarlypartitionedintofivepartitionseachofsize80

    thefirstisusedrightawayforprobing,insteadofbeingwrittenout Costof3(80+320)+20+80=1300blocktransfersfor

    hybridhashjoin,insteadof1500withplainhashjoin. HybridhashjoinmostusefulifM>> sb

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    ComplexJoinsComplexJoins

    Joinwithaconjunctivecondition:r12...ns

    Eitherusenestedloops/blocknestedloops,or Computetheresultofoneofthesimplerjoinsris

    finalresultcomprisesthosetuplesintheintermediateresultthatsatisfytheremainingconditions

    1...i1i+1...n Joinwithadisjunctivecondition

    r12...ns Eitherusenestedloops/blocknestedloops,or Computeastheunionoftherecordsinindividualjoinsris:

    (r1s)(r2s)...(rns)

    (appliesonlytothesetversionofunion!)

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    OtherOperationsOtherOperations

    Duplicateeliminationcanbeimplementedviahashingorsorting. Onsortingduplicateswillcomeadjacenttoeachother,andallbut

    onesetofduplicatescanbedeleted. Optimization:duplicatescanbedeletedduringrungenerationaswell

    asatintermediatemergestepsinexternalsortmerge. Hashingissimilarduplicateswillcomeintothesamebucket.

    Projection: performprojectiononeachtuple followedbyduplicateelimination.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    OtherOperations:AggregationOtherOperations:Aggregation

    Aggregationcanbeimplementedinamannersimilartoduplicateelimination.

    Sortingorhashingcanbeusedtobringtuplesinthesamegrouptogether,andthentheaggregatefunctionscanbeappliedoneachgroup.

    Optimization:combinetuplesinthesamegroupduringrungenerationandintermediatemerges,bycomputingpartialaggregatevalues Forcount,min,max,sum:keepaggregatevaluesontuples

    foundsofarinthegroup. Whencombiningpartialaggregateforcount,addupthe

    aggregates Foravg,keepsumandcount,anddividesumbycountatthe

    end

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    OtherOperations:SetOperationsOtherOperations:SetOperations Setoperations(,and):caneitherusevariantofmergejoinafter

    sorting,orvariantofhashjoin. E.g.,Setoperationsusinghashing:

    1. Partitionbothrelationsusingthesamehashfunction2. Processeachpartitioniasfollows.

    1. Usingadifferenthashingfunction,buildaninmemoryhashindexonri.

    2. Processsiasfollows rs:

    1. Addtuplesinsitothehashindexiftheyarenotalreadyinit.2. Atendofsiaddthetuplesinthehashindextotheresult.

    rs:1. outputtuplesinsitotheresultiftheyarealreadythereinthe

    hashindex rs:

    1. foreachtupleinsi,ifitisthereinthehashindex,deleteitfromtheindex.

    2. Atendofsiaddremainingtuplesinthehashindextotheresult.

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    OtherOperations:OuterJoinOtherOperations:OuterJoin

    Outerjoincanbecomputedeitheras Ajoinfollowedbyadditionofnullpaddednonparticipatingtuples. bymodifyingthejoinalgorithms.

    Modifyingmergejointocomputers Inrs,nonparticipatingtuplesarethoseinrR(rs) Modifymergejointocomputers:Duringmerging,forevery

    tupletrfromrthatdonotmatchanytupleins,outputtrpaddedwithnulls.

    Rightouterjoinandfullouterjoincanbecomputedsimilarly. Modifyinghashjointocomputers

    Ifrisproberelation,outputnonmatchingrtuplespaddedwithnulls Ifrisbuildrelation,whenprobingkeeptrackofwhich

    rtuplesmatchedstuples. Atendofsioutputnonmatchedrtuplespaddedwithnulls

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    EvaluationofExpressionsEvaluationofExpressions

    Sofar:wehaveseenalgorithmsforindividualoperations Alternativesforevaluatinganentireexpressiontree

    Materialization:generateresultsofanexpressionwhoseinputsarerelationsorarealreadycomputed,materialize(store)itondisk.Repeat.

    Pipelining:passontuplestoparentoperationsevenasanoperationisbeingexecuted

    Westudyabovealternativesinmoredetail

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    MaterializationMaterialization

    Materializedevaluation:evaluateoneoperationatatime,startingatthelowestlevel.Useintermediateresultsmaterializedintotemporaryrelationstoevaluatenextleveloperations.

    E.g.,infigurebelow,computeandstore

    thencomputethestoreitsjoinwithcustomer,andfinallycomputetheprojectionsoncustomername.

    )(2500 accountbalance

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    Materialization(Cont.)Materialization(Cont.)

    Materializedevaluationisalwaysapplicable Costofwritingresultstodiskandreadingthembackcanbequitehigh

    Ourcostformulasforoperationsignorecostofwritingresultstodisk,so Overallcost=Sumofcostsofindividualoperations+

    costofwritingintermediateresultstodisk Doublebuffering:usetwooutputbuffersforeachoperation,whenone

    isfullwriteittodiskwhiletheotherisgettingfilled Allowsoverlapofdiskwriteswithcomputationandreduces

    executiontime

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    PipeliningPipelining

    Pipelinedevaluation:evaluateseveraloperationssimultaneously,passingtheresultsofoneoperationontothenext.

    E.g.,inpreviousexpressiontree,dontstoreresultof

    instead,passtuplesdirectlytothejoin..Similarly,dontstoreresultof

    join,passtuplesdirectlytoprojection. Muchcheaperthanmaterialization:noneedtostoreatemporaryrelation

    todisk. Pipeliningmaynotalwaysbepossiblee.g.,sort,hashjoin. Forpipeliningtobeeffective,useevaluationalgorithmsthatgenerate

    outputtuplesevenastuplesarereceivedforinputstotheoperation. Pipelinescanbeexecutedintwoways:demanddrivenandproducer

    driven

    )(2500 accountbalance

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    Pipelining(Cont.)Pipelining(Cont.)

    Indemanddrivenorlazyevaluation systemrepeatedlyrequestsnexttuplefromtopleveloperation Eachoperationrequestsnexttuplefromchildrenoperationsas

    required,inordertooutputitsnexttuple Inbetweencalls,operationhastomaintainstatesoitknowswhat

    toreturnnext Inproducerdrivenoreagerpipelining

    Operatorsproducetupleseagerlyandpassthemuptotheirparents Buffermaintainedbetweenoperators,childputstuplesinbuffer,

    parentremovestuplesfrombuffer ifbufferisfull,childwaitstillthereisspaceinthebuffer,andthen

    generatesmoretuples Systemschedulesoperationsthathavespaceinoutputbufferand

    canprocessmoreinputtuples Alternativename:pullandpushmodelsofpipelining

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    Pipelining(Cont.)Pipelining(Cont.)

    Implementationofdemanddrivenpipelining Eachoperationisimplementedasaniteratorimplementingthe

    followingoperations open()

    E.g.filescan:initializefilescan state:pointertobeginningoffile

    E.g.mergejoin:sortrelations; state:pointerstobeginningofsortedrelations

    next() E.g.forfilescan:Outputnexttuple,andadvanceandstore

    filepointer E.g.formergejoin:continuewithmergefromearlierstate

    tillnextoutputtupleisfound.Savepointersasiteratorstate.

    close()

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    EvaluationAlgorithmsforPipeliningEvaluationAlgorithmsforPipelining

    Somealgorithmsarenotabletooutputresultsevenastheygetinputtuples

    E.g.mergejoin,orhashjoin intermediateresultswrittentodiskandthenreadback

    Algorithmvariantstogenerate(atleastsome)resultsonthefly,asinputtuplesarereadin

    E.g.hybridhashjoingeneratesoutputtuplesevenasproberelationtuplesintheinmemorypartition(partition0)arereadin

    Pipelinedjointechnique:Hybridhashjoin,modifiedtobufferpartition0tuplesofbothrelationsinmemory,readingthemastheybecomeavailable,andoutputresultsofanymatchesbetweenpartition0tuples Whenanewr0tupleisfound,matchitwithexistings0tuples,

    outputmatches,andsaveitinr0 Symmetricallyfors0tuples

  • DatabaseSystemConcepts,5thEd.Silberschatz,KorthandSudarshan

    Seewww.dbbook.comforconditionsonreuse

    EndofChapterEndofChapter

    http://www.db-book.com/

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    Figure13.2Figure13.2

  • Silberschatz,KorthandSudarshan13.DatabaseSystemConcepts5thEdition.

    ComplexJoinsComplexJoins

    Joininvolvingthreerelations:loandepositorcustomer Strategy1.Computedepositorcustomer;useresulttocompute

    loan(depositorcustomer) Strategy2.Computerloandepositorfirst,andthenjointheresult

    withcustomer. Strategy3.Performthepairofjoinsatonce.Buildandindexon

    loanforloannumber,andoncustomerforcustomername. Foreachtupletindepositor,lookupthecorrespondingtuples

    incustomerandthecorrespondingtuplesinloan. Eachtupleofdepositisexaminedexactlyonce.

    Strategy3combinestwooperationsintoonespecialpurposeoperationthatismoreefficientthanimplementingtwojoinsoftworelations.