lecture 2 - web.stanford.edu
TRANSCRIPT
Somequestionsfromlasttime• What’sthebestwaytotakenotes?• Iwillpostslidesaheadoftime.• Sometimestheslideswillrefertoexplanationsontheboard– Iencourageyoutotakenotesforthatstuff.
• What’swiththewaitlist?• WegotanotherTA!(DannyWright).Enrollmentwillbeopeningup(stilllimited,buteveryonewhowasonthewaitlistat8amthismorningwillgetaspot).
Logisticssomemore!• Asbefore:http://web.stanford.edu/class/cs161/• Therewillbeofficehours!Andrecitationsections!
MTu WTh
(screenshotsubjecttochange)
Thedealwithsections:
• HeldonMondaysandTuesdays:goovercontentfrompreviousweek,withexampleproblems!
• Don’tneedtosignup,gotoany/eithersection!
• Problems/solutionsfrombothsectionswillbeposted!
Signupfor
Homework!We’regonna haveit!
• AssignmentwillbepostedFRIDAY.(3pm)• DuethenextFRIDAY.(3pm)• SeetheCOURSEWEBSITEforHWguidelines.• SolutionsmostthefollowingMONDAY.
MondayTuesdayWednesdayThursdayFriday
HWASSIGNED
HWDUE
SOLUTIONSRELEASED
HWRETURNED(target)
Lasttime
• Algorithmsareawesomeandpowerful!• Algorithmdesigner’squestion:CanIdobetter?• Interplaybetweenrigorand intuition.
• Karatsubaintegermultiplication• Exampleof“DivideandConquer”• Not-so-rigorousanalysis
Philosophy
TechnicalcontentPluckythepedanticpenguin
Luckythelackadaisical
lemur.
Today
• Sorting• Returnofdivide-and-conquer with MergeSort• Skills:• Analyzingcorrectnessofiterativeandrecursivealgorithms.• Analyzingrunningtimeofrecursivealgorithms.
• Howdowemeasuretheruntimeofanalgorithm?• Worst-caseanalysis• AsymptoticAnalysis
Sorting
• Importantprimitive• Fortoday,we’llpretendallelementsaredistinct.
6 4 3 8 1 5 2 7
1 2 3 4 5 6 7 8
Benchmark:insertionsort
• Saywewanttosort:
• Insertitemsoneatatime.
• Howwouldweactuallyimplementthis?
6 4 3 8 5
We’regoingtogothroughthisinsome
detail– it’sgoodpractice!
46 3 8 5
Insertionsortexample…
6 4 3 8 5
64 3 8 5
Startwiththesecondelement(thefirstelementissortedwithinitself…)
Pull“4”backuntilit’sintherightplace.
64 3 8 5 Nowlookat“3”
4 63 8 5 Pull“3”backuntilit’sintherightplace.
43 6 8 5 “8”isgood…lookat5
43 6 85 (thenfix5andwe’redone)
Insertionsortpseudocode
• (Discussiononboard)
Luckythelackadaisicallemur
Goone-at-a-timeuntilthingsareintherightplace.
Pluckythepedanticpenguin
Insertionsort:runningtime
n-1iterationsoftheouterloop
Intheworstcase,aboutniterationsofthisinnerloop
Runningtimeisaboutn2
Insertionsort:correctness
• Maintainaloopinvariant.
• Initialization:theloopinvariantholdsbeforethefirstiteration.• Maintenance:Ifitistruebeforethet’th iteration,itwillbetruebeforethe(t+1)’st iteration• Termination:Itisusefultoknowthattheloopinvariantistrueattheendofthelastiteration.
Aloopinvariantissomethingthatshouldbetrueateveryiteration.
(Thisisproof-of-correctnessbyinduction)
Insertionsort:correctness
• Loopinvariant:Atthestartofthet’th iteration(oftheouterloop),thefirstt elementsofthearrayaresorted.• Initialization:Atthestartofthefirstiteration,thefirstelementofthearrayissorted.✓• Maintenance:Byconstruction,thepointofthet’thiterationistoputthe(t+1)’st thingintherightplace.• Termination:Atthestartofthe(len(A)+1)’stiteration(aka,attheendofthealgorithm),thefirstlen(A)itemsaresorted.✓
Tosummarize
InsertionSort isanalgorithmthatcorrectlysortsanarbitraryn-element
arrayintimeaboutn2.
Canwedobetter?
Canwedobetter?
• MergeSort:adivide-and-conquer approach• Recallfromlasttime:
Bigproblem
Smallerproblem
Smallerproblem
Yetsmallerproblem
Yetsmallerproblem
Yetsmallerproblem
Yetsmallerproblem
Recurse!
DivideandConquer:
Recurse!
1
MergeSort6 4 3 8 1 5 2 7
6 4 3 8 1 5 2 7
3 4 6 8 1 2 5 7
2 3 4 5 6 7 8
Recursivemagic!Recursivemagic!
(MERGE pseudocode+analysisonboard)
MERGE!
(Agoodexercise:howwouldIdothisinplace?)
MergeSort Pseudocode
• n=length(A)• if n≤ 1:• return A
• L=MERGESORT(A[1:n/2])• R=MERGESORT(A[n/2+1:n])• returnMERGE(L,R)
MERGESORT(A):
IfAhaslength1,Itisalreadysorted!
Sorttherighthalf
Sortthelefthalf
Mergethetwohalves
Schematicofrecursivecalls
64 3 8 1 5 2 7
1 2 5 73 4 6 8
1 2 3 4 5 6 7 8
Merge!Merge!Merge!Merge!
Merge! Merge!
Merge!
4 3 8 1 5 2 76Originalsequence
Sortedsequence!
ItworksLet’sassumen=2t
• Invariant:“Ineveryrecursivecall,
MERGESORTreturnsasortedarray.”
• n=length(A)• if n≤ 1:
• return A
• L=MERGESORT(A[1:n/2])• R=MERGESORT(A[n/2+1:n])• returnMERGE(L,R)
• Basecase(n=1):a1-elementarrayisalwayssorted.
• Maintenance:SupposethatLandRaresorted.ThenMERGE(L,R)issorted.
• Termination:“Inthetoprecursivecall,MERGESORTreturnsasortedarray.”Themaintenancestepneedsmoredetails!!Whyisthisstatementtrue?
Nottechnicallya“loopinvariant,”buta”recursioninvariant,”thatshouldholdatthebeginningofeveryrecursivecall.
It’sfastLet’skeepassumingn=2t
CLAIM:
MERGESORTrequiresatmost6nlog(n)+6noperationstosortn numbers.
Beforeweseewhy…Howdoesthatcomparetothe≈n2 operationsofINSERTIONSORT?
nlog(n)vsn2
• log(n):howmanytimesdoyouneedtodividenby2inordertogetdownto1?
32168421log(32)=5
6432168421log(64)=6
log(128)=7log(256)=8log(512)=9...log(numberofparticlesintheuniverse)<280
nlog(n)vsn2continued
n nlog(n) n^2
8 24 64
16 64 256
32 160 1024
64 384 4096
128 896 16384
256 2048 65536
512 4608 262144
1024 10240 10485760
200000
400000
600000
800000
1000000
0 200 400 600 800 1000 1200
nlog(n) n^2
It’sfast!
CLAIM:
MERGESORTrequiresatmost6nlog(n)+6noperationstosortn numbers.
Asngrows,that’smuchfasterthanthe≈n2 operationsof
INSERTIONSORT!
Analysis
T(n)=T(n/2)+T(n/2)+T(MERGE)Let’ssay
T(MERGEofsizen/2)≤ 6noperations
Wewillseelaterhowtoanalyzerecurrencerelationsliketheseautomagically…buttodaywe’lldoitfromfirstprinciples.
T(n)=timetorunMERGESORTonalistofsizen
T(MERGEtwolistsofsizen/2)isthetimetodo:
• 3variableassignments(counters←1)• ncomparisons• nmoreassignments• 2ncounterincrements
Sothat’s2T(assign)+nT(compare)+nT(assign)+2nT(increment)
or4n+2operationsPluckythe
pedanticpenguin Luckythelackadaisicallemur
=2T(n/2)+6n
Thisiscalledarecurrencerelation:itdescribestherunningtimeofaproblemofsizenintermsoftherunningtimeofsmallerproblems.
…
Recursiontree
Sizen
n/2n/2
n/4
(Size1)
…
n/4n/4n/4
n/2tn/2tn/2tn/2tn/2tn/2t
…
Level
Amountofworkatthislevel
(justMERGEing)
0
#problems
12
t
log(n)
1
2
4
2t
n
Sizeofeach
problem
n
n/2
n/4
n/2t
1
6n
6n
6n
6n
6n
Amountofworkatalevel:(numberofproblems)× 6× (sizeofproblem)
(explanationonboard)
…
Totalruntime…
• 6nstepsperlevel,ateverylevel
• log(n)+1levels
•6nlog(n)+6nstepstotal
Thatwastheclaim!
Afewreasonstobegrumpy
• Sorting
shouldtakezerosteps…• What’swiththisT(MERGE)<6n?• 2+4n<6nisaloosebound.• Differentoperationsdon’ttakethesameamountoftime.
1 2 3 4 5 6 7 8
Today
• Sorting• Returnofdivide-and-conquer with MergeSort• Skills:• Analyzingcorrectnessofiterativeandrecursivealgorithms.• Analyzingrunningtimeofrecursivealgorithms.
• Howdowemeasuretheruntimeofanalgorithm?• Worst-caseanalysis• AsymptoticAnalysis
Worst-caseanalysis
• Inthisclass,wewillfocusonworst-caseanalysis
• Pros:verystrongguarantee• Cons:verystrongguarantee
1 2 3 4 5 6 7 8
Sortingasortedlistshouldbefast!!
Algorithmdesigner
Algorithm:Do the thingDo the stuffReturn the answer
Here is my algorithm!Here is an input!
Big-Onotation
• Whatdowemeanwhenwemeasureruntime?• Weprobablycareaboutwalltime: howlongdoesittaketosolvetheproblem,insecondsorminutesorhours?
• Thisisheavilydependentontheprogramminglanguage,architecture,etc.• Thesethingsareveryimportant,butarenotthepointofthisclass.• Wewantawaytotalkabouttherunningtimeofanalgorithm,independentoftheseconsiderations.
Howlongdoesanoperationtake?Whyarewebeingsosloppyabout
that“6”?
Rememberthisslide?
n nlog(n) n^2
8 24 64
16 64 256
32 160 1024
64 384 4096
128 896 16384
256 2048 65536
512 4608 262144
1024 10240 1048576 0
200000
400000
600000
800000
1000000
0 200 400 600 800 1000 1200
nlog(n) n^2
Changenlog(n)to5nlog(n)….
n 5nlog(n) n^2
8 120 64
16 320 256
32 800 1024
64 1920 4096
128 4480 16384
256 10240 65536
512 23040 262144
1024 51200 1048576 0
200000
400000
600000
800000
1000000
0 200 400 600 800 1000 1200
5nlog(n) n^2
AsymptoticAnalysisHowdoestherunningtimescaleasngetslarge?
• Abstractsawayfromhardware- andlanguage-specificissues.
• Makesalgorithmanalysismuchmoretractable.
• Onlymakessenseifnislarge(comparedtotheconstantfactors).
Pros: Cons:
Onealgorithmis“faster”thananotherifitsruntimegrowsmore“slowly”asngetslarge.
2100000000000000 nis“better”thann2 ?!?!
O(…)meansanupperbound
• Wesay“T(n)isO(f(n))”iff(n)growsatleastasfastasT(n)asngetslarge.
• Formally,𝑇 𝑛 = 𝑂 𝑓 𝑛
⟺∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,
0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑓(𝑛)(Explanationofwhatallthesesymbolsmeanonboard)
Example1
• T(n)=n,f(n)=n2.• T(n)=O(f(n))
T(n)
n
f(n)
c=1
n0 =1
𝑇 𝑛 = 𝑂 𝑓 𝑛 ⟺
∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑓(𝑛)
Whydoweneedcinthedefinition?
(formalproofonboard)
Example2
• g(n)=2,f(n)=1.• g(n)=O(f(n))(andalsof(n)=O(g(n)))
𝑇 𝑛 = 𝑂 𝑓 𝑛 ⟺
∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑓(𝑛)
f(n)
n0=1n
g(n)
2.1⋅ f(n)
1
2
2.1
(Needcbutnotreallyn0)
Example3
• f(n)=1,g(n)asbelow.• g(n)=O(f(n))(andalsof(n)=O(g(n)))
𝑇 𝑛 = 𝑂 𝑓 𝑛 ⟺
∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,0 ≤ 𝑇 𝑛 ≤ 𝑐 ⋅ 𝑓(𝑛)
f(n)
n
g(n)
2.1⋅ f(n)
1
2
2.1
n0
(Needbothcandn0)
Take-awayfromexamples
• ToproveT(n)=O(f(n)),youhavetocomeupwithcandn0sothatthedefinitionissatisfied.
• ToproveT(n)isNOT O(f(n)),onewayisbycontradiction:• Supposethatsomeonegivesyouacandann0 sothatthedefinitionissatisfied.• Showthatthissomeonemustbylyingtoyoubyderivingacontradiction.
Ω(…)meansalowerbound
• Wesay“T(n)isΩ(f(n))”iff(n)growsatmostasfastasT(n)asngetslarge.
• Formally,𝑇 𝑛 = 𝑂 𝑓 𝑛
⟺∃𝑐, 𝑛. > 0𝑠. 𝑡. ∀𝑛 ≥ 𝑛.,
0 ≤ 𝑐 ⋅ 𝑓 𝑛 ≤ 𝑇 𝑛
Switchedthese!!
Yetmoreexamples
• n3 +3n=O(n3 – n2)• n3 +3n=Ω(n3 – n2)• n3 +3n=Θ(n3 – n2)
• 3n isNOTO(2n)• nlog(n)=Ω(n)• log(n)=Θ(2loglog(n) )
Somebrainteasers
• Aretherefunctionsf,gsothatNEITHER f=O(g)norf=Ω(g)?• Aretherenon-decreasing functionsf,gsothattheaboveistrue?• Definethen’th fibonacci numberbyF(0)=1,F(1)=1,F(n)=F(n-1)+F(n-2)forn>2.• 1,1,2,3,5,8,13,21,34,55,…
Trueorfalse:• F(n)=O(2n)• F(n)=Ω(2n)
We’llbeusinglotsofasymptoticnotationfromhereonout
Butweshouldalwaysbecarefulnottoabuseit.
Inthecourse,(almost)everyalgorithmweseewillbeactuallypractical,withoutneedingtotake𝑛 ≥ 𝑛. = 2:........