AdvancedAlgorithmics(6EAP)

MTAT.03.238Orderofgrowth…maths

JaakVilo2016fall

1JaakVilo

Programexecutiononinputofsizen

• Howmanysteps/cyclesaprocessorwouldneedtodo

• Howtorelatealgorithmexecutiontonrofstepsoninputofsizen?f(n)

• e.g.f(n)=n+n*(n-1)/2+17n+n*log(n)

Whathappensininfinity?

• Fastercomputer,largerinput?

• Badalgorithmonfastcomputerwillbeoutcompetedbygoodalgorithmonslow…

Big-Ohnotationclasses

Class Informal Intuition Analogy

f(n)∈ ο (g(n)) fisdominatedbyg Strictlybelow <f(n)∈O(g(n)) Boundedfromabove Upperbound ≤f(n)∈Θ(g(n)) Boundedfrom

aboveand below“equalto” =

f(n)∈Ω(g(n)) Boundedfrombelow Lowerbound ≥f(n)∈ω(g(n)) fdominatesg Strictlyabove >

MathematicalBackgroundJustification

• Asengineers,youwillnotbepaidtosay:MethodAisbetter thanMethodB

orAlgorithmAisfaster thanAlgorithmB

• Suchdescriptionsaresaidtobequalitative innature;fromtheOED:

qualitative,a. a Relatingto,connectedorconcernedwith,qualityorqualities.Nowusuallyinimpliedorexpressedoppositiontoquantitative.

MathematicalBackgroundJustification

• Businessdecisionscannotbebasedonqualitativestatements:– AlgorithmAcouldbebetter thanAlgorithmB,butAlgorithmAwouldrequirethreepersonweekstoimplement,test,andintegratewhileAlgorithmBhasalreadybeenimplementedandhasbeenusedforthepastyear

– therearecircumstanceswhereitmaybeneficialtouseAlgorithmA,butnotbasedonthewordbetter

MathematicalBackgroundJustification

• Thus,wewilllookataquantitativemeansofdescribingdatastructuresandalgorithms

• FromtheOED:

quantitative,a. Relatingtoorconcernedwithquantityoritsmeasurement;thatassessesorexpressesquantity.Inlaterusefreq.contrastedwithqualitative.

Programtimeandspacecomplexity

• Time:countnrofelementarycalculations/operationsduringprogramexecution

• Space:countamountofmemory(RAM,disk,tape,flashmemory,…)– usuallywedonotdifferentiatebetweencache,RAM,…

– Inpractice,forexample,randomaccessontapeimpossible

• Program1.17floatSum(float*a,constintn)

floats=0;for(inti=0;i<n;i++)

s+=a[i];returns;

– Theinstancecharacteristicisn.– Sincea isactuallytheaddressofthefirstelementofa[],andn ispassedbyvalue,thespaceneededbySum()isconstant(Ssum(n)=1).

• Program1.18floatRSum(float*a,constintn)

if(n<=0)return0;elsereturn(RSum(a,n-1)+a[n-1]);

– Eachcallrequiresatleast4words• Thevaluesofn,a,returnvalueandreturnaddress.

– Thedepthoftherecursionisn+1.• Thestackspaceneededis4(n+1).

n=997

n=1000

n=999

n=998

Inputsize=n

• Inputsizeusuallydenotedbyn• Timecomplexityfunction=f(n)

– array[1..n]– e.g.4*n+3

• Graph:nvertices,medgesf(n,m)

• So,wemaybeabletocountanrofoperationsneeded

• Whatdowedowiththisknowledge?

n2

100 n log n

n2

100 n log n

n = 1000

0.00001*n2

100 n log n

n = 10,000,000

0.00001*n2

100 n log n

n = 2,000,000,000

Logscale y

logscale xlogscale y

• plot[1:10]0.01*x*x,5*log(x),x*log(x)/3

Algorithmanalysisgoal

• Whathappensinthe“longrun”,increasingn

• Compare f(n)toreferenceg(n)(orfandg)

• Atsomen0 >0,ifn>n0,alwaysf(n)< g(n)

Θ ,O,andΩ

Figure 2.1 Graphic examples of the Θ , O, and Ω notations. In each part, the value of n0 shown is the minimum possible value; any greater value would also work.

Dominanttermsonly…

• Essentially,weareinterestedinthelargest(dominant)termonly…

• Whenthisgrowslargeenough,itwill“overshadow” allsmallerterms

Theorem1.2

).()( Thes, .for ,)(

have we,1 ,let Therefore,

.1for ,

)(

:Proof).()( then ,...)( If

0

00

0

0

00

01

mm

m

ii

m

ii

m

m

i

mii

m

m

i

ii

m

i

ii

mmm

nOnfnncnnf

nac

nan

nan

nananf

nOnfanananf

=³£

==

³£

£

£=

=+++=

å

å

å

åå

=

=

=

-

==

AsymptoticAnalysis

• Givenanytwofunctionsf(n) andg(n),wewillrestrictourselvesto:– polynomialswithpositiveleadingcoefficient– exponentialandlogarithmicfunctions

• Thesefunctions as• Wewillconsiderthelimitoftheratio:

)g()f(limnn

n ¥®

¥®n¥®

AsymptoticAnalysis

• Ifthetwofunctionf(n) andg(n) describetheruntimesoftwoalgorithms,and

thatis,thelimitisaconstant,thenwecanalwaysrunthesloweralgorithmonafastercomputertogetsimilarresults

¥<<¥® )g(

)f(lim0nn

n

AsymptoticAnalysis

• Toformallydescribeequivalentruntimes,wewillsaythatf(n) = Q(g(n)) if

• Note:thisisnotequality– itwouldhavebeenbetterifitsaidf(n) ∈ Q(g(n)) however,someonepicked=

¥<<¥® )g(

)f(lim0nn

n

AsymptoticAnalysis

• Wearealsointerestedifonealgorithmrunseitherasymptoticallyslowerorfasterthananother

• Ifthisistrue,wewillsaythatf(n) = O(g(n))

¥<¥® )g(

)f(limnn

n

AsymptoticAnalysis

• Ifthelimitiszero,i.e.,

thenwewillsaythatf(n) = o(g(n)) t• Thisisthecaseiff(n) andg(n) arepolynomialswheref hasa

lowerdegree

0)g()f(lim =

¥® nn

n

AsymptoticAnalysis

• Tosummarize:

0)g()f(lim >

¥® nn

n

))(g()f( nn O=

))(g()f( nn Θ=

))(g()f( nn Ω=

¥<¥® )g(

)f(limnn

n

¥<<¥® )g(

)f(lim0nn

n

AsymptoticAnalysis

• Wehaveonefinalcase:

¥=¥® )g(

)f(limnn

n

))(g()f( nn o=

))(g()f( nn Θ=

))(g()f( nn ω=

0)g()f(lim =

¥® nn

n

¥<<¥® )g(

)f(lim0nn

n

AsymptoticAnalysis

• Graphically,wecansummarizetheseasfollows:

Wesay

if

AsymptoticAnalysis

• Allofn2 100000 n2 – 4 n + 19 n2 + 1000000323 n2 – 4 n ln(n) + 43 n + 10 42n2 + 32

n2 + 61 n ln2(n) + 7n + 14 ln3(n) + ln(n)arebig-Q ofeachother

• E.g.,42n2 + 32 = Q( 323 n2 – 4 n ln(n) + 43 n + 10 )

AsymptoticAnalysis• Wewillfocusonthese

Q(1) constantQ(ln(n)) logarithmicQ(n) linearQ(n ln(n)) “n–log–n”Q(n2) quadraticQ(n3) cubic2n, en, 4n, ... exponential

O(1)<O(logn)<O(n)<O(nlogn)<O(n2)<O(n3)<O(2n)<O(n!)

Growthoffunctions

• SeeChapter“GrowthofFunctions” (CLRS)

•

Logarithms

• Algebracheatsheet:http://tutorial.math.lamar.edu/pdf/Algebra_Cheat_Sheet.pdf

Changeofbasea→b

xb

x aa

b loglog1log =

)(loglog xx ab Q=

Big-Ohnotationclasses

Class Informal Intuition Analogy

f(n)∈ ο (g(n)) fisdominatedbyg Strictlybelow <f(n)∈O(g(n)) Boundedfromabove Upperbound ≤f(n)∈Θ(g(n)) Boundedfrom

aboveand below“equalto” =

f(n)∈Ω(g(n)) Boundedfrombelow Lowerbound ≥f(n)∈ω(g(n)) fdominatesg Strictlyabove >

FamilyofBachmann–Landaunotations

!

Notation Name Intuition As , eventually... Definition

Big Omicron; Big O; Big Oh

f is bounded above by g(up to constant factor) asymptotically

for some k

or

(Note that, since the beginning of the 20th century, papers in number theory have been increasingly and widely using this notation in the weaker sense that f = o(g) is false)

Big Omega

f is bounded below by g(up to constant factor) asymptotically

for some positivek

Big Theta f is bounded both above and below bygasymptotically for some positive k1, k2

Small Omicron; Small O; Small Oh

f is dominated by gasymptotically for every ε

Small Omega f dominates gasymptotically for every k

on the order of; "twiddles"

f is equal to gasymptotically

googol

Howmuchtimedoessortingtake?

• Comparison-basedsort:A[i]<=A[j]

– Upperbound– currentbest-knownalgorithm– Lowerbound – theoretical“atleast”estimate– Iftheyareequal,wehavetheoreticallyoptimalsolution

Simplesort

for i=2..nfor j=i ;j>1;j--

if A[j]<A[j-1]then swap(A[j],A[j-1])else nexti

Thedivide-and-conquerdesignparadigm

1. Divide theproblem(instance)intosubproblems.

2. Conquer thesubproblems bysolvingthemrecursively.

3. Combine subproblem solutions.

Mergesort

Merge-Sort(A,p,r)if p<r then q=(p+r)/2

Merge-Sort( A,p,q)Merge-Sort( A,q+1,r)Merge( A,p,q,r)

It was invented by John von Neumann in 1945.

Example

• Applyingthemergesortalgorithm:

Wikipedia/viz.Va

lue

Pos in array

Quicksort

Wikipedia/“video”

Conclusions

• Algorithmcomplexitydealswiththebehaviorinthelong-term– worstcase -- typical– averagecase -- quitehard– bestcase -- bogus,“cheating”

• Inpractice,long-termsometimesnotnecessary– E.g.forsorting20elements,youdon’tneedfancyalgorithms…

n2.3727

n2.376