spring 2015 lecture 2: analysis of algorithms
DESCRIPTION
Analysis of algorithms the formal way …TRANSCRIPT
2IL50 Data Structures
Spring 2015
Lecture 2: Analysis of Algorithms
Analysis of algorithms
the formal way …
Analysis of algorithms Can we say something about the running time of an algorithm
without implementing and testing it?
InsertionSort(A)1. initialize: sort A[1]2. for j = 2 to A.length3. do key = A[j] 4. i = j -15. while i > 0 and A[i] > key6. do A[i+1] = A[i] 7. i = i -18. A[i +1] = key
Analysis of algorithms Analyze the running time as a function of n (# of input elements)
best case average case worst case
elementary operationsadd, subtract, multiply, divide, load, store, copy, conditional and unconditional branch, return …
An algorithm has worst case running time T(n) if for any input of size n the maximal number of elementary operations executed is T(n).
Analysis of algorithms: example
InsertionSort: 15 n2 + 7n – 2
MergeSort: 300 n lg n + 50 n
n=10 n=100 n=1000
1568 150698 1.5 x 107
10466 204316 3.0 x 106
InsertionSort6 x faster
InsertionSort1.35 x faster
MergeSort5 x faster
n = 1,000,000 InsertionSort 1.5 x 1013
MergeSort 6 x 109 2500 x faster !
The rate of growth of the running time as a function of the input is essential!
Θ-notationIntuition: concentrate on the leading term, ignore constants
19 n3 + 17 n2 - 3n becomes Θ(n3)
2 n lg n + 5 n1.1 - 5 becomes
n - ¾ n √n becomes
Θ(n1.1)
---
Θ-notationLet g(n) : N ↦ N be a function. Then we have
Θ(g(n)) = { f(n) : there exist positive constants c1, c2, and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥
n0 }
“Θ(g(n)) is the set of functions that grow as fast as g(n)”
Θ-notationLet g(n) : N ↦ N be a function. Then we have
Θ(g(n)) = { f(n) : there exist positive constants c1, c2, and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0 }
nn0
c1g(n)
c2g(n)
f(n)
0
Notation: f(n) = Θ(g(n))
Θ-notationLet g(n) : N ↦ N be a function. Then we have
Θ(g(n)) = { f(n) : there exist positive constants c1, c2, and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0 }
Claim: 19n3 + 17n2 - 3n = Θ(n3)
Proof: Choose c1 = 19, c2 = 36 and n0 = 1. Then we have for all n ≥ n0:
0 ≤ c1n3 = 19n3 (trivial) ≤ 19n3 + 17n2 - 3n (since 17n2 > 3n for n ≥ 1)
≤ 19n3 + 17n3 (since 17n2 ≤ 17n3 for n ≥1) = c2n3
■
Θ-notationLet g(n) : N ↦ N be a function. Then we have
Θ(g(n)) = { f(n) : there exist positive constants c1, c2, and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0 }
Claim: 19n3 + 17n2 - 3n ≠ Θ(n2)
Proof: Assume that there are positive constants c1, c2, and n0 such that for all n ≥ n0
0 ≤ c1n2 ≤ 19n3 + 17n2 - 3n ≤ c2n2 Since 19n3 + 17n2 - 3n ≤ c2n2 implies 19n3 ≤ c2n2 + 3n – 17n2 ≤ c2n2 (3n – 17n2 ≤
0)we would have for all n ≥ n0
19n ≤ c2.
O-notationLet g(n) : N ↦ N be a function. Then we have
O(g(n)) = { f(n) : there exist positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0 }
“O(g(n)) is the set of functions that grow at most as fast as g(n)”
O-notationLet g(n) : N ↦ N be a function. Then we have
O(g(n)) = { f(n) : there exist positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0 }
nn0
cg(n)
f(n)
0
Notation: f(n) = O(g(n))
Ω-notationLet g(n) : N ↦ N be a function. Then we have
Ω(g(n)) = { f(n) : there exist positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n) for all n ≥ n0 }
“Ω(g(n)) is the set of functions that grow at least as fast as g(n)”
Ω-notationLet g(n) : N ↦ N be a function. Then we have
Ω(g(n)) = { f(n) : there exist positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n) for all n ≥ n0 }
nn0
cg(n)
f(n)
0
Notation: f(n) = Ω(g(n))
Asymptotic notation
Θ(…) is an asymptotically tight bound
O(…) is an asymptotic upper bound
Ω(…) is an asymptotic lower bound
other asymptotic notation
o(…) → “grows strictly slower than”ω(…) → “grows strictly faster than”
“asymptotically equal”
“asymptotically smaller or equal”
“asymptotically greater or equal”
More notation …f(n) = n3 + Θ(n2) means
f(n) = means
O(1) or Θ(1) means
2n2 + O(n) = Θ(n2) means
there is a function g(n) such thatf(n) = n3 + g(n) and g(n) = Θ(n2)
there is one function g(i) such thatf(n) = and g(i) = O(i)
a constant
for each function g(n) with g(n)=O(n) we have 2n2 + g(n) = Θ(n2)
n
1iO(i)
n
1ig(i)
Quiz1. O(1) + O(1) = O(1)
2. O(1) + … + O(1) = O(1)
3. =
4. O(n2) ⊆ O(n3)
5. O(n3) ⊆ O(n2)
6. Θ(n2) ⊆ O(n3)
7. An algorithm with worst case running time O(n log n) is always slower than an algorithm with worst case running time O(n) if n is sufficiently large.
true
false
true
true
false
true
false
n
1iO(i)
n
1ii)O(
Quiz8. n log2 n = Θ(n log n)
9. n log2 n = Ω(n log n)
10. n log2 n = O(n4/3)
11. O(2n) ⊆ O(3n)
12. O(2n) ⊆ Θ(3n)
false
true
true
true
false
Analysis of algorithms
Analysis of InsertionSortInsertionSort(A)1. initialize: sort A[1]2. for j = 2 to A.length3. do key = A[j] 4. i = j -15. while i > 0 and A[i] > key6. do A[i+1] = A[i] 7. i = i -18. A[i +1] = key
Get as tight a bound as possible on the worst case running time.
➨ lower and upper bound for worst case running timeUpper bound: Analyze worst case number of elementary
operationsLower bound: Give “bad” input example
Analysis of InsertionSortInsertionSort(A)1. initialize: sort A[1]2. for j = 2 to A.length3. do key = A[j] 4. i = j -15. while i > 0 and A[i] > key6. do A[i+1] = A[i] 7. i = i -18. A[i +1] = key
Upper bound: Let T(n) be the worst case running time of InsertionSort on an array of length n. We have
T(n) =
Lower bound:
n
2j
O(1)
O(1)
O(1)
worst case:(j-1) ∙ O(1)
O(1) + { O(1) + (j-1)∙O(1) + O(1) }
= O(j) = O(n2)
n
2j
Array sorted in de-creasing order ➨ Ω(n2)
The worst case running time of InsertionSort is Θ(n2).
MergeSort(A) // divide-and-conquer algorithm that sorts array A[1..n] 1. if A.length = 12. then skip3. else 4. n = A.length ; n1 = floor(n/2); n2 = ceil(n/2); 5. copy A[1.. n1] to auxiliary array A1[1.. n1]6. copy A[n1+1..n] to auxiliary array A2[1.. n2]7. MergeSort(A1); MergeSort(A2)8. Merge(A, A1, A2)
Analysis of MergeSort
O(1)
O(1)
O(n)O(n)
O(n)??
T( n/2 ) + T( n/2 )
MergeSort is a recursive algorithm ➨ running time analysis leads to recursion
Analysis of MergeSort Let T(n) be the worst case running time of MergeSort on an array of
length n. We have
O(1) if n = 1 T(n) = T( n/2 ) + T( n/2 ) + Θ(n) if n > 1
frequently omitted since it (nearly) always holds
often written as 2T(n/2)
Solving recurrences
Solving recurrences
Easiest: Master theoremcaveat: not always applicable
Alternatively: Guess the solution and use the substitution method to prove that your guess it is correct.
How to guess:1. expand the recursion2. draw a recursion tree
Let a and b be constants, let f(n) be a function, and let T(n)be defined on the nonnegative integers by the recurrence
T(n) = aT(n/b) + f(n)
Then we have:
1. If f(n) = O(nlog a – ε) for some constant ε > 0, then T(n) = Θ(nlog a).
2. If f(n) = Θ(nlog a), then T(n) = Θ(nlog a log n)
3. If f(n) = Ω(nlog a + ε) for some constant ε > 0, and if af(n/b) ≤ cf(n) for some constant c < 1 and all sufficiently large n, then T(n) = Θ(f(n))
The master theorem
b
b
b
b
can be rounded up or down
note: logba - ε
b
The master theorem: ExampleT(n) = 4T(n/2) + Θ(n3)
Master theorem with a = 4, b = 2, and f(n) = n3
logba = log24 = 2
➨ n3 = f(n) = Ω(nlog a + ε) = Ω(n2 + ε) with, for example, ε = 1
Case 3 of the master theorem gives T(n) = Θ(n3), if the regularity condition holds.
choose c = ½ and n0 = 1
➨ af(n/b) = 4(n/2)3 = n3/2 ≤ cf(n) for n ≥ n0
➨ T(n) = Θ(n3)
b
The substitution method The Master theorem does not always apply
In those cases, use the substitution method:1. Guess the form of the solution.2. Use induction to find the constants and show that the solution
works
Use expansion or a recursion-tree to guess a good solution.
Recursion-treesT(n) = 2T(n/2) + n
n
n/2 n/2
n/4 n/4 n/4 n/4
n/2i n/2i … n/2i
Θ(1) Θ(1) … Θ(1)
Recursion-treesT(n) = 2T(n/2) + n
n
2 ∙ (n/2) = n
4 ∙ (n/4) = n
2i ∙ (n/2i) = n
n ∙ Θ(1) = Θ(n)
log n
+
Θ(n log n)
n
n/2 n/2
n/4 n/4 n/4 n/4
n/2i n/2i … n/2i
Θ(1) Θ(1) … Θ(1)
Recursion-treesT(n) = 2T(n/2) + n2
n2
(n/2)2 (n/2)2
(n/4)2 (n/4)2 (n/4)2 (n/4)2
(n/2i)2 (n/2i)2 … (n/2i)2
Θ(1) Θ(1) … Θ(1)
Recursion-treesT(n) = 2T(n/2) + n2
n2
(n/2)2 (n/2)2
(n/4)2 (n/4)2 (n/4)2 (n/4)2
(n/2i)2 (n/2i)2 … (n/2i)2
Θ(1) Θ(1) … Θ(1)
n2
2 ∙ (n/2)2 = n2/2
4 ∙ (n/4)2 = n2/4
2i ∙ (n/2i) 2 = n2/2i
n ∙ Θ(1) = Θ(n) +
Θ(n2)
Recursion-treesT(n) = 4T(n/2) + n
n
n/2 n/2 n/2 n/2
… n/4 n/4 n/4 n/4 …
Θ(1) Θ(1) … Θ(1)
Recursion-treesT(n) = 4T(n/2) + n
n
n/2 n/2 n/2 n/2
… n/4 n/4 n/4 n/4 …
Θ(1) Θ(1) … Θ(1)
n
4 ∙ (n/2) = 2n
16 ∙ (n/4) = 4n
n2 ∙ Θ(1) = Θ(n2) +
Θ(n2)
The substitution method
Claim: T(n) = O(n log n)
Proof: by induction on n
to show: there are constants c and n0 such that T(n) ≤ c n log n for all n ≥ n0
T(1) = 2 ➨ choose c = 2 (for now) and n0 = 2
Why n0 = 2? How many base cases?
Base cases: n = 2: T(2) = 2T(1) + 2 ≤ 2∙2 + 2 = 6 = c 2 log 2 for c = 3 n = 3: T(3) = 2T(1) + 3 ≤ 2∙2 + 3 = 7 ≤ c 3 log 3
T(n) =2 if n = 1
2T( n/2 ) + n if n > 1
log 1 = 0 ➨ can not prove bound for n = 13/2 = 1, 4/2 = 2 ➨ 3 must also be
base case
The substitution method
Claim: T(n) = O(n log n)
Proof: by induction on n
to show: there are constants c and n0 such that T(n) ≤ c n log n for all n ≥ n0
choose c = 3 and n0 = 2
Inductive step: n > 3T(n) = 2T( n/2 ) + n
≤ 2 c n/2 log n/2 + n (ind. hyp.) ≤ c n ((log n) - 1) + n
≤ c n log n ■
T(n) =2 if n = 1
2T( n/2 ) + n if n > 1
The substitution method
Claim: T(n) = O(n)
Proof: by induction on n
Base case: n = n0
T(2) = 2T(1) + 2 = 2c + 2 = O(2)
Inductive step: n > n0
T(n) = 2T( n/2 ) + n = 2O( n/2 ) + n (ind. hyp.)
= O(n) ■
T(n) =Θ(1) if n = 1
2T( n/2 ) + n if n > 1
Never use O, Θ, or Ω in a proof by induction!
Analysis of algorithms
one more example …
Example (A) // A is an array of length n1. n = A.length2. if n=13. then return A[1]4. else begin5. Copy A[1… n/2 ] to auxiliary array B[1... n/2 ]6. Copy A[1… n/2 ] to auxiliary array C[1… n/2 ]7. b = Example(B); c = Example(C)8. for i = 1 to n9. do for j = 1 to i10. do A[i] = A[j] 11. return 4312. end
Example
Example (A) // A is an array of length n1. n = A.length2. if n=13. then return A[1]4. else begin5. Copy A[1… n/2 ] to auxiliary array B[1... n/2 ]6. Copy A[1… n/2 ] to auxiliary array C[1… n/2 ]7. b = Example(B); c = Example(C)8. for i = 1 to n9. do for j = 1 to i10. do A[i] = A[j] 11. return 4312. end
ExampleLet T(n) be the worst case running time of Example on an array of length n.
Lines 1,2,3,4,11, and 12 take Θ(1) time.Lines 5 and 6 take Θ(n) time.Line 7 takes Θ(1) + 2 T( n/2 ) time.Lines 8 until 10 take
time.
If n=1 lines 1,2,3 are executed,else lines 1,2, and 4 until 12 are executed.
➨ T(n):
➨ use master theorem …
)Θ(nΘ(i)Θ(1) 2n
1i
n
1i
i
1j
Θ(1) if n=12T(n/2) + Θ(n2) if n>1
Tips Analysis of recursive algorithms:
find the recursion and solve with master theorem if possible
Analysis of loops: summations
Some standard recurrences and sums:
T(n) = 2T(n/2) + Θ(n) ➨
½ n(n+1) = Θ(n2)
Θ(n3)
T(n) = Θ(n log n)
n
1i
i
n
1i
2i
Announcements
Group sign-up closes tonight (Wed 04-02-2015) at midnight.
Not signed up for group ➨ steps will be taken Thursday 05-02-2015
Some students might be moved to equalize sizes.
Email your assignment as .pdf to your tutor, not to me.