algorithms and data structures - star -...
TRANSCRIPT
ECE250: Algorithms and Data Structures
Analyzing and Designing Algorithms
Materials from CLRS: Chapter 2
Ladan Tahvildari, PEng, SMIEEE Professor
Software Technologies Applied Research (STAR) Group
Dept. of Elect. & Comp. Eng.
University of Waterloo
Acknowledgements
v The following resources have been used to prepare materials for this course: Ø MIT OpenCourseWare Ø Introduction to Algorithms (CLRS Book) Ø Data Structures and Algorithm Analysis in C++ (M. Wiess) Ø Data Structures and Algorithms in C++ (M. Goodrich)
v Thanks to many people for pointing out mistakes, providing suggestions, or helping to improve the quality of this course over the last ten years: Ø http://www.stargroup.uwaterloo.ca/~ece250/acknowledgment/
Lecture 2 ECE250 2
Lecture 2 ECE250 3
Analysis of Algorithms
v The theoretical study of computer-program performance and resource usage.
v What is more important than performance? Ø Modularity Ø Correctness Ø Maintainability Ø Functionality Ø Robustness
Ø User-Friendliness Ø Simplicity Ø Extensibility Ø Reliability Ø Programmer Time
Lecture 2 ECE250 4
Why Study Algorithms and Performance?
v Performance often draws the line between what is feasible and what is impossible.
v Algorithms help us to understand scalability.
v Algorithmic mathematics provides a language for talking about program behavior.
Lecture 2 ECE250 5
An Example
v A sorting problem:
v Consider two basic algorithms: Ø Insertion Sort Ø Merge Sort
Input: A sequence of numbers 〈a1, a2, …, an〉
Output: A re-ordering 〈a'1, a'2, …, a'n〉 such that a'1 ≤ a'2 ≤ … ≤ a'n
Lecture 2 ECE250 6
Example v Suppose we want to sort a sequence of numbers (input size n) v Insertion Sort: [later…] v Merge Sort: [later…]
v Bad Programmer ( ), compiler, inst/sec, Computer A v Good Programmer ( ), machine language, inst/sec, Computer B
v Computer A: seconds Ø Merge Sort
v Computer B: seconds Ø Insertion Sort
v For numbers 2.3 days vs. 20 minutes v Conclusions???
21nc
710
610
)lg(2 nnc
21 =c502 =c
710910
10010/)10lg()10)(50( 766 ≈
200010/)10)(2( 926 ≈
Lecture 2 ECE250 7
Asymptotic Performance
v In this course, we are concerned with asymptotic performance of the algorithms: Ø What about the behavior of the algorithm as the ‘size’ of
the problem grows? § The running time § The memory/storage requirements
v The first task will be formally define this asymptotic notation.
Lecture 2 ECE250 8
Asymptotic Performance
v Types of Analysis:
Ø Worst Case § Provides an upper bound in the running time § Like an “Absolute Guarantee”
Ø Average Case § Estimate of the expected running time § Can be useful, but what is “average case”?
ü Random (equally likely) inputs ü “Real-Life” inputs, application driven
Lecture 2 ECE250 9
The Problem of Sorting Input: sequence 〈a1, a2, …, an〉 of numbers.
Example: Input: 8 2 4 9 3 6 Output: 2 3 4 6 8 9
Output: permutation 〈a'1, a'2, …, a'n〉 such that a'1 ≤ a'2 ≤ … ≤ a'n .
• Consider two sorting algorithms: Ø Insertion Sort Ø Merge Sort
n2n lg(n)
Lecture 2 ECE250 10
INSERTION-SORT (A, n) for j ← 2 to n do key ← A[ j] i ← j – 1 while i > 0 and A[i] > key do A[i+1] ← A[i] i ← i – 1 A[i+1] = key
“pseudocode”
A:
Insertion Sort
i j
key sorted
1 n
Lecture 2 ECE250 11
Insertion Sort - Example 8 2 4 9 3 6
Lecture 2 ECE250 12
Insertion Sort - Example 8 2 4 9 3 6
Lecture 2 ECE250 13
Insertion Sort - Example 8 2 4 9 3 6
2 8 4 9 3 6
Lecture 2 ECE250 14
Insertion Sort - Example 8 2 4 9 3 6
2 8 4 9 3 6
Lecture 2 ECE250 15
Insertion Sort - Example 8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
Lecture 2 ECE250 16
Insertion Sort - Example 8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
Lecture 2 ECE250 17
Insertion Sort - Example 8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
Lecture 2 ECE250 18
Insertion Sort - Example 8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
Lecture 2 ECE250 19
Insertion Sort - Example 8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
Lecture 2 ECE250 20
Insertion Sort - Example 8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
Lecture 2 ECE250 21
Insertion Sort - Example 8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
2 3 4 6 8 9
Lecture 2 ECE250 22
Running Time
v The running time depends on the input. Ø An already sorted sequence is easier to sort.
v Parameterize the running time by the size of the input. Ø Short sequences are easier to sort than long ones.
v We seek upper bounds on the running time, to have a guarantee of performance.
Lecture 2 ECE250 23
Kinds of Analyses
v Worst-case: (usually) Ø T(n): maximum time of algorithm on any input of
size n. v Average-case: (sometimes)
Ø T(n): expected time of algorithm over all inputs of size n. § Need assumption of statistical distribution of inputs.
v Best-case: (NEVER) Ø Cheat with a slow algorithm that works fast on
some input.
Lecture 2 ECE250 24
Machine-Independent Time
What is insertion sort’s worst-case time?
v Depends on the speed of our computer: Ø Relative speed (on the same machine) Ø Absolute speed (on different machines)
BIG IDEA:
v Ignore machine-dependent constants Ø otherwise impossible to verify and to compare algorithms
v Look at growth of T(n) as n → ∞ .
“Asymptotic Analysis”
Lecture 2 ECE250 25
Asymptotic Analysis
Engineering: v Drop low-order terms; ignore leading constants. v Example: 3n3 + 90n2 – 5n + 6046 = Θ(n3)
Math: Θ(g(n)) = { f (n) : there exist positive constants c1, c2, and
n0 such that 0 ≤ c1 g(n) ≤ f (n) ≤ c2 g(n) for all n ≥ n0 }
Lecture 2 ECE250 26
Asymptotic Performance
n
T(n)
n0
. • Asymptotic analysis is a
useful tool to help to structure our thinking toward better algorithm.
• We should not ignore asymptotically slower algorithms, however.
When n gets large enough, a Θ(n2) algorithm always beats a Θ(n3) algorithm.
Lecture 2 ECE250 27
INSERTION-SORT (A, n) for j ← 2 to n do key ← A[ j] i ← j – 1 while i > 0 and A[i] > key do A[i+1] ← A[i] i ← i – 1 A[i+1] = key
“pseudocode”
A:
Insertion Sort Analysis
i j
key sorted
1 n
Lecture 2 ECE250 28
Insertion Sort Analysis
Worst case: Input reverse sorted. ( )∑
=
Θ=Θ=n
jnjnT
2
2)()( [arithmetic series]
Lecture 2 ECE250 29
Math Review
v Floor and Ceiling
v Arithmetic Series
v
v Geometric Series
v Appendix A, B, C of CLRS as needed during the course
⎣ ⎦ ⎡ ⎤ ⎣ ⎦ 22/3,22/3,12/3 −=−==
)1(21...21
1
+=+++=∑−
nnnkn
k
101024log;loglog 2 =≡
221
0
=∑∞
=ii
Lecture 2 ECE250 30
Insertion Sort Analysis
Worst case: Input reverse sorted. ( )∑
=
Θ=Θ=n
jnjnT
2
2)()(
Average case: All permutations equally likely. ( )∑
=
Θ=Θ=n
jnjnT
2
2)2/()(
Is insertion sort a fast sorting algorithm? v Moderately so, for small n. v Not at all, for large n.
[arithmetic series]
Lecture 2 ECE250 31
The Problem of Sorting Input: sequence 〈a1, a2, …, an〉 of numbers.
Example: Input: 8 2 4 9 3 6 Output: 2 3 4 6 8 9
Output: permutation 〈a'1, a'2, …, a'n〉 such that a'1 ≤ a'2 ≤ … ≤ a'n .
• Consider two sorting algorithms: Ø Insertion Sort Ø Merge Sort
n2n lg(n)
Lecture 2 ECE250 32
Merge Sort
MERGE-SORT A[1 . . n] 1. If n = 1, done. 2. Recursively sort A[ 1 . . ⎡n/2⎤ ]
and A[ ⎡n/2⎤+1 . . n ] . 3. “Merge” the 2 sorted lists.
Key subroutine: MERGE
Lecture 2 ECE250 33
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
Lecture 2 ECE250 34
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
Lecture 2 ECE250 35
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
Lecture 2 ECE250 36
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
Lecture 2 ECE250 37
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
Lecture 2 ECE250 38
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
Lecture 2 ECE250 39
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
Lecture 2 ECE250 40
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
Lecture 2 ECE250 41
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
20
13
12
11
Lecture 2 ECE250 42
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
20
13
12
11
11
Lecture 2 ECE250 43
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
20
13
12
11
11
20
13
12
Lecture 2 ECE250 44
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
20
13
12
11
11
20
13
12
12
Lecture 2 ECE250 45
Merging Two Sorted Arrays
20
13
7
2
12
11
9
1
1
20
13
7
2
12
11
9
2
20
13
7
12
11
9
7
20
13
12
11
9
9
20
13
12
11
11
20
13
12
12
Time = Θ(n) to merge a total of n elements (linear time).
Lecture 2 ECE250 46
Analyzing Merge Sort
MERGE-SORT A[1 . . n] 1. If n = 1, done. 2. Recursively sort A[ 1 . . ⎡n/2⎤ ]
and A[ ⎡n/2⎤+1 . . n ] . 3. “Merge” the 2 sorted lists
T(n) Θ(1) 2T(n/2)
Θ(n)
Sloppiness: Should be T( ⎡n/2⎤ ) + T( ⎣n/2⎦ ) , but it turns out not to matter asymptotically.
Lecture 2 ECE250 47
Recurrence for Merge Sort
T(n) = Θ(1) if n = 1; 2T(n/2) + Θ(n) if n > 1.
v We shall usually omit stating the base case when T(n) = Θ(1) for sufficiently small n, but only when it has no effect on the asymptotic solution to the recurrence.
v Next lecture provides several ways to find
a good upper bound on T(n).
Lecture 2 ECE250 48
Recursion Tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
Lecture 2 ECE250 49
Recursion Tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
T(n)
Lecture 2 ECE250 50
Recursion Tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
T(n/2) T(n/2)
cn
Lecture 2 ECE250 51
Recursion Tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
T(n/4) T(n/4) T(n/4) T(n/4)
cn/2 cn/2
Lecture 2 ECE250 52
Recursion Tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
Lecture 2 ECE250 53
Recursion Tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
h = lg n
Lecture 2 ECE250 54
Recursion Tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
h = lg n
cn
Lecture 2 ECE250 55
Recursion Tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
h = lg n
cn
cn
Lecture 2 ECE250 56
Recursion Tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
h = lg n
cn
cn
cn
…
Lecture 2 ECE250 57
Recursion Tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
h = lg n
cn
cn
cn
#leaves = n Θ(n)
…
Lecture 2 ECE250 58
Recursion Tree Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/4 cn/4 cn/4 cn/4
cn/2 cn/2
Θ(1)
h = lg n
cn
cn
cn
#leaves = n Θ(n) Total = Θ(n lg n)
…
Lecture 2 ECE250 59
Conclusions
v Θ(n lg n) grows more slowly than Θ(n2).
v Therefore, merge sort asymptotically beats insertion sort in the worst case.
v In practice, merge sort beats insertion sort for n > 30 or so.