cmpt 438 algorithms. why study algorithms? necessary in any computer programming problem ▫improve...

47
CMPT 438 Algorithms

Upload: magnus-holt

Post on 02-Jan-2016

218 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

CMPT 438 Algorithms

Page 2: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Why Study Algorithms?• Necessary in any computer programming problem

▫ Improve algorithm efficiency: run faster, process more data, do something that would otherwise be impossible

▫ Solve problems of significantly large size▫ Technology only improves things by a constant factor

• Compare algorithms• Algorithms as a field of study

▫ Learn about a standard set of algorithms▫ New discoveries arise▫ Numerous application areas

• Learn techniques of algorithm design and analysis

Page 3: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

What are Algorithms?

•An algorithm is a sequence of computational steps that transform the input into the output.

•An algorithm is also a tool for solving a well-specified computational problem. ▫E.g., sorting problem:

▫<31, 26, 41, 59, 58> is an instance of the sorting problem.

Page 4: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

•An algorithm is correct if, for every input instance, it halts with the correct output.

Page 5: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Analyzing Algorithms• Predict the amount of resources required:

▫ memory: how much space is needed?▫ computational time: how fast the algorithm runs?

• FACT: running time grows with the size of the input• Input size (number of elements in the input)

▫ Size of an array, # of elements in a matrix, # of bits in the binary representation of the input, vertices and edges in a graph

• Def: Running time = the number of primitive operations (steps) executed before termination▫ Arithmetic operations (+, -, *), data movement, control,

decision making (if, while), comparison

Page 6: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Algorithm Efficiency vs. Speed

•E.g.: sorting n numbers (n = 106)▫Friend’s computer = 109

instructions/second▫Friend’s algorithm = 2n2 instructions (insertion sort)

▫Your computer = 107 instructions/second▫Your algorithm = 50nlgn instructions (merge sort)

Page 7: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Algorithm Efficiency vs. Speed

•To sort 100 million numbers:•Insertion sort takes more than 23 days•Merge sort takes under 4 hours

Page 8: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Typical Running Time Functions• 1 (constant running time):

▫ Instructions are executed once or a few times• logN (logarithmic)

▫A big problem is solved by cutting the original problem in smaller sizes, by a constant fraction at each step

• N (linear)▫A small amount of processing is done on each input

element• N logN

▫A problem is solved by dividing it into smaller problems, solving them independently and combining the solution

Page 9: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Typical Running Time Functions

•N2 (quadratic)▫Typical for algorithms that process all pairs

of data items (double nested loops)•N3 (cubic)

▫Processing of triples of data (triple nested loops)

•NK (polynomial)•2N (exponential)

▫Few exponential algorithms are appropriate for practical use

Page 10: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Why Faster Algorithms?

Page 11: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Insertion Sort

•Idea: like sorting a hand of playing cards▫Remove one card at a time from the table,

and insert it into the correct position in the left hand compare it with each of the cards already in

the hand, from right to left

Page 12: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Example of insertion sort

5 2 4 6 1 3

Page 13: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,
Page 14: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

INSERTION-SORT

INSERTION-SORT (A, n) ⊳ A[1 . . n] for j ←2 to n do key ← A[ j] i ← j –1 while i > 0 and A[i] > key do A[i+1] ← A[i] i ← i –1 A[i+1] = key

Insertion sort sorts the elements in place.

i

A:

sortedkey

nj1

Page 15: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Analysis of Insertion Sort

Page 16: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Analysis of Insertion Sort

Page 17: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Running time

•Parameterize the running time by the size of the input, since short sequences are easier to sort than long ones.

Page 18: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Kinds of analyses

Worst-case: • T(n) =maximum time of algorithm on any

input of size n.Average-case: • T(n) =expected time of algorithm over all

inputs of size n. • Need assumption of statistical distribution of

inputs.Best-case: • Cheat with a slow algorithm that works fast on some input.

Page 19: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

“Asymptotic Analysis”

Machine-independent timeWhat is insertion sort’s worst-case time?

•It depends on the speed of our computer

BIG IDEA:•Ignore machine-dependent constants.•Look at growth of T(n) as n→∞.

Page 20: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Θ-notation

Math:Θ(g(n)) = { f (n): there exist positive constants c1, c2, and n0

such that 0 ≤c1g(n) ≤f (n) ≤c2g(n)

for all n≥n0}

Engineering:•Drop low-order terms; ignore leading

constants.•Example: 3n3 + 90n2–5n+ 6046 = Θ(n3)

Page 21: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Best Case Analysis

•The array is already sorted▫A[i] ≤ key upon the first time the while loop

test is run (when i = j -1)▫tj = 1

Page 22: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Worst Case Analysis

•The array is in reverse sorted order▫Always A[i] > key in while loop test▫Have to compare key with all elements to

the left of the j-th position ▫compare with j-1 elements ▫tj = j

Page 23: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Average Case?

•All permutations equally likely.

Page 24: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Insertion Sort Summary

•Advantages▫Good running time for “almost sorted”

arrays θ(n)•Disadvantages

▫θ(n2) running time in worst and average case

Is insertion sort a fast sorting algorithm?•Moderately so, for small n.•Not at all, for large n.

Page 25: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Worst-Case and Average-Case

•We usually concentrate on finding only the worst-case running time▫an upper bound on the running time▫For some algorithms, the worst case occurs

often. E.g., searching when information is not

present in the DB▫The average case is often as bad as the

worst case.

Page 26: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Merge Sort

Page 27: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Merge Sort

MERGE-SORT A[1 . . n] 1.If n= 1, done. 2.Recursively sort A[ 1 . . .n/2]and A[ [n/2]+1 . . n ] . 3.“Merge” the 2 sorted lists.

Page 28: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Example

Page 29: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Divide-and-Conquer

•Divide the problem into a number of subproblems▫Similar sub-problems of smaller size

•Conquer the sub-problems▫Solve the sub-problems recursively▫Sub-problem size small enough to solve the

problems in straightforward manner•Combine the solutions to the sub-problems

▫Obtain the solution for the original problem

Page 30: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Merge Sort Approach• To sort an array A[p . . r]:• Divide

▫Divide the n-element sequence to be sorted into two subsequences of n/2 elements each

• Conquer▫Sort the subsequences recursively using merge

sort ▫When the size of the sequences is 1 there is

nothing more to do• Combine

▫Merge the two sorted subsequences

Page 31: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Merge sort

Page 32: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Analyzing merge sort

MERGE-SORT A[1 . . n]1.If n= 1, done.2.Recursively sort A[ 1 . . 「 n/2 」 ] and A[ 「 n/2 」 +1 . . n ] .3.“Merge”the 2sorted lists

Sloppiness: Should be T( 「 n/2 」 ) + T( 「 n/2 」 ) , but it turns out not to matter asymptotically.

T(n)Θ(1)2T(n/2)

?

Page 33: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Merging two sorted arrays

20 1213 11 7 9 2 1

Page 34: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Merging two sorted arrays

20 12

13 11

7 9

2 1

20 1213 11 7 9 2

20 1213 11 7 9

20 1213 11 9

20 1213 11

20 1213

1 2 7 9 11 12

Time = Θ(n) to merge a total of n elements (linear time).

Page 35: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

In place sort?Run time?

Page 36: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Analyzing Divide and Conquer Algorithms•The recurrence is based on the three

steps of the paradigm:▫T(n) – running time on a problem of size n▫Divide the problem into a subproblems,

each of size n/b: takes D(n)▫Conquer (solve) the subproblems: takes

aT(n/b)▫Combine the solutions: takes C(n)

•T(n) = aT(n/b) + D(n) + C(n) otherwise

Page 37: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

MERGE – SORT Running Time

•Divide:▫compute q as the average of p and r: D(n)

= θ(1)•Conquer:

▫recursively solve 2 subproblems, each of size n/2 -> 2T (n/2)

•Combine:▫MERGE on an n-element subarray takes

θ(n) time C(n) = θ(n)•T(n) = 2T(n/2) + θ(n) if n > 1

Page 38: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Recurrence for merge sort

•We shall usually omit stating the base case when T(n) = Θ(1) for sufficiently small n, but only when it has no effect on the asymptotic solution to the recurrence.

Θ(1) if n= 1;2T(n/2)+ Θ(n) if n> 1.

T(n) =

Page 39: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Recursion tree

Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.

Page 40: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Recursion treeSolve T(n) = 2T(n/2) + cn, where c > 0 is

constant.cn

cn/2 cn/2

cn/4 cn/4 cn/4 cn/4

Θ(1)

. .

h= lgn

cn

cn

cn

. .

#leaves = n Θ(n)

Total= Θ( n lg n)

Page 41: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Conclusions

• Θ(n lg n) grows more slowly than Θ(n2).• Therefore, merge sort asymptotically

beats insertion sort in the worst case.•Disadvantage

▫Requires extra space Θ (n)• In practice, merge sort beats insertion

sort for n> 30 or so.

Page 42: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Divide-and-Conquer Example:Binary SearchFind an element in a sorted array:1. Divide: Check middle element.2. Conquer: Recursively search 1 subarray.3. Combine: Trivial.

•A[8] = {1, 2, 3, 4, 5, 7, 9, 11} Find 7

Page 43: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Divide-and-Conquer Example:Binary Search• For an ordered array A, finds if x is in the array

A[lo…hi]

Page 44: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Example

•A[8] = {1, 2, 3, 4, 5, 7, 9, 11}•lo = 1 hi = 8 x = 6

Page 45: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Analysis of Binary Search

?

Page 46: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Divide-and-Conquer Example:Powering a Number

?

?

?

Page 47: CMPT 438 Algorithms. Why Study Algorithms? Necessary in any computer programming problem ▫Improve algorithm efficiency: run faster, process more data,

Readings

•Chapter 3•Appendix A

•Homework 1•Quiz