bİm 202 algorithms

ÖĞR. GÖR. ZEYNEP TURGUT

BİM 202 ALGORITHMS

References - Books

Introduction to AlgorithmsCormen, Leiserson and Rivest and Stein

Data Structures & Algorithm Analysis in C++Mark Allen Weiss

Recommended Readings:Algoritmalar Prof. Dr. Vasif Vagifoglu NabiyevVeri Yapıları ve Algoritmalar

Dr. Rifat Çölkesen

Grading Guidelines

Two Exams: 20% midterm, 60% final.

%20 projects & homeworks & quizzes & attitude to the lesson.

Minimum 70% attendance to lesson required to clear the course.

Topics

Introduction to AlgorithmsAlgorithm AnalysisAlgorithm TypesSorting AlgorithmsSearching AlgorithmsGraph AlgorithmsTree AlgorithmsNP-Complete Problems

WARNING: This lecture contains mathematical content that may be shocking to some students.

Good, better, best. Never let it rest. ‘Till your good is better and your better is best.Saint Jerome

The Computer as a Tool

Much like the microscope does not define biology or the test tube does not define chemistry, the computer doesn't define Computer Science.

The computer is a tool by which Computer Scientists accomplish their goals – to solve problems.

What is Computer Science?

NOT about coding or hardware or software!

Computer Science is about PROBLEM SOLVING

Computer Science is about DEVELOPING ALGORITHMS to solve complex problems

Problem Solving: Main Steps

1. Problem definition2. Algorithm design / Algorithm specification3. Algorithm analysis4. Implementation5. Testing6. [Maintenance]

1. Problem Definition

What is the task to be accomplished? Calculate the average of the grades for a given

student Understand the talks given out by politicians and

translate them in Chinese

What are the time / space / speed / performance requirements ?

2. Algorithm Design / Specifications

Algorithm: Finite set of instructions that, if followed, accomplishes a particular task.

Describe: in natural language / pseudo-code / diagrams / etc.

Criteria to follow: Input: Zero or more quantities (externally produced) Output: One or more quantities Definiteness: Clarity, precision of each instruction Finiteness: The algorithm has to stop after a finite

(may be very large) number of steps Effectiveness: Each instruction has to be basic enough

and feasible Understand speech Translate to Chinese

Algorithm Questions

Computer Scientists ask themselves four critical questions when they evaluate algorithms …

Does the algorithm solve the stated problem?

Is the algorithm well-defined?

Does the algorithm produce an output?

Does the algorithm end in a reasonable length of time?

Developing an Algorithm

1. Identify the Inputs

2. Identify the Processes

3. Identify the Outputs

4. Develop a HIPO Chart

5. Identify modules

2.1. Identify the Inputs

What data do I need?

How will I get the data?

In what format will the data be?

2.2. Identify the Processes

How can I manipulate data to produce meaningful results?

Data vs. Information

2.3. Identify the Outputs

What outputs do I need to return to the user?

What format should the outputs take?

2.4. Develop a HIPO Chart

Hierarchy of Inputs Processes & Outputs

2.5. Identify Relevant Modules

How can I break larger problems into smaller, more manageable pieces?

What inputs do the modules need?

What processes need to happen in the modules?

What outputs are produced by the modules?

Summary

The goal of Computer Science is to develop sound algorithms for solving complex problems.

An algorithm is a well-developed, detailed approach for solving a problem.

Summary

To develop an algorithm:1. Identify the inputs2. Identify the processes3. Identify the outputs4. Develop a HIPO chart5. Identify relevant modules

Space complexity How much space is required

Time complexity How much time does it take to run the algorithm

Often, we deal with estimates!

3. Algorithm Analysis

4,5,6: Implementation, Testing, Maintainance

Implementation Decide on the programming language to use

C, C++, Lisp, Java, Perl, Prolog, assembly, etc. , etc. Write clean, well documented code

Test, test, test

Integrate feedback from users, fix bugs, ensure compatibility across different versions Maintenance

Space Complexity

Space complexity = The amount of memory required by an algorithm to run to completion [Core dumps = the most often encountered cause is

“memory leaks” – the amount of memory required larger than the memory available on a given system]

Some algorithms may be more efficient if data completely loaded into memory Need to look also at system limitations E.g. Classify 2GB of text in various categories [politics,

tourism, sport, natural disasters, etc.] – can I afford to load the entire collection?

Space Complexity (cont’d)

1. Fixed part: The size required to store certain data/variables, that is independent of the size of the problem:- e.g. name of the data collection- same size for classifying 2GB or 1MB of texts

2. Variable part: Space needed by variables, whose size is dependent on the size of the problem:- e.g. actual text - load 2GB of text VS. load 1MB of text

Space Complexity (cont’d)

S(P) = c + S(instance characteristics) c = constant

Example:void float sum (float* a, int n) {

float s = 0; for(int i = 0; i<n; i++) { s+ = a[i]; } return s;}Space? one word for n, one for a [passed by reference!], one

for i constant space!

Time Complexity

Often more important than space complexity space available (for computer programs!) tends to be larger

and larger time is still a problem for all of us

3-4GHz processors on the market still … researchers estimate that the computation of various

transformations for 1 single DNA chain for one single protein on 1 TerraHZ computer would take about 1 year to run to completion

Algorithms running time is an important issue

Running Time

Problem: prefix averages Given an array X Compute the array A such that A[i] is the average of

elements X[0] … X[i], for i=0..n-1Sol 1

At each step i, compute the element X[i] by traversing the array A and determining the sum of its elements, respectively the average

Sol 2 At each step i update a sum of the elements in the

array A Compute the element X[i] as sum/I

Big question: Which solution to choose?

Running time

Input

1 ms

2 ms

3 ms

4 ms

5 ms

A B C D E F G

worst-case

best-case}average-case?

Suppose the program includes an if-then statement that may execute or not: variable running timeTypically algorithms are measured by their worst case

Experimental Approach

Write a program that implements the algorithm

Run the program with data sets of varying size.

Determine the actual running time using a system call to measure time (e.g. system (date) );

Problems?

Experimental Approach

It is necessary to implement and test the algorithm in order to determine its running time.

Experiments can be done only on a limited set of inputs, and may not be indicative of the running time for other inputs.

The same hardware and software should be used in order to compare two algorithms. – condition very hard to achieve!

Use a Theoretical Approach

Based on high-level description of the algorithms, rather than language dependent implementations

Makes possible an evaluation of the algorithms that is independent of the hardware and software environments

Generality

Algorithm Description

How to describe algorithms independent of a programming language

Pseudo-Code = a description of an algorithm that is more structured than usual prose but less formal than a programming language

(Or diagrams)Example: find the maximum element of an array.

Algorithm arrayMax(A, n):Input: An array A storing n integers.Output: The maximum element in A.currentMax A[0]for i 1 to n -1 doif currentMax < A[i] then currentMax A[i]return currentMax

Pseudo Code

Expressions: use standard mathematical symbols use for assignment ( ? in C/C++) use = for the equality relationship (? in C/C++)

Method Declarations: -Algorithm name(param1, param2) Programming Constructs:

decision structures: if ... then ... [else ..] while-loops while ... do repeat-loops: repeat ... until ... for-loop: for ... do array indexing: A[i]

Methods calls: object method(args) returns: return value

Use comments Instructions have to be basic enough and feasible!

Low Level Algorithm Analysis

Based on primitive operations (low-level computations independent from the programming language)

E.g.:Make an addition = 1 operationCalling a method or returning from a method = 1 operation Index in an array = 1 operationComparison = 1 operation etc.

Method: Inspect the pseudo-code and count the number of primitive operations executed by the algorithm

Example

Algorithm arrayMax(A, n):Input: An array A storing n integers.Output: The maximum element in A.

currentMax A[0]for i 1 to n -1 doif currentMax < A[i] then

currentMax A[i]return currentMax

How many operations ?

Types of Algorithms

37

Algorithm classification

Algorithms that use a similar problem-solving approach can be grouped together

We’ll talk about a classification scheme for algorithms

This classification scheme is neither exhaustive nor disjoint

The purpose is not to be able to classify an algorithm as one type or another, but to highlight the various ways in which a problem can be attacked

38

A short list of categories

Algorithm types we will consider include: Simple recursive algorithms Backtracking algorithms Divide and conquer algorithms Dynamic programming algorithms Greedy algorithms Branch and bound algorithms Brute force algorithms Randomized algorithms

39

Simple recursive algorithms I

A simple recursive algorithm: Solves the base cases directly Recurs with a simpler subproblem Does some extra work to convert the solution to the

simpler subproblem into a solution to the given problem

I call these “simple” because several of the other algorithm types are inherently recursive

40

Example recursive algorithms

To count the number of elements in a list: If the list is empty, return zero; otherwise, Step past the first element, and count the remaining

elements in the list Add one to the result

To test if a value occurs in a list: If the list is empty, return false; otherwise, If the first thing in the list is the given value, return

true; otherwise Step past the first element, and test whether the value

occurs in the remainder of the list

41

L8

Recursive Algorithms

long factorial(int n){if (n<=0) return 1;return n*factorial(n-1);

}

Compute 5!

42

L8



}

f(5)=5·f(4)

43

L8



}

f(4)=4·f(3)

f(5)=5·f(4)

44

L8



}

f(3)=3·f(2)

f(4)=4·f(3)

f(5)=5·f(4)

45

L8



}

f(2)=2·f(1)

f(3)=3·f(2)

f(4)=4·f(3)

f(5)=5·f(4)

46

L8



}

f(1)=1·f(0)

f(2)=2·f(1)

f(3)=3·f(2)

f(4)=4·f(3)

f(5)=5·f(4)

47

L8



}

f(0)=1

f(1)=1·f(0)

f(2)=2·f(1)

f(3)=3·f(2)

f(4)=4·f(3)

f(5)=5·f(4)

48

L8



}

1·1=1

f(2)=2·f(1)

f(3)=3·f(2)

f(4)=4·f(3)

f(5)=5·f(4)

49

L8



}

2·1=2

f(3)=3·f(2)

f(4)=4·f(3)

f(5)=5·f(4)

50

L8



}

3·2=6

f(4)=4·f(3)

f(5)=5·f(4)

51

L8



}

4·6=24

f(5)=5·f(4)

52

L8



}

5·24=

120

53

L8



}

Return 5! = 120

54

Backtracking algorithms

Backtracking algorithms are based on a depth-first recursive search

A backtracking algorithm: Tests to see if a solution has been found, and if so,

returns it; otherwise For each choice that can be made at this point,

Make that choice Recur If the recursion returns a solution, return it

If no choices remain, return failure

55

Example backtracking algorithm

To color a map with no more than four colors: color(Country n):

If all countries have been colored (n > number of countries) return success; otherwise,

For each color c of four colors, If country n is not adjacent to a country that

has been colored c• Color country n with color c• recursively color country n+1• If successful, return success

If loop exits, return failure

56

Backtracking

Suppose you have to make a series of decisions, among various choices, where You don’t have enough information to know what

to choose Each decision leads to a new set of choices Some sequence of choices (possibly more than

one) may be a solution to your problemBacktracking is a methodical way of

trying out various sequences of decisions, until you find one that “works”

57

Solving a maze

Given a maze, find a path from start to finishAt each intersection, you have to decide

between three or fewer choices: Go straight Go left Go right

You don’t have enough information to choose correctly

Each choice leads to another set of choicesOne or more sequences of choices may (or may not)

lead to a solutionMany types of maze problem can be solved with

backtracking

58

Coloring a map

You wish to color a map withnot more than four colors red, yellow, green, blue

Adjacent countries must be indifferent colors

You don’t have enough information to choose colorsEach choice leads to another set of choicesOne or more sequences of choices may (or may not)

lead to a solutionMany coloring problems can be solved with

backtracking

59

Solving a puzzle

In this puzzle, all holes but oneare filled with white pegs

You can jump over one pegwith another

Jumped pegs are removedThe object is to remove all

but the last pegYou don’t have enough information to jump

correctlyEach choice leads to another set of choicesOne or more sequences of choices may (or may

not) lead to a solutionMany kinds of puzzle can be solved with

backtracking

60

Divide and Conquer

A divide and conquer algorithm consists of two parts: Divide the problem into smaller subproblems of the

same type, and solve these subproblems recursively Combine the solutions to the subproblems into a

solution to the original problem Traditionally, an algorithm is only called

“divide and conquer” if it contains at least two recursive calls

61

Examples

Quicksort: Partition the array into two parts (smaller numbers

in one part, larger numbers in the other part) Quicksort each of the parts No additional work is required to combine the two

sorted parts Mergesort:

Cut the array in half, and mergesort each half Combine the two sorted arrays into a single sorted

array by merging them

62

Binary tree lookup

Here’s how to look up something in a sorted binary tree: Compare the key to the value in the root

If the two values are equal, report success If the key is less, search the left subtree If the key is greater, search the right subtree

This is not a divide and conquer algorithm because, although there are two recursive calls, only one is used at each level of the recursion

63

Fibonacci numbers

To find the nth Fibonacci number: If n is zero or one, return one; otherwise, Compute fibonacci(n-1) and fibonacci(n-2) Return the sum of these two numbers

This is an expensive algorithm It requires O(fibonacci(n)) time This is equivalent to exponential time, that is, O(2n)

64

Dynamic programming algorithms

A dynamic programming algorithm remembers past results and uses them to find new results

Dynamic programming is generally used for optimization problems Multiple solutions exist, need to find the “best” one Requires “optimal substructure” and “overlapping

subproblems” Optimal substructure: Optimal solution contains optimal

solutions to subproblems Overlapping subproblems: Solutions to subproblems can

be stored and reused in a bottom-up fashionThis differs from Divide and Conquer, where

subproblems generally need not overlap

65

Fibonacci numbers again

To find the nth Fibonacci number: If n is zero or one, return one; otherwise, Compute, or look up in a table, fibonacci(n-1) and

fibonacci(n-2) Find the sum of these two numbers Store the result in a table and return it

Since finding the nth Fibonacci number involves finding all smaller Fibonacci numbers, the second recursive call has little work to do

The table may be preserved and used again later

66

Greedy algorithms

An optimization problem is one in which you want to find, not just a solution, but the best solution

A “greedy algorithm” sometimes works well for optimization problems

A greedy algorithm works in phases: At each phase: You take the best you can get right now, without

regard for future consequences You hope that by choosing a local optimum at each

step, you will end up at a global optimum

67

Example: Counting moneySuppose you want to count out a certain

amount of money, using the fewest possible bills and coins

A greedy algorithm would do this would be:At each step, take the largest possible bill or coin that does not overshoot Example: To make $6.39, you can choose:

a $5 bill a $1 bill, to make $6 a 25¢ coin, to make $6.25 A 10¢ coin, to make $6.35 four 1¢ coins, to make $6.39

For US money, the greedy algorithm always gives the optimum solution

68

A failure of the greedy algorithmIn some (fictional) monetary system,

“krons” come in 1 kron, 7 kron, and 10 kron coins

Using a greedy algorithm to count out 15 krons, you would get A 10 kron piece Five 1 kron pieces, for a total of 15 krons This requires six coins

A better solution would be to use two 7 kron pieces and one 1 kron piece This only requires three coins

The greedy algorithm results in a solution, but not in an optimal solution

69

Branch and bound algorithms

Branch and bound algorithms are generally used for optimization problems As the algorithm progresses, a tree of subproblems is

formed The original problem is considered the “root problem” A method is used to construct an upper and lower

bound for a given problem At each node, apply the bounding methods

If the bounds match, it is deemed a feasible solution to that particular subproblem

If bounds do not match, partition the problem represented by that node, and make the two subproblems into children nodes

Continue, using the best known feasible solution to trim sections of the tree, until all nodes have been solved or trimmed

70

Example branch and bound algorithm

Traveling salesman problem: A salesman has to visit each of n cities (at least) once each, and wants to minimize total distance traveled

Consider the root problem to be the problem of finding the shortest route through a set of cities visiting each city once

Split the node into two child problems: Shortest route visiting city A first Shortest route not visiting city A first

Continue subdividing similarly as the tree grows

71

Brute force algorithm

A brute force algorithm simply tries all possibilities until a satisfactory solution is found Such an algorithm can be:

Optimizing: Find the best solution. This may require finding all solutions, or if a value for the best solution is known, it may stop when any best solution is found Example: Finding the best path for a traveling

salesman Satisficing: Stop as soon as a solution is found that is

good enough Example: Finding a traveling salesman path that is

within 10% of optimal

72

Improving brute force algorithms

Often, brute force algorithms require exponential time

Various heuristics and optimizations can be used Heuristic: A “rule of thumb” that helps you decide

which possibilities to look at first Optimization: In this case, a way to eliminate certain

possibilities without fully exploring them

73

Randomized algorithms

A randomized algorithm uses a random number at least once during the computation to make a decision

Example: In Quicksort, using a random number to choose a pivot

Example: Trying to factor a large number by choosing random numbers as possible divisors

Simple Sorting Algorithms

75

Bubble sort

Compare each element (except the last one) with its neighbor to the right If they are out of order, swap them This puts the largest element at the very end The last element is now in the correct and final place

Compare each element (except the last two) with its neighbor to the right If they are out of order, swap them This puts the second largest element next to last The last two elements are now in their correct and final

placesCompare each element (except the last three) with

its neighbor to the right Continue as above until you have no unsorted elements on

the left

76

Example of bubble sort

7 2 8 5 4

2 7 8 5 4

2 7 8 5 4

2 7 5 8 4

2 7 5 4 8

2 7 5 4 8

2 5 7 4 8

2 5 4 7 8

2 7 5 4 8

2 5 4 7 8

2 4 5 7 8

2 5 4 7 8

2 4 5 7 8

2 4 5 7 8

(done)

77

Code for bubble sort

public static void bubbleSort(int[] a) { int outer, inner; for (outer = a.length - 1; outer > 0; outer--) { // counting down for (inner = 0; inner < outer; inner++) { // bubbling up if (a[inner] > a[inner + 1]) { // if out of order... int temp = a[inner]; // ...then swap a[inner] = a[inner + 1]; a[inner + 1] = temp; } } }}

78Analysis of bubble sort

for (outer = a.length - 1; outer > 0; outer--) { for (inner = 0; inner < outer; inner++) { if (a[inner] > a[inner + 1]) { // code for swap omitted} } }

Let n = a.length = size of the array The outer loop is executed n-1 times (call it n, that’s

close enough) Each time the outer loop is executed, the inner loop is

executed Inner loop executes n-1 times at first, linearly dropping

to just once On average, inner loop executes about n/2 times for

each execution of the outer loop In the inner loop, the comparison is always done

(constant time), the swap might be done (also constant time)

Result is n * n/2 * k, that is, O(n2/2 + k) = O(n2)

79

Loop invariants

You run a loop in order to change thingsOddly enough, what is usually most important in

understanding a loop is finding an invariant: that is, a condition that doesn’t change

In bubble sort, we put the largest elements at the end, and once we put them there, we don’t move them again The variable outer starts at the last index in the array and

decreases to 0 Our invariant is: Every element to the right of outer is in the

correct place That is, for all j > outer, if i < j, then a[i] <= a[j] When this is combined with the loop exit test, outer == 0, we

know that all elements of the array are in the correct place

80

Selection sort

Given an array of length n, Search elements 0 through n-1 and select the smallest

Swap it with the element in location 0 Search elements 1 through n-1 and select the smallest



Swap it with the element in location 3 Continue in this fashion until there’s nothing left to

search

81

Example and analysis of selection sort

The selection sort might swap an array element with itself--this is harmless, and not worth checking for

Analysis: The outer loop executes n-1 times The inner loop executes about n/2

times on average (from n to 2 times) Work done in the inner loop is

constant (swap two array elements) Time required is roughly (n-1)*(n/2) You should recognize this as O(n2)

7 2 8 5 4

2 7 8 5 4

2 4 8 5 7

2 4 5 8 7

2 4 5 7 8

82

Code for selection sort

public static void selectionSort(int[] a) { int outer, inner, min; for (outer = 0; outer < a.length - 1; outer++) { min = outer; for (inner = outer + 1; inner < a.length; inner++) { if (a[inner] < a[min]) { min = inner; } // Invariant: for all i, if outer <= i <= inner, then a[min] <= a[i] }

// a[min] is least among a[outer]..a[a.length - 1] int temp = a[outer]; a[outer] = a[min]; a[min] = temp; // Invariant: for all i <= outer, if i < j then a[i] <= a[j] }}

83

Invariants for selection sort

For the inner loop: This loop searches through the array, incrementing inner

from its initial value of outer+1 up to a.length-1 As the loop proceeds, min is set to the index of the

smallest number found so far Our invariant is:

for all i such that outer <= i <= inner, a[min] <= a[i]For the outer (enclosing) loop:

The loop counts up from outer = 0 Each time through the loop, the minimum remaining

value is put in a[outer] Our invariant is:

for all i <= outer, if i < j then a[i] <= a[j]

84

Insertion sort

The outer loop of insertion sort is: for (outer = 1; outer < a.length; outer++) {...}

The invariant is that all the elements to the left of outer are sorted with respect to one another For all i < outer, j < outer, if i < j then a[i] <= a[j] This does not mean they are all in their final correct place;

the remaining array elements may need to be inserted When we increase outer, a[outer-1] becomes to its left; we

must keep the invariant true by inserting a[outer-1] into its proper place

This means: Finding the element’s proper place Making room for the inserted element (by shifting over other

elements) Inserting the element

85

One step of insertion sort

3 4 7 12 14 14 20 21 33 38 10 55 9 23 28 16

sorted next to be inserted

3 4 7 55 9 23 28 16

10temp

3833212014141210

sorted

less than 10

86

Analysis of insertion sort

We run once through the outer loop, inserting each of n elements; this is a factor of n

On average, there are n/2 elements already sorted The inner loop looks at (and moves) half of these This gives a second factor of n/4

Hence, the time required for an insertion sort of an array of n elements is proportional to n2/4

Discarding constants, we find that insertion sort is O(n2)

87

Summary

Bubble sort, selection sort, and insertion sort are all O(n2)

As we will see later, we can do much better than this with somewhat more complicated sorting algorithms

Within O(n2), Bubble sort is very slow, and should probably never be used

for anything Selection sort is intermediate in speed Insertion sort is usually the fastest of the three--in fact, for

small arrays (say, 10 or 15 elements), insertion sort is faster than more complicated sorting algorithms

Selection sort and insertion sort are “good enough” for small arrays

88

Overview

Divide and Conquer

Merge Sort

Quick Sort

Heap Sort

89

Divide and Conquer1. Base Case, solve the problem

directly if it is small enough

2. Divide the problem into two or more similar and smaller subproblems

3. Recursively solve the subproblems

4. Combine solutions to the subproblems

90Divide and Conquer - Sort

Problem: Input: A[left..right] – unsorted array of

integers Output: A[left..right] – sorted in non-

decreasing order

91

Divide and Conquer - Sort1. Base case

at most one element (left ≥ right), return2. Divide A into two subarrays: FirstPart,

SecondPart Two Subproblems:

sort the FirstPart sort the SecondPart

3. Recursively sort FirstPart sort SecondPart

4. Combine sorted FirstPart and sorted SecondPart

92

Overview

Divide and Conquer

Merge Sort

Quick Sort

Heap Sort

93

Merge Sort: Idea

Merge

Recursively sort

Divide intotwo halves FirstPart SecondPart

FirstPart SecondPart

A

A is sorted!

94Merge Sort: Algorithm

Merge-Sort (A, left, right)

if left ≥ right return

else

middle ← b(left+right)/2

Merge-Sort(A, left, middle)

Merge-Sort(A, middle+1, right)

Merge(A, left, middle, right)

Recursive Call

95

A[middle]A[left]

Sorted FirstPart

Sorted SecondPart

Merge-Sort: Merge

A[right]

merge

A:

A:

Sorted

96

6 10 14 223 5 15 28L: R:

Temporary Arrays

5 15 28 30 6 10 145

Merge-Sort: Merge Example

2 3 7 8 1 4 5 6A:

97Merge-Sort: Merge Example

3 5 15 28 30 6 10 14

L:

A:

3 15 28 30 6 10 14 22R:

i=0

j=0

k=0

2 3 7 8 1 4 5 6

1


1 5 15 28 30 6 10 14

L:

A:

3 5 15 28 6 10 14 22R:

k=1

2 3 7 8 1 4 5 6

2

i=0

j=1


1 2 15 28 30 6 10 14

L:

A:

6 10 14 22R:

i=1

k=2

2 3 7 8 1 4 5 6

3

j=1

100


1 2 3 6 10 14

L:

A:

6 10 14 22R:

i=2

j=1

k=3

2 3 7 8 1 4 5 6

4

101


1 2 3 4 6 10 14

L:

A:

6 10 14 22R:

j=2

k=4

2 3 7 8 1 4 5 6

i=2

5

102


1 2 3 4 5 6 10 14

L:

A:

6 10 14 22R:

i=2

j=3

k=5

2 3 7 8 1 4 5 6

6

103


1 2 3 4 5 6 14

L:

A:

6 10 14 22R:

k=6

2 3 7 8 1 4 5 6

7

i=2

j=4

104


1 2 3 4 5 6 7 14

L:

A:

3 5 15 28 6 10 14 22R:

2 3 7 8 1 4 5 6

8

i=3

j=4

k=7

105


1 2 3 4 5 6 7 8

L:

A:

3 5 15 28 6 10 14 22R:

2 3 7 8 1 4 5 6

i=4

j=4

k=8

106

Merge(A, left, middle, right)1. n1 ← middle – left + 12. n2 ← right – middle3. create array L[n1], R[n2]4. for i ← 0 to n1-1 do L[i] ← A[left +i]5. for j ← 0 to n2-1 do R[j] ← A[middle+j]6. k ← i ← j ← 07. while i < n1 & j < n2 8. if L[i] < R[j] 9. A[k++] ← L[i++]10. else11. A[k++] ← R[j++]12. while i < n1

13. A[k++] ← L[i++]14. while j < n2

15. A[k++] ← R[j++] n = n1+n2

Space: nTime : cn for some constant c

107

6 2 8 4 3 7 5 16 2 8 4 3 7 5 1

Merge-Sort(A, 0, 7)Divide

A:

108

6 2 8 4

3 7 5 1

6 2 8 4

Merge-Sort(A, 0, 3)

, divideA:

Merge-Sort(A, 0, 7)

109

3 7 5 1

8 4

6 26 2

Merge-Sort(A, 0, 1)

, divideA:

Merge-Sort(A, 0, 7)

110

3 7 5 1

8 4

6

2

Merge-Sort(A, 0, 0), base caseA:

Merge-Sort(A, 0, 7)

111

3 7 5 1

8 4

6 2

Merge-Sort(A, 0, 0), returnA:

Merge-Sort(A, 0, 7)

112

3 7 5 1

8 4

6

2

Merge-Sort(A, 1, 1)

, base caseA:

Merge-Sort(A, 0, 7)

113

3 7 5 1

8 4

6 2


Merge-Sort(A, 0, 7)

114

3 7 5 1

8 4

2 6

Merge(A, 0, 0, 1)A:

Merge-Sort(A, 0, 7)

115

3 7 5 1

8 42 6


Merge-Sort(A, 0, 7)

116

3 7 5 1

8 4

2 6

Merge-Sort(A, 2, 3)

48

, divideA:

Merge-Sort(A, 0, 7)

117

3 7 5 1

4

2 6

8


Merge-Sort(A, 0, 7)

118

3 7 5 1

4

2 6

8


Merge-Sort(A, 0, 7)

119

4

2 6

8


Merge-Sort(A, 0, 7)

120

3 7 5 1

4

2 6

8


Merge-Sort(A, 0, 7)

121

3 7 5 1

2 6

4 8

Merge(A, 2, 2, 3)A:

Merge-Sort(A, 0, 7)

122

3 7 5 1

2 6 4 8


Merge-Sort(A, 0, 7)

123

3 7 5 1

2 4 6 8

Merge(A, 0, 1, 3)A:

Merge-Sort(A, 0, 7)

124

3 7 5 12 4 6 8Merge-Sort(A, 0, 3), returnA:

Merge-Sort(A, 0, 7)

125

3 7 5 1

2 4 6 8Merge-Sort(A, 4, 7)

A:

Merge-Sort(A, 0, 7)

126

1 3 5 7

2 4 6 8A:Merge (A, 4, 5, 7)

Merge-Sort(A, 0, 7)

127

1 3 5 72 4 6 8Merge-Sort(A, 4, 7), returnA:

Merge-Sort(A, 0, 7)

128

1 2 3 4 5 6 7 8Merge(A, 0, 3, 7)

A:

Merge-Sort(A, 0, 7)Merge-Sort(A, 0, 7), done!

129

Merge-Sort Analysis cn

2 × cn/2 = cn

4 × cn/4 = cn

n/2 × 2c = cn

log n levels

• Total running time: (nlogn)• Total Space: (n)

Total: cn log n

n

n/2 n/2

n/4 n/4 n/4 n/4

2 2 2

130

Merge-Sort Summary

Approach: divide and conquerTime

Most of the work is in the merging Total time: (n log n)

Space: (n), more space than other sorts.

131

Overview

Divide and Conquer

Merge Sort

Quick Sort

Heap Sort

132

Quick Sort

Divide: Pick any element p as the pivot, e.g, the

first element Partition the remaining elements into

FirstPart, which contains all elements < pSecondPart, which contains all elements ≥ p

Recursively sort the FirstPart and SecondPart

Combine: no work is necessary since sorting

is done in place

133

Quick Sort

x < p

p p ≤ x

PartitionFirstPart SecondPart

ppivot

A:

Recursive call

x < p

p p ≤ x

SortedFirstPart

SortedSecondPart

Sorted

134

Quick Sort

Quick-Sort(A, left, right)if left ≥ right return

else middle ← Partition(A, left, right) Quick-Sort(A, left, middle–1 ) Quick-Sort(A, middle+1, right)

end if

135

Partition

p

p x < p

p ≤ x

p p ≤ xx < p

A:

A:

A:p

136

Partition Example

A: 4 8 6 3 5 1 7 2

137

Partition Example

A: 4 8 6 3 5 1 7 2

i=0

j=1

138

Partition Example

A:

j=1

4 8 6 3 5 1 7 2

i=0

8

139

Partition Example

A: 4 8 6 3 5 1 7 26

i=0

j=2

140

Partition Example

A: 4 8 6 3 5 1 7 2

i=0

383

j=3

i=1

141

Partition Example

A: 4 3 6 8 5 1 7 2

i=1

5

j=4

142

Partition Example

A: 4 3 6 8 5 1 7 2

i=1

1

j=5

143

Partition Example

A: 4 3 6 8 5 1 7 2

i=2

1 6

j=5

144

Partition Example

A: 4 3 8 5 7 2

i=2

1 6 7

j=6

145

Partition Example

A: 4 3 8 5 7 2

i=2

1 6 22 8

i=3

j=7

146

Partition Example

A: 4 3 2 6 7 8

i=3

1 5

j=8

147

Partition Example

A: 4 1 6 7 8

i=3

2 542 3

148

A: 3 6 7 81 542

x < 4 4 ≤ x

pivot incorrect position

Partition Example

149

Partition(A, left, right)1. x ← A[left]2. i ← left3. for j ← left+1 to right4. if A[j] < x then 5. i ← i + 16. swap(A[i], A[j])7. end if8. end for j9. swap(A[i], A[left])10. return i

n = right – left +1 Time: cn for some constant c Space: constant

150

4 8 6 3 5 1 7 22 3 1 5 6 7 84

Quick-Sort(A, 0, 7)Partition

A:

151

2 3 1

5 6 7 84

2 1 3

Quick-Sort(A, 0, 7)Quick-Sort(A, 0, 2)

A:

, partition

152

2

5 6 7 84

1

1 3


, base case, return

153

2

5 6 7 84

1

33


, base case

154

5 6 7 842 1 3

2 1 3

Quick-Sort(A, 0, 7)Quick-Sort(A, 2, 2), returnQuick-Sort(A, 0, 2), return

155

42 1 3

5 6 7 86 7 85

Quick-Sort(A, 0, 7)Quick-Sort(A, 2, 2), returnQuick-Sort(A, 4, 7), partition

156

4

5

6 7 87 866

2 1 3

Quick-Sort(A, 0, 7)Quick-Sort(A, 5, 7), partition

157

4

5

6

7 887

2 1 3

Quick-Sort(A, 0, 7)Quick-Sort(A, 6, 7), partition

158

4

5

6

7

2 1 3


8

, return, base case

8

159

4

5

6 87

2 1 3


, return

160

4

5

2 1 3


, return

6 87

161

42 1 3


, return5 6 87

162

42 1 3


, done!5 6 87

163

Quick-Sort: Best Case Even Partition

Total time: (nlogn)

cn

2 × cn/2 = cn

4 × c/4 = cn

n/3 × 3c = cn

log n levels

n

n/2 n/2

n/4

3 3 3

n/4n/4n/4

164

cn

c(n-1)

3c

2c

n

n-1

n-2

3

2

c(n-2)

Happens only if input is sortd input is reversely

sorted

Quick-Sort: Worst Case Unbalanced Partition

Total time: (n2)

165

Quick-Sort: an Average CaseSuppose the split is 1/10 : 9/10

Quick-Sort: an Average Case

cn

cn

cn

≤cn

n

0.1n 0.9n

0.01n 0.09n 0.09n

Total time: (nlogn)

0.81n

2

2

log10n

log10/9n

≤cn

166

Quick-Sort Summary

Time Most of the work done in partitioning. Average case takes (n log(n)) time. Worst case takes (n2) time

Space Sorts in-place, i.e., does not require additional space

167

SummaryDivide and Conquer

Merge-Sort Most of the work done in Merging (n log(n)) time (n) space

Quick-Sort Most of the work done in partitioning Average case takes (n log(n)) time Worst case takes (n2) time (1) space

168

Why study Heapsort?

It is a well-known, traditional sorting algorithm you will be expected to know

Heapsort is always O(n log n)Quicksort is usually O(n log n) but in the

worst case slows to O(n2)Quicksort is generally faster, but Heapsort

is better in time-critical applicationsHeapsort is a really cool algorithm!

169

What is a “heap”?

Definitions of heap:1. A large area of memory from which the

programmer can allocate blocks as needed, and deallocate them (or allow them to be garbage collected) when no longer needed

2. A balanced, left-justified binary tree in which no node has a value greater than the value in its parent

These two definitions have little in common

Heapsort uses the second definition

170

Balanced binary trees

Recall: The depth of a node is its distance from the root The depth of a tree is the depth of the deepest node

A binary tree of depth n is balanced if all the nodes at depths 0 through n-2 have two children

Balanced Balanced Not balanced

n-2n-1n

171

Left-justified binary trees

A balanced binary tree of depth n is left-justified if: it has 2n nodes at depth n (the tree is “full”),

or it has 2k nodes at depth k, for all k < n, and all

the leaves at depth n are as far left as possible

Left-justified Not left-justified

172

Plan of attack

First, we will learn how to turn a binary tree into a heap

Next, we will learn how to turn a binary tree back into a heap after it has been changed in a certain way

Finally (this is the cool part) we will see how to use these ideas to sort an array

173

The heap property

A node has the heap property if the value in the node is as large as or larger than the values in its children

All leaf nodes automatically have the heap property

A binary tree is a heap if all nodes in it have the heap property

12

8 3Blue node has heap property

12


12

8 14Blue node does not have heap property

174

siftUpGiven a node that does not have the heap

property, you can give it the heap property by exchanging its value with the value of the larger child

This is sometimes called sifting upNotice that the child may have lost the heap

property

14


12

8 14Blue node does not have heap property

175

Constructing a heap I

A tree consisting of a single node is automatically a heap

We construct a heap by adding nodes one at a time: Add the node just to the right of the rightmost

node in the deepest level If the deepest level is full, start a new level

Examples:Add a new node here

Add a new node here

176

Constructing a heap II

Each time we add a node, we may destroy the heap property of its parent node

To fix this, we sift upBut each time we sift up, the value of the

topmost node in the sift may increase, and this may destroy the heap property of its parent node

We repeat the sifting up process, moving up in the tree, until either We reach nodes whose values don’t need to be

swapped (because the parent is still larger than both children), or

We reach the root

177

Constructing a heap III

8 8

10

10

8

10

8 5

10

8 5

12

10

12 5

8

12

10 5

8

1 2 3

4

178

Other children are not affected

The node containing 8 is not affected because its parent gets larger, not smaller

The node containing 5 is not affected because its parent gets larger, not smaller

The node containing 8 is still not affected because, although its parent got smaller, its parent is still greater than it was originally

12

10 5

8 14

12

14 5

8 10

14

12 5

8 10

179

A sample heapHere’s a sample binary tree after it has been

heapified

Notice that heapified does not mean sortedHeapifying does not change the shape of the

binary tree; this binary tree is balanced and left-justified because it started out that way

19

1418

22

321

14

119

15

25

1722

180

Removing the root (animated)Notice that the largest number is now in the

rootSuppose we discard the root:

How can we fix the binary tree so it is once again balanced and left-justified?

Solution: remove the rightmost leaf at the deepest level and use it for the new root

19

1418

22

321

14

119

15

1722

11

181

The reHeap method IOur tree is balanced and left-justified, but no longer a heapHowever, only the root lacks the heap property

We can siftUp() the rootAfter doing this, one and only one of its

children may have lost the heap property

19

1418

22

321

14

9

15

1722

11

182

The reHeap method IINow the left child of the root (still the number

11) lacks the heap property

We can siftUp() this nodeAfter doing this, one and only one of its

children may have lost the heap property

19

1418

22

321

14

9

15

1711

22

183

The reHeap method IIINow the right child of the left child of the root

(still the number 11) lacks the heap property:

We can siftUp() this nodeAfter doing this, one and only one of its children may

have lost the heap property —but it doesn’t, because it’s a leaf

19

1418

11

321

14

9

15

1722

22

184

The reHeap method IVOur tree is once again a heap, because every

node in it has the heap property

Once again, the largest (or a largest) value is in the rootWe can repeat this process until the tree becomes emptyThis produces a sequence of values in order largest to smallest

19

1418

21

311

14

9

15

1722

22

185

Sorting

What do heaps have to do with sorting an array?

Here’s the neat part: Because the binary tree is balanced and left justified,

it can be represented as an array Danger Will Robinson: This representation works well

only with balanced, left-justified binary trees All our operations on binary trees can be represented

as operations on arrays To sort:

heapify the array; while the array isn’t empty { remove and replace the root; reheap the new root node;

}

186

Mapping into an array

Notice: The left child of index i is at index 2*i+1 The right child of index i is at index 2*i+2 Example: the children of node 3 (19) are 7 (18) and 8

(14)

19

1418

22

321

14

119

15

25

1722

25 22 17 19 22 14 15 18 14 21 3 9 11 0 1 2 3 4 5 6 7 8 9 10 11 12

187

Removing and replacing the root

The “root” is the first element in the arrayThe “rightmost node at the deepest level” is the last

elementSwap them...

...And pretend that the last element in the array no longer exists—that is, the “last index” is 11 (containing the value 9)

25 22 17 19 22 14 15 18 14 21 3 9 11 0 1 2 3 4 5 6 7 8 9 10 11 12

11 22 17 19 22 14 15 18 14 21 3 9 25 0 1 2 3 4 5 6 7 8 9 10 11 12

188

Reheap and repeatReheap the root node (index 0, containing 11)...

Remember, though, that the “last” array index is changedRepeat until the last becomes first, and the array is sorted!

22 22 17 19 21 14 15 18 14 11 3 9 25 0 1 2 3 4 5 6 7 8 9 10 11 12

9 22 17 19 22 14 15 18 14 21 3 22 25 0 1 2 3 4 5 6 7 8 9 10 11 12

11 22 17 19 22 14 15 18 14 21 3 9 25 0 1 2 3 4 5 6 7 8 9 10 11 12

...And again, remove and replace the root node

189

Analysis I

Here’s how the algorithm starts: heapify the array;

Heapifying the array: we add each of n nodes Each node has to be sifted up, possibly as far as

the root Since the binary tree is perfectly balanced, sifting up

a single node takes O(log n) time Since we do this n times, heapifying takes n*O(log

n) time, that is, O(n log n) time

190

Analysis II

Here’s the rest of the algorithm: while the array isn’t empty { remove and replace the root; reheap the new root node;

}We do the while loop n times (actually, n-1

times), because we remove one of the n nodes each time

Removing and replacing the root takes O(1) time

Therefore, the total time is n times however long it takes the reheap method

191

Analysis III

To reheap the root node, we have to follow one path from the root to a leaf node (and we might stop before we reach a leaf)

The binary tree is perfectly balancedTherefore, this path is O(log n) long

And we only do O(1) operations at each node Therefore, reheaping takes O(log n) times

Since we reheap inside a while loop that we do n times, the total time for the while loop is n*O(log n), or O(n log n)

192

Analysis IV

Here’s the algorithm again: heapify the array; while the array isn’t empty { remove and replace the root; reheap the new root node;

}We have seen that heapifying takes O(n log

n) timeThe while loop takes O(n log n) timeThe total time is therefore O(n log n) + O(n

log n)This is the same as O(n log n) time

193

Homework

1. What is the running time of Merge-Sort if the array is already sorted? What is the best case running time of Merge-Sort?

2. Demonstrate the working of Partition on sequence (13, 5, 14, 11, 16, 12, 1, 15). What is the value of i returned at the completion of Partition?

Searching Algorithms

195

Searching and sorting algorithms with Big O values.

Algorithm Location Big O

Searching Algorithms:

Linear Search Section 24.2.1 O(n)

Binary Search Section 24.2.2 O(log n)

Recursive Linear Search Exercise 24.8 O(n)

Recursive Binary Search Exercise 24.9 O(log n)

Sorting Algorithms:

Selection Sort Section 24.3.1 O(n2)

Insertion Sort Section 24.3.2 O(n2)

Merge Sort Section 24.3.3 O(n log n)

Bubble Sort Exercises 24.5–24.6 O(n2)

196

Number of comparisons for common Big O notations.

n = O(log n) O(n) O(n log n) O(n2)

1 0 1 0 1

2 1 2 2 4

3 1 3 3 9

4 1 4 4 16

5 1 5 5 25

10 1 10 10 100

100 2 100 200 10000

1,000 3 1000 3000 106

1,000,000 6 1000000 6000000 1012

1,000,000,000 9 1000000000 9000000000 1018

bİm 202 algorithms

Documents

algorithm end

computer doesnt

algorithm welldefined

saint jeromethe computer

outputswhat outputs

inputswhat data

larger problems

stated problem