analysis of algorithms cs 477/677 instructor: monica nicolescu lecture 5
TRANSCRIPT
![Page 1: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/1.jpg)
Analysis of AlgorithmsCS 477/677
Instructor: Monica Nicolescu
Lecture 5
![Page 2: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/2.jpg)
CS 477/677 - Lecture 5 2
The Sorting Problem
• Input:
– A sequence of n numbers a1, a2, . . . , an
• Output:
– A permutation (reordering) a1’, a2’, . . . , an’ of the
input sequence such that a1’ ≤ a2’ ≤ · · · ≤ an’
![Page 3: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/3.jpg)
CS 477/677 - Lecture 5 3
Why Study Sorting Algorithms?
• There are a variety of situations that we can encounter– Do we have randomly ordered keys?– Are all keys distinct?– How large is the set of keys to be ordered?– Need guaranteed performance?– Does the algorithm sort in place?– Is the algorithm stable?
• Various algorithms are better suited to some of these situations
![Page 4: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/4.jpg)
CS 477/677 - Lecture 5 4
Stability
• A STABLE sort preserves relative order of records with equal keys
Sort file on first key:
Sort file on second key:
Records with key value 3 are not in order on first key!!
![Page 5: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/5.jpg)
CS 477/677 - Lecture 5 5
Insertion Sort
• Idea: like sorting a hand of playing cards– Start with an empty left hand and the cards facing
down on the table– Remove one card at a time from the table, and insert
it into the correct position in the left hand• compare it with each of the cards already in the hand, from
right to left
– The cards held in the left hand are sorted• these cards were originally the top cards of the pile on the
table
![Page 6: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/6.jpg)
CS 477/677 - Lecture 5 6
Example
![Page 7: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/7.jpg)
CS 477/677 - Lecture 5 7
INSERTION-SORT
Alg.: INSERTION-SORT(A)
for j ← 2 to n
do key ← A[ j ] Insert A[ j ] into the sorted sequence A[1 . . j -1]
i ← j - 1 while i > 0 and A[i] > key
do A[i + 1] ← A[i] i ← i – 1
A[i + 1] ← key• Insertion sort – sorts the elements in place
a8a7a6a5a4a3a2a1
1 2 3 4 5 6 7 8
key
![Page 8: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/8.jpg)
CS 477/677 - Lecture 5 8
Loop Invariant for Insertion Sort
Alg.: INSERTION-SORT(A)
for j ← 2 to n
do key ← A[ j ]
Insert A[ j ] into the sorted sequence A[1 . . j -1]
i ← j - 1 while i > 0 and A[i] > key
do A[i + 1] ← A[i] i ← i – 1
A[i + 1] ← key
Invariant: at the start of each iteration of the for loop, the elements in A[1 . . j-1] are in sorted order
![Page 9: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/9.jpg)
CS 477/677 - Lecture 5 9
Proving Loop Invariants
• Proving loop invariants works like induction
• Initialization (base case): – It is true prior to the first iteration of the loop
• Maintenance (inductive step): – If it is true before an iteration of the loop, it remains true before
the next iteration
• Termination: – When the loop terminates, the invariant gives us a useful
property that helps show that the algorithm is correct
![Page 10: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/10.jpg)
CS 477/677 - Lecture 5 10
Loop Invariant for Insertion Sort
• Initialization:
– Just before the first iteration, j = 2:
the subarray A[1 . . j-1] = A[1],
(the element originally in A[1]) – is
sorted
![Page 11: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/11.jpg)
CS 477/677 - Lecture 5 11
Loop Invariant for Insertion Sort
• Maintenance: – the while inner loop moves A[j -1], A[j -2], A[j -3],
and so on, by one position to the right until the proper position for key (which has the value that started out in A[j]) is found
– At that point, the value of key is placed into this position.
![Page 12: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/12.jpg)
CS 477/677 - Lecture 5 12
Loop Invariant for Insertion Sort
• Termination: – The outer for loop ends when j = n + 1 j-1 = n– Replace n with j-1 in the loop invariant:
• the subarray A[1 . . n] consists of the elements originally in A[1 . . n], but in sorted order
• The entire array is sorted!
jj - 1
![Page 13: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/13.jpg)
CS 477/677 - Lecture 5 13
Analysis of Insertion Sort
cost times
c1 n
c2 n-1
0 n-1
c4 n-1
c5
c6
c7
c8 n-1
n
j jt2
n
j jt2)1(
n
j jt2)1(
)1(11)1()1()( 82
72
62
5421
nctctctcncncncnTn
jj
n
jj
n
jj
INSERTION-SORT(A)
for j ← 2 to n
do key ← A[ j ] Insert A[ j ] into the sorted sequence A[1 . . j -1]
i ← j - 1
while i > 0 and A[i] > key
do A[i + 1] ← A[i]
i ← i – 1
A[i + 1] ← key
![Page 14: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/14.jpg)
CS 477/677 - Lecture 5 14
Best Case Analysis
• The array is already sorted
– A[i] ≤ key upon the first time the while loop test is run
(when i = j -1)
– tj = 1
• T(n) = c1n + c2(n -1) + c4(n -1) + c5(n -1) +
c8(n-1) = (c1 + c2 + c4 + c5 + c8)n - (c2 + c4
+ c5 + c8)
= an + b = (n)
“while i > 0 and A[i] > key”
)1(11)1()1()( 82
72
62
5421
nctctctcncncncnTn
jj
n
jj
n
jj
![Page 15: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/15.jpg)
CS 477/677 - Lecture 5 15
Worst Case Analysis
• The array is in reverse sorted order– Always A[i] > key in while loop test
– Have to compare key with all elements to the left of the j-th
position compare with j-1 elements tj = j
a quadratic function of n
• T(n) = (n2) order of growth in n2
2
)1()1(1
2
)1(
22
nnjand
nnj
n
j
n
j
)1(2
)1(
2
)1(1
2
)1()1()1()( 8765421
nc
nnc
nnc
nncncncncnT
cbnan 2
“while i > 0 and A[i] > key”
![Page 16: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/16.jpg)
CS 477/677 - Lecture 5 16
Comparisons and Exchanges in Insertion Sort
INSERTION-SORT(A)
for j ← 2 to n
do key ← A[ j ]
Insert A[ j ] into the sorted sequence A[1 . . j
-1]
i ← j - 1
while i > 0 and A[i] > key
do A[i + 1] ← A[i]
i ← i – 1
A[i + 1] ← key
cost
times
c1 n
c2 n-
1
0 n-
1
c4 n-
1
c5
c6
c7
c8 n-
1
n
j jt2
n
j jt2)1(
n
j jt2)1(
n2/2 comparisons
n2/2 exchanges
![Page 17: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/17.jpg)
CS 477/677 - Lecture 5 17
Insertion Sort - Summary
• Idea: like sorting a hand of playing cards– Start with an empty left hand and the cards facing
down on the table.– Remove one card at a time from the table, and insert
it into the correct position in the left hand
• Advantages– Good running time for “almost sorted” arrays (n)
• Disadvantages– (n2) running time in worst and average case– n2/2 comparisons and n2/2 exchanges
![Page 18: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/18.jpg)
CS 477/677 - Lecture 5 18
Bubble Sort
• Idea:– Repeatedly pass through the array– Swaps adjacent elements that are out of order
• Easier to implement, but slower than Insertion sort
1 2 3 n
i
1329648
j
![Page 19: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/19.jpg)
CS 477/677 - Lecture 5 19
Example
1329648i = 1 j
3129648i = 1 j
3219648
i = 1 j
3291648i = 1 j
3296148i = 1 j
3296418
i = 1 j
3296481
i = 1 j
3296481
i = 2 j
3964821
i = 3 j
9648321
i = 4 j
9684321
i = 5 j
9864321
i = 6 j
9864321
i = 7j
![Page 20: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/20.jpg)
CS 477/677 - Lecture 5 20
Bubble Sort
Alg.: BUBBLESORT(A)
for i 1 to length[A]do for j length[A] downto i + 1 do if A[j] < A[j -1]
then exchange A[j] A[j-1]
1329648i = 1 j
i
![Page 21: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/21.jpg)
CS 477/677 - Lecture 5 21
Bubble-Sort Running Time
T(n) = (n2)
222
)1()(
1 1
22
1
nnnnninin
n
i
n
i
n
i
Alg.: BUBBLESORT(A)
for i 1 to length[A]do for j length[A] downto i + 1 do if A[j] < A[j -1]
then exchange A[j] A[j-1]
T(n) = c1(n+1) +
n
i
in1
)1(c2 c3
n
i
in1
)( c4
n
i
in1
)(
= (n) +(c2 + c3 + c4)
n
i
in1
)(
Comparisons: n2/2 Exchanges: n2/2
![Page 22: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/22.jpg)
CS 477/677 - Lecture 5 22
Selection Sort
• Idea:– Find the smallest element in the array– Exchange it with the element in the first position– Find the second smallest element and exchange it with
the element in the second position– Continue until the array is sorted
• Invariant:– All elements to the left of the current index are in
sorted order and never changed again• Disadvantage:
– Running time depends only slightly on the amount of order in the file
1329648
![Page 23: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/23.jpg)
CS 477/677 - Lecture 5 23
Example
1329648
8329641
8349621
8649321
8964321
8694321
9864321
9864321
![Page 24: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/24.jpg)
CS 477/677 - Lecture 5 24
Selection Sort
Alg.: SELECTION-SORT(A)
n ← length[A]for j ← 1 to n - 1
do smallest ← j for i ← j + 1 to n
do if A[i] < A[smallest] then smallest ← i
exchange A[j] ↔ A[smallest]
1329648
![Page 25: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/25.jpg)
CS 477/677 - Lecture 5 25
»n2/2 comparisons
Analysis of Selection Sort
Alg.: SELECTION-SORT(A)
n ← length[A]
for j ← 1 to n - 1
do smallest ← j
for i ← j + 1 to n
do if A[i] < A[smallest]
then smallest ← i
exchange A[j] ↔ A[smallest]
cost
times
c1 1
c2 n
c3 n-
1
c4
c5
c6
c7 n-
1
1
1)1(
n
jjn
1
1)(
n
jjn
1
1)(
n
jjn
»nexchanges
T(n) = (n2)
![Page 26: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/26.jpg)
CS 477/677 - Lecture 5 26
Divide-and-Conquer
• Divide the problem into a number of subproblems
– Similar sub-problems of smaller size
• Conquer the sub-problems
– Solve the sub-problems recursively
– Sub-problem size small enough solve the problems in
straightforward manner
• Combine the solutions to the sub-problems
– Obtain the solution for the original problem
![Page 27: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/27.jpg)
CS 477/677 - Lecture 5 27
Merge Sort Approach
• To sort an array A[p . . r]:
• Divide– Divide the n-element sequence to be sorted into two
subsequences of n/2 elements each
• Conquer– Sort the subsequences recursively using merge sort
– When the size of the sequences is 1 there is nothing
more to do
• Combine– Merge the two sorted subsequences
![Page 28: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/28.jpg)
CS 477/677 - Lecture 5 28
Merge Sort
Alg.: MERGE-SORT(A, p, r)
if p < r Check for base case
then q ← (p + r)/2 Divide
MERGE-SORT(A, p, q) Conquer
MERGE-SORT(A, q + 1, r) Conquer
MERGE(A, p, q, r) Combine
• Initial call: MERGE-SORT(A, 1, n)
1 2 3 4 5 6 7 8
62317425
p rq
![Page 29: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/29.jpg)
CS 477/677 - Lecture 5 29
Example – n Power of 2
1 2 3 4 5 6 7 8
q = 462317425
1 2 3 4
7425
5 6 7 8
6231
1 2
25
3 4
74
5 6
31
7 8
62
1
5
2
2
3
4
4
7 1
6
3
7
2
8
6
5
Example
![Page 30: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/30.jpg)
CS 477/677 - Lecture 5 30
Merging
• Input: Array A and indices p, q, r such that p ≤ q < r– Subarrays A[p . . q] and A[q + 1 . . r] are
sorted
• Output: One single sorted subarray A[p . . r]
1 2 3 4 5 6 7 8
63217542
p rq
![Page 31: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/31.jpg)
CS 477/677 - Lecture 5 31
Merging
• Idea for merging:
– Two piles of sorted cards• Choose the smaller of the two top cards
• Remove it and place it in the output pile
– Repeat the process until one pile is empty
– Take the remaining input pile and place it face-down
onto the output pile
![Page 32: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/32.jpg)
CS 477/677 - Lecture 5 32
Merge - Pseudocode
Alg.: MERGE(A, p, q, r)1. Compute n1 and n2
2. Copy the first n1 elements into L[1 . . n1 + 1] and the next n2 elements into R[1 . . n2 + 1]
3. L[n1 + 1] ← ; R[n2 + 1] ←
4. i ← 1; j ← 15. for k ← p to r6. do if L[ i ] ≤ R[ j ]7. then A[k] ← L[ i ]8. i ←i + 19. else A[k] ← R[ j ]10. j ← j + 1
p q
7542
6321rq + 1
L
R
1 2 3 4 5 6 7 8
63217542
p rq
n1 n2
![Page 33: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/33.jpg)
CS 477/677 - Lecture 5 33
Running Time of Merge
• Initialization (copying into temporary arrays):– (n1 + n2) = (n)
• Adding the elements to the final array (the for
loop):– n iterations, each taking constant time (n)
• Total time for Merge:– (n)
![Page 34: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/34.jpg)
CS 477/677 - Lecture 5 34
Analyzing Divide and Conquer Algorithms
• The recurrence is based on the three steps of the paradigm:– T(n) – running time on a problem of size n– Divide the problem into a subproblems, each of size
n/b: takes D(n)– Conquer (solve) the subproblems: takes aT(n/b) – Combine the solutions: takes C(n)
(1) if n ≤ c
T(n) = aT(n/b) + D(n) + C(n)otherwise
![Page 35: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/35.jpg)
CS 477/677 - Lecture 5 35
MERGE – SORT Running Time
• Divide: – compute q as the average of p and r: D(n) = (1)
• Conquer: – recursively solve 2 subproblems, each of size n/2
2T (n/2)• Combine:
– MERGE on an n-element subarray takes (n) time C(n) = (n)
(1) if n =1
T(n) = 2T(n/2) + (n) if n > 1
![Page 36: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/36.jpg)
CS 477/677 - Lecture 5 36
Solve the Recurrence
T(n) = c if n = 12T(n/2) + cn if n > 1
Use Master’s Theorem:
Compare n with f(n) = cnCase 2: T(n) = Θ(nlgn)
![Page 37: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/37.jpg)
CS 477/677 - Lecture 5 37
Merge Sort - Discussion
• Running time insensitive of the input• Advantages:
– Guaranteed to run in (nlgn)
• Disadvantage– Requires extra space N
• Applications– Maintain a large ordered data file– How would you use Merge sort to do this?
![Page 38: Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5](https://reader035.vdocuments.mx/reader035/viewer/2022062308/56649cef5503460f949be154/html5/thumbnails/38.jpg)
CS 477/677 - Lecture 5 38
Readings
• Chapter 2