lecture 1 introduction to analysis of algorithmsreha/473/01-intro.pdf · lecture 1 . introduction...
TRANSCRIPT
![Page 1: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/1.jpg)
1
CS473 - Algorithms I
CS 473 – Lecture 1
Lecture 1 Introduction to Analysis of Algorithms
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
View in slide-show mode
![Page 2: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/2.jpg)
5 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Algorithm Definition
Algorithm: A sequence of computational steps that transform the input to the desired output
Procedure vs. algorithm An algorithm must halt within finite time with the right output
Example: Sorting
Algorithm a sequence of n numbers
sorted permutation of input sequence
![Page 3: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/3.jpg)
6 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Many Real World Applications
Bioinformatics Determine/compare DNA sequences
Internet Manage/manipulate/route data
Information retrieval Search and access information in large data
Security Encode & decode personal/financial/confidential data
Electronic design automation Minimize human effort in chip-design process
![Page 4: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/4.jpg)
7 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Course Objectives
Learn basic algorithms & data structures Gain skills to design new algorithms
Focus on efficient algorithms
Design algorithms that are fast use as little memory as possible are correct!
![Page 5: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/5.jpg)
8 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Outline of Lecture 1
Study two sorting algorithms as examples Insertion sort: Incremental algorithm Merge sort: Divide-and-conquer
Introduction to runtime analysis
Best vs. worst vs. average case Asymptotic analysis
![Page 6: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/6.jpg)
9 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Sorting Problem
Input: Sequence of numbers
⟨a1, a2,…,an⟩ Output: A permutation
Π= ⟨ Π (1), Π(2),…, Π (n)⟩ such that
aΠ(1)≤aΠ(2) ≤ … ≤ aΠ(n)
![Page 7: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/7.jpg)
10 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort
![Page 8: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/8.jpg)
11 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort: Basic Idea
Assume input array: A[1..n] Iterate j from 2 to n
iter j
j already sorted
after iter j
j
insert into sorted array
sorted subarray
![Page 9: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/9.jpg)
12 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Pseudo-code notation
Objective: Express algorithms to humans in a clear and concise way
Liberal use of English
Indentation for block structures
Omission of error handling and other details needed in real programs
![Page 10: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/10.jpg)
13 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Algorithm: Insertion Sort (from Section 2.2)
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
![Page 11: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/11.jpg)
14 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Algorithm: Insertion Sort
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
Iterate over array elts j
Loop invariant: The subarray A[1..j-1] is always sorted
j already sorted
key
![Page 12: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/12.jpg)
15 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Algorithm: Insertion Sort
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
Shift right the entries in A[1..j-1] that are > key
j already sorted
> key < key
j > key < key
![Page 13: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/13.jpg)
16 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Algorithm: Insertion Sort
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
> key < key
Insert key to the correct location End of iter j: A[1..j] is sorted
key
j
now sorted
![Page 14: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/14.jpg)
17 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort - Example
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
5 2 4 6 1 3
![Page 15: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/15.jpg)
18 CS 473 – Lecture 1
5 4 6 1 3 2
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort - Example: Iteration j=2
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
j
sorted
5 2 4 6 1 3 initial
> 2 j
5 2 4 6 1 3 shift
key=2
sorted insert key
![Page 16: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/16.jpg)
19 CS 473 – Lecture 1
? ? ? ? ? ?
2 5 4 6 1 3
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort - Example: Iteration j=3
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
j
sorted
initial
key=4
What are the entries at the end of iteration j=3?
![Page 17: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/17.jpg)
20 CS 473 – Lecture 1
2 5 6 1 3
2 5 4 6 1 3
2 5 4 6 1 3
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort - Example: Iteration j=3
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
j
sorted
initial
> 4 j
shift
key=4
sorted insert key
< 4
4
![Page 18: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/18.jpg)
21 CS 473 – Lecture 1
2 4 5 1 3
2 4 5 6 1 3
2 4 5 6 1 3
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort - Example: Iteration j=4
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
j
sorted
initial
< 6 j
shift
key=6
sorted insert key 6
![Page 19: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/19.jpg)
22 CS 473 – Lecture 1
2 4 5 6 1 3
? ? ? ? ? ?
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort - Example: Iteration j=5
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
j
sorted
initial
key=1
What are the entries at the end of iteration j=5?
![Page 20: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/20.jpg)
23 CS 473 – Lecture 1
2 4 5 6 3
2 4 5 6 1 3
2 4 5 6 1 3
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort - Example: Iteration j=5
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
j
sorted
initial
>1 j
shift
key=1
sorted insert key
>1 >1 >1
1
![Page 21: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/21.jpg)
24 CS 473 – Lecture 1
1 2 4 5 6
1 2 4 5 6 3
1 2 4 5 6 3
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort - Example: Iteration j=6
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
j
sorted
initial
>3 j
shift
key=3
sorted insert key
>3 >3 <3
3
![Page 22: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/22.jpg)
25 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort Algorithm - Notes
Items sorted in-place Elements rearranged within array At most constant number of items stored outside the array
at any time (e.g. the variable key) Input array A contains sorted output sequence when the
algorithm ends
Incremental approach
Having sorted A[1..j-1], place A[j] correctly so that A[1..j] is sorted
![Page 23: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/23.jpg)
26 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Running Time
Depends on: Input size (e.g., 6 elements vs 6M elements) Input itself (e.g., partially sorted)
Usually want upper bound
![Page 24: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/24.jpg)
27 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Kinds of running time analysis
Worst Case (Usually) T(n) = max time on any input of size n
Average Case (Sometimes) T(n) = average time over all inputs of size n
Assumes statistical distribution of inputs Best Case (Rarely)
T(n) = min time on any input of size n BAD*: Cheat with slow algorithm that works fast on some inputs GOOD: Only for showing bad lower bound
*Can modify any algorithm (almost) to have a low best-case running time Check whether input constitutes an output at the very beginning of the algorithm
![Page 25: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/25.jpg)
28 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Running Time
For Insertion-Sort, what is its worst-case time? Depends on speed of primitive operations Relative speed (on same machine) Absolute speed (on different machines)
Asymptotic analysis
Ignore machine-dependent constants Look at growth of T(n) as n→∞
![Page 26: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/26.jpg)
29 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Θ Notation
Drop low order terms Ignore leading constants e.g. 2n2+5n + 3 = Θ(n2) 3n3+90n2-2n+5= Θ(n3)
Formal explanations in the next lecture.
![Page 27: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/27.jpg)
CS473 – Lecture 1 Cevdet Aykanat - Bilkent University Computer Engineering Department
30
• As n gets large, a Θ(n2) algorithm runs faster than a Θ(n3) algorithm
T(n)
n
min value for n0
Runtime larger asymptotically
![Page 28: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/28.jpg)
31 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort – Runtime Analysis
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
Cost c1
c2
c3 c4
c5 c6
c7
tj: The number of times while loop
test is executed for j
![Page 29: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/29.jpg)
32 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
How many times is each line executed?
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
# times n
n-1
n-1 k4
k5 k6
n-1
![Page 30: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/30.jpg)
33 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort – Runtime Analysis
Sum up costs:
What is the best case runtime?
What is the worst case runtime?
( ) ++−+−+= ∑=
n
jjtcncncncnT
24321 )1()1(
![Page 31: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/31.jpg)
34 CS 473 – Lecture 1
2 4 5 6 1 3
2 4 5 6 1 3
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Question: If A[1...j] is already sorted, tj = ?
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
j
sorted
initial
< 6 j shift none
key=6
tj = 1
![Page 32: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/32.jpg)
35 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort – Best Case Runtime
Original function:
Best-case: Input array is already sorted
tj = 1 for all j
( ) ++−+−+= ∑=
n
jjtcncncncnT
24321 )1()1(
![Page 33: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/33.jpg)
36 CS 473 – Lecture 1
2 4 5 6 1 3
2 4 5 6 1 3
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Q: If A[j] is smaller than every entry in A[1..j-1], tj = ?
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
j
sorted
initial
>1 j shift all
key=1
>1 >1 >1
tj = j
![Page 34: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/34.jpg)
37 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort – Worst Case Runtime
Worst case: The input array is reverse sorted tj = j for all j
After derivation, worst case runtime:
![Page 35: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/35.jpg)
38 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Insertion Sort – Asymptotic Runtime Analysis
Insertion-Sort (A) 1. for j ← 2 to n do 2. key ← A[j]; 3. i ← j - 1; 4. while i > 0 and A[i] > key do 5. A[i+1] ← A[i]; 6. i ← i - 1; endwhile 7. A[i+1] ← key; endfor
Θ(1)
Θ(1)
Θ(1)
![Page 36: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/36.jpg)
CS473 – Lecture 1 Cevdet Aykanat - Bilkent University Computer Engineering Department
39
Asymptotic Runtime Analysis of Insertion-Sort • Worst-case (input reverse sorted)
– Inner loop is Θ(j)
• Average case (all permutations equally likely)
– Inner loop is Θ(j/2)
• Often, average case not much better than worst case
• Is this a fast sorting algorithm? – Yes, for small n. No, for large n.
( ) ( ) ( )2
22njjnT
n
j
n
jΘ=
Θ=Θ= ∑∑
==
( ) ( ) ( ) ( )2
222 njjnT
n
j
n
jΘ=Θ=Θ= ∑∑
==
![Page 37: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/37.jpg)
40 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Merge Sort
![Page 38: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/38.jpg)
41 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Merge Sort: Basic Idea
Divide Input array A
Conquer
sort this half sort this half
merge two sorted halves Combine
![Page 39: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/39.jpg)
CS473 – Lecture 1 Cevdet Aykanat - Bilkent University Computer Engineering Department
42
Merge-Sort (A, p, r) if p = r then return; else q ← (p+r)/2; (Divide)
Merge-Sort (A, p, q); (Conquer) Merge-Sort (A, q+1, r); (Conquer) Merge (A, p, q, r); (Combine) endif
• Call Merge-Sort(A,1,n) to sort A[1..n] • Recursion bottoms out when subsequences have length 1
![Page 40: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/40.jpg)
43 CS 473 – Lecture 1
Merge-Sort (A, p, r) if p = r then return else q ← (p+r)/2 Merge-Sort (A, p, q) Merge-Sort (A, q+1, r) Merge(A, p, q, r) endif
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Merge Sort: Example
5 2 4 6 1 3
p r q
2 4 5 1 3 6
p r q
1 2 3 4 5 6
![Page 41: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/41.jpg)
44 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
How to merge 2 sorted subarrays?
HW: Study the pseudo-code in the textbook (Sec. 2.3.1) What is the complexity of this step? Θ(n)
2 4 5
1 3 6
A[p..q]
A[q+1..r]
1 2 3 4 5 6
![Page 42: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/42.jpg)
45 CS 473 – Lecture 1
Merge-Sort (A, p, r) if p = r then return else q ← (p+r)/2 Merge-Sort (A, p, q) Merge-Sort (A, q+1, r) Merge(A, p, q, r) endif
Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Merge Sort: Correctness
Base case: p = r Trivially correct
Inductive hypothesis: MERGE-SORT is correct for any subarray that is a strict (smaller) subset of A[p, q].
General Case: MERGE-SORT is correct for A[p, q]. From inductive hypothesis and correctness of Merge.
![Page 43: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/43.jpg)
46 CS 473 – Lecture 1
Merge-Sort (A, p, r) if p = r then return else q ← (p+r)/2 Merge-Sort (A, p, q) Merge-Sort (A, q+1, r) Merge(A, p, q, r) endif
Cevdet Aykanat and Mustafa Ozdal
Computer Engineering Department, Bilkent University
Merge Sort: Complexity
Θ(1)
T(n)
Θ(1)
T(n/2)
T(n/2)
Θ(n)
![Page 44: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/44.jpg)
47 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Merge Sort – Recurrence
Describe a function recursively in terms of itself To analyze the performance of recursive algorithms
For merge sort:
Θ(1) if n=1 2T(n/2) + Θ(n) otherwise
T(n) =
![Page 45: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/45.jpg)
48 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
How to solve for T(n)?
Generally, we will assume T(n) = Θ(1) for sufficiently small n
The recurrence above can be rewritten as: T(n) = 2 T(n/2) + Θ(n)
How to solve this recurrence?
Θ(1) if n=1 2T(n/2) + Θ(n) otherwise
T(n) =
![Page 46: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/46.jpg)
49 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Solve Recurrence: T(n) = 2T (n/2) + Θ(n)
Θ(n)
T(n/2) T(n/2)
![Page 47: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/47.jpg)
50 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Solve Recurrence: T(n) = 2T (n/2) + Θ(n)
Θ(n)
Θ(n/2)
T(n/4) T(n/4) T(n/4) T(n/4)
T(n/2) Θ(n/2)
2x
subp
robs
each
size
ha
lved
![Page 48: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/48.jpg)
51 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Solve Recurrence: T(n) = 2T (n/2) + Θ(n)
Θ(n)
Θ(n/2) Θ(n/2)
T(n/4) T(n/4) T(n/4) T(n/4)
Θ(1) Θ(1) Θ(1) Θ(1) Θ(1) Θ(1) Θ(1) Θ(1) Θ(1)
Θ(n)
Θ(lg
n)
Θ(n)
Θ(n)
Θ(n)
Total: Θ(nlgn)
![Page 49: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/49.jpg)
52 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Merge Sort Complexity
Recurrence: T(n) = 2T(n/2) + Θ(n)
Solution to recurrence: T(n) = Θ(nlgn)
![Page 50: Lecture 1 Introduction to Analysis of Algorithmsreha/473/01-Intro.pdf · Lecture 1 . Introduction to Analysis of Algorithms . Cevdet Aykanat and Mustafa Ozdal Computer Engineering](https://reader033.vdocuments.mx/reader033/viewer/2022042310/5ed7ba52498700329150e641/html5/thumbnails/50.jpg)
53 CS 473 – Lecture 1 Cevdet Aykanat and Mustafa Ozdal Computer Engineering Department, Bilkent University
Conclusions: Insertion Sort vs. Merge Sort
Θ(nlgn) grows more slowly than Θ(n2)
Therefore Merge-Sort beats Insertion-Sort in the worst case
In practice, Merge-Sort beats Insertion-Sort for n>30 or so.