Introduction to Algorithms

• Algorithms

1. What are Algorithms? Design of Algorithms. Types of Algo-rithms.

• Sorting

1. Insertion Sort. Merge Sort. QuickSort.

Reading: Weiss, Chapter 5.

1

What are algorithms?

An algorithm is a well-defined finite set of rules that specifies a sequentialseries of elementary operations to be applied to some data called the input,producing after a finite amount of time some data called the output.

An algorithm solves some computational problem.

Algorithms (along with data structures) are the fundamental “building blocks”from which programs are constructed. Only by fully understanding them isit possible to write very effective programs.

2

Design and Analysis

An algorithmic solution to a computational problem will usually involve de-signing an algorithm, and then analysing its performance.

Design A good algorithm designer must have a thorough background knowl-edge of algorithmic techniques, but especially substantial creativity and imag-ination. Often the most obvious way of doing something is inefficient, anda better solution will require thinking “out of the box”. In this respect,algorithm design is as much an art as a science.

Analysis A good algorithm analyst must be able to carefully estimate orcalculate the resources (time, space or other) that the algorithm will usewhen running. This requires logic, care and often some mathematical ability.

The aim of this course is to give you sufficient background to understand andappreciate the issues involved in the design and analysis of algorithms.

3

Design and Analysis

In designing and analysing an algorithm we should consider the followingquestions:

1. What is the problem we have to solve?

2. Does a solution exist?

3. Can we find a solution (algorithm), and is there more than one solution?

4. Is the algorithm correct?

5. How efficient is the algorithm?

4

The importance of design

By far the most important thing in a program is the design of the algorithm.It is far more significant than the language the program is written in, or theclock speed of the computer.

To demonstrate this, we consider the problem of computing the Fibonaccinumbers.

The Fibonacci sequence is the sequence of integers starting

1, 1, 2, 3, 5, 8, 13, 21, 34, 55, . . .

which is formally defined by

F1 = F2 = 1 and Fn = Fn−1 + Fn−2.

Let us devise an algorithm to compute Fn.

5

The naive solution

The naive solution is to simply write a recursive method that directly modelsthe problem.

static int fib(int n) {

return (n<3 ? 1 : fib(n-1) + fib(n-2));

}

Is this a good algorithm/program in terms of resource usage?Timing it on a (2005) iMac gives the following results (the time is in

seconds and is for a loop calculating Fn 10000 times).

Value TimeF20 1.65F21 2.51F22 3.94F23 6.29

Value TimeF24 9.946F25 15.95F26 25.68F27 41.40

How long will it take to compute F30, F40 or F50?

6

Theoretical results

Each method call to fib() does roughly the same amount of work (just twocomparisons and one addition), so we will have a very rough estimate of thetime taken if we count how many method calls are made.

Exercise: Show the number of method calls made to fib() is 2Fn − 1.

7

Re-design the algorithm

We can easily re-design the algorithm as an iterative algorithm.

static int fib(int n) {

int f_2; /* F(i+2) */

int f_1 = 1; /* F(i+1) */

int f_0 = 1; /* F(i) */

for (int i = 1; i < n; i++) {

f_2 = f_1 + f_0; /* F(i+2) = F(i+1) + F(i) */

f_0 = f_1; /* F(i) = F(i+1) */

f_1 = f_2; /* F(i+1) = F(i+2) */

}

return f_0;

}

8

An Iterative Algorithm

An iterative algorithm gives the following times:

Value TimeF20 0.23F21 0.23F22 0.23F23 0.23

Value TimeF103 0.25F104 0.48F105 2.20F106 20.26

9

What is an algorithm?

We need to be more precise now what we mean by a problem, a solution andhow we shall judge whether or not an algorithm is a good solution to theproblem.

A computational problem consists of a general description of a question tobe answered, usually involving some free variables or parameters.

An instance of a computational problem is a specific question obtained byassigning values to the parameters of the problem.

An algorithm solves a computational problem if when presented with anyinstance of the problem as input, it produces the answer to the question asits output.

10

A computational problem: Sorting

Instance: A sequence L of comparable objects.

Question: What is the sequence obtained when the elements of L are placedin ascending order?

An instance of Sorting is simply a specific list of comparable items, such as

L = [25, 15, 11, 30, 101, 16, 21, 2]

orL = [“dog”, “cat”, “aardvark”, “possum”].

11

A computational problem: Travelling Salesman

Instance: A set of “cities” X together with a “distance” d(x, y) between anypair x, y ∈ X.

Question: What is the shortest circular route that starts and ends at a givencity and visits all the cities?

An instance of Travelling Salesman is a list of cities, together with the dis-tances between the cities, such as

X = {A,B,C,D,E, F}

d =

A B C D E FA 0 2 4 ∞ 1 3B 2 0 6 2 1 4C 4 6 0 1 2 1D ∞ 2 1 0 6 1E 1 1 2 6 0 3F 3 4 1 1 3 0

12

An algorithm for Sorting

One simple algorithm for Sorting is called Insertion Sort. The basicprinciple is that it takes a series of steps such that after the i-th step, thefirst i objects in the array are sorted. Then the (i + 1)-th step inserts the(i+1)-th element into the correct position, so that now the first i+1 elementsare sorted.

procedure INSERTION-SORT(A)for j ← 2 to length[A]

do key ← A[j]. Insert A[j] into the sorted sequence A[1 . . . j − 1]i = j − 1while i > 0 and A[i] > key

do A[i + 1]← A[i]i = i− 1

A[i + 1]← key

13

Pseudo-code

Pseudo-code provides a way of expressing algorithms in a way that is in-dependent of any programming language. It abstracts away other programdetails such as the type system and declaring variables and arrays. Somepoints to note are:

• The statement blocks are determined by indentation, rather than { and} delimiters as in Java.

• Control statements, such as if... then...else and while have similarinterpretations to Java.

• The character . is used to indicate a comment line.

14

Pseudo-code (contd)

• A statement v ← e implies that expression e should be evaluated andthe resulting value assigned to variable v. Or, in the case of v1 ← v2 ←e, to variables v1 and v2.

• All variables should be treated as local to their procedures.

• Arrays indexation is denoted by A[i] and arrays are assumed to beindexed from 1 to N (rather than 0 to N − 1, the approach followed byJava).

But to return to the insertion sort: What do we actually mean by a goodalgorithm?

15

Evaluating Algorithms

There are many considerations involved in this question.

• Correctness

1. Theoretical correctness

2. Numerical stability

• Efficiency

1. Complexity

2. Speed

16

Correctness of insertion sort

Insertion sort can be shown to be correct by a proof by induction.

procedure INSERTION-SORT(A)for j ← 2 to length[A]

do key ← A[j]. Insert A[j] into the sorted sequence A[1 . . . j − 1]i = j − 1while i > 0 and A[i] > key

do A[i + 1]← A[i]i = i− 1

A[i + 1]← key

We do the induction over the loop variable j.

The base case of the induction is: “the first element is sorted”,

and the inductive step is:“given the first j elements are sorted after the jth iteration, the first j + 1elements will be sorted after the j + 1th iteration.”

17

Proof by Induction

To show insertion sort is correct, let p(n) be the statement “after the nth

iteration, the first n + 1 elements of the array are sorted”

To show p(0) we simply note that a single element is always sorted.

Given p(i) is true for all i < n, we must show that p(n) is true:After the (n− 1)th iteration the first n elements of the array are sorted.The nth iteration takes the (n + 1)th element and inserts it after the lastelement that a) comes before it, and b) is less than it.Therefore after the nth iteration, the first n + 1 elements of the array aresorted.

18

Aside: Proof by Contradiction

Another proof technique you may need is proof by contradiction.

Here, if you want to show some property p is true, you assume p is not true,and show this assumption leads to a contradiction (something we know isnot true, like i < i).

For example, two sorted arrays of integers, L, containing exactly the sameelements, must be identical.Proof by contradiction: Suppose M 6= N are two distinct, sorted arrayscontaining the same elements. Let i be the least number such that M [i] 6=N [i]. Suppose a = M [i] < N [i]. Since M and N contain the same elements,and M [j] = N [j] for all j < i, we must have a = N [k] for some k > i. Butthen N [k] < N [i] so N is not sorted: contradiction.

19

Complexity of insertion sort

For simple programs, we can directly calculate the number of basic operationsthat will be performed:

procedure INSERTION-SORT(A)1 for j ← 2 to length[A]2 do key ← A[j]3 i = j − 14 while i > 0 and A[i] > key5 do A[i + 1]← A[i]6 i = i− 17 A[i + 1]← key

Lines 2-7 will be executed n times, lines 4-5 will be executed up to j timesfor j=1 to n.

20

Efficiency

An algorithm is efficient if it uses as few resources as possible. Typically theresources which we are interested in are

• Time, and• Space (memory)

Other resources are important in practical terms, but are outside the scopeof the design and analysis of algorithms.

In many situations there is a trade-off between time and space, in that analgorithm can be made faster if it uses more space or smaller if it takes longer.

Although a thorough analysis of an algorithm should consider both time andspace, time is considered more important, and this course will focus on timecomplexity.

21

Measuring time

How should we measure the time taken by an algorithm?

We can do it experimentally by measuring the number of seconds it takes fora program to run — this is often called benchmarking and is often seen inpopular magazines. This can be useful, but depends on many factors:

• The machine on which it is running.• The language in which it is written.• The skill of the programmer.• The instance on which the program is being run, both in termsof size and which particular instance it is.

So it is not an independent measure of the algorithm, but rather a measureof the implementation, the machine and the instance.

22

Complexity

The complexity of an algorithm is a “device-independent” measure of howmuch time it consumes. Rather than expressing the time consumed in sec-onds, we attempt to count how many “elementary operations” the algorithmperforms when presented with instances of different sizes.

The result is expressed as a function, giving the number of operations in termsof the size of the instance. This measure is not as precise as a benchmark,but much more useful for answering the kind of questions that commonlyarise:

• I want to solve a problem twice as big. How long will that takeme?• We can afford to buy a machine twice as fast? What size ofproblem can we solve in the same time?

The answers to questions like this depend on the complexity of the algorithm.

23

Asymptotic Complexity

In computer science it is most common to compare the complexity of twoalgorithms by comparing how the time taken grows with the size of the input.

This is know as asymptotic complexity, or less formally as Big O notation.

If an algorithm runs in O(n2) (big O n squared), then if we double the sizeof the input, we quadruple the time taken. (Because (2n)2 = 4n2).

If an algorithm runs in time O(2n) then if we increase the size of the inputby 1, we double the amount of time taken. (Because 2n+1 = 2.2n).

We will look at this more formally later in the unit.

24

A better sorting algorithm (in time)

procedure MERGE-SORT(A, p, r)if p < r then

q ← b(p + r)/2cMERGE-SORT(A, p, q); MERGE-SORT(A, q + 1, r); MERGE(A, p, q, r)

procedure MERGE(A, p, q, r)n1 ← q − p + 1; n2 ← r − qfor i← 1 to n1 do L[i]← A[p + i− 1]for j ← 1 to n2 do R[j]← A[q + j]i← 1; j ← 1; k ← pwhile i ≤ n1 and j ≤ n2 do

if L[i] ≤ R[j] then A[k + +]← L[i + +]else A[k + +]← R[j + +]

while i ≤ n1 do A[k + +]← L[i + +]while j ≤ n2 do A[k + +]← R[j + +]

25

A better sorting algorithm in space

Quicksort has worst case complexity worse than Merge-Sort, but it’s averagecomplexity and space usage is better than Merge-sort! (CLRS Chapter 7)

procedure QUICKSORT(A, p, r)if p < r

then q ← PARTITION(A, p, r)QUICKSORT(A, p, q − 1); QUICKSORT(A, q + 1, r)

procedure PARTITION(A, p, r)x← A[r]; i← p− 1for j ← p to r − 1

do if A[j] ≤ xthen i← i + 1

exchange A[i]↔ A[j]exchange A[i + 1]↔ A[r]return i + 1

26

Summary

1. An algorithm is a well defined set of rules for solving a computationalproblem.

2. A well designed algorithm should be efficient for problems of all sizes.

3. Algorithms are generally evaluated with respect to correctness, stabil-ity, and efficiency (for space and speed).

4. Theoretical correctness can be established using mathematical proof.

27

Summary (cont.)

5. Insertion sort is a sorting algorithm that runs in time O(n2).

6. Merge sort is a sorting algorithm that runs in time O(nlgn).

7. Quicksort is a sorting algorithm that runs in time O(n2) but is fasterthan Merge sort in the average case.

8. Polynomial algorithms (e.g. O(n), O(nlgn), O(nk)) are regarded asfeasible.

9. Exponential algorithms (e.g. O(2n), O(n)) are regarded as infeasible.

28