cs420 lecture one problems, algorithms, decidability, tractability

CS420 lecture oneProblems, algorithms, decidability, tractability

Problems, solutions, algorithms

• In this course we study questions such as:– given a problem, how do we find an (efficient)

algorithm solving it?– How do we measure the complexity (time, space

requirements) of an algorithm.• Problem: set of instances of a question– is 2 a prime?, .., is 4 a prime?, is 2100000 a prime

• Solution: set of answers– yes, .., no, no

Algorithm• Problem: question + input(s)• Algorithm: effective procedure– mapping input to output

• effective: unambiguous, executable– Turing defined it as: "like a Turing machine"– program = effective procedure

• the size of the problem: an integer n,• # inputs (for sorting problem)• #digits of input (for prime problem)• sometimes more than one integer

Is there an algorithm for each problem?

No1. the problem needs to be effectively specified

"how many angels can dance on the head of a pin?" not effective

2. even if it is effectively specified, there is not always an algorithm to provide an answer

Ulam's problem

f(n): if (n==1) return 1 else if (odd(n)) return f(3*n+1) else return f(n/2)

Try a few inputs, does f(n) always stop?

Ulam's problem


f(1) f(2), f(4), f(8), f(2n) .... f(3) f(5) f(7) f(9)

Ulam's problem


-Nobody has ever found an n for which f does not stop-Nobody has ever found a proof: (so there can be no algorithm deciding this.)

€

∀n : f (n) stops

Halting problem is undecidable

• Given a program P an an input x will P(x) stop? we can prove: the halting problem cannot be solvedie there is no algorithm Halt(P,x) that for any

program P and input x decides whether P(x) stops.

Undecidability

A problem P(x) is undecidable, if there is no algorithm solving P for any legal x

Halting problem is undecidableAssume there is a program P(P1,D) that for any P1 and D, outputs Yes (if P1(D) halts) or No (if P1(D) loops) Then construct Q'(P2): if (P(P2,P2)) loop else halt Now execute Q'(Q') if P(Q', Q') says Q'(Q') halts it loops if P(Q', Q') says Q'(Q') loops it haltsCONTRADICTION -> P does not exist!!

Verification/equivalence undecidable

• Given a specification S and a program P, there is no algorithm that decides whether P executes according to S

• given two programs P1 and P2, there is no algorithm that decides x: P1(x)=P2(x)

• Does this mean we should not build program verifiers? €

∀

Intractability

• Suppose we have a program, – does it execute a in a reasonable time? – Eg, towers of Hanoi.

Three pegs, one with n smaller and smaller disks, move (1 disk at the time) to another peg without ever placing a larger disk on a smaller

f(n) = # moves for tower of size n Monk: before a tower of Hanoi of size 100 is

moved, the world will have vanished

13

Example: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one.

14


15


f(n) = f(n-1) + …

16

Recursive AlgorithmsExample: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one.

f(n) = f(n-1) + 1 + …

17


f(n) = f(n - 1) + 1 + f(n - 1) f(n) = 2f(n - 1) + 1, f(1) = 1

f(n): #moves

• f(n) = 2f(n-1) + 1, f(1)=1• f(1) = 1, f(2) = 3, f(3) = 7, f(4) = 15

• f(n) = 2n-1 How can you show that?

• Can you write an iterative Hanoi algorithm?

Was the monk right?

• 2100 moves, say 1 per second.....

How many years???

Was the monk right?

• 2100 moves, say 1 per second.....

2100 ~ 1030 ~ 1025 days ~ 3.1022 years

more than the age of the universe

Is there a better algorithm?


• Pile(n-1) must be off peg1 and on one other peg before disk n can be moved to its destination

• so (inductively) all moves are necessary

Algorithm complexity

• Measures in units of time and space• Linear Search X in dictionary D i=1; while (not at end and X!= D[i]) {i++}• We don't know if X is in D, and we don't know

where it is, so we can only give worst or average time bounds

• We don't know the time for atomic actions, so we only determine Orders of Magnitude

time complexity

• In the worst case we search all of D, so the loop body is executed n times

• In the average case the loop body is executed about n/2 times. Why?

time complexity

• In the worst case we search all of D, so the loop body is executed n times

• In the average case the loop body is executed about n/2 times. In average case analysis we sum the products of the probability of each outcome and the cost of that outcome, here

€

1/n * i =1/n i = (n(n +1)i=1

n

∑ /2) /n ≈ n /2i=1

n

∑

Units of time

• 1 microsecond ?

• 1 machine instruction?

• #code fragments that take constant time?

Units of time

• 1 microsecond ? no, too specific and machine dependent• 1 machine instruction? no, still too specific• #code fragments that take constant time? yes

unit of space

• bit?

• int?

unit of space

• bit? very detailed but sometimes necessary

• int? nicer, but dangerous: we can code a whole

program or array (or disk) in one arbitrary int, so we have to be careful with space analysis (take value ranges into account when needed)

Orders of magnitude

• If an algorithm with input size n takes less than c.n2 steps to execute, we say it is an order n squared algorithm, notation O(n2)

• In general g(n) = O(f(n)) if there is a constant c such that g(n) <= c.f(n)

for all but a finite number of values for n. In other words, there is a n0 such that

g(n) <= c.f(n) for all n>n0

Worst / average case complexity

• Worst case : maximal number of steps / bits / words an algorithm needs for inputs of size n

• Average case: the expected number of steps / bits / words an algorithm needs:

€

Pii∈I n

∑ .Ci

Pi : probability input i occurs

Ci : complexity given input i

In : all possible inputs of size n


• Is there a better algorithm than linear search?• What does that mean?


• Is there a better algorithm than linear search?• What does that mean? 2*better? better order of magnitude? better worst case? better average case?


• Is there a better algorithm than linear search?• What does that mean? 2*better? NO better order of magnitude? better worst case? better average case? usually better worst case

Binary Search• Because dictionaries are sorted BS(x,lo,hi): if (lo>hi) not found else { m = (lo+hi)/2; if (x==D[m]) found else if (x<D[m]) BS(x,lo,m-1) else BS(x,m+1,hi) }

Binary Search• Because dictionaries are sorted BS(x,lo,hi): if (lo>hi) not found else { m = (lo+hi)/2; if (x==D[m]) found else if (x<D[m]) BS(x,lo,m-1) else BS(x,m+1,hi) } why m-1 and m+1?

Binary search worst case time

• Every 'chop' takes half of the remaining possibilities away

• The number of times we can divide n>0 by 2 until it gets to 0:

log2(n)

so, worst case O(log(n)) why?

Is there an even better algorithm?

• uhhh...

Is there an even better algorithm?

• now we have to consider all possible algorithms• Lower bound of a problem– given a set of "allowed steps" • taking constant time

– lower bound is minimal worst case complexity of any algorithm, using only the allowed steps, solving it

Searching in an ordered array

• allowed steps:– comparison to /assignment of array element– arithmetic on index– some action on found or not found

eg NOT if (isElement(x,D)) ... why NOT?

Event / decision trees

• Given these steps each algorithm is associated with a set of event or decision trees:– comparison is an internal node with two children– (not) found is a leaf– other ops are branches leading to new decisions

or leaves• For some input a path through the tree to a

particular leaf is taken

event tree for linear searchcompare

foundnot found

foundfound action

comparefound

not foundfoundfound action

comparefound

not foundfoundfound action

...

foundnot found action

event tree for binary search

comparefound

not found

foundfound action

compare

found

not found

foundfound actioncomparefound

not found

foundfound action

Height of the event tree

• The event trees for any search are binary trees. Why?• Any tree associated with the search has at least n+1

leaves (n found, 1 not found)• Theorem: A binary tree with n leaves has a height of at

least log(n). Check out some cases.• There is at least one path from root to a leaf of length

at least log(n), so any search algorithm takes at least O(log(n)) time for some of its inputs. (In the next

lecture we will use big Omega for lower bounds.)

Lower bound for search

• Lower bound for search in an ordered array is O(log(n))

• There is no asymptotically better search than binary search

• But what if I know more, eg search in an array with a fixed range of values...

Upper and lower bounds

• Upper bound of a problem: – the lowest worst case complexity of any known

algorithm solving for the problem.– eg search: O(log(n)) binary search

• Lower bound of a problem– Maximum (the minimum number of steps we can

prove any algorithm will need to execute) – eg any search algorithm takes at least O(log(n))

steps

Closed problems, algorithmic gaps

• Closed problem lower bound = upper bound eg search in ordered array

• Algorithmic gap lower bound < upper bound eg NP Complete problems

NP Complete problems

• NPC: a certain class of problems with algorithmic gaps

upper bound: exponential lower bound: polynomial

Trial and error type algorithms are the only ones we have so far to find an exact solution

Some NPC problems

• TSP: Travelling Salesman given cities c1,c2,...,cn and distances between

all of these, find a minimal tour connecting all cities.

• SAT: Satisfiability given a boolean expression E with boolean

variables x1,x2,...,xn determine a truth assignment to all xi making E true

Back tracking

• Back tracking searches (walks) a state space, at each choice point it guesses a choice.

• In a leaf (no further choices) if solution found OK, else go back to last choice point and pick another move.

• A Non Deterministic algorithm is one with the ability to select the sequence of right guesses

(avoiding back track)

P vs NP

• NP is the class of problems solvable in polynomial time by a non deterministic algorithm

• P is the class of problems solvable in polynomial time by a deterministic algorithm

the $1,000,000 question: is P = NP ?

• a $1 question: is Hanoi in NP?

Coping with intractability• NP problems become intractable quickly TSP for 100 cities? How would you

enumerate all possible tours? How many?• Coping with intractability:– Approximation: Find a nearly optimal tour– Randomization: use a probabilistic algorithm using

"coin tosses" (eg prime witnesses)

cs420 lecture one problems, algorithms, decidability, tractability

Documents