computational complexity. problem complexity vs. algorithmic complexity upper bounds vs. lower...

Computational Complexity

Computational Complexity

• Problem complexity vs. algorithmic complexity

• Upper bounds vs. lower bounds

• Exact solutions vs. approximate solutions

• Deterministic vs. randomized algorithms

• Time vs. space

Problem complexity vs. Algorithmic complexity

• The computational complexity of an algorithm addresses the resources needed to run the algorithm.

• The computational complexity of a problem addresses the resources needed to solve the problem.

• These are not the same!

Upper vs. Lower bounds

• Upper Bounds (Algorithmic Theory)– What we can compute

• I can solve problem X with resources R

– Proofs are almost always to give an algorithm that meets the resource bounds

• Lower bounds– How do we show that something can’t be

done?

Exact vs. Approximate Algorithms

• Finding the best Maximum Parsimony tree is NP-hard, but we can find a tree whose maximum parsimony score is no more than twice that of the optimal – and we can do that in polynomial time.

• So approximating the optimal solution is polynomial time, but an exact solution is NP-hard.

• However, for some problems, even finding approximate solutions is NP-hard!

Deterministic vs. Randomized

• Sometimes using randomization allows you to obtain polynomial time algorithms to “solve” the problem.

• But “solving” the problem using randomness is a little subtle! (You solve the problem exactly with high probability, but not with probability 1.)

Time vs. Space

• We have mostly focused on time needed to solve a problem.

• What if we allowed exponential time but only polynomial space? Does this change what we can solve?

– The class P– Decision/Optimization/Construction problems– Reductions between problems (Karp and Cook)– NP-hard– NP-complete– The class NP– The class Co-NP

The class P

• The class “P” refers to decision problems that can be solved exactly in time that is at most polynomial in the length of the input.

• So you have to think carefully about how much space you need to represent the input. Sometimes this is tricky.

Representing the input

• Maximum Clique: the input is just a graph, so the representation uses O(n2) space, where the graph has n nodes.

• Primality: the input is an integer B, so the representation uses O(log B) space.

• Subset Sum: the input is a list of N integers, each at most B. The representation uses O(N log B) space.

Relationships between problems

• We focus on how to use algorithms for one problem to solve another problem. (Think of this as designing an algorithm by calling external subroutines or functions.)

• Key idea: polynomial reducibility

Independent SetFor graph G = (V, E), a subset S of the vertices is an independent set if there are no edges between vertices in S.

S={1,4,6} is an independent set.

S={1,4,6,7} is not an independent set.

1

3

2

6 7

4 5

Clique

For graph G = (V, E), a subset S of the vertices is a clique if every pair of vertices in S are adjacent.

S={2,3} is a clique. S={1,2,3,4} is not a clique.

1

3

2

6 7

4 5

Cliques and Independent Sets

Consider the following two problems:•Clique(G,k): Does G have a clique of size k?•IndependentSet(G,k): Does G have an independent set of size k?

Can we solve Clique(G,k) using an oracle for IndependentSet(G,k)?

Can we solve IndependentSet(G,k) using an oracle for Clique(G,k)?

Complement of a Graph• Def: Gc= (V,Ec) is the complement of

G=(V,E) if Ec has exactly the edges that E does not have.

1

3

2

6 7

4 5

1

3

2

6 7

4 5


• Lemma: Let G=(V,E) be a graph. Then S is a clique in G if and only if S is an independent set in Gc.

• Proof: – If S is a clique in G, then for all vertices u and

v in S, (u,v) is an edge in G. Hence for all vertices u,v in S, (u,v) is not an edge in Gc. Therefore if S is a clique in G, then S is an independent set in Gc.

– (The other direction is also easy.)



Can we solve Clique(G,k) using an oracle for IndependentSet(G,k)?

Can we solve IndependentSet(G,k) using an oracle for Clique(G,k)?

Algorithm to solve Clique(G,k)

Assume we have an oracle for IndependentSet(G,k), for arbitrary graphs G and integers k:

IndependentSet(G,k) =

“Yes” if G has an independent

set of size k, and

“No” otherwise

Algorithm to solve Clique(G,k)

Assume we have an oracle for IndependentSet(G,k), for arbitrary graphs G and integers k.

Given graph G=(V,E) and integer k

•Step 1: Construct Gc

•Step 2: If IndependentSet(Gc,k) = “Yes”, then return “Yes”, else return “No”

And vice-versa

Suppose we have an oracle for Clique(G,k). To solve IndependentSet(G,k), we do essentially the same algorithm:

Input: G=(V,E), integer k

•Step 1: Construct Gc.

•Step 2: If Clique(Gc,k) = “Yes”, then return “Yes”, otherwise return “No”



Can we solve Clique(G,k) using an oracle for IndependentSet(G,k)? YES

Can we solve IndependentSet(G,k) using an oracle for Clique(G,k)? YES

What we showed

• We showed that if IndependentSet(G,k) can be solved in polynomial time, then so can Clique(G,k).

• We also did this using a particular technique: we defined a function F that maps “instances” of Clique(G,k) into “instances” of IndependentSet(G,k), so that – yes-instances map to yes-instances, and

no-instances map to no-instances– F can be implemented in polynomial time

• This is called a “Karp” reduction, after Richard Karp

Karp Reductions• Function F maps Instances of Y to instances of X. • Yes-instances are mapped to yes-instances, and no-instances are mapped

to no-instances. • F is polynomial time.

F(Instances(Y))

Instances of XInstances of Y

F

Karp Reductions• Function F maps Instances of Y to instances of X. • Yes-instances are mapped to yes-instances, and no-instances are mapped to

no-instances. • F is polynomial time.

F(Instances(Y))


F

If X can be solved in polynomial time, then so can Y.

What we showed

• Clique(G,k) reduces to Independent Set(G,k)

• Independent Set (G,k) reduces to

Clique(G,k)

• Therefore, the two problems are polynomially equivalent

Cook Reductions

• Y is Polynomial Time Reducible to X– Solve problem Y with a polynomial number of

computation steps and a polynomial number of calls to a black box that solves X

– Notation: Y <P X

Another kind of reduction



• Therefore, if X can be solved in polynomial time, then so can Y.

This is called a “Cook”-reduction, after Steven Cook.

Cook reduction vs. Karp reduction

• In a Karp reduction, you are only allowed one call to the oracle. Also, yes-instances must map to yes-instances, and no-instances must map to no-instances.

• In a Cook reduction, you can do whatever you like – as long as you only call the oracle a polynomial number of times.

• In both cases, you are only allowed a polynomial amount of work.

Notation: Y <P X

• While Cook and Karp reductions are a bit different from each other, we will use the same notation for both.

• If Y reduces to X (whether by a Karp or a Cook reduction), we write:

Y <P X

Solving 2-colorability

• Suppose you have an oracle to determine if a graph has a proper 3-coloring– 3color(G) = “Yes” if G had a proper 3-coloring,

“No” otherwise

• Can you use this oracle to determine if a graph has a proper 2-coloring?

2-color(G)<P 3-color(G)

• Let G =(V,E). Construct graph G’ = (V’,E’) by– Add one vertex v*– Make v* adjacent to every vertex in V

• Lemma: G can be 2-colored if and only if G’ can be 3-colored.

• Hence, 2-color(G)<P 3-color(G)

Vertex Cover

For graph G = (V, E), a subset S of the vertices is a vertex cover if every edge in E has at least one endpoint in S.

S = {2,3,7} is a vertex cover

S = {1,2,3,4,5} is not a vertex cover

1

3

2

6 7

4 5

Independent Sets and Vertex Covers

1

3

2

6 7

4 5

1

3

2

6 7

4 5

Find an independent set S Confirm that V-S is a vertex cover

Independent Set <P Vertex Cover

• Lemma: Let G = (V,E) be a graph. Then a set S is independent iff V-S is a vertex cover.

• Proof (by contradiction). • (1) Suppose S is an independent set but V-S is not a

vertex cover. Since V-S is not a vertex cover, there is an edge (u,v) in E with neither endpoint in V-S. Hence both u and v are in S. But then S is not an independent set.

• (2) The converse is just as easy to prove.

Independent Set <P Vertex Cover

Suppose we have an oracle for VertexCover(G,k): – VertexCover(G,k) = “YES” if G has a vertex cover of

size k

How can we use this oracle to solve IndependentSet(G,k)?

Answer:•Given graph G =(V,E) and integer k, where G has n vertices, we set:

– IndependentSet(G,k) = VertexCover(G,n-k)

(Note: This is a Karp Reduction, where F maps (G,k) to (G,n-k).)

Y <P X

We have provided Karp reductions to prove

•2-colorability <P 3-colorability

•Clique <P Independent Set

•Independent Set <P VertexCover




• In both cases, you are only allowed a polynomial amount of work.

Three types of problems

• Consider the following three problems– Does graph G have a clique of size k?

(Decision problem)– What is the largest k so that graph G has a

clique of size k? (Optimization problem)

– Find the largest clique in G. (Construction problem)

• Are they polynomially equivalent?

Three types of problems

• Consider the following three problems– Does graph G have a clique of size k?

(Decision problem)– What is the largest k so that graph G has a

clique of size k? (Optimization problem)

– Find the largest clique in G. (Construction problem)

• Are they polynomially equivalent??

Decision Problem

Clique(G,k):

Does graph G have a clique of size k?

1

3

2

6

4 5

7

Optimization Problem

Max-Clique(G):

What is the largest size of any clique in G?

1

3

2

6

4 5

7

Construction Problem

Construct-Max-Clique(G):

Given G, output the largest clique in G.

1

3

2

6

4 5

7

Using oracles (again)

• Suppose you have an oracle which can answer Yes/No questions like this:

Does graph G=(V,E) have a clique of size k?

• How could you use this oracle to – Find the size of a maximum clique?– Construct a maximum clique?

Using Oracles

• To find the largest clique size, do:– For k=n down to 1, DO

• If Clique(G,k) = “Yes”, then Return(k)

• So oracles for decision (Yes/No) problems can be used to solve the optimization problem.

• What about constructing the maximum clique?

Constructing a maximum clique

• Given G=(V,E), and V={v1,v2,…,vn}, DO:

– Let S be the size of the maximum clique.– For i=1 up to n DO

• If Clique(G-{vi},S) = “Yes”, then G:=G-{vi}

– Return vertex set of G

• Note that the graph G is changing during the algorithm, with vertices removed if they are not needed in the maximum clique.

Constructing a maximum clique

• Note that we used the oracle O(n) times – at most n times to find the size S of the maximum clique, and n times to construct it after knowing S.

• Hence, the maximum clique construction problem can be solved with O(n) calls to an oracle that solves the decision problem.

Vertex Cover

For graph G = (V, E), a subset S of the vertices is a vertex cover if every edge in E has at least one endpoint in S.

S = {2,3,7} is a vertex cover

S = {1,2,3,4,5} is not a vertex cover

1

3

2

6 7

4 5

Decision Problem

VertexCover(G,k):

Does graph G have a vertex cover of size k?

1

3

2

6 7

4 5


Min-VertexCover(G):

What is the smallest size of any vertex cover in G?

1

3

2

6 7

4 5


Construct-Min-VertexCover(G):

Given G, output the minimum size vertex cover.

1

3

2

6 7

4 5


Suppose we have an oracle for VertexCover(G,k). Can we use it to construct a smallest vertex cover?

1

3

2

6 7

4 5

Constructing the minimum vertex cover

• Input: Graph G=(V,E), with V={v1,v2,…,vn}

• Step 1: Use the oracle to find S, the size of the minimum vertex cover. Set VC = {}.

• Step 2: For i=1…n DO– If G has no edges, return(VC)

– If VertexCover(G-{vi},S-1) = “YES”, then

• add vi to VC, set S=S-1, and set G=G-{vi}

• Return(VC)

Vertex-ColoringA (proper) vertex-coloring of the graph G = (V, E) is an assignment of colors (or integers) to the vertices of G so that no two adjacent nodes get the same color.

The following is a proper vertex coloring with 3 colors:

c(1)=c(4)=c(5)=c(6) = RED, c(2)=c(7) = GREEN, and c(3) = BLUE.

The following is not a proper vertex-coloring: all vertices colored RED

1

3

2

6 7

4 5

Decision Problem

VertexColor(G,k):

Does graph G have a vertex coloring with k colors?

1

3

2

6 7

4 5


Min-VertexColor(G):

Given G, find the smallest number k so that G has a k-coloring of the vertices.

1

3

2

6 7

4 5


Construct-Min-VertexColor(G):

Given G, find a k-coloring, where G does not have a (k-1)-coloring

1

3

2

6 7

4 5

Constructing the minimum coloring using the oracle

Algorithm•Step 1: Use the oracle to find S, the minimum number of colors needed to vertex-color G. Put every node in G into its own equivalence class.

•Step 2: List all pairs of vertices in G that are not adjacent, p1,p2,…,pk.

•For i=1 up to k do

– Let pi =(x,y), and let Gi = G + (x,y)

– If VertexColor(Gi,S) = “Yes”, then set G = Gi,

•Gc (the complement of G) has S components, and each component is a clique. Number these components 1…S. Color each vertex in G by the number of its component in Gc .

Using Oracles

• In each case, we showed how to solve some problem X by using a polynomial number of calls to an oracle, and a polynomial amount of other work.

• Therefore, if the oracle can be implemented to run in polynomial time, then X is in P.

Using Karp or Cook Reductions

• When we show that problem Y can be solved using an oracle for problem X, it means that if X can be solved in polynomial time, then so can Y.

• Therefore, if Y <P X and X is solvable in polynomial time, then so is Y.

Part II: NP, Co-NP, NP-hard, NP-Complete, etc.

• we’ll talk about– P– Reductions between problems– The class NP– NP-hard– NP-complete– The class Co-NP

Why focus on decision problems?

• We have shown that if a decision problem can be solved in polynomial time, then so can:– the optimization problem– The construction problem

• Also, easier mathematics

Polynomial time decision problems

These decision problems are solvable in polynomial time:

•2-colorability

•Does G have a clique of size 5?

•Does G have a vertex cover of size 5?

•Does G have an independent set of size k?

But what about these?

•3-colorability

•Does G has a clique of size k (where k is part of the input)?

•Does G have an independent set of size k (where k is part of the input)?

•Does G have a vertex cover of size k (where k is part of the input)?

Maybe some problems cannot be solved in polynomial time?

• 3-colorability

• Max Clique

• Max Independent Set

• Minimum Vertex Cover

• Travelling Salesman

How to answer this?

• Fascinating mathematical theory to address what can and cannot be solved using polynomial time

• Basic concepts:– NP (non-deterministic polynomial time)– NP-hard and NP-complete problems – Polynomial reductions between problems:

show that if a problem X can be solved in polynomial time, then so can problem Y

Cook Reductions



– Notation: Y <P X

• Therefore, if X can be solved in polynomial time, then so can Y



F(Instances(Y))


F

If X can be solved in polynomial time, then so can Y.




• In both cases, you are only allowed a polynomial amount of work, and if Y reduces to X and X is in P, then Y is in P.




• In both cases, you are only allowed a polynomial amount of work, and if Y reduces to X and Y is not in P, then X not in P.

What is NP?

• Problems solvable in “non-deterministic polynomial time” . . .

• Problems where “yes-instances” have polynomial time checkable certificates

Certificate examples

• Independent set of size k– The independent set of size k

• Hamiltonian Circuit Problem– A cycle including all of the vertices

• k-coloring a graph– Assignment of k colors to the vertices

• Composite number– A factor of the input integer

What is NP?

• Problems where “yes-instances” can be efficiently verified

– Hamiltonian Circuit, Eulerian graph, 3-colorability, 2-colorability

• Succinct certificate property (so “proof” is short)

• Notes

– a problem in NP may or may not be easy to solve

– being in NP says nothing about having a short proof for the “no-instances”.

What does “X is NP-Complete” mean?

• Hand-waving: X is one of the hardest problems in NP

• More precise definition: X is in NP and every problem in NP reduces to X.

• “X is NP-hard” means that every problem in NP reduces to X. (In other words, X can be NP-hard but not in NP.)

NP-Completeness

• A problem X is NP-complete if – X is in NP

– For every Y in NP, Y <P X (i.e., X is NP-hard)

• In other words, X is a “hardest” problem in NP

NP-Completeness

• A problem X is NP-complete if – X is in NP

– For every Y in NP, Y <P X (i.e., X is NP-hard)

• Lemma 1: If X is NP-hard and X <P Z, then Z is NP-hard

• Lemma 2: If X is NP-Complete, Z is in NP and X <P Z, then Z is NP-Complete



F(Instances(Y))


F

If Y is NP-Complete and X is in NP, then X is NP-Complete.

Can an NP-hard problem be solved in polynomial time?

• If X is NP-hard but also in P, then every problem in NP can be solved in polynomial time.

• This would mean that NP=P.

• Very few people believe this.

• Therefore, don’t try to find a polynomial time algorithm for an NP-hard problem.

History

• Jack Edmonds– Identified NP

• Steve Cook– Cook’s Theorem – the definition of NP-Completeness,

and existence of NP-Complete problems

• Dick Karp– Identified “standard” collection of NP-Complete

Problems and defined “Karp reductions”

• Leonid Levin– Independent discovery of NP-Completeness in USSR

Cook’s Theorem

Theorem: The Circuit Satisfiability Problem is NP-Complete (i.e., every problem in NP reduces to Circuit SAT).

Circuit SAT

AND

OR

AND

AND

OR OR

AND NOT OR NOT

x1 x2x3 x4 x5

AND

ANDNOT

NOT

AND

OR

NOT

AND

OR

AND

Find a satisfying assignment

Circuit SAT

• Circuit Satisfiability– Given a boolean circuit, determine if there is

an assignment of boolean values to the input to make the output true

Cook’s Theorem

Theorem: The Circuit Satisfiability Problem is NP-Complete (Circuit SAT is in NP and every problem in NP reduces to Circuit SAT).

Since there are an infinite number of problems in NP, this cannot be proven individually for each problem in NP. Instead, Cook found a general construction, for any problem in NP.

89

•Observation. All problems below are NP-complete and polynomial reduce to one another!

CIRCUIT-SAT

3-SAT

DIR-HAM-CYCLEINDEPENDENT SET

VERTEX COVER

3-SAT reduces to

INDEPENDENT SET

GRAPH 3-COLOR

HAM-CYCLE

TSP

SUBSET-SUM

SCHEDULINGPLANAR 3-COLOR

SET COVER

NP-Completeness

by definition of NP-completeness

Satisfiability

• Given a boolean formula, does there exist a truth assignment to the variables to make the expression true?

• Examples:– (If A then B) and !B– (A or B or !C) and !A and !B and C– (A iff B) and (If A then !B)– If A then !A

Conjunctive Normal Form (CNF)

• Boolean variables: x1, …, xn

• Term: xi or its negation !xi

• Clause: disjunction of terms– t1 or t2 or … tj

• Problem:– Given a collection of clauses C1, . . ., Ck, does

there exist a truth assignment that makes all the clauses true

– (x1 or !x2), (!x1 or !x3), (x2 or !x3)

3-SAT

• Each clause has exactly 3 terms

• Variables x1, . . ., xn

• Clauses C1, . . ., Ck

– Cj = (tj1 or tj2 or tj3)

• All clauses must be satisfied

• Fact: Every instance of SAT can be converted in polynomial time to an equivalent instance of 3-SAT

Populating the NP-Completeness Universe

• Circuit Sat <P 3-SAT• 3-SAT <P Independent Set• Independent Set <P Vertex Cover• 3-SAT <P Hamiltonian Circuit• Hamiltonian Circuit <P Traveling Salesman• 3-SAT <P Integer Linear Programming• 3-SAT <P Graph Coloring• 3-SAT <P Subset Sum• Subset Sum <P Scheduling with Release times

and deadlines

94

Some NP-Complete Problems

•Six basic genres of NP-complete problems and paradigmatic examples.

– Packing problems: SET-PACKING, INDEPENDENT SET.– Covering problems: SET-COVER, VERTEX-COVER.– Constraint satisfaction problems: SAT, 3-SAT.– Sequencing problems: HAMILTONIAN-CYCLE, TSP.– Partitioning problems: 3D-MATCHING 3-COLOR.– Numerical problems: SUBSET-SUM, KNAPSACK.

•Practice. Most NP problems are either known to be in P or NP-complete.

•Notable exceptions. Factoring, graph isomorphism, Nash equilibrium.

95

More Hard Computational Problems

•Aerospace engineering: optimal mesh partitioning for finite elements.

•Biology: protein folding.

•Chemical engineering: heat exchanger network synthesis.

•Civil engineering: equilibrium of urban traffic flow.

•Economics: computation of arbitrage in financial markets with friction.

•Electrical engineering: VLSI layout.

•Environmental engineering: optimal placement of contaminant sensors.

•Financial engineering: find minimum risk portfolio of given return.

•Game theory: find Nash equilibrium that maximizes social welfare.

•Genomics: phylogeny reconstruction.

•Mechanical engineering: structure of turbulence in sheared flows.

•Medicine: reconstructing 3-D shape from biplane angiocardiogram.

•Operations research: optimal resource allocation.

•Physics: partition function of 3-D Ising model in statistical mechanics.

•Politics: Shapley-Shubik voting power.

•Pop culture: Minesweeper consistency.

•Statistics: optimal experimental design.

97

Asymmetry of NP•Asymmetry of NP. We only need to have short proofs of yes-instances.

•Ex 1. SAT vs. TAUTOLOGY.– Can prove a CNF formula is satisfiable by giving such an assignment.– How could we prove that a formula is not satisfiable?

•Ex 2. HAM-CYCLE vs. NO-HAM-CYCLE.– Can prove a graph is Hamiltonian by giving a Hamiltonian cycle.– How could we prove that a graph is not Hamiltonian?

.Ex 3. 3-colorable vs. not 3-colorable- Can prove that a graph has a 3-coloring by providing it- How could we prove that a graph doesn’t have a 3-coloring?

98

NP and co-NP• NP. Decision problems for which there is a poly-time certifier.• Ex. SAT, HAM-CYCLE, COMPOSITES.

• Def. Given a decision problem X, its complement X is the same problem with the yes and no answers reversed.

• Ex. X = { 0, 1, 4, 6, 8, 9, 10, 12, 14, 15, … }•Ex. X = { 2, 3, 5, 7, 11, 13, 17, 23, 29, … }

• co-NP. Complements of decision problems in NP.• Ex. TAUTOLOGY, NO-HAM-CYCLE, PRIMES.

99

•Fundamental question. Does NP = co-NP?– Do yes instances have succinct certificates iff no instances do?– Consensus opinion: no.

•Theorem. If NP co-NP, then P NP.•Pf idea.

– P is closed under complementation.– If P = NP, then NP is closed under complementation.– In other words, NP = co-NP.– This is the contrapositive of the theorem.

NP = co-NP ?

100

Good Characterizations

•Observation. P NP co-NP.•Fundamental open question:

– Does P = NP co-NP?–Mixed opinions.

•Fact. Factoring is in NP co-NP, but not known to be in P.

if poly-time algorithm for factoring,can break RSA cryptosystem

Can we …

• Can we find a max clique in polynomial time?• Can we solve 3-SAT in polynomial time?• Can we solve Travelling Salesman in Polynomial

time?

All of these are NP-hard problems.

If we could solve any of these in polynomial time, then all can be solved in polynomial time.

What we don’t know

• P vs. NP

NP-Complete

NP

P

NP = P

If P != NP, then

NP-Complete

NP

P

Nothing NP-Complete could be Solved in polynomial time!

And there are even some problemsthat are “NP-intermediate” (e.g., graph-isomorphism)

Most people believe this.

What we don’t know

• Does P = NP?

NP-Complete

NP

P

NP = P

108

Extent and Impact of NP-Completeness

•Extent of NP-completeness. [Papadimitriou 1995] – Prime intellectual export of CS to other disciplines.– 6,000 citations per year (title, abstract, keywords).

• more than "compiler", "operating system", "database"

– Broad applicability and classification power.– "Captures vast domains of computational, scientific,

mathematical endeavors, and seems to roughly delimit what mathematicians and scientists had been aspiring to compute feasibly."

Summary

• Decision Problems• P and NP• Certificates• Polynomial time reduction

– Y <P X

• NP-Completeness Definition– Z in NP, for all Y in NP,

Y <P Z

• NP-Completeness Proof:– A is NPC, B in NP,

A <PB, then B is NPC

NP-Complete

NP

P

computational complexity. problem complexity vs. algorithmic complexity upper bounds vs. lower...

Documents

graph g

complement of g

space slide

computational complexity

independentsetg c

construct g c step

cliqueg c

arbitrary graphs g