chapter 9 greedy technique copyright © 2007 pearson addison-wesley. all rights reserved

Chapter 9 Chapter 9

Greedy TechniqueGreedy Technique

Copyright © 2007 Pearson Addison-Wesley. All rights reserved.

9-2Copyright © 2007 Pearson Addison-Wesley. All rights reserved. A. Levitin “Introduction to the Design & Analysis of Algorithms,” 2nd ed., Ch. 9

Greedy TechniqueGreedy Technique

Constructs a solution to an Constructs a solution to an optimization problemoptimization problem piece by piece by piece through a sequence of choices that are:piece through a sequence of choices that are:

feasible, i.e. satisfying the constraintsfeasible, i.e. satisfying the constraints

locallylocally optimal ( optimal (with respect to some neighborhood definitionwith respect to some neighborhood definition))

greedy (in terms of some measure), and irrevocablegreedy (in terms of some measure), and irrevocable

For some problems, it yields a For some problems, it yields a globallyglobally optimal solution for every optimal solution for every instance. For most, does not but can be useful for fast instance. For most, does not but can be useful for fast approximations. We are mostly interested in the former case approximations. We are mostly interested in the former case in this class.in this class.

Defined by an objective function and a set of constraints


Applications of the Greedy StrategyApplications of the Greedy Strategy

Optimal solutions:Optimal solutions:

• change making for “normal” coin denominationschange making for “normal” coin denominations

• minimum spanning tree (MST)minimum spanning tree (MST)

• single-source shortest paths single-source shortest paths

• simple scheduling problemssimple scheduling problems

• Huffman codesHuffman codes

Approximations/heuristics:Approximations/heuristics:

• traveling salesman problem (TSP)traveling salesman problem (TSP)

• knapsack problemknapsack problem

• other combinatorial optimization problemsother combinatorial optimization problems


Change-Making ProblemChange-Making Problem

Given unlimited amounts of coins of denominations Given unlimited amounts of coins of denominations dd1 1 > … > > … > ddm m , ,

give change for amount give change for amount n n with the least number of coinswith the least number of coins

Example: Example: dd1 1 = 25c, = 25c, dd2 2 =10c, =10c, dd3 3 = 5c, = 5c, dd4 4 = 1c and = 1c and n = n = 48c48c

Greedy solution: Greedy solution:

Greedy solution isGreedy solution is optimal for any amount and “normal’’ set of denominationsoptimal for any amount and “normal’’ set of denominations

may not be optimal for arbitrary coin denominationsmay not be optimal for arbitrary coin denominations

<1, 2, 0, 3>

For example, d1 = 25c, d2 = 10c, d3 = 1c, and n = 30c

Ex: Prove the greedy algorithm is optimal for the above denominations.

Q: What are the objective function and constraints?


Minimum Spanning Tree (MST)Minimum Spanning Tree (MST)

Spanning treeSpanning tree of a connected graph of a connected graph GG: a connected acyclic : a connected acyclic subgraph of subgraph of G G that includes all of that includes all of GG’s vertices’s vertices

Minimum spanning treeMinimum spanning tree of a weighted, connected graph of a weighted, connected graph GG: : a spanning tree of a spanning tree of GG of the minimum total weight of the minimum total weight

Example:Example:

c

db

a6

2

4

3

1

c

db

a

2

3

1

c

db

a6

4 1


Prim’s MST algorithmPrim’s MST algorithm

Start with tree Start with tree TT11 consisting of one (any) vertex and “grow” consisting of one (any) vertex and “grow”

tree one vertex at a time to produce MST through tree one vertex at a time to produce MST through a series of a series of expanding subtrees Texpanding subtrees T11, T, T22, …, T, …, Tnn

On each iteration, On each iteration, construct Tconstruct Tii+1+1 from T from Ti i by adding vertex by adding vertex

not in not in TTi i that is that is closest to those already in closest to those already in TTii (this is a (this is a

“greedy” step!)“greedy” step!)

Stop when all vertices are includedStop when all vertices are included


ExampleExample

c

db

a4

26

1

3

c

db

a4

26

1

3

c

db

a4

26

1

3

c

db

a4

26

1

3

c

db

a4

26

1

3


Notes about Prim’s algorithmNotes about Prim’s algorithm

Proof by induction that this construction actually yields an Proof by induction that this construction actually yields an MST (MST (CLRS, Ch. 23.1CLRS, Ch. 23.1). Main property is given in the next ). Main property is given in the next page.page.

Needs priority queue for locating closest fringe vertex. The Needs priority queue for locating closest fringe vertex. The Detailed algorithm can be found in Levitin, P. 310.Detailed algorithm can be found in Levitin, P. 310.

EfficiencyEfficiency• O(O(nn22)) for weight matrix representation of graph and array for weight matrix representation of graph and array

implementation of priority queue implementation of priority queue • OO((mm log log nn) for adjacency lists representation of graph with ) for adjacency lists representation of graph with

n n vertices and vertices and m m edges and min-heap implementation of the edges and min-heap implementation of the priority queuepriority queue


The Crucial Property behind Prim’s AlgorithmThe Crucial Property behind Prim’s Algorithm

ClaimClaim: Let : Let G = (V,E)G = (V,E) be a weighted graph and be a weighted graph and (X,Y)(X,Y) be a be a partition of V (called a partition of V (called a cutcut). Suppose ). Suppose e = (x,y)e = (x,y) is an edge is an edge of of EE across the cut, where across the cut, where xx is in is in XX and and yy is in is in YY, and , and ee has has the minimum weight among all such crossing edges the minimum weight among all such crossing edges (called a (called a lightlight edge). Then there is an MST containing edge). Then there is an MST containing ee..

y

x

X

Y


Another greedy algorithm for MST: Kruskal’sAnother greedy algorithm for MST: Kruskal’s

Sort the edges in nondecreasing order of lengthsSort the edges in nondecreasing order of lengths

““Grow” tree one edge at a time to produce MST through Grow” tree one edge at a time to produce MST through a a series of expanding forests Fseries of expanding forests F11, F, F22, …, F, …, Fn-n-11

On each iteration, add the next edge on the sorted list On each iteration, add the next edge on the sorted list unless this would create a cycle. (If it would, skip the edge.)unless this would create a cycle. (If it would, skip the edge.)


ExampleExample

c

db

a

4

26

1

3

c

db

a

26

1

3

c

db

a

4

26

1

3

c

db

a

4

26

1

3

c

db

a

4

26

1

3

c

db

a

2

1

3


Notes about Kruskal’s algorithmNotes about Kruskal’s algorithm

Algorithm looks easier than Prim’s but is harder to Algorithm looks easier than Prim’s but is harder to implement (checking for cycles!)implement (checking for cycles!)

Cycle checking: a cycle is created iff added edge connects Cycle checking: a cycle is created iff added edge connects vertices in the same connected componentvertices in the same connected component

Union-find Union-find algorithms – see section 9.2algorithms – see section 9.2

Runs in Runs in O(m log m)O(m log m) time, with time, with m = |E|. m = |E|. The time is mostly The time is mostly spent on sorting.spent on sorting.


Minimum spanning tree vs. Steiner treeMinimum spanning tree vs. Steiner tree

c

db

a 1

1 1

1

c

db

a

vs s

In general, a Steiner minimal tree (SMT) can be much shorter than a minimum spanning tree (MST), but SMTs are hard to compute.


Shortest paths – Dijkstra’s algorithmShortest paths – Dijkstra’s algorithm

Single Source Shortest Paths ProblemSingle Source Shortest Paths Problem: Given a weighted : Given a weighted

connected (directed) graph G, find shortest paths from source vertex connected (directed) graph G, find shortest paths from source vertex ss

to each of the other verticesto each of the other vertices

Dijkstra’s algorithmDijkstra’s algorithm: Similar to Prim’s MST algorithm, with : Similar to Prim’s MST algorithm, with

a different way of computing numerical labels: Among verticesa different way of computing numerical labels: Among vertices

not already in the tree, it finds vertex not already in the tree, it finds vertex uu with the smallest with the smallest sumsum

ddvv + + ww((vv,,uu))

where where

vv is a vertex for which shortest path has been already found is a vertex for which shortest path has been already found on preceding iterations (such vertices form a tree rooted at on preceding iterations (such vertices form a tree rooted at ss))

ddvv is the length of the shortest path from source is the length of the shortest path from source ss to to vv

w w((vv,,uu) is the length (weight) of edge from ) is the length (weight) of edge from vv to to uu


ExampleExample

d4

Tree vertices Remaining verticesTree vertices Remaining vertices

a(-,0) b(a,3) c(-,∞) d(a,7) e(-,∞)

a

b4

e

37

62 5

c

a

b

d

4c

e

3

7 4

62 5

a

b

d

4c

e

3

7 4

62 5

a

b

d

4c

e

3

7 4

62 5

b(a,3) c(b,3+4) d(b,3+2) e(-,∞)

d(b,5) c(b,7) e(d,5+4)

c(b,7) e(d,9)

e(d,9)

d

a

b

d

4c

e

3

7 4

62 5


Notes on Dijkstra’s algorithmNotes on Dijkstra’s algorithm

Correctness can be proven by induction on the number of vertices.Correctness can be proven by induction on the number of vertices.

Doesn’t work for graphs with negative weights (whereas Floyd’s Doesn’t work for graphs with negative weights (whereas Floyd’s algorithm does, as long as there is no negative cycle). Can you find a algorithm does, as long as there is no negative cycle). Can you find a counterexamplecounterexample for Dijkstra’s algorithm? for Dijkstra’s algorithm?

Applicable to both undirected and directed graphsApplicable to both undirected and directed graphs

EfficiencyEfficiency• O(|V|O(|V|22) for graphs represented by weight matrix and array ) for graphs represented by weight matrix and array

implementation of priority queueimplementation of priority queue• O(|E|log|V|) for graphs represented by adj. lists and min-heap O(|E|log|V|) for graphs represented by adj. lists and min-heap

implementation of priority queueimplementation of priority queue

Don’t mix up Dijkstra’s algorithm with Prim’s algorithm! More details of Don’t mix up Dijkstra’s algorithm with Prim’s algorithm! More details of the algorithm are in the text and ref books.the algorithm are in the text and ref books.

We prove the invariants: (i) when a vertex is added to the tree, its correct distance is calculated and (ii) the distance is at least those of the previously added vertices.


Coding ProblemCoding Problem

CodingCoding: assignment of bit strings to alphabet characters: assignment of bit strings to alphabet characters

CodewordsCodewords: bit strings assigned for characters of alphabet: bit strings assigned for characters of alphabet

Two types of codes:Two types of codes: fixed-length encodingfixed-length encoding (e.g., ASCII) (e.g., ASCII) variable-length encodingvariable-length encoding (e,g., Morse code) (e,g., Morse code)

Prefix-free codes (or Prefix-free codes (or prefix-codesprefix-codes)): no codeword is a prefix of another codeword: no codeword is a prefix of another codeword

Problem: If frequencies of the character occurrences areProblem: If frequencies of the character occurrences are known, what is the best binary prefix-free code?known, what is the best binary prefix-free code?

It allows for efficient (online) decoding! E.g. consider the encoded string (msg) 10010110…

E.g. We can code {a,b,c,d} as {00,01,10,11} or {0,10,110,111} or {0,01,10,101}.

The one with the shortest average code length. The average code length represents on the average how many bits are required to transmit or store a character.

E.g. if P(a) = 0.4, P(b) = 0.3, P(c) = 0.2, P(d) = 0.1, then the average length of code #2 is 0.4 + 2*0.3 + 3*0.2 + 3*0.1 = 1.9 bits


Huffman codesHuffman codes

Any binary tree with edges labeled with 0’s and 1’s yields a prefix-free Any binary tree with edges labeled with 0’s and 1’s yields a prefix-free code of characters assigned to its leavescode of characters assigned to its leaves

Optimal binary tree minimizing the averageOptimal binary tree minimizing the average length of a codeword can be constructed length of a codeword can be constructed as follows:as follows:

Huffman’s algorithmHuffman’s algorithm

Initialize Initialize nn one-node trees with alphabet characters and the tree weights one-node trees with alphabet characters and the tree weights

with their frequencies.with their frequencies.

Repeat the following step Repeat the following step nn-1 times: join two binary trees with smallest -1 times: join two binary trees with smallest

weights into one (as left and right subtrees) and make its weight equal the weights into one (as left and right subtrees) and make its weight equal the

sum of the weights of the two trees.sum of the weights of the two trees.

Mark edges leading to left and right subtrees with 0’s and 1’s, respectively.Mark edges leading to left and right subtrees with 0’s and 1’s, respectively.

0 1

10

1

represents {00, 011, 1}


ExampleExample

character Acharacter A B C D B C D __frequency 0.35 0.1 0.2 0.2 0.15frequency 0.35 0.1 0.2 0.2 0.15

codeword 11 100 00 01 101codeword 11 100 00 01 101

average bits per character: 2.25average bits per character: 2.25

for fixed-length encoding: 3for fixed-length encoding: 3

compression ratiocompression ratio: (3-2.25)/3*100% = 25%: (3-2.25)/3*100% = 25%

0.25

0.1

B

0.15

_

0.2

C

0.2

D

0.35

A

0.2

C

0.2

D

0.35

A

0.1

B

0.15

_

0.4

0.2

C

0.2

D

0.6

0.25

0.1

B

0.15

_

0.6

1.0

0 1

0.4

0.2

C

0.2

D0.25

0.1

B

0.15

_

0 1 0

0

1

1

0.25

0.1

B

0.15

_

0.35

A

0.4

0.2

C

0.2

D

0.35

A

0.35

A

chapter 9 greedy technique copyright © 2007 pearson addison-wesley. all rights reserved

Documents

example c d b

constraints b

kruskals b

b efficiency

b feasible

denominations d

prims mst algorithm

c greedy solution