CSCE 411Design and Analysis
of Algorithms
Background ReviewProf. Jennifer Welch
CSCE 411, Background Review 1
Outline Algorithm Analysis Tools Sorting Graph Algorithms
CSCE 411, Background Review 2
Algorithm Analysis Tools Big-Oh notation Basic probability
CSCE 411, Background Review 3
How to Measure Efficiency Machine-independent way:
analyze "pseudocode" version of algorithm assume idealized machine model
one instruction takes one time unit "Big-Oh" notation
order of magnitude as problem size increases Worst-case analyses
safe, often occurs most often, average case often just as bad
CSCE 411, Background Review 4
Faster Algorithm vs. Faster CPU A faster algorithm running on a slower machine will
always win for large enough instances Suppose algorithm S1 sorts n keys in 2n2 instructions Suppose computer C1 executes 1 billion instruc/sec
When n = 1 million, takes 2000 sec Suppose algorithm S2 sorts n keys in 50nlog2n
instructions Suppose computer C2 executes 10 million instruc/sec
When n = 1 million, takes 100 sec
CSCE 411, Background Review 5
Caveat No point in finding fastest algorithm
for part of the program that is not the bottleneck
If program will only be run a few times, or time is not an issue (e.g., run overnight), then no point in finding fastest algorithm
CSCE 411, Background Review 6
Review of Big-Oh Notation Measure time as a function of input size
and ignore multiplicative constants lower order terms
f(n) = O(g(n)) : f is “≤” g f(n) = Ω(g(n)) : f is “≥” g f(n) = (g(n)) : f is “=” g f(n) = o(g(n)) : f is “<” g f(n) = (g(n)) : f is “>” g
CSCE 411, Background Review 7
Big-Oh f(n) = O(g(n)) means, informally, that g is an
upper bound on f Formally: there exist c > 0, n0 > 0 s.t.
f(n) ≤ cg(n) for all n ≥ n0. To show that f(n) = O(g(n)), either find c and
n0 that satisfy the definition, or use this fact: if limn->∞ f(n)/g(n) = some nonnegative constant,
then f(n) = O(g(n)).
CSCE 411, Background Review 8
Big-Omega f(n) = Ω(g(n)) means, informally, that g is an
lower bound on f Formally: there exist c > 0, n0 > 0 s.t.
f(n) ≥ cg(n) for all n ≥ n0. To show that f(n) = Ω(g(n)), either find c and
n0 that satisfy the definition, or use this fact: if limn->∞ f(n)/g(n) = some positive constant or ∞,
then f(n) = Ω(g(n)).
CSCE 411, Background Review 9
Big-Theta f(n) = Θ(g(n)) means, informally, that g is a
tight bound on f Formally: there exist c1 > 0, c2 > 0, n0 > 0 s.t.
c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0.
To show that f(n) = Θ(g(n)), either find c1, c2, n0 that satisfy the definition, or use this fact: if limn->∞ f(n)/g(n) = some positive constant,
then f(n) = Θ(g(n)).
CSCE 411, Background Review 10
Tying Back to Algorithm Analysis Algorithm has worst case running time O(g(n))
if there exist c > 0 and n0 > 0 s.t. for all n ≥ n0, every execution (including the slowest) on an input of size n takes at most cg(n) time.
Algorithm has worst case running time Ω(g(n)) if there exist c > 0 and n0 > 0 s.t. for all n ≥ n0, at least one execution (including the slowest) on an input of size n takes at least cg(n) time.
CSCE 411, Background Review 11
More Definitions f(n) = o(g(n)) means, informally, g is a
non-tight upper bound on f limn->∞ f(n)/g(n) = 0 pronounced “little-oh”
f(n) = ω(g(n)) means, informally, g is a non-tight lower bound on f limn->∞ f(n)/g(n) = ∞ pronounced “little-omega”
CSCE 411, Background Review 12
CSCE 411, Spring 2012: Set 3 13
How To Solve Recurrences Ad hoc method:
expand several times guess the pattern can verify with proof by induction
Master theorem general formula that works if recurrence has the form
T(n) = aT(n/b) + f(n) a is number of subproblems n/b is size of each subproblem f(n) is cost of non-recursive part
CSCE 411, Spring 2012: Set 10 14
Probability Every probabilistic claim ultimately refers to
some sample space, which is a set of elementary events
Think of each elementary event as the outcome of some experiment Ex: flipping two coins gives sample space
HH, HT, TH, TT
An event is a subset of the sample space Ex: event "both coins flipped the same" is HH, TT
CSCE 411, Spring 2012: Set 10 15
Sample Spaces and Events
S
A
HH THTT
HT
CSCE 411, Spring 2012: Set 10 16
Probability Distribution A probability distribution Pr on a
sample space S is a function from events of S to real numbers s.t. Pr[A] ≥ 0 for every event A Pr[S] = 1 Pr[A U B] = Pr[A] + Pr[B] for every two
non- intersecting ("mutually exclusive") events A and B
Pr[A] is the probability of event A
CSCE 411, Spring 2012: Set 10 17
Probability DistributionUseful facts: Pr[Ø] = 0 If A B, then Pr[A] ≤ Pr[B] Pr[S — A] = 1 — Pr[A] // complement Pr[A U B] = Pr[A] + Pr[B] – Pr[A B] ≤ Pr[A] + Pr[B]
CSCE 411, Spring 2012: Set 10 18
Probability Distribution
A
B
Pr[A U B] = Pr[A] + Pr[B] – Pr[A B]
CSCE 411, Spring 2012: Set 10 19
Example Suppose Pr[HH] = Pr[HT] = Pr[TH] =
Pr[TT] = 1/4. Pr["at least one head"]
= Pr[HH U HT U TH] = Pr[HH] + Pr[HT] + Pr[TH] = 3/4. Pr["less than one head"]
= 1 — Pr["at least one head"] = 1 — 3/4 = 1/4
HH THTT
HT1/4
1/4
1/4
1/4
CSCE 411, Spring 2012: Set 10 20
Specific Probability Distribution discrete probability distribution: sample
space is finite or countably infinite Ex: flipping two coins once; flipping one coin
infinitely often uniform probability distribution: sample
space S is finite and every elementary event has the same probability, 1/|S| Ex: flipping two fair coins once
CSCE 411, Spring 2012: Set 10 21
Flipping a Fair Coin Suppose we flip a fair coin n times Each elementary event in the sample space is
one sequence of n heads and tails, describing the outcome of one "experiment"
The size of the sample space is 2n. Let A be the event "k heads and nk tails occur". Pr[A] = C(n,k)/2n.
There are C(n,k) sequences of length n in which k heads and n–k tails occur, and each has probability 1/2n.
CSCE 411, Spring 2012: Set 10 22
Example n = 5, k = 3 Event A is
HHHTT, HHTTH, HTTHH, TTHHH,HHTHT, HTHTH, THTHH,HTHHT, THHTH, THHHT
Pr[3 heads and 2 tails] = C(5,3)/25
= 10/32
CSCE 411, Spring 2012: Set 10 23
Flipping Unfair Coins Suppose we flip two coins, each of
which gives heads two-thirds of the time
What is the probability distribution on the sample space?
HH THTT
HT4/9
2/9
2/9
1/9
Pr[at least one head] = 8/9
CSCE 411, Spring 2012: Set 10 24
Independent Events Two events A and B are independent if
Pr[A B] = Pr[A]·Pr[B] I.e., probability that both A and B
occur is the product of the separate probabilities that A occurs and that B occurs.
CSCE 411, Spring 2012: Set 10 25
Independent Events ExampleIn two-coin-flip example with fair coins: A = "first coin is heads" B = "coins are different"
HH THTT
HT1/4
1/4
1/4
1/4
A B Pr[A] = 1/2Pr[B] = 1/2Pr[A B] = 1/4 = (1/2)(1/2)so A and B are independent
CSCE 411, Spring 2012: Set 10 26
Discrete Random Variables A discrete random variable X is a function
from a finite or countably infinite sample space to the real numbers.
Associates a real number with each possible outcome of an experiment
Define the event "X = v" to be the set of all the elementary events s in the sample space with X(s) = v.
Pr["X = v"] is the sum of Pr[s] over all s with X(s) = v.
CSCE 411, Spring 2012: Set 10 27
Discrete Random Variable
X=v
X=v
X=v
X=vX=v
Add up the probabilities of all the elementary events inthe orange event to get the probability that X = v
CSCE 411, Spring 2012: Set 10 28
Random Variable Example Roll two fair 6-sided dice. Sample space contains 36 elementary
events (1:1, 1:2, 1:3, 1:4, 1:5, 1:6, 2:1,…) Probability of each elementary event is 1/36 Define random variable X to be the
maximum of the two values rolled What is Pr["X = 3"]? It is 5/36, since there are 5 elementary
events with max value 3 (1:3, 2:3, 3:3, 3:2, and 3:1)
CSCE 411, Spring 2012: Set 10 29
Independent Random Variables It is common for more than one random
variable to be defined on the same sample space. E.g.: X is maximum value rolled Y is sum of the two values rolled
Two random variables X and Y are independent if for all v and w, the events "X = v" and "Y = w" are independent.
CSCE 411, Spring 2012: Set 10 30
Expected Value of a Random Variable Most common summary of a random
variable is its "average", weighted by the probabilities called expected value, or expectation, or
mean
Definition: E[X] = ∑ v Pr[X = v]
v
CSCE 411, Spring 2012: Set 10 31
Expected Value Example Consider a game in which you flip two fair coins. You get $3 for each head but lose $2 for each tail. What are your expected earnings? I.e., what is the expected value of the random
variable X, where X(HH) = 6, X(HT) = X(TH) = 1, and X(TT) = —4?
Note that no value other than 6, 1, and —4 can be taken on by X (e.g., Pr[X = 5] = 0).
E[X] = 6(1/4) + 1(1/4) + 1(1/4) + (—4)(1/4) = 1
CSCE 411, Spring 2012: Set 10 32
Properties of Expected Values E[X+Y] = E[X] + E[Y], for any two
random variables X and Y, even if they are not independent!
E[a·X] = a·E[X], for any random variable X and any constant a.
E[X·Y] = E[X]·E[Y], for any two independent random variables X and Y
CSCE 411, Spring 2012: Set 10 33
Conditional Probability Formalizes having partial knowledge about
the outcome of an experiment Example: flip two fair coins.
Probability of two heads is 1/4 Probability of two heads when you already know
that the first coin is a head is 1/2
Conditional probability of A given that B occurs, denoted Pr[A|B], is defined to be
Pr[AB]/Pr[B]
CSCE 411, Spring 2012: Set 10 34
Conditional Probability
A
B
Pr[A] = 5/12Pr[B] = 7/12Pr[AB] = 2/12Pr[A|B] = (2/12)/(7/12) = 2/7
CSCE 411, Spring 2012: Set 10 35
Conditional Probability
Definition is Pr[A|B] = Pr[AB]/Pr[B]
Equivalently, Pr[AB] = Pr[A|B]·Pr[B]
Sorting Insertion Sort Heapsort Mergesort Quicksort
CSCE 411, Background Review 36
CSCE 411, Spring 2012: Set 2 37
Insertion Sort Review How it works:
incrementally build up longer and longer prefix of the array of keys that is in sorted order
take the current key, find correct place in sorted prefix, and shift to make room to insert it
Finding the correct place relies on comparing current key to keys in sorted prefix
Worst-case running time is (n2)
CSCE 411, Spring 2012: Set 2 38
Heapsort Review How it works:
put the keys in a heap data structure repeatedly remove the min from the heap
Manipulating the heap involves comparing keys to each other
Worst-case running time is (n log n)
CSCE 411, Spring 2012: Set 2 39
Mergesort Review How it works:
split the array of keys in half recursively sort the two halves merge the two sorted halves
Merging the two sorted halves involves comparing keys to each other
Worst-case running time is (n log n)
CSCE 411, Spring 2012: Set 3 40
Mergesort Example
1 32 42 5 66
2 64 5 1 2 63
5 2 64 1 3 62
5 2 64
2 5 64
1 3
1 3
62
62
5 62 4
5 62 14 3 62
1 3 62
CSCE 411, Spring 2012: Set 3 41
Recurrence Relation for Mergesort Let T(n) be worst case time on a sequence
of n keys If n = 1, then T(n) = (1) (constant) If n > 1, then T(n) = 2 T(n/2) + (n)
two subproblems of size n/2 each that are solved recursively
(n) time to do the merge Solving recurrence gives T(n) = (n log
n)
CSCE 411, Spring 2012: Set 2 42
Quicksort Review How it works:
choose one key to be the pivot partition the array of keys into those keys
< the pivot and those ≥ the pivot recursively sort the two partitions
Partitioning the array involves comparing keys to the pivot
Worst-case running time is (n2)
Graphs Mathematical definitions Representations
adjacency list, adjacency matrix
Traversals breadth-first search, depth-first search
Minimum spanning trees Kruskal’s algorithm, Prim’s algorithm
Single-source shortest paths Dijkstra’s algorithm, Bellman-Ford algorithm
CSCE 411, Background Review 43
Graphs Directed graph G = (V,E)
V is a finite set of vertices E is the edge set, a binary relation on E (ordered pairs of
vertice) In an undirected graph, edges are unordered pairs of
vertices If (u,v) is in E, then v is adjacent to u and (u,v) is
incident from u and incident to v; v is neighbor of u in an undirected graph, u is also adjacent to v, and (u,v) is
incident on u and v Degree of a vertex is number of edges incident from
or to (or on) the vertex
CSCE 411, Spring 2012: Set 4 44
More on Graphs A path of length k from vertex u to vertex u’ is
a sequence of k+1 vertices s.t. there is an edge from each vertex to the next in the sequence In this case, u’ is reachable from u
A path is simple if no vertices are repeated A path forms a cycle if last vertex = first vertex A cycle is simple if no vertices are repeated
other than first and last
CSCE 411, Spring 2012: Set 4 45
Graph Representations So far, we have discussed graphs purely as
mathematical concepts How can we represent a graph in a
computer program? We need some data structures.
Two most common representations are: adjacency list – best for sparse graphs (few edges) adjacency matrix – best for dense graphs (lots of
edges)
CSCE 411, Spring 2012: Set 4 46
Adjacency List Representation
Given graph G = (V,E) array Adj of |V| linked lists, one for each vertex in V The adjacency list for vertex u, Adj[u], contains all
vertices v s.t. (u,v) is in E each vertex in the list might be just a string,
representing its name, or it might be some other data structure representing the vertex, or it might be a pointer to the data structure for that vertex
Requires Θ(V+E) space (memory) Requires Θ(degree(u)) time to check if (u,v) is in E
CSCE 411, Spring 2012: Set 4 47
CSCE 411, Spring 2012: Set 8 48
Adjacency List Representation
a
c d
b
e
a
b
c
d
e
b c
a d e
a d
b c e
b d
Space-efficient for sparse graphs
Adjacency Matrix Representation
Given graph G = (V,E) Number the vertices 1, 2, ..., |V| Use a |V| x |V| matrix A with (i,j)-entry of A
being 1 if (i,j) is in E 0 if (i,j) is not in E
Requires Θ(V2) space (memory) Requires Θ(1) time to check if (u,v) is in E
CSCE 411, Spring 2012: Set 4 49
CSCE 411, Spring 2012: Set 8 50
Adjacency Matrix Representation
a
c d
b
e
a
b
c
d
e
Check for an edge in constant time
a b c d e0 1 1 0 0
1 0 0 1 1
1 0 0 1 0
0 1 1 0 1
0 1 0 1 0
More on Graph Representations Read Ch 22, Sec 1 for
more about directed vs. undirected graphs how to represented weighted graphs
(some value is associated with each edge) pros and cons of adjacency list vs.
adjacency matrix
CSCE 411, Spring 2012: Set 4 51
CSCE 411, Spring 2012: Set 8 52
Graph Traversals Ways to traverse/search a graph
Visit every vertex exactly once
Breadth-First Search Depth-First Search
CSCE 411, Spring 2012: Set 8 53
Breadth First Search (BFS) Input: G = (V,E) and source s in V for each vertex v in V do
mark v as unvisited mark s as visited enq(Q,s) // FIFO queue Q while Q is not empty do
u := deq(Q) for each unvisited neighbor v of u do
mark v as visited enq(Q,v)
CSCE 411, Spring 2012: Set 8 54
BFS Example start at s and consider in alphabetical
order
a
c
d
b
s
CSCE 411, Spring 2012: Set 8 55
BFS Tree We can make a spanning tree rooted
at s by remembering the "parent" of each vertex
CSCE 411, Spring 2012: Set 8 56
Breadth First Search #2 Input: G = (V,E) and source s in V for each vertex v in V do
mark v as unvisited parent[v] := nil
mark s as visited parent[s] := s enq(Q,s) // FIFO queue Q
CSCE 411, Spring 2012: Set 8 57
Breadth First Search #2 while Q is not empty do
u := deq(Q) for each unvisited neighbor v of u do
mark v as visited parent[v] := u enq(Q,v)
CSCE 411, Spring 2012: Set 8 58
BFS Tree Example
a
c
d
b
s
CSCE 411, Spring 2012: Set 8 59
BFS Trees BFS tree is not necessarily unique for
a given graph Depends on the order in which
neighboring vertices are processed
CSCE 411, Spring 2012: Set 8 60
BFS Numbering During the breadth-first search, assign
an integer to each vertex Indicate the distance of each vertex
from the source s
CSCE 411, Spring 2012: Set 8 61
Breadth First Search #3 Input: G = (V,E) and source s in V for each vertex v in V do
mark v as unvisited parent[v] := nil d[v] := infinity
mark s as visited parent[s] := s d[s] := 0 enq(Q,s) // FIFO queue Q
CSCE 411, Spring 2012: Set 8 62
Breadth First Search #3 while Q is not empty do
u := deq(Q) for each unvisited neighbor v of u do
mark v as visited parent[v] := u d[v] := d[u] + 1 enq(Q,v)
CSCE 411, Spring 2012: Set 8 63
BFS Numbering Example
a
c
d
b
sd = 0
d = 1
d = 1
d = 2
d = 2
CSCE 411, Spring 2012: Set 8 64
Shortest Path Tree Theorem: BFS algorithm
visits all and only vertices reachable from s sets d[v] equal to the shortest path
distance from s to v, for all vertices v, and sets parent variables to form a shortest
path tree
CSCE 411, Spring 2012: Set 8 65
BFS Running Time Initialization of each vertex takes O(V) time Every vertex is enqueued once and
dequeued once, taking O(V) time When a vertex is dequeued, all its
neighbors are checked to see if they are unvisited, taking time proportional to number of neighbors of the vertex, and summing to O(E) over all iterations
Total time is O(V+E)
CSCE 411, Spring 2012: Set 8 66
Depth-First Search Input: G = (V,E) for each vertex u in V do
mark u as unvisited
for each unvisited vertex u in V do call recursiveDFS(u)
recursiveDFS(u): mark u as visited for each unvisited neighbor v of u do
call recursiveDFS(v)
CSCE 411, Spring 2012: Set 8 67
DFS Example start at a and consider in alphabetical
order a
c
d
b
e
f g h
CSCE 411, Spring 2012: Set 8 68
Disconnected Graphs What if graph is disconnected or is
directed? call DFS on several vertices to visit all
vertices purpose of second for-loop in non-recursive
wrapper a
cb
d
e
CSCE 411, Spring 2012: Set 8 69
DFS Tree Actually might be a DFS forest
(collection of trees) Keep track of parents
CSCE 411, Spring 2012: Set 8 70
Depth-First Search #2 Input: G = (V,E) for each vertex u in
V do mark u as unvisited parent[u] := nil
for each unvisited vertex u in V do parent[u] := u // a
root call recursive DFS(u)
recursiveDFS(u): mark u as visited for each unvisited
neighbor v of u do parent[v] := u call recursiveDFS(v)
CSCE 411, Spring 2012: Set 8 71
More Properties of DFS Need to keep track of more
information for each vertex: discovery time: when its recursive
call starts finish time: when its recursive call
ends
CSCE 411, Spring 2012: Set 8 72
Depth-First Search #3 Input: G = (V,E) for each vertex u in V do
mark u as unvisited parent[u] := nil
time := 0 for each unvisited
vertex u in V do parent[u] := u // a root call recursive DFS(u)
recursiveDFS(u): mark u as visited time++ disc[u] := time for each unvisited
neighbor v of u do parent[v] := u call recursiveDFS(v)
time++ fin[u] := time
CSCE 411, Spring 2012: Set 8 73
Running Time of DFS initialization takes O(V) time second for loop in non-recursive wrapper
considers each vertex, so O(V) iterations one recursive call is made for each vertex in recursive call for vertex u, all its
neighbors are checked; total time in all recursive calls is O(E)
Total time is O(V+E)
CSCE 411, Spring 2012: Set 4 74
Minimum Spanning Tree
7
1645
6 8
11
15
14
17
10
13
3
12
29
18
Given a connected undirected graph with edge weights,find subset of edges that spans all the nodes,creates no cycle, and minimizes sum of weights
CSCE 411, Spring 2012: Set 4 75
Facts About MSTs There can be many spanning trees of
a graph In fact, there can be many minimum
spanning trees of a graph But if every edge has a unique
weight, then there is a unique MST
CSCE 411, Spring 2012: Set 9 76
Generic MST Algorithm input: weighted undirected graph
G = (V,E,w) T := empty set while T is not yet a spanning tree of G
find an edge e in E s.t. T U e is a subgraph of some MST of G
add e to T return T (as MST of G)
CSCE 411, Spring 2012: Set 4 77
Kruskal's MST algorithm
7
1645
6 8
11
15
14
17
10
13
3
12
29
18
consider the edges in increasing order of weight,add in an edge iff it does not cause a cycle
CSCE 411, Spring 2012: Set 4 78
Implementing Kruskal's Alg. Sort edges by weight
efficient algorithms known How to test quickly if adding in the
next edge would cause a cycle? use disjoint set data structure, later
CSCE 411, Spring 2012: Set 9 79
Kruskal's Algorithm as a Special Case of Generic Algorithm Consider edges in increasing order of
weight Add the next edge iff it doesn't cause
a cycle At any point, T is a forest (set of
trees); eventually T is a single tree
CSCE 411, Spring 2012: Set 4 80
Another MST Algorithm Kruskal's algorithm maintains a forest
that grows until it forms a spanning tree Alternative idea is keep just one tree and
grow it until it spans all the nodes Prim's algorithm
At each iteration, choose the minimum weight outgoing edge to add greedy!
CSCE 411, Spring 2012: Set 9 81
Idea of Prim's Algorithm Instead of growing the MST as possibly
multiple trees that eventually all merge, grow the MST from a single vertex, so that there is only one tree at any point.
Also a special case of the generic algorithm: at each step, add the minimum weight edge that goes out from the tree constructed so far.
CSCE 411, Spring 2012: Set 9 82
Prim's Algorithm input: weighted undirected graph G = (V,E,w) T := empty set S := any vertex in V while |T| < |V| - 1 do
let (u,v) be a min wt. outgoing edge (u in S, v not in S) add (u,v) to T add v to S
return (S,T) (as MST of G)
CSCE 411, Spring 2012: Set 9 83
Prim's Algorithm Example
a
b
h
i
c
g
d
f
e
4
8 7
8
11
2
7 6
1 2
4 14
9
10
CSCE 411, Spring 2012: Set 9 84
Implementing Prim's Algorithm How do we find minimum weight
outgoing edge? First cut: scan all adjacency lists at
each iteration. Results in O(VE) time. Try to do better.
CSCE 411, Spring 2012: Set 9 85
Implementing Prim's Algorithm Idea: have each vertex not yet in the
tree keep track of its best (cheapest) edge to the tree constructed so far.
To find min wt. outgoing edge, find minimum among these values use a priority queue to store the best edge
info (insert and extract-min operations)
CSCE 411, Spring 2012: Set 9 86
Implementing Prim's Algorithm When a vertex v is added to T, some
other vertices might have their best edges affected, but only neighbors of v add decrease-key operation to the priority
queueu
vTi
Ti+1 w
w's best edge to Ti
check if this edge is cheaper for w
x
x's best edge to Ti
v's best edge to Ti
CSCE 411, Spring 2012: Set 9 87
Details on Prim's AlgorithmAssociate with each vertex v two fields: best-wt[v] : if v is not yet in the tree, then
it holds the min. wt. of all edges from v to a vertex in the tree. Initially infinity.
best-node[v] : if v is not yet in the tree, then it holds the name of the vertex (node) u in the tree s.t. w(v,u) is v's best-wt. Initially nil.
CSCE 411, Spring 2012: Set 9 88
Details on Prim's Algorithm input: G = (V,E,w)// initialization initialize priority queue Q to contain all
vertices, using best-wt values as keys let v0 be any vertex in V decrease-key(Q,v0,0)
// last line means change best-wt[v0] to 0 and adjust Q accordingly
CSCE 411, Spring 2012: Set 9 89
Details on Prim's Algorithm while Q is not empty do
u := extract-min(Q) // vertex w/ smallest best-wt
if u is not v0 then add (u,best-node[u]) to T for each neighbor v of u do
if v is in Q and w(u,v) < best-wt[v] then best-node[v] := u decrease-key(Q,v,w(u,v))
return (V,T) // as MST of G
CSCE 411, Spring 2012: Set 9 90
Running Time of Prim's AlgorithmDepends on priority queue implementation. Let Tins be time for insert Tdec be time for decrease-key Tex be time for extract-minThen we have |V| inserts and one decrease-key in the
initialization: O(VTins+Tdec) |V| iterations of while
one extract-min per iteration: O(VTex) total
CSCE 411, Spring 2012: Set 9 91
Running Time of Prim's Algorithm Each iteration of while includes a for
loop. Number of iterations of for loop varies,
depending on how many neighbors the current vertex has
Total number of iterations of for loop is O(E).
Each iteration of for loop: one decrease key, so O(ETdec) total
CSCE 411, Spring 2012: Set 9 92
Running Time of Prim's Algorithm O(V(Tins + Tex) + ETdec) If priority queue is implemented with
a binary heap, then Tins = Tex = Tdec = O(log V) total time is O(E log V)
(Think about how to implement decrease-key in O(log V) time.)
CSCE 411, Spring 2012: Set 9 93
Shortest Paths in a Graph Let's review two important single-
source shortest path algorithms: Dijkstra's algorithm Bellman-Ford algorithm
CSCE 411, Spring 2012: Set 9 94
Single Source Shortest Path Problem Given: directed or undirected graph G
= (V,E,w) and source vertex s in V Find: For each t in V, a path in G from
s to t with minimum weight Warning! Negative weights are a
problem:
s t4
5
CSCE 411, Spring 2012: Set 9 95
Shortest Path Tree Result of a SSSP algorithm can be
viewed as a tree rooted at the source Why not use breadth-first search? Works fine if all weights are the same:
weight of each path is (a multiple of) the number of edges in the path
Doesn't work when weights are different
CSCE 411, Spring 2012: Set 9 96
Dijkstra's SSSP Algorithm Assumes all edge weights are nonnegative Similar to Prim's MST algorithm Start with source vertex s and iteratively
construct a tree rooted at s Each vertex keeps track of tree vertex that
provides cheapest path from s (not just cheapest path from any tree vertex)
At each iteration, include the vertex whose cheapest path from s is the overall cheapest
CSCE 411, Spring 2012: Set 9 97
Prim's vs. Dijkstra's
s
5
4
1
6
Prim's MST
s
5
4
1
6
Dijkstra's SSSP
CSCE 411, Spring 2012: Set 9 98
Implementing Dijkstra's Alg. How can each vertex u keep track of its
best path from s? Keep an estimate, d[u], of shortest path
distance from s to u Use d as a key in a priority queue When u is added to the tree, check each of
u's neighbors v to see if u provides v with a cheaper path from s: compare d[v] to d[u] + w(u,v)
CSCE 411, Spring 2012: Set 9 99
Dijkstra's Algorithm input: G = (V,E,w) and source vertex
s// initialization d[s] := 0 d[v] := infinity for all other vertices v initialize priority queue Q to contain
all vertices using d values as keys
CSCE 411, Spring 2012: Set 9 100
Dijkstra's Algorithm while Q is not empty do
u := extract-min(Q) for each neighbor v of u do
if d[u] + w(u,v) < d[v] thend[v] := d[u] + w(u,v)decrease-key(Q,v,d[v])parent(v) := u
CSCE 411, Spring 2012: Set 9 101
Dijkstra's Algorithm Example
a b
d e
c
2
84 9
2
4
12
10 6 3
source is vertex a
0 1 2 3 4 5Q abcd
ebcde cde de d Ø
d[a]
0 0 0 0 0 0
d[b]
∞ 2 2 2 2 2
d[c] ∞ 12 10 10 10 10
d[d]
∞ ∞ ∞ 16 13 13
d[e]
∞ ∞ 11 11 11 11
iteration
CSCE 411, Spring 2012: Set 9 102
Running Time of Dijkstra's Alg. initialization: insert each vertex once
O(V Tins)
O(V) iterations of while loop one extract-min per iteration => O(V Tex) for loop inside while loop has variable number of
iterations…
For loop has O(E) iterations total one decrease-key per iteration => O(E Tdec)
Total is O(V (Tins + Tex) + E Tdec)
CSCE 411, Spring 2012: Set 9 103
Using Different Heap Implementations O(V(Tins + Tex) + ETdec) If priority queue is implemented with a
binary heap, then Tins = Tex = Tdec = O(log V) total time is O(E log V)
There are fancier implementations of the priority queue, such as Fibonacci heap: Tins = O(1), Tex = O(log V), Tdec = O(1) (amortized) total time is O(V log V + E)
CSCE 411, Spring 2012: Set 9 104
Using Simpler Heap Implementations O(V(Tins + Tex) + ETdec) If graph is dense, so that |E| = (V2), then it
doesn't help to make Tins and Tex to be at most O(V).
Instead, focus on making Tdec be small, say constant.
Implement priority queue with an unsorted array: Tins = O(1), Tex = O(V), Tdec = O(1) total is O(V2)
CSCE 411, Spring 2012: Set 9 105
What About Negative Edge Weights? Dijkstra's SSSP algorithm requires all
edge weights to be nonnegative even more restrictive than outlawing
negative weight cycles Bellman-Ford SSSP algorithm can
handle negative edge weights even "handles" negative weight cycles by
reporting they exist
CSCE 411, Spring 2012: Set 9 106
Bellman-Ford Idea Consider each edge (u,v) and see if u
offers v a cheaper path from s compare d[v] to d[u] + w(u,v)
Repeat this process |V| - 1 times to ensure that accurate information propgates from s, no matter what order the edges are considered in
CSCE 411, Spring 2012: Set 9 107
Bellman-Ford SSSP Algorithm input: directed or undirected graph G = (V,E,w)//initialization initialize d[v] to infinity and parent[v] to nil for all v in V
other than the source initialize d[s] to 0 and parent[s] to s// main body for i := 1 to |V| - 1 do
for each (u,v) in E do // consider in arbitrary order if d[u] + w(u,v) < d[v] then
d[v] := d[u] + w(u,v) parent[v] := u
CSCE 411, Spring 2012: Set 9 108
Bellman-Ford SSSP Algorithm// check for negative weight cycles for each (u,v) in E do
if d[u] + w(u,v) < d[v] then output "negative weight cycle exists"
CSCE 411, Spring 2012: Set 9 109
Running Time of Bellman-Ford O(V) iterations of outer for loop O(E) iterations of inner for loop O(VE) time total
CSCE 411, Spring 2012: Set 9 110
Bellman-Ford Example
s
c
a
b3
—44
2
1
process edges in order(c,b)(a,b)(c,a)(s,a)(s,c)
<board work>
CSCE 411, Spring 2012: Set 9 111
Speeding Up Bellman-Ford The previous example would have
converged faster if we had considered the edges in a different order in the for loop move outward from s
If the graph is a DAG (no cycles), we can fully exploit this idea to speed up the running time