cse 202: design and analysis of algorithms

CSE 202: Design and Analysis of Algorithms

Lecture 5

Instructor: Kamalika Chaudhuri

Announcements

• Kamalika’s Office hours today moved to tomorrow,12:15-1:15pm

• Homework 1 due now

• Midterm on Feb 14

Algorithm Design Paradigms

• Exhaustive Search

• Greedy Algorithms: Build a solution incrementally piece by piece

• Divide and Conquer: Divide into parts, solve each part, combine results

• Dynamic Programming: Divide into subtasks, perform subtask by size. Combine smaller subtasks to larger ones

• Hill-climbing: Start with a solution, improve it

Divide and Conquer

• Integer Multiplication

• Closest pair of points on a Plane

• Strassen’s algorithm for matrix multiplication

Integer Multiplication: The Grade School Way

[22] 1 0 1 1 0[5] 1 0 1 --------------------------------------------------- 1 0 1 1 0 0 0 0 0 0 1 0 1 1 0 ---------------------------------------------------[110] 1 1 0 1 1 1 0

How to multiply two n-bit numbers?

1.Create array of n intermediate sums2. Add up the sums

Time per addition = O(n)Total time = O(n2).

Can we do better?

A Simple Divide and Conquer

x

xL xR

=

Problem: How to multiply two n-bit numbers x and y?

y

yL yR

=

y = yL 2n/2 + yR x = xL 2n/2 + xR

A Simple Divide and Conquer

xy = (xL 2n/2 + xR)(yL 2n/2 + yR)= xLyL 2n + (xR yL + xLyR) 2n/2 + xRyR

x

xL xR

=

Problem: How to multiply two n-bit numbers x and y?

y

yL yR

=

y = yL 2n/2 + yR x = xL 2n/2 + xR

A Simple Divide and Conquer

Operations:

1. 4 multiplications of n/2 bit #s2. Shifting by n bits3. 3 additions

xy = (xL 2n/2 + xR)(yL 2n/2 + yR)= xLyL 2n + (xR yL + xLyR) 2n/2 + xRyR

x

xL xR

=

Problem: How to multiply two n-bit numbers x and y?

y

yL yR

=

y = yL 2n/2 + yR x = xL 2n/2 + xR

A Simple Divide and Conquer

Operations:

1. 4 multiplications of n/2 bit #s2. Shifting by n bits3. 3 additions

Recurrence:T(n) = 4T(n/2) + O(n)

xy = (xL 2n/2 + xR)(yL 2n/2 + yR)= xLyL 2n + (xR yL + xLyR) 2n/2 + xRyR

x

xL xR

=

Problem: How to multiply two n-bit numbers x and y?

y

yL yR

=

y = yL 2n/2 + yR x = xL 2n/2 + xR

A Simple Divide and Conquer

xy = (xL 2n/2 + xR)(yL 2n/2 + yR)= xLyL 2n + (xR yL + xLyR) 2n/2 + xRyR

Operations:

1. 4 multiplications of n/2 bit #s2. Shifting by n bits3. 3 additions

Recurrence:T(n) = 4T(n/2) + O(n)T(n) = O(n2)

What is the base case?

x

xL xR

=

Problem: How to multiply two n-bit numbers x and y?

y

yL yR

=

y = yL 2n/2 + yR x = xL 2n/2 + xR

A Better Divide and Conquer

xy = (xL 2n/2 + xR)(yL 2n/2 + yR)= xLyL 2n + (xR yL + xLyR) 2n/2 + xRyR

Need: xLyL, xRyR, (xRyL + xLyR)

Computed by 3 multiplications as:(xRyL + xLyR) = (xL + xR)(yL + yR) - xLyL - xRyR

x

xL xR

=

Problem: How to multiply two n-bit numbers x and y?

y

yL yR

=

y = yL 2n/2 + yR x = xL 2n/2 + xR

A Better Divide and Conquer

xy = (xL 2n/2 + xR)(yL 2n/2 + yR)= xLyL 2n + (xR yL + xLyR) 2n/2 + xRyR

Need: xLyL, xRyR, (xRyL + xLyR)

Computed by 3 multiplications as:(xRyL + xLyR) = (xL + xR)(yL + yR) - xLyL - xRyR

Operations:

1. 3 multiplications of n/2 bit #s2. Shifting by n bits3. 6 additions

x

xL xR

=

Problem: How to multiply two n-bit numbers x and y?

y

yL yR

=

y = yL 2n/2 + yR x = xL 2n/2 + xR

A Better Divide and Conquer

xy = (xL 2n/2 + xR)(yL 2n/2 + yR)= xLyL 2n + (xR yL + xLyR) 2n/2 + xRyR

Need: xLyL, xRyR, (xRyL + xLyR)

Computed by 3 multiplications as:(xRyL + xLyR) = (xL + xR)(yL + yR) - xLyL - xRyR

Operations:

1. 3 multiplications of n/2 bit #s2. Shifting by n bits3. 6 additions

Recurrence:T(n) = 3T(n/2) + O(n)

x

xL xR

=

Problem: How to multiply two n-bit numbers x and y?

y

yL yR

=

y = yL 2n/2 + yR x = xL 2n/2 + xR

A Better Divide and Conquer

xy = (xL 2n/2 + xR)(yL 2n/2 + yR)= xLyL 2n + (xR yL + xLyR) 2n/2 + xRyR

Operations:

1. 3 multiplications of n/2 bit #s2. Shifting by n bits3. 6 additions

Recurrence:T(n) = 3T(n/2) + O(n)T(n) = O(n1.59)

x

xL xR

=

Problem: How to multiply two n-bit numbers x and y?

y

yL yR

=

y = yL 2n/2 + yR x = xL 2n/2 + xR

Need: xLyL, xRyR, (xRyL + xLyR)

Computed by 3 multiplications as:(xRyL + xLyR) = (xL + xR)(yL + yR) - xLyL - xRyR

A Better Divide and Conquer

Operations:

1. 3 multiplications of n/2 bit #s2. Shifting by n bits3. 6 additions

Recurrence:T(n) = 3T(n/2) + O(n)T(n) = O(n1.59)

Best: O(n log n 2O(log * n)) [Furer07])

x

xL xR

=

Problem: How to multiply two n-bit numbers x and y?

y

yL yR

=

y = yL 2n/2 + yR x = xL 2n/2 + xR

Need: xLyL, xRyR, (xRyL + xLyR)

Computed by 3 multiplications as:(xRyL + xLyR) = (xL + xR)(yL + yR) - xLyL - xRyR

xy = (xL 2n/2 + xR)(yL 2n/2 + yR)= xLyL 2n + (xR yL + xLyR) 2n/2 + xRyR

Divide and Conquer

• Integer Multiplication

• Closest pair of points on a Plane

• Strassen’s algorithm for matrix multiplication

Closest Pair of Points on a Plane

Given a set P of n points on the plane, find the two points p and q in P s.t d(p, q) is minimum

p q

Closest Pair of Points on a Plane

Given a set P of n points on the plane, find the two points p and q in P s.t d(p, q) is minimum

Naive solution = O(n2)

p q

What about one dimension?

Given a set P of n points on the line, find the two points p and q in P s.t d(p, q) is minimum

p q

What about one dimension?

Given a set P of n points on the line, find the two points p and q in P s.t d(p, q) is minimum

p q

Property: The closest points are adjacent in sorted order

What about one dimension?

Given a set P of n points on the line, find the two points p and q in P s.t d(p, q) is minimum

Algorithm 11. Sort the points 2. Find a point pi in the sorted set s.t d(pi, pi+1) is minimum

p q

Property: The closest points are adjacent in sorted order

What about one dimension?

Given a set P of n points on the line, find the two points p and q in P s.t d(p, q) is minimum

Algorithm 11. Sort the points 2. Find a point pi in the sorted set s.t d(pi, pi+1) is minimum

p q

Property: The closest points are adjacent in sorted order

Running Time = O(n log n)

Does this work in 2D?

p

q

a

b (a, b) : closest in x-coordinate

(a, q) : closest in y-coordinate

(p, q) : closest

Sorting the points by x or y coordinate, and looking at adjacent pairs does not work!

A Divide and Conquer Approach

Find-Closest-Pair(Q):

1. Find the median x coordinate lx2. L = Points to the left of lx3. R = Rest of the points4. Find-closest-pair(L)5. Find-closest-pair(R)6. Combine the results

How to combine?

lx

RL

Problem Case: p in L, q in R (or vice versa)

A Divide and Conquer Approach

Find-Closest-Pair(Q):

1. Find the median x coordinate lx2. L = Points to the left of lx3. R = Rest of the points4. Find-closest-pair(L)5. Find-closest-pair(R)6. Combine the results

How to combine?

1. Naive combination

2. Can we do better?

Time = O(n2)

lx

RL

A Property of Closest Points

Let: t1 = min x, y in L d(x, y)t2 = min x, y in R d(x, y)t = min (t1, t2)

If d(a, b) < t, where a is in L, b in R, then, a and b lie within distance t of lx

RL

lx

A Property of Closest Points

Let: t1 = min x, y in L d(x, y)t2 = min x, y in R d(x, y)t = min (t1, t2)

If d(a, b) < t, where a is in L, b in R, then, a and b lie within distance t of lx

lx

RL>ta

A Property of Closest Points

Let: t1 = min x, y in L d(x, y)t2 = min x, y in R d(x, y)t = min (t1, t2)

If d(a, b) < t, where a is in L, b in R, then, a and b lie within distance t of lx

lx

RL>t

>t

a

b

A Property of Closest Points

Let: t1 = min x, y in L d(x, y)t2 = min x, y in R d(x, y)t = min (t1, t2)

If d(a, b) < t, where a is in L, b in R, then, a and b lie within distance t of lx

lx

RL>t

>t

a

b

t tWe can safely rule out all points in the grey regions!

Another Property

Let: t1 = min x, y in L d(x, y)t2 = min x, y in R d(x, y)t = min (t1, t2)Sy = all points within distance t of lx in sorted order of y coords

If d(a, b) < t, a in L, b in R, then a and b occur within 15 positions of each other in Sy

lx

t/2

t/2

t

t1L

lx

t2 R

If d(a,b) < t, a in L, b in R, a, b are > 15 positions apart

Fact: Each box in figure can contain at most one point.

then, there are >= 3 rows between a and b

Boxes

Thus, d(a, b) >= 3t/2. Contradiction!

a

b

A Divide and Conquer Approach

Find-Closest-Pair(Q):

1. Find the median x coordinate lx2. L = Points to the left of lx3. R = Rest of the points4. (a1, b1, t1) = Find-closest-pair(L)5. (a2, b2, t2) = Find-closest-pair(R)6. t = min(t1, t2)7. Combine the results

How to combine?

lx

RLS = points in Q

within distance t of Ix

Sy = sort S by y coordsFor p in Sy

Find distance from pto next 15 pointsLet (a, b) be the pair withmin such distance

If d(a, b) < t:Return (a, b, d(a, b))

Else if t1 < t2:Return (a1, b1, t1)

Else:Return (a2, b2, t2)

Property: If a in L, b in R, and if d(a, b) < t, then a and b occur within 15 positions of each other in Sy

A Divide and Conquer Approach

Find-Closest-Pair(Qx, Qy):

1. Find the median x coordinate lx2. L = Points to the left of lx3. R = Rest of the points4. (a1, b1, t1) = Find-closest-pair(Lx, Ly)5. (a2, b2, t2) = Find-closest-pair(Rx, Ry)6. t = min(t1, t2)7. Combine the results

How to combine?

S = points in Q within distance t of Ix

Sy = sort S by y coordsFor p in Sy

Find distance from pto next 15 pointsLet (a, b) be the pair withmin such distance

If d(a, b) < t:Return (a, b, d(a, b))

Else if t1 < t2:Return (a1, b1, t1)

Else:Return (a2, b2, t2)

Property: If a in L, b in R, and if d(a, b) < t, then a and b occur within 15 positions of each other in Sy

Implementing Divide + Combine:1. Maintain sorted lists of the input points:

Px = sorted by x, Py = sorted by y2. Computing Lx,Ly,Rx, Ry from Qx, Qy: O(|Q|) time2. Computing Sy from Qy : O(|S|) time3. Combination procedure: O(|Sy|) time

A Divide and Conquer Approach

Find-Closest-Pair(Qx, Qy):

1. Find the median x coordinate lx2. L = Points to the left of lx3. R = Rest of the points4. (a1, b1, t1) = Find-closest-pair(Lx, Ly)5. (a2, b2, t2) = Find-closest-pair(Rx, Ry)6. t = min(t1, t2)7. Combine the results

How to combine?

S = points in Q within distance t of Ix

Sy = sort S by y coordsFor p in Sy

Find distance from pto next 15 pointsLet (a, b) be the pair withmin such distance

If d(a, b) < t:Return (a, b, d(a, b))

Else if t1 < t2:Return (a1, b1, t1)

Else:Return (a2, b2, t2)

Property: If a in L, b in R, and if d(a, b) < t, then a and b occur within 15 positions of each other in Sy

Implementing Divide + Combine:

T(n) = 2 T(n/2) + cnT(n) = (n log n)Θ

Summary: Closest Pair of Points

Given a set P of n points on the plane, find the two points p and q in P s.t d(p, q) is minimum

Find-Closest-Pair(Qx,Qy):

1. Find the median x coordinate lx2. L = Points to the left of lx3. R = Rest of the points4. (a1, b1, t1) = Find-closest-pair(Lx, Ly)5. (a2, b2, t2) = Find-closest-pair(Rx, Ry)6. t = min(t1, t2)7. Let Sy = points within distance t of Ix sorted by y 8. For each p in Sy

Find distance from p to next 15 points in Sy

Let (a, b) be the closest such pair9. Report the closest pair out of:

(a, b), (a1, b1), (a2, b2)

Recurrence:

T(n) = 2T(n/2) + cnT(n) = O(n log n)

What is a base case?

Divide and Conquer

• Integer Multiplication

• Closest pair of points on a Plane

• Strassen’s algorithm for matrix multiplication

Matrix Multiplication

Problem: Given two n x n matrices X and Y, compute Z = XY

Zij = Xik YkjΣ

X =i

j

(i,j)

X Y Z

Naive Method: O(n3) time

A Simple Divide and Conquer

Problem: Given two n x n matrices X and Y, compute Z = XY

X =A B

C DY =

E F

G H

Z =AE + BG AF + BH

CE + DG CF + DH

Algorithm 1: 1. Compute AE, BG, CE, DG, AF, BH, CF, DH2. Compute AE + BG, CE + DG, AF + BH, CF + DH

Operations: 8 n/2 x n/2 matrix multiplications, 4 additions

Recurrence: T(n) = 8 T(n/2) + O(n2) T(n) = O(n3)

A..H: n/2 x n/2

Strassen’s Algorithm

Problem: Given two n x n matrices X and Y, compute Z = XY

X =A B

C DY =

E F

G H

Z =P5 + P4 - P2 + P6 P1 + P2

P3 + P4 P1 + P5 - P3 - P7

P1 = A(F - H)P2 = (A + B)H

P3 = (C + D)EP4 = D(G - E)

P5 = (A + D)(E + H)P6 = (B - D)(G + H)

P7 = (A - C)(E + F)

Operations: 7 n/2 x n/2 matrix multiplications, O(1) additions

Recurrence: T(n) = 7 T(n/2) + O(n2) T(n) = O(n2.81)

A..H: n/2 x n/2

Strassen’s Algorithm

Problem: Given two n x n matrices X and Y, compute Z = XY

X =A B

C DY =

E F

G H

Z =P5 + P4 - P2 + P6 P1 + P2

P3 + P4 P1 + P5 - P3 - P7

P1 = A(F - H)P2 = (A + B)H

P3 = (C + D)EP4 = D(G - E)

P5 = (A + D)(E + H)P6 = (B - D)(G + H)

P7 = (A - C)(E + F)

Operations: 7 n/2 x n/2 matrix multiplications, O(1) additionsRecurrence: T(n) = 7 T(n/2) + O(n2) T(n) = O(n2.81)

A..H: n/2 x n/2

Algorithm Design Paradigms

• Exhaustive Search

• Greedy Algorithms: Build a solution incrementally piece by piece

• Divide and Conquer: Divide into parts, solve each part, combine results

• Dynamic Programming: Divide into subtasks, perform subtask by size. Combine smaller subtasks to larger ones

• Hill-climbing: Start with a solution, improve it

function Fib1(n)if n = 1 return 1if n = 2 return 1return Fib1(n-1) + Fib1(n-2)

Dynamic Programming (DP): A Simple Example

Problem: Compute the n-th Fibonacci number

Recursive Solution Running Time:

T(n) = T(n-1) + T(n-2) + 1

function Fib1(n)if n = 1 return 1if n = 2 return 1return Fib1(n-1) + Fib1(n-2)

Dynamic Programming (DP): A Simple Example

Problem: Compute the n-th Fibonacci number

Recursive Solution Running Time:

T(n) = T(n-1) + T(n-2) + 1

Running time: O(cn)

T(n) = O(cn)

function Fib1(n)if n = 1 return 1if n = 2 return 1return Fib1(n-1) + Fib1(n-2)

Dynamic Programming (DP): A Simple Example

function Fib2(n)Create an array fib[1..n]fib[1] = 1fib[2] = 1for i = 3 to n: fib[i] = fib[i-1] + fib[i-2]return fib[n]

Problem: Compute the n-th Fibonacci number

Recursive Solution

Dynamic Programming Solution

Running Time:

T(n) = T(n-1) + T(n-2) + 1

Running time: O(cn)

T(n) = O(cn)

function Fib1(n)if n = 1 return 1if n = 2 return 1return Fib1(n-1) + Fib1(n-2)

Dynamic Programming (DP): A Simple Example

function Fib2(n)Create an array fib[1..n]fib[1] = 1fib[2] = 1for i = 3 to n: fib[i] = fib[i-1] + fib[i-2]return fib[n]

Problem: Compute the n-th Fibonacci number

Recursive Solution

Dynamic Programming Solution

Running Time:

T(n) = T(n-1) + T(n-2) + 1

Running time: O(cn)

T(n) = O(n)

Running Time:

Running time: O(n)

T(n) = O(cn)

function Fib1(n)if n = 1 return 1if n = 2 return 1return Fib1(n-1) + Fib1(n-2)

Why does DP do better?

function Fib2(n)Create an array fib[1..n]fib[1] = 1fib[2] = 1for i = 3 to n: fib[i] = fib[i-1] + fib[i-2]return fib[n]

Problem: Compute the n-th Fibonacci number

Recursive Solution

Dynamic Programming Solution

Running time: O(cn)

Running time: O(n)

F(5)

F(4)

F(2)F(3)

F(2) F(1)

F(3)

F(1)F(2)

Recursion Tree

Dynamic Programming

Main Steps:

1. Divide the problem into subtasks

2. Define the subtasks recursively (express larger subtasks in terms of smaller ones)

3. Find the right order for solving the subtasks (but do not solve them recursively!)

Dynamic Programming

Main Steps:

1. Divide the problem into subtasks: compute fib[i]

2. Define the subtasks recursively (express larger subtasks in terms of smaller ones)

3. Find the right order for solving the subtasks (i = 1,..,n)

function Fib2(n)Create an array fib[1..n]fib[1] = 1fib[2] = 1for i = 3 to n: fib[i] = fib[i-1] + fib[i-2]return fib[n]

Dynamic Programming Solution

Running time: O(n)

Dynamic Programming

• String Reconstruction

• ...