algorithms and computational complexity: an overview · algorithms and computational complexity: an...
TRANSCRIPT
![Page 1: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/1.jpg)
1
Algorithms and Computational Complexity: an Overview
Winter 2011 Larry Ruzzo
Thanks to Paul Beame, James Lee, Kevin Wayne for some slides
![Page 2: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/2.jpg)
2
goals
Design of Algorithms – a taste design methods
common or important types of problems analysis of algorithms - efficiency
![Page 3: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/3.jpg)
goals
Complexity & intractability – a taste solving problems in principle is not enough
algorithms must be efficient
some problems have no efficient solution
NP-complete problems important & useful class of problems whose solutions (seemingly) cannot be found efficiently
3
![Page 4: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/4.jpg)
4
complexity example
Cryptography (e.g. RSA, SSL in browsers) Secret: p,q prime, say 512 bits each Public: n which equals p x q, 1024 bits
In principle there is an algorithm that given n will find p and q: ���try all 2512 possible p’s, but an astronomical number
In practice no fast algorithm known for this problem (on non-quantum computers) security of RSA depends on this fact
(and research in “quantum computing” is strongly driven by the possibility of changing this)
![Page 5: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/5.jpg)
5
algorithms versus machines
Moore’s Law and the exponential improvements in hardware...
Ex: sparse linear equations over 25 years
10 orders of magnitude improvement!
![Page 6: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/6.jpg)
107
106
105
104
103
102
101
100
Seco
nds
G.E. / CDC 3600
CDC 6600
CDC 7600
Cray 1
Cray 2
Cray 3 (Est.)
1960 1970 1980 1990 2000
Source: Sandia, via M. Schultz!
algorithms or hardware?
25 years progress solving sparse linear systems
Hardware ���alone: 4 orders of magnitude
6
G.E. = Gaussian Elimination
![Page 7: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/7.jpg)
107
106
105
104
103
102
101
100
Seco
nds
G.E. / CDC 3600
CDC 6600
CDC 7600
Cray 1
Cray 2
Cray 3 (Est.)
Sparse G.E.
Gauss-Seidel
SOR CG
1960 1970 1980 1990 2000
Source: Sandia, via M. Schultz!
algorithms or hardware?
25 years progress solving sparse linear systems
Hardware ���alone: 4 orders of magnitude
Software alone: 6 orders of magnitude
7
G.E. = Gaussian Elimination SOR = Successive OverRelaxation CG = Conjugate Gradient
![Page 8: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/8.jpg)
algorithms or hardware?
The ���N-Body ���Problem:
in 30 years��� 107 hardware��� 1010 software
8
Source: T.Quinn!
![Page 9: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/9.jpg)
9
algorithms: a definition
Procedure to accomplish a task or solve a well-specified problem
Well-specified: know what all possible inputs look like and what output looks like given them
“accomplish” via simple, well-defined steps
Ex: sorting names (via comparison)
Ex: checking for primality (via +, -, *, /, ≤)
![Page 10: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/10.jpg)
10
algorithms: a sample problem
Printed circuit-board company has a robot arm that solders components to the board
Time: proportional to total distance the arm must move from initial rest position around the board and back to the initial position
For each board design, find best order to do the soldering
![Page 11: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/11.jpg)
11
printed circuit board
![Page 12: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/12.jpg)
12
printed circuit board
![Page 13: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/13.jpg)
13
more precise problem definition
Input: Given a set S of n points in the plane Output: The shortest cycle tour that visits each point in the set S.
Better known as “TSP”
How might you solve it?
![Page 14: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/14.jpg)
14
nearest neighbor heuristic
Start at some point p0
Walk first to its ���nearest neighbor p1
Walk to the nearest unvisited neighbor p2, then nearest unvisited p3, … until all points have been visited Then walk back to p0
heuristic:���A rule of thumb, simplifica-tion, or educated guess that reduces or limits the search for solutions in domains that are difficult and poorly understood. May be good, but usually not guaranteed to give the best or fastest solution.
![Page 15: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/15.jpg)
15
nearest neighbor heuristic
p0!p1!
p6!
![Page 16: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/16.jpg)
16
an input where nn works badly
p0!
.9!1! 2!4! 8!16!
length ~ 84
![Page 17: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/17.jpg)
17
an input where nn works badly
p0!
.9!1! 2!4! 8!16!
optimal soln for this example���length ~ 64
![Page 18: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/18.jpg)
Repeatedly join the closest pair of points (such that result can still be part of a ���single loop in the end. I.e., join ���endpoints, but not points in middle, ���of path segments already created.)
How does this work on our bad example?
18 p0!
.9!1! 2!4! 8!16!
revised heuristic – closest pairs first
?
![Page 19: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/19.jpg)
19
a bad example for closest pair
1!
1.5! 1.5!
![Page 20: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/20.jpg)
20
a bad example for closest pair
1!
1.5! 1.5!
6+√10 = 9.16 !
vs !
8!
![Page 21: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/21.jpg)
21
something that works
“Brute Force Search”: For each of the n! = n(n-1)(n-2)…1 orderings of the points, check the length of the cycle; Keep the best one
![Page 22: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/22.jpg)
22
two notes
The two incorrect algorithms were greedy Often very natural & tempting ideas They make choices that look great “locally” (and never reconsider them)
When greed works, the algorithms are typically efficient BUT: often does not work - you get boxed in
Our correct alg avoids this, but is incredibly slow 20! is so large that checking one billion per second would take 2.4 billion seconds (around 70 years!) And growing: n! ~ √2 π n • (n/e)n ~ 2O(n log n)
![Page 23: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/23.jpg)
23
the morals of the story
Algorithms are important Many performance gains outstrip Moore’s law
Simple problems can be hard Factoring, TSP, many others
Simple ideas don’t always work Nearest neighbor, closest pair heuristics
Simple algorithms can be very slow Brute-force factoring, TSP
A point we hope to make: for some problems, even the best algorithms are slow
![Page 24: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/24.jpg)
my plan
A brief overview of the theory of algorithms Efficiency & asymptotic analysis Some scattered examples of simple problems where clever algorithms help
A brief overview of the theory of intractability Especially NP-complete problems
“Basics every educated CSE student should know”
24
![Page 25: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/25.jpg)
26
computational complexity
The complexity of an algorithm associates a number T(n), the worst-case time the algorithm takes, with each problem size n.
Mathematically, T: N+ → R+
i.e.,T is a function mapping positive integers (problem sizes) to positive real numbers (number of steps).
![Page 26: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/26.jpg)
27
Problem size !
Tim
e!
T(n)!
computational complexity
![Page 27: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/27.jpg)
28
computational complexity: general goals
Characterize growth rate of worst-case run time as a function of problem size, up to a constant factor Why not try to be more precise? Average-case, e.g., is hard to define, analyze Technological variations (computer, compiler, OS, …) easily 10x or more Being more precise is a ton of work A key question is “scale up”: if I can afford this today, how much longer will it take when my business is 2x larger? (E.g. today: cn2, next year: c(2n)2 = 4cn2 : 4 x longer.)
![Page 28: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/28.jpg)
29
Problem size !
Tim
e!
T(n)!
computational complexity
2n log2n!
n log2n!
![Page 29: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/29.jpg)
30
asymptotic analysis & big-O
Given two functions f and g: N→R, f(n) is O(g(n)) iff ∃ constant c > 0 so that f(n) is eventually always ≤ c g(n)
Example:
10n2-16n+100 is O(n2) (and also O(n3)…)
why?:
10n2-16n+100 ≤ 11n2 for all n ≥ 10
![Page 30: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/30.jpg)
31
polynomial vs exponential
For all r > 1 (no matter how small) ���and all d > 0, (no matter how large) ���nd = O(rn).
n100 1.01n
In short, every exponential grows faster than every polynomial!
![Page 31: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/31.jpg)
32
the complexity class P: polynomial time
P: Running time O(nd) for some constant d ��� (d is independent of the input size n)
Nice scaling property: there is a constant c s.t. doubling n, time increases only by a factor of c. ��� (E.g., c ~ 2d)
Contrast with exponential: For any constant c, there is a d such that n → n+d increases time by a factor of more than c.
(E.g., 2n vs 2n+1)
![Page 32: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/32.jpg)
33
22n
2n/10
1000n2
22n!
2n/10!
1000n2!
polynomial vs exponential growth
![Page 33: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/33.jpg)
34
why it matters
not only get very big, but do so abruptly, which likely yields erratic performance on small instances
![Page 34: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/34.jpg)
Next year's computer will be 2x faster. If I can solve problem of size n0 today, how large a problem can I solve in the same time next year?
35
Complexity Increase E.g. T=1012
O(n) n0 → 2n0 1012 → 2 x 1012
O(n2) n0 → √2 n0 106 → 1.4 x 106
O(n3) n0 → 3√2 n0 104 → 1.25 x 104
2n /10 n0 → n0+10 400 → 410 2n n0 → n0 +1 40 → 41
another view of poly vs exp
![Page 35: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/35.jpg)
complexity summary
Typical initial goal for algorithm analysis is to find an
asymptotic upper bound on worst case running time
as a function of problem size
This is rarely the last word, but often helps separate good algorithms from blatantly poor ones - concentrate on the good ones!
36
![Page 36: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/36.jpg)
why “polynomial”?
Point is not that n2000 is a nice time bound, or that the differences among n and 2n and n2 are negligible.
Rather, simple theoretical tools may not easily capture such differences, whereas exponentials are qualitatively different from polynomials, so more amenable to theoretical analysis.
“My problem is in P” is a starting point for a more detailed analysis
“My problem is not in P” may suggest that you need to shift to a more tractable variant, or otherwise readjust expectations
37
![Page 37: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/37.jpg)
algorithm design techniques
39
![Page 38: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/38.jpg)
40
algorithm design techniques
We will survey two: Later: Dynamic programming Orderly solution of many smaller sub-problems, typically non-disjoint
Can give exponential speedups compared to more brute-force approaches
Today: Divide & Conquer Reduce problem to one or more sub-problems of the same type, typically disjoint
Often gives significant, usually polynomial, speedup
![Page 39: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/39.jpg)
41
algorithm design techniques
Divide & Conquer Reduce problem to one or more sub-problems of the same type Each sub-problem’s size a fraction of the original
Subproblem’s typically disjoint Often gives significant, usually polynomial, speedup Examples:
Mergesort, Binary Search, Strassen’s Algorithm, Quicksort (roughly)
![Page 40: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/40.jpg)
D&C in a nutshell
Suppose we've already invented DumbSort, taking time n2
Try Just One Level of divide & conquer:
DumbSort(first n/2 elements)
DumbSort(last n/2 elements)
Merge results
Time: 2 (n/2)2 + n = n2/2 + n << n2
Almost twice as fast!
42
divide & conquer – the key idea
![Page 41: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/41.jpg)
43
d&c approach, cont.
Moral 1: “two halves are better than a whole” Two problems of half size are better than one full-size problem, even given O(n) overhead of recombining, since the base algorithm has super-linear complexity.
Moral 2: “If a little's good, then more's better” Two levels of D&C would be almost 4 times faster, 3 levels almost 8, etc., even though overhead is growing.
In the limit: you’ve just rediscovered mergesort.
![Page 42: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/42.jpg)
44
mergesort (review)
Mergesort: (recursively) sort 2 half-lists, then merge results.
T(n) = 2T(n/2)+cn, n≥2 T(1) = 0 Solution: O(n log n) ���
Log
n le
vels!
O(n)workperlevel!
![Page 43: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/43.jpg)
A Divide & Conquer Example: Closest Pair of Points
46
![Page 44: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/44.jpg)
closest pair of points: non-geometric version
Given n points and arbitrary distances between them, find the closest pair. (E.g., think of distance as airfare – definitely not Euclidean distance!)
Must look at all n choose 2 pairwise distances, else ���any one you didn’t check might be the shortest.
Also true for Euclidean distance in 1-2 dimensions?
(… and all the rest of the (n) edges…) 2
47
![Page 45: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/45.jpg)
closest pair of points: 1 dimensional version
Given n points on the real line, find the closest pair
Closest pair is adjacent in ordered list Time O(n log n) to sort, if needed Plus O(n) to scan adjacent pairs
Key point: do not need to calc distances between all pairs: exploit geometry + ordering
48
![Page 46: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/46.jpg)
closest pair of points. 2d, Euclidean distance: 1st try
Divide. Sub-divide region into 4 quadrants.
L
50
![Page 47: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/47.jpg)
closest pair of points: 1st try
Divide. Sub-divide region into 4 quadrants. Obstacle. Impossible to ensure n/4 points in
each piece.
L
51
![Page 48: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/48.jpg)
closest pair of points
Algorithm. Divide: draw vertical line L with ≈ n/2 points on each side.
52
L
![Page 49: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/49.jpg)
closest pair of points
Algorithm. Divide: draw vertical line L with ≈ n/2 points on each side.
Conquer: find closest pair on each side, recursively.
53
12
21
L
![Page 50: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/50.jpg)
closest pair of points
Algorithm. Divide: draw vertical line L with ≈ n/2 points on each side.
Conquer: find closest pair on each side, recursively.
Combine: find closest pair with one point in each side. Return best of 3 solutions.
54
12
21 8
L
seems ���like ���Θ(n2) ?
![Page 51: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/51.jpg)
closest pair of points
Find closest pair with one point in each side, assuming that distance < δ.
55
12
21
δ = min(12, 21)
L
![Page 52: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/52.jpg)
closest pair of points
Find closest pair with one point in each side, assuming that distance < δ.
Observation: suffices to consider points within δ of line L.
56
12
21
δ
L
δ = min(12, 21)
![Page 53: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/53.jpg)
closest pair of points
Find closest pair with one point in each side, assuming that distance < δ.
Observation: suffices to consider points within δ of line L. Almost the one-D problem again: Sort points in 2δ-strip by their y coordinate.
57
12
21
1 2
3
4 5 6
7
δ
L
δ = min(12, 21)
![Page 54: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/54.jpg)
closest pair of points
Find closest pair with one point in each side, assuming that distance < δ.
Observation: suffices to consider points within δ of line L. Almost the one-D problem again: Sort points in 2δ-strip by their y coordinate. Only check pts within 8 in sorted list!
58
12
21
1 2
3
4 5 6
7
δ
L
δ = min(12, 21)
![Page 55: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/55.jpg)
closest pair of points
Def. Let si be the point in the 2δ-strip, with the ith smallest y-coordinate.
Claim. If |i – j| > 8, then the distance between si and sj ���is > δ. Pf: No two points lie in same ���½δ-by-½δ box; only 8 boxes ���within δ 59
δ
29 30
31
28
26
25
δ
½δ
½δ
39
i
j
27
![Page 56: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/56.jpg)
closest pair of points: analysis
Number of pairwise distance calculations:
(A mostly superfluous detail: straightforward implementation gives a running time that is a factor of log n larger, due to sorting in the various subproblems. Run time can be reduced to O(n log n) also, roughly by the trick of sorting by x at the top level, and returning/merging y-sorted lists from the subcalls.
Regardless of this nuance, the big picture is the same: divide-and-conquer allows sharp speed gain over a naive n2 method.)
61
€
D(n) ≤0 n =1
2D n /2( ) + 7n n >1⎧ ⎨ ⎩
⎫ ⎬ ⎭
⇒ D(n) = O(n logn)
![Page 57: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/57.jpg)
Integer Multiplication
63
![Page 58: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/58.jpg)
integer arithmetic
Add. Given two n-digit ���integers a and b, ���compute a + b. O(n) bit operations.
Multiply. Given two n-digit ���integers a and b, ���compute a × b. ���The “grade school” method: ���Θ(n2) bit operations.
64
1
0 1 1 1
1 1 0 1 +
0 1 0 1
1 1 1
0 1 0 1
0 1 1 1
1 0 0 0
1 0 1 1 1
Add
1
1
0
0
1
1
1
0
0
1
1
1
1
0
0
1
1
1
1
0
1
0
1
0 0 0 0 0 0 0
1 0 1 0 1 0 1
1 0 1 0 1 0 1
1 0 1 0 1 0 1
1 0 1 0 1 0 1
1 0 1 0 1 0 1
1 0 0 0 0 0 0 0 0 0 0 1 0 1 1
1
0
1
1
1
1
1
0
*
Multiply
0 0 0 0 0 0 0 0
![Page 59: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/59.jpg)
integer arithmetic
Add. Given two n-digit ���integers a and b, ���compute a + b. O(n) bit operations.
Multiply. Given two n-digit ���integers a and b, ���compute a × b. ���The “grade school” method: ���Θ(n2) bit operations.
65
1
0 1 1 1
1 1 0 1 +
0 1 0 1
1 1 1
0 1 0 1
0 1 1 1
1 0 0 0
1 0 1 1 1
Add
1
1
0
0
1
1
1
0
0
1
1
1
1
0
0
1
1
1
1
0
1
0
1
0 0 0 0 0 0 0
1 0 1 0 1 0 1
1 0 1 0 1 0 1
1 0 1 0 1 0 1
1 0 1 0 1 0 1
1 0 1 0 1 0 1
1 0 0 0 0 0 0 0 0 0 0 1 0 1 1
1
0
1
1
1
1
1
0
*
Multiply
0 0 0 0 0 0 0 0
![Page 60: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/60.jpg)
divide-and-conquer multiplication: warmup
To multiply two 2-digit integers: Multiply four 1-digit integers.
Add, shift some 2-digit integers to obtain result.
Same idea works for long integers –
can split them into 4 half-sized ints
66
€
x = 10⋅ x1 + x0y = 10⋅ y1 + y0
xy = 10⋅ x1 + x0( ) 10⋅ y1 + y0( )= 100 ⋅ x1y1 + 10⋅ x1y0 + x0y1( ) + x0y0
5
2
4
3
0 4 4 1
0 1
8 0
5 1
2 1
x0⋅y0
x0⋅y1
x1⋅y0
x1⋅y1
x1 x0
y1 y0
![Page 61: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/61.jpg)
divide-and-conquer multiplication: warmup
To multiply two n-digit integers: Multiply four n/2-digit integers.
Add, shift some n/2-digit integers to obtain result.
67
€
T(n) = 4T n /2( )recursive calls
+ Θ(n)add, shift ⇒ T(n) =Θ(n2 )
€
x = 2n / 2 ⋅ x1 + x0
y = 2n / 2 ⋅ y1 + y0
xy = 2n / 2 ⋅ x1 + x0( ) 2n / 2 ⋅ y1 + y0( )= 2n ⋅ x1y1 + 2n / 2 ⋅ x1y0 + x0y1( ) + x0y0
assumes n is a power of 2
1
1
0
0
1
1
0
1
1
1
0
1
1
1
1
0
1 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0
*
1 0 0 0 0 0 1 0
1 0 0 1 0 1 0 1
1 1 0 0 0 1 0 0
1 1 0 1 1 0 1 0
x0⋅y0
x0⋅y1
x1⋅y0
x1⋅y1
x1 x0
y1 y0
![Page 62: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/62.jpg)
key trick: 2 multiplies for the price of 1
68
€
x = 2n / 2 ⋅ x1 + x0
y = 2n / 2 ⋅ y1 + y0
xy = 2n / 2 ⋅ x1 + x0( ) 2n / 2 ⋅ y1 + y0( )= 2n ⋅ x1y1 + 2n / 2 ⋅ x1y0 + x0y1( ) + x0y0
€
α = x1 + x0
β = y1 + y0
αβ = x1 + x0( ) y1 + y0( )= x1y1 + x1y0 + x0y1( ) + x0y0
x1y0 + x0y1( ) = αβ − x1y1 − x0y0
Well, ok, 4 for 3 is more accurate…
![Page 63: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/63.jpg)
Karatsuba multiplication
To multiply two n-digit integers: Add two ½n digit integers. Multiply three ½n-digit integers.
Add, subtract, and shift ½n-digit integers to obtain result.
Theorem. [Karatsuba-Ofman, 1962] Can multiply two n-digit integers in O(n1.585) bit ops.
69
€
x = 2n / 2 ⋅ x1 + x0
y = 2n / 2 ⋅ y1 + y0
xy = 2n ⋅ x1y1 + 2n / 2 ⋅ x1y0 + x0 y1( ) + x0 y0
= 2n ⋅ x1y1 + 2n / 2 ⋅ (x1 + x0 ) (y1 + y0 ) − x1y1 − x0 y0( ) + x0 y0
€
T(n) ≤ 3T n /2( )recursive calls + O(n)
add, subtract, shift
⇒ T(n) = O(n log 2 3 ) = O(n1.585 )
A B C A C
![Page 64: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/64.jpg)
multiplication – the bottom line
Naïve: Θ(n2) Karatsuba: Θ(n1.59…) Amusing exercise: generalize Karatsuba to do
5 size n/3 subproblems → Θ(n1.46…) Best known: Θ(n log n loglog n)
"Fast Fourier Transform" but mostly unused in practice (unless you need really big numbers - a billion digits of π, say)
High precision arithmetic IS important for crypto
71
![Page 65: Algorithms and Computational Complexity: an Overview · Algorithms and Computational Complexity: an Overview Winter 2011! Larry Ruzzo! Thanks to Paul Beame, James Lee, Kevin Wayne](https://reader033.vdocuments.mx/reader033/viewer/2022041922/5e6c8276d5955e2fdf062b41/html5/thumbnails/65.jpg)
d & c summary
Idea: “Two halves are better than a whole”
if the base algorithm has super-linear complexity.
“If a little's good, then more's better” repeat above, recursively
Applications: Many. Binary Search, Merge Sort, (Quicksort), Closest points, Integer multiply,…
73