fast fourier transform
DESCRIPTION
Fast Fourier Transform. Jean Baptiste Joseph Fourier (1768-1830). These lecture slides are adapted from CLRS. Fast Fourier Transform. Applications. Perhaps single algorithmic discovery that has had the greatest practical impact in history. - PowerPoint PPT PresentationTRANSCRIPT
Princeton University • COS 423 • Theory of Algorithms • Spring 2002 • Kevin Wayne
Fast Fourier Transform
Jean Baptiste Joseph Fourier (1768-1830)
These lecture slides are adapted from CLRS.
2
Fast Fourier Transform
Applications. Perhaps single algorithmic discovery that has had the greatest
practical impact in history. Optics, acoustics, quantum physics, telecommunications, systems
theory, signal processing, speech recognition, data compression. Progress in these areas limited by lack of fast algorithms.
History. Cooley-Tukey (1965) revolutionized all of these areas. Danielson-Lanczos (1942) efficient algorithm. Runge-König (1924) laid theoretical groundwork. Gauss (1805, 1866) describes similar algorithm. Importance not realized until advent of digital computers.
3
Polynomials: Coefficient Representation
Degree n polynomial.
Addition: O(n) ops.
Evaluation: O(n) using Horner's method.
Multiplication (convolution): O(n2).
11
2210)(
nn xaxaxaaxp
11
2210)(
nn xbxbxbbxq
1111100 )()()()()(
nnn xbaxbabaxqxp
))))(((()( 12210 nn axaxaxaxaxp
2211
0011000 )()()()()()(
nnn
j
k
jkjk xbaxbaxbababaxqxp
4
Polynomials: Point-Value Representation
Degree n polynomial. Uniquely specified by knowing p(x) at n different values of x.
)( where ,),(x,),,(x),,(x 11-n1100 kkn xpyyyy
x
y = p(x)
xj
yj
5
Polynomials: Point-Value Representation
Degree n polynomial.
Addition: O(n).
Multiplication: O(n), but need 2n points.
Evaluation: O(n2) using Lagrange's formula.
)( where ,),(x,),,(x),,(x 11-n1100 kkn xpyyyy
),(x,),,(x),,(x 111-n111000 nn zyzyzy
1
0 )(
)(
)(n
kkj
jk
kjj
k xx
xx
yxp
),(x,,),(x),,(x 12121-2n111000 nn zyzyzy
)(z where ,),(x,),,(x),,(x 11-n1100 kkn xqzzz
6
Best of Both Worlds
Can we get "fast" multiplication and evaluation?
Yes! Convert back and forth between two representations.
coefficient
Representation
O(n2)
Multiplication
O(n)
Evaluation
point-value O(n) O(n2)
FFT O(n log n) O(n log n)
1-n10
1-n10
,,b,b
a,,a,a
b
2-2n10 ,,,c cc
)(,),(),q(x
)(,),(),p(x
1-2n10
1-2n10
xqxq
xpxp
)(,),(),r(x 1-2n10 xrxr O(n)
point-value multiplication
O(n2)
coefficient multiplication
O(n log n)evaluationFFT
interpolationinverse FFT
O(n log n)
7
Converting Between Representations: Naïve Solution
Evaluation (coefficient to point-value). Given a polynomial p(x) =a0 + a2x
1 + . . . + an-1 xn-1, choose n distinct
points {x0, x1, . . . , xn-1 } and compute yk = p(xk), for each k using
Horner's method. O(n2).
Interpolation (point-value to coefficient). Given n distinct points {x0, x1, . . . , xn-1 } and yk = p(xk), compute the
coefficients {a0, a1, . . . , an-1 } by solving the following linear system
of equations. Note Vandermonde matrix is invertible iff xk are distinct.
O(n3).
1
2
1
0
1
2
1
0
11
211
12
222
11
211
10
200
1
1
1
1
nnnnnn
n
n
n
y
y
y
y
a
a
a
a
xxx
xxx
xxx
xxx
8
Fast Interpolation: Key Idea
Key idea: choose {x0, x1, . . . , xn-1 } to make computation easier!
Set xk = xj?
1
2
1
0
1
2
1
0
11
211
12
222
11
211
10
200
1
1
1
1
nnnnnn
n
n
n
y
y
y
y
a
a
a
a
xxx
xxx
xxx
xxx
9
1
2
1
0
1
2
1
0
132
132
132
132
)5()5()5(51
)17()17()17(171
)5()5()5(51
)17()17()17(171
nnn
n
n
n
y
y
y
y
a
a
a
a
Fast Interpolation: Key Idea
Key idea: choose {x0, x1, . . . , xn-1 } to make computation easier!
Set xk = xj?
Use negative numbers: set xk = -xj so that (xk )2 = (xj )
2.
– set xk = -xn/2 + k
– E = peven(x2) = a0 + a2172 + a4174 + a6176 + . . . + an-217n-2
– O = x podd(x2) = a117 + a3173 + a5175 + a7177 + . . . + an-117n - 1
– y0 = E + O, yn/2 = E - O
10
Fast Interpolation: Key Idea
Key idea: choose {x0, x1, . . . , xn-1 } to make computation easier!
Set xk = xj?
Use negative numbers: set xk = -xj so that (xk )2 = (xj )
2.
– set xk = -xn/2 + k
Use complex numbers: set x k = k where is nth root of unity.
– (xk )2 = (-xn/2 + k )
2
– (xk )4 = (-xn/4 + k )
4
– (xk )8 = (-xn/8 + k )
8
1
3
2
1
0
)1)(1()1(3)1(21
)1(3963
)1(2642
1321
1
3
2
1
0
1
1
1
1
11111
nnnnnn
n
n
n
n a
a
a
a
a
y
y
y
y
y
11
Roots of Unity
An nth root of unity is a complex number z such that zn = 1. = e 2 i / n = principal n th root of unity. e i t = cos t + i sin t. i2 = -1. There are exactly n roots of unity: k, k = 0, 1, . . . , n-1.
0 = 1
1
2 = i
3
4 = -1
5
6 = -i
7
12
Roots of Unity: Properties
L1: Let be the principal nth root of unity. If n > 0, then n/2 = -1. Proof: = e 2 i / n n/2 = e i = -1. (Euler's formula)
L2: Let n > 0 be even, and let and be the principal nth and (n/2)th roots of unity. Then (k ) 2 = k .
Proof: (k ) 2 = e (2k)2 i / n = e (k) 2 i / (n / 2) = k .
L3: Let n > 0 be even. Then, the squares of the n complex nth roots of unity are the n/2 complex (n/2) th roots of unity.
Proof: If we square all of the nth roots of unity, then each (n/2) th root is obtained exactly twice since:
– L1 k + n / 2 = - k
– thus, (k + n / 2) 2 = (k ) 2
– L2 both of these = k
k + n / 2 and k have the same square
13
Divide-and-Conquer
Given degree n polynomial p(x) = a0 + a1x1 + a2 x
2 + . . . + an-1 xn-1.
Assume n is a power of 2, and let be the principal n th root of unity. Define even and odd polynomials:
– peven(x) := a0 + a2x1 + a4x
2 + a6x3 + . . . + an-2 x
n/2 - 1
– podd(x) := a1 + a3x1 + a5x
2 + a7x3 + . . . + an-1 x
n/2 - 1
– p(x) = peven(x2) + x podd(x
2)
Reduces problem of– evaluating degree n polynomial p(x) at 0, 1, . . . , n-1
to– evaluating two degree n/2 polynomials at: (0)2, (1)2, . . . , (n-1)2 .
L3 peven(x) and podd(x) only evaluated at n/2 complex (n/2) th roots of
unity.
14
FFT Algorithm
if (n == 1) // n is a power of 2 return a0
e 2 i / n
(e0,e1,e2,...,en/2-1) FFT(n/2, a0,a2,a4,...,an-2)(d0,d1,d2,...,dn/2-1) FFT(n/2, a1,a3,a5,...,an-1)
for k = 0 to n/2 - 1 yk ek + k dk
yk+n/2 ek - k dk
return (y0,y1,y2,...,yn-1)
FFT (n, a0, a1, a2, . . . , an-1)
O(n) complex multiplies if we pre-compute k.
)log()()()2/(2)( nnOnTnOnTnT
15
Recursion Tree
a0, a1, a2, a3, a4, a5, a6, a7
a1, a3, a5, a7a0, a2, a4, a6
a3, a7a1, a5a0, a4 a2, a6
a0 a4 a2 a6 a1 a5 a3 a7
"bit-reversed" order
16
Proof of Correctness
Proof of correctness. Need to show yk = p(k ) for each k = 0, . . . , n-1, where is the principaln th root of unity.
Base case. n = 1 = 1. Algorithm returns y0 = a0 = a0 0 .
Induction step. Assume algorithm correct for n / 2.– let be the principal (n/2) th root of unity
– ek = peven(k ) = peven(2k ) by Lemma 2
– dk = podd(k ) = podd(2k ) by Lemma 2
– recall p(x) = peven(x2) + x podd(x
2)
1
0)(
n
j
jkj
k ap
)(
)(
)( 2odd
2even
k
k
kk
kkk
p
p
p
deyk
)(
)()(
)()(
)()(
2/
2odd
2/2even
2odd
2/2even
2odd
2even
2/
nk
nknknk
knkk
kknk
p
pp
pp
pp
deyk
kk
k
17
Best of Both Worlds
Can we get "fast" multiplication and evaluation?
Yes! Convert back and forth between two representations.
coefficient
Representation
O(n2)
Multiplication
O(n)
Evaluation
point-value O(n) O(n2)
FFT O(n log n) O(n log n)
1-n10
1-n10
,,b,b
a,,a,a
b
2-2n10 ,,,c cc
)(,),(),q(x
)(,),(),p(x
1-2n10
1-2n10
xqxq
xpxp
)(,),(),r(x 1-2n10 xrxr O(n)
point-value multiplication
O(n2)
coefficient multiplication
O(n log n)evaluationFFT
interpolationinverse FFT
O(n log n)
18
Forward FFT: given {a0, a1, . . . , an-1 } , compute {y0, y1, . . . , yn-1 } .
Inverse FFT: given {y0, y1, . . . , yn-1 } compute {a0, a1, . . . , an-1 }.
1
3
2
1
0
)1)(1()1(3)1(21
)1(3963
)1(2642
1321
1
3
2
1
0
1
1
1
1
11111
nnnnnn
n
n
n
n a
a
a
a
a
y
y
y
y
y
1
3
2
1
0
1
)1)(1()1(3)1(21
)1(3963
)1(2642
1321
1
3
2
1
0
1
1
1
1
11111
nnnnnn
n
n
n
n y
y
y
y
y
a
a
a
a
a
Inverse FFT
19
Great news: same algorithm as FFT, except use -1 as "principal" nth root of unity (and divide by n).
)1)(1()1(3)1(2)1(
)1(3963
)1(2642
)1(321
1
1
1
1
11111
nnnnn
n
n
n
nF
)1)(1()1(3)1(2)1(
)1(3963
)1(2642
)1(321
1
1
1
1
1
11111
1
nnnnn
n
n
n
n nF
Inverse FFT
20
Inverse FFT: Proof of Correctness
Summation lemma. Let be a primitive n th root of unity. Then
If k is a multiple of n then k = 1. Each nth root of unity k is a root of
xn - 1 = (x - 1) (1 + x + x2 + . . . + xn-1),if k 1 we have: 1 + k + k(2) + . . . + k(n-1) = 0.
Claim: Fn and Fn-1 are inverses.
otherwise0
mod01
0
nknn
j
jk
otherwise0
if1
1 1
0
)(
1
0
1
ii
n
nFF
n
j
jii
n
j
ijij
iinn
21
Inverse FFT: Algorithm
if (n == 1) // n is a power of 2 return a0
e-2i/n
(e0,e1,e2,...,en/2-1) FFT(n/2, a0,a2,a4,...,an-2)(d0,d1,d2,...,dn/2-1) FFT(n/2, a1,a3,a5,...,an-1)
for k = 0 to n/2 - 1 yk (ek + k dk ) / n
yk+n/2 (ek - k dk ) / n
return (y0,y1,y2,...,yn-1)
IFFT (n, a0, a1, a2, . . . , an-1)
22
Best of Both Worlds
Can we get "fast" multiplication and evaluation?
Yes! Convert back and forth between two representations.
coefficient
Representation
O(n2)
Multiplication
O(n)
Evaluation
point-value O(n) O(n2)
FFT O(n log n) O(n log n)
1-n10
1-n10
,,b,b
a,,a,a
b
2-2n10 ,,,c cc
)(,),(),q(x
)(,),(),p(x
1-2n10
1-2n10
xqxq
xpxp
)(,),(),r(x 1-2n10 xrxr O(n)
point-value multiplication
O(n2)
coefficient multiplication
O(n log n)evaluationFFT
interpolationinverse FFT
O(n log n)
23
Integer Arithmetic
Multiply two n-digit integers: a = an-1 . . . a1a0 and b = bn-1 . . . b1b0.
Form two degree n polynomials. Note: a = p(10), b = q(10).
Compute product using FFT in O(n log n) steps. Evaluate r(10) = a b. Problem: O(n log n) complex arithmetic steps.
Solution. Strassen (1968): carry out arithmetic to suitable precision.
– T(n) = O(n T(log n)) T(n) = O(n log n (log log n)1+ )
Schönhage-Strassen (1971): use modular arithmetic.– T(n) = O(n log n log log n)
1
0)(
n
j
jjxaxp
1
0)(
n
j
jjxbxq
)()()( xqxpxr
Princeton University • COS 423 • Theory of Algorithms • Spring 2002 • Kevin Wayne
Extra Slides
25
Polynomials: Representation and Addition
Degree n polynomial. Coefficient representation.
Point-value representation.
Addition. O(n) for either representation.
1
0)(
n
j
jjxaxp
)(that such
),(x,),,(x),,(x 11-n1100
kk
n
xpy
yyy
1
0)(
n
j
jjxbxq
)(zthat such
),(x,),,(x),,(x 11-n1100
kk
n
xq
zzz
1
0)()()()(
n
j
jjj xbaxqxpxr
),(x,),,(x),,(x 111-n111000 nn zyzyzy
26
Polynomials: Evaluation and Multiplication
Multiplication (convolution). Brute force.
O(n2).
Use 2n points.O(n).
Evaluation. Horner's method.
O(n).
Lagrange's formula.O(n2).
))))(((()( 12210 nn axaxaxaxaxp
1
0 )(
)(
)(n
kkj
jk
kjj
k xx
xx
yxp
),(x,),,(x),,(x 12121-n111000 nn zyzyzy
j
kkjkj
n
j
jj
bac
xcxqxpxr
0
22
0)()()(