enhanced compressive sensing using iterative support...

Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments Conclusions

Enhanced Compressive Sensing usingIterative Support Detection

Yilun Wang

Department of Computational and Applied MathematicsRice University

06-22-2009

1 / 47


Acknowledgement

Contributors:

Wotao Yin

Thesis Advisors:

Wotao Yin and Yin Zhang

Committee Members:

William W. Symes

Kevin F. Kelly (ECE)

2 / 47


Outline

1 IntroductionReview: CS Reconstruction AlgorithmsAn Iterative Support Detection Algorithm (ISD)

2 Theoretical Results of ISDThe Truncated Null Space PropertyRecoverability ImprovementReconstruction Error BoundsAlgorithmic Convergence Behavior

3 Support Detection for Fast Decaying Signals

4 Numerical ExperimentsReview of Compared AlgorithmsExperiment SetupNumerical Results

5 Conclusions

3 / 47

Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview: CS Reconstruction Algorithms An Iterative Support Detection Algorithm (ISD)

Outline





5 Conclusions

4 / 47


Compressive Sensing

Assumption: x̄ ∈ Rn: unknown sparse signal, has ≤ k nonzeros(k � n).

Objective: reconstruct x̄ from m linear measurements (m� n)

b = Ax̄ , where A ∈ Rm×n.

The ideal reconstruction model: `0 minimization.

min ‖x‖0 s.t . Ax = b

Advantage: requires fewest measurements: O(k).Disadvantage: computational cost is prohibitive.

5 / 47


Practical Alternatives

Reasonably fast, but requires at least O(k log(n)) measurements

Basis pursuit methods (BP)Greedy methods

Reasonably fast, requires slightly less than O(k log(n))(empirically) measurements

Smooth `0 algorithm (SL0)

Slower, but requires much less than O(k log(n)) measurementsIterative Reweighted `1 Minimization (IRL1)Iterative Reweighted Least Squares Method (IRLS) (smallest m)

The faster the better; the less measurements.

6 / 47


Advantages of the New Algorithm

BP: replace the `0 norm by the `1 norm.

(BP) min ‖x‖1 s.t . Ax = b

ISD, the new algorithm, is a variant of BP.As fast as BP.Requires as few measurements as IRLS.

7 / 47


Algorithmic Framework

ISD (iterative support detection algorithm)

Input: A and b; output: x∗

1 Set the iteration counter k ← 0 and initialize the set of detected entries I(0) ← ∅;

2 While the stopping condition is not met, do

1 T (k) ← (I(k))C = {1, 2, . . . , n} \ I(k);2 x (k) ← solve a truncated `1 minimization problem:

x (k) = arg min ‖xT (k)‖1 s.t . Ax = b.

3 I(k+1) ← detect partial support using x (k) as the reference; e.g.,

I(k+1) = {i : |(x (k))i | > ε(k)}

where ε(k) = ‖x (k)‖∞/5k+1, for example.4 k ← k + 1.

3 x∗ ← x (k).

Note that a BP problem is solved in the first iteration

8 / 47


A Demo (1)

Test data generation:x̄ : sparse Gaussian signal with length n = 200 and sparsity levelk = 25.Number of measurements m = 60.Gaussian matrix A ∈ Rm×n.

m is too small for BP to reconstruct x̄ successfully.

9 / 47


A Demo (2)

0 50 100 150 200−2

−1.5

−1

−0.5

0

0.5

1

1.5

20−th iter. (total,det,c−det,w−det)=(25,18,14,4), Err = 4.38e−001

true signaltrue nonzerofalse nonzero

Reconstruction by BP.Reconstruction error is large.14 true nonzeros are detected.

10 / 47


A Demo (3)

0 50 100 150 200−2

−1.5

−1

−0.5

0

0.5

1

1.5

21−st iter. (total,det,c−det,w−det)=(25,27,21,6), Err = 1.69e−001


Reconstruction by the truncated `1 minimization.Reconstruction error is smaller.21 true nonzeros are detected.

11 / 47


A Demo (4)

0 50 100 150 200−2

−1.5

−1

−0.5

0

0.5

1

1.5

22−nd iter. (total,det,c−det,w−det)=(25,25,25,0), Err = 1.83e−015


Reconstruction by the truncated `1 minimization.Reconstruction error is very small.25 true nonzeros are all detected.

12 / 47


A Demo (5)

0 50 100 150 200−2

−1.5

−1

−0.5

0

0.5

1

1.5

23−rd iter. (total,det,c−det,w−det)=(25,25,25,0), Err = 9.16e−016


Reconstruction by the truncated `1 minimization.Exact reconstruction.

13 / 47

Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsThe Truncated Null Space Property Recoverability Improvement Reconstruction Error Bounds Algorithmic Convergence Behavior

Outline





5 Conclusions

14 / 47


Definition of the Truncated Null Space Property

A ∈ Rm×n has the Truncated Null Space Property (t-NSP) withorder L, γ > 0 and 0 < t ≤ n, if

‖vS‖1 ≤ γ‖vT\S‖1,

for each T ⊂ {1, . . . , n} with |T | = t and each index set S ⊂ Twith |S| ≤ L and each v ∈ N (A).

t=n: reduces to Null Space Property.

15 / 47


Example of the Truncated Null Space Property

Let t = 4, L = 2, v =(

0.1 0.5 0.2 1.5 2.4)T

,

T =[

2 3 4 5]

and S =[

2 4]⊂ T and

T \ S =[

3 5].

2 = ‖vS‖1 ≤ γ‖vT\S‖1 = 2.6γ

So γ ≥ 2/2.6.

t-NSP represented by (t , L, γ̄), where γ̄ is the minimum of allfeasible γ for all v ∈ N (A).

γ̄ is monotonic in L for any t fixed.

16 / 47


Sufficient Condition For a Single Truncated `1

Minimization (1)

Theorem

For given T, suppose that A has t-NSP for (|T |, L, γ̄ < 1). For anygiven x̄ ∈ Rn, it is the unique minimizer of

min{‖xT‖1 : Ax = Ax̄}

if ‖x̄T‖0 ≤ L.

T = Zn: classic result for BP.

17 / 47


Sufficient Condition For a Single Truncated `1

Minimization (2)

In particular, Gaussian measurement matrices have the t-NSP.

‖x̄‖0 < c m1+log n

m(BP)

‖x̄‖0 < dc + c m−d

1+log (n−d)(m−d)

(Truncated `1 Minimization)

d = dc + dw .

dc : number of correct detections.dw : number of wrong detections.

Objective:dc + c m−d

1+log (n−d)(m−d)

> c m1+log n

m

18 / 47


The Truncated `1 Minimization Over BP

Define k(d) = c m−d1+log (n−d)

(m−d)

Objective:

dc + k(0) > k(d)

⇔ dc >

∫ d

0|k ′(d)| (1)

(1) is reasonably likely to reach:|k ′(d)| < 1, and often smaller than 2/5 in practice.

19 / 47


Reconstruction Error Bounds of the Truncated `1

Minimization

Theorem

For given T, suppose that A has t-NSP for (|T |, L, γ̄ < 1). For anygiven x̄ ∈ Rn with ‖x̄T‖0 > L, x∗ is the minimizer of

min{‖xT‖1 : Ax = Ax̄}.

Then ‖(x∗ − x̄)T‖1 ≤ 2cσL(x̄T )1, where c = 1+γ̄1−γ̄ , σL(x̄T )1 is the `1

error of the best L-term approximation of x̄T .

20 / 47


Convergence Behavior of ISD (1)

Assume A has t-NSP (t , L, γ̄)and consider the k -th iteration of ISD.

k -th iteration:

min ‖xT (k)‖1 s.t . Ax = b.

Let t (k) ← |T (k)|, L(k) ← ‖x̄T (k)‖0, how about corresponding γ̄(k)?

If γ̄(k) < 1, succeed in reconstruction.If γ̄(k) ≥ 1, go to next iteration.

Objective: γ̄(1) > γ̄(2) > . . . > γ̄(k) > γ̄(k+1) > . . ., until below 1.

Condition on support detection: ?

21 / 47


Convergence Behavior of ISD (2)

Theorem

Suppose that A has t-NSP for (t1, L1, γ̄1) as well as (t2, L2, γ̄2) witht2 < t1 and L2 < L1. If (L1 − L2) > γ̄1(t1 − t2 − (L1 − L2)), thenγ̄2 < γ̄1.

Comments:(L1 − L2) represents the number of increased correct detectionsand (t1 − t2 − (L1 − L2)) represents the number of increasedwrong detections.

Number of increased correct detections should be larger thannumber of increased wrong detections by a certain amount.

22 / 47


Summary

The theoretical properties of the truncated `1 minimization arestudied.

These theoretical results indicate an iterative support detectionalgorithm is likely to work, given effective support detection.

These theoretical results do not rely on the concreteimplementations of support detection.

23 / 47


Outline





5 Conclusions

24 / 47


Practical Support Detection Methods

The implementations of practical support detection,

rely on features of different signals.vary for different kinds of signals.

25 / 47


Threshold-Based ISD for Fast Decaying Signals

Examples:

True signals: fast decaying signals (e.g. sparse Gaussiansignals)

Threshold based support detection method:

I(k+1) = {i : |x (k)i | > ε(k)}, k = 0, 1, 2, . . . ,

Threshold-based support detection is numerically provedeffective for fast decaying signals.

Large (in magnitude) components of the reconstruction likely toindicated true large nonzeros of the true signal.

26 / 47


How to Choose Threshold Value?

For sparse Gaussian signals: look for “first significant jump” in theincreasingly sorted reconstruction

0 50 100 150 200−1.5

−1

−0.5

0

0.5

1

1.5n=200, m=60, k=25, Err = 4.53e−001


(a) Failed reconstruction

0 50 100 150 2000

0.2

0.4

0.6

0.8

1

1.2

1.4n=200, m=60, k=25, Err = 4.53e−001

false nonzerotrue nonzeroadopted threshold vauereference threshold value

(b) Threshold value based on jump de-tection

|x (k)i+1| − |x

ki | > τ (k), where τ (k) =‖x (k)‖∞/m.

27 / 47

Acknowledgement Introduction Theoretical Results of ISD Support Detection for Fast Decaying Signals Numerical Experiments ConclusionsReview of Compared Algorithms Experiment Setup Numerical Results

Outline





5 Conclusions

28 / 47


Acceleration of ISD

Loose stopping tolerances of each truncated `1 minimizationduring support detection.

Warm-starting scheme: the output of the current truncated `1minimization as the initial point of the next truncated `1minimization.

29 / 47


Review of Compared Algorithms (1)

Noise-free measurements b = Ax̄

IRL1:

x (k) ← minx{

nX

i=1

w (k)i |xi | : Ax = b},

wherew (k)

i = (|x (k−1)i + η|)−1

,

IRLS:x (k) ← min

x{X

i

w̃ (k)i |xi |2 : Ax = b},

wherex (k) = Qk AT(AQk AT)−1b

Qk is the diagonal matrix with entries 1/w̃ (k)i and the weights are set as

w̃ (k)i = (|x (k−1)

i |2 + ζ)−1

30 / 47


Review of Compared Algorithms (2)

Noisy measurements b = Ax̄ + z, where z ∼ N(0, σ).

L1/L2:

minx‖x‖1 +

12ρ‖b − Ax‖2

2}

ISD:

x (k) ← minx‖x

T (k)‖1 +1

2ρ‖b − Ax‖2

2}

IRL1:

x (k) ← minx{

nX

i=1

w (k)i |xi | +

12ρ‖b − Ax‖2

2},

IRLS:x (k) ← min

x{X

i

w̃ (k)i |xi |2 : Ax = b},

wherex (k) = Qk AT(AQk AT)−1b

31 / 47


Comparison Settings

BP (L1/L2), IRL1, IRLS, and ISD.

BP (L1/L2), each reweighted `1 subproblem of IRL1, eachtruncated `1 subproblem of ISD, are all solved by YALL1package.

The same final stopping tolerance for these 4 algorithms, in orderfor fair comparison.

On the same computing platform: MATLAB on Linux.

32 / 47


Experimental Settings

b = Ax̄ + z, where z ∼ N(0, σ). A were Gaussian matrices generated by randn(m,n) andtransformed to have orthonormal rows.

# Nonzero Entries Noise σ Dimension n Sparsity k Range of m Tests for each m

1 Gaussian 0 600 40 [80:10:220] 100Gaussian 0 3000 100 [200:50:800] 100

2Gaussian 0.0001 2000 100 325 200Gaussian 0.001 2000 100 325 200Gaussian 0.01 2000 100 325 200

3 Bernoulli 0 600 40 [80:10:220] 1004 2D Phantom 0.002 65536 4765 9830 1

Table: Summary of test sets.

33 / 47


Test set 1

80 100 120 140 160 180 200 2200

0.5

1

1.5

2

2.5

m

Run

ning

tim

e (s

)

n=600, k=40, m=80:10:220

BPISDIRLSIRL1

(c) CPU time

200 300 400 500 600 700 8000

10

20

30

40

50

60

70

80

m

Run

ning

tim

e (s

)

n=3000, k=100, m=200:50:800

BPISDIRLSIRL1

(d) CPU time

Figure: Test set 1: Comparisons in CPU time.

34 / 47


Test set 1

80 100 120 140 160 180 200 2200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

m

Sta

rdar

d V

aria

tion

of r

unni

ng ti

me

(s)

n=600, k=40, m=80:10:220

BPISDIRLSIRL1

(a) Standard variation in CPU time

200 300 400 500 600 700 8000

5

10

15

20

25

30

35

m

Sta

rdar

d va

riatio

n of

run

ning

tim

e (s

)

n=3000, k=100, m=200:50:800

BPISDIRLSIRL1

(b) Standard variation in CPU time

Figure: Test set 1: Comparisons in standard variation of CPU time.

35 / 47


Test set 1

80 100 120 140 160 180 200 2200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

m

Exa

ct r

econ

stru

ctio

n fr

eque

ncy

n=600, k=40, m=80:10:220

BPISDIRLSIRL1

(a) Exact reconstruction frequency

200 300 400 500 600 700 8000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

m

Exa

ct r

econ

stru

ctio

n fr

eque

ncy

n=3000, k=100, m=200:50:800

BPISDIRLSIRL1

(b) Exact reconstruction frequencies

Figure: Test set 1: Comparisons in recoverability.

36 / 47


Test set 2

0 50 100 150 2000

5

10

15

20

25

30

35

Trials

Run

ning

tim

e (s

)

n=2000, m=325, k=100

L1/L2ISDIRLSIRL1

(a) CPU time

0 50 100 150 20010

−4

10−3

10−2

10−1

100

Trials

Rel

ativ

e re

cons

truc

tion

erro

rs in

L2−

norm

n=2000, m=325, k=100

L1/L2ISDIRLSIRL1

(b) Reconstruction errors in`2 norm

0 50 100 150 20010

−4

10−3

10−2

10−1

100

101

Trials

Rel

ativ

e re

cons

truc

tion

erro

rs in

L1−

norm

n=2000, m=325, k=100

L1/L2ISDIRLSIRL1

(c) Reconstruction errors in`1 norm

Figure: Test set 2 with σ = 0.0001: Comparisons in CPU time andreconstruction errors

37 / 47


Test set 2

0 50 100 150 2000

5

10

15

20

25

30

Trials

Run

ning

tim

e (s

)

n=2000, m=325, k=100

L1/L2ISDIRLSIRL1

(a) CPU time

0 50 100 150 20010

−3

10−2

10−1

100

Trials

Rel

ativ

e re

cons

truc

tion

erro

rs in

L2−

norm

n=2000, m=325, k=100

L1/L2ISDIRLSIRL1


0 50 100 150 20010

−3

10−2

10−1

100

101

Trials

Rel

ativ

e re

cons

truc

tion

erro

rs in

L1−

norm

n=2000, m=325, k=100

L1/L2ISDIRLSIRL1



38 / 47


Test set 2

0 50 100 150 2000

5

10

15

20

25

30

Trials

Run

ning

tim

e (s

)

n=2000, m=325, k=100

L1/L2ISDIRLSIRL1

(a) CPU time

0 50 100 150 20010

−2

10−1

100

Trials

Rel

ativ

e re

cons

truc

tion

erro

rs in

L2−

norm

n=2000, m=325, k=100

L1/L2ISDIRLSIRL1


0 50 100 150 20010

−2

10−1

100

Trials

Rel

ativ

e re

cons

truc

tion

erro

rs in

L1−

norm

n=2000, m=325, k=100

L1/L2ISDIRLSIRL1



39 / 47


Test set 3

80 100 120 140 160 180 200 2200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

m

Exa

ct r

econ

stru

ctio

n fr

eque

ncy

n=600, k=40, m=80:10:220

BPISDIRLSIRL1

Figure: Test set 3 with sparse Bernoulli signals: Comparisons inrecoverability.

Threshold-based support detection does not works for sparse Bernoullisignals.

40 / 47


Test set 4

Original

256 x 256

(a)

Figure: n=65536, m=9830, k=4765.

The true image can be sparsely represented by the 2D Haar wavelets.

41 / 47


Test set 4

BP: SNR=10.67dB, Err=2.54e−01, CPU time=44.02 s

Sample ratio: 15%

(a)

Figure: Reconstruction by L1/L2.

42 / 47


Test set 4

IRL1: SNR=19.08dB, Err=9.64e−02, CPU time=519.10 s

Sample ratio: 15%

(a)

ISD: SNR=37.47dB, Err=1.16e−02, CPU time=173.55 s

Sample ratio: 15%

(b)

Figure: The left plot is the reconstruction by IRL1; the right one is thereconstruction by ISD.

43 / 47


Outline





5 Conclusions

44 / 47


Conclusions

Effective support detection improves CS reconstruction from boththeoretical points of view and practical performances.

In particular, iterative thresholding is effective on sparse signalswith fast decaying distribution of nonzero values.

On-going and future work:Other specific support detection methods for 2D images, video, etc.Other priors: TV(x).ISD applied to other reconstruction algorithms such as SL0 andgreedy methods.

45 / 47


Figure: From http://blog.mozilla.com/sumo/2008/09/15/the-vision-for-sumo-5/

46 / 47


80 100 120 140 160 180 200 2200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

m

Exa

ct r

econ

stru

ctio

n fr

eque

ncy

n=600, k=40, m=80:10:220

BP2 truncated iterations4 truncated iterations6 truncated iterations8 truncated iterations

(a) Exact reconstruction frequencies

200 300 400 500 600 700 8000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

m

Exa

ct r

econ

stru

ctio

n fr

eque

ncy

n=3000, k=100, m=200:50:800

BP2 truncated iterations4 truncated iterations6 truncated iterations8 truncated iterations

(b) Exact reconstruction frequencies

Figure: Test set 1: Comparisons in recoverability.

47 / 47

enhanced compressive sensing using iterative support...

Documents