fast algorithms for nonconvex compressive sensing

Fast algorithms for nonconvexcompressive sensing

Rick Chartrand

Los Alamos National Laboratory

New Mexico Consortium

February 25, 2009

Slide 1 of 16

Operated by Los Alamos National Security, LLC for NNSA

Outline

Motivating example

Nonconvex compressive sensing

Algorithms

Summary

Slide 2 of 16

Operated by Los Alamos National Security, LLC for NNSA

Motivating example

Sparse tomographySuppose we want to reconstruct animage from samples of its Fouriertransform. How many samples dowe need?

Shepp-Logan phantom

Consider radial sampling,such as in MRI or (roughly)CT.

Ω

Slide 3 of 16

Operated by Los Alamos National Security, LLC for NNSA

Motivating example

Nonconvexity

Fewer measurements are needed with nonconvex minimization:

minu‖∇u‖pp, subject to (Fu)|Ω = (Fx)|Ω.

With p = 1, solution is u = x with 18 lines ( |Ω||x| = 6.9%).

With p = 1/2, 10 lines suffice ( |Ω||x| = 3.8%).

backprojection, 18 lines p = 1, 18 lines p = 12

, 10 lines p = 1, 10 lines

Slide 4 of 16

Operated by Los Alamos National Security, LLC for NNSA

Motivating example

Nonconvexity

Fewer measurements are needed with nonconvex minimization:

minu‖∇u‖pp, subject to (Fu)|Ω = (Fx)|Ω.

With p = 1, solution is u = x with 18 lines ( |Ω||x| = 6.9%).

With p = 1/2, 10 lines suffice ( |Ω||x| = 3.8%).

backprojection, 18 lines p = 1, 18 lines

p = 12

, 10 lines p = 1, 10 lines

Slide 4 of 16

Operated by Los Alamos National Security, LLC for NNSA

Motivating example

Nonconvexity

Fewer measurements are needed with nonconvex minimization:

minu‖∇u‖pp, subject to (Fu)|Ω = (Fx)|Ω.

With p = 1, solution is u = x with 18 lines ( |Ω||x| = 6.9%).

With p = 1/2, 10 lines suffice ( |Ω||x| = 3.8%).

backprojection, 18 lines p = 1, 18 lines p = 12

, 10 lines p = 1, 10 lines

Slide 4 of 16

Operated by Los Alamos National Security, LLC for NNSA

Motivating example

New results

These are old results (Mar. 2006); what’s new?

I Reconstruction (to 50 dB) in 13 seconds (versus literature-best1–3 minutes).

I Exact reconstruction from 9 lines (3.5% of Fourier transform).

10 lines fastest 10-line re-covery

9 lines recovery fromfewest samples

Slide 5 of 16

Operated by Los Alamos National Security, LLC for NNSA

Motivating example

New results

These are old results (Mar. 2006); what’s new?I Reconstruction (to 50 dB) in 13 seconds (versus literature-best

1–3 minutes).

I Exact reconstruction from 9 lines (3.5% of Fourier transform).

10 lines fastest 10-line re-covery

9 lines recovery fromfewest samples

Slide 5 of 16

Operated by Los Alamos National Security, LLC for NNSA

Motivating example

New results

These are old results (Mar. 2006); what’s new?I Reconstruction (to 50 dB) in 13 seconds (versus literature-best

1–3 minutes).I Exact reconstruction from 9 lines (3.5% of Fourier transform).

10 lines fastest 10-line re-covery

9 lines recovery fromfewest samples

Slide 5 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

Optimization for sparse recovery

I Let x ∈ RN be sparse: ‖Ψx‖0 = K, K N .I Suppose Φ is an M ×N matrix, M N , with Φ and Ψ

incoherent. For example, Φ = (ϕij), i.i.d. ϕij ∼ N(0, σ2).

minu‖Ψu‖0, s.t. Φu = Φx.

minu‖Ψu‖1, s.t. Φu = Φx.

minu‖Ψu‖pp, s.t. Φu = Φx,

Unique solution is u = x with opti-mally small M , but is NP-hard.

M ≥ 2K suffices w.h.p.

Can be solved efficiently; requiresmore measurements for reconstruc-tion. M ≥ CK log(N/K)

where 0 < p < 1. Solvable in prac-tice; requires fewer measurementsthan `1.M ≥ C1(p)K+pC2(p)K log(N/K)(with V. Staneva)

Slide 6 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

Optimization for sparse recovery

I Let x ∈ RN be sparse: ‖Ψx‖0 = K, K N .I Suppose Φ is an M ×N matrix, M N , with Φ and Ψ

incoherent. For example, Φ = (ϕij), i.i.d. ϕij ∼ N(0, σ2).

minu‖Ψu‖0, s.t. Φu = Φx.

minu‖Ψu‖1, s.t. Φu = Φx.

minu‖Ψu‖pp, s.t. Φu = Φx,

Unique solution is u = x with opti-mally small M , but is NP-hard.

M ≥ 2K suffices w.h.p.

Can be solved efficiently; requiresmore measurements for reconstruc-tion. M ≥ CK log(N/K)

where 0 < p < 1. Solvable in prac-tice; requires fewer measurementsthan `1.M ≥ C1(p)K+pC2(p)K log(N/K)(with V. Staneva)

Slide 6 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 2:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.1p

Slide 7 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 2:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.2p

Slide 7 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 2:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.3p

Slide 7 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 2:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.4p

Slide 7 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 2:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.5p

Slide 7 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.1p

Slide 8 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.2p

Slide 8 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.3p

Slide 8 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.4p

Slide 8 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.5p

Slide 8 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.6p

Slide 8 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1:

x

Φu = Φx

|u1|p + |u2|p + |u3|p = 0.7p

Slide 8 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1/2:

x

|u1|p + |u2|p + |u3|p = 0.1p

Φu = Φx

Slide 9 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1/2:

x

|u1|p + |u2|p + |u3|p = 0.2p

Φu = Φx

Slide 9 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1/2:

x

|u1|p + |u2|p + |u3|p = 0.3p

Φu = Φx

Slide 9 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1/2:

x

|u1|p + |u2|p + |u3|p = 0.4p

Φu = Φx

Slide 9 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1/2:

x

|u1|p + |u2|p + |u3|p = 0.5p

Φu = Φx

Slide 9 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1/2:

x

|u1|p + |u2|p + |u3|p = 0.6p

Φu = Φx

Slide 9 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1/2:

x

|u1|p + |u2|p + |u3|p = 0.7p

Φu = Φx

Slide 9 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1/2:

x

|u1|p + |u2|p + |u3|p = 0.8p

Φu = Φx

Slide 9 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1/2:

x

|u1|p + |u2|p + |u3|p = 0.9p

Φu = Φx

Slide 9 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

The geometry of `p

minu ‖u‖pp, subject to Φu = Φxp = 1/2:

x

|u1|p + |u2|p + |u3|p = 1p

Φu = Φx

Slide 9 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

Why might global minimization be possible?

Consider the ε-regularized, constraint-eliminated objective:

Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,

whereR(V ) = N (Φ). A moderate ε fills in the local minima.

Slide 10 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

Why might global minimization be possible?

Consider the ε-regularized, constraint-eliminated objective:

Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,

whereR(V ) = N (Φ). A moderate ε fills in the local minima.

ε = 0

Slide 10 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

Why might global minimization be possible?

Consider the ε-regularized, constraint-eliminated objective:

Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,

whereR(V ) = N (Φ). A moderate ε fills in the local minima.

ε = 1

Slide 10 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

Why might global minimization be possible?

Consider the ε-regularized, constraint-eliminated objective:

Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,

whereR(V ) = N (Φ). A moderate ε fills in the local minima.

ε = 0.1

Slide 10 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

Why might global minimization be possible?

Consider the ε-regularized, constraint-eliminated objective:

Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,

whereR(V ) = N (Φ). A moderate ε fills in the local minima.

ε = 0.01

Slide 10 of 16

Operated by Los Alamos National Security, LLC for NNSA

Nonconvex compressive sensing

Why might global minimization be possible?

Consider the ε-regularized, constraint-eliminated objective:

Fε(t) =N∑i=1

[xi + (V t)i

]2 + ε

p/2,

whereR(V ) = N (Φ). A moderate ε fills in the local minima.

ε = 0.001

Slide 10 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

Semiconvex regularization

Now we generalize an approach of J. Yang, W. Yin, Y. Zhang, and Y.Wang. Consider a mollified `p objective on R2:

ϕ(t) =

γ|t|2 if |t| ≤ α|t|p/p− δ if |t| > α

The parameters are chosen tomake ϕ ∈ C1.

ϕ(t)

t1α

Now we seek ψ such that

ϕ(t) = minw

ψ(w) + (β/2)|t− w|22

This can be found by convex duality, as |t|22/2−ϕ(t)/β is convex ifβ = αp−2.

Slide 11 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

Semiconvex regularization

Now we generalize an approach of J. Yang, W. Yin, Y. Zhang, and Y.Wang. Consider a mollified `p objective on R2:

ϕ(t) =

γ|t|2 if |t| ≤ α|t|p/p− δ if |t| > α

The parameters are chosen tomake ϕ ∈ C1.

ϕ(t)

t1α

Now we seek ψ such that

ϕ(t) = minw

ψ(w) + (β/2)|t− w|22

This can be found by convex duality, as |t|22/2−ϕ(t)/β is convex ifβ = αp−2.

Slide 11 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

A splitting approach

Now we consider an unconstrained `p minimization problem, andreplace

minu

N∑i=1

ϕ((Du)i) + (µ/2)‖Φu− b‖22

with the split version

minu,w

N∑i=1

ψ(wi) + (β/2)‖Du− w‖22 + (µ/2)‖Φu− b‖22,

which we solve by alternate minimization.

Slide 12 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

Easy iterations

Holding u fixed, the w-subproblem is separable, and its solutioncomes from the convex duality:

wi = max

0, |(Du)i| −

|(Du)i|p−1

β

(Du)i|(Du)i|

.

This generalizes shrinkage ( or soft thresholding ) to `p.

Holding w fixed, the u-problem is quadratic:

(βDTD + µΦTΦ)u = βDTw + µΦT b.

If Φ is a Fourier sampling operator, we can solve this in the Fourierdomain. This is very fast!

Slide 13 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

Easy iterations

Holding u fixed, the w-subproblem is separable, and its solutioncomes from the convex duality:

wi = max

0, |(Du)i| −

|(Du)i|p−1

β

(Du)i|(Du)i|

.

This generalizes shrinkage ( or soft thresholding ) to `p.

Holding w fixed, the u-problem is quadratic:

(βDTD + µΦTΦ)u = βDTw + µΦT b.

If Φ is a Fourier sampling operator, we can solve this in the Fourierdomain. This is very fast!

Slide 13 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

Easy iterations

Holding u fixed, the w-subproblem is separable, and its solutioncomes from the convex duality:

wi = max

0, |(Du)i| −

|(Du)i|p−1

β

(Du)i|(Du)i|

.

This generalizes shrinkage ( or soft thresholding ) to `p.

Holding w fixed, the u-problem is quadratic:

(βDTD + µΦTΦ)u = βDTw + µΦT b.

If Φ is a Fourier sampling operator, we can solve this in the Fourierdomain. This is very fast!

Slide 13 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

Enforcing equality

minu,w

N∑i=1

ψ(wi) + (β/2)‖Du− w‖22 + (µ/2)‖Φu− b‖22

Typically one enforces w = Du and Φu = b by iteratively growingβ and µ larger (continuation).

We get better results from Bregman iteration (generalizing T.Goldstein, S. Osher):

minu,w

N∑i=1

ψ(wi)+(β/2)‖Du−w−Bw‖22 +(µ/2)‖Φu−b−Bu‖22,

and update Bn+1w = Bnw + w −Du (inner loop),

Bm+1u = Bmu + b− Φu (outer loop, if desired).

Slide 14 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

Enforcing equality

minu,w

N∑i=1

ψ(wi) + (β/2)‖Du− w‖22 + (µ/2)‖Φu− b‖22

Typically one enforces w = Du and Φu = b by iteratively growingβ and µ larger (continuation).

We get better results from Bregman iteration (generalizing T.Goldstein, S. Osher):

minu,w

N∑i=1

ψ(wi)+(β/2)‖Du−w−Bw‖22 +(µ/2)‖Φu−b−Bu‖22,

and update Bn+1w = Bnw + w −Du (inner loop),

Bm+1u = Bmu + b− Φu (outer loop, if desired).

Slide 14 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

A surprise

I An early coding mistake led to the use of p = −1/2. This iswhat made the 9-line phantom reconstruction possible.

I Results decay slowly as p is decreased below −1/2.I Negative p-values were used by Rao and Kreutz-Delgado in an

iteratively-reweighted least squares (IRLS) algorithm. Theirnegative p results were worse than their positive p results,which were hampered by getting stuck in local minima.

I A mollified IRLS approach (with Wotao Yin) apparently avoidslocal minima, but negative p results are not better than positivep results.

Slide 15 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

A surprise

I An early coding mistake led to the use of p = −1/2. This iswhat made the 9-line phantom reconstruction possible.

I Results decay slowly as p is decreased below −1/2.

I Negative p-values were used by Rao and Kreutz-Delgado in aniteratively-reweighted least squares (IRLS) algorithm. Theirnegative p results were worse than their positive p results,which were hampered by getting stuck in local minima.

I A mollified IRLS approach (with Wotao Yin) apparently avoidslocal minima, but negative p results are not better than positivep results.

Slide 15 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

A surprise

I An early coding mistake led to the use of p = −1/2. This iswhat made the 9-line phantom reconstruction possible.

I Results decay slowly as p is decreased below −1/2.I Negative p-values were used by Rao and Kreutz-Delgado in an

iteratively-reweighted least squares (IRLS) algorithm. Theirnegative p results were worse than their positive p results,which were hampered by getting stuck in local minima.

I A mollified IRLS approach (with Wotao Yin) apparently avoidslocal minima, but negative p results are not better than positivep results.

Slide 15 of 16

Operated by Los Alamos National Security, LLC for NNSA

Algorithms

A surprise

I An early coding mistake led to the use of p = −1/2. This iswhat made the 9-line phantom reconstruction possible.

I Results decay slowly as p is decreased below −1/2.I Negative p-values were used by Rao and Kreutz-Delgado in an

iteratively-reweighted least squares (IRLS) algorithm. Theirnegative p results were worse than their positive p results,which were hampered by getting stuck in local minima.

I A mollified IRLS approach (with Wotao Yin) apparently avoidslocal minima, but negative p results are not better than positivep results.

Slide 15 of 16

Operated by Los Alamos National Security, LLC for NNSA

Summary

Summary

I Nonconvex compressive sensing allows sparse signals to berecovered with even fewer measurements than “traditional”compressive sensing.

I Decreasing p also improves robustness to noise, and speedsup convergence.

I Regularizing the objective appears to keep algorithms fromconverging to nonglobal minima.

I For Fourier-sampling measurements, such as MRI, a very fastalgorithm is available.

math.lanl.gov/~rick

Slide 16 of 16

Operated by Los Alamos National Security, LLC for NNSA