a dual certificate analysis of compressive off-the-grid recoverya dual certificate analysis of...
Post on 17-Oct-2020
1 Views
Preview:
TRANSCRIPT
A dual certificate analysis of compressive off-the-grid recovery
Nicolas KerivenEcole Normale Supérieure (Paris)
CFM-ENS chair in Data Science
Joint work with Clarice Poon (Cambridge Uni.), Gabriel Peyré (ENS)
Journée GdR MIA, May 3rd 2018
Discrete compressive sensing
1/14
Discrete compressive sensing
1/14
Discrete compressive sensing
1/14
• Signal: vector
Discrete compressive sensing
1/14
• Signal: vector• Sparsity: few non-zeros coefficients
Discrete compressive sensing
1/14
• Signal: vector• Sparsity: few non-zeros coefficients• Dimensionality reduction (often random matrix)
Discrete compressive sensing
1/14
• Signal: vector• Sparsity: few non-zeros coefficients• Dimensionality reduction (often random matrix)• Recovery: convex relaxation LASSO
Discrete compressive sensing
1/14
Off-the-grid recovery: « super-resolution »
2/14
Off-the-grid recovery: « super-resolution »
2/14
- Fourier
- Convolution
Off-the-grid recovery: « super-resolution »
2/14
- Fourier
- Convolution
Off-the-grid recovery: « super-resolution »
2/14
- Fourier
- Convolution
Off-the-grid recovery: « super-resolution »
• Signal: Radon measure
2/14
- Fourier
- Convolution
Off-the-grid recovery: « super-resolution »
• Signal: Radon measure• Sparsity:
2/14
- Fourier
- Convolution
Off-the-grid recovery: « super-resolution »
• Signal: Radon measure• Sparsity:• Dimensionality reduction (e.g. first Fourier coefficients)
2/14
- Fourier
- Convolution
Off-the-grid recovery: « super-resolution »
• Signal: Radon measure• Sparsity:• Dimensionality reduction (e.g. first Fourier coefficients)• Recovery: convex relaxation?
See Keriven 2017, Gribonval 2017
2/14
- Fourier
- Convolution
Off-the-grid recovery: « super-resolution »
• Signal: Radon measure• Sparsity:• Dimensionality reduction (e.g. first Fourier coefficients)• Recovery: convex relaxation? BLASSO [De Castro, Gamboa 2012]
See Keriven 2017, Gribonval 2017
2/14
- Fourier
- Convolution
Off-the-grid recovery: « super-resolution »
• Signal: Radon measure• Sparsity:• Dimensionality reduction (e.g. first Fourier coefficients)• Recovery: convex relaxation? BLASSO [De Castro, Gamboa 2012]
Other approaches: « Prony-like » ESPRIT, MUSIC… (but only 1d noiseless Fourier)
See Keriven 2017, Gribonval 2017
2/14
Example of applications
Fluorescence microscopy[Betzig 2006]
3/14
Example of applications
Fluorescence microscopy[Betzig 2006]
Astronomy[Puschmann 2017]
3/14
Example of applications
Fluorescence microscopy[Betzig 2006]
Compressive k-means(GMM…) [Keriven 2017]
Astronomy[Puschmann 2017]
3/14
Example of applications
Fluorescence microscopy[Betzig 2006]
Compressive k-means(GMM…) [Keriven 2017]
Astronomy[Puschmann 2017]
• Neuro-imaging with EEG [Gramfort 2013]
• 1-layer neural network [Bach 2017]
• Radar• Geophysics• …
3/14
A seminal result [Candès, Fernandez-Granda 2012]
4/14
A seminal result [Candès, Fernandez-Granda 2012]
First Fourier coefficients
4/14
A seminal result [Candès, Fernandez-Granda 2012]
First Fourier coefficients
Inverse Fourier
4/14
A seminal result [Candès, Fernandez-Granda 2012]
First Fourier coefficients
Inverse Fourier
BLASSO
4/14
A seminal result [Candès, Fernandez-Granda 2012]
First Fourier coefficients
• Minimal separation
Inverse Fourier
BLASSO
4/14
A seminal result [Candès, Fernandez-Granda 2012]
First Fourier coefficients
• Minimal separation
• Regular Fourier on the Torus
Inverse Fourier
BLASSO
4/14
A seminal result [Candès, Fernandez-Granda 2012]
First Fourier coefficients
• Minimal separation
• Regular Fourier on the Torus
• Reconstruction: formulated as SDP (other: Frank-Wolfe,
greedy, Prony-like…)
Inverse Fourier
BLASSO
4/14
A seminal result [Candès, Fernandez-Granda 2012]
First Fourier coefficients
• Minimal separation
• Regular Fourier on the Torus
• Reconstruction: formulated as SDP (other: Frank-Wolfe,
greedy, Prony-like…)
• Noise: weak convergence (mass of concentrated
around true Diracs)
Inverse Fourier
BLASSO
4/14
A seminal result [Candès, Fernandez-Granda 2012]
First Fourier coefficients
• Minimal separation
• Regular Fourier on the Torus
• Reconstruction: formulated as SDP (other: Frank-Wolfe,
greedy, Prony-like…)
• Noise: weak convergence (mass of concentrated
around true Diracs)
Inverse Fourier
BLASSO
4/14
Randomness ? Numberof measurements ?
Compressed sensing off-the-grid [Tang, Recht 2013]
5/14
Compressed sensing off-the-grid [Tang, Recht 2013]
Random Fourier coefficients
random
5/14
Compressed sensing off-the-grid [Tang, Recht 2013]
Random Fourier coefficients
BLASSO
random
5/14
Compressed sensing off-the-grid [Tang, Recht 2013]
Random Fourier coefficients
• As in compressive sensing, randomFourier sampling is possible
But:• Limited to 1d Regular Fourier (relies
heavily on previous work by Candès)
• Random signs assumption
BLASSO
random
5/14
Compressed sensing off-the-grid [Tang, Recht 2013]
Random Fourier coefficients
• As in compressive sensing, randomFourier sampling is possible
But:• Limited to 1d Regular Fourier (relies
heavily on previous work by Candès)
• Random signs assumption
BLASSO
Questions:
• More general sampling scheme ?
• Multi-dimensional result ?
• Get rid of random signs ?
random
5/14
Outline
Background on dual certificates
Compressive off-the-grid recovery
Conclusion, outlooks
Dual certificates
Measurements
Hilbert space
6/14
Dual certificates
Measurements
Hilbert space
BLASSO
6/14
Dual certificates
Measurements
Hilbert space
BLASSO
First-order conditions
solution ofBLASSO
6/14
solution of
Dual certificates
Measurements
Hilbert space
BLASSO
First-order conditions
Dual certificate (noiseless case)
solution ofBLASSO
6/14
What is a dual certificate ?
What does it look like ?
7/14
What is a dual certificate ?
What does it look like ?
7/14
What is a dual certificate ?
What does it look like ?
7/14
Case :
What is a dual certificate ?
What does it look like ?
7/14
Case :
What is a dual certificate ?
What does it look like ?
7/14
Case :
What is a dual certificate ?
What does it look like ?
7/14
Ensures uniqueness and robustness…
Non-degenerate dual certif.
Recovery results
8/14
Recovery results
8/14
Recovery results
Theorem (refinement of [Azaïs et al. 2015])
8/14
Recovery results
Theorem (refinement of [Azaïs et al. 2015])
Hyp: there exists a ND dual certif.
8/14
Recovery results
Theorem (refinement of [Azaïs et al. 2015])
Hyp: there exists a ND dual certif.Result: (wrt Bregman divergence)
- not necessarily sparse, but-
8/14
Recovery results
Theorem (refinement of [Azaïs et al. 2015])
Hyp: there exists a ND dual certif.Result: (wrt Bregman divergence)
- not necessarily sparse, but- Mass of concentrated around
8/14
Recovery results
Theorem (refinement of [Azaïs et al. 2015])
Hyp: there exists a ND dual certif.Result: (wrt Bregman divergence)
- not necessarily sparse, but- Mass of concentrated around- Concentration increases when:
noise
8/14
Recovery results
Theorem (refinement of [Azaïs et al. 2015])
Hyp: there exists a ND dual certif.Result: (wrt Bregman divergence)
- not necessarily sparse, but- Mass of concentrated around- Concentration increases when:
noise
Theorem ([Duval Peyré 2015])
8/14
Recovery results
Theorem (refinement of [Azaïs et al. 2015])
Hyp: there exists a ND dual certif.Result: (wrt Bregman divergence)
- not necessarily sparse, but- Mass of concentrated around- Concentration increases when:
noise
Theorem ([Duval Peyré 2015])
Hyp: the minimal norm certificate is non-degenerate
8/14
Recovery results
Theorem (refinement of [Azaïs et al. 2015])
Hyp: there exists a ND dual certif.Result: (wrt Bregman divergence)
- not necessarily sparse, but- Mass of concentrated around- Concentration increases when:
noise
Theorem ([Duval Peyré 2015])
Hyp: the minimal norm certificate is non-degenerateResult: (in the small noise regime)
- is sparse, with the right number of components
-
8/14
Outline
Background on dual certificates
Compressive off-the-grid recovery
Conclusion, outlooks
Goal
Goal: random sampling
Low-dim, random
9/14
Goal
Strategy: Start with « high »-dimensional problemGoal: random sampling
Low-dim, random Ex: full Fourier (high-dim), full convolution (infinite-dim)
9/14
Goal
Strategy: Start with « high »-dimensional problemGoal: random sampling
Step 1: Build ND certificate with full kernel
Low-dim, random Ex: full Fourier (high-dim), full convolution (infinite-dim)
9/14
Goal
Strategy: Start with « high »-dimensional problemGoal: random sampling
Step 1: Build ND certificate with full kernel
Low-dim, random Ex: full Fourier (high-dim), full convolution (infinite-dim)
9/14
Goal
Strategy: Start with « high »-dimensional problemGoal: random sampling
Step 1: Build ND certificate with full kernel
Low-dim, random Ex: full Fourier (high-dim), full convolution (infinite-dim)
9/14
Goal
Strategy: Start with « high »-dimensional problemGoal: random sampling
Step 1: Build ND certificate with full kernel
Step 2: Use Random Features on full kernel to define sampling
Low-dim, random Ex: full Fourier (high-dim), full convolution (infinite-dim)
9/14
Goal
Strategy: Start with « high »-dimensional problemGoal: random sampling
Step 1: Build ND certificate with full kernel
Step 2: Use Random Features on full kernel to define sampling
Assuming
Define
Low-dim, random Ex: full Fourier (high-dim), full convolution (infinite-dim)
9/14
Goal
Strategy: Start with « high »-dimensional problemGoal: random sampling
Step 1: Build ND certificate with full kernel
Step 2: Use Random Features on full kernel to define sampling
Assuming
Define
Then
Low-dim, random Ex: full Fourier (high-dim), full convolution (infinite-dim)
9/14
Goal
Strategy: Start with « high »-dimensional problemGoal: random sampling
Step 1: Build ND certificate with full kernel
Step 2: Use Random Features on full kernel to define sampling
Assuming
Define
Then
Low-dim, random Ex: full Fourier (high-dim), full convolution (infinite-dim)
9/14
Goal
Strategy: Start with « high »-dimensional problemGoal: random sampling
Step 1: Build ND certificate with full kernel
Step 2: Use Random Features on full kernel to define sampling
m=10
Assuming
Define
Then
Low-dim, random Ex: full Fourier (high-dim), full convolution (infinite-dim)
9/14
Goal
Strategy: Start with « high »-dimensional problemGoal: random sampling
Step 1: Build ND certificate with full kernel
Step 2: Use Random Features on full kernel to define sampling
m=10 m=50
Assuming
Define
Then
Low-dim, random Ex: full Fourier (high-dim), full convolution (infinite-dim)
9/14
Goal
Strategy: Start with « high »-dimensional problemGoal: random sampling
Step 1: Build ND certificate with full kernel
Step 2: Use Random Features on full kernel to define sampling
m=500m=10 m=50
Assuming
Define
Then
Low-dim, random Ex: full Fourier (high-dim), full convolution (infinite-dim)
9/14
Step 1: Acceptable full kernels
10/14
Step 1: Acceptable full kernels
10/14
Step 1: Acceptable full kernels
10/14
Step 1: Acceptable full kernels
10/14
Step 1: Acceptable full kernels
Min. Separation
10/14
Step 1: Acceptable full kernels
Min. Separation
1: kernel at each saturation point
10/14
Step 1: Acceptable full kernels
Min. Separation
2: Small adjustements: minimal separation
1: kernel at each saturation point
10/14
Step 1: Acceptable full kernels
Min. Separation
2: Small adjustements: minimal separation
- Multi-d square Féjer kernel (regular Fourier on Torus)
- Multi-d Gaussian kernel
1: kernel at each saturation point
10/14
Step 2: sampling
Thm: Ideal scaling in sparsity: infinite-dim. golfing scheme
11/14
Step 2: sampling
Thm: Ideal scaling in sparsity: infinite-dim. golfing scheme
Depend on kernel
11/14
Step 2: sampling
Thm: Ideal scaling in sparsity: infinite-dim. golfing scheme
• There exists a ND dual certificate (however not the minimal norm certificate)
Depend on kernel
11/14
Step 2: sampling
Thm: Ideal scaling in sparsity: infinite-dim. golfing scheme
• There exists a ND dual certificate (however not the minimal norm certificate)
• No need for random signs
Depend on kernel
11/14
Step 2: sampling
Thm: Ideal scaling in sparsity: infinite-dim. golfing scheme
• There exists a ND dual certificate (however not the minimal norm certificate)
• No need for random signs
• Proof: golfing scheme [Gross 2009, Candès Plan 2011]
Depend on kernel
11/14
Step 2: sampling
Thm: Ideal scaling in sparsity: infinite-dim. golfing scheme
• There exists a ND dual certificate (however not the minimal norm certificate)
• No need for random signs
• Proof: golfing scheme [Gross 2009, Candès Plan 2011]
Thm: Minimal norm certificate (adaptation of [Tang, Recht 2013])
Depend on kernel
With random signs Without random signs
11/14
Step 2: sampling
Thm: Ideal scaling in sparsity: infinite-dim. golfing scheme
• There exists a ND dual certificate (however not the minimal norm certificate)
• No need for random signs
• Proof: golfing scheme [Gross 2009, Candès Plan 2011]
Thm: Minimal norm certificate (adaptation of [Tang, Recht 2013])
Depend on kernel
With random signs Without random signs
11/14
Fun application: convex approach for automatic estimation of number of components in a GMM
Number of measurements in practice ?
Nicolas Keriven
Compressive k-means [Keriven 2017]
Relative number of measurements m/(sd)
12/14
Outline
Background on dual certificates
Compressive off-the-grid recovery
Conclusion, outlooks
Summary, outlooks
• Summary: generalization of existing results on super-resolutionwith random measurements (and minimal separation)
• Beyond Fourier on the Torus (« acceptable » kernels)• Multi-d• No need for random signs for basic recovery result• Support recovery when random signs (or quadratic number of
measurements)
13/14
Summary, outlooks
• Summary: generalization of existing results on super-resolutionwith random measurements (and minimal separation)
• Beyond Fourier on the Torus (« acceptable » kernels)• Multi-d• No need for random signs for basic recovery result• Support recovery when random signs (or quadratic number of
measurements)
13/14
• Outlooks• Other kernels, very different from translation-invariant• More quantified treatment of dimension• Other practical applications (eg 1-layer neural networks with continuum of
neurons [Bach 2017])
Poon, Keriven, Peyré. A Dual Certificates Analysis of Compressive Off-the-Grid Recovery. Preprint arxiv:1802.08464
Code: sketchml.gforge.inria.fr,github: nkeriven
Thank you !
14/14
data-ens.github.ioEnter the data challenges!Come to the colloquium!Come to the Laplace seminars!
top related