sparse approximation: from image restoration to high...
TRANSCRIPT
![Page 1: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/1.jpg)
Sparse Approximation: from Image Restoration to High Dimensional Classification
Bin Dong
Workshop on Frame Theory and Sparse Representation for Complex Data
Institute of Mathematical Science
National University of Singapore
29 May - 2 June 2017
Beijing International Center for Mathematical ResearchBeijing Institute of Big Data Research
Peking University
1
![Page 2: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/2.jpg)
OutlinesI. Brief review of image restoration models
II. Wavelet frame transforms and differential operators under variational and PDE framework
III. Sparse approximation for high-dimensional data classification
IV. Conclusions and Future work
2
![Page 3: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/3.jpg)
Image Restoration Model Image Restoration Problems
Challenges: large-scale & ill-posed
• Denoising, when is identity operator
• Deblurring, when is some blurring operator
• Inpainting, when is some restriction operator
• CT/MR Imaging, when is partial Radon/Fourier
transform
3
![Page 4: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/4.jpg)
Image Restoration Models: A Quick Review
Image restoration:
Variational and Optimization Models
Total variation (TV) and generalizations:
Wavelet frame based:
Others: total generalized variation, low rank, NLM, BM3D, dictionary learning, etc.
PDEs and Iterative Algorithms Perona-Malik equation, shock-filtering (Rudin & Osher), etc
Iterative shrinkage algorithm
What do they have in common?
Shrinkage in sparse domain under transformation!“Dong and Shen, Image restoration: a data-driven perspective, Proceedings of the International Congress of Industrial and Applied Mathematics (ICIAM), 2015”
4
![Page 5: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/5.jpg)
WAVELET FRAME TRANSFORMS
AND DIFFERENTIAL OPERATORS
Bridging discrete and continuum
5
J. Cai, B. Dong, S. Osher and Z. Shen, Image restoration: total variation; wavelet frames; and beyond, Journal of American Mathematical Society, 25(4), 1033-1089, 2012. B. Dong, Q. Jiang and Z. Shen, Image restoration: wavelet frame shrinkage, nonlinear evolution PDEs, and beyond, MMS, 15(1),606-660, 2017. Jian-Feng Cai, B. Dong and Zuowei Shen, Image restorations: a wavelet frame based model for piecewise smooth functions and beyond, Applied and Computational Harmonic Analysis, 41(1), 94-138, 2016. Bin Dong, Zuowei Shen and Peichu Xie, Image restoration: a general wavelet frame based model and its asymptotic analysis, SIAM Journal on Mathematical Analysis, 49(1), 421-445, 2017.
![Page 6: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/6.jpg)
MRA-Based Tight Wavelet Frames Refinable and wavelet functions
Unitary extension principle (UEP)
Discrete 2D transformation:
Perfect reconstruction: Further reading: [Dong and Shen, MRA-Based Wavelet Frames and
Applications, IAS Lecture Notes Series,2011]6
![Page 7: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/7.jpg)
Connections: Motivation
Difference operators in wavelet frame transform:
Thus,
More rigorously [Choi, Dong and Zhang, preprint, 2017]
Haar Filters
Transform
Approximation
7
![Page 8: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/8.jpg)
Connections: Analysis Based Model and Variational Model [Cai, Dong, Osher and Shen, JAMS, 2012]:
The connections give us
Leads to new applications of wavelet frames:
Converges
• Geometric interpretations of the wavelet frame transform (WFT)
• WFT provides flexible and good discretization for differential operators
• Different discretizations affect reconstruction results
• Good regularization should contain differential operators with varied orders (e.g., total
generalized variation [Bredies, Kunisch, and Pock, 2010])
Image segmentation: [Dong, Chien and Shen, 2010] Surface reconstruction from point clouds: [Dong and Shen, 2011]
For any differential operator when proper parameter is chosen.
Standard Discretization Piecewise Linear WFT
8
![Page 9: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/9.jpg)
Relations: Wavelet Shrinkage and Nonlinear PDEs [Dong, Jiang and Shen, MMS, 2017]
Theoretical justification available for quasilinear parabolic equations. Lead to new PDE models such as:
Lead to new wavelet frame shrinkage algorithms:
where
9
![Page 10: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/10.jpg)
Summary
10
VariationalModel
PDEs
Optimization
Iterative Algorithms
Continuum Discrete
![Page 11: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/11.jpg)
Summary
11
J. Cai, B. Dong, S. Osher and Z. Shen, Image restoration: total variation; wavelet frames; and beyond, Journal of American Mathematical Society, 25(4), 1033-1089, 2012.Jian-Feng Cai, B. Dong and Z. Shen, Image restorations: a wavelet frame based model for piecewise smooth functions and beyond, Applied and Computational Harmonic Analysis, 41(1), 94-138, 2016. B. Dong, Z. Shen and P. Xie, Image restoration: a general wavelet frame based model and its asymptotic analysis, SIAM Journal on Mathematical Analysis, 49(1), 421-445, 2017.
B. Dong, Q. Jiang and Z. Shen, Image restoration: wavelet frame shrinkage, nonlinear evolution PDEs, and beyond, Multiscale Modeling & Simulation, 15(1),606-660, 2017.
VariationalModel
PDEs
Optimization
Iterative Algorithms
Continuum Discrete
![Page 12: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/12.jpg)
SPARSE APPROXIMATION IN
HIGH-DIMENSIONAL DATA
CLASSIFICATION
Ning Hao, Bin Dong and Jianqing Fan, Sparsifying the Fisher Linear Discriminant by Rotation, Journal of the Royal Statistical Society Series B, 2015. Bin Dong, Sparse Representation on Graphs by Tight Wavelet Frames and Applications, Applied and Computational Harmonic Analysis, 2015. Bin Dong and Ning Hao, Semi-supervised high dimensional clustering by tight wavelet frames, Proceedings of SPIE,2015.
12
![Page 13: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/13.jpg)
• Network data
• Webpage data
• Text
• Image
• Video
• AudioGame data Webpage data
1.Broad applications
Environmental
Data Sales data
Astronomical data
2.Variety
What is data science? Extracting knowledge from data to make intelligent observations and decisions.
Introduction
![Page 14: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/14.jpg)
- Different area has different focus. Some has tight link with another.
- Broader links?
Data Science
Applied Mathematics
Optimization StatisticsMachine Learning
Deep Theory Applications
Introduction
![Page 15: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/15.jpg)
Importance of the merge:
Combining merits
New insights on classical problems
Introduction
Data Science
Applied Mathematics
Optimization StatisticsMachine Learning
Deep Theory Applications
Algorithms for Data ScienceThinking in depth, aiming for the best.
![Page 16: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/16.jpg)
Introduction
Typical big data set: with huge
Classical v.s. modern
Classical: n<p
Modern: n>p
n
p
p
n
16
![Page 17: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/17.jpg)
Introduction
Typical big data set: with huge
Classical v.s. modern
Classical: n<p Modern: n>p
17
![Page 18: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/18.jpg)
Introduction Sparsity is key What is sparsity for general data sets?
How do we harvest sparsity?
Examples: PCA and its siblings
Low rank approximation
Wavelet frame transform
Dictionary learning
Isomaps, LLE, diffusion maps
Autoencoder
… etc.
Essential information is of much lower dimension than the dimension of the data itself.
Sparse under certain (nonlinear) transformation.
18
![Page 19: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/19.jpg)
MODERN SCENARIONonlinear Classification
Bin Dong, Sparse Representation on Graphs by Tight Wavelet Frames and Applications, Applied and Computational Harmonic Analysis, 2015. Bin Dong and Ning Hao, Semi-supervised high dimensional clustering by tight wavelet frames, Proceedings of SPIE,2015.
19
![Page 20: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/20.jpg)
Nonlinear ClassifierWhen we have enough observations, nonlinear
classifier leads to more accurate classification.
Explicit ClassifierImplicit Classifier
20
![Page 21: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/21.jpg)
PDE Method Ginzburg–Landau (GL) functional [Andrea and Flenner, 2011]
GL model
where
![Page 22: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/22.jpg)
PDE Method Splitting E to a sum of convex and concave parts
Convex splitting scheme
At each iteration, we need to solve a Laplace equation on graph. Fast graph Laplacian solver is needed, such as Nystrom’smethod.
![Page 23: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/23.jpg)
Key idea: Eigenfunctions of Laplace-Beltrami operator (graph Laplacian in discrete setting) are understood as Fourier basis on manifolds (graphs in discrete setting) and the associated eigenvalues as frequency components.
Spectrum of Laplace-Beltrami operator on
Fourier transform Plancherel and Parseval’s identities
Eigenvalues and eigenfunctions:
Wavelet Frame Method
![Page 24: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/24.jpg)
Wavelet Frame Method Asymptotic properties of eigenfunctions and
eigenvalues:
Wavelet system (semi-continuous) on manifold :
Question: how to construct so that is a tight frame on ?
Eigenvalues Eigenfunctions
Dilation Translation
• Weyl’s asymptotic formula(1912):
• Uniform bound (Grieser, 2002):
![Page 25: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/25.jpg)
Wavelet Frame Method Further restriction on :
Question: how to construct so that is a tight frame on ?
Benefits of such restriction Grants a natural transition from continuum to discrete setting
Makes construction of tight frames on manifolds/graphs painless
Grants fast decomposition and reconstruction algorithms (Chebyshev polynomial approximation)
Given and
let
![Page 26: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/26.jpg)
Wavelet Frame Method Sparsity based semi-supervised learning models
Transform W is the fast tight wavelet frame transform on graphs [B. Dong, ACHA, 2015].
Model L2:
Exact Model:
Robust Model:
26
![Page 27: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/27.jpg)
Wavelet Frame Method Classification – Real Datasets
o MNIST data set (http://yann.lecun.com/exdb/
mnist/)o Banknote authentication dataset (UCI machine learning repository)
Results: Model L2 [Dong, ACHA, 2015]
• Max-Flow & PAL : [Merkurjev, Bae, Bertozzi, and Tai, preprint, 2014]• Binary MBO: [Merkurjev, Kostic, and Bertozzi, 2013]• GL: [Bertozzi and Flenner, 2012]
27
~3-5% label
![Page 28: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/28.jpg)
Further Studies of Wavelet Frame Transform on Graphs High dimensional classification [Dong and Hao, SPIE 2015]
V.S. LDA methods: Leukemia (n=72, p=7129) and Lung (n=181, p=12533)
Application in super-resolution diffusion MRI [Yap, Dong, Zhang, Shen, MICCAI 2016]
Error % (Std %) NSC IR ROAD RS-ROAD Exact Model
Leukemia 8.51 (3.0) 4.27 (8.4) 6.35 (6.0) 4.46 (3.1) 5.57 (4.2)
Lung Cancer 10.44 (1.4) 3.47 (7.3) 1.37 (1.1) 0.93 (0.9) 0.59 (0.6)
−30
−20
−10
0
10
20
−30−20−10010203040−40
−30
−20
−10
0
10
20
30
Leukemia
−40
−20
0
20
40
−40
−20
0
20
40−30
−20
−10
0
10
20
30
Lung
28
![Page 29: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/29.jpg)
CONCLUDING REMARKS
29
![Page 30: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/30.jpg)
Conclusions and Future Work Conclusions
o Bridging wavelet frame transforms and differential operators
o New insights, models/algorithms and applications
o Sparse approximation for general data analysis
Future worko Idea of “end-to-end” in classical problems such as imaging
o Learning PDEs from data
30
![Page 31: Sparse Approximation: from Image Restoration to High ...ims.nus.edu.sg/events/2017/data/files/bindong2.pdfWavelet frame transforms and differential operators under variational and](https://reader033.vdocuments.mx/reader033/viewer/2022051901/5fefb4b42db4fc0c42305803/html5/thumbnails/31.jpg)
Thanks for Your Attentionand
Questions?
http://bicmr.pku.edu.cn/~dongbin
31