sparse and overcomplete representations
TRANSCRIPT
![Page 1: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/1.jpg)
Machine Learning for Signal Processing
Sparse and Overcomplete Representations
Bhiksha Raj (slides from Sourish Chaudhuri)
Oct 18, 2016
1
![Page 2: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/2.jpg)
Key Topics in this Lecture
• Basics – Component-based representations
– Overcomplete and Sparse Representations,
– Dictionaries
• Pursuit Algorithms
• How to learn a dictionary
• Why is an overcomplete representation powerful?
Sparse and Overcomplete Representations 2
![Page 3: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/3.jpg)
Representing Data
Dictionary (codebook)
3 Sparse and Overcomplete Representations
![Page 4: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/4.jpg)
Representing Data
Atoms Dictionary
4 Sparse and Overcomplete Representations
![Page 5: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/5.jpg)
Representing Data
Atoms Dictionary
Each atom is a basic unit that can be used to “compose” larger units.
5 Sparse and Overcomplete Representations
![Page 6: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/6.jpg)
Representing Data
Atoms
6 Sparse and Overcomplete Representations
![Page 7: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/7.jpg)
Representing Data
Atoms
7 Sparse and Overcomplete Representations
![Page 8: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/8.jpg)
Representing Data
Atoms
8 Sparse and Overcomplete Representations
![Page 9: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/9.jpg)
Representing Data
9 Sparse and Overcomplete Representations
![Page 10: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/10.jpg)
Representing Data
10 Sparse and Overcomplete Representations
![Page 11: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/11.jpg)
Representing Data
11 Sparse and Overcomplete Representations
![Page 12: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/12.jpg)
Representing Data
6.44
0.004
12.19
4.00
9.12
0.02
0.01
…………. 12 Sparse and Overcomplete Representations
![Page 13: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/13.jpg)
Representing Data
w1
………….
w6
w4
w3
w2
w5
w7
13 Sparse and Overcomplete Representations
![Page 14: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/14.jpg)
Representing Data
=
14 Sparse and Overcomplete Representations
![Page 15: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/15.jpg)
Representing Data
=
15 Sparse and Overcomplete Representations
![Page 16: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/16.jpg)
Representing Data
= D α X
Input 16 Sparse and Overcomplete Representations
![Page 17: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/17.jpg)
Representing Data
= D α X
Unknown
Input 17 Sparse and Overcomplete Representations
![Page 18: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/18.jpg)
Quick Linear Algebra Refresher
• Remember, #(Basis Vectors)= #unknowns
D.α = X
Basis Vectors (from Dictionary)
Input data
Weights
18 Sparse and Overcomplete Representations
![Page 19: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/19.jpg)
Overcomplete Representations
• What is the dimensionality of the input image? (say 64x64 image)
• What is the dimensionality of the dictionary? (each image = 64x64 pixels)
4096
19 Sparse and Overcomplete Representations
4096 x N
![Page 20: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/20.jpg)
Overcomplete Representations
• What is the dimensionality of the input image? (say 64x64 image)
• What is the dimensionality of the dictionary? (each image = 64x64 pixels)
4096
4096 x N
???
20 Sparse and Overcomplete Representations
![Page 21: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/21.jpg)
Overcomplete Representations
• What is the dimensionality of the input image? (say 64x64 image)
• What is the dimensionality of the dictionary? (each image = 64x64 pixels)
4096
VERY LARGE!!!
21 Sparse and Overcomplete Representations
4096 x N
![Page 22: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/22.jpg)
Overcomplete Representations
• What is the dimensionality of the input image? (say 64x64 image)
• What is the dimensionality of the dictionary? (each image = 64x64 pixels)
4096
VERY LARGE!!!
If N > 4096 (as it likely is) we have an overcomplete representation
22 Sparse and Overcomplete Representations
4096 x N
![Page 23: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/23.jpg)
Overcomplete Representations
• What is the dimensionality of the input image? (say 64x64 image)
• What is the dimensionality of the dictionary? (each image = 64x64 pixels)
4096
VERY LARGE!!!
More generally:
If #(basis vectors) > dimensions of input
we have an overcomplete representation
23 Sparse and Overcomplete Representations
4096 x N
![Page 24: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/24.jpg)
Quick Linear Algebra Refresher
• Remember, #(Basis Vectors)= #unknowns
D.α = X
Basis Vectors (from Dictionary)
Input data
Weights
24 Sparse and Overcomplete Representations
![Page 25: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/25.jpg)
• Conventional characterization: Images
– Store Pixel values (positions implicit)
• Image: 128 x pixel.at.(1,1) + 134 x pixel.at.(1,2) + … + 127 x pixel.at.(2,1)….
• Store only the numbers (128,134, …, 127)
• Bases are “pixel.at.(1,1)”, “pixel.at.(1,2)” etc..
– Or rather [1 0 0 0… ], [0 1 0 0 …]
– Pixel positions are implicit
Sparse and Overcomplete Representations 25
Recap – Conventional : Images
![Page 26: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/26.jpg)
Recap – Conventional : Images
• Storing an Image
Sparse and Overcomplete Representations 26
• “Bases” are unit-valued pixels at specific locations
– Only weights are stored
– Basis matrix is implicit (everyone knows what it is)
B.P = Image Pixel position
(rows)
Bases
(columns)
Stored
Pixel values
Actual image
No. of bases = dimension of bases Basis matrix is square
![Page 27: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/27.jpg)
Recap – Conventional: Sound
• Signal = 3 x sample.at.t=1 + 4 x sample.at.t=2 +
3.5 x sample.at.t=3 ….
• Store only the numbers [3, 4, 3.5…]
• Bases are “sample.at.t=1”, “sample.at.t=2”, …
– Or rather [..0 1 0 0 0 …], [..0 0 1 0 0 0 …], ….
– “Time” is implicit
Sparse and Overcomplete Representations 27
t
![Page 28: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/28.jpg)
Recap – Conventional : Sound
• Storing a sound
Sparse and Overcomplete Representations 28
• “Bases” are unit-valued samples at specific time instants
– Only weights are stored
– Basis matrix is implicit (everyone knows what it is)
B.S = Recording
Sample position
(rows)
Bases
(columns)
Stored
signal values
Actual signal
t
No. of bases = dimension of bases Basis matrix is square
![Page 29: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/29.jpg)
Recap: Component-based representations
Sparse and Overcomplete Representations 29
Bases
(columns)
Stored
signal values
Actual signal
• Bases may be deterministic, e.g. sinusoids/wavelets or
derived, e.g. PCA / ICA / NMF bases
• Only store w to represent individual signals. Bases
matrix B stored separately as a one-time deal
Typically, no. of bases (columns) is no more than no. of dimensions Basis matrix is tall or square
![Page 30: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/30.jpg)
Dictionary based Representations
= D α X
Input 30 Sparse and Overcomplete Representations
• Overcomplete “dictionary”-based representations are composition-based representations with more bases than the dimensionality of the data
Bases matrix is wide (more bases than dimensions)
![Page 31: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/31.jpg)
Why Dictionary-based Representations?
• Dictionary based representations are semantically more meaningful
• Enable content-based description – Bases can capture entire structures in data
– E.g. notes in music
– E.g. image structures (such as faces) in images
• Enable content-based processing – Reconstructing, separating, denoising, manipulating speech/music
signals
– Coding, compression, etc.
• Statistical reasons: We will get to that shortly..
Sparse and Overcomplete Representations 31
![Page 32: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/32.jpg)
Problems
= D α X
Input 32 Sparse and Overcomplete Representations
• How to obtain the dictionary
– Which will give us meaningful representations
• How to compute the weights?
![Page 33: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/33.jpg)
Problems
= D α X
Input 33 Sparse and Overcomplete Representations
• How to obtain the dictionary
– Which will give us meaningful representations
• How to compute the weights?
![Page 34: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/34.jpg)
Quick Linear Algebra Refresher
• Remember, #(Basis Vectors)= #unknowns
D.α = X
Basis Vectors
Input data
Unknowns
34 Sparse and Overcomplete Representations
![Page 35: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/35.jpg)
Quick Linear Algebra Refresher
• Remember, #(Basis Vectors)= #unknowns
Basis Vectors
Input data
Unknowns
When can we solve for α?
D.α = X
35 Sparse and Overcomplete Representations
![Page 36: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/36.jpg)
Quick Linear Algebra Refresher
• When #(Basis Vectors) = dim(Input Data), we have a unique solution
• When #(Basis Vectors) < dim(Input Data), we may have no exact solution
• When #(Basis Vectors) > dim(Input Data), we have infinitely many solutions
D.α = X
36 Sparse and Overcomplete Representations
![Page 37: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/37.jpg)
Quick Linear Algebra Refresher
• When #(Basis Vectors) = dim(Input Data), we have a unique solution
• When #(Basis Vectors) < dim(Input Data), we may have no solution
• When #(Basis Vectors) > dim(Input Data), we have infinitely many solutions
D.α = X
37 Sparse and Overcomplete Representations
Our Case
![Page 38: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/38.jpg)
Overcomplete Representation
= D α X
Unknown
#(Basis Vectors) > dimensions of the input 38 Sparse and Overcomplete Representations
![Page 39: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/39.jpg)
Overcompleteness and Sparsity
• To solve an overcomplete system of the type:
• Make assumptions about the data.
• Suppose, we say that X is composed of no more than a fixed number (k) of “bases” from D (k ≤ dim(X))
– The term “bases” is an abuse of terminology..
• Now, we can find the set of k bases that best fit the data point, X.
D.α = X
39 Sparse and Overcomplete Representations
![Page 40: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/40.jpg)
Representing Data
Using bases that we know…
But no more than k=4 bases
40 Sparse and Overcomplete Representations
![Page 41: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/41.jpg)
Overcompleteness and Sparsity
Atoms
But no more than k=4 bases are “active”
41 Sparse and Overcomplete Representations
![Page 42: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/42.jpg)
Overcompleteness and Sparsity
Atoms
But no more than k=4 bases
42 Sparse and Overcomplete Representations
![Page 43: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/43.jpg)
No more than 4 bases
= D
α
X
43 Sparse and Overcomplete Representations
0 0
0 0
0
0
0
0
0
0 0
0
![Page 44: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/44.jpg)
No more than 4 bases
= D
α
X
44 Sparse and Overcomplete Representations
0 0
0 0
0
0
0
0
0
0 0
0
ONLY THE a COMPONENTS CORRESPONDING TO THE 4 KEY DICTIONARY ENTRIES ARE NON-ZERO
![Page 45: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/45.jpg)
No more than 4 bases
= D
α
X
45 Sparse and Overcomplete Representations
0 0
0 0
0
0
0
0
0
0 0
0
ONLY THE a COMPONENTS CORRESPONDING TO THE 4 KEY DICTIONARY ENTRIES ARE NON-ZERO
MOST OF a IS ZERO!! a IS SPARSE
![Page 46: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/46.jpg)
Sparsity- Definition
• Sparse representations are representations that account for most or all information of a signal with a linear combination of a small number of atoms.
(from: www.see.ed.ac.uk/~tblumens/Sparse/Sparse.html)
46 Sparse and Overcomplete Representations
![Page 47: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/47.jpg)
The Sparsity Problem
• We don’t really know k
• You are given a signal X
• Assuming X was generated using the dictionary, can we find α that generated it?
47 Sparse and Overcomplete Representations
![Page 48: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/48.jpg)
The Sparsity Problem
• We want to use as few basis vectors as possible to do this.
Mina
a0
s.t. X Da
48 Sparse and Overcomplete Representations
![Page 49: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/49.jpg)
The Sparsity Problem
• We want to use as few basis vectors as possible to do this.
Mina
a0
s.t. X Da
49 Sparse and Overcomplete Representations
Counts the number of non-zero elements in α
![Page 50: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/50.jpg)
The Sparsity Problem
• We want to use as few basis vectors as possible to do this
– Ockham’s razor: Choose the simplest explanation invoking the fewest variables
Mina
a0
s.t. X Da
50 Sparse and Overcomplete Representations
![Page 51: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/51.jpg)
The Sparsity Problem
• We want to use as few basis vectors as possible to do this.
Mina
a0
s.t. X Da
How can we solve the above?
51 Sparse and Overcomplete Representations
![Page 52: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/52.jpg)
Obtaining Sparse Solutions
• We will look at 2 algorithms:
– Matching Pursuit (MP)
– Basis Pursuit (BP)
Sparse and Overcomplete Representations 52
![Page 53: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/53.jpg)
Matching Pursuit (MP)
• Greedy algorithm
• Finds an atom in the dictionary that best matches the input signal
• Remove the weighted value of this atom from the signal
• Again, find an atom in the dictionary that best matches the remaining signal.
• Continue till a defined stop condition is satisfied.
53 Sparse and Overcomplete Representations
![Page 54: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/54.jpg)
Matching Pursuit
• Find the dictionary atom that best matches the given signal.
Weight = w1
54 Sparse and Overcomplete Representations
![Page 55: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/55.jpg)
Matching Pursuit
• Remove weighted image to obtain updated signal
Find best match for this signal from the dictionary
55 Sparse and Overcomplete Representations
![Page 56: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/56.jpg)
Matching Pursuit
• Find best match for updated signal
56 Sparse and Overcomplete Representations
![Page 57: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/57.jpg)
Matching Pursuit
• Find best match for updated signal
Iterate till you reach a stopping condition, norm(ResidualInputSignal) < threshold
57 Sparse and Overcomplete Representations
![Page 58: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/58.jpg)
Matching Pursuit
From http://en.wikipedia.org/wiki/Matching_pursuit
58 Sparse and Overcomplete Representations
![Page 59: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/59.jpg)
Matching Pursuit
• Problems ???
59 Sparse and Overcomplete Representations
![Page 60: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/60.jpg)
Matching Pursuit
• Main Problem
– Computational complexity
– The entire dictionary has to be searched at every iteration
60 Sparse and Overcomplete Representations
![Page 61: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/61.jpg)
Comparing MP and BP
Matching Pursuit Basis Pursuit
Hard thresholding
Greedy optimization at each step
Weights obtained using greedy rules
(remember the equations)
61 Sparse and Overcomplete Representations
![Page 62: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/62.jpg)
Basis Pursuit (BP)
• Remember,
Mina
a0
s.t. X Da
62 Sparse and Overcomplete Representations
![Page 63: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/63.jpg)
Basis Pursuit
• Remember,
Mina
a0
s.t. X Da
In the general case, this is intractable
63 Sparse and Overcomplete Representations
![Page 64: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/64.jpg)
Basis Pursuit
• Remember,
Mina
a0
s.t. X Da
In the general case, this is intractable
64 Sparse and Overcomplete Representations
Requires combinatorial optimization
![Page 65: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/65.jpg)
Basis Pursuit
• Replace the intractable expression by an expression that is solvable
Mina
a1
s.t. X Da
65 Sparse and Overcomplete Representations
![Page 66: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/66.jpg)
Basis Pursuit
• Replace the intractable expression by an expression that is solvable
Mina
a1
s.t. X Da
66 Sparse and Overcomplete Representations
This will provide identical solutions when D obeys the Restricted
Isometry Property.
![Page 67: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/67.jpg)
Basis Pursuit
• Replace the intractable expression by an expression that is solvable
Mina
a1
s.t. X Da
67 Sparse and Overcomplete Representations
Constraint
Objective
![Page 68: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/68.jpg)
Basis Pursuit
• We can formulate the optimization term as:
Mina{ X Da
2 a
1}
68 Sparse and Overcomplete Representations
Constraint Objective
![Page 69: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/69.jpg)
Basis Pursuit
• We can formulate the optimization term as:
Mina{ X Da
2 a
1}
λ is a penalty term on the non-zero elements and promotes sparsity
69 Sparse and Overcomplete Representations
![Page 70: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/70.jpg)
Basis Pursuit
• We can formulate the optimization term as:
Mina{ X Da
2 a
1}
λ is a penalty term on the non-zero elements and promotes sparsity
Equivalent to LASSO; for more details, see this paper by Tibshirani http://www-stat.stanford.edu/~tibs/ftp/lasso.ps
70 Sparse and Overcomplete Representations
![Page 71: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/71.jpg)
Basis Pursuit
• ||a||1 is not differentiable at aj= 0
• Gradient of ||a||1 for gradient descent update
• At optimum, following conditions hold
Sparse and Overcomplete Representations 71
}{1
2aa
aDXMin
0at 1
0at [-1,1]
0at 11
j
j
j
j a
a
a
a
a
0 if ,
0 if ,0)(2
2
jj
jjj
X
signX
aa
aaa
D
D
![Page 72: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/72.jpg)
Basis Pursuit
• There are efficient ways to solve the LASSO formulation.
– http://web.stanford.edu/~hastie/glmnet_matlab/
• Simplest solution: Coordinate descent algorithms
– On webpage..
72 Sparse and Overcomplete Representations
![Page 73: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/73.jpg)
L1 vs L0
• L0 minimization
– Two-sparse solution
– ANY pair of bases can explain X with 0 error 73
a
aa
DXts
Min
..
0
Overcomplete set of 6 bases
X
![Page 74: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/74.jpg)
L1 vs L0
• L1 minimization
– Two-sparse solution
– All else being equal, the two closest bases are chosen Sparse and Overcomplete Representations 74
a
aa
DXts
Min
..
1
Overcomplete set of 6 bases
X
![Page 75: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/75.jpg)
Comparing MP and BP
Matching Pursuit Basis Pursuit
Hard thresholding
Soft thresholding
Greedy optimization at each step
Global optimization
Weights obtained using greedy rules
Can force N-sparsity with appropriately
chosen weights
(remember the equations)
75 Sparse and Overcomplete Representations
![Page 76: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/76.jpg)
General Formalisms
• L0 minimization
• L0 constrained optimization
• L1 minimization
• L1 constrained optimization
Sparse and Overcomplete Representations 76
a
aa
DXts
Min
..
0
a
aa
DXts
Min
..
1
Cts
XMin
0
2
2
.. a
aa
D
Cts
XMin
1
2
2
.. a
aa
D
![Page 77: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/77.jpg)
Many Other Methods..
• Iterative Hard Thresholding (IHT)
• CoSAMP
• OMP
• …
77 Sparse and Overcomplete Representations
![Page 78: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/78.jpg)
Problems
= D α X
Input 78 Sparse and Overcomplete Representations
• How to obtain the dictionary
– Which will give us meaningful representations
• How to compute the weights?
![Page 79: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/79.jpg)
Dictionaries: Compressive Sensing
• Just random vectors!
79 Sparse and Overcomplete Representations
D
![Page 80: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/80.jpg)
More Structured ways of Constructing Dictionaries
• Dictionary entries must be structurally “meaningful”
– Represent true compositional units of data
• Have already encountered two ways of building dictionaries
– NMF for non-negative data
– K-means ..
80 Sparse and Overcomplete Representations
![Page 81: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/81.jpg)
K-Means for Composing Dictionaries
• Every vector is approximated by the centroid of the cluster it falls into
• Cluster means are “codebook” entries
– Dictionary entries
– Also compositional units the compose the data
81 Sparse and Overcomplete Representations
Train the codebook from training data using K-means
![Page 82: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/82.jpg)
K-Means for Dictionaries
• a must be 1 sparse
• Only a entry must 1
82 Sparse and Overcomplete Representations
Each column is a codeword (centroid) from the codebook
D α X
0 0
0 0
0
0
0
0
0
0
1
0
0
0
|| ∝ ||𝟎 = 𝟏 || ∝ ||𝟏 = 𝟏
![Page 83: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/83.jpg)
K-Means
• Learn Codewords to minimize the total squared length of the training vectors from the closest codeword
83 Sparse and Overcomplete Representations
![Page 84: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/84.jpg)
Length-unconstrained K-Means for Dictionaries
• a must be 1 sparse
• No restriction on a value
84 Sparse and Overcomplete Representations
Each column is a codeword (centroid) from the codebook
D α X
0 0
0 0
0
0
0
0
0
0
3
0
0
0
|| ∝ ||𝟎 = 𝟏
![Page 85: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/85.jpg)
SVD K-Means
• Learn Codewords to minimize the total squared projection error of the training vectors from the closest codeword
85 Sparse and Overcomplete Representations
![Page 86: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/86.jpg)
SVD K–means
86
1. Initialize a set of centroids randomly
2. For each data point x, find the projection from the centroid for each cluster
•
3. Put data point in the cluster of the closest centroid
• Cluster for which pcluster is maximum
4. When all data points are clustered, recompute centroids
5. If not converged, go back to 2
cluster
T
cluster mxp
})|({rEigenvecto Principal clusterxxmcluster
![Page 87: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/87.jpg)
Problem
• Only represents Radial patterns
87 Sparse and Overcomplete Representations
![Page 88: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/88.jpg)
What about this pattern?
• Dictionary entries that represent radial patterns will not capture this structure
– 1-sparse representations will not do 88
![Page 89: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/89.jpg)
What about this pattern?
• We need AFFINE patterns
Sparse and Overcomplete Representations 89
![Page 90: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/90.jpg)
What about this pattern?
• We need AFFINE patterns
• Each vector is modeled by a linear combination of K (here 2) bases 90
![Page 91: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/91.jpg)
What about this pattern?
• We need AFFINE patterns
• Each vector is modeled by a linear combination of K (here 2) bases 91
Every line is a
(constrined) combination
of two bases
2-sparse
Constraint:
Line = a.b1 + (1-a)b2
![Page 92: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/92.jpg)
Codebooks for K sparsity?
• a must be k sparse
• No restriction on a value
92 Sparse and Overcomplete Representations
Each column is a codeword (centroid) from the codebook
D α X
0 0
0 0
0
0
0
0
0
3
0
0
|| ∝ ||𝟎 = 𝒌
1
4
![Page 93: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/93.jpg)
K SVD
• Initialize Codebook
1. For every vector, compute K-sparse alphas
– Using any pursuit algorithm
93 Sparse and Overcomplete Representations
0 0 2 0 0
1 0 0 0 0
0 0 0 1 0
0 7 0 0 0
0 0 0 1 0
3 0 0 0 0
0 5 0 0 1
0 0 1 0 2
a =
D =
![Page 94: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/94.jpg)
K-SVD
2. For each codeword (k):
• For each vector x
• Subtract the contribution of all other codewords to obtain ek(x)
• Codeword-specific residual
• Compute the principal Eigen vector of {ek(x)}
3. Return to step 1
94 Sparse and Overcomplete Representations
0 0 2 0 0
1 0 0 0 0
0 0 0 1 0
0 7 0 0 0
0 0 0 1 0
3 0 0 0 0
0 5 0 0 1
0 0 1 0 2
𝑒𝑘 𝑥 = 𝑥- ∝𝑗 𝐷𝑗𝑗≠𝑘
a =
D =
Dj j != 1
aj(x) j != 1
![Page 95: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/95.jpg)
K-SVD
• Termination of each iteration: Updated dictionary
• Conclusion: A dictionary where any data vector can be composed of at most K dictionary entries
– More generally, sparse composition
95 Sparse and Overcomplete Representations
D
![Page 96: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/96.jpg)
Problems
= D α X
Input 97 Sparse and Overcomplete Representations
• How to obtain the dictionary
– Which will give us meaningful representations
• How to compute the weights?
![Page 97: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/97.jpg)
Applications of Sparse Representations
• Many many applications
– Signal representation
– Statistical modelling
– ..
– We’ve seen one: Compressive sensing
• Another popular use
– Denoising
98 Sparse and Overcomplete Representations
![Page 98: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/98.jpg)
Denoising
• As the name suggests, remove noise!
99 Sparse and Overcomplete Representations
![Page 99: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/99.jpg)
Denoising
• As the name suggests, remove noise!
• We will look at image denoising as an example
100 Sparse and Overcomplete Representations
![Page 100: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/100.jpg)
Image Denoising
• Here’s what we want
101 Sparse and Overcomplete Representations
![Page 101: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/101.jpg)
Image Denoising
• Here’s what we want
102 Sparse and Overcomplete Representations
![Page 102: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/102.jpg)
Image Denoising
• Here’s what we want
103 Sparse and Overcomplete Representations
![Page 103: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/103.jpg)
The Image Denoising Problem
• Given an image
• Remove Gaussian additive noise from it
104 Sparse and Overcomplete Representations
![Page 104: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/104.jpg)
Image Denoising
Y = X + ε
Noisy Input Orig. Image
Gaussian Noise ε = Ν(0, σ)
105 Sparse and Overcomplete Representations
![Page 105: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/105.jpg)
Image Denoising
• Remove the noise from Y, to obtain X as best as possible.
106 Sparse and Overcomplete Representations
![Page 106: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/106.jpg)
Image Denoising
• Remove the noise from Y, to obtain X as best as possible
• Using sparse representations over learned dictionaries
107 Sparse and Overcomplete Representations
![Page 107: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/107.jpg)
Image Denoising
• Remove the noise from Y, to obtain X as best as possible
• Using sparse representations over learned dictionaries
• We will learn the dictionaries
108 Sparse and Overcomplete Representations
![Page 108: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/108.jpg)
Image Denoising
• Remove the noise from Y, to obtain X as best as possible
• Using sparse representations over learned dictionaries
• We will learn the dictionaries
• What data will we use? The corrupted image itself!
109 Sparse and Overcomplete Representations
![Page 109: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/109.jpg)
Image Denoising
• We use the data to be denoised to learn the dictionary.
• Training and denoising become an iterated process.
• We use image patches of size √n x √n pixels (i.e. if the image is 64x64, patches are 8x8)
110 Sparse and Overcomplete Representations
![Page 110: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/110.jpg)
Image Denoising
• The data dictionary D
– Size = n x k (k > n)
– This is known and fixed, to start with
– Every image patch can be sparsely represented using D
111 Sparse and Overcomplete Representations
![Page 111: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/111.jpg)
Image Denoising
• Recall our equations from before.
• We want to find α so as to minimize the value of the equation below:
Mina{ X Da
2 a
0}
112 Sparse and Overcomplete Representations
}{1
2aa
aDXMin
![Page 112: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/112.jpg)
Image Denoising
• In the above, X is a patch.
Mina{ X Da
2 a
1}
113 Sparse and Overcomplete Representations
![Page 113: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/113.jpg)
Image Denoising
• In the above, X is a patch.
• If the larger image is fully expressed by the every patch in it, how can we go from patches to the image?
Mina{ X Da
2 a
1}
114 Sparse and Overcomplete Representations
![Page 114: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/114.jpg)
Mina ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
115 Sparse and Overcomplete Representations
![Page 115: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/115.jpg)
Mina ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
(X – Y) is the error between the input and denoised image. μ is a penalty on the error.
116 Sparse and Overcomplete Representations
![Page 116: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/116.jpg)
Mina ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
Error bounding in each patch -Rij selects the (ij)th patch -Terms in summation = no. of patches
117 Sparse and Overcomplete Representations
![Page 117: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/117.jpg)
Mina ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
λ forces sparsity
118 Sparse and Overcomplete Representations
![Page 118: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/118.jpg)
Image Denoising
• But, we don’t “know” our dictionary D.
• We want to estimate D as well.
119 Sparse and Overcomplete Representations
![Page 119: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/119.jpg)
Image Denoising
• But, we don’t “know” our dictionary D.
• We want to estimate D as well.
MinD,a ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
We can use the previous equation itself!!!
120 Sparse and Overcomplete Representations
![Page 120: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/120.jpg)
Image Denoising
MinD,a ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
How do we estimate all 3 at once?
121 Sparse and Overcomplete Representations
![Page 121: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/121.jpg)
Image Denoising
MinD,a ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
How do we estimate all 3 at once?
We cannot estimate them at the same time!
122 Sparse and Overcomplete Representations
![Page 122: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/122.jpg)
Image Denoising
MinD,a ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
How do we estimate all 3 at once?
Fix 2, and find the optimal 3rd.
123 Sparse and Overcomplete Representations
![Page 123: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/123.jpg)
Image Denoising
MinD,a ij ,X
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Initialize X = Y
124 Sparse and Overcomplete Representations
![Page 124: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/124.jpg)
Image Denoising
Mina ij
{ X Y2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Initialize X = Y, initialize D
0
You know how to solve the remaining portion for α – MP, BP!
125 Sparse and Overcomplete Representations
![Page 125: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/125.jpg)
Image Denoising
• Now, update the dictionary D.
• Update D one column at a time, following the K-SVD algorithm
• K-SVD maintains the sparsity structure
126 Sparse and Overcomplete Representations
![Page 126: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/126.jpg)
Image Denoising
• Now, update the dictionary D.
• Update D one column at a time, following the K-SVD algorithm
• K-SVD maintains the sparsity structure
• Iteratively update α and D
127 Sparse and Overcomplete Representations
![Page 127: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/127.jpg)
Image Denoising
Sparse and Overcomplete Representations 128
Learned Dictionary for Face Image denoising
From: M. Elad and M. Aharon, Image denoising via learned dictionaries and sparse representation, CVPR, 2006.
![Page 128: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/128.jpg)
MinX{ X Y
2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
We know D and α
Const. wrt X
The quadratic term above has a closed-form solution
129 Sparse and Overcomplete Representations
![Page 129: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/129.jpg)
MinX{ X Y
2
2 RijX Da ij
2
2
ij
ij a ij0
ij
}
Image Denoising
We know D and α
Const. wrt X
ij
ij
T
ij
ij
ij
T
ij DRYRRIX )()( 1 a
130 Sparse and Overcomplete Representations
![Page 130: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/130.jpg)
Image Denoising
• Summarizing… We wanted to obtain 3 things
Sparse and Overcomplete Representations 131
![Page 131: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/131.jpg)
Image Denoising
• Summarizing… We wanted to obtain 3 things
Weights α
Dictionary D
Denoised Image X
Sparse and Overcomplete Representations 132
![Page 132: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/132.jpg)
Image Denoising
• Summarizing… We wanted to obtain 3 things
Weights α – Your favorite pursuit algorithm
Dictionary D – Using K-SVD
Denoised Image X
Sparse and Overcomplete Representations 133
![Page 133: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/133.jpg)
Image Denoising
• Summarizing… We wanted to obtain 3 things
Weights α – Your favorite pursuit algorithm
Dictionary D – Using K-SVD
Denoised Image X
Sparse and Overcomplete Representations 134
Iterating
![Page 134: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/134.jpg)
Image Denoising
• Summarizing… We wanted to obtain 3 things
Weights α
Dictionary D
Denoised Image X- Closed form solution
Sparse and Overcomplete Representations 135
![Page 135: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/135.jpg)
Image Denoising
• Here’s what we want
136 Sparse and Overcomplete Representations
![Page 136: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/136.jpg)
Image Denoising
• Here’s what we want
137 Sparse and Overcomplete Representations
![Page 137: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/137.jpg)
Comparing to Other Techniques
Non-Gaussian data
PCA of ICA Images from Lewicki and Sejnowski, Learning Overcomplete Representations, 2000.
138 Sparse and Overcomplete Representations
Which is which?
![Page 138: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/138.jpg)
Comparing to Other Techniques
Non-Gaussian data
PCA ICA
139 Sparse and Overcomplete Representations
Images from Lewicki and Sejnowski, Learning Overcomplete Representations, 2000.
![Page 139: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/139.jpg)
Comparing to Other Techniques
Non-Gaussian data
PCA ICA
Predicts data here
Doesn’t predict data here
Does pretty well
140 Sparse and Overcomplete Representations
Images from Lewicki and Sejnowski, Learning Overcomplete Representations, 2000.
![Page 140: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/140.jpg)
Comparing to Other Techniques
Data still in 2-D space
ICA Overcomplete
Doesn’t capture the underlying representation, which Overcomplete representations can do…
141 Sparse and Overcomplete Representations
![Page 141: Sparse and Overcomplete Representations](https://reader031.vdocuments.mx/reader031/viewer/2022022804/621b0f85469538214b3ae9dc/html5/thumbnails/141.jpg)
Summary
• Overcomplete representations can be more powerful than component analysis techniques.
• Dictionary can be learned from data.
• Relative advantages and disadvantages of the pursuit algorithms.
142 Sparse and Overcomplete Representations