sparse representation and compressive sensing

Advanced Signal Processing

Sparse Representation and Compressive Sensing

Dr. M. Sabarimalai Manikandan

Assistant ProfessorCenter for Excellence in Computational Engineering and Networking

Amrita University, Coimbatore CampusE-mail: [email protected]

September 16, 2011

Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing


Why Signal Processing?

◮ Most natural signals are non-stationary and have highlycomplex time-varying spectro-temporal characteristics.

◮ Mixture of many sources

◮ Composition of mixed events

◮ Various kinds of noise and artifacts

◮ The SP is challenging task because the natural signals aretypically having different shapes, amplitudes, durations andfrequency content, which are not known in many differentapplications and systems



Signal Representation Using Basis Functions

◮ A set {ψn}Nn=1 is called a orthonormal basis for RN if thevectors in the set span R

N and are linearly independent

◮ Let x ∈ RN×1 be the input signal that is spanned by N basis

functions {ψn}Nn=1. Then, a discrete-time signal x can berepresented as

x =

N∑

n=1

αnψn = Ψα (1)

where α = [α1, α2, α3, ......αN ] is the transform coefficientsvector that is computed as αn = 〈x ,ψn〉.

◮ For some transform matrix, the transform coefficients vector αhas a small number of large amplitude coefficients and a largenumber of small amplitude coefficients



Some of Representation or Transform Matrices

◮ Fourier transform matrix

◮ discrete cosines (DCT matrix) and discrete sines (DST matrix)

◮ Haar transform matrix

◮ wavelet and wavelet packets matrices

◮ Gabor filters

◮ curvelets, ridgelets, contourlets, bandelets, shearlets

◮ directionlets, grouplets, chirplets

◮ Hermite polynomials, and so on



Limitations of Fixed Representation Matrix

◮ The Fourier transform is suitable for analysis of thesteady-state sinusoidal signals but it fails to capture the sharpchanges and discontinuities in the signals.

◮ In the STFT-based methods, the choices for widths of thetime-window affect the frequency and time resolution.

◮ The common problem in well-known wavelet transform-basedmethods is which mother wavelet function and characteristicscale provides the best time-frequency resolution for detectionof transients and non-transients.



Sparse Representation/Recovery

◮ Definition: The sparse representation theory has shown thatsparse signals can be exactly reconstructed from a smallnumber of elementary signals (or atoms).

◮ The sparse representation of natural signals can be achievedby exploiting its sparsity or compressibility.

◮ A natural signal is said to be sparse signal if that can becompactly expressed as a linear combination of a few smallnumber of basis vectors.

◮ Sparse representation has become an invaluable tool ascompared to direct time-domain and transform-domain signalprocessing methods.



Sparse Representation: Applications

◮ audio/image/video processing tasks (compression, denoising,deblurring, inpainting, and superresolution)

◮ speech enhancement and recognition

◮ signal detection and classification

◮ face recognition, array processing, blind source separation

◮ sensor networks and cognitive radios

◮ power quality disturbances

◮ underwater acoustic communications

◮ data acquisition and imaging technologies



Sparse Signal: Sparsity

◮ Definition: A signal can be sparse or compressible in sometransform matrix Ψ when the transform coefficients vector αhas a small number of large amplitude coefficients and a largenumber of small amplitude coefficients.

◮ Observations: Most of the energy is concentrated in a fewtransform coefficients in a vector α

◮ The other N − K coefficients have less contribution inrepresenting a signal vector x ∈ R

N×1.

◮ The insignificant coefficients are set to zero in coding scheme.



Sparse Signal: Sparsity

◮ The value of K is computed as K = ‖α‖0, where ‖.‖0 denotesthe ℓ0-norm which counts the number of non-zero entries inα.

◮ Concluding Remarks: A sparse signal x can be exactlyrepresented or approximated by the linear combination of Kbasis functions with shorter transform coefficients vector.

◮ In such a reconstruction process, the reconstruction error by aK -term representation decays exponentially as K increases.



Sparse Representation: Dictionary Learning

◮ Need for Dictionary Learning: In practice, a signal iscomposed of impulsive and oscillatory transients, spikes andlow-frequency components.

◮ Nature: The composite signal may not exhibit sparsity in onetransform basis matrix because some of its components aresparse in one domain while other components are sparse inanother domain.

◮ The signals may exhibit sparsity in either time-domain orfrequency-domain.

◮ For example, the 50 Hz powerline signal is sparse in thefrequency-domain and the impulse or spikes component issparse in the time-domain.



Sparse Representation: Dictionary Learning

◮ Problem with Fixed Basis: In practice, the composite signal(spike is superimposed on powerline signal) exhibits sparsity inneither time-domain nor frequency-domain.

◮ In such cases, a fixed orthogonal basis functions are notflexible enough to capture the complex local waves of a signal.

◮ For example, a fixed elementary cosine waveforms of discretecosine transform (DCT) matrix fails to capture transient partsof biosignals.

◮ Detection and suppression of impulsive noise in speechwaveform.

◮ Compression of slow varying signals with spikes.



Sparse Representation: Need for Best Basis Functions

◮ Remedies: To improve the sparsity of composite signals, onehas to construct a transform matrix with the best basisfunctions.

◮ One way to process such signal is to work with an largedictionary matrix.

◮ A best basis set from a dictionary matrix used to sparsify thedata may yield highly compact representations of manynatural signals.

◮



Sparse Representation: What is Dictionary?

◮ Dictionary: A dictionary is a collection of elementarywaveforms or prototype atoms or basis functions.

◮ A dictionary matrix D of dimension N × L can be representedas D = {ψ1|ψ2|ψ3|..........|ψL}.

◮ The column vectors {ψl}Ll=1 of an dictionary D arediscrete-time elementary signals of length N × 1, calleddictionary atoms or basis functions.

◮ The atoms in the pre-defined dictionary may be pairwiseorthogonal, linear independent, linear dependent, or notorthogonal.



Sparse Representation: Classification of Dictionaries

◮ Based on the number of atoms L and the signal length N, thepre-defined dictionary, D ∈ R

N×M , could be classified intothree categories:

◮ (i) when L > N, D is called overcomplete, or redundantdictionary.

◮ (ii) when L < N, D is called undercomplete dictionary.

◮ (iii) D is said to be complete dictionary if L = N.



Some of Sparse Transform Matrices

◮ dirac and heaviside functions

◮ Fourier transform matrix and Fourier, short-time Fouriertransform (STFT)

◮ discrete cosines (DCT matrix) and discrete sines (DST matrix)

◮ Haar transform matrix

◮ wavelet and wavelet packets matrices

◮ Gabor filters

◮ curvelets, ridgelets, contourlets, bandelets, shearlets

◮ directionlets, grouplets, chirplets

◮ Hermite polynomials, and so on



Sparse Representation: Research Problems

◮ The dirac dictionary can be used to detect the spikes in asignal and the discrete cosines dictionary can providesinusoidal waveforms

◮ The SR from redundant dictionaries may provide better waysto reveal/capture the structures in nonstationaryenvironments

◮ The SR may offer better performance in signal modeling andclassification problems

◮ An efficient and flexible dictionary matrix has to be built forseparation of mixtures of events

◮ Many researchers have attempted to build dictionary forspecific signal processing tasks

◮ How to learn the dictionary from the training datasets



The CS Measurement System

◮ Performing reduction/compression when sensing analogsignals

◮ The CS is a new data acquisition theory

◮ The number of measurements is typically below the number ofsamples obtained from the Nyquist sampling theorem

◮ The nonadaptive linear measurements of the input signalvector are computed as

y = Φx (2)

where y is an M × 1 measurement vector, M ≪ N and Φ isan M × N measurement/sensing matrix.

◮ Measurements using a second basis matrix Φ ∈ RM×N that is

incoherent with the sparsity basis matrix Ψ ∈ RN×N



The CS System: Reduction and Information

◮ The measurement system actually performs dimensionalityreduction

◮ Measurements are able to completely capture the usefulinformation content embedded in a sparse signal

◮ Measurements are information of the signals and thus can beused as features for signal modeling

◮ If the Φ consists of elementary sinusoid waveforms, then α isa vector of Fourier coefficients.

◮ If the Φ consists of Dirac delta functions, then α is a vector ofsampled values of continuous time signal x(t).



The CS Recovery: Issues

◮ The y can be written as

y = ΦΨα (3)

◮ We define the matrix D = ΦΨ with a size of M × N.

◮ The major problem associated with CS concept is that wehave to solve an underdetermined system of equations torecover the original signal x from the measurement vector y .

◮ This system has infinitely many solutions since the number ofequations is less than the number of unknowns

◮ It is necessary to impose constraints such as “sparsity” and“incoherence” that are introduced for for this signal recoveryto be efficient



CS Reconstruction: Incoherence

◮ the CS recovery relies on two basic principles [?]:

◮ (i) the row vectors of the measurement matrix Φ cannotsparsely represent the column vectors of the sparsity matrix Ψ,and vice versa

◮ (ii) the number of measurements M is greater thanO(cKlog(N

K))

◮ these conditions can ensure that it is possible to recover theset of nonzero elements of sparse vector α from measurementsy .

◮ the input signal x can be reconstructed by the lineartransformation of α: .



CS Reconstruction: Incoherence

◮ Sparsity basis matrix Ψ is orthonormal and the sensing matrixΦ consists of M row vectors drawn randomly from some basismatrix Φ̃ ∈ R

N×N

◮ The mutual coherence is computed as:

µ(Φ,Ψ) =√N max

1≤k,j≥N|〈Φk ,Ψj〉| (4)

◮ It measures the largest correlation between any two elementsof Φ and Ψ and will take a value between 1 and

√N .

◮ The value of coherence is large when the elements of Φ andΨ are highly correlated and thus CS system requires moremeasurements.

◮ The smaller value µ(Φ,Ψ) indicates maximally incoherentbases and hence, the number of measurements will be less



CS Recovery: How many measurements?

◮ The recovery performance is perfect and optimal when thebases are perfectly incoherent, and unavoidably decreaseswhen the mutual coherence µ increases.

◮ The number of measurements M required for perfect signalreconstruction can be computed as:

M ≥ c · K · µ2(Φ,Ψ) · log(N) (5)

where c is positive constant, µ is the mutual coherence, K isthe sparsity factor, and N is the length of the input vector.

◮ The value of coherence is large when the elements of Φ andΨ are highly correlated and thus CS system requires moremeasurements.

◮ The smaller value µ(Φ,Ψ) indicates maximally incoherentbases and hence, the number of measurements in (4) can bethe smallest



CS Recovery: How many measurements?

◮ Under very low mutual coherence value, k-sparse signal canbe reconstructed from k .log(N) measurements using basispursuit

◮ Examples of such pairs (maximal mutual incoherence)are: Φ is the spike basis and Ψ is the Fourier basis

◮ Φ is the noiselet basis and Ψ is the wavelet basis. Noiseletsare also maximally incoherent with spikes and incoherent withthe Fourier basis.

◮ Φ is a random matrix and Ψ is any fixed basis



Compressive Sensing/Measurement Matrices

◮ The the entries of Φ are: (i) samples of independent andidentically distributed (iid) Gaussian or Bernoulli entries

◮ (ii) randomly selected rows of an orthogonal N × N matrix

◮ The RIP says that D acts as an approximate isometry on theset of vectors that are K -sparse, and a matrix D satisfies theK−restricted isometry property if there exists the smallestnumber, δs ∈ [0 1], such that(1− δs)‖α‖22 ≤ ‖Dα‖22 ≤ (1 + δs)‖α‖22.The constant δs depends on K , Φ, and α.



CS Recovery by ℓ1-norm Optimization

◮ The goal of a sparse recovery algorithm is to obtain anestimate of α given only y and D = ΦΨ

◮ The recovery of the K -sparse signal x from the measurementsy is ill-posed since M < N

◮ The CS system of equations is underdetermined

◮ the sparest vector is computed by solving the well-knownunderdetermind problem with sparsity constraint,

α̂ = argminα

‖α‖0 subject to y = ΦΨα = Dα (6)

where ‖ • ‖ denotes ℓ0-norm that counts the number ofnonzero entries in a vector.




◮ By allowing a certain degree of reconstruction error given bythe magnitude of the noise

◮ the optimization constraint is now relaxed:

α̂ = argminα

{‖α‖0 +λ

2‖y −ΦΨα‖22} (7)

where λ ∈ R+, which controls the relative importance applied

to the reconstruction error term and the sparseness term.

◮ the solution needs a combinatorial search among all possiblesparse α, which is infeasible for most problems of interest




◮ To overcome this problem, many nonlinear optimization-basedmethods have been proposed to obtain sparest vector α byconverting (6) into a convex problem which relaxes theℓ0-norm to an ℓ1-norm problem

α̂ = argminα

‖α‖1 subject to y = ΦΨα = Dα (8)

α̂ = argminα

{‖α‖1 +λ

2‖y −ΦΨα‖22} (9)

◮ which can be solved by linear programming such as BP, MPand OMP

◮ The solution to equation (8) is exact or optimal if the numberof measurements K is large enough compared to the sparsityfactor K , K < M < N and the measurements are chosenuniformly at random



Analog to Information Converter

�

Figure: The block diagram of AIC



SR Applications: Transients Detection

0 20 40 60 80 100−2

−1

0

1

2

ampli

tude

0 20 40 60 80−1

0

1

−1

0

1

ampli

tude

0 20 40 60 80 100 120 140−2

−1

0

1

sample number

ampli

tude

(b)

(a)

(c)

Figure: Examples of measured transients with 50 Hz power supplywaveforms: (a) spike, (b) microinterruption, and (c) oscillatory transient.




◮ The over-complete dictionary Ψ with size of N × 2N isconstructed as

Ψ = [ I C] (10)

where I is the N × N identity (or spike-like) matrix, and C isthe N × N DCT matrix.

◮

Cij =

{

1√

M, i = 0, 0 ≤ j ≤ N − 1

√

2M

cos(π(2j+1)i

2N), 1 ≤ i ≤ N − 1, 0 ≤ j ≤ N − 1

(11)

and the spike like matrix is constructed as

Iij =

1 0 · · · 0 00 1 · · · 0 00 0 1 0 0...

......

. . . 00 0 0 · · · 1

N×N

(12)




◮ the signal x can be written as

x ≈ Ψα̃ = [ I C]α̃ = Id+ Ca. (13)

and can be rewritten as

y =

N∑

n=1

dnIn +

N∑

n=1

anCn. (14)

◮ The common problem in well-known wavelet transform-basedmethods is which mother wavelet function and characteristicscale provides the best time-frequency resolution for detectionof transients and non-transients.




1. Input: N × 1 input signal vector y .2. Specify the value of regularization parameter λ.3. Read the N × 2N over-complete dictionary matrix Ψ.4. Solve the ℓ1-norm minimization problem:

α̃ = argminα{‖Ψα− y‖22 + λ‖α‖1}5. Obtain the detail and approximation coefficient vectors.6. Process detail vector for detecting boundaries of transient event.7. Output: time-instants and transient portions



SR Applications: Impulsive Transients

−1

0

1Powerline with impulsive noise

orig

inal

−0.5

0

0.5

deta

ilco

mpo

nent

0 0.02 0.04 0.06 0.08 0.1 0.12

−0.5

0

0.5

1

Time (sec)

appr

oxim

atio

nco

mpo

nent

Figure: Illustrates the detail and approximation components extracted byusing the proposed method. The power supply waveform is corrupted byspikes.




0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1−1

0

150 Hz powerline with microinterruption

orig

inal

0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10

0.2

0.4

0.6

0.8

deta

il co

mpo

nent

0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1−1

0

1

Time (sec)

appr

oxim

atio

n co

mpo

nent

Figure: Illustrates the detail and approximation components extracted byusing the proposed method. The The power supply waveform withmicrointerruption.




0 0.01 0.02 0.03 0.04 0.05 0.06

−1

0

1

orig

inal

sig

nal

0 0.01 0.02 0.03 0.04 0.05 0.06−1

−0.5

0

0.5

1

dete

cted

tran

sien

t(d

etai

l sig

nal e

xtra

cted

)

0 0.01 0.02 0.03 0.04 0.05 0.06

−1

0

1

Time (sec)

appr

oxim

atio

n s

igna

l ext

ract

ed

0 0.01 0.02 0.03 0.04 0.05 0.06

−1

0

1

orig

nal s

igna

l

0 0.01 0.02 0.03 0.04 0.05 0.06−1.5

−1

−0.5

0

0.5

1

dete

cted

tran

sien

t (d

etai

l sig

nal e

xtra

cted

)

0 0.01 0.02 0.03 0.04 0.05 0.06

−1

0

1

Time (sec)ap

prox

imat

ion

sig

nal e

xtra

cted

(a)

(f)

low−amplitude transients

(b)

(e)

(d)(c)

high−amplitude transients

Figure: Example of waveforms S1 and S2 illustrates signals corrupted bylow-amplitude transient S1 and high-amplitude transient S2 due tocapacitor switching, respectively. The detected transient events by usingour method.




0.01 0.02 0.03 0.04 0.05

−0.2

0

0.2

0.4

spik

e w

ith n

oise

0.01 0.02 0.03 0.04 0.050

0.1

0.2

Time (sec)

dete

cted

spi

ke

by o

ur m

etho

d

0 0.01 0.02 0.03 0.04 0.05

−0.2

0

0.2

0.4

wav

elet

met

hod

(Firs

t Det

ail)

0 0.01 0.02 0.03 0.04

−0.2

0

0.2

0.4

wav

elet

met

hod

(Sec

ond

Det

ail)

0 0.01 0.02 0.03 0.04 0.05−1

0

1

sign

al w

ithsp

ike

0 0.01 0.02 0.03 0.04 0.05−1

0

1

Time (sec)de

tect

ed s

pike

by o

ur m

etho

d

0 0.01 0.02 0.03 0.04 0.05

−1

0

1

wav

elet

met

hod

(Firs

t Det

ail)

0 0.01 0.02 0.03 0.04 0.05−1

0

1

wav

elet

met

hod

(Sec

ond

Det

ail)

(d1)

(b1)

(a1)

(c2)

(a2)

(d2)

(b2)

(c1)

Figure: Example of transient signals S3 and S4: (a1) the spike buried instrong noise with SNR value of -10 dB; (a2) the 50 Hz sinusoidal signalaffected by a superimposed spike; Plots are the outputs from thewavelet-based methods and the proposed method.



SR Applications: Removal of Powerline

200 400 600 800 1000 1200 1400 1600 1800 2000

-0.5

0

0.5

ampl

itude

Time (sec)

original ECG signal

200 400 600 800 1000 1200 1400 1600 1800 2000

-0.5

0

0.5

original ECG signal plus powerline (10 degree)

ampl

itude

Time (sec)

200 400 600 800 1000 1200 1400 1600 1800 2000

-0.5

0

0.5

Output of CS-based approach

ampl

itude

Time (sec)

200 400 600 800 1000 1200 1400 1600 1800 2000

-0.5

0

0.5

ampl

itude

Time (sec)

original ECG signal

200 400 600 800 1000 1200 1400 1600 1800 2000

-0.5

0

0.5

original ECG signal plus powerline (86 degree)

ampl

itude

Time (sec)

200 400 600 800 1000 1200 1400 1600 1800 2000

-0.5

0

0.5


ampl

itude

Time (sec)

Figure: Removal of Powerline from ECG Signal



SR Applications: Removal of Artifacts

500 1000 1500 2000 2500 3000-1

-0.5

0

0.5

ampli

tude

Time (sec)

original ECG signal

500 1000 1500 2000 2500 3000

-1

-0.5

0

0.5

original ECG signal plus powerline

ampli

tude

Time (sec)

500 1000 1500 2000 2500 3000

-0.5

0

0.5


ampli

tude

Time (sec)

500 1000 1500 2000 2500 3000

-0.4

-0.2

0

0.2

0.4

Output of CS-based baseline wander removal

ampli

tude

Time (sec)

Figure: Simultaneous removal of Powerline and LF artifact from ECGSignal



Sparse Representation and Compressive Sensing

Thanks for your Attention!


sparse representation and compressive sensing

Education

sparse signal x

input signal

composite signal

sparse representation

transform matrix

discretetime signal

recognition signal detection

signal vector x rn1