analysis sparse representation for nonnegative signals...

Research ArticleAnalysis Sparse Representation for Nonnegative Signals Basedon Determinant Measure by DC Programming

Yujie Li 1 Benying Tan2 Atsunori Kanemura 1 Shuxue Ding2 and Wuhui Chen 3

1National Institute of Advanced Industrial Science and Technology (AIST) Tsukuba Japan2School of Computer Science and Engineering University of Aizu Aizuwakamatsu Japan3School of Data and Computer Science Sun Yat-sen University Guangzhou China

Correspondence should be addressed to Wuhui Chen chenwuhmailsysueducn

Received 1 September 2017 Accepted 15 March 2018 Published 24 April 2018

Academic Editor Tsendsuren Munkhdalai

Copyright copy 2018 Yujie Li et al This is an open access article distributed under the Creative Commons Attribution License whichpermits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Analysis sparse representation has recently emerged as an alternative approach to the synthesis sparse model Most existingalgorithms typically employ the ℓ0-norm which is generally NP-hard Other existing algorithms employ the ℓ1-norm to relax theℓ0-norm which sometimes cannot promote adequate sparsityMost of these existing algorithms focus on general signals and are notsuitable for nonnegative signals However many signals are necessarily nonnegative such as spectral data In this paper we presenta novel and efficient analysis dictionary learning algorithm for nonnegative signals with the determinant-type sparsity measurewhich is convex and differentiable The analysis sparse representation can be cast in three subproblems sparse coding dictionaryupdate and signal update because the determinant-type sparsity measure would result in a complex nonconvex optimizationproblem which cannot be easily solved by standard convex optimization methods Therefore in the proposed algorithms we usea difference of convex (DC) programming scheme for solving the nonconvex problem According to our theoretical analysis andsimulation study the main advantage of the proposed algorithm is its greater dictionary learning efficiency particularly comparedwith state-of-the-art algorithms In addition our proposed algorithm performs well in image denoising

1 Introduction

Real signals around our daily life are always distributed in ahigh dimensional space however low-dimensional structuresare found in the signals then we can represent the signalswith a proper model by only a few parameters [1] A propermodel should be simple while matching the signals In thepast decades the sparse and redundant representationmodelhas been proven to be an efficient and beneficial model [2ndash5] The theoretical background for sparse models is given bycompressed sensing (CS) [6ndash8] CS mathematically declaresthat if a signal is sparse or compressive this original signalcan be reconstructed by a few measurements which aremuch fewer than the counts suggested by previous theories[6 7 9ndash11] Sparse representation has also been describedas an extraordinary powerful solution for a wide rangeof real-word applications especially in image processingsuch as image denoising deblurring inpainting restoration

superresolution and also in the field of machine learningcomputer vision and so on [12ndash21]

Sparse representation can be formulated by either asynthesis model or an analysis model The synthesis model ispopular and mature The analysis model has been less inves-tigated for sparse representation although several analysisdictionary learning algorithms have been proposed such asthe analysis K-SVD [12] GreedyAnalysis Pursuit (GAP) [22]and the analysis thresholding algorithm [23]

In practice some signals such as chemical concentrationsin experimental results and pixels in video frames and imageshave inherent nonnegativity and dedicated factorizationmethods have been proposed [24 25] The above analysissparse representation algorithms are all for general sig-nals which contain nonnegative and negative elements Themethods for general signals directly applied to nonnegativesignals cannot achieve satisfying results An existing analysisdictionary learning method for nonnegative signals [26]

HindawiComplexityVolume 2018 Article ID 2685745 12 pageshttpsdoiorg10115520182685745

2 Complexity

which uses blocked determinants as the sparseness measureis quite difficult or computationally expensiveThepurpose ofthis paper is to address the problem on nonnegative analysissparse representation

In this paper we present a novel algorithm of analysissparse representation for nonnegative signals which is par-allel to the synthesis sparse representation in the principleand structure Though this model has been studied in thepast there is still not a matured field for nonnegative analysisrepresentation while the algorithms designed for generalsignal cannot sufficiently be applied to the nonnegativesignals Thus we focus on the nonnegative sparse represen-tation with the analysis model We cast the analysis sparserepresentation into three subproblems (analysis dictionaryupdate sparse coding and signal recovery) and use analternating scheme to obtain an optimization solution Weutilize the determinant-type of sparseness measure as thesparseness constraint which is convex and differentiableThe objective function for sparse coding is nonconvex andstandard convex optimization methods cannot be employedFortunately the objective function is the difference of twoconvex functions then we can introduce difference of convex(DC) programming to solve this nonconvex optimizationproblem

The remainder of this paper is organized as follows Theconventional sparse representation problem is reviewed inSection 2 In Section 3 we introduce the analysis represen-tation In Section 4 we describe the problem formulation foranalysis sparse representation and present the optimizationframework The experiments described in Section 5 demon-strate the practical advantages of the proposed algorithmscomparedwith state-of-the-art algorithms bothwith artificialand real-world datasets Finally we present our conclusionsin Section 6

11 Notations Here we list notations used in this paper Aboldface uppercase letter like X is defined as a matrix anda lowercase letter like 119909119894119895 is defined as the 119894119895th entry of X Aboldface lowercase letter such as x is defined as a vector anda lowercase letter 119909119895 is defined as the 119895th entry of x MatrixslicesX119894 andX119895 are defined as the 119894th row and the 119895th columnof matrix X respectively The Frobenius norm of matrix Xis defined as X119865 = (sum119894119895 |119909119894119895|2)12 The determinant valueof a matrix is denoted by det(sdot) Note that in this paper allparameters take real values

2 Preliminaries

21 Sparse Representation Sparse representation decom-poses observed signals into a product of a dictionary matrixwhich contains signal bases and a sparse coefficient matrix[13ndash17] and there are two different structures synthesismodel and analysis model The synthesis model is the firstproposed sparse model and more popular We first review itin this section

Assume that we want to model the signals X isin R119898times119873where119898 is the signal dimensionality and119873 is the number ofmeasurements The synthesis sparse model suggests that thesignals could be expressed as

X = DH (1)

or

X asymp DHst X minusDH2119865 le 120576

(2)

where D isin R119898times119899 refer to as a dictionary H isin R119899times119873 is arepresentation coefficientmatrix and 120576 ge 0 is a small residual[27] Here 119899 is the number of bases which are also calleddictionary atoms We further assume that the representationmatrix H is sparse (ie many zero entries) to obtain sparserepresentations of the signals Equations (1) or (2) mean thateach signal can be represented as a linear combination of afew atoms from the dictionary matrixD

A key issue in the sparse representation is the choice of thedictionarywhich the observed signals are used to decomposeOne choice is a predefined dictionary such as discreteFourier transform (DFT) discrete cosine transform (DCT)and wavelets [28] which can be employed for learning asignal-specific dictionary from observed signals Anotherchoice the learned dictionary results in better matching tothe contents of signals In addition the learned dictionaryoften exhibits better performance compared to predefineddictionaries in real-world applications [29 30]

Intriguingly there exists a ldquotwinrdquo of the synthesis modelcalled the analysis model [31] Assume that there is a matrixΩ isin R119899times119898 that produces a sparse coefficient matrix H bybeing multiplied to the signal matrixH = ΩX This equationcan be obtained as the solution to a minimization problemof the error function H minus ΩX119865 Remarkably this errorfunction is convex and standard optimization methods canbe employed Since error functions in the synthesis modelare nonconvex optimization in the analysis model is ofteneasier We call Ω isin R119899times119898 the analysis dictionary Atomsin the analysis dictionary Ω are its rows rather columnsin the synthesis dictionary D The term ldquoanalysisrdquo meansthe dictionary analyzes the signal to produce a sparse result[32] To emphasize the difference between the analysis andsynthesis models the term ldquocosparsityrdquo has been introducedin the literature [31 33] which counts the number of 0-valuedelements of ΩX that is zero elements coproduced by Ω andX [34] The analysis sparse model is also called the cosparsemodel and then the analysis dictionary is also called thecosparse dictionary

Now we look more closely at the analysis sparse modelThe analysis model for one signal x isin R119898 which is a columnin the signal matrix X can be represented using a properanalysis dictionary Ω isin R119899times119898 The 119894th row namely 119894thatom in Ω is denoted by 120596119894 We want to make the analysisrepresentation vector h = Ωx sparse This is formulated byintroducing a sparsity measure 119872(h) so that it negativelybehaves with the sparsity of h and minimizing 119872(h) yieldsthe sparsest solution

Ω = argminΩ

119872(h) st h = Ωx

(3)

Complexity 3

0

1

05

2 33 2

1

0

05

1

1

23

3 21

0

05

1

1

2 33 2

1

Figure 1 Illustration of various degrees of sparseness

Although employing the ℓ0-norm that is setting 119872(h) =h0 yields the sparsest solution [35] the optimizationproblem is combinatorial and oftenNP-hardTherefore othersparsity measures such as the ℓ1-norm are employed to haveeasier optimization problems Nevertheless it is known thatthe ℓ1-norm often overpenalizes large elements and solutionsare too sparse

22 Sparseness Measure The ℓ119901-norms where 119901 = 0 1or 2 are popular measures for assessing the sparseness ofa vector Since the ℓ0-norm yields an NP-hard problem itsconvex relaxation the ℓ1-norm is often preferred [36 37]The ℓ1-norm of a vector h is defined to be the sum of theabsolute values of h namely h1 = sum119894 |h119894| If the vectoris nonnegative that is h isin R+ the ℓ1-norm of h is justh1 = sum119894 ℎ119894 For nonnegative vectors their ℓ1-norm isdifferentiable and smooth and gradient methods can beused in optimization Some authors introduce the ℓ2-normwith nonnegative matrix factorization since the nonnegativeconstraints yield sparse solutions However the results withthe ℓ2-norm are not sparser than those with the ℓ0-norm orℓ1-norm [38]

The sparsity measures mentioned above can reflect theinstantaneous sparseness of one single signal [35] but theyare not suitable for evaluating sparsity across differentmeasurements [39] In order to describe the joint sparse-ness of the nonnegative sources we introduce determinant-type of sparsity measure In spectral unmixing for remotesensing image interpretation where signals are nonnega-tive the determinant-type of sparsity measure [40] can getgood results similar to the other sparseness-based methodsfrom the results of numerical experiments [41] Thus thedeterminant-type measure can explicitly measure the sparse-ness of nonnegative matrices

The determinant-type sparse measure has several goodqualities If a nonnegative matrix is normalized we canfind that the determinant value of the nonnegative matrixis well bounded and its value interpolates monotonouslybetween two extremes of 0 and 1 along with the increasingof the sparsity For instance if the nonnegative matrix V isnonsparse with its rows and satisfies sum-to-one then thedeterminant of VV119879 det(VV119879) is close to 0 On the otherhand det(VV119879) approaches to 1 if and only if the matrix is

the most sparse [42] Namely the determinant value satisfiesthat 0 le det(VV119879) le 1 where det(VV119879) = 0 if all entries ofVV119879 are the same and det(VV119879) = 1when the following twocriteria are satisfied at the same time

(1) For all 119894 isin 1 2 119898 only one element in k119894 isnonzero

(2) For all 119894 119895 isin 1 2 119898 and 119894 = 119895 k119894 and k119895 areorthogonal that is k119879119894 k119895 = 0

The detailed proof can be found in [40] Thus we can use thedeterminant measure in the cost function Figure 1 illustratesthe sparseness degrees of three different matrices gaugedby the determinant measure The determinant values of thematrices from left to right are 00625 05 and 1 We can seethat the sparser thematrix is the larger value the determinantmeasure isThus the sparse coding problemwith determinantconstraints can be expressed as an optimization problem

maxV

det (VV119879) = minV

minus det (VV119879) (4)

23 Related Works The existing methods for sparse repre-sentation with the analysis model employ various sparsityconstraints The analysis K-SVD algorithm [12] minimizesthe error between the noisy (observed) signals and theestimated (reconstructed) signals with the ℓ0-norm as thesparsity constraint The optimal backward greedy algorithm(OBG) is employed to estimate the analysis dictionary Ωwhich has higher complexity In [43ndash46] the ℓ1-norm isimposed as sparse constraint on the sparse coefficients and aprojected subgradient-based algorithm is employed to learnthe analysis dictionaryThe analysis operator learning (AOL)algorithm [43] utilizing the ℓ1-norm constraint restricts thedictionary to a uniform normalized tight frame (UNTF) andlearns the analysis dictionary as entire matrix by solving acheap soft thresholding and then projecting the dictionaryonto the UNTF set Li et al [26] propose a novel analysissparse model based on the determinant-type measure wherethe optimization problem is solved by an iterative sparsenessmaximization scheme In their work the problem can becast into row-to-row optimizationwith respect to the analysisatoms and then utilize the quadratic programming (QP)technique to solve the optimization problem with respect

4 Complexity

to each row However this method has no signal recoverystage and cannot apply to denoising and other applicationsIn recently DC programming is introduced to deal with thenonconvex optimization problems for sparse representation[47] DC programming and DC algorithm (DCA) constitutethe backbone of smoothness and nonsmoothness nonconvexprogramming and global optimization [48] Thus it is con-sidered to be a powerful tool for nonconvex optimizations[48 49] The exiting DC programming based sparse repre-sentation algorithms are with synthesis model and we firstpropose DCA to analysis model

3 Formulation

Now we turn to describe the problem of analysis sparserepresentation We consider the following assumption givenan observed signal vector y isin R119898+ we assume that y is anoisy version of a signal x isin R119898+ Thus y = x + u where uis additive positive white Gaussian noise Using an analysisdictionary Ω isin R119899times119898 each row of which defines 1 times 119898analysis atom we assume that x satisfies Ωx0 = 119898 minus 119897where 119897 is the cosparsity of the signal defied to be the numberof zero elements Then we extend to signals matrix and weuse matrix X to define signals with each column as onesignal Here we use119872(sdot) as the sparse measure Taking intoaccount the noise in the measured signals we formulate anoptimization task for analysis dictionary learning by

minΩX

119872(ΩX) st Y minus X2119865 le 120590Ω isinL

(5)

where 120590 is a noise-level parameter L is the constraint onthe analysis dictionary and 119872 is the sparse regularizationWe prefer to use an alternative regularized version of aboveequation with penalty multipliers Here we employ Z as anapproximation of ΩX which makes the learning easier andfaster in practice Thus the analysis sparse coding is cheapand can be obtained exactly by thresholding the product ofΩX and the sparsity measure on Z Thus the analysis sparserepresentation can be rewritten as

minΩXZ

119872(Z) + 120582 Y minus X2119865 + 120573 ΩX minus Z2119865 st 100381710038171003817100381712059611989410038171003817100381710038172 = 1 forall119894

(6)

where Z isin R119899times119873 is the representation coefficient matrixHaving a sparse representation means the representationmatrix Z is sparse We introduce normalization constraint(forall119894 1205961198942 = 1) to analysis dictionary which helps remove ascale ambiguity

We can cast the entire problem into three variable-wise optimization subproblems The first one is the analysisdictionary update subproblem and the next two problems aresparse coefficient matrix and signal update subproblems

A pseudocode for such an analysis dictionary learning ispresented in Algorithm 1 (ADLA)

(1) InitializationΩ0Z0X0 = Y 119894 = 0(2) while not converged do(3) Ω119894+1 = min

ΩΩXminusZ2119865 stforall119894 1205961198942 = 1

(4) Z119894+1 = minZ119872(Z) + 120573ΩXminusZ2119865(5) X119894+1 = minX120582Y minus X2119865 + 120573ΩXminusZ2119865(6) 119894 = 119894 + 1(7) end while

Algorithm 1 Analysis dictionary learning algorithm (ADLA)

31 Analysis Sparse Representation with Determinant Con-straint In this section we introduce a novel determinant-type constrained sparse method to learn an analysis dictio-nary for sparse representation of nonnegative signals Whatis more we set the parameters 120582 and 120573 in (6) as positive con-stants Considering these factors our minimization problemfor analysis sparse dictionary learning with the determinant-type constraint can be expressed as

minΩXZ

minus det (ZZ119879) + 120582 Y minus X2119865 + 120573 ΩX minus Z2119865 st Z gt 0

100381710038171003817100381712059611989410038171003817100381710038172 = 11003817100381710038171003817z11989410038171003817100381710038172 = 1

forall119894

(7)

The above objective function is nonconvex which is hard tosolve by traditional convex optimization methods

4 Proposed Algorithm

Theanalysis representationwith determinant-type constraintproblem can be cast into three subproblems an analysisdictionary update stage a sparse coefficient coding stage anda signal recovery stage The corresponding formulations areas follows

Analysis Dictionary Update Stage

minΩ

ΩX minus Z2119865 st 100381710038171003817100381712059611989410038171003817100381710038172 = 1 forall119894

(8)

Sparse Coefficient Coding Stage

minZ

minus det (ZZ119879) + 120573 ΩX minus Z2119865 st Z gt 0

(9)

1003817100381710038171003817z11989410038171003817100381710038172 = 1 forall119894 (10)

Signal Recovery Stage

minX

120582 Y minus X2119865 + 120573 ΩX minus Z2119865 (11)

Complexity 5

We will describe each stage in detail in the followingsubsections

41 Analysis Dictionary Update We use a projected subgra-dient type algorithm to solve the analysis dictionary updatesubproblem as follows

minΩ


(12)

Since the objective function above is a quadratic functionof Ω the solution can be analytically found as the zero-gradient point The gradient of the objective function is

120597119891 (Ω)120597Ω = (ΩX minus Z)X119879

= ΩXX119879 minus ZX119879(13)

By solving 120597119891(Ω)120597Ω = 0 we obtain

Ω = ZX119879 (XX119879)minus1 (14)

The projection of a dictionary onto a space with fixedrow norms can be easily done for nonzero rows by scalingeach row to have a unit norm We use 119875UN to denote thisprojection If a row is zero we set the row to a normalizedrandom vector Namely 119875UN is not uniquely defined This isdue to the fact that the set of uniformly normalized vectors isnot convex The projection can be found by

119875UN (Ω) = [119875UN (120596119894)]119894

119875UN (120596) =

1205961205962 1205962 = 0] otherwise

(15)

where ] is a random vector on the unit sphere

42 Sparse Coding by DC Programming Optimization forestimating sparse coefficient Z is a core problemThe formu-lation is the subtraction of two convex functions making theobjective function not generally nonconvex Thus we cannotuse traditional convex optimization methods to solve thisproblem

In this paper to solve the objective function we introducethe DC programming scheme to translate the minimiza-tion optimization problem into DC programming problemAccording to the theory of DC programming [48 49]we construct a DC function for our problem Then thereformulated DC objective function 119865 can be expressed as

119865 (Z) = 1198911 (Z) minus 1198912 (Z) (16)

where

1198911 (Z) = 120573 ΩX minus Z2119865 1198912 (Z) = det (ZZ119879) (17)

Here 1198911(Z) and 1198912(Z) are two convex functions Then thesparse coding problem can be reformulated as a DC program

min 1198911 (Z) + 120594R119899times119873+

(Z) minus 1198912 (Z) Z isin R119899times119873+ (18)

where 120594R119899+

(Z) is an indicator function which is defined by120594R119899+

(Z) = 0 if Z isin R119899times119873+ and +infin otherwiseWe utilize the scheme of DCA [48] At each iteration 119896

Z(119896) = nabla1198912 (Z(119896)) = (det (ZZ119879)) (ZZ119879)minus1 Z (19)

At the iteration 119896 + 1Z(119896+1) = argmin

Z1198911 (Z) minus ⟨Z(119896)Z⟩ Z isin R

119899+

= argminZ

120573 ΩX minus Z2119865 minus ⟨Z(119896)Z⟩ Z isin R119899+

(20)

Consider that Z(119896+1) is separable according to the columns ofZ the above equation can be rewritten as

Z(119896+1) = argminZ

119873

sum119895=1

(120573 (ΩX)119879119895 (ΩX)119895 minus 2120573 (ΩX)119879119895 Z119895 + 120573Z119879119895Z119895 minus Z(119896)119879119895 Z119895) Z isin R119899+

(21)

For the sparse coding stage in iteration scheme we fix thedictionary Ω and signals X and update only Z in (16) and(17) To keep the coefficient matrix Z nonnegative the updateprocedure of sparse coefficient Z can be briefly described asfollows

Z(119896+1)119895 = ((ΩX)119895 + 12120573 Z(119896)119895 )+

forall119895 (22)

where (sdot)+ = max(0 sdot) takes the positive part Algorithm 2shows the procedures of the proposed algorithm for sparsecoding based on DC programming

43 Signal Recovery Then we consider the signal recoverystage In this stage the analysis dictionary Ω and the coef-ficients Z are fixed The optimization problem for signalrecovery is formulated as

minX

119891 (X) = 120582 Y minus X2119865 + 120573 ΩX minus Z2119865 (23)

The above objective function (23) is a quadratic functionof signals X Thus the optimal solution of above objec-tive function can be analytically obtained by setting its

6 Complexity

1 2 3 4 5 6 7 8 9 10123456789

10

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

432

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

395

(a)

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

2 3 4 5 6 7 8 9 10123456789

10

347

2 3 4 5 6 7 8 9 10123456789

10

306

2 3 4 5 6 7 8 9 10123456789

10

236

2 3 4 5 6 7 8 9 10123456789

10167

(b)

Figure 2 Illustration of phase transition (a) determinant-type measure119873 = 50 100 150 200 (b) ℓ1-norm119873 = 50 100 150 200 (from leftto right)

Require119898 times119873matrix X 119899 times 119898matrixΩ and119898 lt 119899 lt 119873(1) Initialize Z let 119896 = 1(2) while not converged do(3) Compute Z(119896) using (19)(4) Update Z(119896+1) using (22)(5) end while

Algorithm 2 Sparse coding based on DC programming (SCDC)

gradient equal to zero The gradient of the objective functionis

120597119891 (X)120597X = 120582 (X minus Y) + 120573Ω119879 (ΩX minus Z) (24)

By solving 120597119891(X)120597X = 0 we obtain

X = 120582Y + 120573Ω119879Z120582I + 120573Ω119879Ω (25)

5 Experiments and Discussion

This section presents the results of numerical experimentsto evaluate the performance of the proposed algorithmsThe programs were coded in Matlab (R2016b) and run ona machine with a 33 GHz Intel Core i7 CPU and 16GB ofmemory under Microsoft Windows 10

51 Phase Transition The goal of this phase transitionexperiment is to study how the proposed algorithm withthe determinant-type sparsity measure can identify nonzeroelements in a sparse matrix Z isin R119898times119873 Specifically weexamine when the success or failure of nonzero elementidentification switches according to the sparsity of thematrix

We varied the number 119904 of nonzero elements in sparse matrixZ the number 119898 of the dimensionality of the dictionaryatoms and the number of measurements 119873 The range for119904 was from 1 to 10119898 from 1 to 10 and119873 from 50 to 200

Figure 2 compares the phase transition diagrams ofthe determinant-type measure and the ℓ1-norm The whiteregions indicate success of identifying nonzero elementswhereas the black regions indicate failure It can be seenthat the area of white for the determinant-type measureis larger than that of the ℓ1-norm It demonstrates thatthe determinant-type measure outperforms the ℓ1-normespecially when the number of measurements119873 increases

52 Dictionary Recovery This subsection examines if theproposed algorithm can recover the true analysis dictionaryTo quantitatively evaluate the performance we used syntheticsignals with a known ground-truth dictionary

Then to evaluate the sparsity of the coefficient we use theHoyer sparsity [50] which can be normalized to satisfy moreof the sparsity constraints [51]

HoyerSparsity (x) =radic119899 minus (sum 10038161003816100381610038161199091198941003816100381610038161003816) radicsum1199092119894

radic119899 minus 1 (26)

where x is a column of the coefficient matrix The Hoyersparsity for a matrix is defined as

HoyerSparsity (X) = 1119873119873

sum119894=1

HoyerSparsity (x119894) (27)

The larger the Hoyer sparsity measure the sparser the coeffi-cientmatrixNote that the sparse representation optimizationproblem with Hoyer sparsity constraint is hard to solve thusit is generally used for evaluation

Complexity 7

0 20 40 60 80 100Iterations

85

90

95

100

Reco

very

rate

s = 6s = 5s = 4

s = 3s = 2s = 1

Figure 3 The recovery curves

s = 6s = 5s = 4

s = 3s = 2s = 1

0 20 40 60 80 100Iterations

07

075

08

085

09

095

1

Hoy

er sp

arsit

y

Figure 4 The Hoyer sparsity

521 Experimental Setup A ground-truth dictionary andobserved signals were built as follows A random synthesisdictionary W isin R119898times119898+ was generated by taking the absolutevalues of iid zero mean unit variance normal randomvariables The analysis dictionary was set as Ω = Wminus1 Aset of observation signals X isin R119898times119873+ was generated byX = WZ namely each signal column x was generated by alinear combination of 119904 different atoms in the dictionary with119904 nonnegative corresponding coefficients in unity randomand independent locations Naturally the elements of theobserved signals became nonnegative with different sparsi-ties In this experiment we chose a set of size 119873 = 1000 ofsuch observational signal matrices

0 20 40 60 80 100Iterations

82

84

86

88

90

92

94

96

98

100

Reco

very

rate

041044047

050052055


0 20 40 60 80 100Iterations

07

075

08

085

09

095

Hoy

er sp

arsit

y

041044047

050052055


To initialize the proposed algorithm we used a linearmodel to generate the initial estimate by combining theground-truth dictionaryΩ0 and a normalized randommatrix120598 that isΩinit = (1minus120588)Ω0+120588120598 and then projecting it to satisfythe uniformly normalized and orthogonal constraints When120588 is zero we actually initialize Ωinit with the ground-truthdictionaryΩ0 and when 120588 = 1 the initial analysis dictionarywill be random

The learned analysis dictionary by our algorithm wascompared with the ground-truth dictionary Since there is arow-shuffle indeterminacy we find corresponding rows bysweeping all rows of the learned dictionary and ground-truth dictionary and finding the closest rows between the two

8 Complexity

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 1

s = 1

SADL-DETProposed

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1

Hoy

er sp

arsit

y

SADL-DETProposed

Figure 7 The comparison with SADL-DET when 119904 = 1

dictionaries The distance less than 001 was considered as asuccess

max119895

min119894(1 minus 100381610038161003816100381610038161003816Ω1015840119894

119879Ω0119895

100381610038161003816100381610038161003816) lt 001 (28)

where Ω1015840119894 was an atom from the learned dictionary and Ω119895was an atom from the ground-truth dictionary

First we use different number of nonzeros 119904 We set119904 = 1 2 3 4 5 6 and 120588 = 09 to see the performancebetween different sparsity The Hoyer sparsities of ground-truth coefficient matrices are 1 (119904 = 1) 095 (119904 = 2) 091(119904 = 3) 087 (119904 = 4) 084 (119904 = 5) and 081 (119904 = 6) Figure 3presents the recovery curves of the analysis dictionary Ω0 ofsize 50 times 50 from which we can see that the results canreach to 100 in the cases of 119904 = 1 5 and 6 and more than99 in the cases of 119904 = 2 3 and 4 Figure 4 presents theHoyer sparsity of the learned sparse coefficients which shows

s = 3

s = 3

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1H

oyer

spar

sity

ProposedSADL-DET


the convergence of Hoyer sparsity value of different 119904 andthey nearly converge to the Hoyer sparsities of ground-truthcoefficient matrices

Next we investigated the role of 120588 We chose 120588 =07 08 09 10 11 12 with 119904 = 3 and drew the recoverycurves and the Hoyer sparsity Figure 5 shows the recoverycurves of the analysis dictionary with different 120588 From thisfigure we can see that the ratio of recovery can reach about100 inmanydifferent situations Figure 6 presents theHoyersparsity of the learned sparse coefficient which shows theHoyer sparsity converges to the same value which nearlyequal to the ground-truth Hoyer sparsity 091 (119904 = 3)

Then we compared our proposed algorithm with theexisting determinant-type based algorithm SADL-DET [26]which is also applied for nonnegative signals We comparedthe recovery rate and the Hoyer sparsity for these algorithmsin the case of 119904 = 1 3 5 Figure 7 shows the results in thecase of 119904 = 1 which shows our proposed algorithm converges

Complexity 9

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 5

s = 5

ProposedSADL-DET

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

Reco

very

rate


faster than SADL-DET Figure 8 shows the results in the caseof 119904 = 3 which shows our proposed algorithm also convergefaster in the recovery curve and can obtain a better recoveryrate Figure 9 shows the results in the case of 119904 = 5 whichshows that our proposed algorithm can achieve good results

We compared the computational consumption Figure 10indicates that our proposed algorithm is nearly 103 timesfaster compared with SADL-DET

53 Image Denoising We tested our proposed method witha practical problem of image denoising [18] The noise leveland the quality of denoising were evaluated by the peaksignal-to-noise ratios (PSNRs (PSNR (dB) is defined as10log10(2552119909 minus 119910) where 119909 and 119910 denote the originalsignal and the signal polluted by noise resp)) The testedimages as also the tested noise levels are from the CroppedYale face database (from httpvisionucsdeduextyalebCroppedYaleBZipCroppedYalezip) Note that to make sure

1 3 5Sparsity

0

05

1

15

2

1000

2000

3000

4000

Com

puta

tiona

l tim

e (s)

SADL-DETProposed

Figure 10 The computational consumption

Table 1 Image denoising performance of PSNR (dB)

Face ID Noisy level COAOL Proposed FD01 107662 110553 192743 11055302 107675 110686 190550 11015203 107381 110324 187899 10984904 107542 110452 189949 11002505 107918 110881 184613 11041006 107502 110408 189840 10995907 107844 110838 192423 11033508 108030 110926 187126 11054109 107905 110792 185088 11040010 107528 110408 183358 11000011 107764 110641 187527 11024212 107623 110429 186075 11009213 107268 110181 174434 10973515 108005 110786 189708 11048716 107784 110660 186266 11026117 107745 110626 191770 11022418 107546 110398 190336 11003119 107442 110307 192140 10992020 107765 110703 189491 110255

that the observed signals are nonnegative the noises weadded to the original images are nonnegative uniformlydistributed pseudorandom values

Figure 11 shows original noisy and denoised imagesby our proposed algorithm Then we compare our pro-posed algorithm with the analysis model based algorithmConstrained Overcomplete Analysis Operator Learning(COAOL) and we also represent the denoised results usingfixed Finite Difference (FD) operator The average PSNRsof the denoised results for different face cases are presentedin Table 1 which shows that our algorithm performs well

10 Complexity

(a) (b) (c)

Figure 11 Face image denoising (a) Original face images (b) Noisy images (c) Denoised faces using our proposed algorithm

(a) (b) (c) (d) (e)

Figure 12 Face image denoising (a) Original face images (b) Noisy images (c) Denoised faces using COAOL (d) Denoised faces using ourproposed algorithm (e) Denoised faces using FD

in image denoising To intuitively see the denoising perfor-mance with different methods we take faces 05 10 15 and20 for instance Figure 12 shows the original faces noisyfaces and denoised faces with different methods COAOLours and FD Although COAOL has a good performance forgeneral signals it cannot achieve good results for nonnegativenoise Our proposed algorithm has outstanding advantage innonnegative image denoising

6 Conclusion

In this study we have proposed a novel and efficient anal-ysis sparse representation algorithm with the determinant-type sparse measure which focuses on nonnegative signalsprocessing We separated the whole problem into threesubproblems analysis dictionary updates sparse coding andsignals recovery In the sparse coding stage we employ DC

Complexity 11

programming to solve the nonconvex minimization prob-lems The experimental results verify the effectiveness of theproposed algorithm and the determinant measure of sparsityRemarkable advantages of the proposed algorithm includethe faster running time and good performance in imagedenoising Moving forward we can employ the proposedalgorithm for more applications such as image inpaintingsuperresolution and other applications

Conflicts of Interest

The authors declare that there are no conflicts of interestregarding the publication of this paper

Acknowledgments

This work was supported in part by NEDO and JSPS KAK-ENHI 26730130 and 16K00335

References

[1] R G Baraniuk V Cevher andM BWakin ldquoLow-dimensionalmodels for dimensionality reduction and signal recovery Ageometric perspectiverdquo Proceedings of the IEEE vol 98 no 6pp 959ndash971 2010

[2] B K Natarajan ldquoSparse approximate solutions to linear sys-temsrdquo SIAM Journal on Computing vol 24 no 2 pp 227ndash2341995

[3] M Huang W Yang J Jiang et al ldquoBrain extraction based onlocally linear representation-based classificationrdquo NeuroImagevol 92 pp 322ndash339 2014

[4] A M Bruckstein D L Donoho and M Elad ldquoFrom sparsesolutions of systems of equations to sparse modeling of signalsand imagesrdquo SIAM Review vol 51 no 1 pp 34ndash81 2009

[5] J Mairal G Sapiro and M Elad ldquoLearning multiscale sparserepresentations for image and video restorationrdquo MultiscaleModeling amp Simulation vol 7 no 1 pp 214ndash241 2008

[6] D L Donoho ldquoCompressed sensingrdquo Institute of Electrical andElectronics Engineers Transactions on Information Theory vol52 no 4 pp 1289ndash1306 2006

[7] R G Baraniuk ldquoCompressive sensingrdquo IEEE Signal ProcessingMagazine vol 24 no 4 pp 118ndash121 2007

[8] Y Wang J Zeng Z Peng X Chang and Z Xu ldquoLinearconvergence of adaptively iterative thresholding algorithms forcompressed sensingrdquo IEEE Transactions on Signal Processingvol 63 no 11 pp 2957ndash2971 2015

[9] E J Candes and M B Wakin ldquoAn introduction to compressivesampling A sensingsampling paradigm that goes against thecommon knowledge in data acquisitionrdquo IEEE Signal ProcessingMagazine vol 25 no 2 pp 21ndash30 2008

[10] Y Tsaig and D L Donoho ldquoExtensions of compressed sensingrdquoSignal Processing vol 86 no 3 pp 549ndash571 2006

[11] E Candes and J Romberg ldquoSparsity and incoherence incompressive samplingrdquo Inverse Problems vol 23 no 3 pp 969ndash985 2007

[12] R Rubinstein T Peleg and M Elad ldquoAnalysis K-SVD adictionary-learning algorithm for the analysis sparse modelrdquoIEEE Transactions on Signal Processing vol 61 no 3 pp 661ndash677 2013

[13] M Elad M A T Figueiredo and Y Ma ldquoOn the role ofsparse and redundant representations in image processingrdquoProceedings of the IEEE vol 98 no 6 pp 972ndash982 2010

[14] M G Jafari and M D Plumbley ldquoFast dictionary learningfor sparse representations of speech signalsrdquo IEEE Journal ofSelected Topics in Signal Processing vol 5 no 5 pp 1025ndash10312011

[15] M J Fadili J-L Starck and F Murtagh ldquoInpainting andzooming using sparse representationsrdquo The Computer Journalvol 52 no 1 pp 64ndash79 2009

[16] A Adler V Emiya M G Jafari M Elad R Gribonval and MD Plumbley ldquoAudio inpaintingrdquo IEEE Transactions on AudioSpeech and Language Processing vol 20 no 3 pp 922ndash932 2012

[17] C F Caiafa and A Cichocki ldquoComputing sparse represen-tations of multidimensional signals using Kronecker basesrdquoNeural Computation vol 25 no 1 pp 186ndash220 2013

[18] M Elad and M Aharon ldquoImage denoising via sparse andredundant representations over learned dictionariesrdquo IEEETransactions on Image Processing vol 15 no 12 pp 3736ndash37452006

[19] W Dong L Zhang and G Shi ldquoCentralized sparse represen-tation for image restorationrdquo in Proceedings of the 2011 IEEEInternational Conference on Computer Vision ICCV 2011 pp1259ndash1266 esp November 2011

[20] Z Li S Ding Y Li Z Yang S Xie and W Chen ldquoManifoldoptimization-based analysis dictionary learning with an ℓ12-norm regularizerrdquo Neural Networks vol 98 pp 212ndash222 2018

[21] Z Li T Hayashi S Ding and Y Li ldquoDictionary learning withthe ℓ12-regularizer and the coherence penalty and its conver-gence analysisrdquo International Journal of Machine Learning andCybernetics pp 1ndash14 2017

[22] R Giryes S Nam M Elad R Gribonval and M E DaviesldquoGreedy-like algorithms for the cosparse analysismodelrdquoLinearAlgebra and its Applications vol 441 pp 22ndash60 2014

[23] T Peleg and M Elad ldquoPerformance guarantees of the thresh-olding algorithm for the cosparse analysis modelrdquo Institute ofElectrical and Electronics Engineers Transactions on InformationTheory vol 59 no 3 pp 1832ndash1845 2013

[24] V P Pauca J Piper and R J Plemmons ldquoNonnegative matrixfactorization for spectral data analysisrdquo Linear Algebra and itsApplications vol 416 no 1 pp 29ndash47 2006

[25] A B Hamza and D J Brady ldquoReconstruction of reflectancespectra using robust nonnegative matrix factorizationrdquo IEEETransactions on Signal Processing vol 54 no 9 pp 3637ndash36422006

[26] Y Li S Ding and Z Li ldquoDictionary learning with the cosparseanalysis model based on summation of blocked determinantsas the sparseness measurerdquoDigital Signal Processing vol 48 pp298ndash309 2016

[27] Y Li S Ding and Z Li ldquoA dictionary-learning algorithmfor the analysis sparse model with a determinant-type ofsparsity measurerdquo in Proceedings of the 2014 19th InternationalConference on Digital Signal Processing DSP 2014 pp 152ndash156Hong Kong August 2014

[28] I W Selesnick R G Baraniuk and N G Kingsbury ldquoThedual-tree complex wavelet transformrdquo IEEE Signal ProcessingMagazine vol 22 no 6 pp 123ndash151 2005

[29] M Elad Sparse and Redundant Representations FromTheory toApplications in Signal and Image Processing Springer NewYorkNY USA 2010

12 Complexity

[30] J-L Starck F Murtagh and J M Fadili Sparse image and signalprocessing Cambridge University Press Cambridge 2010

[31] S NamM E Davies M Elad and R Gribonval ldquoThe cosparseanalysis model and algorithmsrdquo Applied and ComputationalHarmonic Analysis vol 34 no 1 pp 30ndash56 2013

[32] S Ravishankar and Y Bresler ldquoLearning sparsifying trans-formsrdquo IEEE Transactions on Signal Processing vol 61 no 5 pp1072ndash1086 2013

[33] S Nam M E Davies M Elad and R Gribonval ldquoCosparseanalysis modelingmdashuniqueness and algorithmsrdquo in Proceedingsof the IEEE International Conference on Acoustics Speech andSignal Processing (ICASSP rsquo11) pp 5804ndash5807 Prague CzechRepublic May 2011

[34] S Hawe M Kleinsteuber and K Diepold ldquoAnalysis operatorlearning and its application to image reconstructionrdquo IEEETransactions on Image Processing vol 22 no 6 pp 2138ndash21502013

[35] J M Duarte-Carvajalino and G Sapiro ldquoLearning to sensesparse signals simultaneous sensingmatrix and sparsifying dic-tionary optimizationrdquo IEEE Transactions on Image Processingvol 18 no 7 pp 1395ndash1408 2009

[36] D L Donoho and M Elad ldquoOptimally sparse representationin general (nonorthogonal) dictionaries via 1198971 minimizationrdquoProceedings of the National Acadamy of Sciences of the UnitedStates of America vol 100 no 5 pp 2197ndash2202 2003

[37] C A Zala I Barrodale and J S Kennedy ldquoHigh-resolutionsignal and noise field estimation using the L1 (least absolutevalues) normrdquo IEEE Journal of Oceanic Engineering vol 12 no1 pp 253ndash264 1987

[38] Z Li S Ding Y Li Z Tang and W Chen ldquoImprovingdictionary learning using the Itakura-Saito divergencerdquo inProceedings of the 2nd IEEE China Summit and InternationalConference on Signal and Information Processing IEEEChinaSIP2014 pp 733ndash737 chn July 2014

[39] B Willmore and D J Tolhurst ldquoCharacterizing the sparsenessof neural codesrdquo Network Computation in Neural Systems vol12 no 3 pp 255ndash270 2001

[40] Z Yang Y Xiang S Xie S Ding and Y Rong ldquoNonnegativeblind source separation by sparse component analysis based ondeterminant measurerdquo IEEE Transactions on Neural Networksand Learning Systems vol 23 no 10 pp 1601ndash1610 2012

[41] L Miao and H Qi ldquoEndmember extraction from highly mixeddata using minimum volume constrained nonnegative matrixfactorizationrdquo IEEE Transactions on Geoscience and RemoteSensing vol 45 no 3 pp 765ndash777 2007

[42] Z Yang G Zhou S Xie S Ding J-M Yang and J ZhangldquoBlind spectral unmixing based on sparse nonnegative matrixfactorizationrdquo IEEE Transactions on Image Processing vol 20no 4 pp 1112ndash1125 2011

[43] M Yaghoobi S Nam R Gribonval and M E Davies ldquoCon-strained overcomplete analysis operator learning for cosparsesignal modellingrdquo IEEE Transactions on Signal Processing vol61 no 9 pp 2341ndash2355 2013

[44] M Yaghoobi and M E Davies ldquoRelaxed analysis operatorlearningrdquo in NIPS Workshop on Analysis Operator Learning vsDictionary Learning Fraternal Twins in Sparse Modeling 2012

[45] M Yaghoobi S Nam R Gribonval and M E Davies ldquoNoiseaware analysis operator learning for approximately cosparsesignalsrdquo in Proceedings of the IEEE International Conference onAcoustics Speech and Signal Processing (ICASSP rsquo12) pp 5409ndash5412 March 2012

[46] G Peyre and Fadili J M ldquoLearning analysis sparsity priorsrdquo inProceedings of the International Conference on Sampling Theoryand Applications (SampTA) 4 pages 2011

[47] B Tan Y Li S Ding and X Li ldquoRecovering nonnegativesparse signals with a determinant-type of sparse measure anddc programmingrdquo in Proceedings of International Conference onApplied Computer and Communication Technologies (ComCom2017) 2017

[48] H A LeThi X T Vo and T PhamDinh ldquoEfficient nonnegativematrix factorization by DC programming and DCArdquo NeuralComputation vol 28 no 6 pp 1163ndash1216 2016

[49] L T An and P D Tao ldquoTheDC (difference of convex functions)programming and DCA revisited with DC models of realworld nonconvex optimization problemsrdquoAnnals of OperationsResearch vol 133 pp 23ndash46 2005

[50] P O Hoyer ldquoNon-negativematrix factorization with sparsenessconstraintsrdquo Journal of Machine Learning Research vol 5 pp1457ndash1469 2004

[51] N Hurley and S Rickard ldquoComparing measures of sparsityrdquoInstitute of Electrical and Electronics Engineers Transactions onInformation Theory vol 55 no 10 pp 4723ndash4741 2009

Hindawiwwwhindawicom Volume 2018

MathematicsJournal of


Mathematical Problems in Engineering

Applied MathematicsJournal of


Probability and StatisticsHindawiwwwhindawicom Volume 2018

Journal of


Mathematical PhysicsAdvances in

Complex AnalysisJournal of


OptimizationJournal of



Engineering Mathematics

International Journal of


Operations ResearchAdvances in

Journal of


Function SpacesAbstract and Applied AnalysisHindawiwwwhindawicom Volume 2018

International Journal of Mathematics and Mathematical Sciences


Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom

The Scientific World Journal

Volume 2018

Hindawiwwwhindawicom Volume 2018Volume 2018

Numerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisNumerical AnalysisAdvances inAdvances in Discrete Dynamics in

Nature and SocietyHindawiwwwhindawicom Volume 2018

Hindawiwwwhindawicom

Dierential EquationsInternational Journal of

Volume 2018


Decision SciencesAdvances in


AnalysisInternational Journal of


Stochastic AnalysisInternational Journal of

Submit your manuscripts atwwwhindawicom

2 Complexity

which uses blocked determinants as the sparseness measureis quite difficult or computationally expensiveThepurpose ofthis paper is to address the problem on nonnegative analysissparse representation

In this paper we present a novel algorithm of analysissparse representation for nonnegative signals which is par-allel to the synthesis sparse representation in the principleand structure Though this model has been studied in thepast there is still not a matured field for nonnegative analysisrepresentation while the algorithms designed for generalsignal cannot sufficiently be applied to the nonnegativesignals Thus we focus on the nonnegative sparse represen-tation with the analysis model We cast the analysis sparserepresentation into three subproblems (analysis dictionaryupdate sparse coding and signal recovery) and use analternating scheme to obtain an optimization solution Weutilize the determinant-type of sparseness measure as thesparseness constraint which is convex and differentiableThe objective function for sparse coding is nonconvex andstandard convex optimization methods cannot be employedFortunately the objective function is the difference of twoconvex functions then we can introduce difference of convex(DC) programming to solve this nonconvex optimizationproblem

The remainder of this paper is organized as follows Theconventional sparse representation problem is reviewed inSection 2 In Section 3 we introduce the analysis represen-tation In Section 4 we describe the problem formulation foranalysis sparse representation and present the optimizationframework The experiments described in Section 5 demon-strate the practical advantages of the proposed algorithmscomparedwith state-of-the-art algorithms bothwith artificialand real-world datasets Finally we present our conclusionsin Section 6

11 Notations Here we list notations used in this paper Aboldface uppercase letter like X is defined as a matrix anda lowercase letter like 119909119894119895 is defined as the 119894119895th entry of X Aboldface lowercase letter such as x is defined as a vector anda lowercase letter 119909119895 is defined as the 119895th entry of x MatrixslicesX119894 andX119895 are defined as the 119894th row and the 119895th columnof matrix X respectively The Frobenius norm of matrix Xis defined as X119865 = (sum119894119895 |119909119894119895|2)12 The determinant valueof a matrix is denoted by det(sdot) Note that in this paper allparameters take real values

2 Preliminaries

21 Sparse Representation Sparse representation decom-poses observed signals into a product of a dictionary matrixwhich contains signal bases and a sparse coefficient matrix[13ndash17] and there are two different structures synthesismodel and analysis model The synthesis model is the firstproposed sparse model and more popular We first review itin this section

Assume that we want to model the signals X isin R119898times119873where119898 is the signal dimensionality and119873 is the number ofmeasurements The synthesis sparse model suggests that thesignals could be expressed as

X = DH (1)

or

X asymp DHst X minusDH2119865 le 120576

(2)

where D isin R119898times119899 refer to as a dictionary H isin R119899times119873 is arepresentation coefficientmatrix and 120576 ge 0 is a small residual[27] Here 119899 is the number of bases which are also calleddictionary atoms We further assume that the representationmatrix H is sparse (ie many zero entries) to obtain sparserepresentations of the signals Equations (1) or (2) mean thateach signal can be represented as a linear combination of afew atoms from the dictionary matrixD

A key issue in the sparse representation is the choice of thedictionarywhich the observed signals are used to decomposeOne choice is a predefined dictionary such as discreteFourier transform (DFT) discrete cosine transform (DCT)and wavelets [28] which can be employed for learning asignal-specific dictionary from observed signals Anotherchoice the learned dictionary results in better matching tothe contents of signals In addition the learned dictionaryoften exhibits better performance compared to predefineddictionaries in real-world applications [29 30]

Intriguingly there exists a ldquotwinrdquo of the synthesis modelcalled the analysis model [31] Assume that there is a matrixΩ isin R119899times119898 that produces a sparse coefficient matrix H bybeing multiplied to the signal matrixH = ΩX This equationcan be obtained as the solution to a minimization problemof the error function H minus ΩX119865 Remarkably this errorfunction is convex and standard optimization methods canbe employed Since error functions in the synthesis modelare nonconvex optimization in the analysis model is ofteneasier We call Ω isin R119899times119898 the analysis dictionary Atomsin the analysis dictionary Ω are its rows rather columnsin the synthesis dictionary D The term ldquoanalysisrdquo meansthe dictionary analyzes the signal to produce a sparse result[32] To emphasize the difference between the analysis andsynthesis models the term ldquocosparsityrdquo has been introducedin the literature [31 33] which counts the number of 0-valuedelements of ΩX that is zero elements coproduced by Ω andX [34] The analysis sparse model is also called the cosparsemodel and then the analysis dictionary is also called thecosparse dictionary

Now we look more closely at the analysis sparse modelThe analysis model for one signal x isin R119898 which is a columnin the signal matrix X can be represented using a properanalysis dictionary Ω isin R119899times119898 The 119894th row namely 119894thatom in Ω is denoted by 120596119894 We want to make the analysisrepresentation vector h = Ωx sparse This is formulated byintroducing a sparsity measure 119872(h) so that it negativelybehaves with the sparsity of h and minimizing 119872(h) yieldsthe sparsest solution

Ω = argminΩ

119872(h) st h = Ωx

(3)

Complexity 3

0

1

05

2 33 2

1

0

05

1

1

23

3 21

0

05

1

1

2 33 2

1










maxV




4 Complexity


3 Formulation


minΩX


(5)


minΩXZ


(6)









minΩXZ


100381710038171003817100381712059611989410038171003817100381710038172 = 11003817100381710038171003817z11989410038171003817100381710038172 = 1

forall119894

(7)





minΩ


(8)


minZ


(9)

1003817100381710038171003817z11989410038171003817100381710038172 = 1 forall119894 (10)


minX


Complexity 5



minΩ


(12)


120597119891 (Ω)120597Ω = (ΩX minus Z)X119879

= ΩXX119879 minus ZX119879(13)


Ω = ZX119879 (XX119879)minus1 (14)


119875UN (Ω) = [119875UN (120596119894)]119894

119875UN (120596) =

1205961205962 1205962 = 0] otherwise

(15)




119865 (Z) = 1198911 (Z) minus 1198912 (Z) (16)

where



min 1198911 (Z) + 120594R119899times119873+


where 120594R119899+






119899+

= argminZ


(20)


Z(119896+1) = argminZ

119873

sum119895=1


(21)


Z(119896+1)119895 = ((ΩX)119895 + 12120573 Z(119896)119895 )+

forall119895 (22)



minX



6 Complexity

1 2 3 4 5 6 7 8 9 10123456789

10

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

432

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

395

(a)

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

2 3 4 5 6 7 8 9 10123456789

10

347

2 3 4 5 6 7 8 9 10123456789

10

306

2 3 4 5 6 7 8 9 10123456789

10

236

2 3 4 5 6 7 8 9 10123456789

10167

(b)







X = 120582Y + 120573Ω119879Z120582I + 120573Ω119879Ω (25)












sum119894=1



Complexity 7

0 20 40 60 80 100Iterations

85

90

95

100

Reco

very

rate

s = 6s = 5s = 4

s = 3s = 2s = 1


s = 6s = 5s = 4

s = 3s = 2s = 1

0 20 40 60 80 100Iterations

07

075

08

085

09

095

1

Hoy

er sp

arsit

y



0 20 40 60 80 100Iterations

82

84

86

88

90

92

94

96

98

100

Reco

very

rate

041044047

050052055


0 20 40 60 80 100Iterations

07

075

08

085

09

095

Hoy

er sp

arsit

y

041044047

050052055




8 Complexity

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 1

s = 1

SADL-DETProposed

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1

Hoy

er sp

arsit

y

SADL-DETProposed



max119895

min119894(1 minus 100381610038161003816100381610038161003816Ω1015840119894

119879Ω0119895

100381610038161003816100381610038161003816) lt 001 (28)



s = 3

s = 3

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1H

oyer

spar

sity

ProposedSADL-DET





Complexity 9

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 5

s = 5

ProposedSADL-DET

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

Reco

very

rate





1 3 5Sparsity

0

05

1

15

2

1000

2000

3000

4000

Com

puta

tiona

l tim

e (s)

SADL-DETProposed






10 Complexity

(a) (b) (c)


(a) (b) (c) (d) (e)



6 Conclusion


Complexity 11




Acknowledgments


References






























12 Complexity






























Journal of












Journal of







Volume 2018






Volume 2018








Complexity 3

0

1

05

2 33 2

1

0

05

1

1

23

3 21

0

05

1

1

2 33 2

1










maxV




4 Complexity


3 Formulation


minΩX


(5)


minΩXZ


(6)









minΩXZ


100381710038171003817100381712059611989410038171003817100381710038172 = 11003817100381710038171003817z11989410038171003817100381710038172 = 1

forall119894

(7)





minΩ


(8)


minZ


(9)

1003817100381710038171003817z11989410038171003817100381710038172 = 1 forall119894 (10)


minX


Complexity 5



minΩ


(12)


120597119891 (Ω)120597Ω = (ΩX minus Z)X119879

= ΩXX119879 minus ZX119879(13)


Ω = ZX119879 (XX119879)minus1 (14)


119875UN (Ω) = [119875UN (120596119894)]119894

119875UN (120596) =

1205961205962 1205962 = 0] otherwise

(15)




119865 (Z) = 1198911 (Z) minus 1198912 (Z) (16)

where



min 1198911 (Z) + 120594R119899times119873+


where 120594R119899+






119899+

= argminZ


(20)


Z(119896+1) = argminZ

119873

sum119895=1


(21)


Z(119896+1)119895 = ((ΩX)119895 + 12120573 Z(119896)119895 )+

forall119895 (22)



minX



6 Complexity

1 2 3 4 5 6 7 8 9 10123456789

10

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

432

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

395

(a)

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

2 3 4 5 6 7 8 9 10123456789

10

347

2 3 4 5 6 7 8 9 10123456789

10

306

2 3 4 5 6 7 8 9 10123456789

10

236

2 3 4 5 6 7 8 9 10123456789

10167

(b)







X = 120582Y + 120573Ω119879Z120582I + 120573Ω119879Ω (25)












sum119894=1



Complexity 7

0 20 40 60 80 100Iterations

85

90

95

100

Reco

very

rate

s = 6s = 5s = 4

s = 3s = 2s = 1


s = 6s = 5s = 4

s = 3s = 2s = 1

0 20 40 60 80 100Iterations

07

075

08

085

09

095

1

Hoy

er sp

arsit

y



0 20 40 60 80 100Iterations

82

84

86

88

90

92

94

96

98

100

Reco

very

rate

041044047

050052055


0 20 40 60 80 100Iterations

07

075

08

085

09

095

Hoy

er sp

arsit

y

041044047

050052055




8 Complexity

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 1

s = 1

SADL-DETProposed

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1

Hoy

er sp

arsit

y

SADL-DETProposed



max119895

min119894(1 minus 100381610038161003816100381610038161003816Ω1015840119894

119879Ω0119895

100381610038161003816100381610038161003816) lt 001 (28)



s = 3

s = 3

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1H

oyer

spar

sity

ProposedSADL-DET





Complexity 9

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 5

s = 5

ProposedSADL-DET

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

Reco

very

rate





1 3 5Sparsity

0

05

1

15

2

1000

2000

3000

4000

Com

puta

tiona

l tim

e (s)

SADL-DETProposed






10 Complexity

(a) (b) (c)


(a) (b) (c) (d) (e)



6 Conclusion


Complexity 11




Acknowledgments


References






























12 Complexity






























Journal of












Journal of







Volume 2018






Volume 2018








4 Complexity


3 Formulation


minΩX


(5)


minΩXZ


(6)









minΩXZ


100381710038171003817100381712059611989410038171003817100381710038172 = 11003817100381710038171003817z11989410038171003817100381710038172 = 1

forall119894

(7)





minΩ


(8)


minZ


(9)

1003817100381710038171003817z11989410038171003817100381710038172 = 1 forall119894 (10)


minX


Complexity 5



minΩ


(12)


120597119891 (Ω)120597Ω = (ΩX minus Z)X119879

= ΩXX119879 minus ZX119879(13)


Ω = ZX119879 (XX119879)minus1 (14)


119875UN (Ω) = [119875UN (120596119894)]119894

119875UN (120596) =

1205961205962 1205962 = 0] otherwise

(15)




119865 (Z) = 1198911 (Z) minus 1198912 (Z) (16)

where



min 1198911 (Z) + 120594R119899times119873+


where 120594R119899+






119899+

= argminZ


(20)


Z(119896+1) = argminZ

119873

sum119895=1


(21)


Z(119896+1)119895 = ((ΩX)119895 + 12120573 Z(119896)119895 )+

forall119895 (22)



minX



6 Complexity

1 2 3 4 5 6 7 8 9 10123456789

10

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

432

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

395

(a)

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

2 3 4 5 6 7 8 9 10123456789

10

347

2 3 4 5 6 7 8 9 10123456789

10

306

2 3 4 5 6 7 8 9 10123456789

10

236

2 3 4 5 6 7 8 9 10123456789

10167

(b)







X = 120582Y + 120573Ω119879Z120582I + 120573Ω119879Ω (25)












sum119894=1



Complexity 7

0 20 40 60 80 100Iterations

85

90

95

100

Reco

very

rate

s = 6s = 5s = 4

s = 3s = 2s = 1


s = 6s = 5s = 4

s = 3s = 2s = 1

0 20 40 60 80 100Iterations

07

075

08

085

09

095

1

Hoy

er sp

arsit

y



0 20 40 60 80 100Iterations

82

84

86

88

90

92

94

96

98

100

Reco

very

rate

041044047

050052055


0 20 40 60 80 100Iterations

07

075

08

085

09

095

Hoy

er sp

arsit

y

041044047

050052055




8 Complexity

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 1

s = 1

SADL-DETProposed

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1

Hoy

er sp

arsit

y

SADL-DETProposed



max119895

min119894(1 minus 100381610038161003816100381610038161003816Ω1015840119894

119879Ω0119895

100381610038161003816100381610038161003816) lt 001 (28)



s = 3

s = 3

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1H

oyer

spar

sity

ProposedSADL-DET





Complexity 9

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 5

s = 5

ProposedSADL-DET

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

Reco

very

rate





1 3 5Sparsity

0

05

1

15

2

1000

2000

3000

4000

Com

puta

tiona

l tim

e (s)

SADL-DETProposed






10 Complexity

(a) (b) (c)


(a) (b) (c) (d) (e)



6 Conclusion


Complexity 11




Acknowledgments


References






























12 Complexity






























Journal of












Journal of







Volume 2018






Volume 2018








Complexity 5



minΩ


(12)


120597119891 (Ω)120597Ω = (ΩX minus Z)X119879

= ΩXX119879 minus ZX119879(13)


Ω = ZX119879 (XX119879)minus1 (14)


119875UN (Ω) = [119875UN (120596119894)]119894

119875UN (120596) =

1205961205962 1205962 = 0] otherwise

(15)




119865 (Z) = 1198911 (Z) minus 1198912 (Z) (16)

where



min 1198911 (Z) + 120594R119899times119873+


where 120594R119899+






119899+

= argminZ


(20)


Z(119896+1) = argminZ

119873

sum119895=1


(21)


Z(119896+1)119895 = ((ΩX)119895 + 12120573 Z(119896)119895 )+

forall119895 (22)



minX



6 Complexity

1 2 3 4 5 6 7 8 9 10123456789

10

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

432

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

395

(a)

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

2 3 4 5 6 7 8 9 10123456789

10

347

2 3 4 5 6 7 8 9 10123456789

10

306

2 3 4 5 6 7 8 9 10123456789

10

236

2 3 4 5 6 7 8 9 10123456789

10167

(b)







X = 120582Y + 120573Ω119879Z120582I + 120573Ω119879Ω (25)












sum119894=1



Complexity 7

0 20 40 60 80 100Iterations

85

90

95

100

Reco

very

rate

s = 6s = 5s = 4

s = 3s = 2s = 1


s = 6s = 5s = 4

s = 3s = 2s = 1

0 20 40 60 80 100Iterations

07

075

08

085

09

095

1

Hoy

er sp

arsit

y



0 20 40 60 80 100Iterations

82

84

86

88

90

92

94

96

98

100

Reco

very

rate

041044047

050052055


0 20 40 60 80 100Iterations

07

075

08

085

09

095

Hoy

er sp

arsit

y

041044047

050052055




8 Complexity

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 1

s = 1

SADL-DETProposed

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1

Hoy

er sp

arsit

y

SADL-DETProposed



max119895

min119894(1 minus 100381610038161003816100381610038161003816Ω1015840119894

119879Ω0119895

100381610038161003816100381610038161003816) lt 001 (28)



s = 3

s = 3

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1H

oyer

spar

sity

ProposedSADL-DET





Complexity 9

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 5

s = 5

ProposedSADL-DET

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

Reco

very

rate





1 3 5Sparsity

0

05

1

15

2

1000

2000

3000

4000

Com

puta

tiona

l tim

e (s)

SADL-DETProposed






10 Complexity

(a) (b) (c)


(a) (b) (c) (d) (e)



6 Conclusion


Complexity 11




Acknowledgments


References






























12 Complexity






























Journal of












Journal of







Volume 2018






Volume 2018








6 Complexity

1 2 3 4 5 6 7 8 9 10123456789

10

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

432

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

407

1 2 3 4 5 6 7 8 9 10123456789

10

395

(a)

00102030405060708091

00102030405060708091

00102030405060708091

00102030405060708091

2 3 4 5 6 7 8 9 10123456789

10

347

2 3 4 5 6 7 8 9 10123456789

10

306

2 3 4 5 6 7 8 9 10123456789

10

236

2 3 4 5 6 7 8 9 10123456789

10167

(b)







X = 120582Y + 120573Ω119879Z120582I + 120573Ω119879Ω (25)












sum119894=1



Complexity 7

0 20 40 60 80 100Iterations

85

90

95

100

Reco

very

rate

s = 6s = 5s = 4

s = 3s = 2s = 1


s = 6s = 5s = 4

s = 3s = 2s = 1

0 20 40 60 80 100Iterations

07

075

08

085

09

095

1

Hoy

er sp

arsit

y



0 20 40 60 80 100Iterations

82

84

86

88

90

92

94

96

98

100

Reco

very

rate

041044047

050052055


0 20 40 60 80 100Iterations

07

075

08

085

09

095

Hoy

er sp

arsit

y

041044047

050052055




8 Complexity

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 1

s = 1

SADL-DETProposed

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1

Hoy

er sp

arsit

y

SADL-DETProposed



max119895

min119894(1 minus 100381610038161003816100381610038161003816Ω1015840119894

119879Ω0119895

100381610038161003816100381610038161003816) lt 001 (28)



s = 3

s = 3

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1H

oyer

spar

sity

ProposedSADL-DET





Complexity 9

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 5

s = 5

ProposedSADL-DET

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

Reco

very

rate





1 3 5Sparsity

0

05

1

15

2

1000

2000

3000

4000

Com

puta

tiona

l tim

e (s)

SADL-DETProposed






10 Complexity

(a) (b) (c)


(a) (b) (c) (d) (e)



6 Conclusion


Complexity 11




Acknowledgments


References






























12 Complexity






























Journal of












Journal of







Volume 2018






Volume 2018








Complexity 7

0 20 40 60 80 100Iterations

85

90

95

100

Reco

very

rate

s = 6s = 5s = 4

s = 3s = 2s = 1


s = 6s = 5s = 4

s = 3s = 2s = 1

0 20 40 60 80 100Iterations

07

075

08

085

09

095

1

Hoy

er sp

arsit

y



0 20 40 60 80 100Iterations

82

84

86

88

90

92

94

96

98

100

Reco

very

rate

041044047

050052055


0 20 40 60 80 100Iterations

07

075

08

085

09

095

Hoy

er sp

arsit

y

041044047

050052055




8 Complexity

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 1

s = 1

SADL-DETProposed

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1

Hoy

er sp

arsit

y

SADL-DETProposed



max119895

min119894(1 minus 100381610038161003816100381610038161003816Ω1015840119894

119879Ω0119895

100381610038161003816100381610038161003816) lt 001 (28)



s = 3

s = 3

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1H

oyer

spar

sity

ProposedSADL-DET





Complexity 9

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 5

s = 5

ProposedSADL-DET

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

Reco

very

rate





1 3 5Sparsity

0

05

1

15

2

1000

2000

3000

4000

Com

puta

tiona

l tim

e (s)

SADL-DETProposed






10 Complexity

(a) (b) (c)


(a) (b) (c) (d) (e)



6 Conclusion


Complexity 11




Acknowledgments


References






























12 Complexity






























Journal of












Journal of







Volume 2018






Volume 2018








8 Complexity

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 1

s = 1

SADL-DETProposed

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1

Hoy

er sp

arsit

y

SADL-DETProposed



max119895

min119894(1 minus 100381610038161003816100381610038161003816Ω1015840119894

119879Ω0119895

100381610038161003816100381610038161003816) lt 001 (28)



s = 3

s = 3

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

1H

oyer

spar

sity

ProposedSADL-DET





Complexity 9

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 5

s = 5

ProposedSADL-DET

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

Reco

very

rate





1 3 5Sparsity

0

05

1

15

2

1000

2000

3000

4000

Com

puta

tiona

l tim

e (s)

SADL-DETProposed






10 Complexity

(a) (b) (c)


(a) (b) (c) (d) (e)



6 Conclusion


Complexity 11




Acknowledgments


References






























12 Complexity






























Journal of












Journal of







Volume 2018






Volume 2018








Complexity 9

0 20 40 60 80 100Iterations

0

20

40

60

80

100

Reco

very

rate

s = 5

s = 5

ProposedSADL-DET

ProposedSADL-DET

0 20 40 60 80 100Iterations

02

03

04

05

06

07

08

09

Reco

very

rate





1 3 5Sparsity

0

05

1

15

2

1000

2000

3000

4000

Com

puta

tiona

l tim

e (s)

SADL-DETProposed






10 Complexity

(a) (b) (c)


(a) (b) (c) (d) (e)



6 Conclusion


Complexity 11




Acknowledgments


References






























12 Complexity






























Journal of












Journal of







Volume 2018






Volume 2018








10 Complexity

(a) (b) (c)


(a) (b) (c) (d) (e)



6 Conclusion


Complexity 11




Acknowledgments


References






























12 Complexity






























Journal of












Journal of







Volume 2018






Volume 2018








Complexity 11




Acknowledgments


References






























12 Complexity






























Journal of












Journal of







Volume 2018






Volume 2018








12 Complexity






























Journal of












Journal of







Volume 2018






Volume 2018















Journal of












Journal of







Volume 2018






Volume 2018








analysis sparse representation for nonnegative signals...

Documents