robust kernel-based regression using orthogonal matching pursuit

Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work

ROBUST KERNEL-BASED REGRESSIONUSING ORTHOGONAL MATCHING PURSUIT

(OMP)

G. Papageorgiou, P. Bouboulis, S. Theodoridis

Department of Informatics and Telecommunications, National and KapodistrianUniversity of Athens, Greece

MLSP SEP. 22-25, 2013,Southampton, UK

G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece

ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)


Outline

1 Introduction

2 Problem formulation

3 Unravelling KROMP

4 Experimental Results

5 Conclusions and future work




Introduction

Let (yk , xk), k = 1, 2, ..., n, with yk ∈ R, xk ∈ Rm. A typicalregression task is to estimate the input-output relation satisfying:

yk = f (xk) + ηk , k = 1, 2, ..., n.

The problem is solved:

f linear: Least Squares in the Euclidean space.

f nonlinear: Assume that f ∈ H, where H will be assumed tobe a RKHS and the minimizer of

minf

n∑k=1

(yk − f (xk)

)2+ µ||f ||2H, µ > 0

admits a representation f̂ (x) =∑n

i=1 αiκ(x, xi ) (LS in H).




Problem formulation

What if the noise samples contain outliers?

Is it possible to remove other type of noise too?

Let uk be impulse noise samples of unknown “energy”, assumed tobe outliers, hence sparse. Since LS is not robust against outlierswe reformulate:

Problem model

yk = f (xk) + uk + ηk , k = 1, 2, ..., n,




Problem formulation

Assuming that the unknown function is expressed as a linearcombination of kernel functions of the formf =

∑nk=1 αkκ(·, xk) + c , we intend to solve:

Minimization Task

minα,c,u

‖u‖0s.t. ‖Kα + c1 + u− y‖22 + λ‖α‖22 + λc2 ≤ ε




Problem formulation

Minimization Task

minα,c,u

||u||0 s.t. ||Kα + c1 + u− y||22 + λ||α||22 + λc2 ≤ ε

Remarks:

Keeps the square error low.

Regularization guards against overfitting.

Use of nonconvex function.

Exploits greedy sparse approximation algorithms from thefamily of (GA).




Sparse approximation (denoising) algorithms

l0 (pseudo)-norm minimization:

minx||x||0 s.t. ||Ax− b||2 ≤ ε

Greedy Algorithms (GA)

(MP) (OMP) (OLS) (LARS)




Sparse approximation (denoising) algorithms

l1 norm minimization:

minx||x||1 s.t. ||Ax− b||2 ≤ ε

or minx{λ||x||1 + 1

2 ||Ax− b||22} (

for appropriate Lagrangemultiplier λ = λ(A,b, ε)

)Homotopy(TNIPM)GradientProjection (GP)(IRLS)

IterativeShrinkage-ThresholdingProximalGradient (PG)

(CoSaMP)

AugmentedLagrangianMethods




Column selection from matrix A

We recast the optimization task in matrix notation:

minz||u||0 s.t. ||Az− y||22 + λ||Bz||22 ≤ ε,

where A =[K 1 In

], z =

αcu

, B =

In 0 On

0T 1 0T

On 0 On

, meaning

that only the last part of z is sparse.





Initially (k := 0) we set A(0) = [K 1] and B(0) = In+1.

At each step:

We find one position (index) for an outlier (say 1 ≤ jk ≤ n).Then, A is augmented by the column jk of In and B isaugmented by zeros in order to match the appropriatedimensions (actually this is a projection matrix B2 = B).Finally, we solve minz ‖A(k)z− y‖22 + λ‖B(k)z‖22.





Green columns: columns from matrix A, which have not beenselected yet.Red columns: columns that have already been selected at previoussteps. These participate in the regularized minimization problem.

Step k := 0 : A(0) = [K 1]

Step k := 1 : A(1) = [K 1 e2]

Step k := 2 : A(2) = [K 1 e2 e3]




Kernel Regularized OMP (KROMP): Initialization

Select the kernel for the kernel matrix K , the regularizationparameter λ and the error threshold ε.

Step k := 0 : Solve the regularized LS problem and computethe residual z(0) := arg minz ||A(0)z− y||22 + λ||B(0)z||22,

r(0) = y − A(0)z(0), where S(0)act = {1, ..., n + 1},

A(0) = [K 1], B(0) = In+1.




Kernel Regularized OMP (KROMP): Iteration cycle

Iteration: While ||r(k−1)||2 > ε

1 k := k + 1.2 Find the most correlated direction jk := arg maxj /∈Sact

|r (k−1)j |.3 Update the set: S

(k)act := S

(k−1)act ∪ {jk}

A(k) := [A(k−1) ejk ], B(k) :=

[B(k−1) 0

0T 0

].

4 Compute current solution and residual:z(k) := arg minz ||A(k)z− y||22 + λ||B(k)z||22, r(k) = y− A(k)z(k).

Remark: The residual obtained at each cycle is strictly decreasing.




Reducing Complexity

As complexity at each step is O(n + k)3, k << n, there is aneed for cost reduction.

Initially, the matrix C (0) = [A(0)TA(0) + λB(0)] is inverted, inorder to solve the minimization task.

Similarly, at step k the matrix C (k) = [A(k)TA(k) + λB(k)] isinverted.




Reducing Complexity

However, as large part of the inverted matrix at each stepremains unchanged, we could avoid the inversion by applyingthe matrix inversion lemma (MIL):

C (k)−1=

[C (k−1)−1

+ 1tC

(k−1)−1αT

jkαjkC

(k−1)−1 −1tC

(k−1)−1αT

jk

−1tαjkC

(k−1)−1 1t

],

where αjk = eTjkA(k−1), t = 1−αjkC

(k−1)−1αT

jk.

Achieved computational cost: O(n + k)2 per step.




sinc : 1-D case

−1 −0.5 0 0.5 1−40

−20

0

20

40Noisy data versus estimated data with KROMP

−1 −0.5 0 0.5 1−10

0

10

20Original data versus estimated data with MSE=0.05671

(a) KROMP with λ = 1.

−1 −0.5 0 0.5 1−40

−20

0

20

40Noisy data versus estimated data with ADM

−1 −0.5 0 0.5 1−10

0

10

20Original data versus ADM estimated data with MSE=0.09278

(b) ADM with λ = 1 and µ = 0.1.

Figure: Reconstruction of y = 20sinc(2πx) over the presence of 20dBGaussian noise and 10% outliers with values taken randomly in [−20, 20].(a) using KROMP and (b) using ADM (Giannakis-Mateos 2012). TheMSE is computed over 10000 data sets of 201 points.




sinc : 2-D case

−10

1

−1−0.8−0.6−0.4−0.200.20.40.60.81−40

−20

0

20

40

60

Noisy data versus estimated data

−10

1

−1−0.8−0.6−0.4−0.200.20.40.60.81−20

0

20

40

60

Original data versus estimated data

Figure: Reconstruction of f (x , y) = 50sinc(π√x2 + y2) over the

presence of 15dB Gaussian noise and 10% outliers with values takenrandomly in [−20, 20]. KROMP attains a MSE = 1.022, while ADMattains a MSE = 2.205 (computed over 10000 data sets of21× 21 = 421 points).




Application in image denoising: Lena

Original Image Noisy Image, PSNR=20.39dB

Figure: The original image of Lena (512× 512 pixels) and its noisycounterpart. Image corrupted with 20dB Gaussian noise and impulsenoise uniformly distributed with values ±100.




Application in image denoising: Lena

Restored Image with KROMP, PSNR=31.2dB Restored Image with ADM, PSNR=27.07dB

Figure: Left: reconstructed image using KROMP attains aPSNR = 31.2dB. Right: reconstructed image using ADM attains aPSNR = 27.07dB. In both algorithms parameters are set usingcross-correlation so that the MSE is minimized.




Application in image denoising: Boat

Original Image Noisy Image, PSNR=20.2dB

Figure: The original image of Boat (512× 512 pixels) and the noisyimage. Image corrupted with 20dB Gaussian noise and impulse noiseuniformly distributed with values ±100.




Application in image denoising: Boat

Restored Image with KROMP, PSNR=29.32dB Restored Image with ADM, PSNR=26.23dB

Figure: Left:reconstructed image using KROMP attains aPSNR = 29.32dB. Right:reconstructed image using ADM attains aPSNR = 26.23dB. In both algorithms parameters are set usingcross-correlation so that the MSE is minimized.




Conclusions

An (OMP)-based algorithm is proposed for use in nonlinearestimation

The performance of the algorithm exceeds its predecessor(ADM) in numerous examples

The residual obtained at each cycle is proved to be strictlydecreasing

Recent applications in image-denoising seem to verify ourclaim towards better performance

The heavy computational cost is reduced using MIL; otherpopular techniques (e.g., Cholesky and QR factorization)seem to work also

(OMP) is a well studied algorithm, hence rich properties arethere to reinforce our work




Future work

1 Construction of a more suitable stopping criterion

2 Complexity reduction

KROMP using QR decompositionKROMP using Cholesky factorization

3 Theoretical results for stability of the method

Conditions on the recovery of the exact support of the outliervectorBounds on the approximation of the solution

4 Applications of KROMP as an image or audio denoisingalgorithm

Codes and more at: http://bouboulis.mysch.gr/kernels.html




Thank you

Thank you!Questions please



robust kernel-based regression using orthogonal matching pursuit

Education