Total Variation and Euler's Elastica for Supervised Learning

Tong Lin, Hanlin Xue, Ling Wang, Hongbin Zha

Contact: tonglin123@gmail.com

Peking University, China

2012-6-29

1Key Lab. Of Machine Perception, School of EECS,

Peking University, China

Background• Supervised Learning:

• Definition: Predict u: x -> y, with training data (x1, y1), …, (xN, yN)

• Two tasks: Classification and Regression

• Prior Work：• SVM:

• RLS: Regularized Least Squares, Rifkin, 2002

2

Hinge loss:

Squared loss:

3

Background• Prior Work (Cont.)：

• Laplacian Energy: “Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples,” Belkin et al., JMLR 7:2399-2434, 2006

• Hessian Energy: “Semi-supervised Regression using Hessian Energy with an Application to Semi-supervised Dimensionality Reduction,” K.I. Kim, F. Steinke, M. Hein, NIPS 2009

• GLS: “Classification using geometric level sets,” Varshney & Willsky, JMLR 11:491-516, 2010

4

Motivation

SVM Our Proposed Method

5

Large margin should not be the sole criterion; we argue sharper edges and smoother boundaries can play significant roles.

3D display of the output classification function u(x) by the proposed EE model

6

• General：

• Laplacian Regularization (LR)：

• Total Variation (TV)：

• Euler’s Elastica (EE)：

1min ( ( ), ) ( )

n

i iiuL u x y S u

2 2min ( ) | |u

u y dx u dx

2 2min ( ) ( ) | |u

u y dx a b u dx

2min ( ) | |u

u y dx u dx

| |

u

u

Models

7

TV&EE in Image Processing• TV: a measure of total quantity of the value change• Image denoising (Rudin, Osher, Fatemi, 1992)

• Elastica was introduced by Euler in 1744 on modeling torsion-free elastic rods

• Image inpainting (Chan et al., 2002)

8

• TV can preserve sharp edges, while EE can produce smooth boundaries

• For details, see T. Chan & J. Shen’s textbook: Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods, SIAM, 2005

9

Decision boundary

The mean curvature k in high dimensional space can have same expression except the constant 1/(d-1).

Framework

10

11

• The calculus of variations → Euler-Lagrange PDE

3

1 1( ) ( '( ) | |) ( ( '( ) | |))

| | | |V n u u u u

u u

2min [ ] ( ) ( )J u u y dx S u

2( ) | |LRS u u dx

( ) | |TVS u u dx

2( ) ( ) | |EES u a b u dx

2( ) 0 (#)u u y 2( ) 0| |

uu y

u

2( ) 0V u y

2( ) a b

Energy Functional Minimization

12

Solutions

a. Laplacian Regularization (LR)

Radial Basis Function Approximation

b. TV & EE: We develop two solutions• Gradient descent time marching (GD)• Lagged linear equation iteration (LagLE)

13

Experiments: Two-Moon Data

SVM

EE

Both methods can achieve 100% accuracies with different parameter combinations

14

Experiments: Binary Classification

15

Experiments: Multi-class Classification

16

Experiments: Multi-class Classification

Note: Results of TV and EE are computed by the LagLE method.

17

Experiments: Regression

18

Conclusions• Contributions:

• Introduce TV&EE to the ML community

• Demonstrate the significance of curvature and gradient empirically

• Achieve superior performance for classification and regression

• Future Work：• Hinge loss

• Other basis functions

• Extension to semi-supervised setting

• Existence and uniqueness of the PDE solutions

• Fast algorithm to reduce the running time

End, thank you!