discrete thin-plate splines - institute for mathematical … · · 2017-10-25... ( mathematical...

Discrete thin-plate splines

Linda Stals

Mathematical Sciences Institute, Australian National University

Linda Stals ( Mathematical Sciences Institute, Australian National University)Discrete thin-plate splines 1 / 65

Thin-Plate Splines

3D Image Recovery

Finger Print Analysis

Image Warping

Medical Image Analysis

Data Mining


Smoothing Splines

Given a set of attributes vectors x = (x1, x2, · · · , xd)T , build a predictivemodel

y = f (x).

y ≈ f (x).

To estimate f by a 2nd-order smoothing spline minimise:

Jα(f ) =1

n

n∑i=1

(f (x(i))− y (i))2 + α

∫Ω

∑|ν|=2

(2

ν

)(Dν f (x))2dx.

The first term penalises lack of fit, the second penalises roughness.


Radial Basis Functions

The standard approach is to represent f as a linear combination of radialbasis functions

f (x) =M∑k=1

aφk(x) + α

n∑i=1

wiU(x, x(i)),

where φk are monomials of order up to 1 and U are suitable radial basisfunctions.Favoured method as it gives an analytical solution.


Radial Basis Functions

2D Eg:

U(x, x(i)) =−1

16πr2ln(r).


Thin Plate Splines

Requires a solution of a dense system of matrices.

System may be ill-conditioned.

Size increases with the number of data points.

Not practical for large data sets.


Finite Element Approximation

Represent f as a linear combination of linear finite elements.In vector notation f will be of the form

f (x) = b(x)Tc.

Minimise Jα over all f of this form

Jα(f ) =1

n

n∑i=1

(f (x(i))− y (i))2 + α

∫Ω

∑|ν|=2

(2

ν

)(Dν f (x))2dx.

[S. Roberts, M. Hegland, I. Altas; SINUM 2003]


Non-Conforming Finite elements

The smoothing term (derivatives) is not defined for piecewise multi-linearfunctions.Use non-conforming finite elements.Represent the gradient of f by u = (bTg1, ...,b

Tgd) where∫Ω∇f (x) · ∇v(x) dx =

∫Ωu(x) · ∇v(x) dx,

for all piecewise multi-linear function v .


Non-Conforming Finite elements

∫Ω∇f (x) · ∇v(x) dx =

∫Ωu(x) · ∇v(x) dx,

is equivalent to

Lc =d∑

s=1

Gsgs ,

where L is a discrete approximation to the negative Laplace operator and(G1, ...,Gd) is a discrete approximation to the transpose of the gradientoperator.


Finite Element Approximation

2nd-order smoothing spline: minimise

Jα(f ) =1

n

n∑i=1

(f (x(i))− y (i))2 + α

∫Ω

∑|ν|=2

(2

ν

)(Dν f (x))2dx.

Finite element approximation: minimise

Jα(c, g1, g2, · · · , gd) =1

n

n∑i=1

(f (x(i))− y (i))2 + α

d∑s=1

gTs Lgs ,

subject to

Lc =d∑

s=1

Gsgs .


2D Formulation

2nd-order smoothing spline: minimise

Jα(f ) =1

n

n∑i=1

(f (x(i))− y (i))2

+ α

∫Ω

((∂2

1 f (x))2

+ 2 (∂1∂2f (x))2 +(∂2

2 f (x))2)dx,

Jα(c, g1, g2) =1

n

n∑i=1

(b(x(i))Tc− y (i))2

+ α

∫Ω∇bT (x)g1.∇bT (x)g1 +∇bT (x)g2.∇bT (x)g2 dx


2D Formulation

Minimise:

Jα(c , g1, g2) = cTAc− 2dTc + ‖y‖2/n + α(gT1 Lg1 + gT2 Lg2)

subject toLc = G1g1 + G2g2.

Where

A =1

n

n∑i=1

b(x(i))b(x(i))T ,

and

d =1

n

n∑i=1

b(x(i))y (i).


Discrete System

Recall :

Minimise:

Jα(c , g1, g2) = cTAc− 2dTc + ‖y‖2/n + α(gT1 Lg1 + gT2 Lg2)

subject toLc = G1g1 + G2g2.

A 0 0 L0 αL 0 −GT

1

0 0 αL −GT2

L −G1 −G2 0

cg1

g2

w

=

d000

−h1

h2

h3

h4

,w is a Lagrange multiplier.The vectors h1, · · · ,h4 store the boundary information. The boundaryconditions can be Dirichlet or Neumann.


Boundary Condition Examples - Dirichlet

x(1) = (0.25, 0.25), x(2) = (0.75, 0.25), x(3) = (0.25, 0.75),x(4) = (0.75, 0.75); y (1) = 1, y (2) = 0, y (3) = 0 and y (4) = 1

hf (x) = tps fit. hf (x) = 0.


Boundary Condition Examples - Neumann

x(1) = (0.25, 0.25), x(2) = (0.75, 0.25), x(3) = (0.25, 0.75),x(4) = (0.75, 0.75); y (1) = 1, y (2) = 0, y (3) = 0 and y (4) = 1

0 0.2

0.4 0.6

0.8 1 0

0.2

0.4

0.6

0.8

1-0.2

0 0.2 0.4 0.6 0.8

1 1.2 1.4 1.6

Boundary Condition Test 3

Neumann


Further Model Problems - Peak

exp

(−50

(0.5− x

(j)1

)2)

exp

(−50

(0.5− x

(j)2

)2)

+ ε,

where ε is a normally distributed random variable with mean 0 and avariance of 0.05

0

0.2

0.4

0.6

0.8

1

0 0.2

0.4 0.6

0.8 1

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

Original Data

x

y

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

0

0.2

0.4

0.6

0.8

1

0 0.2

0.4 0.6

0.8 1

-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Spline Fit

x

y

-0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Boundary conditions: hf = fy , hu = ∇fy and hλ = −α∆fy .


Further Model Problems - Sin

sin(4πx1) sin(4πx2)

Finite element grid of size m = 4225 with different values of α.

0 0.1 0.2 0.3 0.4 0.5 0.3 0.4

0.5 0.6

0.7 0.8

0.9 1

-1

-0.5

0

0.5

1

Test Problem 6 with alpha = 0.0001

0 0.1 0.2 0.3 0.4 0.5 0.3 0.4

0.5 0.6

0.7 0.8

0.9 1

-1

-0.5

0

0.5

1

Test Problem 6 with alpha = 0.000001

Boundary conditions: hf = fy , hu = ∇fy and hλ = −α∆fy .[M. F. Hutchinson; Communications in Statistics 1989]


Sine Example with Holes

y(x , y) = sin(2πx) sin(2πy), such that y(x , y) < 0.

n = 179401, m = 4229 with α = 10−6

Boundary: hf (x) = y(x), hu = ∇hf (x), hλ(x) = −α∆hf .


Use of FEM Grid

(0.25− ‖x(j) − [0.5, 0.5]T‖1)× (0.5− ‖x(j) − [0.5, 0.5]T‖1)

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

y

x

FEM grid - Diamond with Hole


Use of FEM Grid

Examples of initial FEM grid, with the centre.

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

y

x

FEM grid - Diamond


Sphere Example

Grid - 189 Nodes, α = 10−3.


Sphere Example

Grid - 68705 Nodes, α = 10−3.


Semi Sphere Example

Grid - 68705 Nodes, α = 10−3.


Two Sphere Example

Grid - 68705 Nodes, α = 10−7.


Convergence on Smooth Problem

y (i) = fy (x(i)) where ∇4fy = 0.

f (x) = fy (x) =

∥∥∥∥x +

[0.50.5

]∥∥∥∥2

-30

-25

-20

-15

-10

-10-9-8-7-6-5-4-3-2

log2

of

L2

Err

or

log2 of Grid Spacing

L2 Error for Model Problem 3

cg1w

interpol

-30

-25

-20

-15

-10

-10-9-8-7-6-5-4-3-2

log2

of

L2

Err

or



O(h) convergenceO(h2) convergence

f (x) = fy (x) = cosh(2πx1) sin(2πx2)

-20

-15

-10

-5

0

5

-10-9-8-7-6-5-4-3-2

log2

of

L2

Err

or



cg1g2w

interpol

-20

-15

-10

-5

0

5

-10-9-8-7-6-5-4-3-2

log2

of

L2

Err

or



O(h) convergenceO(h2) convergence


Solution Process

Recall: A 0 0 L0 αL 0 −GT

1

0 0 αL −GT2

L −G1 −G2 0

cg1

g2

w

=

d000

−h1

h2

h3

h4

.Scale by 1/α to get

A/α 0 0 L0 L 0 −GT

1

0 0 L −GT2

L −G1 −G2 0

cg1

g2

w

=

d/α

000

−h1/αh2/αh3/αh4

.The vectors h1, · · · ,h4 store the boundary information. (Assume Dirichletfor now.)


Data Projection Matrix

The matrix A = BBT , where

BT =1√n

b1(x(1)) · · · bm(x(1))...

......

b1(x(n)) · · · bm(x(n))

,is symmetric, nonnegative and sparse.B : Data points→ FEM.The properties of the A matrix varies with the distribution of the datapoints.

If no data points lie in the support of basis function bi , the entries inthe ith column are zero. In which case A, has a non-trivial null-space.

If any triangles has less than two data points, the system isunder-determined system and thus A has a non-trivial null-space



Let P =[0 Ip

]be a projection onto the region of the domain containing

data points. ( Ip be an identity matrix).The matrix Ap := PAPT is defined on the region of the domain wherethere are data points.Note that A = PTApP, since A is zero in the region where there are nodata points.



Furthermore, Ap may be decomposed as follows

Ap =

[Boo Bou

0 Buu

] [BToo 0

BTou BT

uu

]= QT

[AToo Bou

BTou Iu

]Q,

where Iu is an identity matrix, Aoo = BooBToo + BouB

Tou and

Q =

[Io 00 BT

uu

]with Io being an identity matrix.



Let

F =

[AToo Bou

BTou Iu

].

Then Ap = QTFQ and A = (PTQT )F (QP).Note that F is SPD.


Example Data Distribution

A

Figure : Example distribution of data in a uniform finite element gridcorresponding to the A matrix



A_p

Figure : Example distribution of data in a uniform finite element gridcorresponding to the Ap matrix. The empty triangles have been removed



F

Figure : Example distribution of data in a uniform finite element gridcorresponding to the F matrix. Work directly on the data points in regions wherethere are not a lot of data points.


SMW formula

Recall:

S =

L −G1 −G2 00 L 0 −GT

1

0 0 L −GT2

A/α 0 0 L

If P1 =

[0 · · · 0 P

]and P2 =

[P 0 · · · 0

]then M + PT

1 ApP2 =M + PT

1 QTFQP2 is equivalent to above matrix where

M =

L −G1 −G2 00 L 0 −GT

1

0 0 L −GT2

0 0 0 L

.This is an upper triangular matrix with the Laplacian matrix sitting alongthe main diagonal. As the L matrices are SPD the above system can bereadily inverted by using back substitution.


SMW formula

Recall:

(M + X1FX

T2

)−1= M−1 −M−1X1

[F−1 + XT

2 M−1X1

]−1XT

2 M−1.

According to the to Sherman–Morrison–Woodbury formula

S−1 = M−1 −M−1PT1 QT

[F−1 + QPHPTQT/α

]−1QP2M

−1,

where H = L−1(∑d

s=1 GsL−1GT

s

)L−1.

M is an upper triangular matrix and is easily inverted.

The Laplacian matrix L is SPD and L−1 is applied to a vector byusing an optimal solver such as the multigrid method.

Most of the work goes into solving systems of the form Eαv = αf,where Eα :=

(αF−1 + QPHPTQT

). The matrix Eα is SPD so a PCG

solver is used.


Eα Condition Number

To measure the conditioning we need to determine the eigenvalues of L, Fand Gs .We firstly look at a uniform finite element grid defined on the onedimensional domain [0, 1] with grid spacing h = 1/(m + 1).If the domain contains a lot of data spread throughout,

Eα = αA−1u + L−1

(G1L

−1GT1

)L−1.

[L. Stals; Journal of Scientific Computing 2013]


Laplacian Matrix

For a uniform finite element grid in one dimension, the stencils for L is

L :1

h

[−1 2 −1

].

Following Hackbusch, we can show the eigenvalues of L are

λµ(L) =4

hsin2

(πµh

2

),

and the eigenvectors are uµ where uµj = sin (πµjh) for 1 ≤ µ ≤ m and1 ≤ j ≤ m.


Data Matrix

Recall:

A =1

n

n∑j=1

b(x(j))b(x(j))T .

If there are a lot of data points throughout the whole domain, i.e. n islarge,

Ak,l ≈∫

Ωbk(x)bl(x) dx.

We construct a matrix Au, where (Au)k,l =∫

Ω bkbl dx, that is defined bythe stencil

Au :h

6

[1 4 1

].

Consequently, Au can be used as an approximation to A.


Data Matrix

By using the same eigenvectors that we used for L, uµ, we get

λµ (Au) =h

3

(2 cos2

(πµh

2

)+ 1

).


G1L−1GT

1

We have uµ is also an eigenvector for G1L−1GT

1 with correspondingeigenvalues

λµ(G1L

−1GT1

)= h cos2

(πµh

2

).


Theorem

IfEα = αA−1

u + L−1(G1L

−1GT1

)L−1,

then

λµ(Eα) =3α

h

1

2 cos2(πµh

2

)+ 1

+h3

16

cos2(πµh

2

)sin4

(πµh

2

) ,λµ(AuEα) = α +

h

3

(2 cos2

(πµh

2

)+ 1

)h3

16

cos2(πµh

2

)sin4

(πµh

2

) ,and

λµ(LEαL) =48α

h3

sin4(πµh2 )

2 cos2(πµh

2

)+ 1

+ h cos2

(πµh

2

),

where 1 ≤ µ ≤ m.


Parallel Implementation

Recall :

We want to solve

S−1 = M−1 −M−1PT1 QT

[F−1 + QPHPTQT/α

]−1QP2M

−1,

where H = L−1(∑d

s=1 GsL−1GT

s

)L−1.

Expensive part:

Eα :=(αF−1 + QPHPTQT

).

(Eg. Eα = αA−1u + L−1

(G1L

−1GT1

)L−1. )


Saddle Point Formulation

Recall: (yet again)A/α 0 0 L

0 L 0 −GT1

0 0 L −GT2

L −G1 −G2 0

cg1

g2

w

=

d/α

000

−h1/αh2/αh3/αh4

.Swap the first and last rows to get

L −G1 −G2 00 L 0 −GT

1

0 0 L −GT2

A/α 0 0 L

cg1

g2

w

=

000

d/α

−

h4

h2/αh3/αh1/α

.


Matrix Properties

Symmetric Indefinite for Dirichlet

Symmetric and Singular for Neumann;

null space =

001

.


Matrix Properties

Conditioning = O(h2), but large constant.


Matrix Properties

Use SQMR method as solver directly on saddle point problem (butmust be careful about breakdowns.)

SQMR same as MINRES for SPD preconditioners. Unlike MINRESSQMR allows the use of an indefinite preconditioner. Not so muchtheory for SQMR.


Model Problem

y(x) = exp(−50 (x − 0.5)2

).

0.0 0.2 0.4 0.6 0.8 1.0x

−0.2

0.0

0.2

0.4

0.6

0.8

1.0 Example splines for different values of α

α=1.0−2

α=1.0−5

α=1.0−8

data

Figure : Spline fit to example test problem with α = 10−2, 10−5, 10−8 and gridspacing h = 2−6.


Convergence Rate I (α = 10−2)

0

1000

2000

3000

4000

5000

2 4 6 8 10 12

Num

ber

of Itera

tions

level

I Preconditioner (α = 10−2

)

248

163264

Dirichlet top Neumann did not converge(Similar for α = 10−5 and α = 10−8.)


Domain Decomposition Method

P−1

DD=

N∑i=1

RTi SiRi .

The boundary conditions on each subdomain is Dirichlet, so can use SMWapproach to solve problem on each subdomain.I did not find much theory on the use of DD methods with indefiniteproblems, so did testing with Python on 1D model problem before movingto proper parallel code.


Convergence Rate DD (α = 10−2)

0

100

200

300

400

500

2 4 6 8 10 12

Num

ber

of Itera

tions

level

DD Preconditioner (α = 10−2

)

248

163264

Dirichlet top Neumann did not converge



0

100

200

300

400

500

2 4 6 8 10 12

Num

ber

of Itera

tions

level


)

248

163264



Domain Decomposition Method With Coarse GridPreconditioner

P−1coarse = RT

0

(R0SRT

0

)−1R0 +

N∑i=1

RTi SiRi .

The coarse grid is designed to move the small eigenvalues away from zero.


Convergence Rate DD+Coarse (α = 10−2)

0

10

20

30

40

50

2 4 6 8 10 12

Num

ber

of Itera

tions

level

Coarse Preconditioner (α = 10−2

)

248

163264

0

10

20

30

40

50

2 4 6 8 10 12

Num

ber

of Itera

tions

level


)

48

163264

Dirichlet top Neumann bottomLinda Stals ( Mathematical Sciences Institute, Australian National University)Discrete thin-plate splines 54 / 65


0

10

20

30

40

50

2 4 6 8 10 12

Num

ber

of Itera

tions

level


)

248

163264

0

10

20

30

40

50

2 4 6 8 10 12

Num

ber

of Itera

tions

level


)

48

163264


Domain Decomposition Method - PBNN Preconditioner

Q = RT0

(R0SRT

0

)−1R0.

R = I − QS.

P−1

BNN= RTP−1

DDR + Q.

This technique is similar to coarse grid correction, but removesaccumulation of errors.


Convergence Rate DD+Projection (α = 10−2)

0

10

20

30

40

50

2 4 6 8 10 12

Num

ber

of Itera

tions

level

Projection Preconditioner (α = 10−2

)

248

163264

0

10

20

30

40

50

2 4 6 8 10 12

Num

ber

of Itera

tions

level


)

48

163264



0

10

20

30

40

50

2 4 6 8 10 12

Num

ber

of Itera

tions

level


)

248

163264

0

10

20

30

40

50

2 4 6 8 10 12

Num

ber

of Itera

tions

level


)

48

163264


2D Example

0 0.2

0.4 0.6

0.8 1

x 0

0.2

0.4

0.6

0.8

1

y

0

0.2

0.4

0.6

0.8

1

1.2

Original Data Set (641601 data points)


2D Example

FEM fit with 4225 nodes and α = 10−8.


Future Work

Finish looking into parallel solver.

Run some large 2D and 3D codes.


Adapative refinement

8

16

24

32

40

48

56

64

Error indicator

Reduce/Remove reading data in from files

Load balancing: take into account connection between FEM grid anddata points.


0

0.2

0.4

0.6

0.8

1

x

0

0.2

0.4

0.6

0.8

1

y

-10 0

10 20 30 40 50 60

Thank You


discrete thin-plate splines - institute for mathematical … · · 2017-10-25... ( mathematical...

Documents