Data Mining Course 2007

Eric Postma

Clustering

Overview

Three approaches to clustering

1. Minimization of reconstruction error• PCA, nlPCA, k-means clustering

2. Distance preservation• Sammon mapping, Isomap, SPE

3. Maximum likelihood density estimation• Gaussian Mixtures

• These datasets have identical statistics up to 2nd order

1. Minimization of reconstruction error

Illustration of PCA (1)

• Face dataset (Rice database)

Illustration of PCA (2)

• Average face

Illustration of PCA (3)

• Top 10 Eigenfaces

Each 39-dimensional data item describes different aspects of the welfare and poverty of one country.

2D PCA projection

Non-linear PCA

• Using neural networks (to be discussed tomorrow)

2. Distance preservation

Sammon mapping

• Given a data set X. The distance between any two samples is defined as Dij

• We consider the projection on a two dimensional plane where the projected points are separated by dij

• Define an Error function

i jiij

ijij

i ji ij DdD

DE

21

Sammon mapping

Main limitations of Sammon

• The Sammon mapping procedure is a gradient descent method

• Main limitation: local minima

• MDS may be preferred because it finds global minima (being based on PCA)

• Both methods have difficulty with “curved or curly subspaces”

Isomap

• Tenenbaum• Build a graph in which each node

represents a data point• Compute shortest distances along the

graph (e.g., Dijkstra’s algorithm)• Store all distances in a matrix D• Perform MDS on the matrix D

Illustration of Isomap (1)

• For two arbitrary points on the manifold Euclidean distance does not always reflect similarity (cf. dashed blue line)

Illustration of Isomap (2)

• Isomap finds the appropriate shortest path along the graph (red curve, for K=7, N=1000)

Illustration of Isomap (3)

• Two-dimensional embedding (red line is the shortest path along the graph, blue line is the true distance in the embedding.

Illustration of Isomap (4)

• Isomaps (●) ability to find the intrinsic dimensionality as compared to PCA and MDS (∆ and o).

Illustration of Isomap (5)

Illustration of Isomap (6)

Illustration of Isomap (7)

• Interpolation along a straight line

Stochastic Proximity Embedding

• SPE algorithm

• Agrafiotis, D.K. and Xu, H. (2002). A self-organizing principle for learning nonlinear manifolds. Proceedings of the National Academy of Sciences U.S.A.

Stress function

Output proximity between points i and j Input proximity between points i and j

2)(),( ijijijij rdrdf

Swiss roll data set

Original 3D set 2D embedding obtained by SPE

Stress as a function of embedding dimension(averaged over 30 runs)

Scalability (# steps for four set sizes)Linear scaling

Conformations of methylpropyletherC1C2C3O4C5

Diamine combinatorial library

Clustering

• Minimize the total within-cluster variance (reconstruction error)

C

c

N

i ci

ic wxkE2

• kic = 1 if a data point belongs to cluster cK-means clustering1. Random selection of C cluster centres2. Partition the data by assigning them to the clusters3. The mean of each partitioning is the new cluster centreA distance threshold may be used…

• Effect of distance threshold on the number of clusters

Main limitation of k-means clustering

• Final partitioning and cluster centres depend on initial configuration

• Discrete partitioning may introduce errors

• Instead of minimizing the reconstruction error, we may maximize the likelihood of the data (given some probabilistic model)

Neural algorithms related to k-means

• Kohonen self-organizing feature maps

• Competitive learning networks

3. Maximum likelihood

Gaussian Mixtures

• Model the pdf of the data using a mixture of distributions

• K is the number of kernels (<< # data points)• Common choice for the component densities p(x|i):

K

iiPixpxp )()|()(

2

2

2/2 2exp

)2(1)|(

i

id

i

xixp

Illustration of EM applied to GM model

The solid line gives the initialization of the EM algorithm: two kernels,P(1) = P(2) = 0:5, μ1 = 0.0752; μ2 = 1.0176, σ1 = σ2 = 0:2356

Convergence after 10 EM steps..

Relevant literature

• L.J.P. van der Maaten, E.O. Postma, and H.J. van den Herik (submitted). Dimensionality Reduction: A Comparative Review.

• http://www.cs.unimaas.nl/l.vandermaaten