spectral clustering jianping fan dept of computer science unc, charlotte
TRANSCRIPT
![Page 1: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/1.jpg)
Spectral Clustering
Jianping Fan
Dept of Computer Science
UNC, Charlotte
![Page 2: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/2.jpg)
2
Lecture Outline
Motivation Graph overview and construction Spectral Clustering Cool implementations
![Page 3: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/3.jpg)
3
Semantic interpretations of clusters
![Page 4: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/4.jpg)
4
Spectral Clustering Example – 2 Spirals
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
-2 -1.5 -1 -0.5 0 0.5 1 1.5 2
Dataset exhibits Dataset exhibits complex cluster shapescomplex cluster shapes
K-meansK-means performs very performs very poorly in this space due poorly in this space due bias toward dense bias toward dense spherical clusters.spherical clusters.
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
-0.709 -0.7085 -0.708 -0.7075 -0.707 -0.7065 -0.706In the embedded space In the embedded space given by two leading given by two leading eigenvectors, clusters eigenvectors, clusters are trivial to separate.are trivial to separate.
![Page 5: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/5.jpg)
Original Points K-means (2 Clusters)
Spectral Clustering Example
Why k-means fail for these two examples?
Geometry vs. Manifold
![Page 6: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/6.jpg)
6
Lecture Outline
Motivation Graph overview and construction Spectral Clustering Cool implementation
![Page 7: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/7.jpg)
7
Graph-based Representation of Data Similarity
![Page 8: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/8.jpg)
8
Graph-based Representation of Data Similarity
similarity
![Page 9: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/9.jpg)
9
Graph-based Representation of Data Relationship
![Page 10: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/10.jpg)
10
Manifold
![Page 11: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/11.jpg)
11
Graph-based Representation of Data Relationships
Manifold
![Page 12: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/12.jpg)
12
Graph-based Representation of Data Relationships
![Page 13: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/13.jpg)
13Data Graph Construction
![Page 14: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/14.jpg)
14
Graph-based Representation of Data Relationships
![Page 15: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/15.jpg)
15
Graph-based Representation of Data Relationships
![Page 16: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/16.jpg)
16
![Page 17: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/17.jpg)
17
Graph-based Representation of Data Relationships
![Page 18: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/18.jpg)
18
Graph-based Representation of Data Relationships
![Page 19: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/19.jpg)
19
Graph Cut
![Page 20: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/20.jpg)
20
Lecture Outline
Motivation Graph overview and construction Spectral Clustering Cool implementations
![Page 21: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/21.jpg)
21
Graph-based Representation of Data Relationships
![Page 22: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/22.jpg)
22
![Page 23: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/23.jpg)
23
Graph Cut
![Page 24: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/24.jpg)
24
![Page 25: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/25.jpg)
25
![Page 26: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/26.jpg)
26
![Page 27: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/27.jpg)
27
![Page 28: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/28.jpg)
28
Graph-based Representation of Data Relationships
![Page 29: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/29.jpg)
29
Graph Cut
![Page 30: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/30.jpg)
30
![Page 31: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/31.jpg)
31
![Page 32: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/32.jpg)
32
![Page 33: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/33.jpg)
33
![Page 34: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/34.jpg)
34
Eigenvectors & Eigenvalues
![Page 35: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/35.jpg)
35
![Page 36: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/36.jpg)
36
![Page 37: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/37.jpg)
37
Normalized Cut
A graph G(V, E) can be partitioned into two disjoint sets A, B
Optimal partition of the graph G is achieved by minimizing the cut
Cut is defined as :
Min) (
![Page 38: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/38.jpg)
38
Normalized Cut
Normalized Cut
Association between partition set and whole graph
![Page 39: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/39.jpg)
39
Normalized Cut
![Page 40: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/40.jpg)
40
Normalized Cut
![Page 41: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/41.jpg)
41
Normalized Cut
![Page 42: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/42.jpg)
42
Normalized Cut
Normalized Cut becomes
Normalized cut can be solved by eigenvalue equation:
![Page 43: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/43.jpg)
43
K-way Min-Max Cut
Intra-cluster similarity
Inter-cluster similarity
Decision function for spectral clustering
![Page 44: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/44.jpg)
44
Mathematical Description of Spectral Clustering
Refined decision function for spectral clustering
We can further define:
![Page 45: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/45.jpg)
45
Refined decision function for spectral clustering
This decision function can be solved as
![Page 46: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/46.jpg)
46
Spectral Clustering Algorithm Ng, Jordan, and Weiss
Motivation Given a set of points
We would like to cluster them into k subsets
1,...,l
nS s s R
![Page 47: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/47.jpg)
47
Algorithm
Form the affinity matrix Define if
Scaling parameter chosen by user
Define D a diagonal matrix whose
(i,i) element is the sum of A’s row i
nxnW Ri j
0iiW
2 2|| || / 2i js s
ijW e
![Page 48: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/48.jpg)
48
Algorithm
Form the matrix
Find , the k largest eigenvectors of L These form the the columns of the new
matrix X Note: have reduced dimension from nxn to nxk
1/ 2 1/ 2L D WD
1 2, ,..., kx x x
![Page 49: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/49.jpg)
49
Algorithm
Form the matrix Y Renormalize each of X’s rows to have unit length Y
Treat each row of Y as a point in Cluster into k clusters via K-means
2 2/( )ij ij ijj
Y X X kR
nxkR
![Page 50: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/50.jpg)
50
Algorithm
Final Cluster Assignment Assign point to cluster j iff row i of Y was
assigned to cluster jis
![Page 51: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/51.jpg)
51
Why?
If we eventually use K-means, why not just apply K-means to the original data?
This method allows us to cluster non-convex regions
![Page 52: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/52.jpg)
52
Some Examples
![Page 53: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/53.jpg)
53
![Page 54: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/54.jpg)
54
![Page 55: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/55.jpg)
55
![Page 56: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/56.jpg)
56
![Page 57: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/57.jpg)
57
![Page 58: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/58.jpg)
58
![Page 59: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/59.jpg)
59
![Page 60: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/60.jpg)
60
![Page 61: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/61.jpg)
61
User’s Prerogative
Affinity matrix construction Choice of scaling factor
Realistically, search over and pick value that gives the tightest clusters
Choice of k, the number of clusters Choice of clustering method
2
![Page 62: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/62.jpg)
62
0
5
10
15
20
25
30
35
40
45
50
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
K
Eig
enva
lue
Largest Largest eigenvalueseigenvalues
of Cisi/Medline of Cisi/Medline datadata
λ1
λ2
How to select k? Eigengap: the difference between two consecutive eigenvalues. Most stable clustering is generally given by the value k that
maximises the expression
1k k k
Choose Choose k=2k=2
12max k
![Page 63: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/63.jpg)
63
Recap – The bottom line
![Page 64: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/64.jpg)
64
Summary
Spectral clustering can help us in hard clustering problems
The technique is simple to understand The solution comes from solving a simple
algebra problem which is not hard to implement
Great care should be taken in choosing the “starting conditions”
![Page 65: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/65.jpg)
Spectral ClusteringSpectral Clustering
![Page 66: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/66.jpg)
Spectral ClusteringSpectral Clustering
![Page 67: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/67.jpg)
Spectral ClusteringSpectral Clustering
![Page 68: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/68.jpg)
Spectral ClusteringSpectral Clustering
![Page 69: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/69.jpg)
Spectral ClusteringSpectral Clustering
![Page 70: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/70.jpg)
Spectral ClusteringSpectral Clustering
![Page 71: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/71.jpg)
Spectral ClusteringSpectral Clustering
![Page 72: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/72.jpg)
Spectral ClusteringSpectral Clustering
![Page 73: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/73.jpg)
Spectral ClusteringSpectral Clustering
![Page 74: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/74.jpg)
Spectral ClusteringSpectral Clustering
![Page 75: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/75.jpg)
Spectral ClusteringSpectral Clustering
![Page 76: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/76.jpg)
Spectral ClusteringSpectral Clustering
![Page 77: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/77.jpg)
Spectral ClusteringSpectral Clustering
![Page 78: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/78.jpg)
![Page 79: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/79.jpg)
![Page 80: Spectral Clustering Jianping Fan Dept of Computer Science UNC, Charlotte](https://reader030.vdocuments.mx/reader030/viewer/2022032703/56649f505503460f94c72a7f/html5/thumbnails/80.jpg)