graph embedding and extensions: a general framework for dimensionality reduction keywords:...
TRANSCRIPT
![Page 1: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/1.jpg)
Graph Embedding and Extensions: A General Framework for Dimensionality
Reduction
Keywords:
Dimensionality reduction, manifold learning, subspace learning, graph embedding framework.
![Page 2: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/2.jpg)
1.Introduction
• Techniques for dimensionality reduction Linear: PCA/LDA/LPP... Nonlinear: ISOMAP/Laplacian Eigenmap/LLE... Linear Nonlinear: kernel trick
• Graph embedding framework A unified view for understanding and explaining many po
pular algorithms such as the ones mentioned above. A platform for developing new dimension reduction algori
thms.
![Page 3: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/3.jpg)
2.Graph embedding2.1Graph embeddingLet m is often very large so we
need to find
Intrinsic graph: --similarity matrix
Penalty graph: --the similarity to be suppressed in the dimension-reduced feature space Y
NNRWWXG , ,
miN RxxxX ],,...[ 1
mmRyyxF m ',,: '
NNppp RWWXG , ,
![Page 4: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/4.jpg)
Our graph-preserving criterion is:
L is called Laplacian matrix
B typically is diagonal for scale normalization or L-matrix of the penalty graph
jiijii
T
dByyjiijji
dByy
WDWDL
LyyWyyyTT
,
minargminarg2*
![Page 5: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/5.jpg)
Linearization:
Kernelization:
Both can be obtained by solving:
wXy T
wXLXwWijyywii
T
dwwordXBXww
ji
dwwordXBXww
'
'
2
' '
minargminarg*
x:
)()(),(
minargminarg*
' '
2
' '
iiji
ii
T
dKordKBK
jT
iT
dKordKBK
xxxxk
KLKWijKK
vBvL TT XBXKBIBKLKXLXLL ,,,;,,
![Page 6: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/6.jpg)
Tensorization:
2.2General Framework for Dimensionality Reduction
ijji
njnidwwf
n WwwXwwXwwn
2
11)...(
1 ......minarg)*...(1
![Page 7: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/7.jpg)
The adjacency graphs for PCA and LDA. (a) Constraint and intrinsic graph in PCA. (b) Penalty and intrinsic graphs in LDA.
![Page 8: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/8.jpg)
2.3 Related Works and Discussions
2.3.1 Kernel Interpretation and Out-of-Sample Extension
• Ham et al. [13] proposed a kernel interpretation of KPCA,ISOMAP, LLE, and Laplacian Eigenmap
• Bengio et al. [4] presented a method for computing the low dimensional representation of out-of-sample data.
• Comparison:
Kernel Interpretation Graph embeding normalized similarity matrix laplacian matrix
unsupervised learning both supervised&unsupervised
![Page 9: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/9.jpg)
2.3.2 Brand’s Work [5]
yWDyy
Wyyy
T
Dyy
T
Dyy
T
T
)(minarg
maxarg
1
*
1
*
Brand’s Work can be viewed as a special case of the graph embedding framework
![Page 10: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/10.jpg)
2.3.3 Laplacian Eigenmap [3] and LPP [10]
• Single graph B=D• Nonnegative similarity matrix• Although [10] attempts to use LPP to explain PC
A and LDA, this explanation is incomplete.
The constraint matrix B is fixed to D in LPP, while the constraint matrix of LDA is comes from a penalty graph that connects all samples with equal weights;hence, LPP cannot explain LPP. Also,a minimization algorithm, does not explain why PCA maximizes the objective function.
![Page 11: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/11.jpg)
3 MARGINAL FISHER ANALYSIS
3.1 Marginal Fisher Analysis• Limitation of LDA:data distribution assumption
limited available projection directions• MFA overcomed the limitation by characterizing intraclass
compactness and interclass separability.
intrinsic graph: each sample is connected to its k1
nearest neighbors of the same class
(intraclass compactness)
penalty graph: each sample is connected to its k2
nearest neighbors of other classes
(interclass separability)
![Page 12: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/12.jpg)
![Page 13: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/13.jpg)
![Page 14: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/14.jpg)
Procedure of MFA
• PCA projection• Constructing the intraclass compactness and int
erclass separability graphs.• Marginal Fisher Criterion
• Output the final linear projection direction
![Page 15: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/15.jpg)
• The available projection directions are much greater than that of LDA
• There is no assumption on the data distribution of each class
• Without prior information on data distributions
Advantages of MFA
![Page 16: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/16.jpg)
KMFA
Projection direction:
The distance between sample xi and xj is
For a new data point x, its projection to the derivedoptimal direction is obtained as
![Page 17: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/17.jpg)
TMFA:
![Page 18: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/18.jpg)
4.Experiments4.1face recognition
4.1.1
MFA>Fisherface(LDA+PCA)>PCA
PCA+MFA>PCA+LDA>PCA
4.1.2Kernel trick
KDA>LDA,KMFA>MFA
KMFA>PCA,Fisherface,LPP
![Page 19: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/19.jpg)
Trainingset Adequate: LPP > Fisherface ,PCA Inadequate: Fisherface > LPP>PCA anyway, MFA>=LPPPerformance can be substantially improved by e
xploring a certain range of PCA dimensions first.PCA+MFA>MFA,Bayesian face >PCA,Fisherface,LPPTensor representation brings encouraging impro
vements compared with vector-based algorithms it is critical to collect sufficient samples for all su
bjects!
![Page 20: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/20.jpg)
4.2 A Non-Gaussian Case
![Page 21: Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,](https://reader035.vdocuments.mx/reader035/viewer/2022062407/56649cae5503460f94972268/html5/thumbnails/21.jpg)
5.CONCLUSION AND FUTURE WORK• All possible extensions of the algorithms m
entioned in this paper
• Combination of the kernel trick and tensorization
• The selection of parameters k1 and k2
• How to utilize higher order statistics of the data set in the graph embedding framework?