the infinite hierarchical factor regression model
DESCRIPTION
The Infinite Hierarchical Factor Regression Model. Piyush Rai and Hal Daume III NIPS 2008. Presented by Bo Chen March 26, 2009. Outline. Introduction The Infinite Hierarchical Factor Regression Model Indian Buffet Process and Beta Process Experiment Summary. Introduction. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/1.jpg)
The Infinite Hierarchical Factor Regression Model
Piyush Rai and Hal Daume IIINIPS 2008
Presented by Bo ChenMarch 26, 2009
![Page 2: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/2.jpg)
Outline
• Introduction • The Infinite Hierarchical Factor Regression
Model• Indian Buffet Process and Beta Process• Experiment• Summary
![Page 3: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/3.jpg)
Introduction• The latent factor representation benefits: 1. Discovering the latent process underlying the data 2. Simpler predictive modeling through a compact data
representation. Large P, Small N. N>=10 · d · C
• The fundamental advantages over standard FA model: 1. not assume known number of factors; 2. not assume factors are independent; 3. not assume all features are relevant to the factor
analysis.
![Page 4: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/4.jpg)
Algorithm Model
:
![Page 5: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/5.jpg)
Graphical Model
T is used to eliminate the spurious genes or noise features.So Tp determines whether the p-th customer will enter restaurant to eat anydish.
![Page 6: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/6.jpg)
Indian Buffet Process--from latent classes to latent features
• For a finite feature model:
(Tom Griffiths, 2006)
• Indian restaurant with countably many infinite dishes
![Page 7: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/7.jpg)
Differences between DP and IBP
DP class matrix
IBP ‘class’ matrix
1. Latent feature 2. Clustering 3. others
Different styles match different problems.
![Page 8: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/8.jpg)
Two-Parameter Finite Model
the first customer samples Poisson( ) dishes the i-th customer
samples a previously sampled dish with probability
then samples new dishes
(Z. Ghahramani et. al., 2006)
![Page 9: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/9.jpg)
Beta Process V.S. IBP
• Beta Process:
the first customer samples Poisson( ) dishes the i-th customer
samples a previously sampled dish with probability
then samples new dishes
![Page 10: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/10.jpg)
Hierarchical Factor Prior• Kingman’s Coalescent It is a distribution over the genealogy of a countably infinite set of
individuals. Construct tree structure
• Brownian diffusion A Markov process which encodes message (mean and covariance)
in each node of the above tree.
Y. W. Teh, H. Daume III, and D. M. Roy. Bayesian Agglomerative Clustering with Coalescents. In NIPS, 2008.
![Page 11: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/11.jpg)
Feature Selection Prior• Some genes are spurious
Before selecting dishes, these ‘spurious’ customers
should leave the restaurant.
![Page 12: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/12.jpg)
Provided by Piyush Rai
![Page 13: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/13.jpg)
Experimental results
E-coli data:100 samples 50 genes8 underlying factors
Breast cancer data:251 samples226 genes5 underlying factors
1. The hierarchy can be used to find factors in order of their prominence.2. Hierarchical modeling results in better predictive performance for the
factor regression task.3. The factor hierarchy leads to faster convergence since most of the unlikely
configurations will never be visited as they are constrained by the hierarchy.
![Page 14: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/14.jpg)
The Comparison of Factor Loading Matrice Learned from Different Methods
Ground Truth NIPS Method
Sparse BPFA on Factor loading VB Sparse BPFA on Factor score VB
![Page 15: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/15.jpg)
Factor Regression
Training and test data are combined together and test responsesare treated as missing values to be imputed.
![Page 16: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/16.jpg)
The Existing Similar FA Models• Putting binary matrix on factor score matrix
David Knowles and Zoubin Ghahramani. Infinite Sparse Factor Analysis and Infinite Independent Components Analysis, ICA 2007John Paisley et. al., Nonparametric Factor Analysis with Beta Process Priors, in submission 2009.
Summary: 1. For ‘large P, small N’ problems, the first one is faster to learn thesmall factor score matrix with KxN. Considering MCMC solution, it is difficult for the second one to handle the problem with tens of thousands of genes . 2. The second one can give an explanation to the relationship between geneand factor (pathway).
• Putting binary matrix on factor loading matrix
Piyush Rai and Hal Daume III. The Infinite Hierarchical Factor Regression Model, NIPS 2008.
![Page 17: The Infinite Hierarchical Factor Regression Model](https://reader035.vdocuments.mx/reader035/viewer/2022081503/568167f6550346895ddd7035/html5/thumbnails/17.jpg)
The New Developments of IBP
F. Doshi, K. T. Miller, J. Van Gael and Y.W. Teh, Variational Inference for the Indian Buffet Process, AISTATS 2009.
Jurgen Van Gael, Yee Whye Teh, Zoubin Ghahramani , The Infinite Factorial Hidden Markov Model, NIPS 2008.
K. A. Heller and Zoubin Ghahramani, A Nonparametric Bayesian Approach to Modeling Overlapping Clusters, AISTATS 2007.