piecewise bounds for estimating bernoulli-logistic latent gaussian models
DESCRIPTION
Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models. Mohammad Emti yaz Khan Joint work with Benjamin Marlin, and Kevin Murphy University of British Columbia June 29, 2011. Modeling Binary Data. Main Topic of our paper - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/1.jpg)
Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models
Mohammad Emtiyaz Khan Joint work with Benjamin Marlin, and Kevin Murphy
University of British Columbia
June 29, 2011
![Page 2: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/2.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Modeling Binary Data
Main Topic of our paper
Bernoulli-Logistic Latent Gaussian Models (bLGMs)
Image from http://thenextweb.com/in/2011/06/06/india-to-join-the-open-data-revolution-in-july/
![Page 3: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/3.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
bLGMs - Classification ModelsBayesian Logistic Regression and Gaussian Process Classification (Jaakkola and Jordan 1996, Rasmussen 2004, Gibbs and Mackay 2000, Kuss and Rasmussen 2006, Nickisch and Rasmussen 2008, Kim and Ghahramani, 2003).
Figures reproduced using GPML toolbox
![Page 4: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/4.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
bLGMs - Latent Factor ModelsProbabilistic PCA and Factor Analysis models (Tipping 1999, Collins, Dasgupta and Schapire 2001, Mohammed, Heller, and Ghahramani 2008, Girolami 2001, Yu and Tresp 2004).
![Page 5: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/5.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Parameter Learning is IntractableLogistic Likelihood is not conjugate to the Gaussian prior.
x
We propose piecewise bounds to obtain tractable lower bounds to marginal likelihood.
![Page 6: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/6.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Learning in bLGMs
![Page 7: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/7.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Bernoulli-Logistic Latent Gaussian Models
Parameter Setz1n
y1n
n=1:N
µ Σ
W
z2n zLn
y2n yDny3n
![Page 8: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/8.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Learning Parameters of bLGMs
x
![Page 9: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/9.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
some other tractable terms in m and V
+
Variational Lower Bound (Jensen’s)
x
![Page 10: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/10.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Quadratic Bounds
• Bohning’s bound (Bohning, 1992)
x
![Page 11: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/11.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Quadratic Bounds
• Bohning’s bound (Bohning, 1992)
x
• Jaakkola’s bound (Jaakkola and Jordan, 1996)
• Both bounds have unbounded error.
![Page 12: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/12.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Problems with Quadratic Bounds1-D example with µ = 2, σ = 2
Generate data, fix µ = 2, and compare marginal likelihood and lower bound wrt σzn
yn
n=1:N
µ σ
As this is a 1-D problem, we can compute lower bounds without Jensen’s inequality. So plots that follow have errors only due to error in bounds.
![Page 13: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/13.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Problems with Quadratic BoundsBohning Jaakkola Piecewise
Q1(x)
Q2(x)
Q3(x)
![Page 14: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/14.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Piecewise Bounds
![Page 15: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/15.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Finding Piecewise Bounds• Find Cut points, and
parameters of each pieces by minimizing maximum error.
• Linear pieces (Hsiung, Kim and Boyd, 2008)
• Quadratic Pieces (Nelder-Mead method)
• Fixed Piecewise Bounds!• Increase accuracy by
increasing number of pieces.
![Page 16: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/16.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Linear Vs Quadratic
![Page 17: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/17.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Results
![Page 18: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/18.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Binary Factor Analysis (bFA)
Z1n
Y1n
n=1:N
W
Z2n ZLn
Y2n YDnY3n
• UCI voting dataset with D=15, N=435.
• Train-test split 80-20%• Compare cross-
entropy error on missing value prediction on test data.
![Page 19: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/19.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
bFA – Error vs Time
BohningJaakkolaPiecewise Linear with 3 piecesPiecewise Quad with 3 piecesPiecewise Quad with 10 pieces
![Page 20: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/20.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
bFA – Error Across Splits
Erro
r with
Pie
cew
ise
Qua
drat
ic
Error with Bohning and Jaakkola
BohningJaakkola
![Page 21: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/21.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Gaussian Process Classification• We repeat the experiments
described in Kuss and Rasmussen, 2006
• We set µ =0 and squared exponential Kernel
Σij =σ exp[(xi-xj)^2/s]
• Estimate σ and s. • We run experiments on
Ionoshphere (D = 200)• Compare Cross-entropy
Prediction Error for test data.
z1
y1
µ Σ
z2 zD
yDy2
X s σ
![Page 22: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/22.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
GP – Marginal Likelihood
![Page 23: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/23.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
GP – Prediction Error
![Page 24: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/24.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
EP vs Variational• We see that the variational approach underestimates
the marginal likelihood in some regions of parameter space.
• However, both methods give comparable results for prediction error.
• In general, the variational EM algorithm for parameter learning is guaranteed to converge when appropriate numerical methods are used,
• Nickisch and Rasmussen (2008) describe the variational approach as more principled than EP.
![Page 25: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/25.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Conclusions
![Page 26: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/26.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
• Fixed piecewise bounds can give a significant improvement in estimation and prediction accuracy relative to variational quadratic bounds.
• We can drive the error in the logistic-log-partition bound to zero by letting the number of pieces increase.
• This increase in accuracy comes with a corresponding increase in computation time.
• Unlike many other frameworks, we have a very fine grained control over the speed-accuracy trade-off through controlling the number of pieces in the bound.
Conclusions
![Page 27: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/27.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Thank You
![Page 28: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/28.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Piecewise-Bounds: Optimization Problem
![Page 29: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/29.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
![Page 30: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/30.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
![Page 31: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/31.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
![Page 32: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/32.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Latent Gaussian Graphical ModelLED dataset, 24 variables, N=2000
z1n
y1n
n=1:N
µ Σ
z2n zDn
yDny3n
![Page 33: Piecewise Bounds for Estimating Bernoulli-Logistic Latent Gaussian Models](https://reader036.vdocuments.mx/reader036/viewer/2022062310/568160a6550346895dcfc7ff/html5/thumbnails/33.jpg)
Piecewise Bounds for Binary Latent Gaussian Models
ICML 2011. Mohammad Emtiyaz Khan
Sparse Version