010_20160216_variational gaussian process
TRANSCRIPT
Variational Gaussian Process
Tran Quoc Hoan
@k09ht haduonght.wordpress.com/
10 February 2016, Paper Alert, Hasegawa lab., Tokyo
The University of Tokyo
Dustin Tran, Rajesh Ranganath, David M.Blei ICLR 2016
2
Background
p(x) = p(x|z)p(z)
………
………
Parameters
………
………
Parameters✓�
Inference model Generative model
Observationsx
Hidden variables zq�(z|x)
p✓(x|z)
z ⇠ p✓(z)
In variational auto encoder (VAE), parameters are displayed as neural networks
3
Summary• Deep generative models provide complex representation
of data
• Variational inference methods require a rich family of approximating distribution
• They develop a powerful variational model - the variational Gaussian process (VGP)
• They prove a universal approximation theorem: the VGP can capture any continuous posterior distribution.
• They derive an efficient black box algorithm.
4
Variational Models• We want to compute posterior p(z|x) (z: latent variables, x: data)
• Variational inference seeks to minimize for a family q(z;�)
KL(q(z;�)||p(z|x))
• Maximizing evidence lower bound (ELBO)
log p(x) � Eq(z;�)[log p(x|z)]�KL(q(z;�)||p(z))
• (Common) Mean-field distribution q(z;�) =Y
i
q(zi;�i)
• Hierarchical variational models
• (Newer) Interpret the family as a variational model for posterior latent variables z (introducing new latent variables)[1]
Lawrence, N. (2000). Variational Inference in Probabilistic Models. PhD thesis.
5
Gaussian Processes
6
Gaussian Processes
7
Variational Gaussian Processes
8
Variational Gaussian Processes
9
Variational Gaussian ProcessesMean-fields parameters
Induces correlation btw latent variables of the variational model
10
Universal Approximation Theorem
11
Variational Lower Bound
auxiliary model
Variational latent variable space
Posterior latent variable space
Data space
12
Auto-Encoding Variational Models
Take both xn, zn as input
13
Black Box Stochastic Optimization
14
Black Box Stochastic Optimization
???
15
Black box inference
16
Experiments
17
Experiments