010_20160216_variational gaussian process

Variational Gaussian Process

Tran Quoc Hoan

@k09ht haduonght.wordpress.com/

10 February 2016, Paper Alert, Hasegawa lab., Tokyo

The University of Tokyo

Dustin Tran, Rajesh Ranganath, David M.Blei ICLR 2016

2

Background

p(x) = p(x|z)p(z)

………

………

Parameters

………

………

Parameters✓�

Inference model Generative model

Observationsx

Hidden variables zq�(z|x)

p✓(x|z)

z ⇠ p✓(z)

In variational auto encoder (VAE), parameters are displayed as neural networks

3

Summary• Deep generative models provide complex representation

of data

• Variational inference methods require a rich family of approximating distribution

• They develop a powerful variational model - the variational Gaussian process (VGP)

• They prove a universal approximation theorem: the VGP can capture any continuous posterior distribution.

• They derive an efficient black box algorithm.

4

Variational Models• We want to compute posterior p(z|x) (z: latent variables, x: data)

• Variational inference seeks to minimize for a family q(z;�)

KL(q(z;�)||p(z|x))

• Maximizing evidence lower bound (ELBO)

log p(x) � Eq(z;�)[log p(x|z)]�KL(q(z;�)||p(z))

• (Common) Mean-field distribution q(z;�) =Y

i

q(zi;�i)

• Hierarchical variational models

• (Newer) Interpret the family as a variational model for posterior latent variables z (introducing new latent variables)[1]

Lawrence, N. (2000). Variational Inference in Probabilistic Models. PhD thesis.

5

Gaussian Processes

6

Gaussian Processes

7

Variational Gaussian Processes

8

Variational Gaussian Processes

9

Variational Gaussian ProcessesMean-fields parameters

Induces correlation btw latent variables of the variational model

10

Universal Approximation Theorem

11

Variational Lower Bound

auxiliary model

Variational latent variable space

Posterior latent variable space

Data space

12

Auto-Encoding Variational Models

Take both xn, zn as input

13

Black Box Stochastic Optimization

14

Black Box Stochastic Optimization

???

15

Black box inference

16

Experiments

17

Experiments