Expectation-Maximization (EM) Algorithm &Monte Carlo Sampling for Inference and Approximation

Expectation-Maximization Algorithm

“The Expectation-Maximization algorithm is a general technique for finding maximum likelyhood* solutions for probabilistic models having latent variables” (Dempster et al., 1977; McLachlan and Krishnan,

1997).

Is an iterative process and consists of two steps: E-step and M-step.

General purpose technique:

- Needs to be adapted for each application- Versatile. Used in machine learning, computer vision, language processing....

Intro: Maximum Likelihood Estimation methods

Maximum Likelihood Estimation (MLE) are methods to estimate parameters of an unknown, parameter-dependent probability density function p( x | θ ) from the observed sample (x1,x2,...,xn).

- When is EM useful?

- When MLE solutions are difficult or not possible to get because there are latent variables involved.

- Either missing values or we decide to get aditional unkown variables for modelling simplicity.

EM summarized

- Given a joint distribution p(Z , X | θ

) over obsereved variables X and latent variables Z, governed by parameters θ, the goal is to

optimize the likelihood function p( X | θ

) with respect to θ.

- Choose an inital setting for the parameters E step.

- Evaluate p(Z | X, )

- M step. Evaluate given by = arg max Q(θ, )

- Where Q(θ, )=Σ p( Z | X, ) ln p( X,Z | θ

)

- Check for convergence. If not satisfied, then ←

-from (Christopher M. Bishop, Patter Recognition and Machine Learning. Springer, 2006)

Monte Carlo Sampling for Inference and Approximation

- Inference – To draw conclusions from gathered data.

- Monte Carlo Sampling – Broad selection of computational algorithms that rely on repeated random sampling to obtain numerical results.

- For a better understanding we have prepared two very simple examples.

Rolling a dice

- We know that the probability of getting a 4 is:

- 1/6 (approx 17%)

- Can we obtain the same result by Monte Carlo simulation?

- More iterations give less error in the result!

Calculating the area of the unit circle

0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

10 iterations of Monte Carlo

Ratio: 2.4

Calculating the area of the unit circle

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1000 iterations of Monte Carlo

Ratio: 3.04

Calculating the area of the unit circle

- 1 million iterations:

- Ratio: 3.1400

- 100 million iterations:

- Ratio: 3.1416

And so forth!

Application of EM

- Pattern Recognition- Image Recognition - Computer vision- Maximum likelihood- Bioinformatics

Application of MC

- Finance- Statistics - Molecular dynamics- Computer Graphics- Fluid mechanics