hidden variables, the em algorithm, and mixtures of gaussians
DESCRIPTION
03/15/12. Hidden Variables, the EM Algorithm, and Mixtures of Gaussians. Computer Vision CS 543 / ECE 549 University of Illinois Derek Hoiem. Today’s Class. Examples of Missing Data Problems Detecting outliers Latent topic models (HW 2, problem 3) Segmentation (HW 2, problem 4) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/1.jpg)
Hidden Variables, the EM Algorithm, and Mixtures of Gaussians
Computer VisionCS 543 / ECE 549
University of Illinois
Derek Hoiem
03/15/12
![Page 2: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/2.jpg)
Today’s Class• Examples of Missing Data Problems
– Detecting outliers– Latent topic models (HW 2, problem 3)– Segmentation (HW 2, problem 4)
• Background– Maximum Likelihood Estimation– Probabilistic Inference
• Dealing with “Hidden” Variables– EM algorithm, Mixture of Gaussians– Hard EM
![Page 3: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/3.jpg)
Missing Data Problems: OutliersYou want to train an algorithm to predict whether a photograph is attractive. You collect annotations from Mechanical Turk. Some annotators try to give accurate ratings, but others answer randomly.
Challenge: Determine which people to trust and the average rating by accurate annotators.
Photo: Jam343 (Flickr)
Annotator Ratings
108928
![Page 4: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/4.jpg)
Missing Data Problems: Object Discovery
You have a collection of images and have extracted regions from them. Each is represented by a histogram of “visual words”.
Challenge: Discover frequently occurring object categories, without pre-trained appearance models.
http://www.robots.ox.ac.uk/~vgg/publications/papers/russell06.pdf
![Page 5: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/5.jpg)
Missing Data Problems: SegmentationYou are given an image and want to assign foreground/background pixels.
Challenge: Segment the image into figure and ground without knowing what the foreground looks like in advance.
Foreground
Background
![Page 6: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/6.jpg)
Missing Data Problems: SegmentationChallenge: Segment the image into figure and ground without knowing what the foreground looks like in advance.
Three steps:1. If we had labels, how could we model the appearance of
foreground and background?2. Once we have modeled the fg/bg appearance, how do we
compute the likelihood that a pixel is foreground?3. How can we get both labels and appearance models at once?
Foreground
Background
![Page 7: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/7.jpg)
Maximum Likelihood Estimation
1. If we had labels, how could we model the appearance of foreground and background?
Foreground
Background
![Page 8: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/8.jpg)
Maximum Likelihood Estimation
nn
N
xp
p
xx
)|(argmaxˆ
)|(argmaxˆ
..1
x
xdata parameters
![Page 9: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/9.jpg)
Maximum Likelihood Estimation
nn
N
xp
p
xx
)|(argmaxˆ
)|(argmaxˆ
..1
x
x
Gaussian Distribution
2
2
2
2
2exp
2
1),|(
n
n
xxp
![Page 10: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/10.jpg)
Maximum Likelihood Estimation
nn
N
xp
p
xx
)|(argmaxˆ
)|(argmaxˆ
..1
x
x
2
2
2
2
2exp
2
1),|(
n
n
xxp
Gaussian Distribution
n
nxN
1̂ n
nxN22 ˆ
1ˆ
![Page 11: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/11.jpg)
Example: MLE
>> mu_fg = mean(im(labels))mu_fg = 0.6012
>> sigma_fg = sqrt(mean((im(labels)-mu_fg).^2))sigma_fg = 0.1007
>> mu_bg = mean(im(~labels))mu_bg = 0.4007
>> sigma_bg = sqrt(mean((im(~labels)-mu_bg).^2))sigma_bg = 0.1007
>> pfg = mean(labels(:));
labelsim
fg: mu=0.6, sigma=0.1bg: mu=0.4, sigma=0.1
Parameters used to Generate
![Page 12: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/12.jpg)
Probabilistic Inference 2. Once we have modeled the fg/bg appearance, how
do we compute the likelihood that a pixel is foreground?
Foreground
Background
![Page 13: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/13.jpg)
Probabilistic Inference
Compute the likelihood that a particular model generated a sample
component or label
),|( nn xmzp
![Page 14: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/14.jpg)
Probabilistic Inference
Compute the likelihood that a particular model generated a sample
component or label
|
|,),|(
n
mnnnn xp
xmzpxmzp
![Page 15: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/15.jpg)
Probabilistic Inference
Compute the likelihood that a particular model generated a sample
component or label
|
|,),|(
n
mnnnn xp
xmzpxmzp
kknn
mnn
xkzp
xmzp
|,
|,
![Page 16: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/16.jpg)
Probabilistic Inference
Compute the likelihood that a particular model generated a sample
component or label
|
|,),|(
n
mnnnn xp
xmzpxmzp
kknknn
mnmnn
kzpkzxp
mzpmzxp
|,|
|,|
kknn
mnn
xkzp
xmzp
|,
|,
![Page 17: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/17.jpg)
Example: Inference
>> pfg = 0.5;>> px_fg = normpdf(im, mu_fg, sigma_fg);>> px_bg = normpdf(im, mu_bg, sigma_bg);>> pfg_x = px_fg*pfg ./ (px_fg*pfg + px_bg*(1-pfg));
imfg: mu=0.6, sigma=0.1bg: mu=0.4, sigma=0.1
Learned Parameters
p(fg | im)
![Page 18: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/18.jpg)
Dealing with Hidden Variables
3. How can we get both labels and appearance parameters at once?
Foreground
Background
![Page 19: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/19.jpg)
Mixture of Gaussians
m
m
mn
m
x
2
2
2 2exp
2
1
mmmnnnn mzxpmzxp ,,|,,,|, 22 πσμ
mnmmn mzpxp |,| 2
mixture component
m
mmmnnn mzxpxp ,,|,,,| 22 πσμ
component priorcomponent model parameters
![Page 20: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/20.jpg)
Mixture of Gaussians
With enough components, can represent any probability density function– Widely used as general purpose pdf estimator
![Page 21: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/21.jpg)
Segmentation with Mixture of Gaussians
Pixels come from one of several Gaussian components– We don’t know which pixels come from which
components– We don’t know the parameters for the
components
![Page 22: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/22.jpg)
Simple solution
1. Initialize parameters
2. Compute the probability of each hidden variable given the current parameters
3. Compute new parameters for each model, weighted by likelihood of hidden variables
4. Repeat 2-3 until convergence
![Page 23: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/23.jpg)
Mixture of Gaussians: Simple Solution
1. Initialize parameters
2. Compute likelihood of hidden variables for current parameters
3. Estimate new parameters for each model, weighted by likelihood
),,,|( )()(2)( tttnnnm xmzp πσμ
nnnm
nnm
tm x
1ˆ )1(
nmnnm
nnm
t
m x 2)1(2 ˆ1
ˆ
N
nnm
tm
)1(ˆ
![Page 24: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/24.jpg)
Expectation Maximization (EM) Algorithm
z
zx
|,logargmaxˆ pGoal:
XfXf EE Jensen’s Inequality
Log of sums is intractable
See here for proof: www.stanford.edu/class/cs229/notes/cs229-notes8.ps
for concave functions f(x)
![Page 25: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/25.jpg)
Expectation Maximization (EM) Algorithm
1. E-step: compute
2. M-step: solve
)(,|
,||,log|,logE )(t
xzpppt
xzzxzx
z
)()1( ,||,logargmax tt pp
xzzxz
z
zx
|,logargmaxˆ pGoal:
![Page 26: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/26.jpg)
Expectation Maximization (EM) Algorithm
1. E-step: compute
2. M-step: solve
)(,|
,||,log|,logE )(t
xzpppt
xzzxzx
z
)()1( ,||,logargmax tt pp
xzzxz
z
zx
|,logargmaxˆ pGoal: XfXf EE
log of expectation of P(x|z)
expectation of log of P(x|z)
![Page 27: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/27.jpg)
EM for Mixture of Gaussians (by hand)
mm
m
mn
m
x
2
2
2exp
2
1 m
mmmnnn mzxpxp ,,|,,,| 22 πσμ
1. E-step:
2. M-step:
)(,|
,||,log|,logE )(t
xzpppt
xzzxzx
z
)()1( ,||,logargmax tt pp
xzzxz
![Page 28: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/28.jpg)
EM for Mixture of Gaussians (by hand)
mm
m
mn
m
x
2
2
2exp
2
1 m
mmmnnn mzxpxp ,,|,,,| 22 πσμ
1. E-step:
2. M-step:
)(,|
,||,log|,logE )(t
xzpppt
xzzxzx
z
)()1( ,||,logargmax tt pp
xzzxz
),,,|( )()(2)( tttnnnm xmzp πσμ
nnnm
nnm
tm x
1ˆ )1(
nmnnm
nnm
t
m x 2)1(2 ˆ1
ˆ
N
nnm
tm
)1(ˆ
![Page 29: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/29.jpg)
EM Algorithm
• Maximizes a lower bound on the data likelihood at each iteration
• Each step increases the data likelihood– Converges to local maximum
• Common tricks to derivation– Find terms that sum or integrate to 1– Lagrange multiplier to deal with constraints
![Page 30: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/30.jpg)
EM Demos
• Mixture of Gaussian demo
• Simple segmentation demo
![Page 31: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/31.jpg)
“Hard EM”• Same as EM except compute z* as most likely
values for hidden variables
• K-means is an example
• Advantages– Simpler: can be applied when cannot derive EM– Sometimes works better if you want to make hard
predictions at the end• But
– Generally, pdf parameters are not as accurate as EM
![Page 32: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/32.jpg)
Missing Data Problems: OutliersYou want to train an algorithm to predict whether a photograph is attractive. You collect annotations from Mechanical Turk. Some annotators try to give accurate ratings, but others answer randomly.
Challenge: Determine which people to trust and the average rating by accurate annotators.
Photo: Jam343 (Flickr)
Annotator Ratings
108928
![Page 33: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/33.jpg)
HW 4, problem 2
The “good/bad” label of each annotator is the missing data
The true scores for each image have a Gaussian distribution
The false scores come from a uniform distribution
Annotators are always “bad” or always “good”
![Page 34: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/34.jpg)
Missing Data Problems: Object Discovery
You have a collection of images and have extracted regions from them. Each is represented by a histogram of “visual words”.
Challenge: Discover frequently occurring object categories, without pre-trained appearance models.
http://www.robots.ox.ac.uk/~vgg/publications/papers/russell06.pdf
![Page 35: Hidden Variables, the EM Algorithm, and Mixtures of Gaussians](https://reader035.vdocuments.mx/reader035/viewer/2022070405/56813c92550346895da64056/html5/thumbnails/35.jpg)
Next class• MRFs and Graph-cut Segmentation