1 applications of belief propagation in low-level vision bill freeman massachusetts institute of...

50
1 Applications of belief propagation in low- level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor, Jonathan Yedidia, Yair Weiss, Thouis Jones, Edward Adelson, Marshall Tappen.

Upload: aidan-howe

Post on 27-Mar-2015

216 views

Category:

Documents


1 download

TRANSCRIPT

Page 1: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

1

Applications of belief propagation in low-level vision

Bill Freeman

Massachusetts Institute of Technology

Jan. 12, 2010

Joint work with: Egon Pasztor, Jonathan Yedidia, Yair Weiss, Thouis Jones, Edward Adelson, Marshall Tappen.

Page 2: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

2

x1MMSE meanx1

sumx2

sumx3

P(x1,x2,x3,y1,y2,y3)

y1

Derivation of belief propagation

),( 11 yx

),( 21 xx

),( 22 yx

),( 32 xx

),( 33 yx

x1

y2

x2

y3

x3

Page 3: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

3

),(),(sum

),(),(sum

),(mean

),(),(

),(),(

),(sumsummean

),,,,,(sumsummean

3233

2122

111

3233

2122

111

3213211

3

2

1

321

321

xxyx

xxyx

yxx

xxyx

xxyx

yxx

yyyxxxPx

x

x

xMMSE

xxxMMSE

xxxMMSE

The posterior factorizes

y1

),( 11 yx

),( 21 xx

),( 22 yx

),( 32 xx

),( 33 yx

x1

y2

x2

y3

x3

Page 4: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

4

Propagation rules

y1

),( 11 yx

),( 21 xx

),( 22 yx

),( 32 xx

),( 33 yx

x1

y2

x2

y3

x3),(),(sum

),(),(sum

),(mean

),(),(

),(),(

),(sumsummean

),,,,,(sumsummean

3233

2122

111

3233

2122

111

3213211

3

2

1

321

321

xxyx

xxyx

yxx

xxyx

xxyx

yxx

yyyxxxPx

x

x

xMMSE

xxxMMSE

xxxMMSE

Page 5: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

5

Propagation rules

y1

),( 11 yx

),( 21 xx

),( 22 yx

),( 32 xx

),( 33 yx

x1

y2

x2

y3

x3

),(),(sum

),(),(sum

),(mean

3233

2122

111

3

2

1

xxyx

xxyx

yxx

x

x

xMMSE

)( ),( ),(sum)( 23222211

21

2

xMyxxxxMx

Page 6: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

6

Propagation rules

y1

),( 11 yx

),( 21 xx

),( 22 yx

),( 32 xx

),( 33 yx

x1

y2

x2

y3

x3

),(),(sum

),(),(sum

),(mean

3233

2122

111

3

2

1

xxyx

xxyx

yxx

x

x

xMMSE

)( ),( ),(sum)( 23222211

21

2

xMyxxxxMx

Page 7: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

7

Belief propagation messages

jii =

ijNk

jkjji

xi

ji xMxxxM

j \)(ij )(),( )(

j

To send a message: Multiply together all the incoming messages, except from the node you’re sending to,then multiply by the compatibility matrix and marginalize over the sender’s states.

A message: can be thought of as a set of weights on each of your possible states

Page 8: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

8

Belief propagation: the nosey neighbor rule

“Given everything that I’ve heard, here’s what I think is going on inside your house”

(Given my incoming messages, affecting my state probabilities, and knowing how my states affect your states, here’s how I think you should modify the probabilities of your states)

Page 9: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

9

Beliefs

j

)(

)( )(jNk

jkjjj xMxb

To find a node’s beliefs: Multiply together all the messages coming in to that node.

(Show this for the toy example.)

Page 10: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

10

Optimal solution in a chain or tree:Belief Propagation

• “Do the right thing” Bayesian algorithm.

• For Gaussian random variables over time: Kalman filter.

• For hidden Markov models: forward/backward algorithm (and MAP variant is Viterbi).

Page 11: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

11

Markov Random Fields

• Allows rich probabilistic models for images.• But built in a local, modular way. Learn local

relationships, get global effects out.

Page 12: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

12

MRF nodes as pixels

Winkler, 1995, p. 32

Page 13: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

13

MRF nodes as patches

image patches

(xi, yi)

(xi, xj)

image

scene

scene patches

Page 14: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

14

Network joint probability

scene

image

Scene-scenecompatibility

functionneighboringscene nodes

local observations

Image-scenecompatibility

function

i

iiji

ji yxxxZ

yxP ),(),(1

),(,

Page 15: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

15

In order to use MRFs:

• Given observations y, and the parameters of the MRF, how infer the hidden variables, x?

• How learn the parameters of the MRF?

Page 16: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

16

Inference in Markov Random Fields

Gibbs sampling, simulated annealingIterated conditional modes (ICM)Belief propagation

Application examples:super-resolutionmotion analysisshading/reflectance separation

Graph cutsVariational methods

Page 17: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

17

Inference in Markov Random Fields

Gibbs sampling, simulated annealingIterated conditional modes (ICM)Belief propagation

Application examples:super-resolutionmotion analysisshading/reflectance separation

Graph cutsVariational methods

Page 18: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

18

),,,,,(sumsummean 3213211321

yyyxxxPxxxx

MMSE

y1

Derivation of belief propagation

),( 11 yx

),( 21 xx

),( 22 yx

),( 32 xx

),( 33 yx

x1

y2

x2

y3

x3

Page 19: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

19

No factorization with loops!

y1

x1

y2

x2

y3

x3

),(),(sum

),(),(sum

),(mean

3233

2122

111

3

2

1

xxyx

xxyx

yxx

x

x

xMMSE

31 ),( xx

Page 20: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

20

Applications of belief propagation in low-level vision

Bill Freeman

Massachusetts Institute of Technology

Jan. 12, 2010

Joint work with: Egon Pasztor, Jonathan Yedidia, Yair Weiss, Thouis Jones, Edward Adelson, Marshall Tappen.

Page 21: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

21

Belief, and message updates

jii =

ijNk

jkjji

xi

ji xMxxxM

j \)(ij )(),( )(

j

)(

)( )(jNk

jkjjj xMxb

Page 22: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

22

Optimal solution in a chain or tree:Belief Propagation

• “Do the right thing” Bayesian algorithm.

• For Gaussian random variables over time: Kalman filter.

• For hidden Markov models: forward/backward algorithm (and MAP variant is Viterbi).

Page 23: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

23

Justification for running belief propagation in networks with loops

• Experimental results:

– Error-correcting codes

– Vision applications

• Theoretical results:

– For Gaussian processes, means are correct.

– Large neighborhood local maximum for MAP.

– Equivalent to Bethe approx. in statistical physics.

– Tree-weighted reparameterization

Weiss and Freeman, 2000

Yedidia, Freeman, and Weiss, 2000

Freeman and Pasztor, 1999;Frey, 2000

Kschischang and Frey, 1998;McEliece et al., 1998

Weiss and Freeman, 1999

Wainwright, Willsky, Jaakkola, 2001

Page 24: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

24

Results from Bethe free energy analysis

• Fixed point of belief propagation equations iff. Bethe approximation stationary point.

• Belief propagation always has a fixed point.• Connection with variational methods for inference: both

minimize approximations to Free Energy,– variational: usually use primal variables.

– belief propagation: fixed pt. equs. for dual variables.

• Kikuchi approximations lead to more accurate belief propagation algorithms.

• Other Bethe free energy minimization algorithms—Yuille, Welling, etc.

Page 25: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

25

References on BP and GBP

• J. Pearl, 1985– classic

• Y. Weiss, NIPS 1998– Inspires application of BP to vision

• W. Freeman et al learning low-level vision, IJCV 1999– Applications in super-resolution, motion, shading/paint

discrimination• H. Shum et al, ECCV 2002

– Application to stereo• M. Wainwright, T. Jaakkola, A. Willsky

– Reparameterization version• J. Yedidia, AAAI 2000

– The clearest place to read about BP and GBP.

Page 26: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

26

Inference in Markov Random Fields

Gibbs sampling, simulated annealingIterated conditional modes (ICM)Belief propagation

Application examples:super-resolutionmotion analysisshading/reflectance separation

Graph cutsVariational methods

Page 27: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

27

Super-resolution

• Image: low resolution image

• Scene: high resolution image

imag

esc

ene

ultimate goal...

Page 28: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

28

Polygon-based graphics images are resolution independent

Pixel-based images are not resolution

independent

Pixel replication

Cubic splineCubic spline, sharpened

Training-based super-resolution

Page 29: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

29

3 approaches to perceptual sharpening

(1) Sharpening; boost existing high frequencies.

(2) Use multiple frames to obtain higher sampling rate in a still frame.

(3) Estimate high frequencies not present in image, although implicitly defined.

In this talk, we focus on (3), which we’ll call “super-resolution”.

spatial frequency

ampl

itud

e

spatial frequencyam

plit

ude

Page 30: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

30

Super-resolution: other approaches

• Schultz and Stevenson, 1994

• Pentland and Horowitz, 1993

• fractal image compression (Polvere, 1998; Iterated Systems)

• astronomical image processing (eg. Gull and Daniell, 1978; “pixons” http://casswww.ucsd.edu/puetter.html)

• Follow-on: Jianchao Yang, John Wright, Thomas S. Huang, Yi Ma: Image super-resolution as sparse representation of raw image patches. CVPR 2008

Page 31: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

31

Training images, ~100,000 image/scene patch pairs

Images from two Corel database categories: “giraffes” and “urban skyline”.

Page 32: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

32

Do a first interpolation

Zoomed low-resolution

Low-resolution

Page 33: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

33

Zoomed low-resolution

Low-resolution

Full frequency original

Page 34: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

34

Full freq. originalRepresentationZoomed low-freq.

Page 35: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

35

True high freqsLow-band input

(contrast normalized, PCA fitted)

Full freq. originalRepresentationZoomed low-freq.

(to minimize the complexity of the relationships we have to learn,we remove the lowest frequencies from the input image,

and normalize the local contrast level).

Page 36: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

36

Training data samples (magnified)

......

Gather ~100,000 patches

low freqs.

high freqs.

Page 37: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

37

True high freqs.Input low freqs.

Training data samples (magnified)

......

Nearest neighbor estimate

low freqs.

high freqs.

Estimated high freqs.

Page 38: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

38

Input low freqs.

Training data samples (magnified)

......

Nearest neighbor estimate

low freqs.

high freqs.

Estimated high freqs.

Page 39: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

39

Example: input image patch, and closest matches from database

Input patch

Closest imagepatches from database

Correspondinghigh-resolution

patches from database

Page 40: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

40

Page 41: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

41

Scene-scene compatibility function, (xi, xj)

Assume overlapped regions, d, of hi-res. patches differ by Gaussian observation noise:

d

Uniqueness constraint,not smoothness.

Page 42: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

42

Image-scene compatibility function, (xi, yi)

Assume Gaussian noise takes you from observed image patch to synthetic sample:

y

x

Page 43: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

43

Markov network

image patches

(xi, yi)

(xi, xj)

scene patches

Page 44: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

44

Iter. 3

Iter. 1

Belief PropagationInput

Iter. 0

After a few iterations of belief propagation, the algorithm selects spatially consistent high resolution

interpretations for each low-resolution patch of the input image.

Page 45: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

45

Zooming 2 octaves

85 x 51 input

Cubic spline zoom to 340x204 Max. likelihood zoom to 340x204

We apply the super-resolution algorithm recursively, zooming

up 2 powers of 2, or a factor of 4 in each dimension.

Page 46: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

46

True200x232

Original50x58

(cubic spline implies thin plate prior)

Now we examine the effect of the prior assumptions made about images on the

high resolution reconstruction.First, cubic spline interpolation.

Page 47: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

47

Cubic splineTrue

200x232

Original50x58

(cubic spline implies thin plate prior)

Page 48: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

48

True

Original50x58

Training images

Next, train the Markov network algorithm on a world of random noise

images.

Page 49: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

49

Markovnetwork

True

Original50x58

The algorithm learns that, in such a world, we add random noise when zoom

to a higher resolution.

Training images

Page 50: 1 Applications of belief propagation in low-level vision Bill Freeman Massachusetts Institute of Technology Jan. 12, 2010 Joint work with: Egon Pasztor,

50

True

Original50x58

Training images

Next, train on a world of vertically oriented rectangles.