スパース正則化項をともなうマルコフ確率場における確率伝...

18 November, 2010 NC201011 (Sendai） 1

スパース正則化項をともなうマルコフ確率場における確率伝搬法

東北大学大学院情報科学研究科

田中和之

http://www.smapip.is.tohoku.ac.jp/~kazu/

CollaboratorsD. M. Titterington (University of Glasgow, UK)


MRF and Belief Propagation

Matthias W. Seeger (2008): J. Machine Learning ResearchBayesian Inference and Optimal Design for the Sparse Linear Model

Variational Bayes and Expectation Propagation

Is it possible to extend the belief propagation to the MRF with Sparsity?

Tanaka and Morita (1995): Physics Letter ABelief Propagation for MRF in Image Processing

Cluster Variation Method =Generalized Belief Propagation

Geman and Geman (1986): IEEE Transactions on PAMIImage Processing by Markov Random Fields (MRF)


What is Sparsity?

=

N

MNMM

N

N

M

f

ff

BBB

BBBBBB

h

hh

2

1

21

22221

12211

2

1

Mg

gg

2

1

),0( 2σN

Bayesian Inference

Nf

ff

2

1

Number of Parameters N > Number of Equations M

−−−∝== pfBfggGfF γ

σ2

221exp}|Pr{

Optimization Problemp

E fBfggf γσ

−−−=222

1)|(

Compressed Sensing

The case of p=0 or p=1is good for compressed sensing.!!


Probabilistic model in the present talk

−−−−−∝== ∑∑∑

∈∈∈ VEV ii

jiji

iii fffgf ||)(2

1)(2

1exp}|Pr{},{

222 γασ

gGfF

Bayesian Inference for Image Restoration

||

2

1

Vg

gg

),0( 2σN

||

2

1

Vf

ff

SmoothingData Dominant Sparsity

−−−∝==

pfBfggGfF γ

σ 2221exp}|Pr{

Bayesian Inference for Compressed Sensing

}Pr{ fF =

18 November, 2010 5 NC201011 (Sendai）

Image Restoration by Gaussian MRF Model (γ=0)

Original Image

MSE:305

MSE: 675 MSE: 447MSE: 260

MSE: 1401

Degraded Image

Lowpass Filter Median Filter

Exact

Wiener Filter

( )2ˆ||

1MSE ∑∈

−=VV i

ii ff

Belief Propagation

MSE:328

プレゼンタープレゼンテーションのノートFinally, we show only the results for the gray-level image restoration. For each numerical experiments, the loopy belief propagation ca give us better results than the ones by conventional filters.

Conventional Belief Propagation


( ) ( ) ( )( ) ( )

= ∏∏

∈∈ EV },{

,1ji jjii

jiij

iii fWfW

ffWfW

ZP f

−−−≡ ||)(

21exp)( 22 iiiii fgffW γσ

−−−−−=

===

∑∑∑∈∈∈ VEV i

iji

jii

ii fffgfZ

P

||)(21)(

21exp1

)(}|Pr{

},{

222 γασ

fgGfF

−−−−−−−−≡ ||||)(

21)(

21)(

21exp),( 222

22 jijijjiijiij ffffgfgfffW γγασσ

Conventional Belief Propagation


[ ] ( )( ) 0ln)( ≥

≡ ∫ ff

ff dPQQPQD

KL Divergence

∫ ∫ ∫+∞

∞−

∞

∞

+∞

∞− +−≡ ||1121||21 ),,,()( ViiVii dfdfdfdfdffffQfQ

( ) ( ) ( )( ) ( )

≅ ∏∏

∈∈ EV },{

,

ji jjii

jiij

iii fQfQ

ffQfQQ f

( ) ( ) ( )( ) ( )

= ∏∏

∈∈ EV },{

,1ji jjii

jiij

iii fWfW

ffWfW

ZP f

∫ ∫ ∫+∞

∞−

∞

∞

+∞

∞− +−+−≡ ||111121||21 ),,,(),( VjjiiVjiij dfdfdfdfdfdfdffffQffQ

)},{( ),( and )( )(for Equtions Integral usSimultaneo EV ∈∀∈∀ jiffQifQ jiijii

Extended Belief Propagation


( ) ( ) ( )( ) ( )

= ∏∏

∈∈ EV },{~~,~1

ji jjii

jiij

iii fWfW

ffWfW

ZP f

−−−−−−≡ 222

22 )(2

1)(2

1)(2

1exp),(~ jijjiijiij ffgfgfffW ασσ

−−−≡ ||)(

21exp)( 22 iiiii fgffW ασ

−−−−−=

===

∑∑∑∈∈∈ VEV i

iji

jii

ii fffgfZ

P

||)(21)(

21exp1

)(}|Pr{

},{

222 γασ

fgGfF

−−≡ 22 )(2

1exp)(~ iiii gffW σ



[ ] ( )( ) 0ln)( ≥

≡ ∫ ff

ff dPQQPQD

KL Divergence

∫ ∫ ∫+∞

∞−

∞

∞

+∞

∞− +−≡ ||1121||21 ),,,()( VV dfdfdfdfdffffQfQ iiii

( ) ( ) ( )( ) ( )

≅ ∏∏

∈∈ EV },{~~,~

ji jjii

jiij

iii fQfQ

ffQfQQ f

( ) ( ) ( )( ) ( )

= ∏∏

∈∈ EV },{~~,~1

ji jjii

jiij

iii fWfW

ffWfW

ZP f

−−∝ 2)(

21exp)(~ ii

iiii mfV

fQ

−−

−−−∝

−

jj

ii

jjji

ijiiiiiijiij mf

mfVVVV

mfmf,ffQ1

),(21exp)(~

Average Varianceff dQfm ii )(∫≡ ff dQmfV iiii )()( 2∫ −≡

Covarianceff dQmfmfV jjiiij )())((∫ −−≡


( )( ) ( ) ( ) ( ) ( )( ) ( ) ( ) ( ) ( )∫ ∫

∫∞+

∞−

∞+

∞− →→→→

+∞

∞− →→→→

→ =211111151141132112

11111151141132112221

,

,

dfdffMfMfMfMffW

dffMfMfMfMffWfM

1

3

4 2

5

13→M

14→M

15→M

21→M

14 2 7

∫+∞

∞− 1df

2

8

1 7

6

=

Message Passing Rule of Belief Propagation

18 November, 2010 10NC201011 (Sendai）

1 2

3 8

5 6

2

111→M

−−

−−−∝

−

jj

ii

jjji

ijiiiiiijii mf

mfVVVV

mfmfffQ1

),(21exp),(~

−−∝ →→→

2)(21exp)( ijiijiij ffM µλ

−−∝ 2)(

21exp)(~ ii

iiii mfV

fQ

プレゼンタープレゼンテーションのノートThe reducibility conditions can be rewritten as the following fixed point equations. This fixed point equations is corresponding to the extremum condition of the Bethe free energy. And the fixed point equations can be numerically solved by using the natural iteration. The algorithm is corresponding to the loopy belief propagation.


( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( ) ( )

Φ←∫

∫∞+

∞− →→→→−

∞+

∞− →→→→−

→

111511411311211||

111511411311211||

21

1

111 ~

~

)(0

0

dffMfMfMfMfWe

dffMfMfMfMfWeff

fMi

i

f

f

α

α

1

3

4 2

5

13→M

14→M

15→M

21→M ∫∞+

∞−

12

1

1 dfff

1

3

4 2

5

=



1

111→M

−−∝ →→→


1

∫∞+

∞−

12

1

1 dfff

)( 11 fQMessage Passing Rules include computations of error functions

−−∝ 211 )(2

1exp)(~ iiii

mfV

fQ

1

3

4 7

6



1

3

4 2

5

13→M

14→M

15→M

21→M

14 2 7



1 2

3 8

5 6

111→M

)(~ ii fQ

−−∝ →→→


1

3

4 2

51

1

3

4 2

51

)( ii fQ),(~

jii ffQ

1

3

4 2

5

13→M

14→M

15→M

21→M

111→M

Approximate Marginal Probability in Belief Propagation

Computation Time O(|V|)


Interpretation of Expectation Propagation for Sparse MRF

Matthias Seeger: Bayesian Inference and Optimal Design for the Sparse Linear Model, Journal of Machine Learning Research vol.9, pp.759-813, 2008.


Thomas Minka: Expectation Propagation for Approximate Bayesian Inference, Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, pp.362–369, 2001.



( ) ( )( )

= ∏

∈Vi ii

ii

fWfWW

ZP ~)(

~1 ff

−−−−≡ ∑∑

∈∈ EV },{

222 )(2

1)(2

1exp)(~ji

jii

ii ffgffW ασ

−−−≡ ||)(

21exp)( 22 iiiii fgffW γσ

−−−−−=

===

∑∑∑∈∈∈ VEV i

iji

jii

ii fffgfZ

P

||)(21)(

21exp1

)(}|Pr{

},{

222 γασ

fgGfF

−−≡ 22 )(2

1exp)(~ iiii gffW σ



[ ] ( )( ) 0ln)( ≥

≡ ∫ ff

ff dPQQPQD

KL Divergence

∫ ∫ ∫+∞

∞−

∞

∞

+∞

∞− +−≡ ||1121||21 ),,,()( VV dfdfdfdfdffffQfQ iiii

( ) ( ) ( )( )

≅ ∏

∈Vi ii

ii

fQfQQQ ~

~ ff

−−∝ 2)(

21exp)(~ ii

iiii mfV

fQ

−−−∝ − )()(

21exp)(~ 1T mfVmffQ

Average Varianceff dQfm ii )(∫≡ ff dQmfV iiii )()( 2∫ −≡

Covarianceff dQmfmfV jjiiij )())((∫ −−≡

( ) ( )( )

= ∏

∈Vi ii

ii

fWfWW

ZP ~)(

~1 ff

This scheme corresponds to Adaptive TAP (Manfred Opper) for Sparse MRF.

−−−∝ − )()(

21exp)(~ 1 mfVmffQ



We have to compute inverse matrices of |V|x|V| matrices

Computation Time O(|V|3)

Message Passing Rule of Expectation Propagation

|V|-Dimensional Gaussian Distribution

)( ii fQ

)( ii fQ


SummaryFormulation of Extended Belief Propagation (EBP) for Sparse Markov Random Fields.Interpretation of Expectation Propagation (EP) by means of Extended Belief Propagation.

Future ProblemsHyperparameter Estimation by means of EM algorithm with Extended Belief Propagation.Comparison of EBP with EP.Perturbative Analysis from GMRF to Sparse MRF based on Plefka Expansion. Application of EBP to Compressed Sensing.

スパース正則化項をともなう�マルコフ確率場における確率伝搬法 MRF and Belief PropagationWhat is Sparsity?Probabilistic model in the present talkImage Restoration by Gaussian MRF Model (g=0)Conventional Belief PropagationConventional Belief PropagationExtended Belief PropagationExtended Belief PropagationExtended Belief PropagationExtended Belief PropagationExtended Belief PropagationInterpretation of Expectation Propagation for Sparse MRFInterpretation of Expectation Propagation for Sparse MRFInterpretation of Expectation Propagation for Sparse MRFInterpretation �of Expectation Propagation for Sparse MRFSummary

スパース正則化項をともなう マルコフ確率場における確率伝...

Documents

スパース正則化項をともなうマルコフ確率場における確率伝...