introduction to probabilistic image processing and ... › ~kazu › tutorial... · bayes formula...
TRANSCRIPT
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 11
Introduction to Introduction to Probabilistic Image Processing and Probabilistic Image Processing and
Bayesian NetworksBayesian NetworksKazuyuki TanakaKazuyuki Tanaka
Graduate School of Information Sciences,Graduate School of Information Sciences,Tohoku University, Sendai, JapanTohoku University, Sendai, Japan
http://www.smapip.is.tohoku.ac.jp/~kazu/http://www.smapip.is.tohoku.ac.jp/~kazu/
CollaboratorsProf. Mike Titterington (University of Glasgow, UK)
Dr. Koji Tsuda (MPI for Biological Cybernetics, Germany)Dr. Muneki Yasuda (Tohoku University, Japan)
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 22
ContentsContents
1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model4.4. Belief PropagationBelief Propagation5.5. Other ApplicationOther Application6.6. Concluding RemarksConcluding Remarks
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 33
ContentsContents
1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model4.4. Belief PropagationBelief Propagation5.5. Other ApplicationOther Application6.6. Concluding RemarksConcluding Remarks
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 44
More is differentMore is different
Probabilistic information processing can give Probabilistic information processing can give us unexpected capacity in a system constructed us unexpected capacity in a system constructed from many cooperating elements with from many cooperating elements with randomness. randomness. The circumstances are similar to everything in The circumstances are similar to everything in the natural world being made up of a lot of the natural world being made up of a lot of molecules with interactions and fluctuations. molecules with interactions and fluctuations. The collection of molecules can give rise to The collection of molecules can give rise to unpredictable phenomena. unpredictable phenomena. This is often called This is often called ““More is differentMore is different”” in in physics. physics.
29 December, 2008 National Tsing Hua University, Taiwan 5
Main InterestsMain InterestsInformation Processing: Information Processing:
DataDataPhysics: Physics:
Material, Material, Natural PhenomenaNatural Phenomena
System of a lot of elements with mutual relationSystem of a lot of elements with mutual relationCommon Concept between Computer Sciences and PhysicsCommon Concept between Computer Sciences and Physics
MaterialMolecule
Materials are constructed from a lot of molecules.
Molecules have interactions of each other.
0,1 101101110001
010011101110101000111110000110000101000000111010101110101010Bit
Data
Data is constructed from many bits
A sequence is formed by deciding the arrangement of bits.
A lot of elements have mutual relation of each otherA lot of elements have mutual relation of each otherSome physical concepts Some physical concepts in Physical models are in Physical models are useful for the design of useful for the design of computational models in computational models in probabilistic probabilistic information processing.information processing.
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 66
More is different in informatics as well.More is different in informatics as well.Our goal is to establish theoretical paradigms Our goal is to establish theoretical paradigms for probabilistic information processing by for probabilistic information processing by means of statistical science and statistical means of statistical science and statistical physics. physics. The probabilistic information processing is The probabilistic information processing is based on both modeling of problems and based on both modeling of problems and design of algorithms, which is often realized design of algorithms, which is often realized as graphical models including Bayesian as graphical models including Bayesian network.network.
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 77
Purpose of My TalkPurpose of My Talk
Review of formulation of probabilistic model for Review of formulation of probabilistic model for image processing by means of conventional image processing by means of conventional statistical schemes.statistical schemes.Review of probabilistic image processing by using Review of probabilistic image processing by using Gaussian graphical model (Gauss Markov Gaussian graphical model (Gauss Markov Random Fields) as the most basic example.Random Fields) as the most basic example.Review of how to construct a belief propagation Review of how to construct a belief propagation algorithm for image processing.algorithm for image processing.
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 88
ContentsContents
1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical Model Gaussian Graphical Model 4.4. Belief PropagationBelief Propagation5.5. Other ApplicationOther Application6.6. Concluding RemarksConcluding Remarks
29 December, 2008 National Tsing Hua University, Taiwan 9
Bayes Formula and Bayesian Network
Posterior Probability
}Pr{}Pr{}|Pr{}|Pr{
BAABBA =
Bayes Rule
Prior Probability
Event B is given as the observed data.Event A corresponds to the original information to estimate. Thus the Bayes formula can be applied to the estimation of the original information from the given data.
A
BBayesian Network
Data-Generating Process
29 December, 2008 National Tsing Hua University, Taiwan 10
Image Restoration by Probabilistic Model
Original Image
Degraded Image
Transmission
Noise
444 3444 21
444 8444 764444444 84444444 76
4444444 84444444 76
Likelihood Marginal
PriorProcess nDegradatio
Posterior
}Image DegradedPr{}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
=
Assumption 1: The degraded image is randomly generated from the original image by according to the degradation process. Assumption 2: The original image is randomly generated by according to the prior probability.
Bayes Formula
29 December, 2008 National Tsing Hua University, Taiwan 11
Image Restoration by Probabilistic Model
Degraded
Image
i
fi: Light Intensity of Pixel iin Original Image
),( iii yxr =r
Position Vector of Pixel i
gi: Light Intensity of Pixel iin Degraded Image
i
Original
Image
The original images and degraded images are represented by f = (f1,f2,…,f|V|)T and g = (g1,g2,…,g|V|)T , respectively.
29 December, 2008 National Tsing Hua University, Taiwan 12
Probabilistic Modeling of Image Restoration
444 8444 764444444 84444444 76
4444444 84444444 76
PriorLikelihood
Posterior
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
∝
∏∈Ψ=
ViiiVV fgfffgggP )|(),,,|,,,( ||21||21 LL
Random Fieldsfi
gi
fi
gi
or
Assumption 1: A given degraded image is obtained from the original image by changing the state of each pixel to another state by the same probability, independently of the other pixels.
29 December, 2008 National Tsing Hua University, Taiwan 13
Probabilistic Modeling of Image Restoration
444 8444 764444444 84444444 76
4444444 84444444 76
PriorLikelihood
Posterior
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
∝
∏∈Φ=Eji
jiV fffffP},{
||21 ),(),,,( L
Random Fields
Assumption 2: The original image is generated according to a prior probability. Prior Probability consists of a product of functions defined on the neighbouring pixels.
i j
Product over All the Nearest Neighbour Pairs of Pixels
29 December, 2008 National Tsing Hua University, Taiwan 14
Prior Probability for Binary Image
== >p p p−
21 p−
21i j Probability of
NeigbouringPixel
∏∈Φ=Eji
ji fffP},{
),()( i j
It is important how we should assume the function Φ(fi,fj) in the prior probability.
)0,1()1,0()0,0()1,1( Φ=Φ>Φ=Φ
We assume that every nearest-neighbour pair of pixels take the same state of each other in the prior probability.
1,0=if
29 December, 2008 National Tsing Hua University, Taiwan 15
Prior Probability for Binary Image
Prior probability prefers to the configuration with the least number of red lines.
Which state should the center pixel be taken when the states of neighbouring pixels are fixed to the white states?
?
>
== >p pi j Probability of
Nearest NeigbourPair of Pixels
29 December, 2008 National Tsing Hua University, Taiwan 16
Prior Probability for Binary ImagePrior Probability for Binary Image
Which state should the center pixel be taken when the states of neighbouring pixels are fixed as this figure?
?-?== >
p p
> >=
Prior probability prefers to the configuration with the least number of red lines.
29 December, 2008 National Tsing Hua University, Taiwan 17
What happens for the case of large number of pixels?
p 0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0lnp
Disordered State Critical Point(Large fluctuation)
small p large p
Covariance between the nearest neghbourpairs of pixels
Sampling by Marko chain Monte Carlo
Ordered State
Patterns with both ordered statesand disordered states are often generated near the critical point.
29 December, 2008 National Tsing Hua University, Taiwan 18
Physical model of ferromagnetism and Probabilistic model of image processingPhysical model of ferromagnetism and Probabilistic model of image processing
p p
p p
>
=
=
x
Ising Model
Markov Random Field (MRF) Model
>
=
=
Up Spin State
Down Spin State
Black State
White State
Probabilistic models for image processing has the similar structure as physical models of ferromagnetism
Regular graph
x
y
29 December, 2008 National Tsing Hua University, Taiwan 19
Pattern near Critical Point of Prior Probability
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2 0.4 0.6 0.8 1.0ln p
similar
small p large p
Covariance between the nearest neghbourpairs of pixels
We regard that patterns generated near the critical point are similar to the local patterns in real world images.
29 December, 2008 National Tsing Hua University, Taiwan 20
Bayesian Image Analysis
fg
)( fP )|( fgP gOriginal Image Degraded Image
Prior Probability
Bayes Formulas=>Posterior Probability
Degradation Process
Image processing is reduced to calculations of averages, variances and co-variances in the posterior probability.
E:Set of all the nearest neighbourpairs of pixels
V:Set of All the pixels
)()()|()|(
gPfPfgPgfP =
∫ ∫ ∫+∞∞−
+∞∞−
+∞∞−
= ||21||21||21 ),,,|,,,(ˆVVVii dfdfdfgggfffPff LLLL
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 2121
ContentsContents
1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model4.4. Belief PropagationBelief Propagation5.5. Other ApplicationOther Application6.6. Concluding RemarksConcluding Remarks
29 December, 2008 National Tsing Hua University, Taiwan 22
Bayesian Image Analysis by Gaussian Graphical Model
0005.0=α 0030.0=α0001.0=α
Patterns are generated by MCMC.
Markov Chain Monte Carlo Method
Prior Probability ( )+∞∞−∈ ,if
E:Set of all the nearest-neghbourpairs of pixels
V:Set of all the pixels
( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛−−∝ ∑
∈EjijiV fffffP
},{
2||21 2
1exp),,,( αL
29 December, 2008 National Tsing Hua University, Taiwan 23
Bayesian Image Analysis by Gaussian Graphical Model
( )2,0~ σNfg ii −Histogram of Gaussian Random Numbers
n Noise GaussianfImage Original gImage Degraded
( )+∞∞−∈ ,, ii gf
Degraded image is obtained by adding a white Gaussian noise to the original image.
Degradation Process is assumed to be the additive white Gaussian noise.
V:Set of all the pixels
( )∏∈
⎟⎠⎞
⎜⎝⎛ −−=
ViiiNN gffffgggP 2
222121 21exp
2
1),,,|,,,(σπσ
LL
29 December, 2008 National Tsing Hua University, Taiwan 24
( ) gCIdfgffPf 12 )( ˆ −+== ∫ ασ
Bayesian Image Analysis by Gaussian Graphical Model
( )+∞∞−∈ ,, ii gf
( )( )
( )⎪⎩
⎪⎨
⎧∈−=
=otherwise0
14
Eijji
jCi
Multi-Dimensional Gaussian Integral Formula
Posterior Probability
Average of the posterior probability can be calculated by using the multi-dimensional Gauss integral Formula
|V|x|V| matrix
E:Set of all the nearest-neghbourpairs of pixels
V:Set of all the pixels
( ) ( ) ⎟⎟⎠
⎞⎜⎜⎝
⎛−−−−∝
=
∑∑},{
222 2
12
1exp
)()()|()|(
jiji
iii ffgf
gPfPfgPgfP
ασ
29 December, 2008 National Tsing Hua University, Taiwan 25
Bayesian Image Analysis by Gaussian Graphical Model
0
0.0002
0.0004
0.0006
0.0008
0.001
0 20 40 60 80 100( )tσ
( )tα
g f̂
gCttItf 12 ))()(()(ˆ −+= σα
Iteration Procedure of EM algorithm in Gaussian Graphical Model
EM
f̂
g( )
),|(max arg)ˆ,ˆ(,
σασασα
gP=
( ) ( )( ) ( ) ( )( )gttQtt ,,1,1 σασα ←++
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 2626
Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters
( )2ˆ||
1MSE ∑∈
−=Vi
ii ffV
315315Statistical MethodStatistical Method
445445(5x5)(5x5)486486(3x3)(3x3)Median Median
FilterFilter
413413(5x5)(5x5)388388(3x3)(3x3)LowpassLowpass
FilterFilter
MSEMSE
(3x3) (3x3) LowpassLowpass (5x5) Median(5x5) MedianMRFMRF
Original ImageOriginal Image Degraded ImageDegraded Image
RestoredRestoredImageImage
VV:Set:Set of all of all the pixelsthe pixels
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 2727
ContentsContents
1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model4.4. Belief PropagationBelief Propagation5.5. Other ApplicationOther Application6.6. Concluding RemarksConcluding Remarks
29 December, 2008 National Tsing Hua University, Taiwan 28
Graphical Representation for Tractable Models
Tractable Model
⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛⎟⎟⎠
⎞⎜⎜⎝
⎛= ∑∑∑
∑ ∑ ∑
===
= = =
1,01,01,0
1,0 1,0 1,0
),(),(),(
),(),(),(
CBA
A B C
DChDBgDAf
DChDBgDAf A
B CD
∑ ∑ ∑= = =1,0 1,0 0,1A B C
Intractable Model∑ ∑ ∑= = =1,0 1,0 1,0
),(),(),(A B C
AChCBgBAf
A
B C
Tree Graph
Cycle Graph
It is possible to calculate each summation independently.
It is hard to calculate each summation independently.
∑ ∑ ∑= = =1,0 1,0 0,1A B C
29 December, 2008 National Tsing Hua University, Taiwan 29
Loopy Belief Propagation for Graphical Model in Image Processing
Graphical model for image processing is represented in terms of the square lattice.Square lattice includes a lot of cycles.Belief propagation are applied to the calculation of statistical quantities as an approximate algorithm.
Every graph consisting of a pixel and its four neighbouring pixels can be regarded as a tree graph.
Loopy Belief PropagationLoopy Belief Propagation
1 2 4
5
31 2 4
5
3
3
1 2
5
4∫+∞∞−
← 2df21
29 December, 2008 National Tsing Hua University, Taiwan 30
Loopy Belief Propagation in Image Processing
We have four kinds of message passing rules for each pixel.
Each massage passing rule includes 3 incoming messages and 1 outgoing message
Visualizations of Passing Messages
29 December, 2008 National Tsing Hua University, Taiwan 31
EM algorithm by means of Belief Propagation
Input
Output
LoopyBP EM
Update Rule of Loopy Belief Propagation
EM Algorithm for Hyperparameter Estimation
( ) ( )( ) ( ) ( )( )gttQtt ,,1,1 σασα ←++
3
1 2
5
4∫+∞∞−
← 2df21
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 3232
Probabilistic Image Processing by Probabilistic Image Processing by EMEM Algorithm and Loopy BP for Algorithm and Loopy BP for
Gaussian Graphical ModelGaussian Graphical Model( ) ( )( ) ( ) ( )( ).,,1,1 gttQtt σασα ←++
gf̂
Loopy Belief PropagationLoopy Belief Propagation
ExactExact
0006000ˆ335.36ˆ
LBP
LBP
.=
=
ασ
0007130ˆ624.37ˆ
Exact
Exact
.=
=
ασ
MSE:327
MSE:315
0
0.0002
0.0004
0.0006
0.0008
0.001
0 20 40 60 80 100( )tσ
( )tα
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 3333
ContentsContents
1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical Model Gaussian Graphical Model 4.4. Belief PropagationBelief Propagation5.5. Other ApplicationOther Application6.6. Concluding RemarksConcluding Remarks
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 3434
Digital Images Digital Images InpaintingInpaintingbased on MRFbased on MRF
Inpu
t
Out
put
MarkovRandomFields
M. Yasuda, J. Ohkubo M. Yasuda, J. Ohkubo and K. Tanaka: and K. Tanaka: Proceedings ofProceedings ofCIMCA&IAWTIC2005. CIMCA&IAWTIC2005.
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 3535
ContentsContents
1.1. IntroductionIntroduction2.2. Probabilistic Image ProcessingProbabilistic Image Processing3.3. Gaussian Graphical ModelGaussian Graphical Model4.4. Belief PropagationBelief Propagation5.5. Other ApplicationOther Application6.6. Concluding RemarksConcluding Remarks
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 3636
SummarySummary
Formulation of probabilistic model for image Formulation of probabilistic model for image processing by means of conventional statistical processing by means of conventional statistical schemes has been summarized.schemes has been summarized.Probabilistic image processing by using Gaussian Probabilistic image processing by using Gaussian graphical model has been shown as the most basic graphical model has been shown as the most basic example.example.It has been explained how to construct a belief It has been explained how to construct a belief propagation algorithm for image processing.propagation algorithm for image processing.
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 3737
ReferencesReferences1. K. Tanaka: “Statistical-Mechanical Approach to Image Processing” (Topical
Review), Journal of Physics A: Mathematical and General, vol.35, no.37, pp.R81-R150, 2002.
2. K. Tanaka: “Probabilistic Inference by Means of Cluster Variation Method andLinear Response Theory,” IEICE Transactions on Information and Systems, vol.E86-D, no.7, pp.1228-1242, 2003.
3. K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: “Accuracy of the Bethe Approximation for Hyperparameter Estimation in Probabilistic Image Processing,”Journal of Physics A: Mathematical and General, vol.37, no.36, pp.8675-8696, 2004.
4. J. Ohkubo, M. Yasuda and K. Tanaka: “Statistical-mechanical Iterative Algorithms on Complex Networks,” Physical Review E, vol.72, no.4, Article No.046135, 2005.
5. K. Tanaka and D. M. Titterington: “Statistical Trajectory of Approximate EM Algorithm for Probabilistic Image Processing,” Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp.11285-11300, 2007.
6. M. Yasuda and K. Tanaka: “The Mathematical Structure of the Approximate Linear Response Relation,” Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp.9993-10007, 2007.
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 3838
SMAPIP ProjectSMAPIP ProjectSMAPIP Project
MEXT Grant-in Aid for Scientific Research on Priority AreasMEXT Grant-in Aid for Scientific Research on Priority Areas
Period: 2002 –2005Head Investigator: Kazuyuki TanakaPeriod: 2002 –2005Head Investigator: Kazuyuki Tanaka
Member:K. Tanaka, Y. Kabashima,H. Nishimori, T. Tanaka, M. Okada, O. Watanabe, N. Murata, ......
Statistical Mechanical Approach to Statistical Mechanical Approach to Probabilistic Information ProcessingProbabilistic Information Processing
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 3939
DEX-SMI ProjectDEXDEX--SMI ProjectSMI Project
http://dex-smi.sp.dis.titech.ac.jp/DEX-SMI/http://dexhttp://dex--smi.sp.dis.titech.ac.jp/DEXsmi.sp.dis.titech.ac.jp/DEX--SMISMI//DEX-SMI GOGO
MEXT GrantMEXT Grant--in Aid for Scientific in Aid for Scientific Research on Priority AreasResearch on Priority Areas
Period: 2006 –2009Head Investigator: Prof. Yoshiyuki Kabashima
(Tokyo Institute of Technology)
Period: 2006 –2009Head Investigator: Prof. Yoshiyuki Kabashima
(Tokyo Institute of Technology)
Deepening and Expansion of Deepening and Expansion of Statistical Mechanical InformaticsStatistical Mechanical Informatics
Member:Y. Kabashima, K. Tanaka, H. Nishimori, T. Tanaka, M. Okada, S. Ishii, M. Hayashi,…
29 December, 200829 December, 2008 National Tsing Hua University, TaiwanNational Tsing Hua University, Taiwan 4040
CERIES ProjectCERIES ProjectCERIES Project
http://www.ecei.tohoku.ac.jp/gcoe/http://http://www.ecei.tohoku.ac.jp/gcoewww.ecei.tohoku.ac.jp/gcoe//
CERIES GOGO
GCOE Program
Period: 2007 –2011Head Investigator:
Prof. Fumiyuki Adachi (Tohoku University)
Period: 2007 –2011Head Investigator:
Prof. Fumiyuki Adachi (Tohoku University)
Center of Education and Research for Center of Education and Research for Information Electronics SystemsInformation Electronics Systems