video image segmentation with graphical modelsvideo image segmentation goal: to label the image...
TRANSCRIPT
![Page 1: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/1.jpg)
Video Image Segmentation with Graphical models
Jieyu ZhaoJieyu Zhao
![Page 2: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/2.jpg)
OutlineFundamentalsFundamentalsDeterministic MethodsDeterministic MethodsStochastic Methods Stochastic Methods Some ResultsSome ResultsConclusionsConclusions
![Page 3: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/3.jpg)
Video Image Segmentation
Goal:Goal: to label the image regions with salient to label the image regions with salient homogeneous propertieshomogeneous properties, , such as color, such as color, texture, motion or texture, motion or spatiospatio--temporal structurestemporal structures
The labeling algorithms based on graphical The labeling algorithms based on graphical models become popular in recent years.models become popular in recent years.
deterministic and stochasticdeterministic and stochastic
![Page 4: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/4.jpg)
![Page 5: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/5.jpg)
![Page 6: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/6.jpg)
Deterministic Algorithms
Belief Propagation, which infers marginal Belief Propagation, which infers marginal probabilities at the nodes of the graph by probabilities at the nodes of the graph by exchanging of messagesexchanging of messagesinitially designed on trees and later generalizedinitially designed on trees and later generalizedMinimum Graph Cut, popular deterministic Minimum Graph Cut, popular deterministic method maps the image segmentation task into method maps the image segmentation task into a Maxa Max--Flow/Min Cut problem Flow/Min Cut problem Other related approaches, such as normalized Other related approaches, such as normalized cutcut
![Page 7: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/7.jpg)
Stochastic Algorithms
Mainly based on the Gibbs sampler, a Markov Mainly based on the Gibbs sampler, a Markov chain Monte Carlo algorithmchain Monte Carlo algorithm
Markov random field approaches Markov random field approaches random walk and diffusion approaches random walk and diffusion approaches the Potts models, the the Potts models, the SwendsenSwendsen--Wang method. Wang method.
Stochastic approaches are usually powerful but Stochastic approaches are usually powerful but timetime--consumingconsuming
![Page 8: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/8.jpg)
Representation
image represented with a weighted graph,image represented with a weighted graph,vertices reflect the states of image pixels and vertices reflect the states of image pixels and weighted edges represent the relationship weighted edges represent the relationship between pixels.between pixels.
44--neighbour structure,neighbour structure,weights represent theweights represent thesimilarities.similarities.
Segmentation ~ Min CutSegmentation ~ Min Cut
![Page 9: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/9.jpg)
Maximum flow / Minimum cut
“Max flow”: maximize the sum ∑u f(u,t)
“Min cut”: Delete the "best" set of edges to disconnect t from s, with the smallest capacity
![Page 10: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/10.jpg)
A weighted graph -- material flowing through the edges (railways, water pipelines)
Maximum flow: maximize the sum ∑u f(u,t)
![Page 11: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/11.jpg)
A cut is a node partition (S, T) such that s is in S and t is in T.
capacity(S, T) = sum of weights of edges leaving S.
![Page 12: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/12.jpg)
a min cut
![Page 13: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/13.jpg)
Max-flow min-cut theorem: The value of the max flow is equal to the capacity of the min cut.
Augmenting path theorem: A flow f is a max flow if and only if there are no augmenting paths.
The following are equivalent:(i) f is a max flow.(ii) There is no augmenting path relative to f.(iii) There exists a cut whose capacity equals
the value of f.
![Page 14: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/14.jpg)
Augmenting path = path in residual graph.Increase flow along forward edges.Decrease flow along backward edges.
original graphresidual graph
Flow f(e). "Undo" flow sent.Edge e = v w w v
![Page 15: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/15.jpg)
![Page 16: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/16.jpg)
Image Segmentation Using Min CutImage Segmentation Using Min CutCalculating weighted graphCalculating weighted graphSetting some seed points, automatically Setting some seed points, automatically or interactivelyor interactivelyMax Flow AlgorithmMax Flow Algorithm
Tends to have small and Tends to have small and biased segmentationbiased segmentationImproved by the normalized Improved by the normalized cut: cut:
![Page 17: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/17.jpg)
History of Worst-Case Running Times
![Page 18: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/18.jpg)
Stochastic Algorithms
Markov random field approachesMarkov random field approachesPotts model, Potts model, SwendsenSwendsen--Wang method Wang method Random walk and diffusion approachesRandom walk and diffusion approaches
![Page 19: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/19.jpg)
Markov random fields
Positive:Positive:
Markovian: state only depends on Markovian: state only depends on neighborsneighbors
Homogenious: probability independent of Homogenious: probability independent of positions of sitespositions of sites
FffP ∈∀> ,0)(
)|()|( }{ iNiiSi ffPffP =−
![Page 20: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/20.jpg)
Markov-Gibbs Equivalence
GRF GRF ---- global property (global property (the Gibbs distributionthe Gibbs distribution) )
MRF MRF ---- local property (local property (the Markovianitythe Markovianity))
The HammersleyThe Hammersley--Clifford theorem, the Clifford theorem, the equivalence of these two:equivalence of these two:
F is an MRF on S with respect to N if and F is an MRF on S with respect to N if and only if F is a GRF on S with respect to N only if F is a GRF on S with respect to N ..
![Page 21: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/21.jpg)
Gibbs distribution: Gibbs distribution:
where where EE is the energy function, is the energy function, TT is the temperature.is the temperature.
(a)(a) maximization of the posterior probability in themaximization of the posterior probability in theBayesian framework Bayesian framework
(b)(b) minimization of the posterior energy minimization of the posterior energy function of a MRFfunction of a MRF
(c)(c) minimization of the energy in a stochastic minimization of the energy in a stochastic recurrent networkrecurrent network
∑∈
−
−
=
Ff
TfE
TfE
eefP )(
)(
)(
![Page 22: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/22.jpg)
Ising/Potts ModelsIsingIsing model has a choice of two possible model has a choice of two possible
spin states at each lattice pointspin states at each lattice point
![Page 23: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/23.jpg)
Potts models have q>2 possible states:Potts models have q>2 possible states:S1, S2, S3, S4, S1, S2, S3, S4, …… SqSq
![Page 24: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/24.jpg)
Segmentation with Potts Models
![Page 25: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/25.jpg)
Swendsen-Wang method
SW method speeds up the timeSW method speeds up the time--consuming consuming process by flipping the color of all vertices process by flipping the color of all vertices in one or all clusters simultaneously in one or all clusters simultaneously
![Page 26: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/26.jpg)
My WorkAdd external fields for segmentationAdd external fields for segmentationWorking at low temperature or deterministicallyWorking at low temperature or deterministicallyNoisy video image segmentationNoisy video image segmentation
Probability is given by:Probability is given by:
EM algorithm developed to estimate the model EM algorithm developed to estimate the model parametersparameters
),21exp(),(),|(
)(
1 ∑ ∑∈ ∈
− +=Si iNj
jti
tiG xxxWP βββ VVVx
![Page 27: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/27.jpg)
![Page 28: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/28.jpg)
Random Walk MethodsLabels: Labels:
L1, L2, L3L1, L2, L3
Weights: in [0,1]Weights: in [0,1]
![Page 29: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/29.jpg)
Probability of reaching L1 Probability of reaching L2Probability of reaching L1 Probability of reaching L2
![Page 30: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/30.jpg)
Probability of reaching L3Probability of reaching L3 Segmentation results
![Page 31: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/31.jpg)
My WorkMake it fast, local and limited stepsMake it fast, local and limited stepsReduce noise while keeping edgesReduce noise while keeping edgesApply to facial feature extractionApply to facial feature extraction
the random walkers eliminate the noise the random walkers eliminate the noise and keep the mutually connected feature and keep the mutually connected feature pixels from vaguenesspixels from vaguenesslike morphology filters but it does not like morphology filters but it does not need to define a structural element in need to define a structural element in advanceadvance
![Page 32: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/32.jpg)
![Page 33: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/33.jpg)
![Page 34: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/34.jpg)
![Page 35: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/35.jpg)
Conclusions and Future Work
Graphical models are powerful and ideal Graphical models are powerful and ideal for image segmentationfor image segmentationChoice of the deterministic and stochastic Choice of the deterministic and stochastic algorithms, tradealgorithms, trade--off off To make them more robust and develop To make them more robust and develop some applicationssome applications
![Page 36: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/36.jpg)
Thank You !
![Page 37: Video Image Segmentation with Graphical modelsVideo Image Segmentation Goal: to label the image regions with salient homogeneous properties, such as color, texture, motion or spatio-temporal](https://reader034.vdocuments.mx/reader034/viewer/2022042804/5f5979a1afa4520ae1630d60/html5/thumbnails/37.jpg)