robust higher order potentials for enforcing label consistency pushmeet kohli microsoft research...
Post on 19-Dec-2015
220 views
TRANSCRIPT
Robust Higher Order Potentials Robust Higher Order Potentials For Enforcing Label ConsistencyFor Enforcing Label Consistency
Pushmeet KohliPushmeet KohliMicrosoft Research CambridgeMicrosoft Research Cambridge
Lubor LadickyLubor Ladicky Philip TorrPhilip TorrOxford Brookes University, OxfordOxford Brookes University, Oxford
CVPR 2008CVPR 2008
Image labelling Problems
Image Denoising
Geometry Estimation
Object Segmentation
Assign a label to each image Assign a label to each image pixelpixel
Building
Sky
Tree Grass
Object Segmentation using CRFs
Unary potentials based on
Colour, Location and Texture
features
CRF Energy
(Shotton et al. ECCV 2006)
Encourages label consistency in adjacent pixels
Limitations of Pairwise CRFs
Image
Unary Potential
MAP-CRF
Solution
• Encourages short boundaries (Shrinkage bias)
• Can only enforce label consistency in adjacent pixels
• Inability to incorporate region based features
Label Consistency in Image Regions
• Pixels constituting some regions belong to- Same plane (Orientation)
(Hoiem, Efros, & Herbert, ICCV’05)- Same object
(Russel, Efros, Sivic, Freeman, & Zisserman, CVPR06)
Image (MSRC) Segmentation (Mean shift)
Image labelling using segments
Image Object Labelling
UnsupervisedSegmentation
• Geometric Context [Hoiem et al, ICCV05]
• Object Segmentation[He et al. ECCV06, Yang et al. CVPR07, Rabinovich et al. ICCV07, Batra et al. CVPR08]
• Interactive Video Segmentation[Wang, SIGGRAPH 2005 ]
Not robust to Inconsistent Segments!
Our Higher Order CRF Model
cMultiple Segmentation
s
Encourages label consistency in
regions
Label Consistency in Segments
• Encourages consistency within super-pixels
• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]
c
Label Consistency in Segments
Cost: 0
c
• Encourages consistency within super-pixels
• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]
Label Consistency in Segments
Cost: f (|c|)
c
• Encourages consistency within super-pixels
• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]
• Encourages consistency within super-pixels
• Takes the form of a PN Potts model[Kohli et al. CVPR 2007]
Label Consistency in Segments
Cost: f (|c|)
c
Does not distinguish between Good/Bad
Segments !
Quality based Label Consistency
How to measure quality G(c)?[Ren and Malik ICCV03, Rabinovich et al. ICCV07, many others]
• Colour and Texture Similarity• Contour Energy
Label inconsistency cost depends on segment quality
Measure quality from variance in feature responses
Higher order generalization of contrast-sensitive pairwise potential
Quality based Label Consistency
MSRC imageMean shift
segmentation
Segment Quality(darker is better)
Robust Consistency Potentials
Inconsistent Pixels
max
0 1
PN Potts
Too Rigid!
max
0 1 T
Robust
Inconsistent Pixels
Robust Consistency Potentials
Number of Inconsistent
Pixels
SlopeMaximum
Inconsistency Cost
max
0 1 T
Robust
Inconsistent Pixels
Minimizing Higher order Energy Functions
• Message passing is computationally expensive– High runtime and space complexity - O(LN) – L = Number of Labels, N = Size of Clique
• Efficient BP for Higher Order MRFs [Lan et al. ECCV 06, Potetz CVPR 2007]
– 2x2 clique potentials for Image Denoising – Take minutes per iteration (Hours to
converge)
Minimizing Higher order Energy Functions
• Graph Cut based move making algorithm[Kohli et al. CVPR 2007]
– Can handle very high order energy functions – Extremely efficient: computation time in the order of
seconds– Only applicable to some classes of functions (PN Potts)– Cannot handle robust consistency potential
• This paper– Can minimize a much larger class of higher order energy
functions– Same time complexity as [Kohli et al. CVPR 2007]
Move making algorithms
Expansion and Swap move algorithms[Boykov Veksler and Zabih, PAMI 2001]
• Makes a series of changes to the solution (moves)• Each move results in a solution with smaller energy
Current SolutionCurrent Solution
Generate pseudo-boolean move
function
Generate pseudo-boolean move
function
Minimize move function to get optimal move
Minimize move function to get optimal move
Move to new
solution
Move to new
solution
How to minimize
move functions?
Minimizing Move Functions using Graph Cuts
st-mincut
(Positive weights)
st-mincut
(Positive weights)
Second orderPseudo-boolean
Function Minimization (submodular)
Second orderPseudo-boolean
Function Minimization (submodular)
Optimal moves can be found extremely efficiently using graphs cuts
Most pairwise CRF models used in Computer Vision lead to submodular
move functions
Minimizing Higher Order Energy Functions: Our results
• Result 1– We show that a large class of higher order potentials
lead to higher order submodular move functions– Can be minimized in polynomial time
• Submodular Function Minimization– Minimizing general submodular functions is
computationally expensive– Complexity O(n6)– Cannot handle large problems!
Details in Technical ReportDetails in Technical Report
Minimizing Higher Order Energy Functions: Our results
• Minimizing Higher order functions using Graph cuts– Higher order functions can be transformed to second
order functions by adding auxillary variables– Exponential number of auxillary variables needed in
general
• Result 2– Our higher order functions can be transformed to
second-order functions using ≤2 auxillary variables per potential.
– Can be minimized extremely efficiently – Complexity << O(n6) Details in Technical ReportDetails in Technical Report
Overview of our Method
Unary Potentials [Shotton et al. ECCV
2006]
+Contrast Sensitive Pairwise
Potentials
Higher Order Potentials
(Multiple Segmentations)
Segmentation Solution
+
Higher Order Energy
Energy Minimizatio
n
Experimental results
Datasets: MSRC (21), Sowerby (7) [Shotton et al. ECCV 2006] [He et al. CVPR 04]
Qualitative Results
Image(MSRC-
21)
Pairwise CRF
Higher order CRF
Ground Truth
SheepSheep
GrassGrass
Qualitative Results (Contd..)
Image(MSRC-
21)
Pairwise CRF
Higher order CRF
Ground Truth
Results can be improved using image specific colour models Rother et al. SIGGRAPH 2004 Shotton et al. ECCV
2006
Quantitative Results: Problems
Rough ground truth segmentations
Fine structures have small influence on overall pixel accuracy
Generating Accurate Segmentations
Image(MSRC-21)
OriginalSegmentation
NewSegmentation
• Generated accurate segmentation of 27 images• 30 minutes per image
Relationship between Qualitative and Quantitative
Results
Image(MSRC-21)
Pairwise CRF
Higher order CRF
Ground Truth
95.8%Overall Pixel
Accuracy
98.7%
Small changes in pixel accuracy can lead to large improvements in segmentation results.
Quantitative Accuracy• Measure accuracy in labelling boundary pixels.• Accuracy evaluated in boundary bands of
variable width
Image(MSRC-21)
Hand-labelled Segmentation
Trimap(8-pixels)
Trimap(16-pixels)
Quantitative Accuracy
• Measure accuracy in labelling boundary pixels.
• Accuracy evaluated in boundary bands of variable width
Conclusions
• Method to enforce label consistency in image regions
• Generalization of the commonly used Pairwise CRF model
• Allows integration of pixel and region level features for image labelling problems
Thanks
Running Time ResultsTim
e (
sec)
Number of Segmentations
Inconsistency Cost
Qualitative Results (Contd..)
Image(MSRC-
21)
Pairwise CRF
Higher order CRF
Ground Truth
Transformation to second order functions
cAuxiliary variables