university of orontotfidler/teaching/2015/slides/csc2523/... · 2016. 5. 17. · otalt energy...
TRANSCRIPT
Instance Segmentation
Min Bai
University of Toronto
April 4, 2016
Min Bai (UofT) Instance Segmentation April 4, 2016 1 / 21
Segmentation Review
Task in computer vision
Assign object class label to each pixel in image
Source: Taegyu Lim
Min Bai (UofT) Instance Segmentation April 4, 2016 2 / 21
What is instance segmentation?
Problem:
How many cows are there?How many cars are there?
Source: Taegyu Lim
Min Bai (UofT) Instance Segmentation April 4, 2016 3 / 21
Why do we care?
Richer information about world
Object localization, tracking
Interactions with objects
Source: Japan TimesMin Bai (UofT) Instance Segmentation April 4, 2016 4 / 21
Current Approaches
Convolutional Neural Networks + Conditional Random Fields fordepth ordering
Z. Zhang et al, Monocular Object Instance Segmentation and DepthOrdering with CNNs (ICCV 2015)Z. Zhang et al, Instance-Level Segmentation with Deep DenselyConnected MRFs (CVPR 2016)
Recurrent Neural Networks
Parades et al, Recurrent Instance Segmentation
Min Bai (UofT) Instance Segmentation April 4, 2016 5 / 21
Monocular Object Instance Segmentation and Depth
Ordering with CNNs
Instance Segmentation and Depth Ordering Network
MRF for Patch Merging
Reasons about a globally consistent depth ordering instances
Min Bai (UofT) Instance Segmentation April 4, 2016 6 / 21
Monocular Object Instance Segmentation and Depth
Ordering with CNNs
Instance Segmentation and Depth Ordering Network
Source: Zhang et al
Min Bai (UofT) Instance Segmentation April 4, 2016 7 / 21
MRF Patch Merging
Total energy function to be minimized
Source: Zhang et al
Min Bai (UofT) Instance Segmentation April 4, 2016 8 / 21
MRF Patch Merging
Total energy function to be minimized
First term: Global ordering should always ≥ ordering within patch
seen by CNN
Min Bai (UofT) Instance Segmentation April 4, 2016 9 / 21
MRF Patch Merging
Total energy function to be minimized
Second term: Connected components are ordered vertically. Depth
label should ≥ vertical ordering
Min Bai (UofT) Instance Segmentation April 4, 2016 10 / 21
MRF Patch Merging
Total energy function to be minimized
Third term: depth labeling for pixels belonging to di�erent connected
components should be di�erent
Sparse, random connectivity
Min Bai (UofT) Instance Segmentation April 4, 2016 11 / 21
MRF Patch Merging
Total energy function to be minimized
Fourth term: depth labeling of neighboring pixels in same connected
component should be the same
Min Bai (UofT) Instance Segmentation April 4, 2016 12 / 21
MRF Patch Merging
Total energy function to be minimized
Minimized via quadratic pseudo-boolean optimization and graph cut
Min Bai (UofT) Instance Segmentation April 4, 2016 13 / 21
MRF Patch Merging
Results - Left: input, middle: ground truth, right: result
Source: Zhang et al
Min Bai (UofT) Instance Segmentation April 4, 2016 14 / 21
Instance-Level Segmentation with Deep Densely Connected
MRFs, Zhang et al
Extends previous project
Instead of locally connected MRF, uses fully connected MRF within
each patch and between connected components of di�erent patches
Source: Zhang et alMin Bai (UofT) Instance Segmentation April 4, 2016 15 / 21
Instance-Level Segmentation with Deep Densely Connected
MRFs, Zhang et al
Similar instance segmentation and depth ordering network
Densely connected MRF
Does not reason about depth ordering - only instance identitiesLonger range smoothnessInnovative method for MRF energy minimization
Min Bai (UofT) Instance Segmentation April 4, 2016 16 / 21
Instance-Level Segmentation with Deep Densely Connected
MRFs, Zhang et al
Complete MRF energy term
Source: Zhang et al
Min Bai (UofT) Instance Segmentation April 4, 2016 17 / 21
Instance-Level Segmentation with Deep Densely Connected
MRFs, Zhang et al
Complete MRF energy term
First term: encourages smoothness of instance label assignments
Min Bai (UofT) Instance Segmentation April 4, 2016 18 / 21
Instance-Level Segmentation with Deep Densely Connected
MRFs, Zhang et al
Complete MRF energy term
Second term: encourages global instance assignments of pixels to besame if CNN assigns them to be the same, and di�erent otherwise
Min Bai (UofT) Instance Segmentation April 4, 2016 19 / 21
Instance-Level Segmentation with Deep Densely Connected
MRFs, Zhang et al
Complete MRF energy term
Third term: encourages assignments of labels to pixels belonging todi�erent inter-connected-components to be di�erent
Min Bai (UofT) Instance Segmentation April 4, 2016 20 / 21
Instance-Level Segmentation with Deep Densely Connected
MRFs, Zhang et al
Results, from left to right: input, ground truth, previous paper, this
paper
Source: Zhang et al
Min Bai (UofT) Instance Segmentation April 4, 2016 21 / 21