weakly supervised semantic segmentation with image-level...
TRANSCRIPT
![Page 1: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/1.jpg)
Yunchao Wei
Weakly Supervised Semantic Segmentation with Image-level Annotation
https://weiyc.github.io
![Page 2: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/2.jpg)
The trend of weakly-supervised learning
2014 2015 2016
CVPR ICCV/ECCV
2013 2014
5
3
8
6
8 8
12 12
![Page 3: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/3.jpg)
Why do we need WSL?
•The success of DCNN-based object recognition approaches rely on alarge number of labeled images
•Labeling a large amount of images is very costly in terms of bothfinance and human effort.
•Object detection
•Semantic Segmentation
![Page 4: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/4.jpg)
Semantic Segmentation
… …
Fully-convolutional Segmentation Network
Loss
Segmentation
Task
•Fully supervised scheme
![Page 5: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/5.jpg)
Semantic Segmentation
•Weakly-supervised scheme with image-level annotation
person
horsetable
images
annotations
…
…
Weakly-supervised
Semantic Segmentation
Test Image
![Page 6: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/6.jpg)
Proposal-based Solution
Learning to segment with image-level annotations. PR 2016
![Page 7: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/7.jpg)
Proposal-based Solution
Hypotheses-CNN-Pooling
HCP: A flexible CNN framework for multi-label image classification Yunchao Wei, etc. TPAMI 2016
Localization Map Generation
•Exhaustedly examine each proposal togenerate localization
•Time consuming
•Introducing false negative pixels(background)
![Page 8: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/8.jpg)
STC: Simple to Complex
Simple Images Complex Images
•Motivation
STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017
![Page 9: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/9.jpg)
STC: Simple to Complex
•Simple images with the corresponding saliency maps
STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017
![Page 10: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/10.jpg)
STC: Simple to Complex
•Framework•Initial-DCNN
•Enhanced-DCNN
•Powerful-DCNN
STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017
![Page 11: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/11.jpg)
STC: Simple to Complex
•Flickr-Clean(40K)
![Page 12: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/12.jpg)
STC: Simple to Complex
Networks Training Set mIoU
I-DCNN Flickr-Clean 44.1
E-DCNN Flickr-Clean 46.8
P-DCNN Flickr-Clean+VOC 49.8
Ablation Analysis on Pascal VOC12 val
Comparisons on Pascal VOC12 test
Methods mIoU
MIL-FCN (ICLR 2015) 24.9
CCNN (ICCV 2015) 35.5
EM-Adapt (ICCV 2015) 39.6
MIL-ILP-Seg (CVPR 2015) 40.6
STC (ours) 51.2
STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017
![Page 13: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/13.jpg)
STC: Simple to Complex
•Testing Results
•Shortcomings•Depend on a large number of simple images for training.
Image Result GT Image Result GT
![Page 14: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/14.jpg)
Object Region Mining with AE
•Top-down attention• Class Activation Mapping [1]; Excitation Backpropagation [2]
[1] Learning Deep Features for Discriminative Localization. Bolei Zhou etc. CVPR 2016
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
[2] Top-down Neural Attention by Excitation Backprop. Jianmin Zhang etc. ECCV 2016
CVPR 2016 ECCV 2016
![Page 15: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/15.jpg)
Object Region Mining with AE
•Motivation•Classification networks are only responsive to small and sparse discriminative regions from object of interest
•How to obtain dense and integral object-related regions for learning to semantic segmentation?
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
![Page 16: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/16.jpg)
Object Region Mining with AE
•Solution: Adversarial erasing (AE)
Some visualized samples
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
![Page 17: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/17.jpg)
Object Region Mining with AE
•Framework of AE
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
![Page 18: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/18.jpg)
Object Region Mining with AE
•Examples of mined object regions produced by the AE approach
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
![Page 19: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/19.jpg)
Object Region Mining with AE
•Online prohibitive segmentation learning (PSL) for Semantic Segmentation
PSL
Producing Segmentation Mask
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
![Page 20: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/20.jpg)
Object Region Mining with AE
•Ablation Analysis on Pascal VOC12 val
0 150
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Epoch
Loss
AE-step4
AE-step3
AE-step2
AE-step1
AE-Steps mIoU
AE-step1 44.9
AE-step2 49.5
AE-step3 50.9
AE-step4 48.8
Training Schemes mIoU
w/o PSL 50.9
w/ PSL 54.1
w/ PSL+ 55.0
Adversarial erasing
Prohibitive Segmentation Learning
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
![Page 21: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/21.jpg)
Object Region Mining with AE
•Comparisons on Pascal VOC12 testMethods mIoU
MIL-FCN (ICLR 2015) 24.9
CCNN (ICCV 2015) 35.5
EM-Adapt (ICCV 2015) 39.6
MIL-ILP-Seg (CVPR 2015) 40.6
STC (PAMI 2016) 51.2
DCSM (ECCV 2016) 45.1
BFBP (ECCV 2016) 48.0
SEC (ECCV 2016) 51.7
AF-SS(ECCV 2016) 52.7
AE-PSL (ours) 55.7
images predictions GT
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
![Page 22: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/22.jpg)
Future work
22
•Simultaneous weakly-supervised object detection and semanticsegmentation
•Semi-supervised object detection and semantic segmentation
![Page 23: Weakly Supervised Semantic Segmentation with Image-level …valser.org/webinar/slide/slides/20170510/魏云超-valse... · 2019-01-11 · MIL-ILP-Seg (CVPR 2015) 40.6 STC (ours) 51.2](https://reader033.vdocuments.mx/reader033/viewer/2022050206/5f58fead1b112e346e6370ba/html5/thumbnails/23.jpg)
23