learning structured visual concepts with few-shot...
TRANSCRIPT
![Page 1: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/1.jpg)
12/5/2019 1
Learning Structured Visual
Concepts with Few-shot Supervision
Xuming He 何旭明
ShanghaiTech University
![Page 2: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/2.jpg)
Outline
Introduction
Learning from very limited annotated data
Background in few-shot learning
Few-shot classification
Meta-learning framework
Towards few-shot representation learning in vision tasks
Spatio-temporal patterns in videos [CVPR 2018]
Visual object & task representation [AAAI 2019]
Summary and future directions
212/5/2019
![Page 3: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/3.jpg)
Data-driven visual scene understanding
Deep Neural Networks require large amount of annotated data
Introduction
312/5/2019
Semantic segmentation Instance segmentation&detection
Depth estimation Image-level description
![Page 4: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/4.jpg)
Real-world scenarios
Data annotation is costly
Many specific domain and cross modality tasks
Visual concept learning in wild
412/5/2019
Medical image understanding
(image credit: 廖飞. 胰腺影像学. 2015.)Biological image analysis
(Zhang and He, 2019) Vision & Language (MSCOCO)
(Liu et al CVPR 2019)
![Page 5: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/5.jpg)
Challenges
Limitation in naïve transfer learning
Insufficient instance variations of novel classes
Fine-tuning usually fails given a few examples per class
Human (child) performance is much better
How do we achieve such data efficiency?
What representations are used?
What are the underlying learning algorithms?
512/5/2019
Image Credit: Ravi & Larochelle et al 2017
![Page 6: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/6.jpg)
Main intuitions in few-shot learning
Prior knowledge in different vision tasks
Similarity between visual categories
Feature representations, etc.
Similarity between visual recognition tasks
Learning a classifier, etc.
Focusing on generic aspects of similar tasks
Generic visual representations
Not category-specific
Transferrable learning strategies
Very data-efficient
612/5/2019
Task 1
Task 2
![Page 7: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/7.jpg)
Outline
Introduction
Learning from very limited annotated data
Background in few-shot learning
Few-shot classification
Meta-learning framework
Towards few-shot representation learning in vision tasks
Spatio-temporal patterns in videos [CVPR 2018]
Visual object & task representation [AAAI 2019]
Summary and future directions
712/5/2019
![Page 8: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/8.jpg)
Few-shot learning problem
Learning from (very) limited annotated data
Typical setting:
Classification using a few training examples per visual category
Formally, given a small dataset
N categories
K shot: each class has K examples, or
The goal is to learn a model F parametrized by to minimize
812/5/2019
Image Credit: Weng, Lil-log, 2018
![Page 9: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/9.jpg)
Few-shot learning problem
For a single isolated task, this is difficult
But if we have access to many similar few-shot learning tasks, we can exploit such prior knowledge.
Main idea is to consider task-level learning
Learn a representation shared by all those tasks
Learn an efficient classifier learning algorithm that can be applied to all the tasks
912/5/2019
Image Credit: Weng, Lil-log, 2018
![Page 10: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/10.jpg)
Meta-learning framework
Problem formulation
Each few-shot classification problem as a task
Each task (or an episode) consists of
Task-train (support) set
Task-test set (query)
For each task, we adopt an learning algorithm
to learn its own classifier via
to perform well on the task-test set
1012/5/2019
![Page 11: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/11.jpg)
Meta-learning formulation
Key assumptions:
The learning algorithm is shared across tasks
We can sample many tasks to learn a good
A meta-learning strategy
Input: meta-training set
Output: algorithm parameter
Objective: good performance on meta-test set
Minimizing the empirical loss on the meta-training set
Each meta-train task
1112/5/2019
![Page 12: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/12.jpg)
Meta-learning formulation
Analogy to standard supervised learning
1212/5/2019 Image Credit: Ravi & Larochelle et al 2017
![Page 13: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/13.jpg)
Overview of existing methods
Depending on the meta-learners used in few-shot tasks
1312/5/2019Slide Credit: Vinyals, NIPS 2017
![Page 14: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/14.jpg)
Metric-based methods
Basic idea: Learn a generic distance metric
1412/5/2019
Typical methods
Siamese network (Koch, Zemel & Salakhutdinov, 2015)
Matching network (Vinyals et al, 2016)
Relation network (Sung et al. 2018)
Prototypical network (Snell, Swersky & Zemel, 2017)
![Page 15: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/15.jpg)
Optimization-based methods
Basic idea: Adjust the optimization in model learning so that the model can effectively learn from a few examples
1512/5/2019
Typical methods
LSTM meta-learner (Ravi &
Larochelle, 2017)
MAML (Finn, et al. 2017)
Reptile (Nichol, Achiam & Schulman, 2018)
![Page 16: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/16.jpg)
Model-based methods
Basic idea: Building a neural network with specific architecture for fast learning
1612/5/2019
Typical methods
Memory-augmented network (Santoro et al., 2016)
Meta networks (Munkhdalai & Yu,
2017)
SNAIL (Mishra et al., 2018)
![Page 17: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/17.jpg)
Main limitations
A global representation of inputs
Sensitive to nuisance parameters: background clutter, occlusions, etc.
Mixed representation and predictor learning
Complex architecture, difficult to interpret
Sometimes slow convergence
Focusing on classification tasks
Non-trivial to apply to other vision tasks: localization, segmentation, etc.
1712/5/2019
![Page 18: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/18.jpg)
Our proposed solutions
Structure-aware data representation
Spatial/temporal representations for semantic objects/actions
Decoupling representation and classifier learning
Improving representation learning
Generalizing to other visual tasks
Instance localization and detection with few-shot learning
1812/5/2019
![Page 19: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/19.jpg)
Outline
Introduction
Learning from very limited annotated data
Background in few-shot learning
Few-shot classification
Meta-learning framework
Towards few-shot representation learning in vision tasks
Spatio-temporal patterns in videos [CVPR 2018]
Visual object & task representation [AAAI 2019]
Summary and future directions
1912/5/2019
![Page 20: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/20.jpg)
Temporal action localization
Our goal: Jointly classify action instances and localizethem in an untrimmed video
Important for detailed video understanding
Broad range of applications in video surveillance/analytics
2012/5/2019
![Page 21: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/21.jpg)
Our problem setting
We conceptualize an example-based action localization strategy
Few-shot learning of action classes and
Being sensitive to action boundaries
2112/5/2019
Few-shot Action
Localization Network
![Page 22: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/22.jpg)
Main ideas
Meta-learning problem formulation
Learning how to transfer the labels of a few action examples to a test video
Encode action instance into a structured representation
Learn to match (partial) action instances
Exploit the matching correlation scores
2212/5/2019
![Page 23: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/23.jpg)
Overview of our method
2312/5/2019
![Page 24: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/24.jpg)
Video encoder network
Embed an action video into a segment-based representation
Maintain its temporal structure
Allows partial matching between two actions
2412/5/2019
![Page 25: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/25.jpg)
Similarity Network
Generate a matching score between labeled examples (support set) and a test window
2512/5/2019
![Page 26: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/26.jpg)
Similarity Network
Full context embedding (FCE)
Capture context of the entire support set and enrich the action representations
2612/5/2019
![Page 27: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/27.jpg)
Similarity Network
Similarity scores
Cosine distance between two action instances
Nearest neighbor for classification, but what about localization?
2712/5/2019
![Page 28: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/28.jpg)
Labeling network
Cache correlation scores for sliding windows
Exploit patterns in the score matrix to predict the locations
2812/5/2019
![Page 29: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/29.jpg)
Matching examples
2912/5/2019
Matching score trajectories
![Page 30: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/30.jpg)
Meta-learning strategy
Meta-training phase
Meta-training set
Task-train (support set)
Task-test (query)
Loss function
Our loss function
Localization loss: foreground vs background (cross entropy)
Classification loss: action class (log loss)
Ranking loss: replacing localization loss to encourage partial alignment
3012/5/2019
![Page 31: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/31.jpg)
Experimental evaluation
Few-shot performance summary
~80 classes for meta-training and ~20 for meta-test
3112/5/2019
Thumos14
ActivityNet
Fully supervised Few-shot
![Page 32: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/32.jpg)
Ablative Study
Effect of the similarity net
Effect of temporal structure
3212/5/2019
![Page 33: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/33.jpg)
Outline
Introduction
Learning from very limited annotated data
Background in few-shot learning
Few-shot classification
Meta-learning framework
Towards few-shot representation learning in vision tasks
Spatio-temporal patterns in videos [CVPR 2018]
Visual object & task representation [AAAI 2019]
Summary and future directions
3312/5/2019
![Page 34: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/34.jpg)
Task: Few-shot image classification
Our goal: An efficient modular meta-learner for visual concepts
A better image representation
An easy-to-interpret encoding method for support set
3412/5/2019
Image Credit: Ravi & Larochelle et al 2017
![Page 35: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/35.jpg)
Main idea
Exploiting attention mechanism in representation learning
Spatial attention to localize the foreground object
Task attention to encode the task context for label prediction
3512/5/2019
![Page 36: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/36.jpg)
Main idea
Exploiting attention mechanism in representation learning
Recurrent attention to refine the representation
3612/5/2019
![Page 37: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/37.jpg)
Dual-attention structure
Spatial attention
Extracting relevant features on Conv-feature maps
Using test image feature as query
3712/5/2019
Pooling
![Page 38: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/38.jpg)
Dual-attention structure
Task attention
Encoding the support set by selecting relevant training examples
3812/5/2019
Support-set representation
![Page 39: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/39.jpg)
Dual-attention structure
Recurrent attention
Refining task-test (query image) features with support set
3912/5/2019
![Page 40: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/40.jpg)
Network architecture
4012/5/2019
![Page 41: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/41.jpg)
Example results
4112/5/2019
![Page 42: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/42.jpg)
Example results
4212/5/2019
![Page 43: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/43.jpg)
A hybrid loss function
4312/5/2019
Standard meta-learning loss + global classification loss
![Page 44: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/44.jpg)
Validation of hybrid loss
We train our models from scratch (no pre-training)
4412/5/2019
![Page 45: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/45.jpg)
Quantitative results
MiniImageNet:
80 classes for meta-training and 20 for meta-test
Roughly 100K tasks for training and 1K for test
4512/5/2019
![Page 46: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/46.jpg)
Quantitative results
MiniImageNet:
80 classes for meta-training and 20 for meta-test
Roughly 100K tasks for training and 1K for test
4612/5/2019
![Page 47: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/47.jpg)
Failure cases
Large variations in scale/viewpoint
4712/5/2019
![Page 48: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/48.jpg)
Research questions I
Task similarity
A new benchmark: Meta-CIFAR100
4812/5/2019
![Page 49: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/49.jpg)
Research questions I
Preliminary results on Meta-CIFAR100
Task similarity plays a key role in few-shot performance
4912/5/2019
![Page 50: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/50.jpg)
Research questions II
From few-shot to low-shot learning
Novel classifier: incremental few-shot learning
How do we exploit unlabeled data?
5012/5/2019
![Page 51: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/51.jpg)
Summary and future directions
Few-shot visual concept learning
Structured representation is important
Modularized, interpretable network design
Extension to multiple vision tasks
Future directions
Studying impact of different task distributions
Connecting few-shot learning to continual learning
Exploring few-shot learning in real-world applications
5112/5/2019
![Page 52: Learning Structured Visual Concepts with Few-shot Supervisionvalser.org/webinar/slide/slides/20191204/few_shot_supervision_web… · 4/12/2019 · Basic idea: Building a neural network](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbdcfb6450b76092a516456/html5/thumbnails/52.jpg)
Acknowledgement
PhD students
Hongtao Yang @ANU
Songyang Zhang and Shipeng Yan @ShanghtaiTech
5212/5/2019
Thank You & Question!