constrained semi-supervised learning using attributes and comparative attributes
DESCRIPTION
Constrained Semi-Supervised Learning using Attributes and Comparative Attributes. Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta The Robotics Institute Carnegie Mellon University. Supervision. Big-Data. Active Learning. Supervised. - PowerPoint PPT PresentationTRANSCRIPT
CONSTRAINED SEMI-SUPERVISED LEARNING
USING ATTRIBUTES AND COMPARATIVE ATTRIBUTESAbhinav Shrivastava, Saurabh Singh, Abhinav Gupta
The Robotics InstituteCarnegie Mellon University
2
SUPERVISION
SUPERVISED
ACTIVELEARNING
BIG-DATA
[von Ahn and Dabbish, 2004], [Russell et al., IJCV 2009][Prakash and Parikh, ECCV 2012] [Vijayanarasimhan and Grauman, CVPR 2011] [Kapoor et al., ICCV 2007][Qi et al., CVPR 2008] [Joshi et al., CVPR 2009] [Siddiquie and Gupta, CVPR 2010]
3
SUPERVISION
SUPERVISED
UNSUPERVISED
ACTIVELEARNING
[Russell et al., CVPR 2006] [Kang et al., ICCV 2011]
4
SUPERVISION
SUPERVISED
UNSUPERVISED
ACTIVELEARNINGSEMI-
SUPERVISED
[Zhu, TR, 2005], [Chunsheng Fang, Slides, 2009]
5
SUPERVISION
SUPERVISED
UNSUPERVISED
ACTIVELEARNINGSEMI-
SUPERVISEDBOOTSTRAPPINGLabeled Seed
ExamplesAmphitheatre
Unlabeled Data
Select Candidates
TrainModels
Add to Labeled Set
RetrainModels
Amphitheatre
6
SUPERVISION
SUPERVISED
UNSUPERVISED
ACTIVELEARNINGSEMI-
SUPERVISEDBOOTSTRAPPINGRetrainModels
Labeled Seed Examples
Amphitheatre
Unlabeled Data
Select Candidates
Add to Labeled Set
Amphitheatre
25th Iteration
[Curran et al., PACL 2007]
Semantic Drift
Amphitheatre + Auditorium
7
SUPERVISION
SUPERVISED
UNSUPERVISED
ACTIVELEARNINGSEMI-
SUPERVISEDGRAPH-BASED METHODS
[Ebert et al., ECCV 2010] [Fergus et al., NIPS 2009]
8
Amphitheatre Amphitheatre
OUR APPROACHAmphitheatre
Auditorium
Amphitheatre
Auditorium
9
OUR APPROACHAmphitheatre
Auditorium
Amphitheatre
Auditorium
Joint Learning
[Carlson et al., NAACL HLT Workshop on SSL for NLP 2009]
Share Data
AmphitheatreAmphitheatre
AuditoriumAuditorium
BanquetHall
BanquetHall
Conference Room
Conference Room
BINARY ATTRIBUTES (BA)
Indoor Man-madeTables and Chairs Large Seating CapacityIndoor Man-madeTables and Chairs Large Seating Capacity
10[Farhadi et al., CVPR 2009] [Lampert et al., CVPR 2009]
BINARY ATTRIBUTES (BA)Tables and Chairs
Conference Room
BanquetHall
Auditorium
Amphitheatre
Indoor
Large Seating Capacity
Man-made
[Patterson and Hays, CVPR 2012]
Tables and Chairs
Conference Room
BanquetHall
Auditorium
Amphitheatre
Indoor
Large Seating Capacity
Man-made
12
AuditoriumIndoor Has Seat Rows
✗
13
SHARING VIA DISSIMILARITY
Amphitheatre Auditorium
Has Larger Circular Structures
[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]
14
Amphitheatre AuditoriumHas Larger
Circular Structures
?
15
✗
Amphitheatre AuditoriumHas Larger
Circular Structures
16
DISSIMILARITY
Has Larger Circular Structures
[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]
COMPARATIVE ATTRIBUTES
17
DISSIMILARITYCOMPARATIVE ATTRIBUTES
Has Larger Circular
Structures
[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]
……
……
……
……
Features• GIST• RGB (Tiny Image)• Line Histogram of:
Length Orientation• LAB histogram
[Hays and Efros, SIGGRAPH 2007][Oliva and Torralba, 2006][Torralba et al., PAMI, 2008]
18
……
……
DISSIMILARITYCOMPARATIVE ATTRIBUTES
[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]
……
…… Has Larger
Circular Structures
ClassifierBoosted Decision Tree[Hoiem et al., IJCV 2007]
✗or
Has Larger Circular
Structures
19
COMPARATIVE ATTRIBUTES
[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]
Amphitheatre > BarnAmphitheatre > Conference Room
Desert > Barn
Is More Open
Church (Outdoor) > CemeteryBarn > Cemetery
Has Taller Structures
20
Amphitheatre
Auditorium
Amphitheatre
Auditorium
Labeled Seed Examples Bootstrapping
Selected Candidates
21
Labeled Seed Examples
Amphitheatre
Auditorium
Amphitheatre
Auditorium
Bootstrapping
Amphitheatre
Auditorium
Our Approach (Constrained Bootstrapping)
Selected Candidates
Indoor
Has Seat Rows
Attributes
Has Larger Circular
Structures
ComparativeAttributes
22
Banq
uet
Bedr
oom
Labeled Data
Unlabeled Data
has more space
has larger structures
Training Pairwise Data
Promoted InstancesConference Room Banquet Hall
[Gupta and Davis, ECCV 2008]
Comparative Attribute Classifiers
mor
e sp
ace
larg
er st
ruct
ures
Attribute Classifiersin
door
has g
rass
Scene Classifiers
bedr
oom
banq
uet h
all
23
Conference Room
Itera
tion
1Ite
ratio
n 40
Seed
Ex
ampl
es
Introspection
24
Boot
stra
ppin
gBA
Con
stra
ints
AmphitheatreO
ur A
ppro
ach
Seed
Imag
es
26
EXPERIMENTAL EVALUATION
27
CONTROL EXPERIMENTS# Images
(SUN Database)
15 Scene Classes 2 (seed)
Black-box Binary Attributes (BA) 25 (separate)
Black-box Comparative Attributes (CA) 25 (separate)
Unlabeled Dataset 18,000(Distractors) (9,500)Test Set 50
SUN Database : [Xiao et al., CVPR 2010]
28
CONTROL EXPERIMENTS
29
CONTROL EXPERIMENTS
30
CONTROL EXPERIMENTS
31
CONTROL EXPERIMENTS
32
CONTROL EXPERIMENTS
\\
33
CONTROL EXPERIMENTS
\\
Eigen Functions: [Fergus et al., NIPS 2009]
34
1
40
Banquet Hall
10
Itera
tions
Seed
Imag
es
35
CONTROL EXPERIMENTS# Images
(SUN Database)
15 Scene Classes 2 (seed)
Black-box Binary Attributes (BA) 25 (separate)
Black-box Comparative Attributes (CA) 25 (separate)
Unlabeled Dataset 18,000(Distractors) (9,500)Test Set 50
SUN Database : [Xiao et al., CVPR 2010]
36
CO-TRAINING (SMALL SCALE)# Images
(SUN Database)
15 Scene Classes 2 (seed)
Black-box Binary Attributes (BA) 15x2 (seed)
Black-box Comparative Attributes (CA) 15x2 (seed)
Unlabeled Dataset 18,000(Distractors) (9,500)Test Set 50
SUN Database : [Xiao et al., CVPR 2010]
37
Itera
tion-
1Ite
ratio
n-60
Boot
stra
ppin
gO
ur A
ppro
ach
Itera
tion-
1Ite
ratio
n-60
Seed
Imag
esBedroom
38
SCENE CLASSIFICATION
Eigen Functions: [Fergus et al., NIPS 2009]
39
CO-TRAINING (LARGE SCALE)• 15 Scene Categories
25 Seed images / category
• Unlabeled Set 1Million (SUN Database + ImageNet) >95% distractors
SUN Database: [Xiao et al., CVPR 2010]ImageNet: [Deng et al., CVPR 2009]
Improve 12 out of 15 scene classifiers
40
CONCLUSION• Sharing via Dissimilarities
• Constrained Bootstrapping
AuditoriumAmphitheatre
Has Larger Circular
Structures
Labeled Seed Examples
Amphitheatre
Auditorium
Amphitheatre
Auditorium
Bootstrapping
Amphitheatre
Auditorium
Constrained Bootstrapping
41
Banq
uet
Bedr
oom
Labeled Data
Unlabeled Data
has more space
has larger structures
Training Pairwise Data
Promoted InstancesConference Room Banquet Hall
Comparative Attribute Classifiers
mor
e sp
ace
larg
er st
ruct
ures
Attribute Classifiersin
door
has g
rass
Scene Classifiers
bedr
oom
banq
uet h
all
42
43
Unary
Binary
44
EIGEN FUNCTIONSSettings from Fergus et al. NIPS, 2009• GIST descriptor• 32D space using PCA– k=64 eigen functions
45
FEATURES• 960D GIST• 75D RGB (image is resized to 5x5)• 30D histogram of line lengths• 200D histogram of orientation of lines• 784D 3D-histogram Lab color space (14x14x4)
• Total 2049
[Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al., PAMI, 2008]
46
CATEGORIES
47
BINARY ATTRIBUTE RELATIONSHIP
48
COMPARATIVE ATTRIBUTE RELATIONSHIPS
49
CLASSIFIERS• Boosted Decision Trees– From [Hoiem et al., IJCV 2007]
• Scene– 20 Trees, 8 Nodes
• Binary Attribute– 40 Trees, 8 Nodes
• Comparative Attribute• 20 Trees, 4 Nodes• Differential features as in [Gupta and Davis, ECCV 2008]
50
SUPERVISION
SUPERVISED
ACTIVELEARNING
[Torralba et al., PAMI 2008]
51
SUPERVISION
SUPERVISED
UNSUPERVISED
ACTIVELEARNINGSEMI-
SUPERVISEDGRAPH-BASED METHODS
Train Bus
52
MUTUAL EXCLUSION (ME)
53
EXPERIMENTAL EVALUATION
Experiments• Control• Small-scale• Large-scale
Evaluation Metrics• Scene Classification– Mean AP
• Purity of Labels
Datasets• SUN Database• ImageNet
15 Scene Categories
54
BASELINES• Bootstrapping (Self-learning)
- Binary Classifiers- Multi-class Classifiers
• Eigen Function based Graph Laplacian
55
SCENE CLASSIFICATION
Eigen Functions: [Fergus et al., NIPS 2009]
56
PURITY OF LABELING
Eigen Functions: [Fergus et al., NIPS 2009]
57
58
1
40
Banquet Hall
90
10Itera
tions
Seed
Imag
es
59
60
PURITY OF LABELING
Eigen Functions: [Fergus et al., NIPS 2009]
61
SCENE CLASSIFICATION
Mean