Large-scale CelebA face attribute dataset
• 200K celebrity images, each with 40 attributes
• Highly imbalanced: average positive class rate 23%
• Total accuracy =𝑡𝑝 + 𝑡𝑛
𝑁𝑝 + 𝑁𝑛Balanced accuracy =
1
2
𝑡𝑝
𝑁𝑝+
𝑡𝑛
𝑁𝑛
Edge detection on BSDS500 dataset
• Retrieve from 2M edge label patches with long-tail distribution
Learning Deep Representation for Imbalanced ClassificationChen Huang1,2, Yining Li1, Chen Change Loy1, Xiaoou Tang1
1The Chinese University of Hong Kong 2SenseTime Group Limited
{chuang, ly015, ccloy, xtang}@ie.cuhk.edu.hk
Data imbalance is common in visual classification
1. Motivation
Wearing
hat
Not wearing
hat
…
Minority class
Majority class
Face attribute example
Deep embedding: Class-level cluster- & class-level constraint
Study traditional re-sampling and cost-sensitive learning scheme
2. Main Idea
Triplet embedding
Class 1
minority
Class 2
majority
Class 2 majority
Quintuplet embedding
Class 1 minority
…
Cluster j
Cluster 2
Cluster 1 Cluster 1
𝐷 𝑓 𝑥𝑖 , 𝑓 𝑥𝑖𝑝+
< 𝐷(𝑓 𝑥𝑖 , 𝑓(𝑥𝑖𝑝−))
< 𝐷(𝑓 𝑥𝑖 , 𝑓(𝑥𝑖𝑝−−
)) < 𝐷(𝑓 𝑥𝑖 , 𝑓(𝑥𝑖𝑛))
𝑥𝑖 – an anchor
𝑥𝑖𝑝+
– the anchor’s most distant within-cluster neighbor
𝑥𝑖𝑝−
– the nearest within-class neighbor of the anchor, but from a different cluster
𝑥𝑖𝑝−−
– the most distant within-class neighbor of the anchor
𝑥𝑖𝑛 – the nearest between-class neighbor of the anchor
Triple-header hinge loss
Network architecture
• Equal class re-sampling & class costs assignment in batches
Training step
3. Large Margin Local Embedding (LMLE)
s.t.:
● Clustering by k-means
● Generate quintuplets from
cluster & class membership
● Re-sample batches equally from each class
● Forward their quintuplets to CNN to
compute loss
● Back-propagation
Feature-based clustering
Feature learning/updating
Every 5000 iterations
CNN
CNN
CNN
CNN
CNN
Trip
le-h
ead
er h
ing
e lo
ss
Mini-
batches
Training
samples …
EmbeddingQuintuplet
Shared parameters
Large margin cluster-wise kNN: fast & imbalance resistant
4. Cluster-wise kNN search
-8 -6 -4 -2 0 2 4 6 8-5
-4
-3
-2
-1
0
1
2
3
4
5
-15 -10 -5 0 5 10-20
-15
-10
-5
0
5
10
15
-10 -8 -6 -4 -2 0 2 4 6 8-8
-6
-4
-2
0
2
4
6
8
NC2
NC3
NC4
NC5
PC1
PC2
NC1
-8 -6 -4 -2 0 2 4 6 8-5
-4
-3
-2
-1
0
1
2
3
4
5
-15 -10 -5 0 5 10-20
-15
-10
-5
0
5
10
15
-10 -8 -6 -4 -2 0 2 4 6 8-8
-6
-4
-2
0
2
4
6
8
NC2
NC3
NC4
NC5
PC1
PC2
NC1
-8 -6 -4 -2 0 2 4 6 8-5
-4
-3
-2
-1
0
1
2
3
4
5
-15 -10 -5 0 5 10-20
-15
-10
-5
0
5
10
15
-10 -8 -6 -4 -2 0 2 4 6 8-8
-6
-4
-2
0
2
4
6
8
NC2
NC3
NC4
NC5
PC1
PC2
NC1
-8 -6 -4 -2 0 2 4 6 8-5
-4
-3
-2
-1
0
1
2
3
4
5
-15 -10 -5 0 5 10-20
-15
-10
-5
0
5
10
15
-10 -8 -6 -4 -2 0 2 4 6 8-8
-6
-4
-2
0
2
4
6
8
NC2
NC3
NC4
NC5
PC1
PC2
NC1
Class 1: cluster 1
Class 1: cluster 2
Class 2: cluster 1
Class 2: cluster 2
Class 2: cluster 3
Class 2: cluster 4
Class 2: cluster 5
-8 -6 -4 -2 0 2 4 6 8-5
-4
-3
-2
-1
0
1
2
3
4
5
-15 -10 -5 0 5 10-20
-15
-10
-5
0
5
10
15
-10 -8 -6 -4 -2 0 2 4 6 8-8
-6
-4
-2
0
2
4
6
8
NC2
NC3
NC4
NC5
PC1
PC2
NC1
-8 -6 -4 -2 0 2 4 6 8-5
-4
-3
-2
-1
0
1
2
3
4
5
-15 -10 -5 0 5 10-20
-15
-10
-5
0
5
10
15
-10 -8 -6 -4 -2 0 2 4 6 8-8
-6
-4
-2
0
2
4
6
8
NC2
NC3
NC4
NC5
PC1
PC2
NC1
DeepID2 Triplet Our LMLE
5. Results
Total acc. Balanced acc.
Triplet-kNN 83 72
Anet 87 80
LMLE-kNN 90 84
Ground truthSketchToken
ODS 0.73
DeepContour
ODS 0.76
LMLE-kNN
ODS 0.78
6. Conclusion
Cluster- & class-level quintuplets preserve both locality
across clusters and discrimination between classes,
irrespective of class imbalance
Large margin classification by fast cluster-wise kNN search