Attribute Learning for Understanding Unstructured Social Activity

Yanwei Fu, Timothy M. Hospedales, Tao Xiang,

and Shaogang Gong

School of EECS, Queen Mary University of London, UK

Presented by Amr El-Labban

VGG Reading Group, Dec 5th 2012

Contributions

1. Unstructured social activity attribute (USAA) dataset

2. Semi-latent attribute space

3. Topic model based attribute learning

Objective

Automatic classification of unstructured group social activity

Use an attribute based approach

Start with sparse, user defined attributes

Add latent ones

Learn jointly

Dataset

1500 videos, 8 classes

69 visual/audio attributes manually labelled

Weak labelling

SIFT, STIP and MFCC features used

Data available (features, attributes, YouTube IDs)

Semi-Latent Attribute Space

Space consisting of:

User defined attributes

Discriminative latent attributes

Non-discriminative (background) latent attributes

Topic modelling

P(x|d)

d

x

x – low level features (‘words’)

y – attributes (‘topics’)

d – ‘documents’

= P(x|y)

y

x

P(y|d)

d

y

Latent Dirichlet Allocation

y x

x – low level features

y – attributes (user defined and latent)

θ – attribute distribution

φ – word distribution

α, β – Dirichlet parameters

Aside: Dirichlet disribution

Distribution over multinomial distributions

Parameterised by α

α = (6,2,2)

α = (6,2,6)

α = (3,7,5)

α = (2,3,4)

Aside: Dirichlet disribution

Important things to know:

- peak is closer to larger values - large gives small variance <1 gives more sparse distributions

Latent Dirichlet Allocation

y x

x – low level features

y – attributes (user defined and latent)

θ – attribute distribution

φ – word distribution

α, β – Dirichlet parameters

Latent Dirichlet Allocation

y x

Generative modelfor each document:

Choose θ ~ Dir(α)Choose φ ~ Dir(β)for each word:Choose y ~ Multinomial(θ) Choose x ~ Multinomial(φ y)

Latent Dirichlet Allocation

y x

𝑃 (𝐷|𝛼 , 𝛽)=∏𝑘=1

𝐾

𝑃 (𝜑𝑘∨𝛽)∏𝑚=1

𝑀

𝑃 (𝜃𝑚∨𝛼)∏𝑛=1

𝑁

𝑃 (𝑦𝑚 ,𝑛|𝜃𝑚 )𝑃 (𝑥𝑚 ,𝑛∨𝜑 𝑦𝑚 ,𝑛)

Latent Dirichlet Allocation

y x

EM to learn Dirichlet parameters:

Approximate inference for posterior:

SLAS

User defined part Per instance prior on α. Set to zero when attribute isn’t present in ground truth

Latent part First half “class conditional”

One α per class. All but one constrained to zero.

Second half “background” Unconstrained

Classification

Use SLAS posterior to map from raw data to attributes

Use standard classifier (logistic regression) from attributes to classes

N-shot transfer learning

Split data into two partitions – source and target

Learn attribute models on source data

Use N examples from target to learn attribute-class mapping

Zero-shot learning

Detect novel class Manually defined attribute-class “prototype” Improve with self-training algorithm:

1. Infer attributes for novel data

2. NN matching in user defined space against protoype

3. For each novel class:

a) Find top K matches

b) Train new prototype in full attribute space (mean of top K)

4. NN matching in the full space

Experiments

Compare three models:

Direct: KNN or SVM on raw data

SVM-UD+LR: SVM to map raw data to attributes, LR maps attributes to classes

SLAS+LR: SLAS to map raw data to attributes, LR learns classes based on user-defined and class conditional attributes.

MASSIVE HACK

“The UD part of the SLAS topic profile is estimating the same thing as the SVM attribute classifiers, however the latter are slightly more reliable due to being discriminatively optimised. As input to LR, we therefore actually use the SVM attribute classier outputs in conjunction with the latent part of our topic profile.”

Results - classification

SLAS+LR better as number if training data and user defined attributes decreases

Copes with 25% wrong attribute bits

Results - classification

KNN and SVM have vertical bands – consistent misclassification

Results – N-shot transfer learning

Vary number of user defined attributes

SVM+LR cannot cope with zero attributes

Results – Zero-shot transfer learning

Two cases: Continuous prototype – mean attribute profile Binary prototype – thresholded mean

Tested without background latent attributes (SLAS(NF))

Conclusion

Augmenting SVM and user defined attributes with latent ones definitely helps.

Experimental hacks make it hard to say how good the model really is…