Relative AttributesDevi Parikh, Kristen Grauman

[Website] [Paper] [Slides] [Code] [Demo]

Slides by: Dèlia Fernández Cañellas Computer Vision Reading Group: 26/04/2016

1. Introduction and

Motivation

1. Introduction and MotivationWhen dealing with recognition tasks, human-nameable visual "attributes" can be beneficial, but:

- Sometimes it is hard to make a binary decision on whether an image satisfies an attribute.- Comparisons are easier.

A B C

1. Introduction and MotivationRelative Attributes

- Semantically rich way to describe and compare objects in the world.- Indicate the strength of an attribute in an image with respect to other images.- Allow relating images and categories to each other.

A B C

A is more natural than B C less natural than B

2. Learning Relative

Attributes

2. Learning Relative AttributesModel Relative Attributes: Learn ranking function for each attribute

Set of training images: I = { i } represented by feature vectors { xi } and a set of M attributes A = { am }.

Set of ordered pair of images →

Set of un-ordered paris →

2. Learning Relative Attributes

2. Learning Relative Attributes

2. Learning Relative AttributesWide-margin binary classifier VS Wide-margin ranking function

3. Applications

3. Applications- Zero-Shot Learning: learn a classifier f : X → Y that must predict novel values of Y that

were omitted from the training set.

- Can predict new classes based on their relationships to existing classes – without training images.

- Describing images: Automatically generating relative textual descriptions of images.

3. Applications: Zero-Shot LearningConsidering the following set up:

- N total categories N = S + U- S seen categories:

- Associated images are available- Described relative to each other via attributes.

- U unseen categories: - No images are available for these categories- Described relative to seen categories for a subset of attributes.

3. Applications: Zero-Shot LearningTraining: Creating categories model

1. Train a set of relative attributes using S categories.a. Learn all M relative attribute am.

b. Predict real-valued rank for all images: each

image is represented by a M-dimensional vector indicating its rank score for all M attributes.

2. Build generative model (Gaussian) for each S category using the responses of the relative attributes.

3. Infer the parameters of the generative models of U categories by using their relative descriptions.

3. Applications: Zero-Shot LearningTest: classify new images

1. Compute i indicating its relative attribute ranking-scores for the image.

2. Assign it to the seen or unseen category that assigns it the highest likelihood:

3. Applications: Describing Images→ Goal: Automatically generating relative textual descriptions of images.

Given an image I to be described:

1. We evaluate all learnt ranking functions on I.

3. Applications: Describing Images

Given an image I to be described:

1. We evaluate all learnt ranking functions on I.

2. Identify 2 reference images from which image I will be described.

3. Applications: Describing Images

Given an image I to be described:

1. We evaluate all learnt ranking functions on I.

2. Identify 2 reference images from which image I will be described.

3. Can also describe an image relative to other categories.

→ Purely textual description

4. Experiments and Results

4. Experiments and Results:USED DATA BASES:

4. Experiments and Results:- The list of attributes used for each dataset:

4. Experiments and Results: Zero-Shot Learning- Experiment:

Compare zero-shot results to two baselines.

- Direct Attribute Prediction (DAP): uses binary attribute descriptions for all categories.

- Score-based Relative Attributes (SRA): uses scores of a binary classifier instead of a ranking function.

4. Experiments and Results: Zero-Shot Learning- Results:

4. Experiments and Results: Describing Images- Experiment:

4. Experiments and Results: Describing Images- Experiment:

4. Experiments and Results: Describing Images- Results: Relative descriptions are better than binary

5. Conclusions

5. Conclusions- Richer language of supervision and description than the commonly used binary attributes.

- Natural Descriptions: Leverages a natural mode of description.

- Flexibility: Allows use of as many attributes for defining relations among as many classes.

THANKS!¿ Any Question ?