second’internaonal’workshop’on’parts’and’a5ributes...

39
Second Interna,onal Workshop on Parts and A5ributes ECCV 2012, Firenze, Italy October, 2012 Discovering a Lexicon of Parts and Attributes Subhransu Maji Research Assistant Professor Toyota Technological Institute at Chicago

Upload: hoangdang

Post on 21-Apr-2018

218 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Second  Interna,onal  Workshop  on  Parts  and  A5ributesECCV  2012,  Firenze,  Italy

October,  2012Discovering a Lexicon of Parts and Attributes

Subhransu MajiResearch Assistant Professor

Toyota Technological Institute at Chicago

Page 2: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Motivation

• Detailed object recognition

• Communication requires a lexicon

Diverse Visual Categories

High%heel(Blue(Shoe(

Berg et al., 10 Farhadi et al, 09

CUB 200 dataset, Visipedia project

Page 3: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Source of part and attribute lexicons

• Field guides provide exhaustive lists when available

• Expert vs. Layman

• Task specific vs. not

Page 4: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Source of part and attribute lexicons

• Captioned images

Limited by sources of such text

Image from Berg et al., ECCV’10

Descriptions are often not visual

Page 5: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

What are good attribute lexicons?

• Properties

• It should be easy to communicate

• It should be easy to differentiate instances from one another

images from google, cub 200

Page 6: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Discriminative description task

Describe the (visual) differences between the two

Page 7: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Discriminative description task

Page 8: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Discriminative description task

!"#$

•  %&'()$•  *'"$)(+,()$•  -).$/0&0-$•  *'"$-1..)-$

!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$

$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$

-01(.$-1..)-$!"#$%0,(45$-1..)-$

!"#$%+",)')-.)#%

Description

Page 9: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Discriminative description task

!"#$

•  %&'()$•  *'"$)(+,()$•  -).$/0&0-$•  *'"$-1..)-$

!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$

$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$

-01(.$-1..)-$!"#$%0,(45$-1..)-$

!"#$%+",)')-.)#%

Description

!"#$

•  %&'()$•  *'"$)(+,()$•  -).$/0&0-$•  *'"$-1..)-$

!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$

$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$

-01(.$-1..)-$!"#$%0,(45$-1..)-$

!"#$%+",)')-.)#%

Discriminative Description

Helps elicit a lexicon that enables fine grained discriminationIs task specific by design

Page 10: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Discriminative description task

!"#$

•  %&'()$•  *'"$)(+,()$•  -).$/0&0-$•  *'"$-1..)-$

!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$

$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$

-01(.$-1..)-$!"#$%0,(45$-1..)-$

!"#$%+",)')-.)#%

Description

!"#$

•  %&'()$•  *'"$)(+,()$•  -).$/0&0-$•  *'"$-1..)-$

!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$

$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$

-01(.$-1..)-$!"#$%0,(45$-1..)-$

!"#$%+",)')-.)#%

Discriminative Description

Helps elicit a lexicon that enables fine grained discriminationIs task specific by design

Non-accidental properties, Biederman 87Pragmatics of Language, Levinson 83

Page 11: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Collecting descriptions on MTurkInterface on AMT

free form text(separated by ‘vs’)

Example annotations

Minimizes instruction biasKeeps the interface simple

Page 12: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Example annotations: airplanespair 82/999; 5 good

facing left facing rightturbofan powered plane propeller powered planelonger tail shorter tailgreen rudder white rudderpassenger door open baggage hold door open"

pair 65/999; 5 good

propeller to the body propeller to the wingone rudder two ruddersthin body fat bodylow wings high wingsfacing towards left side facing slightly towards"

Images from airliners.net

Page 13: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Example annotations: birdspair 10/1600; 5 good

black and white wings spotted wingswhite body spotted bodylarge eyes small eyessmall tail long tailv shaped beak pointed beak"

pair 43/1600; 5 good

yellow black body orange brown bodypointy beak shape beakshort tail long tailblack spot over head brown stripe over headshort leg long leg"

Images from CUB 200 dataset

Page 14: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Analyzing the text: instance specific properties

Different properties are revealed for different instances

bird familyfur color

beak shapetail size

beak color

beak sizetail size

wing colorhead colorleg color

beak colorsitting vs. flyingfeather color

tail sizeleg color

Page 15: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Analyzing the text: instance specific properties

Frequency of usage is a measure of its discriminability

Page 16: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Analyzing the text: instance specific properties

Frequency of usage is a measure of its discriminability

Page 17: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

red rudder vs. white rudderpointy nose vs. round nose

sentence pairs

Analyzing the text : discovering a lexicon

!"#$

•  %&'()$•  *'"$)(+,()$•  -).$/0&0-$•  *'"$-1..)-$

!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$

$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$

-01(.$-1..)-$!"#$%0,(45$-1..)-$

!"#$%+",)')-.)#%

Page 18: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

red rudder vs. white rudderpointy nose vs. round nose

sentence pairs

Analyzing the text : discovering a lexicon

!"#$

•  %&'()$•  *'"$)(+,()$•  -).$/0&0-$•  *'"$-1..)-$

!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$

$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$

-01(.$-1..)-$!"#$%0,(45$-1..)-$

!"#$%+",)')-.)#%

ruddernose

parts

{red, white}{pointy, round}

modifiers

red rudder vs. white rudderpointy nose vs. round nose

sentence pairs

Page 19: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

red rudder vs. white rudderpointy nose vs. round nose

sentence pairs

Analyzing the text : discovering a lexicon

!"#$

•  %&'()$•  *'"$)(+,()$•  -).$/0&0-$•  *'"$-1..)-$

!"#$%&'(&)'*)#%$%-0%)&&)-$%&'()$!"#$%'"")(+)-$%&'()$

$$0()$)(+,()$!"#$201-$)(+,()"$$$$$-).$/0&0-$!"#$3*,4)$/0&0-$

-01(.$-1..)-$!"#$%0,(45$-1..)-$

!"#$%+",)')-.)#%

ruddernose

parts

{red, white}{pointy, round}

modifiers

red rudder vs. white rudderpointy nose vs. round nose

sentence pairs

Key ideanouns : words that repeatmodifiers : words that are differentEach sentence has only one noun and modifier

Page 20: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Analyzing the text : discovering a lexicon

Sentence alignment

yellowbeak

blackandwhitebeak

Used in NLP to initialize translation tables (IBM models)

Page 21: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Analyzing the text : discovering a lexicon

Sentence alignment

yellowbeak

blackandwhitebeak

Used in NLP to initialize translation tables (IBM models)

GoalsDiscover a lexicon of parts, modifiers and part-modifier relations

Modifiers should be shared across attributes Estimate the frequency of each attribute

Page 22: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

A generative model of sentence pairs

Page 23: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

A generative model of sentence pairs

z

a

f

e

N

J

tI

⌦ �

Page 24: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

A generative model of sentence pairs

z

a

f

e

N

J

tI

⌦ �

yellow beak vs.

black and white beak

Page 25: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

A generative model of sentence pairs

z

a

f

e

N

J

tI

⌦ �

yellow beak vs.

black and white beak

beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...

part-modifier topic pairs

Page 26: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

A generative model of sentence pairs

zbeak-color

z

a

f

e

N

J

tI

⌦ �

yellow beak vs.

black and white beak

beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...

part-modifier topic pairs

Page 27: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

A generative model of sentence pairs

zbeak-color

ttopic:colortopic:beak

z

a

f

e

N

J

tI

⌦ �

yellow beak vs.

black and white beak

beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...

part-modifier topic pairs

Page 28: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

A generative model of sentence pairs

yellowbeak

NULL

ezbeak-color

ttopic:colortopic:beak

z

a

f

e

N

J

tI

⌦ �

yellow beak vs.

black and white beak

beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...

part-modifier topic pairs

Page 29: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

A generative model of sentence pairs

yellowbeak

NULL

ezbeak-color

ttopic:colortopic:beak

z

a

f

e

N

J

tI

⌦ �

yellow beak vs.

black and white beak

beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...

part-modifier topic pairs a

Page 30: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

A generative model of sentence pairs

yellowbeak

NULL

eblackandwhitebeak

fzbeak-color

ttopic:colortopic:beak

z

a

f

e

N

J

tI

⌦ �

yellow beak vs.

black and white beak

beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...

part-modifier topic pairs a

Page 31: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

A generative model of sentence pairs

yellowbeak

NULL

eblackandwhitebeak

fzbeak-color

ttopic:colortopic:beak

z

a

f

e

N

J

tI

⌦ �

yellow beak vs.

black and white beak

beak-sizebeak-colorbird-sizewing-colorbird-kindleg-colorhead-color...

part-modifier topic pairs a

Initialize part and modifier topics using word alignments

Page 32: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Parts, modifiers and attributes of airplanes

wheelwheels plane

engineengines rudder

wingswing

frontback nose facing body tail GLOBAL

onetwono

singlethreedoublefour

colorblackskylight

whiteblueordinarycoloreddark

whitegreenwhitered

pointyroundflat

pointedsharppointsquare

propellerpassenger

jetonly

militarycargo

smallbiglarge

medium

whiteredbluegreenyellowgrayorangebrown

topbottommiddledown

openclosedopenedclose

rightleft

slightly

onnearoff

200 images, 1000 random pairsImages from airliners.net

Page 33: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Parts, modifiers and attributes of airplanes

wheelwheels plane

engineengines rudder

wingswing

frontback nose facing body tail GLOBAL

onetwono

singlethreedoublefour

colorblackskylight

whiteblueordinarycoloreddark

whitegreenwhitered

pointyroundflat

pointedsharppointsquare

propellerpassenger

jetonly

militarycargo

smallbiglarge

medium

whiteredbluegreenyellowgrayorangebrown

topbottommiddledown

openclosedopenedclose

rightleft

slightly

onnearoff

200 images, 1000 random pairsImages from airliners.net

1

Page 34: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Parts, modifiers and attributes of airplanes

wheelwheels plane

engineengines rudder

wingswing

frontback nose facing body tail GLOBAL

onetwono

singlethreedoublefour

colorblackskylight

whiteblueordinarycoloreddark

whitegreenwhitered

pointyroundflat

pointedsharppointsquare

propellerpassenger

jetonly

militarycargo

smallbiglarge

medium

whiteredbluegreenyellowgrayorangebrown

topbottommiddledown

openclosedopenedclose

rightleft

slightly

onnearoff

200 images, 1000 random pairsImages from airliners.net

12

Page 35: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Parts, modifiers and attributes of airplanes

wheelwheels plane

engineengines rudder

wingswing

frontback nose facing body tail GLOBAL

onetwono

singlethreedoublefour

colorblackskylight

whiteblueordinarycoloreddark

whitegreenwhitered

pointyroundflat

pointedsharppointsquare

propellerpassenger

jetonly

militarycargo

smallbiglarge

medium

whiteredbluegreenyellowgrayorangebrown

topbottommiddledown

openclosedopenedclose

rightleft

slightly

onnearoff

200 images, 1000 random pairsImages from airliners.net

12 3

Page 36: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Parts, modifiers and attributes of airplanes

wheelwheels plane

engineengines rudder

wingswing

frontback nose facing body tail GLOBAL

onetwono

singlethreedoublefour

colorblackskylight

whiteblueordinarycoloreddark

whitegreenwhitered

pointyroundflat

pointedsharppointsquare

propellerpassenger

jetonly

militarycargo

smallbiglarge

medium

whiteredbluegreenyellowgrayorangebrown

topbottommiddledown

openclosedopenedclose

rightleft

slightly

onnearoff

200 images, 1000 random pairsImages from airliners.net

12 3localized vs. global

Page 37: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Parts, modifiers and attributes of birds

bird wingsfeatherfeathers tail beak like body leg legs eyes neck head in fur GLOBAL

longshortsmalllargebig

vpointypointedpointbendbendedpointlyslightly

sparrowducksparowcroweagledovepigeonhumming

kiteparrot

brownblueyellowgrayredgreenspottedashlight

blackwhiteorange

fatslimsilmlean

sharproundflat

normalshapedcurvedbluntlittle

roundedordinaryoval

200 images, 1600 pairs1 image per category

CUB 200 dataset

beak-size, wing-color, tail-size, body color, bird type

Page 38: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Parts, modifiers and attributes of people

handhands hair facing

snapsnape

thepicture spectacles in glass bag watch glasses GLOBAL

towardsleftright

backwardsforwardsidewards

manlady

womanboygirlladies

babyadultchildkid

childrenadults

shirttshirtjacketdresscoattshirts

wearingnot

smilinghaving

asiancaucasianafricansasianslatin

western

sidebackfront

backsidefrontaltowardturnturnedrear

sidewaysuprightus

blackwhitebluebrownblonderedgreengrayyellowcoloredpinkorange

singlecouple2non

doublegroupcouplesmany3

darklightfairteeskyshowbrightlighterdesignmediumthick

fatslim

normalthinleanfit

averageskinny

femalemale

fullhalfof

sleeveonlyfully

sleevelessjeensclosepartialrain

thinningtorsowaist

indooroutdoordoorhomelystage

insideoutsideout

handed

longshortsmallshot

womenmen

womens

youngoldoldermiddlematureelderly

twoonethreeboth

somebodyweight

sittingstandingwalkingridingcyclingsleepingdancingdriving

nowith

withoutalonebald

400 images, 1600 pairsRandom images from

PASCAL VOC 11

location, shirt vs tshirt, hair length, gender

Page 39: Second’Internaonal’Workshop’on’Parts’and’A5ributes …ttic.uchicago.edu/~smaji/presentations/lexicon-discovery-eccv2012.pdf · one rudder two rudders thin body fat body

Conclusions

• Discriminative description is an effective way to elicit a lexicon of attributes that are useful for fine-grained distinction

• Simple analysis of sentence pairs can help discover

• a lexicon of parts

• a lexicon of modifiers

• relationships between parts and modifiers (attributes)

• the relative frequency of these in a dataset

• A useful tool to bootstrap annotation collection process