Annotation Propagation in Large Image Databases viaDense Image Correspondence

Supplemental Material

1 Additional Results

Here we show more results of our system on the datasets we experimented with. De-scription of the datasets and the experiment setups are given in Section 5 in the paper.We recommend viewing the results electronically and zoom in for more details. No-tice that some figures span more than one page. We indexed the figures in a grid toallow referencing particular results. We note that all the results are for images that areoriginally untagged and unlabeled in the database (the subset of images I \ It).

In Fig. 1 we show more results on LabelMe Outdoors (LMO) dataset [1]. In additionto the final result, we also show the MAP labels based on local evidence alone (theappearance model in Section 3, Eqn. 2), similar to Fig. 3(c) in the paper.

Fig. 2 shows the estimated spatial prior of each word (Eqn. 7) in LMO’s vocabulary.It can be seen that the prior agrees with the true spatial prior, computed from the groundtruth labels, for more frequent words. The estimated prior is somewhat blurrier than theground truth, indicating some errors in classification, however the general layout iscaptured correctly. For example, sky is mostly at the top of the image, building is in themiddle, and road and sea are at the bottom.

Fig. 3 shows more results on SUN dataset [2], as well as comparison with the resultsby [3], similar to Fig. 8 in the paper. The results of [3] were produced using the authors’original implementation (available online), modified by us to account for tagged imagesas described in Section 3 in their paper (termed “weak supervision”). Taken togetherwith Fig. 1, these results show that the algorithm can handle large variety of both indoorand outdoor scenes. Notice that while SUN has a relatively large vocabulary (500+words), the tags inferred by the algorithm tend to correspond to words with higherfrequency in the dataset. That is because words that occur frequently, and co-occurfrequently with other words, are considered more probable by the algorithm (Eqn. 3).

Fig. 4 and 5 show more results on the ESP game dataset [4] and IAPR bench-mark [5], where we used the same images and vocabulary as in [6] (available online).These two datasets are much noisier in terms of both image content and vocabulary,and so are more challenging for the algorithm. In particular, both datasets include moreabstract words (e.g. smile, night) that are harder to model, as well as words that mightnot correspond to a particular image region (e.g. photo).

Finally, Fig. 6 shows more failure cases on all datasets. Limitations of the system in-clude incorrect classification under similar visual appearance or insufficient exemplarsof particular words (e.g. row 1 columns 2-3, row 3 columns 1,3 in (a), row 1 column 1 in(b), row 2 columns 2-3 in (c)), and errors due to incorrect inter-image correspondence(e.g. row 1 column 1, row 5 columns 1,3 in (a), row 2 column 3 in (b), row 1 column 1in (c)).

2

1 2

1

mountain

plant

river

sky

tree

mountain

plant

river

sky

tree

building

mountain

sea

sky

tree

building

mountain

sea

sky

tree

2

building

mountain

plant

sea

sky

building

mountain

plant

sea

sky

building

mountain

sea

sky

tree

building

sea

sky

tree

3

building

person

sand

sea

sky

building

sand

sea

sky

river

sky

tree

unlabeled

river

sky

tree

4

building

grass

mountain

sky

tree

building

grass

mountain

sky

tree

grass

person

sky

tree

grass

person

sky

tree

5person

plant

rock

sky

tree

unlabeled

person

plant

sky

tree

grass

mountain

plant

sky

tree

grass

mountain

plant

sky

tree

6

plant

river

rock

sky

tree

plant

river

rock

sky

tree

mountain

river

sky

tree

unlabeled

mountain

river

sky

tree

7

mountain

plant

river

sky

tree

mountain

plant

sky

tree

building

mountain

person

sky

tree

building

mountain

sky

tree

8

mountain

person

river

sky

tree

mountain

person

river

sky

tree

plant

river

sky

tree

unlabeled

plant

river

sky

tree

9building

car

crosswalk

road

sky

streetlight

building

car

road

sky

car

road

sign

sky

streetlight

tree

unlabeled

car

road

sign

sky

tree

10

car

road

sign

sky

tree

car

road

sign

sky

tree

car

road

sign

sky

tree

car

road

sign

sky

tree

Source Appearance model Result Source Appearance model Result

Title Suppressed Due to Excessive Length 3

1 2

11

car

road

sign

sky

tree

unlabeled

car

road

sign

sky

tree

car

road

sign

sky

tree

car

road

sign

sky

tree

12car

mountain

road

sky

tree

unlabeled

car

mountain

road

sky

tree

building

car

road

sidewalk

sky

unlabeled

building

road

sidewalk

sky

13

building

door

sky

tree

window

building

door

sky

tree

window

building

grass

sky

window

unlabeled

building

grass

sky

window

14building

door

road

sidewalk

window

unlabeled

building

door

road

sidewalk

window

building

door

road

sky

window

unlabeled

building

door

road

sky

window

15

building

door

road

sidewalk

window

building

door

road

sidewalk

window

building

car

road

sidewalk

tree

unlabeled

building

car

road

sidewalk

tree

16

building

car

road

sky

tree

building

car

road

sky

tree

building

car

road

sidewalk

sky

building

car

road

sidewalk

sky

17awning

building

car

tree

window

unlabeled

awning

building

car

tree

window

mountain

road

sky

tree

mountain

road

sky

tree

18

mountain

river

sky

tree

mountain

river

sky

tree

bridge

building

car

person

road

sidewalk

sky

tree

bridge

building

car

person

road

sidewalk

sky

tree

19 building

car

mountain

person

road

sidewalk

sky

streetlight

tree

building

car

mountain

road

sidewalk

sky

tree

building

sky

tree

unlabeled

building

sky

tree

Source Appearance model Result Source Appearance model Result

Fig. 1. More results on LMO. For each example, we show the source image on the left, the MAPlabeling using the appearance model only in the middle (computed independently at each pixel;see Fig. 3 in the paper), and the final result of the annotation propagation algorithm (appearancemodel + spatial regularization + regularization via dense image correspondences) on the right.Note that the final result might not contain all tags from the appearance model result.

4

Estimated Ground truth

sky building mountain tree road

sea field grass river plant

car sand rock sidewalk window

desert door bridge person fence

balcony staircase awning crosswalk

sign streetlight boat pole

bus sun cow bird

moon

Fig. 2. The estimated spatial prior hsl (Eqn. 7) for the LMO vocabulary. Words are ordered

from top left to bottom right according to their frequency in the dataset. For each word, the leftimage is the estimated prior and the right image is the true prior according to human labels. Thecolormap is the same as Fig. 1 above, with saturation corresponding to probability, from white(zero probability) to saturated (high probability).

Title Suppressed Due to Excessive Length 5

1 2

1

ceiling

ceiling lamp

door

floor

wall

balconybarsbathtubboatbottleboxbulletin boardcabinetceilingceiling lampchairchandeliercountercrosswalkcuddly toycurtaindesk lampdresserentranceexercise machine

curtain

floor

person

shelves

arcadebagbalconybookbottlebowlbridgebulletin boardcandleceilingceiling lampchaircolumncountercubiclecushiondesk lampdoorentranceextractor hood

2

ceiling lamp

floor

wall

window

bedbenchbleachersbuildingcarceilingcentral reservationcliffcountertopcupboardcurtaindeck chairfloorlarge windowmachinemicrowavemountainoutletpaintingperson

floor

wall

window

balustradeboatbridgebuildingbulletin boardcarchestcliffcupboardcurtaincushiondesk lampdoorextractor hoodfaucetfencefountainglass wallgroundhandrail

3

curtain

floor

person

shelves

airplanealtarpiecearcadeawningbagbarrelbenchbleachersboatbottleboxbuildingbulletin boardcabinetcarceilingceiling lampchairclockcourt

ceiling

ceiling lamp

door

floor

wall

animalarmchairawningbleachersboatbookbridgebuildingcabinetcarceilingceiling lampcloudcoffee makerconveyor beltcurtaindeck chairflagfloorflowers

4

ceiling

floor

plant

wall

window

applearcadearmchairbarrelbasketbeambedbenchbleachersbookboxcabinetcanceilingceiling lampchaircolumncountercuddly toycup

bed

ceiling

floor

wall

window

altarbagbasketbedbellbookbottlebowlbuildingbulletin boardcabinetcanceilingceiling lampchairchandelierchimneycloudcountercountertop

5

ceiling lamp

floor

table

wall

window

awningbagbasketbathtubbedbenchbilliard tablebookbottlebowlboxbreadbulletin boardcanceilingceiling lampchairchandelierchestcountertop

floor

wall

window

armchairawningbalconybasketbathtubbicyclebookbowlboxbrand namebuildingcabinetcandleceiling lampchairchandeliercloudcolumncurbcurtain

6

ceiling

floor

wall

window

alarm clockapplearmchairbalconybicyclebrand namebuildingcardceilingceiling lampchaircolumncountercurtaindesk lampdoordrawergroundmicrowavemirror

table

wall

window

alarm clockbagbalconybarrelbarsbedbellboatbookbowlboxbrand namebreadbridgebucketbuildingcabinetcancandiescandle

7

curtain

floor

person

shelves

air conditioningaltarpiecebagbalconybasketbathtubbedbenchboxcabinetcarceilingceiling lampcountercurtaindrawerelevatorextractor hoodfieldfiles

ground

sky

tree

water

airplaneawningbasketbedbenchbicyclebottleboxbuildingbulletin boardcandlecarcentral reservationchairclothescountercrosswalkdesk lampdoorelephant

8

grass

plant

sky

tree

deck chairfieldfloegrassmountainpathpitchplantpoleseasheepshop windowskyswimming pooltreewall

building

mountain

sky

tree

air conditioningawningbasketbenchbrand namebuildingcabinetcarcaravanceilingclockcloudcoffee makercolumncupboardcurtaindesk lampdolldomedoor

9

building

mountain

road

sky

window

altarbackpackbagbarbecuebarsbasketbathtubbeambenchboardsboatboxbrand namebuildingcabinetcarcardceilingcentral reservationcloud

building

mountain

sky

tree

arcadebarrelbucketbuildingchaircourtcrosswalkdirt trackdoorelephantembankmentfieldfilesgrassgroundlandmagazinesmountainpathperson

10

building

mountain

sky

tree

window

armchairawningbenchbicycleboardsboatbottleboxbucketbuildingcabinetcarceilingchaircitycountercountertopcrosswalkcubiclecurtain

building

grass

sky

tree

balconyblanketbottlebrand namebuildingcabinetcarceiling lampchairchimneycloudcolumncountercountertopcurtaindomedoorelephantfencefloor

Source AP (ours) STF Source AP (ours) STF

6

1 2

11

ceiling

floor

table

wall

armchairbagbarbasketbicycleboatbookbookcasebottleboxbreadbridgebuildingbulletin boardcancarceilingceiling lampchairchandelier

ceiling

ceiling lamp

door

floor

wall

air conditioningarmchairbarsbenchboxbridgebuildingbulletin boardcabinetcarceilingceiling lampconveyor beltcrosscushiondoorfencefloorglass wallground

12

building

road

sky

tree

barrelboatbookbuildingbulletin boardcarcourtcupboarddirt trackdoorfencefieldfloorgrassgroundlandmountainpathplantroad

mountain

road

sky

tree

bulletin boardcabinetcarconveyor beltdoorfountainglass wallladderledgemachinerymountainpersonpipepolerailingriverroadrugseaseparation

13building

door

road

sidewalk

tree

window

air conditioninganimalawningbagbalconyballbarboatboxbridgebucketbuildingbulletin boardcabinetcarceilingceiling lampchairclosetcolumn

building

grass

sky

tree

window

billboardbookboxbucketbuildingcarceiling lampchairdoordummyfaucetfencefieldflowersgrassmountainpaintingpathpersonpitch

14

mountain

sea

sky

fish

mountain

river

sea

sky

swimming pool

ceiling

drawer

stove

tray

wall

bagbalconybenchbicyclebreadcarceilingceiling lampcentral reservationclockcloudcountercurtaincutleryfencefile cabinetfloorframegarage doorglass wall

15

building

mountain

road

sky

carconveyor beltembankmentfaucetfieldglass wallmountainplantpolerailingrefrigeratorriverroadsandseaseparationsignskysnowsnowy ground

floor

plant

wall

window

armchairawningbagbalconybedbenchboatbookbottleboxbuildingcancarceilingceiling lampchairclosetcolumncurtaindecoration

16

building

door

tree

wall

window

awningbagbasketbedbenchbookbowlbuildingbulletin boardcabinetcanceilingceiling lampchairchandeliercloudcountercutlerydesk lampdome

building

mountain

sky

tree

animalarmchairbalustradebarsbedbellboatbookbookcasebridgebulletin boardcarchestcolumncowdecorationdirt trackelephantfencefield

17

mountain

sky

tree

beam

floe

mountain

river

sea

sky

swimming pool

grass

sky

tree

armchairbuildingceilingconveyor beltcurtainembankmentfieldglass wallgrasslandmountainplantrocksheepsignskytreewater

18

building

grass

person

sky

tree

animalbenchboardsboatcabinetdeck chairgrapesgrasshay rollleafpathplantskytoastertree

building

mountain

sky

tree

bagbathtubbellbenchboatbookbucketbuildingbulletin boardcabinetcanchairchestclothescrossextractor hoodfencefloorflowersgate

19

building

mountain

sky

tree

boatbuildingcandlecaravandirt trackembankmentfloefloormountainpersonpianopillowrockskysnowstagestreetlightteddy beartowertoy

building

mountain

sky

animalbarsbedbenchbleachersboatbuildingcarcardceiling lampcentral reservationcloudcolumncrosscutlerydesk lampdomedoordrawerfence

20

floor

person

wall

balconybridgebuildingbulletin boardceilingceiling lampcolumncowdecorationextractor hoodfieldfloorfluorescent tubehandrailjarmirrorpaintingplantsinksky

ceiling

grass

monitor

person

sky

ceiling

grass

machine

sky

tree

Source AP (ours) STF Source AP (ours) STF

Title Suppressed Due to Excessive Length 7

1 2

21

building

car

road

sky

awningbalconyboxbrand namebuildingcarceilingceiling lampchaircoffee makercupboardcurtaindeck chairdesk lampeaselfenceflagflowersgroundmirror

building

road

sky

tree

armchairbalconybleachersbrand namebuildingcarcrossdomedooreaselfencehatheatermezzaninemirrorpaintingpianoprinterroadsand

22

building

road

sky

tree

animalarmchairbagbedbleachersboatbottleboxbrand namebuildingcarceilingceiling lampcentral reservationchairchestclosetclothescuddly toycurtain

ceiling

floor

wall

window

balconybasketbeambookboxbreadceilingceiling lampchairclosetcountercupcurtaindishdrawerdummyfanflagfloorfluorescent tube

23

door

floor

plant

wall

air conditioningbarbarrelbarsbenchbuildingbulletin boardcarceilingceiling lampchaircolumncubicledeckdirt trackdoor knobfencefieldfile cabinetfloor

building

person

road

sky

tree

appleawningbagbasketbenchbleachersbrand namebridgebuildingcarceilingcountercrosscupextractor hoodfaucetflaggroundice rinkmachine

24

building

grass

sky

tree

animalboatdeckdirt trackfencegrapesgrassgroundhay rollpathpersonplantpolerockscreensculptureskystepsstreetlighttree

building

mountain

sky

tree

bicyclebottlebreadbuildingcurtaindummyembankmentfencefishgarage doorguitarmezzaninemountainpersonpianoposterprojection screenroadrocking chairscaffolding

25

building

sky

tree

window

air conditioningaltaranimalawningbagbarsbedbellbleachersbookboxbridgebucketbuildingbulletin boardcabinetcanceilingceiling lampchair

building

ground

sky

tree

buildingcaravanclouddoorelephantfencefieldgrandstandgrassmountainpathpersonplantskytree

26

building

sky

tree

window

air conditioningalarm clockatticawningbalconybasketboatbowlboxbrand namebuildingcabinetcarceiling lampchaircloudcountercupboardcurtaindesk lamp

building

sky

tree

window

bedbilliard tableblanketboardsboatbuildingceilingdoorfencefloefountaingategroundhatmountainpersonplateaupoleprojection screenrefrigerator

27

building

mountain

sea

sky

bagbasketbedbenchbilliard tablebookbowlboxbuildingbulletin boardcabinetcarceiling lampchaircloudcountercupboarddishdoor knobdrawer

building

road

sky

tree

applearmchairawningbagbarsboatboxbreadbuildingbulletin boardcabinetcarcaravanceiling lampchestcowdesk lampdomedrawerelephant

Source AP (ours) STF Source AP (ours) STF

Fig. 3. More results on SUN, and comparison with semantic texton forests (STF) [3]. For someimages, STF assigned too many tags for a clear visualization, and so we limited the legends to 20words (omitting any excess words; for visualization purposes only).

8

1 2 3

1building

church

old

sky

tree

eye

face

girl

hair

woman

light

sky

sun

tree

2flower

green

plant

picture

rectangle

square

white

woman

grass

sky

tree

3

black

circle

plate

red

round

black

face

hair

man

woman

grass

green

house

tree

4

coin

gold

gray

money

silver

flower

grass

green

man

sky

black

face

hat

man

smile

5blue

chart

graph

man

red

round

square

white

yellow

lake

mountain

sky

tree

water

6cloud

ocean

sea

sky

water

city

mountain

sky

street

tower

girl

hair

swim

water

woman

7building

light

tree

white

blue

people

picture

sky

tree

black

eye

hair

man

music

8

flower

grass

lake

tree

water

diagram

drawing

green

white

black

hair

man

picture

Fig. 4. More results on ESP.

Title Suppressed Due to Excessive Length 9

1 2 3

1house

mountain

range

sky

tree

house

sky

statue

tree

water

mountain

river

road

side

tree

2bush

helmet

meadow

people

tree

desert

front

mountain

rock

tourist

bush

house

road

sea

sky

3grass

hill

man

sky

sweater

front

sky

tower

tree

window

canyon

landscape

lookout

road

sky

4chair

forest

front

garden

tourist

beach

palm

sea

sky

water

bush

mountain

people

slope

tree

5lamp

night

square

street

tower

city

cloud

house

sky

view

desert

mountain

people

rock

sky

6front

group

lake

people

sky

forest

front

gravel

road

sign

entrance

front

stone

tourist

wall

7face

rock

sky

stage

waterfall

mountain

rock

sky

slope

woman

building

house

mountain

slope

view

8horizon

sky

sun

tree

front

man

mountain

sky

tree

cloud

man

rock

sky

tree

Fig. 5. More results on IAPR.

10

1 2 3

1building

mountain

road

sky

tree

mountain

rock

sand

sea

sky

mountain

rock

sand

sea

sky

2car

mountain

road

sky

tree

building

field

plant

sky

tree

grass

plant

river

sky

tree

3building

mountain

river

sky

tree

building

mountain

road

sky

tree

building

car

mountain

sky

tree

4building

door

road

sky

building

ground

tree

window

grass

mountain

sky

tree

5building

door

road

sky

window

building

mountain

sea

sky

tree

building

road

sky

(a) SUN (and LMO)

1green

man

people

sky

tree

black

ear

eye

girl

smile

forest

green

rock

sky

tree

2

flower

orange

red

sun

white

circlecoincrossgoldround

night

sky

space

star

tree

(b) ESP

1hat

house

man

sky

woman

cloud

dune

sand

sea

sky

airport

building

plane

sky

tree

2chair

cloud

hill

sky

tree

desert

man

middle

people

sky

lamp

room

table

wall

window

(c) IAPR

Fig. 6. More failure cases on SUN, ESP and IAPR.

Title Suppressed Due to Excessive Length 11

References

1. Russell, B., Torralba, A., Murphy, K., Freeman, W.: Labelme: a database and web-based toolfor image annotation. IJCV 77 (2008) 157–173

2. Xiao, J., Hays, J., Ehinger, K., Oliva, A., Torralba, A.: Sun database: Large-scale scene recog-nition from abbey to zoo. In: CVPR. (2010) 3485–3492

3. Shotton, J., Johnson, M., Cipolla, R.: Semantic texton forests for image categorization andsegmentation. In: CVPR. (2008)

4. Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: SIGCHI. (2004)5. Grubinger, M., Clough, P., Muller, H., Deselaers, T.: The iapr benchmark: A new evaluation

resource for visual information systems. In: LREC. (2006) 13–236. Makadia, A., Pavlovic, V., Kumar, S.: Baselines for image annotation. IJCV 90 (2010)