Annotation Propagation in Large Image Databases viaDense Image Correspondence
Supplemental Material
1 Additional Results
Here we show more results of our system on the datasets we experimented with. De-scription of the datasets and the experiment setups are given in Section 5 in the paper.We recommend viewing the results electronically and zoom in for more details. No-tice that some figures span more than one page. We indexed the figures in a grid toallow referencing particular results. We note that all the results are for images that areoriginally untagged and unlabeled in the database (the subset of images I \ It).
In Fig. 1 we show more results on LabelMe Outdoors (LMO) dataset [1]. In additionto the final result, we also show the MAP labels based on local evidence alone (theappearance model in Section 3, Eqn. 2), similar to Fig. 3(c) in the paper.
Fig. 2 shows the estimated spatial prior of each word (Eqn. 7) in LMO’s vocabulary.It can be seen that the prior agrees with the true spatial prior, computed from the groundtruth labels, for more frequent words. The estimated prior is somewhat blurrier than theground truth, indicating some errors in classification, however the general layout iscaptured correctly. For example, sky is mostly at the top of the image, building is in themiddle, and road and sea are at the bottom.
Fig. 3 shows more results on SUN dataset [2], as well as comparison with the resultsby [3], similar to Fig. 8 in the paper. The results of [3] were produced using the authors’original implementation (available online), modified by us to account for tagged imagesas described in Section 3 in their paper (termed “weak supervision”). Taken togetherwith Fig. 1, these results show that the algorithm can handle large variety of both indoorand outdoor scenes. Notice that while SUN has a relatively large vocabulary (500+words), the tags inferred by the algorithm tend to correspond to words with higherfrequency in the dataset. That is because words that occur frequently, and co-occurfrequently with other words, are considered more probable by the algorithm (Eqn. 3).
Fig. 4 and 5 show more results on the ESP game dataset [4] and IAPR bench-mark [5], where we used the same images and vocabulary as in [6] (available online).These two datasets are much noisier in terms of both image content and vocabulary,and so are more challenging for the algorithm. In particular, both datasets include moreabstract words (e.g. smile, night) that are harder to model, as well as words that mightnot correspond to a particular image region (e.g. photo).
Finally, Fig. 6 shows more failure cases on all datasets. Limitations of the system in-clude incorrect classification under similar visual appearance or insufficient exemplarsof particular words (e.g. row 1 columns 2-3, row 3 columns 1,3 in (a), row 1 column 1 in(b), row 2 columns 2-3 in (c)), and errors due to incorrect inter-image correspondence(e.g. row 1 column 1, row 5 columns 1,3 in (a), row 2 column 3 in (b), row 1 column 1in (c)).
2
1 2
1
mountain
plant
river
sky
tree
mountain
plant
river
sky
tree
building
mountain
sea
sky
tree
building
mountain
sea
sky
tree
2
building
mountain
plant
sea
sky
building
mountain
plant
sea
sky
building
mountain
sea
sky
tree
building
sea
sky
tree
3
building
person
sand
sea
sky
building
sand
sea
sky
river
sky
tree
unlabeled
river
sky
tree
4
building
grass
mountain
sky
tree
building
grass
mountain
sky
tree
grass
person
sky
tree
grass
person
sky
tree
5person
plant
rock
sky
tree
unlabeled
person
plant
sky
tree
grass
mountain
plant
sky
tree
grass
mountain
plant
sky
tree
6
plant
river
rock
sky
tree
plant
river
rock
sky
tree
mountain
river
sky
tree
unlabeled
mountain
river
sky
tree
7
mountain
plant
river
sky
tree
mountain
plant
sky
tree
building
mountain
person
sky
tree
building
mountain
sky
tree
8
mountain
person
river
sky
tree
mountain
person
river
sky
tree
plant
river
sky
tree
unlabeled
plant
river
sky
tree
9building
car
crosswalk
road
sky
streetlight
building
car
road
sky
car
road
sign
sky
streetlight
tree
unlabeled
car
road
sign
sky
tree
10
car
road
sign
sky
tree
car
road
sign
sky
tree
car
road
sign
sky
tree
car
road
sign
sky
tree
Source Appearance model Result Source Appearance model Result
Title Suppressed Due to Excessive Length 3
1 2
11
car
road
sign
sky
tree
unlabeled
car
road
sign
sky
tree
car
road
sign
sky
tree
car
road
sign
sky
tree
12car
mountain
road
sky
tree
unlabeled
car
mountain
road
sky
tree
building
car
road
sidewalk
sky
unlabeled
building
road
sidewalk
sky
13
building
door
sky
tree
window
building
door
sky
tree
window
building
grass
sky
window
unlabeled
building
grass
sky
window
14building
door
road
sidewalk
window
unlabeled
building
door
road
sidewalk
window
building
door
road
sky
window
unlabeled
building
door
road
sky
window
15
building
door
road
sidewalk
window
building
door
road
sidewalk
window
building
car
road
sidewalk
tree
unlabeled
building
car
road
sidewalk
tree
16
building
car
road
sky
tree
building
car
road
sky
tree
building
car
road
sidewalk
sky
building
car
road
sidewalk
sky
17awning
building
car
tree
window
unlabeled
awning
building
car
tree
window
mountain
road
sky
tree
mountain
road
sky
tree
18
mountain
river
sky
tree
mountain
river
sky
tree
bridge
building
car
person
road
sidewalk
sky
tree
bridge
building
car
person
road
sidewalk
sky
tree
19 building
car
mountain
person
road
sidewalk
sky
streetlight
tree
building
car
mountain
road
sidewalk
sky
tree
building
sky
tree
unlabeled
building
sky
tree
Source Appearance model Result Source Appearance model Result
Fig. 1. More results on LMO. For each example, we show the source image on the left, the MAPlabeling using the appearance model only in the middle (computed independently at each pixel;see Fig. 3 in the paper), and the final result of the annotation propagation algorithm (appearancemodel + spatial regularization + regularization via dense image correspondences) on the right.Note that the final result might not contain all tags from the appearance model result.
4
Estimated Ground truth
sky building mountain tree road
sea field grass river plant
car sand rock sidewalk window
desert door bridge person fence
balcony staircase awning crosswalk
sign streetlight boat pole
bus sun cow bird
moon
Fig. 2. The estimated spatial prior hsl (Eqn. 7) for the LMO vocabulary. Words are ordered
from top left to bottom right according to their frequency in the dataset. For each word, the leftimage is the estimated prior and the right image is the true prior according to human labels. Thecolormap is the same as Fig. 1 above, with saturation corresponding to probability, from white(zero probability) to saturated (high probability).
Title Suppressed Due to Excessive Length 5
1 2
1
ceiling
ceiling lamp
door
floor
wall
balconybarsbathtubboatbottleboxbulletin boardcabinetceilingceiling lampchairchandeliercountercrosswalkcuddly toycurtaindesk lampdresserentranceexercise machine
curtain
floor
person
shelves
arcadebagbalconybookbottlebowlbridgebulletin boardcandleceilingceiling lampchaircolumncountercubiclecushiondesk lampdoorentranceextractor hood
2
ceiling lamp
floor
wall
window
bedbenchbleachersbuildingcarceilingcentral reservationcliffcountertopcupboardcurtaindeck chairfloorlarge windowmachinemicrowavemountainoutletpaintingperson
floor
wall
window
balustradeboatbridgebuildingbulletin boardcarchestcliffcupboardcurtaincushiondesk lampdoorextractor hoodfaucetfencefountainglass wallgroundhandrail
3
curtain
floor
person
shelves
airplanealtarpiecearcadeawningbagbarrelbenchbleachersboatbottleboxbuildingbulletin boardcabinetcarceilingceiling lampchairclockcourt
ceiling
ceiling lamp
door
floor
wall
animalarmchairawningbleachersboatbookbridgebuildingcabinetcarceilingceiling lampcloudcoffee makerconveyor beltcurtaindeck chairflagfloorflowers
4
ceiling
floor
plant
wall
window
applearcadearmchairbarrelbasketbeambedbenchbleachersbookboxcabinetcanceilingceiling lampchaircolumncountercuddly toycup
bed
ceiling
floor
wall
window
altarbagbasketbedbellbookbottlebowlbuildingbulletin boardcabinetcanceilingceiling lampchairchandelierchimneycloudcountercountertop
5
ceiling lamp
floor
table
wall
window
awningbagbasketbathtubbedbenchbilliard tablebookbottlebowlboxbreadbulletin boardcanceilingceiling lampchairchandelierchestcountertop
floor
wall
window
armchairawningbalconybasketbathtubbicyclebookbowlboxbrand namebuildingcabinetcandleceiling lampchairchandeliercloudcolumncurbcurtain
6
ceiling
floor
wall
window
alarm clockapplearmchairbalconybicyclebrand namebuildingcardceilingceiling lampchaircolumncountercurtaindesk lampdoordrawergroundmicrowavemirror
table
wall
window
alarm clockbagbalconybarrelbarsbedbellboatbookbowlboxbrand namebreadbridgebucketbuildingcabinetcancandiescandle
7
curtain
floor
person
shelves
air conditioningaltarpiecebagbalconybasketbathtubbedbenchboxcabinetcarceilingceiling lampcountercurtaindrawerelevatorextractor hoodfieldfiles
ground
sky
tree
water
airplaneawningbasketbedbenchbicyclebottleboxbuildingbulletin boardcandlecarcentral reservationchairclothescountercrosswalkdesk lampdoorelephant
8
grass
plant
sky
tree
deck chairfieldfloegrassmountainpathpitchplantpoleseasheepshop windowskyswimming pooltreewall
building
mountain
sky
tree
air conditioningawningbasketbenchbrand namebuildingcabinetcarcaravanceilingclockcloudcoffee makercolumncupboardcurtaindesk lampdolldomedoor
9
building
mountain
road
sky
window
altarbackpackbagbarbecuebarsbasketbathtubbeambenchboardsboatboxbrand namebuildingcabinetcarcardceilingcentral reservationcloud
building
mountain
sky
tree
arcadebarrelbucketbuildingchaircourtcrosswalkdirt trackdoorelephantembankmentfieldfilesgrassgroundlandmagazinesmountainpathperson
10
building
mountain
sky
tree
window
armchairawningbenchbicycleboardsboatbottleboxbucketbuildingcabinetcarceilingchaircitycountercountertopcrosswalkcubiclecurtain
building
grass
sky
tree
balconyblanketbottlebrand namebuildingcabinetcarceiling lampchairchimneycloudcolumncountercountertopcurtaindomedoorelephantfencefloor
Source AP (ours) STF Source AP (ours) STF
6
1 2
11
ceiling
floor
table
wall
armchairbagbarbasketbicycleboatbookbookcasebottleboxbreadbridgebuildingbulletin boardcancarceilingceiling lampchairchandelier
ceiling
ceiling lamp
door
floor
wall
air conditioningarmchairbarsbenchboxbridgebuildingbulletin boardcabinetcarceilingceiling lampconveyor beltcrosscushiondoorfencefloorglass wallground
12
building
road
sky
tree
barrelboatbookbuildingbulletin boardcarcourtcupboarddirt trackdoorfencefieldfloorgrassgroundlandmountainpathplantroad
mountain
road
sky
tree
bulletin boardcabinetcarconveyor beltdoorfountainglass wallladderledgemachinerymountainpersonpipepolerailingriverroadrugseaseparation
13building
door
road
sidewalk
tree
window
air conditioninganimalawningbagbalconyballbarboatboxbridgebucketbuildingbulletin boardcabinetcarceilingceiling lampchairclosetcolumn
building
grass
sky
tree
window
billboardbookboxbucketbuildingcarceiling lampchairdoordummyfaucetfencefieldflowersgrassmountainpaintingpathpersonpitch
14
mountain
sea
sky
fish
mountain
river
sea
sky
swimming pool
ceiling
drawer
stove
tray
wall
bagbalconybenchbicyclebreadcarceilingceiling lampcentral reservationclockcloudcountercurtaincutleryfencefile cabinetfloorframegarage doorglass wall
15
building
mountain
road
sky
carconveyor beltembankmentfaucetfieldglass wallmountainplantpolerailingrefrigeratorriverroadsandseaseparationsignskysnowsnowy ground
floor
plant
wall
window
armchairawningbagbalconybedbenchboatbookbottleboxbuildingcancarceilingceiling lampchairclosetcolumncurtaindecoration
16
building
door
tree
wall
window
awningbagbasketbedbenchbookbowlbuildingbulletin boardcabinetcanceilingceiling lampchairchandeliercloudcountercutlerydesk lampdome
building
mountain
sky
tree
animalarmchairbalustradebarsbedbellboatbookbookcasebridgebulletin boardcarchestcolumncowdecorationdirt trackelephantfencefield
17
mountain
sky
tree
beam
floe
mountain
river
sea
sky
swimming pool
grass
sky
tree
armchairbuildingceilingconveyor beltcurtainembankmentfieldglass wallgrasslandmountainplantrocksheepsignskytreewater
18
building
grass
person
sky
tree
animalbenchboardsboatcabinetdeck chairgrapesgrasshay rollleafpathplantskytoastertree
building
mountain
sky
tree
bagbathtubbellbenchboatbookbucketbuildingbulletin boardcabinetcanchairchestclothescrossextractor hoodfencefloorflowersgate
19
building
mountain
sky
tree
boatbuildingcandlecaravandirt trackembankmentfloefloormountainpersonpianopillowrockskysnowstagestreetlightteddy beartowertoy
building
mountain
sky
animalbarsbedbenchbleachersboatbuildingcarcardceiling lampcentral reservationcloudcolumncrosscutlerydesk lampdomedoordrawerfence
20
floor
person
wall
balconybridgebuildingbulletin boardceilingceiling lampcolumncowdecorationextractor hoodfieldfloorfluorescent tubehandrailjarmirrorpaintingplantsinksky
ceiling
grass
monitor
person
sky
ceiling
grass
machine
sky
tree
Source AP (ours) STF Source AP (ours) STF
Title Suppressed Due to Excessive Length 7
1 2
21
building
car
road
sky
awningbalconyboxbrand namebuildingcarceilingceiling lampchaircoffee makercupboardcurtaindeck chairdesk lampeaselfenceflagflowersgroundmirror
building
road
sky
tree
armchairbalconybleachersbrand namebuildingcarcrossdomedooreaselfencehatheatermezzaninemirrorpaintingpianoprinterroadsand
22
building
road
sky
tree
animalarmchairbagbedbleachersboatbottleboxbrand namebuildingcarceilingceiling lampcentral reservationchairchestclosetclothescuddly toycurtain
ceiling
floor
wall
window
balconybasketbeambookboxbreadceilingceiling lampchairclosetcountercupcurtaindishdrawerdummyfanflagfloorfluorescent tube
23
door
floor
plant
wall
air conditioningbarbarrelbarsbenchbuildingbulletin boardcarceilingceiling lampchaircolumncubicledeckdirt trackdoor knobfencefieldfile cabinetfloor
building
person
road
sky
tree
appleawningbagbasketbenchbleachersbrand namebridgebuildingcarceilingcountercrosscupextractor hoodfaucetflaggroundice rinkmachine
24
building
grass
sky
tree
animalboatdeckdirt trackfencegrapesgrassgroundhay rollpathpersonplantpolerockscreensculptureskystepsstreetlighttree
building
mountain
sky
tree
bicyclebottlebreadbuildingcurtaindummyembankmentfencefishgarage doorguitarmezzaninemountainpersonpianoposterprojection screenroadrocking chairscaffolding
25
building
sky
tree
window
air conditioningaltaranimalawningbagbarsbedbellbleachersbookboxbridgebucketbuildingbulletin boardcabinetcanceilingceiling lampchair
building
ground
sky
tree
buildingcaravanclouddoorelephantfencefieldgrandstandgrassmountainpathpersonplantskytree
26
building
sky
tree
window
air conditioningalarm clockatticawningbalconybasketboatbowlboxbrand namebuildingcabinetcarceiling lampchaircloudcountercupboardcurtaindesk lamp
building
sky
tree
window
bedbilliard tableblanketboardsboatbuildingceilingdoorfencefloefountaingategroundhatmountainpersonplateaupoleprojection screenrefrigerator
27
building
mountain
sea
sky
bagbasketbedbenchbilliard tablebookbowlboxbuildingbulletin boardcabinetcarceiling lampchaircloudcountercupboarddishdoor knobdrawer
building
road
sky
tree
applearmchairawningbagbarsboatboxbreadbuildingbulletin boardcabinetcarcaravanceiling lampchestcowdesk lampdomedrawerelephant
Source AP (ours) STF Source AP (ours) STF
Fig. 3. More results on SUN, and comparison with semantic texton forests (STF) [3]. For someimages, STF assigned too many tags for a clear visualization, and so we limited the legends to 20words (omitting any excess words; for visualization purposes only).
8
1 2 3
1building
church
old
sky
tree
eye
face
girl
hair
woman
light
sky
sun
tree
2flower
green
plant
picture
rectangle
square
white
woman
grass
sky
tree
3
black
circle
plate
red
round
black
face
hair
man
woman
grass
green
house
tree
4
coin
gold
gray
money
silver
flower
grass
green
man
sky
black
face
hat
man
smile
5blue
chart
graph
man
red
round
square
white
yellow
lake
mountain
sky
tree
water
6cloud
ocean
sea
sky
water
city
mountain
sky
street
tower
girl
hair
swim
water
woman
7building
light
tree
white
blue
people
picture
sky
tree
black
eye
hair
man
music
8
flower
grass
lake
tree
water
diagram
drawing
green
white
black
hair
man
picture
Fig. 4. More results on ESP.
Title Suppressed Due to Excessive Length 9
1 2 3
1house
mountain
range
sky
tree
house
sky
statue
tree
water
mountain
river
road
side
tree
2bush
helmet
meadow
people
tree
desert
front
mountain
rock
tourist
bush
house
road
sea
sky
3grass
hill
man
sky
sweater
front
sky
tower
tree
window
canyon
landscape
lookout
road
sky
4chair
forest
front
garden
tourist
beach
palm
sea
sky
water
bush
mountain
people
slope
tree
5lamp
night
square
street
tower
city
cloud
house
sky
view
desert
mountain
people
rock
sky
6front
group
lake
people
sky
forest
front
gravel
road
sign
entrance
front
stone
tourist
wall
7face
rock
sky
stage
waterfall
mountain
rock
sky
slope
woman
building
house
mountain
slope
view
8horizon
sky
sun
tree
front
man
mountain
sky
tree
cloud
man
rock
sky
tree
Fig. 5. More results on IAPR.
10
1 2 3
1building
mountain
road
sky
tree
mountain
rock
sand
sea
sky
mountain
rock
sand
sea
sky
2car
mountain
road
sky
tree
building
field
plant
sky
tree
grass
plant
river
sky
tree
3building
mountain
river
sky
tree
building
mountain
road
sky
tree
building
car
mountain
sky
tree
4building
door
road
sky
building
ground
tree
window
grass
mountain
sky
tree
5building
door
road
sky
window
building
mountain
sea
sky
tree
building
road
sky
(a) SUN (and LMO)
1green
man
people
sky
tree
black
ear
eye
girl
smile
forest
green
rock
sky
tree
2
flower
orange
red
sun
white
circlecoincrossgoldround
night
sky
space
star
tree
(b) ESP
1hat
house
man
sky
woman
cloud
dune
sand
sea
sky
airport
building
plane
sky
tree
2chair
cloud
hill
sky
tree
desert
man
middle
people
sky
lamp
room
table
wall
window
(c) IAPR
Fig. 6. More failure cases on SUN, ESP and IAPR.
Title Suppressed Due to Excessive Length 11
References
1. Russell, B., Torralba, A., Murphy, K., Freeman, W.: Labelme: a database and web-based toolfor image annotation. IJCV 77 (2008) 157–173
2. Xiao, J., Hays, J., Ehinger, K., Oliva, A., Torralba, A.: Sun database: Large-scale scene recog-nition from abbey to zoo. In: CVPR. (2010) 3485–3492
3. Shotton, J., Johnson, M., Cipolla, R.: Semantic texton forests for image categorization andsegmentation. In: CVPR. (2008)
4. Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: SIGCHI. (2004)5. Grubinger, M., Clough, P., Muller, H., Deselaers, T.: The iapr benchmark: A new evaluation
resource for visual information systems. In: LREC. (2006) 13–236. Makadia, A., Pavlovic, V., Kumar, S.: Baselines for image annotation. IJCV 90 (2010)