92.9% 92.1% 88.1% 87.6% 36.8% 33.8% 33.0% 27.4%
93.3% 93.3% 90.8% 88.9% 35.3% 34.7% 33.0% 31.9%
mou
ntai
npa
stur
eam
usem
ent
park HR:
64.2%
HR: 59.2%
HR: 50.2%
84.8% 81.4% 78.8%95.7% 32.3% 31.2% 28.4% 28.4%
HR: 55.6%ai
rpor
tte
rmin
al
91.5% 87.5% 86.9% 85.4% 22.9% 17.1% 13.5% 9.0%
cock
pit
HR: 49.5%
94.9% 91.4% 91.4% 91.2% 21.8% 18.8% 17.1% 15.5%
93% 92% 88% 88% 37% 34% 33% 27%
32%33%35%35%89%91%93%93%
28%28%31%32%79%81%85%96%
15%17%19%22%91%91%91%95%
91% 88% 87% 85% 23% 17% 14% 9%
highest memorability lowest memorability
1.0 s
... ...
1.4 stime
false alarmcorrect rejection hitrepeat
+ + ++ +
Quantifying Context Effects on Image MemorabilityZoya Bylinskii, Phillip Isola, Antonio Torralba, Aude Oliva
Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology
Crowd-sourced (AMT) memory (recognition) games
FIGRIM Dataset (Fine-Grained Image Memorability) 21 scene categories, more than 300 instances/category
Extrinsic Effects of Image Context
Contextually distinct images are more memorablePearson r(D1,HR1) = 0.26Pearson r(D2,HR2) = 0.24Pearson r(∆D,∆HR) = 0.35
More varied image contexts are more memorable overallPearson r(H,HR) = 0.53Pearson r(∆H,∆HR) = 0.74
Extrinsic Effects of Individual Observer
Pc(fi) =1
kCkX
j2C
K(fi � fj)
Intrinsic Effects on Memorability
v v
Memorable within categories
Memorable across categories
We train a classifier to predict whether a set of eye movements will lead to a successful encoding
H(C) = Ec[� logPc(fi)]
Memorability rank of images is consistent across participants and experimentsSpearman r = 0.69-0.86(for each of 21 categories)
Paper: Bylinskii, Z., Isola, P., Bainbridge, C., Torralba, A., Oliva, A. "Intrinsic and Extrinsic Effects on Image Memorability",
Vision Research 2015.
Dataset: http://figrim.mit.edu
Information theoretic model for context
Memorability rank of categories is stableSpearman r = 0.68(across splits of images)
context: set of images from which experimental sequence is sampled
Modeling is done with CNN (convolutional neural net) features which encode image semantics.
Pairwise correlations b/w experiments r > 0.60
more likely to look like images from other categories
more likely to be memorable across different contexts
coarse binning and smoothing
exemplar classifier trained per image
test-time classification
HR (hit rate): ratio of hits to total of hits and misses
* we only use fixation position for prediction
60.09%
66.02%
Eye movements are predictive of whether an image will be remembered later
D(I;C2)�D(I;C1)
HR
c 2(I)�HR
c 1(I)
ï16.5 ï16 ï15.5 ï15 ï14.5 ï14
50
52
54
56
58
60
62
64
Context Entropy
Ave
rage
HR
(%
)
airport
amusement park
badlands
bathroom
bedroom
bridge
castle
cockpit
conference room dining room
golf course
highway
house
kitchen
lighthouse
living room
mountain
pasture
playground
skyscraper tower
Pearson r = 0.53 (p = 0.01)
H(C)
HR
images sorted by expected classifier confidence
...
* figures include scores for within-category experiment
D(I;C) = � logPc(fi)
