![Page 1: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/1.jpg)
Predicting Visual Memory: Behavioral, Neuroscience and
Computational Accounts
Aude Oliva Computer Science and Artificial Intelligence
Lab Massachusetts Institute of Technology
1
![Page 2: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/2.jpg)
Can we predict which images are memorable ?
© Source Unknown. All rights reserved. This content is excluded from our CreativeCommons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.
2
![Page 3: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/3.jpg)
Memorability A metric of the utility of information
Understand human memory Diagnose memory problems
© IEEE. All rights reserved. This content is excluded from our Creative Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/. © Source Unknown. All rights reserved. Source: Isola, Phillip, Jianxiong Xiao, Devi Parikh, Antonio This content is excluded from our Creative Torralba, and Aude Oliva. "What makes a photograph Commons license. For more information,memorable?." IEEE transactions on pattern analysis and see https://ocw.mit.edu/help/faq-fair-use/.machine intelligence 36, no. 7 (2014): 1469-1482.
Data Visualization
© Source Unknown. All rights reserved. Thiscontent is excluded from our CreativeCommons license. For more information, seehttps://ocw.mit.edu/help/faq-fair-use/.
Logos Slogans - words-
© Google. All rights reserved. This content isexcluded from our Creative Commons license.For more information, seehttps://ocw.mit.edu/help/faq-fair-use/.
Education -Individual differences
© Source Unknown. All rights reserved. Thiscontent is excluded from our Creative Commonslicense. For more information, seehttps://ocw.mit.edu/help/faq-fair-use/.
Mobile applications
© Blackberry. All rights reserved. Thiscontent is excluded from our CreativeCommons license. For more information,see https://ocw.mit.edu/help/faq-fair-use/.
Social Networking
Face images, videos Memorability
© Source Unknown. All rights reserved. This© Source Unknown. All rights reserved. This content is content is excluded from our Creativeexcluded from our Creative Commons license. For more Commons license. For more information, seeinformation, see https://ocw.mit.edu/help/faq-fair-use/. https://ocw.mit.edu/help/faq-fair-use/.
Design mnemonic aids
© Source Unknown. All rights reserved. Thiscontent is excluded from our CreativeCommons license. For more information, seehttps://ocw.mit.edu/help/faq-fair-use/.
Retrieve better images
from search
© Source Unknown. All rights reserved. Thiscontent is excluded from our CreativeCommons license. For more information, seehttps://ocw.mit.edu/help/faq-fair-use/.
Computer Graphics
- cognitive saliency
© Source Unknown. All rights reserved. Thiscontent is excluded from our Creative Commons license. For more information, seehttps://ocw.mit.edu/help/faq-fair-use/.
Summarize Bigdata –
3
© Facebook. All rights reserved. This content isexcluded from our Creative Commons license.For more information, see https://ocw.mit.edu/-
. help/faq-fair-use/.
![Page 4: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/4.jpg)
Figure removed due to copyright restrictions. Please see the video.Source: Isola, Phillip, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. "What makes an image memorable?"In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 145-152. IEEE, 2011.
4
Visual Memory Experiments
Phillip Isola
![Page 5: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/5.jpg)
5
Figure removed due to copyright restrictions. Please see the video.Source: Isola, Phillip, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. "What makes an image memorable?"In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 145-152. IEEE, 2011.
Large difference in image memorability& high consistency between observers’ groups
Isola et al (2011). IEEE CVPR
![Page 6: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/6.jpg)
0.75
Unknown. All rights reserved.This content is excluded from our CreativeCommons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.© Source
![Page 7: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/7.jpg)
~ 80 scores per image chance level rank calculated by randomly ordering the images on the x-axis
Consistent ρ = 0.75
Is memorability consistent across different observers? Yes
Averag
e %
mem
orab
ilit
yPr
obab
ility
of co
rrec
tly
dete
ctin
g a
repe
at
Image rank N, according to group 1
7
![Page 8: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/8.jpg)
8
Figure removed due to copyright restrictions. Please see the video.Source: Isola, Phillip, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. "What makes an image memorable?"In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 145-152. IEEE, 2011.
Subjective judgments do not predict image memorability
![Page 9: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/9.jpg)
9
Image memorability is distinct from image aesthetic
Figure removed due to copyright restrictions. Please see the video.Source: Isola, Phillip, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. "What makes an image memorable?"In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 145-152. IEEE, 2011.
![Page 10: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/10.jpg)
10
Is memorability stable across time? Yes
Figure removed due to copyright restrictions. Please see the video.Source: Isola, Phillip, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. "What makes an image memorable?"In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 145-152. IEEE, 2011.
![Page 11: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/11.jpg)
27
When do memorability differences arise?At stage of encoding: This suggests some images (features) are encoded in less
sufficient detail than others
Figure removed due to copyright restrictions. Please see the video.Source: Isola, Phillip, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. "What makes an image memorable?"In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pp. 145-152. IEEE, 2011.
![Page 12: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/12.jpg)
Wilma Bainbridge
Courtesy of The American Psychological Association. Used with permission.Source: Bainbridge, Wilma A., Phillip Isola, and Aude Oliva."The intrinsic memorability offace photographs." Journal of Experimental Psychology: General 142, no. 4 (2013): 1323.
Face dataset 10K: faces selection follows the distribution of the US census. Available on the web
Bainbridge, Isola, Oliva, (2013). J. Exp. Psychology: General. 12
![Page 13: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/13.jpg)
Large difference in face memorability & high consistency between observers’ groups
Consistent ρ = 0.69
Consistent ρ = 0.72
© Journal of Experimental Psychology. All rights reserved. This content is excluded from ourCreative Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.Source: Bainbridge, Wilma A., Phillip Isola, and Aude Oliva. "The intrinsic memorability of facephotographs." Journal of Experimental Psychology: General 142, no. 4 (2013): 1323.
Bainbridge, Isola, Oliva, (2013). J. Exp. Psychology: General. 13
![Page 14: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/14.jpg)
Kyle Mahowald
Words Memorability Phillip Isola
68%68% Av
erag
e %
HIT
Consistent ρ = 0.66
MIT Word Dataset 10K dataset
Aver
age
% H
IT
Consistent ρ = 0.64
Subtlex: American English subtitles
Word rank N, according to group 1 Word rank N, according to group 1
Ev Fedorenko Ted Gibson
Mahowald, Isola, Fedorenko, Oliva, Gibson (submitted, 2015).
![Page 15: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/15.jpg)
Memorable vs. Forgettable words
Forgettable words Excellent Blast Irrational Massive
Memorable words Fabulous Vogue Grotesque Avalanche
© clipartpanda.com. All rights reserved.This content is excludedfrom our Creative Commons license. For more information, seehttps://ocw.mit.edu/help/faq-fair-use/.
Information-theoretic norm (number of synonyms, number of meanings) predict word memorability See Konkle, et al (2011), JEP:General
Mahowald, Isola, Fedorenko, Oliva, Gibson (submitted, 2015). 15
![Page 16: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/16.jpg)
eur r mewor o memor i it Wilma Perception Memory Bainbridge
PPA
RSC OPA
LOC
e tr /dors isu corte edi empor o e
Attention Early visual areas network
![Page 17: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/17.jpg)
Whole Brain ContrastNovel Novel
Memorable Forgettable
17
![Page 18: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/18.jpg)
Equalized Memorable & Forgettable Groups
Interestingness Aesthetic
180 memorable scenes 90 indoor, 90 outdoor (38 landscapes) HIT : 0.97 (above 25% of HR) - FA: 0.10 Color (RGB, Lab), spectral frequency
Attractiveness Emotion Kindness Friendliness Happiness Introversion Confidence
180 memorable faces 90 women, 90 men HIT : 0.72 (above 25% of HR) - FA: 0.11 Race, age, type of emotion, spectral frequency
180 forgettable scenes 90 indoor, 90 outdoor (38 landscapes) HIT : 0.69 (below 25% of HR) - FA: 0.10 Color (RGB, Lab), spectral frequency
180 forgettable faces 90 women, 90 men HIT : 0.32 (below 25% of HR) - FA: 0.11 Race, age, type of emotion, spectral frequency
18
![Page 19: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/19.jpg)
Multi-variate Pattern Analysis Memorable vs. Forgettable
Perirhinal
Hippocampus (Posterior)
Parahippocampal Cortex
A role for the hippocampus in higher-order statistical perception
Bainbridge, Dilks, Oliva (in preparation) MVPA Searchlight preliminary results (3x3x3 cubic voxel grid) 19
![Page 20: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/20.jpg)
Computational Model of Memorability
© Source Unknown. All rights reserved. This content is excluded from our CreativeCommons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/. 20
![Page 21: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/21.jpg)
Figure removed due to copyright restrictions.Source: Khosla, Aditya, Akhil S. Raju, Antonio Torralba, and Aude Oliva."Understanding and predicting image memorability at a large scale." In Proceedings of the IEEE International Conference on Computer Vision, pp. 2390-2398. 2015.
Large scale visual memorability60,000 photographs with memorability scores
Most memorable Less memorable
Aditya Khosla
http://memorability.csail.mit.edu/21
![Page 22: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/22.jpg)
22
0.68
Unknown. All rights reserved. This content is excluded from our Creative Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.© Source
![Page 23: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/23.jpg)
23
Most memorable Less memorable
• Focus
• Enclosed Setting
• Dynamics
• Unusual
60,000 images
• No single focus
• Distant view
• Static
• Common
Figure removed due to copyright restrictions.Source: Khosla, Aditya, Akhil S. Raju, Antonio Torralba, and Aude Oliva."Understanding and predicting image memorability at a large scale." In Proceedings of the IEEE International Conference on Computer Vision, pp. 2390-2398. 2015.
![Page 24: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/24.jpg)
You need to recognize to remember
© Source Unknown. All rights reserved.This content is excluded from our CreativeCommons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.
24
![Page 25: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/25.jpg)
Building first a visual recognition system
25
© Source Unknown. All rights reserved.This content is excluded from our CreativeCommons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.
![Page 26: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/26.jpg)
Scene Understanding:
Context and Objects
d. This content is mons license. For more edu/help/faq
26
26© Dorothy Hays. All rights reserved. This content is excluded from our Creative Common license. For more information, see http://ocw.mit.edu/help/faq-fair-use/.
![Page 27: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/27.jpg)
27
he e o utio o sce e d o ect ce tered d t ses
# images103 105 109107104 106 108
# images103 105 109107104 106 108
COIL-20
Caltech 101
PASCAL (2005)Caltech-4 (2003)
4
MNIST (1998)
2 year old kid
SUN database (2010)8 scenes database (2001)
15 scenes database (2006)
8 scenes database (2001)8 scenes database (2001)
(2009)
![Page 28: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/28.jpg)
© Computer Vision and Pattern Recognition. All rights reserved. This content is excluded from our Creative Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.Source: Xiao, Jianxiong, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba."Sun database:Large-scale scene recognition from abbey to zoo." In Computer vision and pattern recognition (CVPR), 2010IEEE conference on, pp. 3485-3492. IEEE, 2010.
Xiao et al (2010), CVPR, Ehinger et al (2011)
![Page 29: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/29.jpg)
© Google. All rights reserved. This content is excluded from our Creative Commonslicense. For more information, see https://ocw.mit.edu/help/faq-fair-use/.
29
![Page 30: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/30.jpg)
© Google. All rights reserved. This content i s excluded from our Creative Commonslicense. For more information, see https://ocw.mit.edu/help/faq-fair-use/. 30
![Page 31: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/31.jpg)
Improving diversity abandoned, acceptable, accessible, additional, adjacent, advertised, affordable, air-conditioned, alternative, american, amusing, ancient, antique, appealing, appropriate, architectural, asian, astonishing, astounding, attractive, austere, authentic, available, average, awesome, beautiful, beguiling, beloved, best, better, better-known, big, bigger, biggest, bizarre, black, black-and-white, bland, boring, breezy, brick-built, bright, brighter, brightest, brilliant, broken, busiest, business-like, bustling, busy, central, centralized, certain, changed, changing, charming, cheap, cheaper, cheapest, cheerful, cheerless, cheery, cherished, chilling, chilly, civilized, classic, classical, clean, cleaner, clear, clearer, clinical, closer, closest, closing, cloudy, coastal, cold, coldest, colourful, comfortable, comforting, comfortless, comfy, common, comparable, comparative , competitive, complementary, complete, complex, complicated, concealed, conceivable, confined, considerable, contemporary, cool, coolest, cosmopolitan, cost-effective, cosy, cozy, cream-white, creative, crowded, cultivated, cultural, current, damp, dangerous, dark, darkened, darker , darkest, decorative, delightful, designated, designed, desirable, desired, desolate, desolated, different, difficult, dilapidated, dim, dimly-lit, dingy, dirty, disadvantageous, disorderly, do-it-yourself, domestic, double, double-fronted, double-length, downtown, drab, dreadful, driest, dry , dual, dull, duller, dullest, dusty, early, economic, economical, elegant, embarrassing, empty, enormous, especial, european, everyday, exciting, exemplary, exotic, exterior, external, extraordinary, extravagant, familiar, famous, fancy, fantastic, far-away, fascinating, fashionable, fashioned, favourable, fictional, fictitious, filmed, filthy, fine, foggy, foreign, formal, fractured, friendly, frightening, frightful, frosty, frozen, frustrating, full, funny, furnished, fuzzy, gaudy, ghastly, ghostly, glamorous, glassy, glazed, glittering, gloomy, glorious, glossy, godlike, gold-plated, good, gorgeous, graceful, gracious, grand, gray, great, greatest, green, greener, grey, grisly, gruesome, habitable, habitual, handy, happy, harmonious, harrowing, harsh, hazardous, healthful, healthy, heart-breaking, heart-rending, heavy, hideous, hiding, higgledy-piggledy, high, hilarious, historic, historical, holiest, home, horizontal, hospitable, hostile, hot, huge, humid, idyllic, illegal, imaginary, immaculate, im mense, imminent, immortal, impassable, impassioned, impersonal, important, impossible, impressive, improbable, improper, inauspicious, in conceivable, inconvenient, incredible, independent, individual, indoor, industrial, ineffable, inexpensive, informal, inhabited, inhospitable, in itial, innovatory, innumerable, insecure, insignificant, inspiring, integrated, intentional, interesting, intermediate, internal, international, in timidating, intriguing, inviting, irrational, irregular, isolated, joint, joyful, key, known, large, large-scale, largest, less-favored, lesser, licensed , li feless, light, limited, little, little-frequented, little-known, lively, living, local, lofty, logical, lone, long, long-awaited, long-forgotten, long-in habited, long-netting, long-stays, long-term, lost, lousy, lovely, low, low-ceilinged, low-cost, low-energy, lower, lucky, luxury, magical, magnificent, main, majestic, major, marginal, marine, marvellous, massive, masterful, maximum, mean, meaningless, mechanised, medieval, mediocre, medium-sized, melancholy, memorable, messy, middle, middle-order, mighty, miniature, minor, miserable, missing, misty, mixed, modern, moist, mouldy, mountainous, moving, muddy, multi-functional, multiple, mundane, murky, musty, muted, mysterious, mysterious-lo oking, mystic, mystical, mythic, naff, named, nameless, narrow, national, native, natural, naturalistic, nearby, neat, necessary, neglected, neighboring, new, nice, night-time, nineteenth-century, noisy, nondescript, normal, northern, notable, notorious, numerous, odd, odorous, official, old, only, open, open-air, operatic, orderly, ordinary, organic, original, ornamental, out-of-homes, out-of-the-way, outdoor, outlying, outside, outstanding, over-crowded, overgrown, overwhelming, paid, painful, painted, palatial, pastoral, peaceful, peculiar, perfect, periodic, peripheral, permanent, permitted, personal, petty, pictorial, picturesque, pitiful, placid, plain, planted, pleasant, pleasing, poisonous, poor, popular, populated, populous, positive, possible, post-war, posterior, postmodern, potential, powerful, practical, pre-arranged, pre-eminent, precise, predictable, present, present-day, preserved, pretty, previous, pricey, primal, prior, private, privileged, probable, professional, profitable, promising, proven, public, pure, queer, quiet, rainy, rare, real, realistic, reasonable, rebuilt, recent, recognized, recommended, re constructed, recreated, recurring, red, red-brick, redundant, refused, regional, regular, related, relative, relaxing, relevant, reliable, religious, re maining, remarkable, remote, rented, representative, reputable, required, reserved, residential, respectable, respected, restful, restless, re stricted, retail, rich, ridiculous, right, rigid, river-crossing, rocky, romantic, rural, sacred, sad, safe, salubrious, satisfying, scary, scattered, scenic, scientific, secondary, secret, secured, selected, senior, separated, serious, sexy, shiny, shocking, shoddy, short-term, significant, silent, silly, similar, simple, single, sizable, slack, small, smelly, smoke-free, smoking, snowy, sobering, soft, solid, sombre, soothing, sophisticated, sorrowful, sound-filled, southern, spare, spatial, special, specialized, spectacular, sporting, stable, standard, static, steady, stifling, strange, stressful, striking, stunning, stupendous, stupid, stylish, successful, sufficient, sunny, super, superb, superior, surrealistic, suspicious, symbolic, te enage, terrible, terrific, theoretical, thrilling, thriving, tidier, tight, tiny, tough, tragic, unattractive, unbelievable, uncertain, unchanging, uncharted, uncivilized, uncomfortable, unconventional, underground, underwater, undisturbed, uneven, unexpected, unfamiliar, unforgettable, unfriendly, unhappy, unhealthy, unimportant, unknown, unnatural, unnecessary, unparalleled, unpleasant, unsafe, unseemly, unsuitable, unusual, upmarket, urban, vague, valuable, varied, various, vertical, very, vibrant, virtual, visual, vital, vivid, voluntary, vulgar, vulnerable, wacky, waiting, warm, wealthy, weeping, weird, weird-looking, well-assured, well-defended, well-designed, well-hidden, well-insulated, well-known, well-lit, well-loved, well-ordered, well-organized, well-secured, well-sheltered, well-used, wet, white, whole, wicked, wide, widespread, wild, windy, wintering, wonderful, wondrous, wooded, wordless, working, worldly, worldwide, worst, worthwhile, worthy, wretched, wrong, young, yucky, 31
![Page 32: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/32.jpg)
© Google. All rights reserved. This content is excluded from our Creative Commonslicense. For more information, see https://ocw.mit.edu/help/faq-fair-use/.
32
![Page 33: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/33.jpg)
© Google. All rights reserved. This content is excluded from our Creative Commonslicense. For more information, see https://ocw.mit.edu/help/faq-fair-use/.
33
![Page 34: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/34.jpg)
© Google. All rights reserved. This content is excluded from our Creative Commonslicense. For more information, see https://ocw.mit.edu/help/faq-fair-use/. 34
![Page 35: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/35.jpg)
© Google. All rights reserved. This content is excluded from our Creative Commonslicense. For more information, see https://ocw.mit.edu/help/faq-fair-use/.
35
![Page 36: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/36.jpg)
Bolei Aditya Agata Antonio
AlexNet
Khosla Lapedriza Zhou Torralba 10 million images, 460 categories
Courtesy of Zhou, Bolei, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva."Learning deep features for scene recognition using places database." In Advances in neuralinformation processing systems, pp. 487-495. 2014.
places.csail.mit.edu
![Page 37: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/37.jpg)
Web demo: places.csail.mit.edu
37
© CSAIL. All rights reserved. This content is excluded from our Creative Commons license.For more information, see https://ocw.mit.edu/help/faq-fair-use/.
![Page 38: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/38.jpg)
Web demo: places.csail.mit.edu
38
© CSAIL. All rights reserved. This content is excluded from our Creative Commons license.For more information, see https://ocw.mit.edu/help/faq-fair-use/.
![Page 39: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/39.jpg)
Estimating the Receptive Fields
pool1 pool5conv3
Courtesy of Bolei Zhou, Aditya Khosla, Agata Lapedriza, A
More semantically meaningful ude Oliva and Antonio Torralba. Used with permission.
Zhou, Khosla, Lapedriza, Oliva, Torralba (2015), ICLR 39
![Page 40: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/40.jpg)
Spatial maps of correlations between human brain and model layers
AlexNet
V1 Parietal
Ventral
Cichy, Khosla, Pantazis, Torralba, Oliva (submitted) 40
© Radoslaw Cichy, Aditya Khosla, Dimitrios Pantazis, Antonio Torralba and Aude Oliva.All rights reserved. This content is excluded from our Creative Commons license. Formore information, see https://ocw.mit.edu/help/faq-fair-use/.
![Page 41: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/41.jpg)
41
![Page 42: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/42.jpg)
42
![Page 43: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/43.jpg)
43
![Page 44: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/44.jpg)
44
![Page 45: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/45.jpg)
45
![Page 46: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/46.jpg)
46
![Page 47: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/47.jpg)
47
![Page 48: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/48.jpg)
48
![Page 49: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/49.jpg)
49
![Page 50: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/50.jpg)
50
![Page 51: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/51.jpg)
51
![Page 52: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/52.jpg)
52
![Page 53: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/53.jpg)
Object detectors emerge inside the CNN
Courtesy of Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva and Antonio Torralba. Used with permission.
53
![Page 54: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/54.jpg)
em et or redicti m e emor i it
input output
HybridNet from Zhou et al, NIPS 2014 (places.csail.mit.edu)
![Page 55: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/55.jpg)
r i i em et
input output
Scenes and
Objects
![Page 56: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/56.jpg)
r i i em et
input output
LaMem
Backpropagation Fine-tuning
Khosla, Raju, Torralba and Oliva (under review). Understanding and Predicting ImageMemorability at a Large Scale. http://memorability.csail.mit.edu/.
![Page 57: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/57.jpg)
0.65
0.68
© Source Unknown. All rights reserved. This content is excluded from our Creative Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.
![Page 58: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/58.jpg)
Figure removed due to copyright restrictions. Please see the video.Source: Khosla, Aditya, Akhil S. Raju, Antonio Torralba, and Aude Oliva."Understandingand predicting image memorability at a large scale." In Proceedings of the IEEEInternational Conference on Computer Vision, pp. 2390-2398. 2015.
58
Visualizing Neurons RF of Memorability
![Page 59: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/59.jpg)
Cognitive Saliency: Which regions are memorable?
59
![Page 60: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/60.jpg)
60
![Page 61: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/61.jpg)
61
![Page 62: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/62.jpg)
62
![Page 63: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/63.jpg)
63
![Page 64: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/64.jpg)
64
![Page 65: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/65.jpg)
Modeling Human Memory
65
© Source Unknown. All rights reserved. This content is excluded from our Creative Commons license. For more information, see https://ocw.mit.edu/help/faq-fair-use/.
![Page 66: Predicting Visual Memory - MIT OpenCourseWareocw.mit.edu/resources/res-9-003-brains...Equalized Memorable & Forgettable Groups Interestingness Aesthetic 180 memorable scenes 90 indoor,](https://reader034.vdocuments.mx/reader034/viewer/2022050323/5f7c8e1df9b0281a0b175c7f/html5/thumbnails/66.jpg)
MIT OpenCourseWarehttps://ocw.mit.edu
Resource: Brains, Minds and Machines Summer CourseTomaso Poggio and Gabriel Kreiman
The following may not correspond to a p articular course on MIT OpenCourseWare, but has beenprovided by the author as an individual learning resource.
For information about citing these materials or our Terms of Use, visit: https://ocw.mit.edu/terms.