visual attention derek hoiem march 14, 2007 misc reading group
TRANSCRIPT
Saccades and Fixations Saccades and Fixations
• Scope: 2 deg (poor spatial res beyond this)
• Duration: 50-500 ms (mean 250 ms)
• Length: 0.5 to 50 degrees (mean 4 to 12)
• Various types (e.g., regular, tracking, micro)
Saccades and Fixations Saccades and Fixations
Free Examine
What are the material circumstances of the family?
What are their ages?
What were they doing before arrival?
Remember the clothes
Remember object and person positions
How long has the unexpected visitor been away?
[Yarbus 1967]
Visual PhenomenaVisual Phenomena
• Fast scene recognition (100-150 ms)
• Fast “contains animal” (100-150 ms)
• Pop-out
• Attentional blindness
• Change blindness
Pop-out (more texture)Pop-out (more texture)
+V V
V
VVV
VV
V
V
V
V
VV
V
V
V
V
V
VV V
VV
V
V
V
V
VV
V
V
T
Pop-out (harder texture)Pop-out (harder texture)
+
V
VV
VV
V
V
V
V
V V
V
V
V
V
V
V
V
VV
VV V
V
V
V
V
VV
V
V
V
Y
Pop-out (color + texture)Pop-out (color + texture)
+T
V
V
V
V
V
V
V V
V
T
V
V
V
V
V
TV
VT V
T
V
V
V
TT
T
V
TV
Pop-out (layout)Pop-out (layout)
+
T T T T T T T
T T T T T T
T T T T T T T
T T T T T T T
T T T T T T T
T T T T T T T
T
DemosDemos
http://viscog.beckman.uiuc.edu/djs_lab/demos.html
Purpose of attentionPurpose of attention
• Warning (animals, flashes, sudden motion)
• Exploration (find objects, verification)
• Inspection
Bottom-up Attention ModelsBottom-up Attention Models
[Itti Koch Niebur 1998]
Gabor Pyramid + Orientation Filters
Subtract low-res (3-4 octaves) from higher res
Normalize (0..1) + map * ( 1 – maxave )2 + add maps
Average Maps
Inhibition + Excitation
Bottom-Up: NormalizationBottom-Up: Normalization
[Itti Koch Niebur 1998]
• Normalize map values to fixed range [0..1]
• Compute average local maximum m
• Multiply map by (1-m)2
Updates to Bottom-Up ModelUpdates to Bottom-Up Model
[Peters Iyer Itti Koch 2005]
• Cross-orientation suppression
• Long-range contour interactions
• Eccentricity-dependent processing (e-x)
– Goal: better prediction of subsequent fixations
Experiments with Newer ModelExperiments with Newer Model
[Peters Iyer Itti Koch 2005]
Normalized Scanpath Salience Inter-observer Salience
Almost No Benefit to More Complicated ModelsAlmost No Benefit to More Complicated Models
[Peters Iyer Itti Koch 2005]
“Eccentricity-Dependent Filtering” Helps“Eccentricity-Dependent Filtering” Helps
[Peters Iyer Itti Koch 2005]
No EDF EDF
Other Bottom-Up IssuesOther Bottom-Up Issues
• Real viewing vs. images (Gajewski et al. 2005)
– Longer saccades (12 deg vs. 4 deg)
– Short saccades may be due to density of images, rather than movement cost
• Saliency map?
– Evidence for multi-saliency representations (not clear there is a single map)
– Capability to ignore predictable motions is difficult in map formulation
Image Saliency Map
Alternative Bottom-up ModelsAlternative Bottom-up Models
• Itti-Koch accounts for some pop-out effects
– Is it biologically plausible? (peak normalization)
– Is it biologically reasonable?
– No reasonable mechanism for next fixation
• Top-down bias only (Wolfe’s guided search)
– Does not account for free viewing behavior
• Surprise or explanation seeking (expectation-based saliency)
– No saliency map required
– May provide better prediction of next fixation, account for motion prediction
Top-Down Attention ModelsTop-Down Attention Models
• Feature weighting
– Verbal
– Visual
• Location prior
– From memory of scene (direct or indirect)
– From scene information and semantics
Verbal Cueing Feature WeightingVerbal Cueing Feature Weighting
• Faster search if cued as to color or texture
• Faster yet if exemplar is shown
• Searching for mid-level cues (e.g., intensity, size, saturation) is harder
– But may still be cued
[Navalpakkam Itti 2006]
Role of MemoryRole of Memory
• People can remember hundreds or thousands of scenes from single exposure (Shephard 1967)
• After seeing repeated scenes (in random order)
– Faster finding of target (Brockmole and Henderson 2006)
• When mirrored after learning
– First look at original location, then quickly go to new location (still faster)
• Learning of upside-down scenes takes twice as long
Scene ContextScene Context
• Scene-constrained targets detected faster, with fewer eye movements
• Strategy
1st: check target-consistent regions
2nd: check target-inconsistent regions
[Neider Zelinsky 2005]
Scene ContextScene Context
• “Gist” can provide image height prior
[Torralba et al. 2006]
Saliency = inverse probability ^(0.05) * gaussian
Bottom-up + Top-down AttentionBottom-up + Top-down Attention
• Method 1: Weight individual features
• Method 2: Saliency .* Bias
ConclusionsConclusions
• Artificial static scenes and pop-out well-explained by existing models
• Little recent progress in bottom-up models (stuck with Itti-Koch model)
• Only simplistic scene information modeled
SourcesSources• Saliency
– Itti, Koch, Niebur (1998). A model of saliency-based visual attention for rapid scene analysis.
– Itti, Koch (2001). Computational Modelling of Visual Attention.
– Itti (2002). Modeling Primate Visual Attention.
– Itti (2002). Visual Attention.
– Navalpakkam, Arbib, Itti (2004). Attention and Scene Understanding.
– Peters, Iyer, Itti, Koch (2005). Components of bottom-up gaze allocation in natural images.
• Role of memory
– Chun, Jiang (1998). Contextual cueing: implicit learning and memory of visual context guides spatial attention.
– Chun, Jiang (2003). Implicit, long-term spatial contextual memory.
– Brockmole, Henderson (2006). Recognition and attention guidance during contextual cueing in real-world scenes: Evidence from eye movements.
• Top-Down Attention
– Niedur, Zelinksy (2005). Scene context guides eye movements during visual search.
– Navalpakkam, Itti (2006). Top-down attention selection is fine grained.
– Torralba, Oliva, Castelhano, Henderson (2006). Contextual guidance of eye movements and attention in real-world scenes: the role of global features on object search.
SourcesSources• Others (used)
– Rensink, O’Regan, Clark (1997). To see or not to see: the need for attention to perceive changes in scenes.
– Liversedge, Findlay (2000). Saccadic eye movements and cognition.
– Rensink (2000). The dynamic representation of scenes.
– Delorme, Rousselet, Mace, Fabre-Thorpe (2004). Interaction of top-down and bottom-up processing in the fast visual analysis of natural scenes.
– Gajewski, Pearson, Mack, Bartlett, Henderson (2005). Human gaze control in real world search.
– http://www.diku.dk/~panic/eyegaze/node13.html
• Others (not used but potentially interesting)
– Itti, Koch, Braun (2000). Revisiting spatial vision: toward a unifying model.
– Epstein (2005). The cortical basis of visual scene processing.
– Baldi, Itti (2005). Attention: Bits versus Wows.