image retrieval using eye movements fred stentiford & wole oyekoya university college london
Post on 23-Jan-2016
216 views
TRANSCRIPT
![Page 1: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/1.jpg)
Image Retrieval Using Eye
Movements
Fred Stentiford & Wole Oyekoya
University College London
![Page 2: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/2.jpg)
Outline
1. Eye Movement Behaviour
2. Image Identification
3. Image Search
4. Conclusions & Future Work
![Page 3: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/3.jpg)
Eye Gaze Computer (Eye Image Processing) Client Computer
(Application Program)
Chinrest
Eye Monitor
Real-Time Gazepoint
Display
Application Display
Eye Tracking System
![Page 4: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/4.jpg)
Eye Movement Behaviour
saliency mapimage
fixation and saccade map
noROI
![Page 5: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/5.jpg)
Eye Movement Behaviour
clearROI
saliency mapimage
fixation and saccade map
![Page 6: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/6.jpg)
Eye Movement Behaviour – no ROI
participant A participant B
participant Dparticipant C
![Page 7: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/7.jpg)
Eye Movement Behaviour – clear ROI
participant A participant B
participant Dparticipant C
![Page 8: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/8.jpg)
Variance of Attention Measure
149786212261246378Image6
1466120214531432246Image5
8571094687741443Image4
Obvious ROI
197365175389175Image3
629328496479500Image2
532333193325298Image1
Unclear ROI
DCBA
Participants Image Variance
![Page 9: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/9.jpg)
Time Fixating Salient Regions (ms)
1240162098042406
19602220148036805
12802420234028204
Obvious ROI
204001003
4005004205802
1402060401
Unclear ROI
DCBA
ParticipantsImages
![Page 10: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/10.jpg)
Findings
No special fixation sequence although many look at salient regions first
Very salient regions inspected frequently and compared with background
![Page 11: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/11.jpg)
Eye vs Mouse for Image Identification
target image
1. Mouse click
2. Fixation > 40ms
![Page 12: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/12.jpg)
Screen Display Sequence
D D D D D D D D D D D D D D D D D D D DD D D D D D T2 D D D D D D D D D D D D DD D D D T1 D D D D D D D D D D … D D D D DD D D D D D D D D D D D D D D D D D T50 DD D D D D D D D D D D D D T3 D D D D D D
D = distractor Tn = target image
![Page 13: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/13.jpg)
Eye vs Mouse Response Times
INPUT Main Effect
F(1,10) = 8.72; p < 0.0145
1.9
2.0
2.1
2.2
2.3
2.4
2.5
Mouse Eye
Res
po
nse
Tim
e (s
eco
nd
s)
12 participants
![Page 14: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/14.jpg)
Eye vs Mouse Response Times
Mouse First
Eye First
1.8
1.9
2
2.1
2.2
2.3
2.4
2.5
2.6
Mouse Eye
Res
po
nse
Tim
es (
seco
nd
s)
6 participants in each group
![Page 15: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/15.jpg)
Image Search Task
target image
target image
steps to target
1000 images13 participants
![Page 16: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/16.jpg)
Image Selection
• Gaze selection of an image is determined by the sum of all fixations of 80ms or more on that image exceeding a threshold.
• Two thresholds 400ms and 800ms
• Successive sets of 15 images are retrieved based on their similarity with selected image.
• Performance compared with images randomly retrieved
• Participants not told what determines screen changes
![Page 17: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/17.jpg)
Target Images
easy to find hard to find
![Page 18: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/18.jpg)
Similarity Links
![Page 19: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/19.jpg)
![Page 20: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/20.jpg)
Results
13 participants8 sessions
Selection Mode Image Type Steps to target 14 Easy-to-find 15 23
Eye gaze Hard-to-find
21 20
Easy-to-find 16 25
Random selection
Hard-to-find 26
Main effect: Eye gaze 18 steps Random 22 steps p < 0.037
![Page 21: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/21.jpg)
Results – Easy vs Hard Images
12
14
16
18
20
22
24
26
Easy HardImage
Ste
ps
to t
arg
et
Eye gaze Random Selection
![Page 22: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/22.jpg)
Other Selection Criteria
Fixation Threshold
Steps to target
Time to target
(seconds)
Average Time per
display
Fixation Numbers
Average Fixation
Numbers per display
300ms 17 17.9 1.081 53 3
400ms 18 28.1 1.630 86 5
Revisit 16 37.7 2.352 99 6
Revisit/400ms 17 24.0 1.470 72 4
24 participants8 sessions
Main effect: fixation threshold not significant
![Page 23: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/23.jpg)
Results - Lower Fixation Thresholds
Fixation Threshold
Steps to target
Time to target (seconds)
Average Time per Display
Fixation Numbers
Average Fixation Numbers per
Display
100ms 20 8.0 0.394 20 1
200ms 12 7.0 0.634 18 2
300ms 4 5.2 1.139 17 3
6 participants3 sessions
Significant differences between random and 200ms + 300ms.
![Page 24: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/24.jpg)
Results - Lower Fixation Thresholds
0
5
10
15
20
25
30
100ms 200ms 300msFixation Threshold
Ste
ps
to t
arg
et
Eye gaze Random Selection
![Page 25: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/25.jpg)
Conclusions
Eye tracking can be faster than tactile interfaces for visual tasks
Eye tracking interfaces are feasible for fast image search
Pre-attentive vision plays a part in very rapid search
![Page 26: Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London](https://reader036.vdocuments.mx/reader036/viewer/2022062519/56649d2b5503460f94a008f4/html5/thumbnails/26.jpg)
Future Work
Further study of human visual behaviour
Use of higher performance similarity measures
Application to browsing large collections of photos/videos
Shared interaction