eye-based interaction in graphical systems: theory & practice andrew duchowski computer science...

Eye-Based Interaction in Graphical Systems: Eye-Based Interaction in Graphical Systems: Theory & PracticeTheory & Practice


Andrew DuchowskiAndrew DuchowskiComputer ScienceComputer Science

Clemson UniversityClemson University

[email protected]@cs.clemson.edu

Roel VertegaalRoel VertegaalComputing and Information ScienceComputing and Information Science

Queen’s UniversityQueen’s University

[email protected]@acm.org

Fig.1: ScanpathsFig.1: Scanpaths

OverviewOverviewOverviewOverview

• Basics of visual attention, human vision, Basics of visual attention, human vision, eye movements and signal analysiseye movements and signal analysis

• Eye tracking hardware specificationsEye tracking hardware specifications

• Video eye tracker integrationVideo eye tracker integration

• Principal eye tracking system modesPrincipal eye tracking system modes• interactive or diagnosticinteractive or diagnostic

• Example systems and potential applicationsExample systems and potential applications

Course Schedule: Part ICourse Schedule: Part ICourse Schedule: Part ICourse Schedule: Part I

• Introduction to the Human Visual System Introduction to the Human Visual System (HVS)(HVS)• Neurological substrate of the HVSNeurological substrate of the HVS

• Physiological and functional descriptionsPhysiological and functional descriptions

• Visual PerceptionVisual Perception

• Spatial, temporal, color visionSpatial, temporal, color vision

• Eye MovementsEye Movements

• Saccades, fixations, pursuits, nystagmusSaccades, fixations, pursuits, nystagmus

Course Schedule: Part IICourse Schedule: Part IICourse Schedule: Part IICourse Schedule: Part II

• Part II: Eye tracking systemsPart II: Eye tracking systems• The eye trackerThe eye tracker

• early developmentsearly developments

• video-based eye trackersvideo-based eye trackers

• system usesystem use

• Integration issuesIntegration issues

• application designapplication design

• calibrationcalibration

• data collection / analysisdata collection / analysis

Course Schedule: Part III / DemoCourse Schedule: Part III / DemoCourse Schedule: Part III / DemoCourse Schedule: Part III / Demo

• Part III: Potential applicationsPart III: Potential applicationsVRVR Human FactorsHuman Factors

Collaborative systemsCollaborative systems AdvertisingAdvertising

PsychophysicsPsychophysics DisplaysDisplays

• Demonstration: GAZE Groupware SystemDemonstration: GAZE Groupware System



Part IPart I

Introduction to the Human Visual SystemIntroduction to the Human Visual System

A: Visual AttentionA: Visual AttentionA: Visual AttentionA: Visual Attention

• Latin translation: Latin translation: “Many filtered into few “Many filtered into few for perception”for perception”

• Visual scene inspection is performed Visual scene inspection is performed minutatimminutatim (piecemeal), not (piecemeal), not in totoin toto

““When the things are apprehended by the When the things are apprehended by the sensessenses, , the number of them that can be attended to at the number of them that can be attended to at once is small, once is small, `Pluribus intentus, minor est ad `Pluribus intentus, minor est ad singula sensus' singula sensus' ””

— — William JamesWilliam James

A.1: Visual Attention—chronological A.1: Visual Attention—chronological reviewreviewA.1: Visual Attention—chronological A.1: Visual Attention—chronological reviewreview

• Qualitative historical background: a Qualitative historical background: a dichotomous theory of attention—the dichotomous theory of attention—the “what” and “where” of (visual) attention“what” and “where” of (visual) attention• Von Helmholtz (ca. 1900): mainly concerned with eye Von Helmholtz (ca. 1900): mainly concerned with eye

movements to spatial locations, the “where”, I.e., movements to spatial locations, the “where”, I.e., attention as overt mechanism (eye movements)attention as overt mechanism (eye movements)

• James (ca. 1900): defined attention mainly in terms of James (ca. 1900): defined attention mainly in terms of the “what”, i.e., attention as a more internally covert the “what”, i.e., attention as a more internally covert mechanismmechanism

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)


• Broadbent (ca. 1950): defined attention as “selective Broadbent (ca. 1950): defined attention as “selective filter” from auditory experiments; generally agreeing filter” from auditory experiments; generally agreeing with Von Helmholtz’s “where”with Von Helmholtz’s “where”

• Deutsch and Deutsch (ca. 1960): rejected “selective Deutsch and Deutsch (ca. 1960): rejected “selective filter” in favor of “importance weightings”; generally filter” in favor of “importance weightings”; generally corresponding to James’ “what”corresponding to James’ “what”

• Treisman (ca. 1960): proposed unified theory of Treisman (ca. 1960): proposed unified theory of attention—attenuation filter (the “where”) followed by attention—attenuation filter (the “where”) followed by “dictionary units” (the “what”)“dictionary units” (the “what”)



• Main debate at this point: is attention Main debate at this point: is attention parallel (the “where”) or serial (the “what”) parallel (the “where”) or serial (the “what”) in nature?in nature?

• Gestalt view: recognition is a wholistic Gestalt view: recognition is a wholistic process (e.g., Kanizsa figure)process (e.g., Kanizsa figure)

• Theories advanced through early Theories advanced through early recordings of eye movementsrecordings of eye movements



• Yarbus (ca. 1967): demonstrated sequential, but Yarbus (ca. 1967): demonstrated sequential, but variable, viewing patterns over particular image variable, viewing patterns over particular image regions (akin to the “what”)regions (akin to the “what”)

• Noton and Stark (ca. 1970): showed that subjects Noton and Stark (ca. 1970): showed that subjects tend to fixate identifiable regions of interest, tend to fixate identifiable regions of interest, containing “informative details”; coined term containing “informative details”; coined term “scanpath” describing eye movement patterns“scanpath” describing eye movement patterns

• Scanpaths helped cast doubt on the Gestalt Scanpaths helped cast doubt on the Gestalt hypothesishypothesis



Fig.2: Yarbus’ early scanpath Fig.2: Yarbus’ early scanpath recording:recording:

• trace 1: examine at willtrace 1: examine at will

• trace 2: estimate wealthtrace 2: estimate wealth

• trace 3: estimate agestrace 3: estimate ages

• trace 4: guess previous activitytrace 4: guess previous activity

• trace 5: remember clothingtrace 5: remember clothing

• trace 6: remember positiontrace 6: remember position

• trace 7: time since last visittrace 7: time since last visit



• Posner (ca. 1980): proposed attentional “spotlight”, Posner (ca. 1980): proposed attentional “spotlight”, an overt mechanism independent from eye an overt mechanism independent from eye movements (akin to the “where”)movements (akin to the “where”)

• Treisman (ca. 1986): once again unified “what” and Treisman (ca. 1986): once again unified “what” and “where” dichotomy by proposing the Feature “where” dichotomy by proposing the Feature Integration Theory (FIT), describing attention as a Integration Theory (FIT), describing attention as a “glue” which integrates features at particular “glue” which integrates features at particular locations to allow wholistic perceptionlocations to allow wholistic perception



• Summary: the “what” and “where” Summary: the “what” and “where” dichotomy provides an intuitive sense of dichotomy provides an intuitive sense of attentional, foveo-peripheral visual attentional, foveo-peripheral visual mechanismmechanism

• Caution: the “what/where” account is Caution: the “what/where” account is probably overly simplistic and is but one probably overly simplistic and is but one theory of visual attentiontheory of visual attention

B: Neurological Substrate of the B: Neurological Substrate of the Human Visual System (HVS)Human Visual System (HVS)B: Neurological Substrate of the B: Neurological Substrate of the Human Visual System (HVS)Human Visual System (HVS)

• Any theory of visual attention must Any theory of visual attention must address the fundamental properties of address the fundamental properties of early visual mechanismsearly visual mechanisms

• Examination of the neurological substrate Examination of the neurological substrate provides evidence of limited information provides evidence of limited information capacity of the visual system—a capacity of the visual system—a physiological reason for an attentional physiological reason for an attentional mechanismmechanism

B.1: The EyeB.1: The EyeB.1: The EyeB.1: The Eye

Fig. 3: The eye—“the world’s Fig. 3: The eye—“the world’s worst camera”worst camera”

• suffers from numerous suffers from numerous optical imperfections...optical imperfections...

• ...endowed with several ...endowed with several compensatory compensatory mechanismsmechanisms

B.1: The Eye B.1: The Eye (cont’d)(cont’d)B.1: The Eye B.1: The Eye (cont’d)(cont’d)

Fig. 4: Ocular opticsFig. 4: Ocular optics

B.1: The Eye B.1: The Eye (cont’d)(cont’d)B.1: The Eye B.1: The Eye (cont’d)(cont’d)

• Imperfections:Imperfections:• spherical abberationsspherical abberations

• chromatic abberationschromatic abberations

• curvature of fieldcurvature of field

• Compensations:Compensations:• iris—acts as a stopiris—acts as a stop

• focal lens—sharp focusfocal lens—sharp focus

• curved retina—matches curved retina—matches curvature of fieldcurvature of field

B.2: The RetinaB.2: The RetinaB.2: The RetinaB.2: The Retina

• Retinal photoreceptors constitute first Retinal photoreceptors constitute first stage of visual perceptionstage of visual perception

• Photoreceptors Photoreceptors transducers converting transducers converting light energy to electrical impulses (neural light energy to electrical impulses (neural signals)signals)

• Photoreceptors are functionally classified Photoreceptors are functionally classified into two types: into two types: rodsrods and and conescones

B.2: The Retina—rods and conesB.2: The Retina—rods and conesB.2: The Retina—rods and conesB.2: The Retina—rods and cones

• Rods: sensitive to dim and achromatic Rods: sensitive to dim and achromatic light (night vision)light (night vision)

• Cones: respond to brighter, chromatic Cones: respond to brighter, chromatic light (day vision)light (day vision)

• Retinal construction: 120M rods, 7M cones Retinal construction: 120M rods, 7M cones arranged concentricallyarranged concentrically

B.2: The Retina—cellular makeupB.2: The Retina—cellular makeupB.2: The Retina—cellular makeupB.2: The Retina—cellular makeup

• The retina is composed of 3 main layers of The retina is composed of 3 main layers of different cell types (a 3-layer “sandwich”)different cell types (a 3-layer “sandwich”)

• Surprising fact: the retina is “inverted”— Surprising fact: the retina is “inverted”— photoreceptors are found in the bottom photoreceptors are found in the bottom layer (furthest away from incoming light)layer (furthest away from incoming light)

• Connection bundles between layers are Connection bundles between layers are called called plexiformplexiform or or synaptic layerssynaptic layers

B.2: The Retina—cellular makeup B.2: The Retina—cellular makeup (cont’d)(cont’d)B.2: The Retina—cellular makeup B.2: The Retina—cellular makeup (cont’d)(cont’d)

Fig.5: The retinocellular Fig.5: The retinocellular layers (w.r.t. incoming layers (w.r.t. incoming light):light):

• ganglion layerganglion layer

• inner synaptic inner synaptic plexiform layerplexiform layer

• inner nuclear layerinner nuclear layer

• outer synaptic outer synaptic plexiform layerplexiform layer

• outer layerouter layer

B.2: The Retina—cellular makeup B.2: The Retina—cellular makeup (cont’d)(cont’d)B.2: The Retina—cellular makeup B.2: The Retina—cellular makeup (cont’d)(cont’d)

Fig.5 (cont’d): The neuron:Fig.5 (cont’d): The neuron:

• all retinal cells are types all retinal cells are types of neuronsof neurons

• certain neurons mimic a certain neurons mimic a “digital gate”, firing when “digital gate”, firing when activation level exceeds a activation level exceeds a thresholdthreshold

• rods and cones are rods and cones are specific types of specific types of dendritesdendrites

B.2: The Retina—retinogeniculate B.2: The Retina—retinogeniculate organization organization (from outside in, w.r.t. cortex)(from outside in, w.r.t. cortex)

B.2: The Retina—retinogeniculate B.2: The Retina—retinogeniculate organization organization (from outside in, w.r.t. cortex)(from outside in, w.r.t. cortex)

• Outer layerOuter layer: rods and cones: rods and cones

• Inner layerInner layer: horizontal cells, laterally : horizontal cells, laterally connected to photoreceptorsconnected to photoreceptors

• Ganglion layerGanglion layer: ganglion cells, connected : ganglion cells, connected (indirectly) to horizontal cells, project via (indirectly) to horizontal cells, project via the myelinated pathways, to the Lateral the myelinated pathways, to the Lateral Geniculate Nuclei (LGN) in the cortexGeniculate Nuclei (LGN) in the cortex

B.2: The Retina—receptive fieldsB.2: The Retina—receptive fieldsB.2: The Retina—receptive fieldsB.2: The Retina—receptive fields

• Receptive fields: collections of Receptive fields: collections of interconnected cells within the inner and interconnected cells within the inner and ganglion layersganglion layers

• Field organization determines impulse Field organization determines impulse signature of cells, based on cell typessignature of cells, based on cell types

• Cells may depolarize due to light Cells may depolarize due to light increments (+) or decrements (-)increments (+) or decrements (-)

B.2: The Retina—receptive fields B.2: The Retina—receptive fields (cont’d)(cont’d)B.2: The Retina—receptive fields B.2: The Retina—receptive fields (cont’d)(cont’d)

Fig.6: Receptive fields:Fig.6: Receptive fields:

• signal profile signal profile resembles a resembles a “Mexican hat”“Mexican hat”

• receptive field receptive field sizes vary sizes vary concentricallyconcentrically

• color-opposing color-opposing fields also existfields also exist

B.3: Visual PathwaysB.3: Visual PathwaysB.3: Visual PathwaysB.3: Visual Pathways

• Retinal ganglion cells project to the LGN Retinal ganglion cells project to the LGN along two major pathways, distinguished along two major pathways, distinguished by morphological cell types: by morphological cell types: and and cells cells cells project to the cells project to the magnocellularmagnocellular (M-) layers (M-) layers

cells project to the cells project to the parvocellularparvocellular (P-) layers (P-) layers

• Ganglion cells are functionally classified Ganglion cells are functionally classified by three types: X, Y, and W cellsby three types: X, Y, and W cells

B.3: Visual Pathways—functional B.3: Visual Pathways—functional response of ganglion cellsresponse of ganglion cellsB.3: Visual Pathways—functional B.3: Visual Pathways—functional response of ganglion cellsresponse of ganglion cells

• X cells: sustained stimulus, location, and X cells: sustained stimulus, location, and fine detailfine detail• nervate along both M- and P- projectionsnervate along both M- and P- projections

• Y cells: transient stimulus, coarse features, Y cells: transient stimulus, coarse features, and motionand motion• nervate along only the M-projectionnervate along only the M-projection

• W cells: coarse features and motionW cells: coarse features and motion• project to the Superior Colliculus (SC)project to the Superior Colliculus (SC)

B.3: Visual Pathways B.3: Visual Pathways (cont’d)(cont’d)B.3: Visual Pathways B.3: Visual Pathways (cont’d)(cont’d)

Fig.7: Optic tract and radiations Fig.7: Optic tract and radiations (visual pathways):(visual pathways):

• The LGN is of particular The LGN is of particular clinical importanceclinical importance

• M- and P-cellular M- and P-cellular projections are clearly projections are clearly visible under microscopevisible under microscope

• Axons from M- and P-layers Axons from M- and P-layers of the LGN terminate in of the LGN terminate in area V1area V1

B.3: Visual Pathways B.3: Visual Pathways (cont’d)(cont’d)B.3: Visual Pathways B.3: Visual Pathways (cont’d)(cont’d)

Table.1: Functional characteristics of ganglionic Table.1: Functional characteristics of ganglionic projectionsprojections

Characteristics Magno Parvoganglion size large smalltransmission time fast slowreceptive fields large smallsensitivity to small objects poor goodsensitivity to change in light levels large smallsensitivity to contrast low highsensitivity to motion high lowcolor discrimination no yes

B.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyondB.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond

Fig.8: The brain Fig.8: The brain and visual and visual pathways:pathways:

• the cerebral the cerebral cortex is cortex is composed of composed of numerous numerous regions regions classified by classified by their functiontheir function

B.4: The Occipital Cortex and B.4: The Occipital Cortex and Beyond Beyond (cont’d)(cont’d)

B.4: The Occipital Cortex and B.4: The Occipital Cortex and Beyond Beyond (cont’d)(cont’d)

• M- and P- pathways terminate in distinct M- and P- pathways terminate in distinct layers of cortical area V1layers of cortical area V1

• Cortical cells (unlike center-surround Cortical cells (unlike center-surround ganglion receptive fields) respond to ganglion receptive fields) respond to orientation-specific stimulusorientation-specific stimulus

• Pathways emanating from V1 joining Pathways emanating from V1 joining multiple cortical areas involved in vision multiple cortical areas involved in vision are called are called streamsstreams

B.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——directional selectivitydirectional selectivityB.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——directional selectivitydirectional selectivity

• Cortical Directional Selectivity (CDS) of cells Cortical Directional Selectivity (CDS) of cells in V1 contributes to motion perception and in V1 contributes to motion perception and control of eye movementscontrol of eye movements

• CDS cells establish a motion pathway from CDS cells establish a motion pathway from V1 projecting to areas V2 and MT (V5)V1 projecting to areas V2 and MT (V5)

• In contrast, Retinal Directional Selectivity In contrast, Retinal Directional Selectivity (RDS) may not contribute to motion (RDS) may not contribute to motion perception, but is involved in eye movementsperception, but is involved in eye movements

B.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——cortical cellscortical cellsB.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——cortical cellscortical cells

• Two consequences of visual system’s Two consequences of visual system’s motion-sensitive, single-cell organization:motion-sensitive, single-cell organization:• due to motion sensitivity, eye movements are never due to motion sensitivity, eye movements are never

perfectly still (instead tiny jitter is observed, termed perfectly still (instead tiny jitter is observed, termed microsaccademicrosaccade)—if eyes were stabilized, image would )—if eyes were stabilized, image would fade!fade!

• due to single-cell organization, representation of due to single-cell organization, representation of natural images is quite abstract: there is no “retinal natural images is quite abstract: there is no “retinal buffer”buffer”

B.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——2 attentional streams2 attentional streamsB.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——2 attentional streams2 attentional streams

• Dorsal stream:Dorsal stream:• V1, V2, MT (V5), MST, Posterior Parietal CortexV1, V2, MT (V5), MST, Posterior Parietal Cortex

• sensorimotor (motion, location) processingsensorimotor (motion, location) processing

• the attentional “where”?the attentional “where”?

• Ventral (temporal) stream:Ventral (temporal) stream:• V1, V2, V4, Inferotemporal CortexV1, V2, V4, Inferotemporal Cortex

• cognitive processingcognitive processing

• the attentional “what”?the attentional “what”?

B.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——3 attentional regions3 attentional regionsB.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——3 attentional regions3 attentional regions

• Posterior Parietal Cortex (dorsal stream):Posterior Parietal Cortex (dorsal stream):• disengages attentiondisengages attention

• Superior Colliculus (midbrain):Superior Colliculus (midbrain):• relocates attentionrelocates attention

• Pulvinar (thalamus; colocated with LGN):Pulvinar (thalamus; colocated with LGN):• engages, or enhances, attentionengages, or enhances, attention

C: Visual Perception C: Visual Perception (with emphasis on foveo-(with emphasis on foveo-

peripheral distinction)peripheral distinction)C: Visual Perception C: Visual Perception (with emphasis on foveo-(with emphasis on foveo-

peripheral distinction)peripheral distinction)

• Measurable performance parameters may Measurable performance parameters may often (but not always!) fall within ranges often (but not always!) fall within ranges predicted by known limitations of the predicted by known limitations of the neurological substrateneurological substrate

• Example: visual acuity may be estimated by Example: visual acuity may be estimated by knowledge of density and distribution of the knowledge of density and distribution of the retinal photoreceptorsretinal photoreceptors

• In general, performance parameters are In general, performance parameters are obtained empiricallyobtained empirically

• Main parameters sought: visual acuity, Main parameters sought: visual acuity, contrast sensitivitycontrast sensitivity

• Dimensions of retinal features are measured Dimensions of retinal features are measured in terms of projected scene onto retina in in terms of projected scene onto retina in units of degrees visual angle,units of degrees visual angle,

where where SS is the object size and is the object size and DD is distance is distance

C.1: Spatial VisionC.1: Spatial VisionC.1: Spatial VisionC.1: Spatial Vision

D

SA

22arctan

C.1: Spatial Vision—visual angleC.1: Spatial Vision—visual angleC.1: Spatial Vision—visual angleC.1: Spatial Vision—visual angle

Fig.9: Visual angleFig.9: Visual angle

C.1: Spatial Vision—common visual C.1: Spatial Vision—common visual anglesanglesC.1: Spatial Vision—common visual C.1: Spatial Vision—common visual anglesangles

Table 2: Common visual anglesTable 2: Common visual angles

Object Distance Angle subtended

thumbnail arm’s length 1.5-2 degsun or moon - .5 degUS quarter coin arm’s length 2 degUS quarter coin 85 m 1 minUS quarter coin 5 km 1 sec

C.1: Spatial Vision—retinal regionsC.1: Spatial Vision—retinal regionsC.1: Spatial Vision—retinal regionsC.1: Spatial Vision—retinal regions

• Visual field: 180° horiz. Visual field: 180° horiz. 130° vert. 130° vert.• Fovea Centralis (foveola): highest acuityFovea Centralis (foveola): highest acuity

• 1.3° visual angle; 25,000 cones1.3° visual angle; 25,000 cones

• Fovea: high acuity (at 5°, acuity drops to 50%)Fovea: high acuity (at 5°, acuity drops to 50%)

• 5° visual angle; 100,000 cones5° visual angle; 100,000 cones

• Macula: within “useful” acuity region (to about 30°)Macula: within “useful” acuity region (to about 30°)

• 16.7° visual angle; 650,000 cones16.7° visual angle; 650,000 cones

• Hardly any rods in the foveal regionHardly any rods in the foveal region

C.1: Spatial Vision—visual angle and C.1: Spatial Vision—visual angle and receptor distributionreceptor distributionC.1: Spatial Vision—visual angle and C.1: Spatial Vision—visual angle and receptor distributionreceptor distribution

Fig.10: Retinotopic receptor distributionFig.10: Retinotopic receptor distribution

C.1: Spatial Vision—visual acuityC.1: Spatial Vision—visual acuityC.1: Spatial Vision—visual acuityC.1: Spatial Vision—visual acuity

Fig.11: Visual acuity at Fig.11: Visual acuity at eccentricities and light levels:eccentricities and light levels:

• at photopic (day) light levels, at photopic (day) light levels, acuity is fairly constant acuity is fairly constant within central 2°within central 2°

• acuity drops of linearly to 5°; acuity drops of linearly to 5°; drops sharply (exp.) beyonddrops sharply (exp.) beyond

• at scotopic (night) light at scotopic (night) light levels, acuity is poor at all levels, acuity is poor at all eccentricitieseccentricities

C.1: Spatial Vision—measuring C.1: Spatial Vision—measuring visual acuityvisual acuityC.1: Spatial Vision—measuring C.1: Spatial Vision—measuring visual acuityvisual acuity

• Acuity roughly corresponds to foveal Acuity roughly corresponds to foveal receptor distribution in the fovea, but not receptor distribution in the fovea, but not necessarily in the peripherynecessarily in the periphery

• Due to various contributing factors Due to various contributing factors (synaptic organization and later-stage (synaptic organization and later-stage neural elements), effective relative visual neural elements), effective relative visual acuity is generally measured by acuity is generally measured by psychophysical experimentationpsychophysical experimentation

C.2: Temporal VisionC.2: Temporal VisionC.2: Temporal VisionC.2: Temporal Vision

• Visual response to motion is characterized by Visual response to motion is characterized by two distinct facts: two distinct facts: persistence of visionpersistence of vision (POV) (POV) and the and the phi phenomenonphi phenomenon

• POV: essentially describes human temporal POV: essentially describes human temporal sampling ratesampling rate

• Phi: describes threshold above which humans Phi: describes threshold above which humans detect detect apparent movementapparent movement

• Both facts exploited in media to elicit motion Both facts exploited in media to elicit motion perceptionperception

C.2: Temporal Vision—persistence of C.2: Temporal Vision—persistence of visionvisionC.2: Temporal Vision—persistence of C.2: Temporal Vision—persistence of visionvision

Fig.12: Critical Fusion Frequency:Fig.12: Critical Fusion Frequency:

• stimulus flashing at about stimulus flashing at about 50-60Hz appears steady50-60Hz appears steady

• CFF explains why flicker is CFF explains why flicker is not seen when viewing not seen when viewing sequence of still imagessequence of still images

• cinema: 24 fps cinema: 24 fps 3 = 72Hz 3 = 72Hz due to 3-bladed shutterdue to 3-bladed shutter

• TV: 60 fields/sec, interlacedTV: 60 fields/sec, interlaced

C.2: Temporal Vision—phi C.2: Temporal Vision—phi phenomenonphenomenonC.2: Temporal Vision—phi C.2: Temporal Vision—phi phenomenonphenomenon

• Phi phenomenon explains why motion is Phi phenomenon explains why motion is perceived in cinema, TV, graphicsperceived in cinema, TV, graphics

• Besides necessary flicker rate (60Hz), Besides necessary flicker rate (60Hz), illusion of apparent, or stroboscopic, illusion of apparent, or stroboscopic, motion must be maintainedmotion must be maintained

• Similar to old-fashioned neon signs with Similar to old-fashioned neon signs with stationary bulbsstationary bulbs

• Minimum rate: 16 frames per secondMinimum rate: 16 frames per second

C.2: Temporal Vision—peripheral C.2: Temporal Vision—peripheral motion perceptionmotion perceptionC.2: Temporal Vision—peripheral C.2: Temporal Vision—peripheral motion perceptionmotion perception

• Motion perception is not homogeneous across Motion perception is not homogeneous across visual fieldvisual field

• Sensitivity to target motion decreases with Sensitivity to target motion decreases with retinal eccentricity for slow motion...retinal eccentricity for slow motion...• higher rate of target motion (e.g., spinning disk) is higher rate of target motion (e.g., spinning disk) is

needed to match apparent velocity in foveaneeded to match apparent velocity in fovea

• ……but, motion is more salient in periphery than but, motion is more salient in periphery than in fovea (easier to detect moving targets than in fovea (easier to detect moving targets than stationary ones)stationary ones)

C.2: Temporal Vision—peripheral C.2: Temporal Vision—peripheral sensitivity to direction of motionsensitivity to direction of motionC.2: Temporal Vision—peripheral C.2: Temporal Vision—peripheral sensitivity to direction of motionsensitivity to direction of motion

Fig.13: Threshold isograms for Fig.13: Threshold isograms for peripheral rotary movement:peripheral rotary movement:

• periphery is twice as periphery is twice as sensitive to horizontal-sensitive to horizontal-axis movement as to axis movement as to vertical-axis movementvertical-axis movement

• (numbers in diagram (numbers in diagram are rates of pointer are rates of pointer movement in rev./min.)movement in rev./min.)

C.3: Color Vision—cone typesC.3: Color Vision—cone typesC.3: Color Vision—cone typesC.3: Color Vision—cone types

Fig.14: Spectral sensitivity curves Fig.14: Spectral sensitivity curves of cone photoreceptorsof cone photoreceptors

• foveal color vision is foveal color vision is facilitated by three types of facilitated by three types of cone photorecptorscone photorecptors

• a good deal is known about a good deal is known about foveal color vision, relatively foveal color vision, relatively little is known about little is known about peripheral color visionperipheral color vision

• of the 7,000,000 cones, of the 7,000,000 cones, most are packed tightly into most are packed tightly into the central 30° foveal regionthe central 30° foveal region

C.3: Color Vision—peripheral color C.3: Color Vision—peripheral color perception fieldsperception fieldsC.3: Color Vision—peripheral color C.3: Color Vision—peripheral color perception fieldsperception fields

Fig.15: Visual fields for monocular Fig.15: Visual fields for monocular color vision (right eye)color vision (right eye)

• blueblue and and yellowyellow fields are fields are larger than larger than redred and and greengreen fieldsfields

• most sensitive to most sensitive to blueblue, up , up to 83°; to 83°; redred up to 76°; up to 76°; greengreen up to 74° up to 74°

• chromatic fields do not chromatic fields do not have definite borders, have definite borders, sensitivity gradually and sensitivity gradually and irregularly drops off over irregularly drops off over 15-30° range15-30° range

C.4: Implications for Design of C.4: Implications for Design of Attentional DisplaysAttentional DisplaysC.4: Implications for Design of C.4: Implications for Design of Attentional DisplaysAttentional Displays

• Need to consider distinct characteristics of Need to consider distinct characteristics of foveal and peripheral vision, in particular:foveal and peripheral vision, in particular:• spatial resolutionspatial resolution

• temporal resolutiontemporal resolution

• luminance / chrominanceluminance / chrominance

• Furthermore, Furthermore, gaze-contingentgaze-contingent systems must systems must match dynamics of human eye movementmatch dynamics of human eye movement

D: Taxonomy and Models of Eye D: Taxonomy and Models of Eye MovementsMovementsD: Taxonomy and Models of Eye D: Taxonomy and Models of Eye MovementsMovements

• Eye movements are mainly used to Eye movements are mainly used to reposition the foveareposition the fovea

• Five main classes of eye movements:Five main classes of eye movements:

• saccadicsaccadic

• smooth pursuitsmooth pursuit

• vergencevergence

• vestibularvestibular

• physiological nystagmusphysiological nystagmus

• (fixations)(fixations)

• Other types of movements are non-Other types of movements are non-positional positional (adaptation, accommodation)(adaptation, accommodation)

D.1: Extra-Ocular MusclesD.1: Extra-Ocular MusclesD.1: Extra-Ocular MusclesD.1: Extra-Ocular Muscles

Fig.16: Extrinsic muscles of the eyes:Fig.16: Extrinsic muscles of the eyes:

• in general, eyes move within 6 degrees of freedom (6 in general, eyes move within 6 degrees of freedom (6 muscles)muscles)

D.1: Oculomotor PlantD.1: Oculomotor PlantD.1: Oculomotor PlantD.1: Oculomotor Plant

Fig.17: Oculomotor system:Fig.17: Oculomotor system:

• eye movement signals eye movement signals emanate from three emanate from three main distinct regions:main distinct regions:

• occipital cortex (areas occipital cortex (areas 17, 18, 19, 22)17, 18, 19, 22)

• superior colliculus (SC)superior colliculus (SC)

• semicircular canals semicircular canals (SCC)(SCC)

D.1: Oculomotor Plant D.1: Oculomotor Plant (cont’d)(cont’d)D.1: Oculomotor Plant D.1: Oculomotor Plant (cont’d)(cont’d)

• Two pertinent observations:Two pertinent observations:1 eye movement system is, to a large extent, a eye movement system is, to a large extent, a

feedback circuitfeedback circuit

2 controlling cortical regions can be functionally controlling cortical regions can be functionally characterized as:characterized as:

• voluntary (occipital cortex—areas 17, 18, 19, 22)voluntary (occipital cortex—areas 17, 18, 19, 22)

• involuntary (superior colliculus, SC)involuntary (superior colliculus, SC)

• reflexive (semicircular canals, SCC)reflexive (semicircular canals, SCC)

D.2: SaccadesD.2: SaccadesD.2: SaccadesD.2: Saccades

• Rapid eye movements used to reposition Rapid eye movements used to reposition foveafovea

• Voluntary and reflexiveVoluntary and reflexive

• Range in duration from 10ms - 100msRange in duration from 10ms - 100ms

• Effectively blind during transitionEffectively blind during transition

• Deemed Deemed ballisticballistic (pre-programmed) and (pre-programmed) and stereotyped stereotyped (reproducible)(reproducible)

D.2: Saccades—modelingD.2: Saccades—modelingD.2: Saccades—modelingD.2: Saccades—modeling

Fig.18: Linear moving average filter model:Fig.18: Linear moving average filter model:

• sstt = input (pulse), = input (pulse), xxtt = output (step), = output (step), ggkk = filter coefficients = filter coefficients

• e.g., Haar filter {1,-1}e.g., Haar filter {1,-1}

0

110

kktk

ttt

sg

sgsgx

D.3: Smooth PursuitsD.3: Smooth PursuitsD.3: Smooth PursuitsD.3: Smooth Pursuits

• Involved when visually tracking a moving Involved when visually tracking a moving targettarget

• Depending on range of target motion, eyes Depending on range of target motion, eyes are capable of matching target velocityare capable of matching target velocity

• Pursuit movements are an example of a Pursuit movements are an example of a control system with built-in negative control system with built-in negative feedbackfeedback

D.3: Smooth Pursuits—modelingD.3: Smooth Pursuits—modelingD.3: Smooth Pursuits—modelingD.3: Smooth Pursuits—modeling

Fig.19: Linear, time-invariant filter model:Fig.19: Linear, time-invariant filter model:

• sstt = target position, = target position, xxtt = (desired) eye position, = (desired) eye position, hh = filter = filter

• retinal receptors give additive velocity errorretinal receptors give additive velocity error

1 ttt xxsh )(

D.4: NystagmusD.4: NystagmusD.4: NystagmusD.4: Nystagmus

• Conjugate eye movements characterized by Conjugate eye movements characterized by sawtooth-like time course pattern (pursuits sawtooth-like time course pattern (pursuits interspersed with saccades)interspersed with saccades)

• Two types (virtually indistinguishable):Two types (virtually indistinguishable):• Optokinetic: compensation for retinal movement of targetOptokinetic: compensation for retinal movement of target

• Vestibular: compensation for head movementVestibular: compensation for head movement

• May be possible to model with combination May be possible to model with combination of saccade/pursuit filtersof saccade/pursuit filters

D.5: FixationsD.5: FixationsD.5: FixationsD.5: Fixations

• Possibly the most important type of eye Possibly the most important type of eye movement for attentional applicationsmovement for attentional applications• 90% viewing time is devoted to fixations90% viewing time is devoted to fixations

• duration: 150ms - 600msduration: 150ms - 600ms

• Not technically eye movements in their Not technically eye movements in their own right, rather characterized by own right, rather characterized by miniature eye movements:miniature eye movements:• tremor, drift, microsaccadestremor, drift, microsaccades

D.6: Eye Movement AnalysisD.6: Eye Movement AnalysisD.6: Eye Movement AnalysisD.6: Eye Movement Analysis

• Two significant observations:Two significant observations:1 only three types of eye movements are mainly only three types of eye movements are mainly

needed to gain insight into overt localization of visual needed to gain insight into overt localization of visual attention:attention:

• fixationsfixations

• saccadessaccades

• smooth pursuits (to a lesser extent)smooth pursuits (to a lesser extent)

2 all three signals may be approximated by linear, all three signals may be approximated by linear, time-invariant (LTI) filter systemstime-invariant (LTI) filter systems

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—assumptionsassumptionsD.6: Eye Movement Analysis—D.6: Eye Movement Analysis—assumptionsassumptions

• Important point: it is assumed observed eye Important point: it is assumed observed eye movements disclose evidence of movements disclose evidence of overtovert visual visual attentionattention• it is possible to attend to objects it is possible to attend to objects covertlycovertly (without (without

moving eyes)moving eyes)

• Linearity: although practical, this assumption Linearity: although practical, this assumption is an operational oversimplification of is an operational oversimplification of neuronal (non-linear) systemsneuronal (non-linear) systems

D.6: Eye Movement Analysis—goalsD.6: Eye Movement Analysis—goalsD.6: Eye Movement Analysis—goalsD.6: Eye Movement Analysis—goals

• goal of analysis is to locate goal of analysis is to locate regions where signal regions where signal average changes abruptlyaverage changes abruptly• fixation end, saccade startfixation end, saccade start• saccade end, fixation startsaccade end, fixation start

• two main approaches:two main approaches:• summation-basedsummation-based• differentiation-baseddifferentiation-based

• both approaches rely on both approaches rely on empirical thresholdsempirical thresholds

Fig.20: Hypothetical eye movement signalFig.20: Hypothetical eye movement signal

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—denoisingdenoisingD.6: Eye Movement Analysis—D.6: Eye Movement Analysis—denoisingdenoising

Fig.21: Signal denoising—reduce noise due to:Fig.21: Signal denoising—reduce noise due to:• eye instability (jitter), or worse, blinkseye instability (jitter), or worse, blinks• removal possible based on device characteristics (e.g., blink = removal possible based on device characteristics (e.g., blink =

[0,0])[0,0])

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—summation basedsummation basedD.6: Eye Movement Analysis—D.6: Eye Movement Analysis—summation basedsummation based

• Dwell-time fixation detection depends on:Dwell-time fixation detection depends on:• identification of a stationary signal (fixation), andidentification of a stationary signal (fixation), and

• size of time window specifying range of duration size of time window specifying range of duration (and hence temporal threshold)(and hence temporal threshold)

• Example: Example: position-varianceposition-variance method: method:• determine whether determine whether MM of of NN points lie within a certain points lie within a certain

distance distance DD of the mean ( of the mean () of the signal) of the signal

• values values MM, , NN, and , and DD are determined empirically are determined empirically

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—differentiation baseddifferentiation basedD.6: Eye Movement Analysis—D.6: Eye Movement Analysis—differentiation baseddifferentiation based

• Velocity-based saccade/fixation detection:Velocity-based saccade/fixation detection:• calculated velocity (over signal window) is compared calculated velocity (over signal window) is compared

to thresholdto threshold

• if velocity > threshold then saccade, else fixationif velocity > threshold then saccade, else fixation

• Example: Example: velocity detectionvelocity detection method: method:• use short Finite Impulse Response (FIR) filters to use short Finite Impulse Response (FIR) filters to

detect saccade (may be possible in real-time)detect saccade (may be possible in real-time)

• assuming symmetrical velocity profile, can extend to assuming symmetrical velocity profile, can extend to velocity-based predictionvelocity-based prediction

D.6: Eye Movement Analysis D.6: Eye Movement Analysis (cont’d)(cont’d)D.6: Eye Movement Analysis D.6: Eye Movement Analysis (cont’d)(cont’d)

Fig.22: Saccade/fixation detectionFig.22: Saccade/fixation detection

(a) position-variance(a) position-variance (b) velocity-detection(b) velocity-detection

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—exampleexampleD.6: Eye Movement Analysis—D.6: Eye Movement Analysis—exampleexample

Fig.23: FIR filter velocity-detection Fig.23: FIR filter velocity-detection method based on idealized method based on idealized saccade detection:saccade detection:

• 4 conditions on measured 4 conditions on measured acceleration:acceleration:

maxmin

)()(

||

||

TIIT

ISgnISgn

BI

AI

12

12

2

1 • acc. > thresh. Aacc. > thresh. A

• acc. > thresh. Bacc. > thresh. B

• sign changesign change

• duration thresh.duration thresh.

• thresholds derived from thresholds derived from empirical valuesempirical values

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—example example (cont’d)(cont’d)

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—example example (cont’d)(cont’d)

• Amplitude thresholds Amplitude thresholds AA, , BB: derived from expected peak : derived from expected peak saccade velocities: 600°/ssaccade velocities: 600°/s

• Duration thresholds Duration thresholds TTminmin, , TTmaxmax: derived from expected : derived from expected

saccade duration: 120ms - 300mssaccade duration: 120ms - 300ms

Fig.24: FIR filters for saccade detectionFig.24: FIR filters for saccade detection



Part IIPart II

Eye Tracking SystemsEye Tracking Systems

E: The Eye TrackerE: The Eye TrackerE: The Eye TrackerE: The Eye Tracker

• Two broad applications of eye movement Two broad applications of eye movement monitoring techniques:monitoring techniques:• measuring position of eye relative to the headmeasuring position of eye relative to the head• measuring orientation of eye in space, or the “point of measuring orientation of eye in space, or the “point of

regard” (POR)—used to identify fixated elements in a regard” (POR)—used to identify fixated elements in a visual scenevisual scene

• Arguably, the most widely used apparatus Arguably, the most widely used apparatus for measuring the POR is the video-based for measuring the POR is the video-based corneal reflection eye trackercorneal reflection eye tracker

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking TechniquesTechniquesE.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking TechniquesTechniques

• Four broad categories of eye movement Four broad categories of eye movement methodologies:methodologies:• electro-oculography (EOG)electro-oculography (EOG)

• scleral contact lens/search coilscleral contact lens/search coil

• photo-oculography (POG) or video-oculography photo-oculography (POG) or video-oculography (VOG)(VOG)

• video-based combined pupil and corneal reflectionvideo-based combined pupil and corneal reflection

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques Techniques (cont’d)(cont’d)

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques Techniques (cont’d)(cont’d)

• First method for objective eye movement First method for objective eye movement measurements using corneal reflection measurements using corneal reflection reported in 1901reported in 1901

• Techniques using contact lenses to improve Techniques using contact lenses to improve accuracy developed in 1950s (invasive)accuracy developed in 1950s (invasive)

• Remote (non-invasive) trackers rely on visible Remote (non-invasive) trackers rely on visible features of the eye (e.g., pupil)features of the eye (e.g., pupil)

• Fast image processing techniques have Fast image processing techniques have facilitated real-time video-based systemsfacilitated real-time video-based systems

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—EOGTechniques—EOGE.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—EOGTechniques—EOG

Fig.25: EOG measurement:Fig.25: EOG measurement:

• relies on measurement of relies on measurement of skin’s potential differences, skin’s potential differences, using electrodes placed using electrodes placed around the eyearound the eye

• most widely used method most widely used method some 20 years ago (still some 20 years ago (still used today)used today)

• similar to electro-similar to electro-mechanical motion-capturemechanical motion-capture

• measures eye movements measures eye movements relative to head positionrelative to head position

• not generally suitable for not generally suitable for POR measurement (unless POR measurement (unless head is also tracked)head is also tracked)

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—Scleral Contact Techniques—Scleral Contact Lens/Search CoilLens/Search Coil

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—Scleral Contact Techniques—Scleral Contact Lens/Search CoilLens/Search Coil

Fig.26: Scleral coil:Fig.26: Scleral coil:

• search coil embedded in search coil embedded in contact lens and contact lens and electromagnetic field electromagnetic field framesframes

• possibly most precisepossibly most precise

• similar to electromagnetic similar to electromagnetic position/orientation trackers position/orientation trackers used in motion-captureused in motion-capture

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—Scleral Contact Techniques—Scleral Contact Lens/Search Coil Lens/Search Coil (cont’d)(cont’d)

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—Scleral Contact Techniques—Scleral Contact Lens/Search Coil Lens/Search Coil (cont’d)(cont’d)

Fig.27: Example of scleral Fig.27: Example of scleral suction ring insertion:suction ring insertion:

• most intrusive methodmost intrusive method

• insertion of lens requires insertion of lens requires carecare

• wearing of lens causes wearing of lens causes discomfortdiscomfort

• highly accurate, but limited highly accurate, but limited measurement range (~5°)measurement range (~5°)

• measures eye movements measures eye movements relative to head positionrelative to head position

• not generally suitable for not generally suitable for POR measurement (unless POR measurement (unless head is also tracked)head is also tracked)

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—POG / VOGTechniques—POG / VOGE.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—POG / VOGTechniques—POG / VOG

Fig.28: Example of POG / VOG methods and devices:Fig.28: Example of POG / VOG methods and devices:

• wide variety of techniques based on measurement of wide variety of techniques based on measurement of distinguishable ocular features (similar to optical “mocap”)distinguishable ocular features (similar to optical “mocap”)• pupil: apparent shapepupil: apparent shape• limbus: position of iris-sclera limbus: position of iris-sclera

boundaryboundary

• infra-red: corneal reflection of infra-red: corneal reflection of directed light sourcedirected light source

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—Video-Based Combined Techniques—Video-Based Combined Pupil and Corneal ReflectionPupil and Corneal Reflection

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—Video-Based Combined Techniques—Video-Based Combined Pupil and Corneal ReflectionPupil and Corneal Reflection

Fig.29: Table-mounted Fig.29: Table-mounted (remote) video-based eye (remote) video-based eye trackertracker

• compute POR, usually in compute POR, usually in real-timereal-time

• utilize relatively cheap video utilize relatively cheap video cameras and image cameras and image processing hardwareprocessing hardware

• can also allow limited head can also allow limited head movementmovement

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—Video-Based Combined Techniques—Video-Based Combined Pupil and Corneal Reflection Pupil and Corneal Reflection (cont’d)(cont’d)

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—Video-Based Combined Techniques—Video-Based Combined Pupil and Corneal Reflection Pupil and Corneal Reflection (cont’d)(cont’d)

Fig.30: Head-mounted video-Fig.30: Head-mounted video-based eye trackerbased eye tracker

• essentially identical to essentially identical to table-mounted systems, table-mounted systems, but with miniature opticsbut with miniature optics

• most suitable for (graphical) most suitable for (graphical) interactive systems, e.g., interactive systems, e.g., VRVR

• binocular systems also binocular systems also availableavailable

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—Corneal ReflectionTechniques—Corneal ReflectionE.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—Corneal ReflectionTechniques—Corneal Reflection

• Two points of reference on the eye are Two points of reference on the eye are needed to separate eye movements from needed to separate eye movements from head movements, e.g.,head movements, e.g.,• pupil centerpupil center• corneal reflection of nearby, directed light source (IR)corneal reflection of nearby, directed light source (IR)

• Positional difference between pupil center Positional difference between pupil center and corneal reflection changes with eye and corneal reflection changes with eye rotation, but remains relatively constant with rotation, but remains relatively constant with minor head movementsminor head movements

E.1: Brief Survey of Eye Tracking E.1: Brief Survey of Eye Tracking Techniques—Corneal Reflection Techniques—Corneal Reflection (cont’d)(cont’d)


Fig.31: Purkinje imagesFig.31: Purkinje images

• corneal reflections are known corneal reflections are known as the Purkinje images, or as the Purkinje images, or reflections:reflections:1 front surface of corneafront surface of cornea

2 rear surface of cornearear surface of cornea

3 front surface of lensfront surface of lens

4 rear surface of lensrear surface of lens• video-based trackers typically video-based trackers typically

locate the first Purkinje imagelocate the first Purkinje image



• Purkinje images appear as Purkinje images appear as small white dots in close small white dots in close proximity to the (dark) pupilproximity to the (dark) pupil

• tracker calibration is tracker calibration is achieved by measuring achieved by measuring user gazing at properly user gazing at properly positioned grid points positioned grid points (usually 5 or 9)(usually 5 or 9)

• tracker interpolates POR tracker interpolates POR on perpendicular screen in on perpendicular screen in front of userfront of user

Fig.32: Pupil and Purkinje Fig.32: Pupil and Purkinje images as seen by eye images as seen by eye tracker’s cameratracker’s camera



Fig.33: Dual-Purkinje image Fig.33: Dual-Purkinje image (DPI) eye tracker(DPI) eye tracker

• so-called generation-V so-called generation-V trackers measure the 1trackers measure the 1stst and and 44thth Purkinje images Purkinje images

• DPI trackers measure DPI trackers measure rotational and translational rotational and translational eye movementseye movements

• 11stst and 4 and 4thth reflections reflections move together through move together through same distance upon eye same distance upon eye translation, but separate translation, but separate upon eye rotationupon eye rotation

• highly precisehighly precise

• used to be expensive and used to be expensive and difficult to set updifficult to set up

F: Integration Issues and F: Integration Issues and RequirementsRequirementsF: Integration Issues and F: Integration Issues and RequirementsRequirements

• Integration of eye tracker into graphics Integration of eye tracker into graphics system chiefly depends on:system chiefly depends on:• delivery of proper graphics video stream to trackerdelivery of proper graphics video stream to tracker

• subsequent reception of tracker’s 2D gaze datasubsequent reception of tracker’s 2D gaze data

• Gaze data (Gaze data (xx- and - and yy-coordinates) are typically -coordinates) are typically either stored by tracker or sent to graphics either stored by tracker or sent to graphics host via serial cablehost via serial cable

• Discussion focuses on video-based eye Discussion focuses on video-based eye trackertracker

F: Integration Issues and F: Integration Issues and Requirements Requirements (cont’d)(cont’d)


• Video-based tracker’s main advantages Video-based tracker’s main advantages over other systems:over other systems:• relatively non-invasiverelatively non-invasive

• fairly accurate (to about 1° over a 30° field of view)fairly accurate (to about 1° over a 30° field of view)

• for the most part, not difficult to integratefor the most part, not difficult to integrate

• Main limitation: sampling frequency, Main limitation: sampling frequency, typically limited to video frame rate, 60Hztypically limited to video frame rate, 60Hz



Fig.34: Virtual Reality Eye Tracking (VRET) Lab at ClemsonFig.34: Virtual Reality Eye Tracking (VRET) Lab at Clemson

• integration description based on VRET lab equipmentintegration description based on VRET lab equipment

• two systems described:two systems described:• table-mounted, monocular systemtable-mounted, monocular system• HMD-fitted, binocular systemHMD-fitted, binocular system

F: Integration Issues and F: Integration Issues and Requirements—VRET lab equipmentRequirements—VRET lab equipmentF: Integration Issues and F: Integration Issues and Requirements—VRET lab equipmentRequirements—VRET lab equipment

• SGI Onyx2® InfiniteReality™ graphics hostSGI Onyx2® InfiniteReality™ graphics host• dual-rack, dual-pipe, 8 MIPS® R10000™ CPUsdual-rack, dual-pipe, 8 MIPS® R10000™ CPUs• 3Gb RAM, 0.5G texture memory3Gb RAM, 0.5G texture memory

• ISCAN eye trackerISCAN eye tracker• table-mounted pan/tilt camera monocular unittable-mounted pan/tilt camera monocular unit• HMD-fitted binocular unitHMD-fitted binocular unit

• Virtual Research V8 HMDVirtual Research V8 HMD

• Ascension 6 Degree-Of-Freedom (DOF) Flock Of Ascension 6 Degree-Of-Freedom (DOF) Flock Of Birds (FOB) d.c. electromagnetic head trackerBirds (FOB) d.c. electromagnetic head tracker

F: Integration Issues and F: Integration Issues and Requirements—preliminariesRequirements—preliminariesF: Integration Issues and F: Integration Issues and Requirements—preliminariesRequirements—preliminaries

• Primary requirements:Primary requirements:• knowledge of video format required by tracker (e.g., knowledge of video format required by tracker (e.g.,

NTSC, VGA)NTSC, VGA)• knowledge of data format returned by tracker (e.g., byte knowledge of data format returned by tracker (e.g., byte

order, codes)order, codes)

• Secondary requirements—tracker capabilities:Secondary requirements—tracker capabilities:• fine-grained cursor control and readout?fine-grained cursor control and readout?• transmission of tracker’s operating mode along with gaze transmission of tracker’s operating mode along with gaze

data?data?

F: Integration Issues and F: Integration Issues and Requirements—objectivesRequirements—objectivesF: Integration Issues and F: Integration Issues and Requirements—objectivesRequirements—objectives

• Scene alignment:Scene alignment:• required for calibration, display, and data mappingrequired for calibration, display, and data mapping• use tracker’s fine-cursor to measure graphics display use tracker’s fine-cursor to measure graphics display

dimensions—dimensions—it is crucial that graphically displayed it is crucial that graphically displayed calibration points are aligned with those displayed by eye calibration points are aligned with those displayed by eye trackertracker

• Host/tracker synchronization:Host/tracker synchronization:• required for generation of proper graphics display, i.e., required for generation of proper graphics display, i.e.,

calibration or stimuluscalibration or stimulus• use tracker’s operating mode datause tracker’s operating mode data

F.1: System InstallationF.1: System InstallationF.1: System InstallationF.1: System Installation

• Primary wiring considerations:Primary wiring considerations:• video cables—imperative that graphics host generate video cables—imperative that graphics host generate

video signal in format expected by eye trackervideo signal in format expected by eye tracker

• example problem: graphics host generates VGA signal example problem: graphics host generates VGA signal (e.g., as required by HMD), eye tracker expects NTSC(e.g., as required by HMD), eye tracker expects NTSC

• serial line—comparatively simple; serial driver typically serial line—comparatively simple; serial driver typically facilitated by data specifications provided by eye facilitated by data specifications provided by eye tracker vendortracker vendor

F.1: System Installation F.1: System Installation (cont’d)(cont’d)F.1: System Installation F.1: System Installation (cont’d)(cont’d)

Fig.35: Video signal wiring diagram for the Fig.35: Video signal wiring diagram for the VRET lab at ClemsonVRET lab at Clemson

• HMD driven by HMD driven by VGA VGA

• switchbox switchbox controls video controls video between between monitors and monitors and HMDHMD

• 2 VGA-NTSC 2 VGA-NTSC convertersconverters

• TV driven by TV driven by NTSCNTSC

F.1: System Installation—lessons F.1: System Installation—lessons learned at Clemsonlearned at ClemsonF.1: System Installation—lessons F.1: System Installation—lessons learned at Clemsonlearned at Clemson

• Various video switches were needed to Various video switches were needed to control graphics video and eye camera videocontrol graphics video and eye camera video

• Custom VGA cables (13W3-HD15) were Custom VGA cables (13W3-HD15) were needed to feed monitors, HMD, and trackerneeded to feed monitors, HMD, and tracker

• Host VGA signal had to be re-configured Host VGA signal had to be re-configured (horizontal sync (horizontal sync notnot sync-on-green) sync-on-green)

• Switchbox had to be re-wired (missing two Switchbox had to be re-wired (missing two lines for pins 13 and 14!)lines for pins 13 and 14!)

F.2: Application Program F.2: Application Program RequirementsRequirementsF.2: Application Program F.2: Application Program RequirementsRequirements

• Two example applications:Two example applications:• 2D image-viewing program (monocular)2D image-viewing program (monocular)• VR gaze-contingent environment (binocular)VR gaze-contingent environment (binocular)

• Most important common requirement:Most important common requirement:• mapping eye tracker coordinates to application mapping eye tracker coordinates to application

program’s reference frameprogram’s reference frame

• Extra requirements for VR:Extra requirements for VR:• head tracker coordinate mappinghead tracker coordinate mapping• gaze vector calculationgaze vector calculation

F.2.1: Eye Tracker Screen Coordinate F.2.1: Eye Tracker Screen Coordinate Mapping—generalMapping—generalF.2.1: Eye Tracker Screen Coordinate F.2.1: Eye Tracker Screen Coordinate Mapping—generalMapping—general

• The eye tracker returns the user’s POR The eye tracker returns the user’s POR relative to the tracker’s screen reference relative to the tracker’s screen reference frame, e.g., a 512frame, e.g., a 512512 pixel plane512 pixel plane

• Tracker data must be mapped to the Tracker data must be mapped to the dimensions of the application screendimensions of the application screen

• In general, to map In general, to map x'x' [[a,ba,b]] to range to range [[c,dc,d]],,

)(

))((

ab

cdaxcx

F.2.1: Eye Tracker Screen Coordinate F.2.1: Eye Tracker Screen Coordinate Mapping—to 3D viewing frustumMapping—to 3D viewing frustumF.2.1: Eye Tracker Screen Coordinate F.2.1: Eye Tracker Screen Coordinate Mapping—to 3D viewing frustumMapping—to 3D viewing frustum

Fig.36: Eye tracker to VR mappingFig.36: Eye tracker to VR mapping• note the eye tracker origin at top-leftnote the eye tracker origin at top-left

F.2.1: Eye Tracker Screen Coordinate F.2.1: Eye Tracker Screen Coordinate Mapping—to 3D viewing frustum Mapping—to 3D viewing frustum (cont’d)(cont’d)

F.2.1: Eye Tracker Screen Coordinate F.2.1: Eye Tracker Screen Coordinate Mapping—to 3D viewing frustum Mapping—to 3D viewing frustum (cont’d)(cont’d)

• to convert eye tracker coordinates to convert eye tracker coordinates ((x',y'x',y')) to graphics to graphics coordinates coordinates ((x,yx,y)),,

512bottomtop512

bottom

512leftright

left

))((

)(

yy

xx

• the term the term ((512512 - - y'y')) handles the handles the yy-coordinate flip so that eye tracker screen is converted to -coordinate flip so that eye tracker screen is converted to bottom-left of the viewing frustumbottom-left of the viewing frustum

• if dimensions of graphics window are static, e.g., 640if dimensions of graphics window are static, e.g., 640480, above equation can be 480, above equation can be hardcodedhardcoded

F.2.1: Eye Tracker Screen Coordinate F.2.1: Eye Tracker Screen Coordinate Mapping—to 2D image planeMapping—to 2D image planeF.2.1: Eye Tracker Screen Coordinate F.2.1: Eye Tracker Screen Coordinate Mapping—to 2D image planeMapping—to 2D image plane

• Conversion of eye tracker coordinates Conversion of eye tracker coordinates ((x',y'x',y')) to 2D image to 2D image plane coordinates plane coordinates ((x,yx,y) ) is handled similarlyis handled similarly

• For example, if viewable image plane has dimensions For example, if viewable image plane has dimensions 600600450,450,

))(.())((

).()(

87890625512512

450512

1718751512600

yy

y

xx

x

• Note the above mapping assumes eye tracker coordinates are in Note the above mapping assumes eye tracker coordinates are in range range [[0,5120,512]]

• In practice, In practice, usableusable coordinates depend on location of application coordinates depend on location of application window on eye tracking screenwindow on eye tracking screen

F.2.1: Eye Tracker Screen Coordinate F.2.1: Eye Tracker Screen Coordinate Mapping—to 2D image plane Mapping—to 2D image plane (cont’d)(cont’d)

F.2.1: Eye Tracker Screen Coordinate F.2.1: Eye Tracker Screen Coordinate Mapping—to 2D image plane Mapping—to 2D image plane (cont’d)(cont’d)

Fig.37: Application window measurementFig.37: Application window measurement

• use eye tracker’s fine cursor use eye tracker’s fine cursor movement to measure application movement to measure application window’s extentswindow’s extents

• calculate mapping, e.g., for a calculate mapping, e.g., for a 600600450 window,450 window,

)()(

)()(

450153446

53449

600151482

51

yy

xx

F.2.2: Mapping Flock Of Birds F.2.2: Mapping Flock Of Birds Tracker CoordinatesTracker CoordinatesF.2.2: Mapping Flock Of Birds F.2.2: Mapping Flock Of Birds Tracker CoordinatesTracker Coordinates

• For VR applications, position and For VR applications, position and orientation of the head is required orientation of the head is required (obtained from head tracker, e.g., FOB)(obtained from head tracker, e.g., FOB)

• The tracker reports 6 Degree-Of-Freedom The tracker reports 6 Degree-Of-Freedom (DOF) information regarding sensor (DOF) information regarding sensor position and orientationposition and orientation

• Orientation is given in terms of Euler Orientation is given in terms of Euler anglesangles

F.2.2: Mapping Flock Of Birds F.2.2: Mapping Flock Of Birds Tracker Coordinates Tracker Coordinates (cont’d)(cont’d)


Fig.38: Euler anglesFig.38: Euler angles

rot y rot x rot z

yaw pitch roll

azimuth elevation roll

longitude latitude roll

Table 3: Euler angle namesTable 3: Euler angle names

• Euler angles roll, pitch, and Euler angles roll, pitch, and yaw are represented by yaw are represented by RR, , EE, , AA, respectively, respectively

• each describes rotation angle each describes rotation angle about one axisabout one axis



• Euler angles are described by familiar homogeneous Euler angles are described by familiar homogeneous rotation matrices:rotation matrices:

1000

0100

00

00

RR

RR

z

cossin

sincos

R

1000

00

00

0001

EE

EEx cossin

sincosR

1000

00

0010

00

AA

AA

y cossin

sincos

R

1000

0

0

0

AEEAE

AERARERAERAR

AERARERAERAR

yxz coscossinsincos

cossincossinsincoscossinsincoscossin

cossinsinsincoscossinsinsinsincoscos

RRRM

• the composite 4the composite 44 matrix, containing all rotations in one:4 matrix, containing all rotations in one:



• in VR, the composite transformation matrix, returned by the in VR, the composite transformation matrix, returned by the head tracker, is used to transform an arbitrary directional head tracker, is used to transform an arbitrary directional vector, vector, ww = [ = [x y zx y z 1] 1], to align it with the current sensor , to align it with the current sensor (head) orientation:(head) orientation:

1000

0

0

0

1

AEEAE

AERARERAERAR

AERARERAERAR

z

y

xT

coscossinsincos



wMw

• this formulation is used to align the initial view vector, up this formulation is used to align the initial view vector, up vector, and eventually gaze vector with the current head-vector, and eventually gaze vector with the current head-centric reference framecentric reference frame

• note that the FOB matrix may be shifted by 1note that the FOB matrix may be shifted by 1



• e.g., transforming the initial view vector, e.g., transforming the initial view vector, vv = [0 0 -1 1] = [0 0 -1 1]::

1000

0

0

0

1

1

0

0

AEEAE

AERARERAERAR

AERARERAERART

coscossinsincos



vMv

• e.g., transforming the initial up vector, e.g., transforming the initial up vector, uu = [0 1 0 1] = [0 1 0 1]::

1000

0

0

0

1

0

1

0

AEEAE

AERARERAERAR

AERARERAERART

coscossinsincos



uMu

• gaze vector is transformed similarlygaze vector is transformed similarly

F.2.3: 3D Gaze Point and Vector F.2.3: 3D Gaze Point and Vector CalculationCalculationF.2.3: 3D Gaze Point and Vector F.2.3: 3D Gaze Point and Vector CalculationCalculation

• The gaze point calculation in 3-space The gaze point calculation in 3-space depends on only the relative positions of depends on only the relative positions of the two eyes in the horizontal axisthe two eyes in the horizontal axis

• Parameters of interest here are the 3D Parameters of interest here are the 3D virtual (world) coordinates of the gaze virtual (world) coordinates of the gaze point, point, ((xxgg, , yygg, , zzgg))

• These coordinates can be determined These coordinates can be determined from traditional stereo geometryfrom traditional stereo geometry

F.2.3: 3D Gaze Point and Vector F.2.3: 3D Gaze Point and Vector Calculation Calculation (cont’d)(cont’d)


Fig.39: Basic binocular geometryFig.39: Basic binocular geometry

• helmet position is the origin, helmet position is the origin, ((xxhh, , yyhh, , zzhh))

• helmet view vector is the helmet view vector is the optical (viewer-local) optical (viewer-local) zz-axis-axis

• helmet up vector is the helmet up vector is the (viewer-local) (viewer-local) yy-axis-axis

• eye tracker provides eye tracker provides instantaneous viewer-local instantaneous viewer-local gaze coordinates (gaze coordinates (mapped to mapped to viewing frustumviewing frustum))



• given instantaneous binocular gaze coordinates given instantaneous binocular gaze coordinates ((xxll,,yyll)) and and

((xxrr,,yyrr)) at focal distance at focal distance ff along the viewer-local along the viewer-local zz-axis, the -axis, the

gaze point gaze point ((xxgg,,yygg,,zzgg)) can be derived parametrically: can be derived parametrically:

sfzsz

yysysy

xxsxsx

hg

rlhg

rlhg

)(

)/)(()(

)/)(()(

1

21

21

• where the interpolant s is given as:where the interpolant s is given as:

bxx

bs

rl



• the gaze point can be expressed parametrically as a point the gaze point can be expressed parametrically as a point on a ray with origin (xh, yh, zh), the helmet position, with on a ray with origin (xh, yh, zh), the helmet position, with the ray emanating along a vector scaled by parameter s:the ray emanating along a vector scaled by parameter s:

)(

)/)((

)/)((

hhg

hrlhg

hrlhg

zfszz

yyysyy

xxxsxx

2

2

• or, in vector notation, or, in vector notation, gg = = hh + + ssvv, where , where hh is the head is the head position, position, vv is the central view vector, and is the central view vector, and ss is the scale is the scale parameter as defined previouslyparameter as defined previously

• note the view vector here is not related to the view vector note the view vector here is not related to the view vector given by the head trackergiven by the head tracker



• the view vector related to the gaze vector is obtained by the view vector related to the gaze vector is obtained by subtracting the helmet position from the midpoint of the eye subtracting the helmet position from the midpoint of the eye tracked x-coordinate and focal distance to the near view plane,tracked x-coordinate and focal distance to the near view plane,

hm

v

h

h

h

rl

rl

z

y

x

f

yy

xx

2

2

/)(

/)(

where where mm denotes the left and right eye coordinate midpoint denotes the left and right eye coordinate midpoint



• to transform the vector to transform the vector vv to the proper (instantaneous) to the proper (instantaneous) head orientation, this vector should be normalized, then head orientation, this vector should be normalized, then transformed by the orientation matrix returned by the head transformed by the orientation matrix returned by the head trackertracker

• the transformed vector the transformed vector vv gives the gaze direction (ray) gives the gaze direction (ray)

• using the helmet position using the helmet position hh and gaze direction and gaze direction vv, we can , we can express the gaze vector via a parametric representation of express the gaze vector via a parametric representation of a ray with linear interpolant a ray with linear interpolant tt: :

0 tttgaze ,)( vh

F.2.4: Virtual Fixation CoordinatesF.2.4: Virtual Fixation CoordinatesF.2.4: Virtual Fixation CoordinatesF.2.4: Virtual Fixation Coordinates

• The gaze vector can be used in VR to The gaze vector can be used in VR to calculate virtual fixation coordinatescalculate virtual fixation coordinates

• Fixation coordinates are obtained via Fixation coordinates are obtained via traditional ray/polygon intersection traditional ray/polygon intersection calculations, as used in ray tracingcalculations, as used in ray tracing

• The fixated object of interest (polygon) is The fixated object of interest (polygon) is the one closest to the viewer which the one closest to the viewer which intersects the rayintersects the ray

F.2.4: Virtual Fixation Coordinates—F.2.4: Virtual Fixation Coordinates—ray/plane intersectionray/plane intersectionF.2.4: Virtual Fixation Coordinates—F.2.4: Virtual Fixation Coordinates—ray/plane intersectionray/plane intersection

• The calculation of a ray and all polygons The calculation of a ray and all polygons in the scene is obtained via a parametric in the scene is obtained via a parametric representation of the ray:representation of the ray:

do trtray r)(where where rroo defines the ray’s origin (point) and defines the ray’s origin (point) and rrdd defines the ray direction defines the ray direction

(vector)(vector)

• For gaze, use For gaze, use rroo = = hh, the head position, , the head position, rrdd = = vv, the gaze direction vector, the gaze direction vector

F.2.4: Virtual Fixation Coordinates—F.2.4: Virtual Fixation Coordinates—ray/plane intersection ray/plane intersection (cont’d)(cont’d)


• Recall the plane equation Recall the plane equation Ax + By + Cz + D = 0Ax + By + Cz + D = 0, where , where AA22 + B + B22 + C + C22 = 1 = 1, i.e., , i.e., A, B, CA, B, C define the plane normal define the plane normal

• Calculate the ray/plane intersection,Calculate the ray/plane intersection,

d

o

ddd

ooo Dr

CzByAx

DCzBxAxt

rN

N

)(

• Find the closest ray/plane intersection to Find the closest ray/plane intersection to the viewer, where the viewer, where t > 0t > 0



Fig.40: Ray/plane geometryFig.40: Ray/plane geometry

• NN is actually is actually -N-N, to calculate angle , to calculate angle between ray and face normalbetween ray and face normal

• possible divide-by-possible divide-by-zero: need to check zero: need to check for this (if close to 0 for this (if close to 0 then ray and plane then ray and plane don’t intersect)don’t intersect)

• if dot product is if dot product is greater than 0, greater than 0, surface is hidden surface is hidden from viewer (use to from viewer (use to speed up code)speed up code)



Fig.41: Ray/plane intersection Fig.41: Ray/plane intersection algorithmalgorithm

• the parameter the parameter tt defines the defines the point of intersection along the point of intersection along the ray at the plane defined by ray at the plane defined by NN

• if if t > 0t > 0, then point of intersection , then point of intersection pp is given by: is given by: p = rp = roo + t + trrdd

• this only gives the intersection this only gives the intersection of the ray and the (infinite!) of the ray and the (infinite!) planeplane

• need to test whether need to test whether pp lies lies within confines of the polygonal within confines of the polygonal faceface

}

;/

);(

){(if

;

do

do

d

dd

vvt

Dv

v

v

rN

rN

0

F.2.4: Virtual Fixation Coordinates—F.2.4: Virtual Fixation Coordinates—point-in-polygonpoint-in-polygonF.2.4: Virtual Fixation Coordinates—F.2.4: Virtual Fixation Coordinates—point-in-polygonpoint-in-polygon

Fig.42: Point-in-polygon Fig.42: Point-in-polygon problemproblem

for each edge:for each edge:

• calculate plane perpendicular to calculate plane perpendicular to polygon, passing through the polygon, passing through the edge’s two vertices:edge’s two vertices:

N'N' = = NN ( (BB - - AA))

• calculate new plane’s equationcalculate new plane’s equation

• test point p to see if it lies test point p to see if it lies “above” or “below” new plane“above” or “below” new plane

• is p is “above” all planes, p is is p is “above” all planes, p is inside polygoninside polygon

F.3: System Calibration and UsageF.3: System Calibration and UsageF.3: System Calibration and UsageF.3: System Calibration and Usage

• Most video-based eye trackers require Most video-based eye trackers require calibrationcalibration

• Usually composed of simple stimuli (dots, Usually composed of simple stimuli (dots, crosses, etc.) displayed sequentially at far crosses, etc.) displayed sequentially at far extents of viewing windowextents of viewing window

• Application program displaying stimulus Application program displaying stimulus must be able to draw calibration stimulus at must be able to draw calibration stimulus at appropriate locations and at appropriate timeappropriate locations and at appropriate time

F.3: System Calibration and Usage F.3: System Calibration and Usage (cont’d)(cont’d)F.3: System Calibration and Usage F.3: System Calibration and Usage (cont’d)(cont’d)

Fig.43: Usual graphics draw routine augmented by Fig.43: Usual graphics draw routine augmented by mode-sensitive eye tracking codemode-sensitive eye tracking code

sceneProjection();sceneProjection(); // e.g., ortho or perspective// e.g., ortho or perspectivesceneMatrix();sceneMatrix(); // model (object) transformations// model (object) transformationsswitchswitch(eye tracker state) {(eye tracker state) {

case RUN:case RUN:ifif(! hide stimulus)(! hide stimulus)

displayStimulus();displayStimulus();case RESET:case RESET:case CALIBRATE:case CALIBRATE:

drawCalibrateDot(drawCalibrateDot(xx-5, -5, yy-5, -5, xx+5, +5, yy+5);+5);}}swapbuffers();swapbuffers();


• calibration stimulus is displayed in both calibration stimulus is displayed in both RESETRESET and and CALIBRATECALIBRATE states: this facilitates initial alignment of the states: this facilitates initial alignment of the application window (default calibration dot is at center)application window (default calibration dot is at center)

• stimulus scene (e.g., image or VE) is only displayed if display stimulus scene (e.g., image or VE) is only displayed if display condition is satisfied: this can be used to limit duration of condition is satisfied: this can be used to limit duration of displaydisplay

• for VR applications, draw routine may be preceded by left or for VR applications, draw routine may be preceded by left or right viewport calls (for stereoscopic displays)right viewport calls (for stereoscopic displays)

• the main program loop is responsible for: 1) reading eye the main program loop is responsible for: 1) reading eye (and head) tracker data; 2) mapping coordinates; 3) (and head) tracker data; 2) mapping coordinates; 3) starting/stopping timers; 4) recording or acting on gaze starting/stopping timers; 4) recording or acting on gaze coordinatescoordinates


Fig.44: Main loop (2D imaging application)Fig.44: Main loop (2D imaging application)

while(1) {while(1) {getEyeTrackerData(getEyeTrackerData(xx,,yy););mapEyeTrackerData(mapEyeTrackerData(xx,,yy););switch(eye tracker state) {switch(eye tracker state) {

case RUN:case RUN:if(!starting) {start timer; displayStimulus=1;}if(!starting) {start timer; displayStimulus=1;}if(timer() > DURATION) displayStimulus=0;if(timer() > DURATION) displayStimulus=0;else storeData(else storeData(xx,,yy););

case RESET:case RESET:case CALIBRATE:case CALIBRATE:

starting=0;starting=0;}}redraw();redraw();

}}


Fig.45: Main loop (VR application)Fig.45: Main loop (VR application)

whilewhile(1) {(1) {getHeadTrackerData(getHeadTrackerData(eyeeye,,dirdir,,upvupv););getEyeTrackerData(getEyeTrackerData(xxll,,yyll,,xxrr,,yyrr););mapEyeTrackerData(mapEyeTrackerData(xxll,,yyll,,xxrr,,yyrr););s = b/(xs = b/(xll - x - xrr + b) + b) // linear gaze interpolant// linear gaze interpolanthh = [eye = [eyexx, eye, eyeyy, eye, eyezz]] // head position// head positionvv = [(x = [(xll+x+xrr)/2 - x)/2 - xhh, (y, (yll+y+yrr)/2 - y)/2 - yhh, f-z, f-zhh]] // central view vector// central view vectortransformVectorToHead(transformVectorToHead(vv)) // multiply // multiply vv by FOB matrix by FOB matrixgg = = hh + + ssvv // calculate gaze point// calculate gaze pointswitchswitch(eye tracker state) {(eye tracker state) {

......}}redraw();redraw();

}}


• once application program has been developed, system is once application program has been developed, system is ready for use; general manner of usage requires the following ready for use; general manner of usage requires the following steps:steps:• move application window to align it with eye tracker’s move application window to align it with eye tracker’s

default (central) calibration dotdefault (central) calibration dot• adjust the eye tracker’s pupil and corneal reflection adjust the eye tracker’s pupil and corneal reflection

thresholdsthresholds• calibrate the eye trackercalibrate the eye tracker• reset the eye tracker and run (program displays stimulus reset the eye tracker and run (program displays stimulus

and stores data)and stores data)• save recorded datasave recorded data• optionally re-calibrate againoptionally re-calibrate again

F.4: Data Collection and AnalysisF.4: Data Collection and AnalysisF.4: Data Collection and AnalysisF.4: Data Collection and Analysis

• Data collection is fairly straightforward: Data collection is fairly straightforward: store point of regard info along with store point of regard info along with timestamptimestamp

• Use linked list since number of samples Use linked list since number of samples may be largemay be large

typedef struct {typedef struct {queue link;queue link; // linked list node// linked list nodein x,y;in x,y; // POR data// POR datadouble t;double t; // timestamp (usually in ms)// timestamp (usually in ms)

} PORnode;} PORnode;

Fig.46: 2D imaging POR data structureFig.46: 2D imaging POR data structure

F.4: Data Collection and Analysis F.4: Data Collection and Analysis (cont’d)(cont’d)F.4: Data Collection and Analysis F.4: Data Collection and Analysis (cont’d)(cont’d)

• For VR applications, data structure is For VR applications, data structure is similar, but will require similar, but will require zz-component-component

• May also store head positionMay also store head position

• Analysis follows eye movement analysis Analysis follows eye movement analysis models presented previouslymodels presented previously• Goals: 1) eliminate noise; 2) identify fixationsGoals: 1) eliminate noise; 2) identify fixations

• Final point: label stored data appropriately; Final point: label stored data appropriately; with many subjects, experiments tend to with many subjects, experiments tend to generate LOTS of datagenerate LOTS of data

F.4: Data Collection and Analysis F.4: Data Collection and Analysis (cont’d)(cont’d)F.4: Data Collection and Analysis F.4: Data Collection and Analysis (cont’d)(cont’d)

Fig.47: Example Fig.47: Example of 3D gaze of 3D gaze point in VRpoint in VR

• calculated calculated gaze point of gaze point of user in “art user in “art gallery” gallery” environmentenvironment

• raw data, raw data, blinks blinks removedremoved



Part IIIPart III

Potential Gaze-Contingent ApplicationsPotential Gaze-Contingent Applications

G: Applications—IntroductionG: Applications—IntroductionG: Applications—IntroductionG: Applications—Introduction

• Wide variety of eye tracking applications Wide variety of eye tracking applications exist, each class increasingly relying on exist, each class increasingly relying on advanced graphical techniquesadvanced graphical techniquesPsychophysicsPsychophysics Human FactorsHuman Factors

AdvertisingAdvertising DisplaysDisplays

Virtual RealityVirtual Reality HCI & Collaborative SystemsHCI & Collaborative Systems

• Two broad categories: Two broad categories: diagnosticdiagnostic or or interactiveinteractive

H: Psychology, Psychophysics, and H: Psychology, Psychophysics, and NeuroscienceNeuroscienceH: Psychology, Psychophysics, and H: Psychology, Psychophysics, and NeuroscienceNeuroscience

• Applications range from basic research in Applications range from basic research in vision science to investigation of visual vision science to investigation of visual exploration in aesthetics (e.g., perception exploration in aesthetics (e.g., perception of art)of art)

• Examples:Examples:• psychophysics: spatial acuity, contrast sensitivity, ...psychophysics: spatial acuity, contrast sensitivity, ...• perception: reading, natural scenery, ...perception: reading, natural scenery, ...• neuroscience: cognitive loads, with fMRI, ...neuroscience: cognitive loads, with fMRI, ...

H: Psychology, Psychophysics, and H: Psychology, Psychophysics, and Neuroscience Neuroscience (cont’d)(cont’d)

H: Psychology, Psychophysics, and H: Psychology, Psychophysics, and Neuroscience Neuroscience (cont’d)(cont’d)

Fig.48: Perception of artFig.48: Perception of art• small but visible differences in scanpathssmall but visible differences in scanpaths• similar sets of fixated image featuressimilar sets of fixated image features

(a) aesthetic group(a) aesthetic group (b) semantic group(b) semantic group

I: Ergonomics and Human FactorsI: Ergonomics and Human FactorsI: Ergonomics and Human FactorsI: Ergonomics and Human Factors

• Applications range from usability studies Applications range from usability studies to testing effectiveness of cockpit displaysto testing effectiveness of cockpit displays

• Examples:Examples:• evaluation of tool icon groupingsevaluation of tool icon groupings

• comparison of gaze-based and mouse interactioncomparison of gaze-based and mouse interaction

• organization of click-down menusorganization of click-down menus

• testing electronic layout of pilots’ visual flight rulestesting electronic layout of pilots’ visual flight rules

• testing simulators for training effectivenesstesting simulators for training effectiveness

I: Ergonomics and Human Factors I: Ergonomics and Human Factors (cont’d)(cont’d)I: Ergonomics and Human Factors I: Ergonomics and Human Factors (cont’d)(cont’d)

Fig.49: Virtual aircraft cargo-bay Fig.49: Virtual aircraft cargo-bay environmentenvironment

• examination of visual search patterns of experts during examination of visual search patterns of experts during aircraft inspection tasksaircraft inspection tasks

• 3D scanpaths: gaze/wall intersection points3D scanpaths: gaze/wall intersection points

J: Marketing / AdvertisingJ: Marketing / AdvertisingJ: Marketing / AdvertisingJ: Marketing / Advertising

• Applications range from assessing ad Applications range from assessing ad effectiveness (copy testing) in various effectiveness (copy testing) in various media (print, images, video, etc.) to media (print, images, video, etc.) to disclosure research (visibility of fine print) disclosure research (visibility of fine print)

• Examples:Examples:• eye movements over print media (e.g., yellow eye movements over print media (e.g., yellow

pages)pages)

• eye movements over TV ads, magazines, ...eye movements over TV ads, magazines, ...

J: Marketing / Advertising J: Marketing / Advertising (cont’d)(cont’d)J: Marketing / Advertising J: Marketing / Advertising (cont’d)(cont’d)

Fig.50: Scanpaths over magazine adsFig.50: Scanpaths over magazine ads

K: DisplaysK: DisplaysK: DisplaysK: Displays

• Applications range from perceptually-Applications range from perceptually-based image and video display design to based image and video display design to estimation of corrective display functions estimation of corrective display functions (e.g., gamma, color spaces, etc.)(e.g., gamma, color spaces, etc.)

• Examples:Examples:• JPEG/MPEG (no eye tracking JPEG/MPEG (no eye tracking per seper se, but , but

perceptually based, e.g., JPDs)perceptually based, e.g., JPDs)

• gaze-contingent displays (e.g., video-telephony, …)gaze-contingent displays (e.g., video-telephony, …)

• computer (active) visioncomputer (active) vision

K: Displays K: Displays (cont’d)(cont’d)K: Displays K: Displays (cont’d)(cont’d)

Fig.51: Gaze-based foveo-peripheral image codingFig.51: Gaze-based foveo-peripheral image coding• 2 Regions Of Interest (ROIs)2 Regions Of Interest (ROIs)• smooth degradation (wavelet interpolation)smooth degradation (wavelet interpolation)

(a) Haar HVS reconstruction(a) Haar HVS reconstruction (b) wavelet acuity mapping(b) wavelet acuity mapping

L: Graphics and Virtual RealityL: Graphics and Virtual RealityL: Graphics and Virtual RealityL: Graphics and Virtual Reality

• Applications range from eye-slaved foveal Applications range from eye-slaved foveal Region Of Interest (ROI) VR simulators to Region Of Interest (ROI) VR simulators to gaze-contingent geometric modelinggaze-contingent geometric modeling

• Examples:Examples:• flight simulators (peripheral display degradation)flight simulators (peripheral display degradation)

• driving simulators (driver testing)driving simulators (driver testing)

• gaze-based dynamic Level Of Detail modelinggaze-based dynamic Level Of Detail modeling

• virtual terrainsvirtual terrains

L: Graphics and Virtual Reality L: Graphics and Virtual Reality (cont’d)(cont’d)L: Graphics and Virtual Reality L: Graphics and Virtual Reality (cont’d)(cont’d)

Fig.52: Gaze-Fig.52: Gaze-contingent contingent Martian terrainMartian terrain

• subdivided subdivided quad meshquad mesh

• per-block LODper-block LOD

• resolution resolution level based on level based on viewing viewing direction and direction and distancedistance

L: Graphics and Virtual Reality L: Graphics and Virtual Reality (cont’d)(cont’d)L: Graphics and Virtual Reality L: Graphics and Virtual Reality (cont’d)(cont’d)

M: Human-Computer Interaction and M: Human-Computer Interaction and Collaborative SystemsCollaborative SystemsM: Human-Computer Interaction and M: Human-Computer Interaction and Collaborative SystemsCollaborative Systems

• Applications range from eye-based Applications range from eye-based interactive systems to collaborationinteractive systems to collaboration

• Examples:Examples:• intelligent gaze-based informational displays (text intelligent gaze-based informational displays (text

scroll window synchronized to gaze)scroll window synchronized to gaze)

• ““self-disclosing” display where digital characters self-disclosing” display where digital characters responded to user’s gaze (e.g., blushing)responded to user’s gaze (e.g., blushing)

• multiparty VRML environmentsmultiparty VRML environments

M: Human-Computer Interaction and M: Human-Computer Interaction and Collaborative Systems Collaborative Systems (cont’d)(cont’d)

M: Human-Computer Interaction and M: Human-Computer Interaction and Collaborative Systems Collaborative Systems (cont’d)(cont’d)

Fig.53: GAZE Groupware displayFig.53: GAZE Groupware display

Fig.54: GAZE Groupware interfaceFig.54: GAZE Groupware interface

• multiparty tele-multiparty tele-conferencing and conferencing and document sharing document sharing systemsystem

• images rotate to images rotate to show gaze direction show gaze direction (who is talking to (who is talking to whom)whom)

• document “lightspot” document “lightspot” (deictic “look at this” (deictic “look at this” reference)reference)



For further information:For further information:

http://www.vr.clemson.edu/eyetrackinghttp://www.vr.clemson.edu/eyetracking

• SIGGRAPH course notesSIGGRAPH course notes

• Eye Tracking Research & Applications SymposiumEye Tracking Research & Applications Symposium

Eye Tracking Research & ApplicationsEye Tracking Research & ApplicationsSymposium 2000Symposium 2000

November 6th-8th 2000, Palm Beach Gardens, FL, USA

Sponsored by ...

With corporate sponsorship from …

http://www.vr.clemson.edu/eyetracking/et-conf/http://www.vr.clemson.edu/eyetracking/et-conf/

Don’t forget to attend:Don’t forget to attend:Don’t forget to attend:Don’t forget to attend:

LC Technologies Unilever Corp. SensoMotoric Instr.



DemonstrationDemonstration

GAZE Groupware SystemGAZE Groupware System

eye-based interaction in graphical systems: theory & practice andrew duchowski computer science...

Documents

eye tracking systems

eyebased interaction

human visual system

visual attentionchronological

defined attention

gaze groupware system

nystagmus slide

toto visual scene inspection