Analyzing the Integration of Figures and Text Using Eye MovementsJason RubinsteinWith Eileen Kowler, Nick Ross, Elio Santos, Min ZhaoUniversity of Delaware and Rutgers

Introduction• Figures + Text• Same “page” vs. different

“page”• Semantic integration of

information over long timescales• Building up a knowledgebase

• Between saccades• “rapid movements of the eyes

between positions of rest”• Between “page turns”

• Meaning• People tend to limit memory

load when possible

The L–D differences in the ventilatory and metabolic variables occurred even between 15-min epochs of identical activity levels, whether these were below or above the daily mean (Fig. 5). VE/VO2 in the D remained higher than in the L, by 13% when activity was low and 9% at the higher activity level. The results of this analysis did not depend on the animal ’s activity immediately preceding the epoch analyzed, as there was no correlation between the L –D difference in VE for iso-activity epochs and the level of activity prior to the epochs analyzed.

Ballard, Hayhoe, & Pelz, 1995

Epelboim & Suppes, 2001

Question

What strategies do people use to extract meaning from figures and text (in combination)?

Why?• Real world• Textbooks• Educational material

• Standardized tests• SATs

• Newspapers• Websites• Instruction manuals• Captions

The percentage statements during which the participant looked at the person was used as an indicator of gaze seeking. There was not an effect of gaze cueing and no interaction between gaze cueing and full or partial instructions. For full instructions, the person was fixated less often when he provided gaze cues, than when he did not provide gaze cues. For partial instructions, gaze cues were often sought.

The percentage statements during which the participant looked at the person was used as an indicator of gaze seeking. There was not an effect of gaze cueing and no interaction between gaze cueing and full or partial instructions. For full instructions, the person was fixated less often when he provided gaze cues, than when he did not provide gaze cues. For partial instructions, gaze cues were often sought.

Which of the following is true?

A. The percentage of fixations on the person was high for both full and partial instructions independently of whether the gaze cues were provided by the person or not.B. Fixations on the person were more frequent following full instructions with or without gaze cues.C. Gaze cues increased the percentage of fixations on the person for both full and partial instructions.D. For partial instructions, fixations on the person were rare whether gaze cues were provided or not.

Question• What strategies do people use to extract meaning from figures

and text (in combination)?• Patterns in• Rule-governed transitions• Revisits

• “Effector” (motor action)• Eye (simultaneous) or arm (sequential)

vs

VIDEO

Results• Revisit figure after reading key words in text• Words that refer to the legend,

specific aspects of figure, etc

Results• Revisit figure after reading key words in text• Words that refer to the legend,

specific aspects of figure, etc• On average…• Switch more in simultaneous condition (!)

Number of visits to figure Number of visits to text

Results• Revisit figure after reading key words in text• Words that refer to the legend,

specific aspects of figure, etc• On average…• Switch more in simultaneous condition (!)• Though “easier” (cognitively) in alternating

• Less “work”

Results• Revisit figure after reading key words in text• Words that refer to the legend,

specific aspects of figure, etc• On average…• Switch more in simultaneous condition (!)• Though “easier” (cognitively) in alternating

• Less “work”• Periphery -- planning

Results• Revisit figure after reading key words in text• Words that refer to the legend,

specific aspects of figure, etc• On average…• Switch more in simultaneous condition (!)• Though “easier” cognitively in alternating

• Less “work”• Periphery -- planning

• Spend more time in text in both conditions• Longer visits + more time overall

Proportion of time spent in figure Proportion of time spent in text

Results• Revisit figure after reading key words in text• Words that refer to the legend,

specific aspects of figure, etc• On average…• Switch more in simultaneous condition (!)• Though “easier” cognitively in alternating

• Less “work”• Periphery -- planning

• Spend more time in text in both conditions• Longer visits + more time overall

• However, proportion of overall time between simultaneous and alternating about the same

Average length of visits to figure (s) Average length of visits to text (s)

Results• Revisit figure after reading key words in text• Words that refer to the legend,

specific aspects of figure, etc• On average…• Switch more in simultaneous condition (!)• Though “easier” cognitively in alternating

• Less “work”• Periphery -- planning

• Spend more time in text in both conditions• Longer visits + more time overall

• However, proportion of overall time between simultaneous and alternating about the same

• Reading…• Images vs. words?

What next?• MORE• Participants• Stimuli

• Shorter trials/harder questions• Accuracy of various strategies

• Specifics of transitions• Word in text → which part of figure?

• Vice versa• Model revisits

• Simultaneous trials• Saccade triggers stimuli

• Periphery

?

Thank you!• Eileen Kowler• Nick Ross, Elio Santos, Min Zhao• Matthew Stone• Joe, Henry, Avery, Rachel, Lara• The Academy• Everyone else