inferring cognitive states in information science

8/3/2019 Inferring Cognitive States in Information Science

1/53

Dept. of Library and Information Science, Rutgers UniversityNew Brunswick, NJ, USA

Workshop on Inferring Cognitive and Emotional States from Multimodal Measures MMCogEmS2011November 17, 2011

Jacek Gwizdka & Michael J. Cole


2/53

!! Overall research goal: infer and predict mentalstates and context of a person engaged ininteractive information search (e.g., Web search)

!!Completed projects: measures derived fromeye-gaze patterns!! eye-movement patterns and interaction logs to infer

"! task characteristics"! dynamic user states (such as cognitive load/effort)"! persistent user characteristics (such as domain knowledge)

!! On-going projects: multi-modal measures!! eye-tracking + EEG + GSR!! cognitive load + timing of relevance decisions


3/53

3

!!Methodology: Using eye-gaze patterns!! Higher-order patterns: Reading Models!! Measures of cognitive effort in reading

!!Results:!! User study I: journalistic search tasks

"! task characteristics"! cognitive effort

!! User study II: genomics search tasks"! cognitive effort (& learning)"! domain knowledge

!! On-going work


4/53


5/53

5

!! Eye-tracking research havefrequently analyzed eye-gaze position aggregates('hot spots)!! spatiotemporal-intensity

heat maps

!! also sequential scan paths

! Higher-order patterns:

reading models & derived measures


6/53

!! We have developed a new methodology to analyzeeye-gaze patterns:

!! Model thereading processto represent (textual) informationacquisition in search

!! Measure thecognitive effortdue to (textual) informationacquisition

!! Usebothto correlate / infer higher-level constructs (taskcharacteristics, user knowledge, etc.)


7/53

7

Can be represented as units of reading experience:

((F F F) F (F F F) F F F F (F F F F F F) F)

F = fixation


8/53

1.! Eye movements are cognitively controlled (Findlay &Gilchrist, 2003)

2.! Eyes fixate until cognitive processing is completed(Rayner, 1998)

Eye gaze pattern analysis is powerful:

!! Eye gaze is only way to acquire (textual)information

!! 1. + 2. ! Direct causal connection betweenobservable (text) information search behavior andusers mental state


9/53

9

!! We implemented the E-Z Reader reading model(Reichle et al., 2006)

!! Fixation duration >113 ms threshold for lexical processing(Reingold & Rayner, 2006)

!! The algorithm distinguishesreading fixation sequencesfromisolated fixations, called'scanning' fixations

!! Each lexical fixation is classified to (S,R) that is (Scan,Reading)

!! Inputs: eye gaze location, duration!! Add fixation to reading sequence if next saccade:

!! on the same line of text!! and less than 120 pixels to the right!! or is a regression on the same line of text


10/53


11/53

11

!! Two states: reading and scanning!! transition probabilities!! each state characterized by the number of lexical fixations

and duration

ScanRead

1-q

p

1-p

q


12/53

12

Can be represented as units (Fixations) of reading experience:

(F F F) F (F F F) F F F F (F F F F F F) F

Using thereading model:

Reading state R (green); Scanning state S:

R S R S S S S R S


13/53

13


14/53

!! Eyes fixate until cognitive processing is completed(Rayner 1998)

!! While reading, words already understood in theparafoveal region are skipped (Reichle, et al., 2006)

!! Eye gaze patterns depend on cognitive processingof information that is being acquired

!! Hypothesis: Analysis of reading fixation patternsreveal some aspects of cognitive effort

14


15/53

!!Reading Speed

!!Perceptual Span- Average spacing of fixations!!Lexical Fixation Duration Excess(LFDE):

!! Time needed to acquire meaning above the minimum forlexical access

!!Fixation Regressions- Number of regressionfixations in the reading sequence

text acquired= -----------------

processing time

15


16/53

!!Reading speedwill be slower for:!! hard to read text (Rayner & Pollatsek, 1989),!! unfamiliar words (Williams & Morris, 2004),!! words used in less frequent senses (Sereno, ODonnell, &

Rayner, 2006),

!! more complex concepts (Morris, 1994)16

1o (70px)foveal region


17/53

Perceptual span is the spacing of fixations

Perceptual spanreflects a human limitation on the numberanddifficultyof concepts that can be processed (e.g. Pollatsek

et al. 1986).

17


18/53

!! 10-15% of fixations are regressions

!! Reading goal affectsreading regressions!! More regressions when:

!! greater reader domain expertise,!! conceptuallycomplex&difficulttext passages,!! resolution of ambiguous (sense) words

18


19/53

!! GreaterLFDEindicatesless familiarwords &greater conceptualcomplexity

!! LFDE is also correlated with establishing wordmeaning in context

example

19


20/53

20


21/53

!! 32 journalism students!! 4journalistictasks (realistic, created by journalism faculty

and journalists)

!! Journalism tasks can be about any topic, but fewtask types.

!! Tasks designed to vary in ways that affect searchbehavior (Li, 2009)

!! Task difficulty was post-self-rated by participants (7-point Likert scale: very easy to extremely difficult)

21


22/53

!!Complexity- number of steps needed (ex: identify an expert, getcontact information)

!!Task Product(factual vs. intellectual, e.g., fact checking vs.production of a document)

!!Named- is actual search target specified?!!Level- the information object to process (a complete document vs.

a document segment)!!Task Goal- the nature of the task goal (specific vs. amorphous)

!! Note:Copy Editing CPE&Advance Obituary OBIare most dissimilar!! Copy Editingis expected to be easiest,Advance Obituarymost difficult

Task Product Level Named Goal Complexity

BackgroundBIC mixed Document No Specific High

Copy EditingCPE factual Segment Yes Specific LowInterview PreparationINT mixed Document No Mixed A,S Low

Advance ObituaryOBI factual Document No Amorphous High


23/53

23

!! User task characteristics!! Can we detect task characteristics from eye-gaze patterns ?

!! Cognitive effort!! Do the cognitive effort measures correlate with:

"! task properties expected to contribute to task difficulty?"! the effort needed to complete the task?"! user judgment of task difficulty?


24/53

24


25/53

25

!! Task effects on transition probabilities S!R & R!S(all subjects & pages)

(Cole, Gwizdka, Liu, Bierig, Belkin & Zhang, ECCE 2010; IwC 2011)

! For OBI, INT searchersbiased tocontinuereading

! For CPE tocontinuescanning

Searchers are adoptingdifferent reading

strategies for differenttask types

OBI: advanced obituaryINT: interview preparationCPE: copy editingBIC: background information


26/53

26

!! For highly attended pages

Total Text Acquired on

SERPs and Content

per page

Total Text Acquired on

SERPs and Content

OBI: advanced obituaryINT: interview preparation

CPE: copy editingBIC: background information


27/53

27

!! For highly attended pages

State Transitions

on SERPs per page

State Transitions on

Content pages per page

Read !Scan

Read !Scan

Scan! ReadScan! Read

OBI: advanced obituaryINT: interview preparation

CPE: copy editingBIC: background information


28/53

28

Measure Related Task Characteristics

Number ofstate transitions

bias toread Taskleveland taskgoal

level: document;goalotherthan specific (OBI & INT)

bias toscanlevel: segment and taskgoal: specific (CPE)

Total text acquired on SERPsTaskcomplexity: More text acquired inBIC and OBI

Text acquired and number ofstate transitions per page oncontent pages

Tasklevel: segment and taskproduct:factual (CPE)

Cole, Gwizdka, Liu, Bierig, Belkin & Zhang. (2011). Task and User Effects on Reading Patterns in Information

Search Interacting with Computers23(4), 346 362.

Task Product Level Named Goal Complexity

BackgroundBIC mixed Document No Specific High

Copy EditingCPE factual Segment Yes Specific Low

Interview PreparationINT mixed Document No Mixed A,S Low

Advance ObituaryOBI factual Document No Amorphous High

Forhighlyattendedpages

For allpages


29/53


30/53

(self-rated after the task)

BIC CPE INT OBI


31/53

!! Search effort: task time, pages visited, queriesentered

!! Copy Editing (CPE) required the least effort of all tasks!! Advance Obituary (OBI) required overall most effort (although not

the greatest effort of the tasks for every effort measure)

!! For all tasks, for both greater perceived difficulty (self-ratings) and search task effort:

!! higher medianLFDE(Kruskal-Wallis chi-squared =125.02, p = 0.03)!! slower reading speed (ANOVA F-value=5.5 p=0.02)!! Strongest correlations obtained when considering only the

single longest reading sequence on a page


32/53

32


33/53

!!

33


34/53

!!

34


35/53

!!

35


36/53

!! Cognitive effort measures seem valid!! Eye gaze pattern cognitive effort measures match

withsubjective task difficulty

!! Cognitive effort measure results correlate withtaskcharacteristics related to task effort

!! e.g. Complex tasks, amorphous goals

36


37/53

3737


38/53

!! Words areindicativeof concepts and conceptfeatures

!! Reading involves:!! knowledge used to understand words,!! processing concepts expressed in the content, and!! acquisition of information (and concepts) from the content

!! User knowledge controls interaction during search:!! selects the words to read, and!! imposes cognitive processing demands to understand the

concepts associated with the words


39/53

!! Does users knowledge influence information searchbehavior?

!! Is cognitive effort related to domain knowledge?

39


40/53

!! 40 undergraduate and graduate students!! Rated 409 genetics and genomics MeSH terms

!!1: No knowledge, ... to5: Can explain to others

!! Five tasks from 2004 TREC Genomics track!! Tasks were hard!

!! We use the same methodology to create readingmodels and calculate cognitive effort measures as instudy I


41/53

!! Participants domain knowledge (PDK) wasrepresented by sum of term ratings!! participants rated MeSH terms!! normalized by a hypothetical expert

! ki is the term knowledge rating (1-5)! i ranges over all terms! ti is 1 if rated or 0 if not! m number of terms rated by a participant! The sum is normalized by a hypothetical expert

who rated all terms as 'can explain to others'


42/53


43/53

These cognitive effort measures were individuallycorrelated with level of domain knowledge.

Forallreading sequences:

!! higher domain knowledge ~ lower cognitive effort!! perceptual span (Kruskal-Wallis "2 = 4734.254, p < 2.2e-16)!! LFDE (Kruskal-Wallis "2 = 5570.103, p < 2.2e-16)!! reading speed (Kruskal-Wallis "2 = 5570.103, p < 2.2e-16)

Similar correlations found for long reading sequences

Long reading sequencesmight better reflect concept use by participants

during information acquisition because of the attention allocated to acquiringthat text.


44/53

!! For long reading sequences!! We used random forests to construct regression

models from the cognitive effort measures

!! Regression results were clustered(agglomerate hierarchical clustering)

!! Random forest model gave us relative importance ofcognitive effort measures as contributing variablesin a predictive model!! high importance: reading length (px), LDFE, total duration

of reading sequences (sum of lexical fix), perceptual span

!! less important: number of regressions


45/53

Random forest model classification errorsall participants

only native English speakers

Random forest model cog effort !

domain knowledge correlation

with MeSH based domain knowledge

PDKgroups low inter high

low 8! 0! 0!

intermediate 1! 23! 0!

high 0! 0! 6!

PDKgroups low inter high

low 3! 0! 0!

intermediate 0! "#! 0!high 0! 0! $!


46/53

!! Ability to detect knowledge levels indicates apossibility of real-time detection of learning of anew material (new domain)

!! Task phase analysis: beginning, middle, end!! same random forest model across the three phases!! significant difference : LFDE drops from beg to mid to end

phase, while

!! numFix -- not significantly different between phases!! and readingLength increased from middle to end (sig: Kruskal-

Wallis chi^2 = 885.2262, df = 817, p < 0.05)

!! Possible evidence for learning ?


47/53

47

!! Eye tracking enables high resolution analysis ofsearchers activity during interactions withinformation systems

!! There is more beyond eye-gaze locations withtimestamps

!! Eye-tracking data:!! can be used to identification of task characteristics!! cognitive effort!! domain knowledge

!! High potential for implicit detection of a searchersstates


48/53

!! The reading model methodology and cognitive effortmeasures are based on many years of empirical

research.

!! Eye movements have a direct causal connection to theinformation acquisition process.

!! This connection is not mediated!!! Domain independent

!! Document content is not involved!! Culturally and individually independent

!! Method represents the user's experience of the informationacquisition process

!! Real-time modeling of user domain knowledge is possible


49/53

!! Processing requirements are low - just need fixationlocation and duration.

!! Only recent eye movements are needed to calculatecognitive effort.

!! Real-time assessment of cognitive effort!! Early task session detection of user properties, e.g. domain

knowledge and perception of task difficulty

!! Soon enough for a system to make a difference inproviding user support


50/53

50


51/53

51

!! Start witheye-tracking:pupillometry

!! info relevance (Oliveria,Russell, Aula, 2009)

!!

low-level decision timing(Einhuser, et al. 2010)

!! AddsEEG,GSR!! Funded by Google

Research Award

EEG

GSR

Eye

tracking

! Implicit characterization of Information Search Process usingphysiological devices

! Can we detect when searchers make information relevancedecisions?

Tobii T-60eye-tracker

Emotiv EPOCwireless EEGheadset


52/53

52

!! Jacek Gwizdka http://jsg.tel

!! Acknowledgements!! Funding: IMLS Google!! Collaborators:

"! Dr. Nicholas J. Belkin, Dr. Xiangmin Zhang"! Post-Doc: Dr. Ralf Bierig"! Collaborator & PhD student:Michael Cole"! PhD students: Chang Liu, Jingjing Liu"! Master students


53/53

53

!! Eye tracking technology is declining in price and in2-3 years could be part of standard displays.!! Already in luxury cars and semi-trucks (sleep detection)!! Computers with built in eye-trackingTobii / Lenovoproof of concept eye-trackinglaptop - March 2011

inferring cognitive states in information science

Documents