Eye-tracking the Factors of Process Model Comprehension

Tasks

Răzvan Petruşel, Jan MendlingBabeş-Bolyai University

Cluj-Napoca, Romaniarazvan.petrusel@econ.ubbcluj.ro

Wirtschaftsuniversität Wien, Austria

jan.mendling@wu.ac.at

OverviewAddress the gap of research on the factors that influence the comprehension tasks.

2

Motivation and Relevant Region Definition The Experiment Conclusions

Motivation

3

Q1 - Can Z and AA be executed in the same case?Q2 - After O has been executed, and the default path is taken at the next gateway, then Z must always be executed?

Notion of Relevant Region

4

o Notion of Dominator o Relevant Region for activities a and

b includes all elements on a path from the dominator node of a and b

Eye-tracking example

5

Eye-tracking example

6

Eye-tracking example

7

Eye-tracking example

8

HypothesisH1: The Relevant Region elements are fixated a longer time than other model elements by the subjects that provided the correct answer to the comprehension question;H2: More elements of the Relevant Region are fixated than other model elements by the subjects that provided the correct answer;H3: The higher the percentage of time spent fixating the Relevant Region elements, the more likely is a correct answer;H4: The higher the share of Relevant Region elements a person fixates (scan-path recall and/or f-measure), the more likely is a correct answer. 9

Experiment

Conducted using 26 experts from academia and industry in Vienna, Eindhoven and Cluj-Napoca; Eye-tracking of subject behavior while answering 6 comprehension questions based on 2 structured and 4 unstructured models; Measured: Model elements inspected; Time spent fixating each model element.

10

Measurements

11

ATT ADT ALX ALY ARI APW APH AQU AFXTotalTime DeltaTimeX_Gaze Y_Gaze Region PupilWidthPupilHeightQuality Fixation 0.0000 0.0000 0.1869 0.5495 3 0.0759 0.0576 1 0.0167 0.0167 166.523 0.1829 0.5426 3 0.0746 0.0557 1 0.0167 0.0334 167.122 0.1796 0.5335 3 0.0752 0.0572 1 0.0167 0.0500 166.523 0.1759 0.5257 3 0.0758 0.0582 1 0.0167 0.0619 B:ROI[03] for 0.183466 sec 0.0668 167.968 0.1728 0.5149 3 0.0762 0.0568 1 0.0168 0.0835 166.659 0.1708 0.5061 3 0.0696 0.0607 1 0.0167 0.0953 B:Fixation 10263 ( 0.194, 0.501 ) for 0.116743 sec 0.1001 166.579 0.1683 0.4958 3 0.0735 0.0572 1 0.0167 0.1168 166.703 0.1680 0.4936 3 0.0735 0.0559 1 0.0167 0.1335 166.535 0.1694 0.4894 3 0.0698 0.0592 1 0.0167 0.1501 166.661 0.1697 0.4896 3 0.0764 0.0573 1 0.0167 0.1501 A:Drift 7313 ( 0.183, 0.539 ) --> ( 0.170, 0.490 ) = 0.05134 distance for 0.216858 sec 0.1668 167.059 0.1684 0.4895 3 0.0778 0.0572 1 0.0334

ResultsModel_question no 19_0 19_6 29_5 30_3 39_6 50_1 Correct (no.) 18 18 17 6 16 14 Incorrect (no.) 5 6 2 10 0 9 Correct (%) 78.26% 66.67% 89.47% 37.5% 100% 60.87%

Subject 1 2 3 15 25 Question code 10_0 10_0 10_0 10_0 10_0

Outcome 1 1 0 1 0 TIR 47% 86% 12% 65% 59%

Count ME Actually visited 16 5 17 6 12 Count RR Elem 3 3 2 3 3 Total RR Elem 3 3 3 3 3

SPP 0.19 0.60 0.12 0.50 0.25 SPR 1.00 1.00 0.67 1.00 1.00 SPF 0.32 0.75 0.20 0.67 0.40

SPF2 0.54 0.88 0.34 0.83 0,63 Variable F F2 SPP SPR TIR

Sample size 146 146 146 146 146

Arithmetic mean 0.5742 0.5742 0.4745 0.6591 0.5729

Standard deviation 0.2214 0.2214 0.2547 0.2687 0.2839

Results H1 and H2

13

Variable F F2 SPP SPR TIR ANOVA F-ratio 28,247 29.650 17.290 21.446 17.964 ANOVA Significance P<0.001 P<0.001 P<0.001 P<0.001 P<0.001 Simple correlation r 0.405 0.413 0.327 0.360 0.333

Independent Variable Fcrit at 95% F-ratio p-value F 4.61 28.24 P < 0.001 F2 4.80 29.65 P < 0.001 SPP 3.01 17.29 P < 0.001 SPR 3.64 21.44 P < 0.001 TIR 3.11 17.96 P < 0.001

Results H3 and H4

14

Independent variables Coefficient Std. Error rpartial t P (Constant) 0.2253 SPP 0.4232 0.1362 0.2515 3.108 0.0023 SPR 0.4758 0.1290 0.2946 3.687 0.0003

Independent Variables R2 R2-adjusted SPP, SPR 0.1847 0.1733 SPP, TIR 0.1156 0.1032 SPR, TIR 0.1680 0.1564 F, TIR 0.1649 0.1532

Results H3 and H4

15

Variable Coefficient Std. Error P Recall 2.71200 0.94914 0.0043 TIR 1.79751 0.81127 0.0267 Constant -1.5463

Results H3 and H4

16

Variable Coefficient Std. Error P Recall 2.71200 0.94914 0.0043 TIR 1.79751 0.81127 0.0267 Constant -1.5463

Recall TIR Probability of correct answer

0.40 40% 56,40% 0.40 60% 64,95% 0.50 50% 67,01% 0.90 50% 85,73% 1 100% 95,09%

Enabled activities

Investigate structuredness in more detail; Assign a quantitative measure to the difference between experts and novices; How does a comprehension question influence the cognitive process; Mine a cognitive exploration pattern (process model of examining a process model) that provides a higher probability of correctly understanding a model.

17

To take home…

18

Relevant Region notion

Answering correctly a comprehension question is strongly linked to: how many and, how long RR elements are inspected.

It is possible to predict if a comprehension question will be answered correctly.

Thank you for your attention!

Further questions?

19