Comparison of Attention Behaviour Across User Sets through AutomaticIdentification of Common Areas of Interest

Prithiviraj K. Muthumanickam, Jouni Helske, Aida Nordman, Jimmy Johansson, Matthew CooperUniversity of Linkoping, Sweden

<prithiviraj.muthumanickam, aida.vitoria, jimmy.johansson, matthew.cooper>@liu.se jouni.helske@iki.fi

Abstract

Eye tracking is used to analyze and compare userbehaviour within numerous domains, but long durationeye tracking experiments across multiple users generatemillions of eye gaze samples, making the data analysisprocess complex. Usually the samples are labelled intoAreas of Interest (AoI) or Objects of Interest (OoI),where the AoI approach aims to understand how auser monitors different regions of a scene while OoIidentification uncovers distinct objects in the scene thatattract user attention. Using scalable clustering andcluster merging techniques that require minimal userinput, we label AoIs across multiple users in longduration eye tracking experiments. Using the commonAoI labels then allows direct comparison of the usersas well as the use of such methods as Hidden MarkovModels and Sequence mining to uncover common anddistinct behaviour between the users which, until now,has been prohibitively difficult to achieve.

1. Introduction

Eye-tracking has become a valuable tool to analyzeand compare visual scanning behaviour across multipleusers in numerous application domains including userinterface evaluation studies, air traffic control, drivingand maritime simulators etc. Typically, the processinvolves manual specification of regions of interestin the displayed application by an analyst and theeye-tracking system then recording how the user’s visualattention moves between these pre-defined regions asthe session progresses. This can be an effectiveapproach but misses a level of detail which goes beyondthe analyst’s expectations of user behaviour.This hasbeen a focus of recent research which has looked at waysto derive the areas of interest (AoIs) directly from thedata, rather than through the analyst’s expectations [1,2]. This approach can, therefore, better capture theuser’s attention over time. Earlier methods [2] identifiedthe AoIs in long duration eye tracking experiments using

scalable clustering and cluster merging. The results havebeen demonstrated across gaze data collected from AirTraffic Controllers working in a simulated remote towerenvironment [3]. But manual comparison of visualscanning behaviour across multiple air traffic controllerswas a complex process due to the fact that the IDs orlabels of the AoIs found for a single subject do notcorrelate with those of others.

In this work, we have developed a method forthe cross-correlation of AoIs between sets of usersperforming long-duration eye-tracking experiments.This allows us to create a common set of AoI labelsbetween the members of a set of subjects, allowingdirect visual comparison. The identified labels can befurther examined using sequence analysis methods sothat an analyst can identify similarities and differencesin visual scanning behaviour amongst the subjects.Also, real-world eye tracking experiments are proneto noise due to users moving away from the fieldof view of the sensor or due to sensor errors. Inour example application scenario, ∼30% of the eyetracking data collected from an air traffic controlsimulation was noisy. Taking this into account, wehave explored the use of sequence analysis approachessuch as multinomial Hidden Markov Models [4] and asequence mining application named ELOQUENCE [5].Our method extends the approach developed in [2]to identify common AoIs labels across multiple users,which can be then utilized to analyze the similaritiesand differences in visual behaviour between all of theusers. The main contributions of our approach are thefollowing:

• Automatic identification of common AoI labelsacross multiple subjects with minimal user input.

• Analysis of user behaviour using multinomialHidden Markov Model and sequence miningbased approaches in the presence of noisy eyegaze data.

Proceedings of the 53rd Hawaii International Conference on System Sciences | 2020

Page 1360URI: https://hdl.handle.net/10125/63906978-0-9981331-3-3(CC BY-NC-ND 4.0)

2. Related Work

Eye gaze points are labelled using two prominentapproaches - Area of Interest (AoI) and Object ofInterest (OoI) based methods. The distinction betweenthese two approaches lies in what questions they aimto answer. AoI based approaches reveal how a userinspects and monitors different regions of the displayedscene, while OoI based approaches reveal which distinctobjects in the scene attract user attention. AoI basedapproaches can have a greater level of detail thanOoI, as the total number of OoI labels is tied to thenumber of distinct objects in the scene. We classifythe related work section into three parts: (1) and(2): comparison of visual behaviour using OoI or AoIlabelling, respectively, and (3): comparison methodswithout any labelling. For recent reviews of eye trackingdata analysis and visualization readers can refer to [6, 7].

2.1. Comparison using OoI labelling.

Image processing algorithms were used in [8] tocompute OoIs in images. Scale-invariant featuretransforms [9] and spectral clustering [10] were usedfor automatic detection of OoIs in videos. Differentvisualization approaches such as transition graphs,‘scarf plots’, and OoI timelines are then used to compareeye gaze behaviour between multiple users [11]. Thefocus of our work is to reveal how different regions of ascene (AoIs) attract visual attention providing a greaterlevel of detail in visual attention labelling than OoIs.

2.2. Comparison using AoI labelling.

Manual annotation of AoIs is very difficult forlong duration eye tracking data due to the variation inbehaviours which may occur. Automated approachescompute AoIs using pre-defined grids or data-drivenmethods. Predefined grid-based methods divide thescene into a user defined grid and gaze points fallingwithin a grid cell are labelled with the cell id, butthey can fail to capture the level of detail in thevisual scanning behaviour. Labelling the AoIs basedon spatio-temporal characteristics of the data providesa higher resolution in the identified AoIs and bettercaptures the evolution of the users’ attention over time.The ScanMatch [12] approach allows manual definitionof AoI labels through simple binning of the scene intoa discrete number of regular bins. It was extendedin [13] using a percentile mapping approach. Datadriven methods such as Mean-shift clustering were usedin iComp [14] to cluster the fixation points into AoIlabels. The fixation clusters are represented as ellipsesand an intersection is performed across the ellipses to

label AoIs across users. Over et. al. [15] constructedVoronoi cells around fixation points based on theirdensity, where the areas of high fixation density lead tosmall Voronoi cells and vice-versa. Unfortunately noneof these methods handle the problem of data saturation,a common issue with very long duration eye trackingdatasets [2], well. The Levenshtein distance [14]is used to compute a similarity score between twousers but this approach cannot define the relationshipbetween AoIs and how close they are in space andtime. The Needleman-Wunsch algorithm [12] canbe used to model relationships between AoIs using asimilarity matrix but it has to be defined manually.Multidimensional scaling [16] can then be applied onthese similarity matrices to identify groups of usersexhibiting similar gaze behaviour. A more exhaustivestate of the art in comparing AoIs between multipleusers is available in [17]. Long duration eye trackingexperiments tend to generate noise, due to sensorerrors or users moving out of the field of view ofthe sensors, and the challenges in dealing with thisuncertainty in the data are not discussed extensively inthe previous methods. Probabilistic transition matrixbased methods [18] compute the amount of transitionbetween the AoI labels of the stimulus and can beused to compute the variability between a pair ofusers. However, the number of transition matrices acrossmultiple time steps and users increases dramatically fora large dataset.

2.3. Comparison without labelling.

Geometric distance based comparison methods areused to map neighbouring fixations using Mannandistance [19] but the challenge is that several fixationsfrom a single eye gaze sequence can be mapped to asingle fixation in the other. This leads to overestimatedsimilarity scores and hence a disproportionate impactof individual fixations. The MultiMatch algorithm [20]computes distances across multiple dimensions suchas the shape of eye gaze samples, saccade lengthand direction, fixation position and duration andconverts the data into a vector based representation.Similarity between users is statistically tested usingthe Kolmogorov-Smirnov test. Heatmaps [21] can becomputed based on the density of fixation points andcompared across users using similarity measures such ascorrelation coefficient, Kullback-Leibler distance, andearth mover distance. A state of the art discussion onavailable comparison methods and metrics is presentedin Holmqvist et. al. [22]. Without the application ofany data abstraction techniques such as data labelling orclustering, however, analyzing very large eye tracking

Page 1361

tUser-1

tUser-2

tUser-n

. . . . . . . . .

Unconstrained identi�cationof similar AoIs over time

Step-2

Step-3

t t t

t

t

t

t

t

t

Unconstrained AoIidenti�cation

Interactive common AoI identi�cation across users

Step-1

Figure 1. Method pipeline: Common AoI identification across user sets. Step1 - Identification of fixation

clusters. Step2 - Similar AoI identification for every user. Step3 - Common AoI identification across all users.

datasets collected across multiple users hugely increasesthe computational complexity of the analysis process.

The main contributions of our proposed methodare, (1) Computation of common AoI labels acrossmultiple users using scalable clustering and clustermerging that is not constrained by the total numberof parameters that need to be ‘tweaked’ by ananalyst (Sections 3.1, 3.2, 3.3), (2) Comparison ofuser behaviour using two prominent sequence analysisapproaches in the presence of noise in the eye gaze data(see section 3.4).

3. Our method

As discussed in Section 2, clustering methods foridentifying AoIs in long duration eye tracking datasuffer from data saturation. The work of [2] highlightsthis problem when applying Mean-shift clustering forlong duration eye gaze data. To overcome thisproblem, they propose a clustering and hierarchicalcluster merging to compute AoIs. The AoI labels arenot, however, synchronized across all subjects and hencethe comparison of visual scanning behaviour betweensubjects can be only performed manually. Also themethod requires a number of parameters to be defined byan analyst and it becomes difficult if we want to comparebehaviour between a large number of subjects. In orderto avoid data saturation and to remove the difficultyaround modifying multiple parameters, we propose thefollowing list of improvements for labelling AoIs.

3.1. Step1 - Clustering raw eye gaze data

For each subject, the raw eye gaze data withits timestamp information is divided into overlappingtime windows. The earlier work of [2] used 2DMinimum spanning trees (MST) to cluster eye gazepoints. Classification of eye gaze data into fixationclusters and saccades are performed based on thespatio-temporal characteristics of consecutive eye gazepoints. Hence, clustering the data without temporalinformation can lead to mis-classification. This problemcan, to some extent, be avoided by setting small

overlapping time windows but that approach introducesadditional parameter tuning. Instead, we propose a 3DMST based clustering approach where the nodes of thetree represent 2D eye gaze data with its correspondingtimestamp. The edge weight of the MST is computedusing the Euclidean distance metric. Fixation clustersfrom the 3D MST can then be computed by deletingits inconsistent edges. As described in the earlierwork of [2], Zahn’s clustering algorithm requires tuningof input parameters such as neighbourhood depth andinconsistency factor for each eye tracking data set.In order to avoid the complexity of tuning inputparameters for each eye tracking data set, we extendthe Maximum Standard Deviation Reduction (MSDR)algorithm of [23] to delete inconsistent edges in the 3DMST. Let σ(T ) represent the standard deviation of allthe edge weights of the MST. At each iteration, i, anedge is removed such that it maximizes the reductionof the weighted standard deviation σ(T i) of the MSTsub-graphs. If the edges of the MST sub-graph at the ith

iteration is T i : {T i1, T i2, ..., T im}, the standard deviationof edge weights of the sub-graph be σ(T ij ) , the σ(T i) iscomputed using equation 1.

σ(T i) =

∑|T i|j=1 |T ij | · σ(T ij )∑|T i|

j=1 |T ij |(1)

The local minimum of a higher order polynomialfit on the difference between successive σ2

i valuescorresponds to the optimal iteration to terminate theprocess (see [23]). Each MST sub-graph representsan eye gaze cluster and as the maximum duration ofa saccade is 80ms, MST sub-graphs with less than∼5 gaze points can be discarded as saccades foran eye tracker of refresh rate 60 Hz. The convexhulls of the MST sub-graphs can be represented usingpolygons and in the subsequent steps, only the polygonsare considered for further processing. Any fixationclassification algorithm can also be used in this step, butour approach removes the complexity of working withmultiple input parameters.

Page 1362

0 50 100 150 200 250 300iterations

30

40

50

60

70

Aver

age

edge

wei

ght

Figure 2. Significant drop in average edge weights

of the MST is highlighted by the knee point in red.

dj(A,B) = 1− area(A ∩B)

area(A ∪B)(2)

3.2. Step2 - Hierarchical cluster merging

Spatially neighbouring AoIs that are separatedin time may end up having different AoI clusterlabels. In order to compute maximum spatial similarityamong the AoIs, minimum spanning trees can beeffective in solving this combinatorial problem. Forevery pair of adjacent time windows, we computean adjacency matrix of the cluster polygons. Eachentry of the adjacency matrix corresponds to the spatialsimilarity between a pair of cluster polygons and iscomputed using a symmetric Jaccard distance metric(see Equation 2). An MST is computed from thismatrix, where the weight of an edge corresponds to theamount of dissimilarity between two cluster polygons.The longest edges in the MST correspond to clusterpolygons of high spatial dissimilarity. The edges aresorted and removed in an iterative manner starting fromthe longest edge. The average edge weights representingthe average value of the Jaccard distance metric can beplotted against the iteration step as shown in Figure 2. Aknee point can be computed using [24] which indicatesthe iteration index at which there is a significant dropin the dissimilarity. This entire computation requiresno user intervention and can be parallelized acrossevery adjacent pair of time windows. Cluster polygonswith high spatial similarity are merged together andthis process of merging adjacent time windows can berepeated in an hierarchical manner, as shown in Figure 1.

3.3. Step3 - Common AoI labelling

The topmost time window of the hierarchy diagram(see Figure 1) contains cluster polygons representingareas of interest from all the subjects taking part in thesame scenario of an eye tracking experiment. In orderto compute common AoI labels across all the subjects,a similarity matrix is computed from all the clusterpolygons as explained in Step 2. Cluster polygons

which exhibit strong similarity are merged based onthe computation of the knee point. At this juncture,an analyst can also modify the knee point to controlthe degree of similarity while merging the clusters (seeFigure 2). Moving the knee point towards the left leadsto a higher degree of cluster merging leading to very fewareas of interest and vice-versa.

3.4. Step4 - Comparison of multiple subjects

The common AoI labels from the topmost timewindow of the hierarchy (see Figure 1) can bepropagated back to the lowest level containing the raweye gaze points where each point gets labelled either asnoise, saccade or an AoI label id. The raw eye gaze datais transformed into a symbol sequence for each subjectand sequence analysis methods can be used to identifypatterns of visual gaze behaviour across the subjects. Weanalyze the data with two prominent sequence analysisapproaches, Hidden Markov Models and pattern-growthbased methodology [25] for sequence mining, in orderto explore subject behaviour across multiple subjects.

Hidden Markov models (HMM). Instead ofinspecting the similarities in the recorded eye-trackingdata directly, a probabilistic modelling approach whichtakes account of the uncertainties in the data due tonoise in the eye-tracking sensors can be preferable. Forcategorical time series of AoI labels, an HMM [26] is anatural option to find a probabilistic description of theAoI patterns, both for smoothing noisy high-frequencydata and for predicting missing observations (e.g. dueto sensor malfunction). As each move between AoIs istypically separated by a saccade, a discrete-time HMMwhich only considers simple transitions between timepoints is insufficient as this would result in a modelwhere each transition between two AoIs is through a“saccade state”. While a continuous time HMM couldbe applied only for AoIs of interest, this approach wouldalso rely heavily on potentially noisy local transitions.Hence, we consider the following multinomial HMMwith aggregated time intervals. Our starting point is theaggregated counts of AoI labels during time intervals oflength ∆. By denoting the time interval [t, t+∆] simplyas t, we define

• Observed label set sequence Y = (Y1, . . . , YT )with Yt = (n1t , . . . , n

Lt ), where nlt is the number

of observed labels l at time t.• Hidden state sequence x = (x1, x2, . . . , xT ) with

hidden states s ∈ {1, . . . , S}.• Initial state probability vector π = {πs} of lengthS, where πs is the probability of starting from thehidden state s.

• Transition probability matrix A = {asr} of size

Page 1363

S×S, where asr is the probability of moving fromthe hidden state s at time t− 1 to the hidden stater at time t.

• Emission probabilities p(Yt)s, s = 1, . . . , S,where each p(Yt)s is a multinomial distributiondefining the state specific emission probabilitiesof observations Yt.

For a more detailed description of HMM see,for example, [4]. Given K, the number of hiddenstates and the observations Y, we can estimatethe initial probability vector, transition probabilitiesand multinomial distributions, either via maximumlikelihood or by a Bayesian approach.

Sequence mining. The path of AoIs followed bythe subject’s eyes during an eye-tracking experimentcan be seen as a sequence of events, where eachevent corresponds to a visit to an AoI. This makesit possible to use sequence mining techniques toinvestigate the subjects visual scanning strategies duringan experiment. In contrast to the methods in sections 2and Hidden Markov models, sequential patterns ofeye movement across multiple subjects, which may beshifted by a time span, can be easily detected. Input fora sequence mining algorithm consists of a data set, D,of event sequences and a minimum support threshold,denoted by σ. In the concrete case of the work presentedhere, an event corresponds to an AoI and an eventsequence captures the sequence of AoIs derived fromeye movement data. A sequence α ≡ e1 → · · · → en isa subsequence of another sequence β ≡ e′1 → · · · → e′m(0 ≤ n ≤ m), if e1 = e′i1 , · · · , en = e′in , for someintegers 1 ≤ i1 < · · · < in ≤ m. The output of asequence mining algorithm is a set of AoI sequences Θ,called frequent patterns. Each frequent pattern p ∈ Θoccurs as a sub-sequence of k ≥ 1 input sequences andk|D| ≥ σ. In the context of the work described in this

paper, sequence mining can be applied

• to investigate eye movement strategies commonto multiple subjects who performed a commonexperiment, or

• to find patterns on the sequence of AoIs obtainedfrom a single subject.

In the former case, it is important that a commonalphabet is used to label the AoIs discovered for themultiple subjects. Thus, the computational flow starts byusing the method described in section 3.3 for computingcommon AoI labels across different subjects, thensequences of AoIs corresponding to the gaze movementsof each subject can be obtained by labelling the raw eyegaze points of each participant. A gaze point in the rawdata that falls within an AoI X (i.e. a cluster polygon) is

labelled with the X’s label. Finally, sequence mining isused to find common eye movement strategies. For thelatter case corresponding to a single subject, a slidingwindow over the input sequence of AoIs generatedby the subject is used and a set of sub-sequencescan be obtained from this window at different pointsin time. This set can then be mined to discoversequential patterns corresponding to frequent occurringeye-movement strategies employed by the subject whilesolving tasks such as monitoring aircraft landings andtake off. The sequence mining algorithm used is basedon a pattern-growth approach [25] and is part of theinteractive visualization system ELOQUENCE [5].

4. Application Example: Simulation ofRemote Multi-Tower Air TrafficControl Scenario

The source of the data for demonstration and testingof our methods are derived from a new way of workingin the field of Air Traffic Control. The ‘RemoteTower’ approach has been in development for more thanten years and offers the opportunity to staff numeroussmaller airports with a single controller. The traditionaltower is replaced with cameras, and the Air trafficcontrol officer (ATCO) works in a virtual tower wherelarge display environments provide the equivalent of atower view and can display more than one airport viewat a time. This approach reduces the number of ATCOsrequired to provide national cover, reducing costs, butthis new way of working presents significant changes inprocedure and is currently a topic of thorough study todetermine best practices and ensure safety of operation.The experimental setup presented in [3] is used for datacollection and it consists of six large screens presentingthe tower view for two airports. The screens on the deskpresent, respectively, radar information for both airports,flight ‘strip’ management for both airports, and a voicecommunication system VCS to control communicationwith the aircraft.

Data collection. Eye tracking data sets werecollected from four operational air traffic controllerofficer (ATCOs) performing the same scenario.SmartEye [27] camera sensors were used to track theeye movements of the controllers in a non-intrusivemanner. On average around 30% of the data samplesexhibited uncertainty due to sampling errors and alsodue to controllers moving their attention away fromany of the nine screens. During the experiment thecontrollers can move in a comfortable manner but arerequired to remain seated. The eye-tracking data isrecorded in the local coordinate system (pixel values)of each of the screens and each experimental scenario

Page 1364

(a) ATCO-A (b) ATCO-B

(c) ATCO-C(d) ATCO-D

Figure 3. Common labelling of AoIs across four controllers: Common AoIs are marked with the same colour in

each sub-figure. The space-time cube (STC) can be rotated, zoomed and filtered along vertical axis to avoid

visual clutter. The two tower views are displayed at the back of the cube while the radar, flight strip and the

voice control displays are arranged across the front part of the cube.

●

●

●

●

●

● ●

●

●

●●

●

●●

● ● ●

●

●●

●

●

●

●

●

●

●

●

● ● ●●

●● ●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●

● ●●

●

●●

● ●

●

●

●

●

●

●

●

●●

● ●● ● ● ●

●

●

●

●

●

●●

●●

●●

● ●

4 5 6

1 2 3

1954

3889

3596

3080 60

635

9727

6841

1025

4326

14 710

662

3650

2445

2966

3585

1477

2293

3293

2558 39

665

541

12 264

1954

3889

3596

3080 60

635

9727

6841

1025

4326

14 710

662

3650

2445

2966

3585

1477

2293

3293

2558 39

665

541

12 264

1954

3889

3596

3080 60

635

9727

6841

1025

4326

14 710

662

3650

2445

2966

3585

1477

2293

3293

2558 39

665

541

12 264

0.0

0.2

0.4

0.0

0.2

0.4

Label

Em

issi

on p

roba

bilit

y

Hidden state ● ● ● ● ● ●1 2 3 4 5 6

Figure 4. Emission probabilities of the states

identified using HMM across all users.

lasted for ∼90 minutes for each controller. For thepurposes of our analysis, the eye tracking data wasremapped into a single space-time volume, as shown inFigure 3, as if the displays were parts of a single, largerflat display environment.

4.1. Common AoI labelling across all users

For each user, the gaze data was divided into 1minute time windows with 50% overlap. These valueswere chosen since any task performed by an ATCOsuch as assessing the flight, radar, runway etc., shouldrequire less than 30 seconds. As our method doesnot depend on user-defined parameters, Steps 1 & 2require no intervention from the analyst and hence caneasily be scaled to any number of subjects. CommonAoI labels are computed across all the users in Step 3and at this stage, an analyst can control the degreeof similarity across AoIs by interactively adjusting theknee point. Moving the knee point towards the leftin Figure 2 leads to a high degree of cluster merging,eventually producing less AoIs and vice-versa. Theresults for four ATCOs carrying out the same scenario

are displayed in Figure 3. All computations wereimplemented in Python on an iMac with a 3.5Ghz IntelCore i7 processor and 16GB RAM. Step1 takes between15 to 20 minutes for a single user, but this processingneeds to be done only once for each data set andadvanced parallel processing libraries can be utilizedto improve the performance. In Steps 2 & 3, findingintersections between polygons were computed usingkd-trees, which makes the computation logarithmic, andthe iterative edge removal requires logarithmic time tosort the edge weights.

4.2. Hidden Markov Model (HMM)

In case of multinomial HMM, we chose 4.5 minutestime intervals, a duration chosen based on the timetaken for any significant event, such as a take-off orlanding, in the simulation, creating in total n = 19 timewindows for each user. While shorter time intervals canprovide more fine-grained details of subject behaviour,longer intervals can better capture groups of commonlyoccurring labels (the estimated hidden states containmultiple large emission probabilities) and reduce theeffect of small time-scale variation and missingness inthe AoI data. As the interest was in comparing thefour participants, we modelled a joint model for allfour participants, leading to common estimates of π, A,B, and thus common definitions of the hidden states.Due to computational costs involved in identifying theparameters, using the full set of extracted AoIs wasnot feasible. Therefore we focused on AoIs whichoccurred commonly for all four users. We first extracteda reduced set of labels with a condition that the totaldwell time of each label was at least two seconds for oneor more users. This resulted in 370 AoIs. From these,

Page 1365

we used only those common labels across all the fourusers (intersection of users’ reduced label sets), leadingto 22 distinct labels.

The HMM was written in Stan [28], and the analysisand visualization was done in the R environment[29] using rstan [30] and ggplot2 [31] packages. Acommon problem in HMM framework is the definitionof K, the number of hidden states. Too smallor large number of hidden makes interpretation ofthe models as well as general model fit difficult.Small K leads to hidden states which combine largenumber of observed symbols, whereas increasing Kproduces typically simpler (sparser) emission patterns,but complex transition structures (see, for example,discussion in [32]). Note however, that with largeK, possibly with sparsity constraints between states,HMM approach can be useful as a pre-processingmethod for creating reduced sets of aggregated labels,which can be further analyzed, for example, with asequential pattern mining approach. As our goal was notprediction but exploring and comparing the behavioursof the participants, we chose K = 6 based on visualinspections of the model parameters, hidden state paths,and the corresponding label patterns mapped against thesimulation environment and observed sequences.

We used Dirichlet priors for π, rows of A, denotedby Ak, and rows of B, denoted by Bk as,

π ∼ Dirichlet(απ1 , . . . , απK), απk = 2K−k,

Bk ∼ Dirichlet(δ, . . . , δ), δ ∼ half-Normal(0, 1)

Ak ∼ Dirichlet(αA,k1 , . . . , αA,kK ),

αA,kj =

12 (1 + T

K ), if |j − k| = 1,

1 + TK , if j = k,

1, otherwise,

where T is the number of time points (19) and δ isan additional concentration parameter. The prior onπ concentrates initial distribution towards first statesfor identifiability (ordering of states is arbitrary), whilethe prior on A emphasises diagonal dominant transitionmatrices a priori, and small values of δ emphasise sparseemission densities. Based on above definitions, weobtained the MAP (maximum a posteriori) estimatesof our model parameters and most probable hiddenstate paths with Stan using L-BFGS-B optimization.Figure 4 describes the composition of each state in termsof emission probabilities. Figure 5 shows the mostprobable hidden paths for each user. The height andopacity of each rectangle is proportional to the labelcount using the common label set (number of labelswithin time-interval belonging to the set intersection of

ATCO-A

ATCO-B

ATCO-C

ATCO-D

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19Time (4.5min intervals)

Hidden state 1 2 3 4 5 6

Multiple touch and go + one take-off event Event of snow in two airports Transition to State6

Figure 5. State sequence for four controllers with

information on contextual significance of each state

given by colour intensity and size.

all AoI labels) to the label count using the reduced labelset (370, set union of all labels), with the maximumheight corresponding to 62% and the minimum to 7%.We can see large similarities especially with ATCOsB and C, while ATCO D is spending relatively largeproportion of time looking at something outside of thecommon label set. From the state sequence diagramof Figure 5, we can infer the common groups ofobserved AoI labels (labels which have large emissionprobabilities in same hidden state) and transitionsbetween these label-groups (hidden states).

Case 1:First 22 minutes of the simulation FromFigure 5, we see that in the first 22 minutes ofthe experiment, ATCOs B, C and D exhibit verysimilar behaviour , while ATCO A handles the scenariodifferently. Based on the available context data, onlythe left airport was active with multiple ‘touch and goevents’ of an aircraft and a single take off event. ATCOsB, C and D are in State2, which exhibits concentratedattention at multiple parts of the airport as seen inFigure 6 (b). ATCO A remains in State3 where the maindifference from other ATCOs is that there is a scanningbehaviour at multiple parts of the airport with significantattention. Even the radar of the right airport receivesmajor attention despite there being no traffic scheduledat that airport.

Case 2: Snow event Between time windows 9 to13 (see Figure 5), we can observe different scanningbehaviours between the ATCOs. From the context data,we find that there is a snowfall drastically reducing thevisibility. During this time, a snowplough parked at acorner of the apron moves to clear the runway. Duringthe snow event, there is also a take-off event in theleft airport and a landing event at the right. ATCO Aremains under State 5 for this entire time period, whileATCOs B and D remain in this state for most of the time.By observing the emission probabilities of State 5, theATCOs exhibit concentrated attention at different AoIsas shown in Figure 7 (a). However, ATCO C remains inState 3 and, as explained above in Case 1, it correspondsto scanning behaviour at multiple parts of the airport,with significant time spent in each AoI to maintain the

Page 1366

Apron(a) ATCOA (b) ATCOB

Left airport Radar

Left airport Zoomed-in view

Intersection fromapron till runway

Touch-and-go point

Wind indicatorRight airport Radar

Figure 6. First 22 minutes of the experiment: (a) Major attention scanning pattern of ATCO A (b)

Concentrated attention of ATCO B

situational awareness. The event of snow in both theairports has modified the behaviour of ATCO C from apreviously concentrated attention to a highly dispersedattention state of monitoring different AoIs.

Case 3: Between time windows 15 and 17 inFigure 5, all the ATCOs make a transition to State 6that represents maximum visual attention at the rightairport radar (see Figure 4). From the context data wefind that there was a take-off event at the left airportand a landing-event at the right. Very high attention waspaid by all ATCOs to the right airport radar to constantlyupdate the position status of the incoming aircraft.

4.3. Common sequences between subjects

Exploratory sequence mining provides the facilityfor analyst-driven exploration of AoIs to identifypatterns of interest in the data and also providessequential information through the order of visits acrossAoIs. The four scanpath data sets from the ATCOs canbe used as input for the sequence mining algorithm.The system used in this work is ELOQUENCE [5].We first investigated the existence of eye-movementstrategies common to all subjects. Thus, we set thesupport constraint σ to 100% and requested the systemto compute sequence patterns of length 3. These patternsare shown in Figure 8(a). We were then interestedin finding which AoIs were visited, by all four usersat some point in time, after visiting AoI 3889. Thus,we have grown the tree branch, corresponding to thesequence pattern 3080 → 606 → 3889. This leadsto the conclusion that all four users used a “nested”back-and-forth scanning behaviour: after visiting AoI3080 the users scan between AoIs 3889 and 1954 and,some time after, return to AoI 3080. These AoIs areshown in figure 8(b). Our next goal was to investigatecommon scanning strategies that occurred before andafter visiting AoI 662. We used the forward andbackward capability of our system to mine interestingpatterns. After selecting an AoI, forward miningcomputes sequence patterns starting with that AoI, whilebackward mining computes sequence patterns endingwith it. An example is given in Figure 8(c). Inthe graph shown in this Figure, each path (without

repeated edges) starting at node 662 corresponds to asub-sequence found in the four input sequences of AoIs(i.e. σ = 100) Finally, we refer to the possibility toset time constraints on both the dwell time of the AoIsand on the time elapsed between AoIs of the computedsequence patterns, in order to find more meaningfulpatterns. Hence, the simplicity of this search mechanismallows for a fine-grained exploration of related activitieswithin the long-duration eye gaze data, finding frequentpatterns which correspond with user behaviour. Thissame information also allows frequent patterns to befiltered out of the data, allowing infrequent patterns tobe identified, meaning that the analyst can explore lessfrequent, even aberrant behaviours.

5. Discussion

By taking into account the characteristics of theeye tracking data, the proposed improvements removesthe restrictions in tuning multiple parameters for AoIidentification. The technique also allows an analystto define the degree of similarity across the identifiedAoI clusters through modification of a knee pointplot. Identification of an optimum knee point can thenbe handled by an analyst using an interactive visualanalysis system. The advantage of this approach is thatthe identification of areas of interest is data driven andthe total number of identified AoIs can be completelycontrolled by an analyst. With the increase in thenumber of identified Areas of interest, it enables moredetailed analysis of the visual scanning patterns butit adds additional computational complexity for thesequence analysis methods.

By transforming the raw eye gaze data into symbolicrepresentation, sequence analysis can be used to identifypatterns in eye gaze behaviour. While differentsequence analysis methods from bio-informatics andtext mining communities are available for analyzingthe symbol sequence, two prominent approaches usingHMM and Sequence mining are utilized to identifypatterns in the data set. HMM approach can beused to infer multiple groups of observed AoIs andtransition of eye gaze behaviour between the groupsof AoIs. But due to the computational complexity

Page 1367

Figure 7. Comparison of ATCOs A and C during the snow event

19543889

3080

6062768

2543

(a)

(b)

(c)

(d)Figure 8. (a) Common eye-movements to all users.

(b) AoIs identified in the shown sequences. Common

eye-movements before and after visiting AoI 662.

of HMMs for analyzing large data sets from multipleusers, interactive exploratory data analysis becomesdifficult. Hence, HMM based approach can be usefulas a pre-processing method for creating reduced sets ofaggregated AoIs, which can then be further analyzed forexample with sequential pattern mining approach. Thesequence mining algorithm based on a pattern-growthapproach [5] enables analyst-driven exploratory dataanalysis to identify patterns of interest. The order ofvisits to different AoIs and their frequency of occurrenceacross multiple users can be identified by an analystusing the interactive visual analysis system. Differentsequence mining approaches from text mining andbio-informatics communities can be evaluated in thefuture to identify the effectiveness of different methodsfor tasks performed by an analyst. But in order toperform an evaluation of different sequence analysistechniques, ground truth information that represents theintention for a visual scanning behaviour can can becollected from ATCOs, using for example, talk outloud methods with predefined code words indicating theintention of the actual activity.

6. Conclusions and Future Work

In this work we have developed a method which,with little or no input from an analyst, computes

common Areas of Interest (AoIs) labels across multipleusers who are undertaking a similar or identicaltask. The common set of AoI labels permits theuse of any number of methods to explore commonand different behaviours between subjects. We haveexemplified this through the application of sequenceanalysis methods such as Hidden Markov models andSequence mining. The specific case described in thiswork relates to ATCOs performing experimental studiesbeing carried out with respect to safe practice in remotetower scenarios, where the ATCO is simultaneouslyresponsible for traffic at two or more different airports.Our colleagues in air traffic management, who haveprovided the data sets, have found these approachesvery interesting, particularly for the long-durationexperiments which they frequently conduct, and wehave begun working to conduct an evaluation of theeffectiveness of these methods in analysing data of thistype in comparison with current techniques.

The HMM framework can also be extended invarious ways. For example, the label occurrencesand hidden state transitions can be explained throughadditional information, such as the timing of variousevents on the screen or the characteristics of theparticipants. In the case of a large number ofparticipants, mixtures of multiple HMMs can be used togroup participants into sub-populations of interest [33].We are also currently studying the optimal combinationsof the label set definitions and time interval lengths forvarious settings.

One aspect of eye-tracking data which must be dealtwith when considering sequence mining is that the datais very noisy, with missing or erroneous samples dueto occlusion, interference or other issues. Our initialversion of sequence mining method has avoided thisissue but not dealt with it directly. We plan to improvethe quality of the sequences through better filtering andbetter use of the temporal information available.

Acknowledgements

We thank Lothar Meyer and Supathida Boonsongof the Swedish Air Navigation service for providingthe data sets and domain expertise. Funding forthis research project was provided by the SwedishTransport Agency, Trafikverket, and the SwedishScience Council.

Page 1368

References

[1] W. Fuhl, T. Kuebler, T. Santini, and E. Kasneci,“Automatic generation of saliency-based areas of interestfor the visualization and analysis of eye-tracking data,” inProceedings of the Conference on Vision, Modeling, andVisualization, pp. 47–54, 2018.

[2] P. K. Muthumanickam, K. Vrotsou, A. Nordman,J. Johansson, and M. Cooper, “Identification oftemporally varying areas of interest in long-durationeye-tracking data sets,” IEEE TVCG, vol. 25, no. 1,pp. 87–97, 2019.

[3] P. K. Muthumanickam, A. Nordman, L. Meyer,S. Boonsong, J. Lundberg, and M. Cooper, “Analysisof long duration eye-tracking experiments in a remotetower environment,” in 13th USA/Europe air trafficmanagement R&D seminar, Vienna, Austria, 2019.

[4] L. Rabiner, “A tutorial on hidden Markov modelsand selected applications in speech recognition,”Proceedings of the IEEE, vol. 77, pp. 257–286, 1989.

[5] K. Vrotsou and A. Nordman, “Exploratory visualsequence mining based on pattern-growth,” IEEE TVCG,vol. PP, pp. 1–1, 06 2018.

[6] G. Andrienko, N. Andrienko, M. Burch, andD. Weiskopf, “Visual analytics methodology foreye movement studies,” IEEE TVCG, vol. 18, no. 12,pp. 2889–2898, 2012.

[7] T. Blascheck, K. Kurzhals, M. Raschke, M. Burch,D. Weiskopf, and T. Ertl, “Visualization of eye trackingdata: A taxonomy and survey,” in Computer GraphicsForum, vol. 36, pp. 260–284, 2017.

[8] L. Paletta, K. Santner, G. Fritz, A. Hofmann, G. Lodron,G. Thallinger, and H. Mayer, “A computer vision systemfor attention mapping in slam based 3d models,” arXivpreprint arXiv:1305.1163, 2013.

[9] T. C. Kubler, D. R. Bukenberger, J. Ungewiss,A. Worner, C. Rothe, U. Schiefer, W. Rosenstiel,and E. Kasneci, “Towards automated comparison ofeye-tracking recordings in dynamic scenes,” in EUVIP,2014, pp. 1–6, IEEE, 2014.

[10] K. Kurzhals, M. Hlawatsch, C. Seeger, and D. Weiskopf,“Visual analytics for mobile eye tracking,” IEEE TVCG,vol. 23, no. 1, pp. 301–310, 2017.

[11] T. Blascheck, M. Schweizer, F. Beck, and T. Ertl, “Visualcomparison of eye movement patterns,” in ComputerGraphics Forum, vol. 36, pp. 87–97, Wiley OnlineLibrary, 2017.

[12] F. Cristino, S. Mathot, J. Theeuwes, and I. D. Gilchrist,“Scanmatch: A novel method for comparing fixationsequences,” Behavior research methods, vol. 42, no. 3,pp. 692–700, 2010.

[13] T. C. Kubler, E. Kasneci, and W. Rosenstiel,“Subsmatch: Scanpath similarity in dynamic scenesbased on subsequence frequencies,” in Proceedingsof the Symposium on Eye Tracking Research andApplications, pp. 319–322, ACM, 2014.

[14] A. T. Duchowski, J. Driver, S. Jolaoso, W. Tan,B. N. Ramey, and A. Robbins, “Scanpath comparisonrevisited,” in Proceedings of the 2010 symposium onETRA, pp. 219–226, ACM, 2010.

[15] E. A. Over, I. T. Hooge, and C. J. Erkelens, “Aquantitative measure for the uniformity of fixationdensity: The voronoi method,” Behavior researchmethods, vol. 38, no. 2, pp. 251–261, 2006.

[16] A. Coltekin, S. I. Fabrikant, and M. Lacayo, “Exploringthe efficiency of users’ visual analytics strategies basedon sequence analysis of eye movement recordings,”International Journal of Geographical InformationScience, vol. 24, no. 10, pp. 1559–1575, 2010.

[17] T. C. Kubler, C. Rothe, U. Schiefer, W. Rosenstiel,and E. Kasneci, “Subsmatch 2.0: Scanpathcomparison and classification based on subsequencefrequencies,” Behavior research methods, vol. 49, no. 3,pp. 1048–1064, 2017.

[18] S. R. Ellis and J. D. Smith, “Patterns of statisticaldependency in visual scanning,” 1985.

[19] S. K. Mannan, K. H. Ruddock, and D. S. Wooding, “Therelationship between the locations of spatial features andthose of fixations made during visual examination ofbriefly presented images.,” Spatial vision, 1996.

[20] R. Dewhurst, M. Nystrom, H. Jarodzka, T. Foulsham,R. Johansson, and K. Holmqvist, “It depends on how youlook at it: Scanpath comparison in multiple dimensionswith multimatch, a vector-based approach,” Behaviorresearch methods, vol. 44, no. 4, pp. 1079–1100, 2012.

[21] R. Caldara and S. Miellet, “iMap: a novel methodfor statistical fixation mapping of eye movement data,”Behavior research methods, pp. 864–878, 2011.

[22] K. Holmqvist, M. Nystrom, R. Andersson, R. Dewhurst,H. Jarodzka, and J. Van de Weijer, Eye tracking: Acomprehensive guide to methods and measures. OUPOxford, 2011.

[23] Y. Zhou, O. Grygorash, and T. F. Hain, “Clustering withminimum spanning trees,” International Journal on AITools, vol. 20, no. 01, pp. 139–177, 2011.

[24] V. Satopaa, J. Albrecht, D. Irwin, and B. Raghavan,“Finding a” kneedle” in a haystack: Detectingknee points in system behavior,” in 2011 31stInternational Conference on Distributed ComputingSystems Workshops, pp. 166–171, IEEE, 2011.

[25] J. Han, J. Pei, and X. Yan, “Sequential pattern miningby pattern-growth: principle and extensions,” Studies inFuzziness and Soft Computing, pp. 183–220, 2005.

[26] L. E. Baum and T. Petrie, “Statistical inference forprobabilistic functions of finite state Markov chains,”The Annals of Mathematical Statistics, vol. 37, no. 6,pp. 1554–1563, 1966.

[27] Smart Eye AB, Gothenburg, Sweden, “Smart Eye Pro.”[28] B. Carpenter, A. Gelman, M. Hoffman, D. Lee,

B. Goodrich, M. Betancourt, M. Brubaker, J. Guo, P. Li,and A. Riddell, “Stan: A probabilistic programminglanguage,” Journal of Statistical Software, Articles,vol. 76, no. 1, pp. 1–32, 2017.

[29] R Core Team, R: A Language and Environment forStatistical Computing. R Foundation for StatisticalComputing, Vienna, Austria, 2019.

[30] Stan Development Team, “RStan: the R interface toStan,” 2018. R package version 2.18.2.

[31] H. Wickham, ggplot2: Elegant Graphics for DataAnalysis. Springer-Verlag New York, 2016.

[32] S. Helske, J. Helske, and M. Eerola, CombiningSequence Analysis and Hidden Markov Models in theAnalysis of Complex Life Sequence Data, pp. 185–200.Cham: Springer International Publishing, 2018.

[33] S. Helske and J. Helske, “Mixture hidden markov modelsfor sequence data: The seqHMM package in R,” Journalof Statistical Software, Articles, no. 3, pp. 1–32, 2019.

Page 1369