1

Predicting the Driverrsquos Focus of Attentionthe DR(eye)VE Project

Andrea Palazzilowast Davide Abatilowast Simone Calderara Francesco Solera and Rita Cucchiara

AbstractmdashIn this work we aim to predict the driverrsquos focus of attention The goal is to estimate what a person would pay attention towhile driving and which part of the scene around the vehicle is more critical for the task To this end we propose a new computer visionmodel based on a multi-branch deep architecture that integrates three sources of information raw video motion and scene semanticsWe also introduce DR(eye)VE the largest dataset of driving scenes for which eye-tracking annotations are available This datasetfeatures more than 500000 registered frames matching ego-centric views (from glasses worn by drivers) and car-centric views (fromroof-mounted camera) further enriched by other sensors measurements Results highlight that several attention patterns are sharedacross drivers and can be reproduced to some extent The indication of which elements in the scene are likely to capture the driverrsquosattention may benefit several applications in the context of human-vehicle interaction and driver attention analysis

Index Termsmdashfocus of attention driverrsquos attention gaze prediction

F

1 INTRODUCTION

According to the J3016 SAE international Standard which definedthe five levels of autonomous driving [26] cars will provide afully autonomous journey only at the fifth level At lower levels ofautonomy computer vision and other sensing systems will stillsupport humans in the driving task Human-centric AdvancedDriver Assistance Systems (ADAS) have significantly improvedsafety and comfort in driving (eg collision avoidance systemsblind spot control lane change assistance etc) Among ADASsolutions the most ambitious examples are related to monitoringsystems [21] [29] [33] [43] they parse the attention behavior ofthe driver together with the road scene to predict potentially unsafemanoeuvres and act on the car in order to avoid them ndash eitherby signaling the driver or braking However all these approachessuffer from the complexity of capturing the true driverrsquos attentionand rely on a limited set of fixed safety-inspired rules Herewe shift the problem from a personal level (what the driver islooking at) to a task-driven level (what most drivers would lookat) introducing a computer vision model able to to replicate thehuman attentional behavior during the driving task

We achieve this result in two stages First we conduct adata-driven study on driversrsquo gaze fixations under differentcircumstances and scenarios The study concludes that thesemantic of the scene the speed and bottom-up features allinfluence the driverrsquos gaze Second we advocate for the existenceof common gaze patterns that are shared among different driversWe empirically demonstrate the existence of such patterns bydeveloping a deep learning model that can profitably learn topredict where a driver would be looking at in a specific situationTo this aim we recorded and annotated 555000 frames (approx

6 hours) of driving sequences in different traffic and weatherconditions the DR(eye)VE dataset For every frame we acquired

bull All authors are with the Department of Engineering ldquoEnzo FerrarirdquoUniversity of Modena and Reggio Emilia ItalyE-mail namesurnameunimoreitlowast indicates equal contribution

(a) RGB frame (b) optical flow

(c) semantic segmentation (d) predicted map

Fig 1 An example of visual attention while driving (d) estimated fromour deep model using (a) raw video (b) optical flow and (c) semanticsegmentation

the driverrsquos gaze through an accurate eye tracking device andregistered such data to the external view recorded from aroof-mounted camera The DR(eye)VE data richness enablesus to train an end-to-end deep network that predicts salientregions in car-centric driving videos The network we propose isbased on three branches which estimate attentional maps froma) visual information of the scene b) motion cues (in terms ofoptical flow) and c) semantic segmentation (Fig 1) In contrast tothe majority of experiments which are conducted in controlledlaboratory settings or employ sequences of unrelated images [11][30] [68] we train our model on data acquired on the fieldFinal results demonstrate the ability of the network to generalizeacross different day times different weather conditions differentlandscapes and different driversEventually we believe our work can be complementary to thecurrent semantic segmentation and object detection literature [13][44] [45] [70] [76] by providing a diverse set of informationAccording to [61] the act of driving combines complex attentionmechanisms guided by the driverrsquos past experience short

arX

iv1

705

0385

4v3

[cs

CV

] 6

Jun

201

8

2

reactive times and strong contextual constraints Thus very littleinformation is needed to drive if guided by a strong focus ofattention (FoA) on a limited set of targets our model aims atpredicting them

The paper is organized as follows In Sec 2 related worksabout computer vision and gaze prediction are provided to frameour work in the current state-of-the-art scenario Sec 3 describesthe DR(eye)VE dataset and some insights about several attentionpatterns that human drivers exhibit Sec 4 illustrates the proposeddeep network to replicate such human behavior and Sec 5 reportsthe performed experiments

2 RELATED WORK

The way humans favor some entities in the scene along withkey factors guiding eye fixations in presence of a given task (egvisual search) has been extensively studied for decades [66][74] The main difficulty that rises when approaching thesubject is the variety of perspectives under which it can be castIndeed visual attention has been approached by psychologistsneurobiologists and computer scientists making the field highlyinterdisciplinary [20] We are particularly interested in thecomputational perspective in which predicting human attention isoften formalized as an estimation task delivering the probabilityof each point in a given scene to attract the observerrsquos gaze

Attention in images and videos Coherently with psychologicalliterature that identifies two distinct mechanisms guiding humaneye fixations [63] computational models for FoA predictionbranch into two families top-down and bottom-up strategiesFormer approaches aim at highlighting objects and cues that couldbe meaningful in the context of a given task For this reasonsuch methods are also known as task-driven Usually top-downcomputer vision models are built to integrate semantic contextualinformation in the attention prediction process [64] This can beachieved by either merging estimated maps at different levelsof scale and abstraction [24] or including a-priori cues aboutrelevant objects for the task at hand [17] [22] [75] Humanfocus in complex interactive environments (eg while playingvideogames) [9] [49] [50] follows task-driven behaviors as well

Conversely bottom-up models capture salient objects orevents naturally popping out in the image independently of theobserver the undergoing task and other external factors Thistask is widely known in literature as visual saliency predictionIn this context computational models focus on spotting visualdiscontinuities either by clustering features or consideringthe rarity of image regions locally [39] [57] or globally [1][14] [77] For a comprehensive review of visual attentionprediction methods we refer the reader to [7] Recently thesuccess of deep networks involved both task-driven attentionand saliency prediction as models have become more powerfulin both paradigms achieving state-of-the-art results on publicbenchmarks [15] [16] [28] [34] [37]In video attention prediction and saliency estimation are morecomplex with respect to still images since motion heavily affectshuman gaze Some models merge bottom-up saliency with motionmaps either by means of optical flow [79] or feature tracking [78]Other methods enforce temporal dependencies between bottom-upfeatures in successive frames Both supervised [59] [79] andunsupervised [42] [72] [73] feature extraction can be employed

and temporal coherence can be achieved either by conditioning thecurrent prediction on information from previous frames [54] or bycapturing motion smoothness with optical flow [59] [79] Whiledeep video saliency models still lack an interesting work is [4]which relies on a recurrent architecture fed with clip encodings topredict the fixation map by means of a Gaussian Mixture Model(GMM) Nevertheless most methods limit to bottom-up featuresaccounting for just visual discontinuities in terms of textures orcontours Our proposal instead is specifically tailored to thedriving task and fuses the bottom-up information with semanticsand motion elements that have emerged as attention factors fromthe analysis of the DR(eye)VE datasetAttention and driving Prior works addressed the task ofdetecting saliency and attention in the specific context of assisteddriving In such cases however gaze and attentive mechanismshave been mainly studied for some driving sub-tasks onlyoften acquiring gaze maps from on-screen images Bremond etal [58] presented a model that exploits visual saliency witha non-linear SVM classifier for the detection of traffic signsThe validation of this study was performed in a laboratorynon-realistic setting emulating an in-car driving session A morerealistic experiment [10] was then conducted with a larger set oftargets eg including pedestrians and bicyclesDriverrsquos gaze has also been studied in a pre-attention context bymeans of intention prediction relying only on fixation maps [52]The study in [68] inspects the driverrsquos attention at T junctions inparticular towards pedestrians and motorbikes and exploits objectsaliency to avoid the looked-but-failed-to-see effect In absence ofeye tracking systems and reliable gaze data [5] [19] [62] [69]focus on driversrsquo head detecting facial landmarks to predict headorientation Such mechanisms are more robust to varying lightingconditions and occlusions but there is no certainty about theadherence of predictions to the true gaze during the driving task

Datasets Many image saliency datasets have been releasedin the past few years improving the understanding of the humanvisual attention and pushing computational models forward Mostof these datasets include no motion information as saliencyground truth maps are built by aggregating fixations of severalusers within the same still image Usually a Gaussian filteringpost-processing step is employed on recorded data in order tosmooth such fixations and integrate their spatial locations Somedatasets such as the MIT saliency benchmark [11] were labeledthrough an eye tracking system while others like the SALICONdataset [30] relied on users clicking on salient image locationsWe refer the reader to [8] for a comprehensive list of availabledatasets On the contrary datasets addressing human attentionprediction in video still lack Up to now Action in the Eye [41]represents the most important contribution since it consistsin the largest video dataset accompanied by gaze and fixationannotations That information however is collected in the contextof action recognition so it is heavily task-driven A few datasetsaddress directly the study of attention mechanisms while drivingas summarized in Tab 1 However these are mostly restrictedto limited settings and are not publicly available In some ofthem [58] [68] fixation and saliency maps are acquired duringan in-lab simulated driving experience In-lab experiments enableseveral attention drifts that are influenced by external factors(eg monitor distance and others) rather than the primary taskof driving [61] A few in-car datasets exist [10] [52] but wereprecisely tailored to force the driver to fulfill some tasks such

3

Fig 2 Examples taken from a random sequence of DR(eye)VE From left to right frames from the eye tracking glasses with gaze data from theroof-mounted camera temporal aggregated fixation maps (as defined in Sec 3) and overlays between frames and fixation maps

as looking at people or traffic signs Coarse gaze informationis also available in [19] while the external road scene imagesare not acquired We believe that the dataset presented in [52]is among the others the closer to our proposal Yet videosequences are collected from one driver only it is not publiclyavailable Conversely our DR(eye)VE dataset is the first datasetaddressing driverrsquos focus of attention prediction that is madepublicly available Furthermore it includes sequences fromseveral different drivers and presents a high variety of landscapes(ie highway downtown and countryside) lighting and weatherconditions

3 THE DR(EYE)VE PROJECT

In this section we present the DR(eye)VE dataset (Fig 2)the protocol adopted for video registration and annotation theautomatic processing of eye-tracker data and the analysis of thedriverrsquos behavior in different conditions

The dataset The DR(eye)VE dataset consists of 555000frames divided in 74 sequences each of which is 5 minutes longEight different drivers of varying age from 20 to 40 including7 men and a woman took part to the driving experiment thatlasted more than two months Videos were recorded in differentcontexts both in terms of landscape (downtown countrysidehighway) and traffic condition ranging from traffic-free to highlycluttered scenarios They were recorded in diverse weatherconditions (sunny rainy cloudy) and at different hours of the day(both daytime and night) Tab 1 recaps the dataset features andTab 2 compares it with other related proposals DR(eye)VE iscurrently the largest publicly available dataset including gaze anddriving behavior in automotive settings

The Acquisition System The driverrsquos gaze informationwas captured using the commercial SMI ETG 2w Eye TrackingGlasses (ETG) ETG capture attention dynamics also in presenceof head pose changes which occur very often during the task ofdriving While a frontal camera acquires the scene at 720p30fpsusers pupils are tracked at 60Hz Gaze information are providedin terms of eye fixations and saccade movements ETG was

manually calibrated before each sequence for every driverSimultaneously videos from the car perspective were acquiredusing the GARMIN VirbX camera mounted on the car roof(RMC Roof-Mounted Camera) Such sensor captures frames at1080p25fps and includes further information such as GPS dataaccelerometer and gyroscope measurements

Video-gaze registration The dataset has been processed tomove the acquired gaze from the egocentric (ETG) view to thecar (RMC) view The latter features a much wider field of view(FoV) and can contain fixations that are out of the egocentricview For instance this can occur whenever the driver takes apeek at something at the border of this FoV but doesnrsquot move hishead For every sequence the two videos were manually alignedto cope with the difference in sensors framerate Videos were thenregistered frame-by-frame through a homographic transformationthat projects fixation points across views More formally at eachtimestep t the RMC frame ItRMC and the ETG frame ItETGare registered by means of a homography matrix HETGrarrRMC computed by matching SIFT descriptors [38] from one viewto the other (see Fig 3) A further RANSAC [18] procedureensures robustness to outliers While homographic mappingis theoretically sound only across planar views - which is notthe case of outdoor environments - we empirically found thatprojecting an object from one image to another always recoveredthe correct position This makes sense if the distance between theprojected object and the camera is far greater than the distancebetween the object and the projective plane In Sec 13 of thesupplementary material we derive formal bounds to explain thisphenomena

Fixation map computation The pipeline discussed aboveprovides a frame-level annotation of the driverrsquos fixations Incontrast to image saliency experiments [11] there is no clear andindisputable protocol for obtaining continuous maps from rawfixations when acquired in task-driven real-life scenarios Thisis even more evident when fixations are collected in task-drivenreal-life scenarios The main motivation resides in the fact thatobserverrsquos subjectivity cannot be removed by averaging different

4

TABLE 1Summary of the DR(eye)VE dataset characteristics The dataset was designed to embody the most possible diversity in the combination of

different features The reader is referred to either the additional material or to the dataset presentation [2] for details on each sequence

Videos Frames Drivers Weather conditions Lighting Gaze Info Metadata Camera Viewpoint

74 555000 8sunny day raw fixations GPS driver (720p)cloudy evening gaze map car speed car (1080p)rainy night pupil dilation car course

TABLE 2A comparison between DR(eye)VE and other datasets

Dataset Frames Drivers Scenarios Annotations Real-world Public

Pugeault et al [52] 158668 ndash Countryside HighwayDowntown

Gaze MapsDriverrsquos Actions Yes No

Simon et al [58] 40 30 Downtown Gaze Maps No NoUnderwood et al [68] 120 77 Urban Motorway ndash No NoFridman et al [19] 1860761 50 Highway 6 Gaze Location Classes Yes No

DR(eye)VE [2] 555000 8 Countryside HighwayDowntown

Gaze MapsGPS Speed Course Yes Yes

observersrsquo fixations Indeed two different observers cannotexperience the same scene at the same time (eg two driverscannot be at the same time in the same point of the street) Theonly chance to average among different observers would be theadoption of a simulation environment but it has been provedthat the cognitive load in controlled experiments is lower thanin real test scenarios and it effects the true attention mechanismof the observer [55] In our preliminary DR(eye)VE release[2] fixation points were aggregated and smoothed by means ofa temporal sliding window In such a way temporal filteringdiscarded momentary glimpses that contain precious informationabout the driverrsquos attention Following the psychological protocolin [40] and [25] this limitation was overcome in the currentrelease where the new fixation maps were computed withouttemporal smoothing Both [40] and [25] highlight the high degreeof subjectivity of scene scanpaths in short temporal windows (lt 1sec) and suggest to neglect the fixations pop-out order withinsuch windows This mechanism also ameliorates the inhibitionof return phenomenon that may prevent interesting objects to beobserved twice in short temporal intervals [27] [51] leading tothe underestimation of their importanceMore formally the fixation map Ft for a frame at time t is builtby accumulating projected gaze points in a temporal slidingwindow of k = 25 frames centered in t For each time step t+ iin the window where i isin minusk2 minus

k2 + 1 k2 minus 1 k2 gaze

points projections on Ft are estimated through the homographytransformation Ht

t+i that projects points from the image plane atframe t+ i namely pt+i to the image plane in Ft A continuousfixation map is obtained from the projected fixations by centeringon each of them a multivariate Gaussian having a diagonalcovariance matrix Σ (the spatial variance of each variable is set to

Fig 3 Registration between the egocentric and roof-mounted cameraviews by means of SIFT descriptor matching

Fig 4 Resulting fixation map from a 1 second integration (25 frames)The adoption of the max aggregation of equation 1 allows to account inthe final map two brief glances towards traffic lights

σ2s = 200 pixels) and taking the max value along the time axis

Ft(x y) = maxiisin(minus k

2 k2 )N ((x y) |Ht

t+i middot pt+iΣ) (1)

The Gaussian variance has been computed by averaging the ETGspatial acquisition errors on 20 observers looking at calibrationpatterns at different distances from 5 to 15 meters The describedprocess can be appreciated in Fig 4 Eventually each map Ft isnormalized to sum to 1 so that it can be considered a probabilitydistribution of fixation points

Labeling attention drifts Fixation maps exhibit a verystrong central bias This is common in saliency annotations [60]and even more in the context of driving For these reasons thereis a strong unbalance between lots of easy-to-predict scenariosand unfrequent but interesting hard-to-predict events

To enable the evaluation of computational models under suchcircumstances the DR(eye)VE dataset has been extended witha set of further annotations For each video subsequences whoseground truth poorly correlates with the average ground truth ofthat sequence are selected We employ Pearsonrsquos CorrelationCoefficient (CC) and select subsequences with CC lt 03 Thishappens when the attention of the driver focuses far from the

5

(a) Acting - 69 719 (b) Inattentive - 12 282

(c) Error - 22 893 (d) Subjective - 3 166

Fig 5 Examples of the categorization of frames where gaze is farfrom the mean Overall 108 060 frames (sim20 of DR(eye)VE) wereextended with this type of information

vanishing point of the road Examples of such subsequencesare depicted in Fig 5 Several human annotators inspected theselected frames and manually split them into (a) acting (b)inattentive (c) errors and (d) subjective eventsbull errors can happen either due to failures in the measuring tool

(eg in extreme lighting conditions) or in the successive dataprocessing phase (eg SIFT matching)

bull inattentive subsequences occur when the driver focuses hisgaze on objects unrelated to the driving task (eg looking atan advertisement)

bull subjective subsequences describe situations in which theattention is closely related to the individual experience ofthe driver eg a road sign on the side might be an interestingelement to focus for someone that has never been on that roadbefore but might be safely ignored by someone who drivesthat road every day

bull acting subsequences include all the remaining onesActing subsequences are particularly interesting as the deviationof driverrsquos attention from the common central pattern denotes anintention linked to task-specific actions (eg turning changinglanes overtaking ) For these reasons subsequences of thiskind will have a central role in the evaluation of predictive modelsin Sec 5

31 Dataset analysisBy analyzing the dataset frames the very first insight is thepresence of a strong attraction of driverrsquos focus towards thevanishing point of the road that can be appreciated in Fig 6 Thesame phenomenon was observed in previous studies [6] [67] inthe context of visual search tasks We observed indeed that driversoften tend to disregard road signals cars coming from the opposite

(a) (b)

Fig 6 Mean frame (a) and fixation map (b) averaged across the wholesequence 02 highlighting the link between driverrsquos focus and the van-ishing point of the road

direction and pedestrians on sidewalks This is an effect of humanperipheral vision [56] that allows observers to still perceive andinterpret stimuli out of - but sufficiently close to - their focus ofattention (FoA) A driver can therefore achieve a larger area ofattention by focusing on the roadrsquos vanishing point due to thegeometry of the road environment many of the objects worth ofattention are coming from there and have already been perceivedwhen distantMoreover the gaze location tends to drift from this central attrac-

tor when the context changes in terms of car speed and landscapeIndeed [53] suggests that our brain is able to compensate spatiallyor temporally dense information by reducing the visual field sizeIn particular as the car travels at higher speed the temporal densityof information (ie the amount of information that the driver needsto elaborate per unit of time) increases this causes the usefulvisual field of the driver to shrink [53] We also observe thisphenomenon in our experiments as shown in Fig 7DR(eye)VE data also highlight that the driverrsquos gaze is attractedtowards specific semantic categories To reach the above conclu-sion the dataset is analysed by means of the semantic segmenta-tion model in [76] and the distribution of semantic classes withinthe fixation map evaluated More precisely given a segmentedframe and the corresponding fixation map the probability for eachsemantic class to fall within the area of attention is computed asfollows First the fixation map (which is continuous in [0 1]) isnormalized such that the maximum value equals 1 Then nine bi-nary maps are constructed by thresholding such continuous valueslinearly in the interval [0 1] As the threshold moves towards 1(the maximum value) the area of interest shrinks around the realfixation points (since the continuous map is modeled by meansof several Gaussians centered in fixation points see previoussection) For every threshold a histogram over semantic labelswithin the area of interest is built by summing up occurrencescollected from all DR(eye)VE frames Fig 8 displays the resultfor each class the probability of a pixel to fall within the region ofinterest is reported for each threshold value The figure providesinsight about which categories represent the real focus of attentionand which ones tend to fall inside the attention region just byproximity with the formers Object classes that exhibit a positivetrend such as road vehicles and people are the real focus ofthe gaze since the ratio of pixels classified accordingly increaseswhen the observed area shrinks around the fixation point In abroader sense the figure suggests that despite while driving ourfocus is dominated by road and vehicles we often observe specificobjects categories even if they contain little information useful todrive

4 MULTI-BRANCH DEEP ARCHITECTURE FORFOCUS OF ATTENTION PREDICTION

The DR(eye)VE dataset is sufficiently large to allow theconstruction of a deep architecture to model common attentionalpatterns Here we describe our neural network model to predicthuman FoA while driving

Architecture design In the context of high level videoanalysis (eg action recognition and video classification) ithas been shown that a method leveraging single frames canbe outperformed if a sequence of frames is used as inputinstead [31] [65] Temporal dependencies are usually modeledeither by 3D convolutional layers [65] tailored to capture short

6

|Σ| = 249times 109 |Σ| = 203times 109 |Σ| = 861times 108 |Σ| = 102times 109 |Σ| = 928times 108

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 7 As speed gradually increases driverrsquos attention converges towards the vanishing point of the road (a) When the car is approximatelystationary the driver is distracted by many objects in the scene (b-e) As the speed increases the driverrsquos gaze deviates less and less from thevanishing point of the road To measure this effect quantitatively a two-dimensional Gaussian is fitted to approximate the mean map for each speedrange and the determinant of the covariance matrix Σ is reported as an indication of its spread (the determinant equals the product of eigenvalueseach of which measures the spread along a different data dimension) The bar plots illustrate the amount of downtown (red) countryside (green)and highway (blue) frames that concurred to generate the average gaze position for a specific speed range Best viewed on screen

Pro

babi

lity

offix

atio

n

road0

01

02

03

04

sidewalk buildings traffic signs trees flowerbed sky vehicles people0

0005

001

cycles

Fig 8 Proportion of semantic categories that fall within the driverrsquos fixation map when thresholded at increasing values (from left to right) Categoriesexhibiting a positive trend (eg road and vehicles) suggest a real attention focus while a negative trend advocates for an awareness of the objectwhich is only circumstantial See Sec 31 for details

conv3D64

conv3D128

conv3D256

conv3D256

conv2D1

pool3D

pool3D

pool3D

up2D

conv3D

64

conv3D conv3D conv3Dpool3D pool3D pool3D

up2D

128 256 256 1

conv2D

Fig 9 The COARSE module is made of an encoder based on C3D network [65] followed by a bilinear upsampling (bringing representations back tothe resolution of the input image) and a final 2D convolution During feature extraction the temporal axis is lost due to 3D pooling All convolutionallayers are preceded by zero paddings in order keep borders and all kernels have size 3 along all dimensions Pooling layers have size and strideof (1 2 2 4) and (2 2 2 1) along temporal and spatial dimensions respectively All activations are ReLUs

range correlations or by recurrent architectures (eg LSTMGRU) that can model longer term dependencies [3] [47] Ourmodel follows the former approach relying on the assumptionthat a small time window (eg half a second) holds sufficientcontextual information for predicting where the driver wouldfocus in that moment Indeed human drivers can take even lesstime to react to an unexpected stimulus Our architecture takes asequence of 16 consecutive frames (asymp 065s) as input (called clipsfrom now on) and predicts the fixation map for the last frame ofsuch clipMany of the architectural choices made to design the networkcome from insights from the dataset analysis presented in Sec31In particular we rely on the following resultsbull the driversrsquo FoA exhibits consistent patterns suggesting that

it can be reproduced by a computational modelbull the driversrsquo gaze is affected by a strong prior on objects

semantics eg drivers tend to focus on items lying on theroad

bull motion cues like vehicle speed are also key factors thatinfluence gaze

Accordingly the model output merges three branches with identi-cal architecture unshared parameters and different input domains

the RGB image the semantic segmentation and the optical flowfield We call this architecture multi-branch model Followinga bottom-up approach in Sec 41 the building blocks of eachbranch are motivated and described Later in Sec 42 it will beshown how the branches merge into the final model

41 Single FoA branch

Each branch of the multi-branch model is a two-input two-output architecture composed of two intertwined streams The aimof this peculiar setup is to prevent the network from learninga central bias that would otherwise stall the learning in earlytraining stages 1 To this end one of the streams is given as input(output) a severely cropped portion of the original image (groundtruth) ensuring a more uniform distribution of the true gaze andruns through the COARSE module described below Similarly theother stream uses the COARSE module to obtain a rough predictionover the full resized image and then refines it through a stack ofadditional convolutions called REFINE model At test time onlythe output of the REFINE stream is considered Both streams rely

1 For further details the reader can refer to Sec 14 and Sec 15 of thesupplementary material

7

last frame

shared weights

Videoclip

CROP

RESIZE

COARSE REFINE outputinput

448

448

cropped videoclip

112

112

112

112

resized videoclip

Fig 10 A single FoA branch of our prediction architecture The COARSE module (see Fig 9) is applied to both a cropped and a resized version ofthe input tensor which is a videoclip of 16 consecutive frames The cropped input is used during training to augment the data and the variety ofground truth fixation maps The prediction of the resized input is stacked with the last frame of the videoclip and fed to a stack of convolutional layers(refinement module) with the aim of refining the prediction Training is performed end-to-end and weights between COARSE modules are shared Attest time only the refined predictions are used Note that the complete model is composed of three of these branches (see Fig 11) each of whichpredicting visual attention for different inputs (namely image optical flow and semantic segmentation) All activations in the refinement module areLeakyReLU with α = 10minus3 except for the last single channel convolution that features ReLUs Crop and resize streams are highlighted by lightblue and orange arrows respectively

on the COARSE module the convolutional backbone (with sharedweights) which provides the rough estimate of the attentional mapcorresponding to a given clip This component is detailed in Fig 9The COARSE module is based on the C3D architecture [65] thatencodes video dynamics by applying a 3D convolutional kernelon the 4D input tensor As opposed to 2D convolutions that stridealong the width and height dimension of the input tensor a 3Dconvolution also strides along time Formally the j-th feature mapin the i-th layer at position (x y) at time t is computed as

vxytij = bij +summ

Piminus1sump=0

Qiminus1sumq=0

Riminus1sumr=0

wpqrijmvx+py+qt+riminus1m (2)

where m indexes different input feature maps wpqrijm is the valueat the position (p q) at time r of the kernel connected to the m-thfeature map and Pi Qi and Ri are the dimensions of the kernelalong width height and temporal axis respectively bij is the biasfrom layer i to layer jFrom C3D only the most general-purpose features are retainedby removing the last convolutional layer and the fully connectedlayers which are strongly linked to the original action recognitiontask The size of the last pooling layer is also modified in order tocover the remaining temporal dimension entirely This collapsesthe tensor from 4D to 3D making the output independent of timeEventually a bilinear upsampling brings the tensor back to theinput spatial resolution and a 2D convolution merges all featuresinto one channel See Fig 9 for additional details on the COARSEmodule

Training the two streams together The architecture of asingle FoA branch is depicted in Fig 10 During training the firststream feeds the COARSE network with random crops forcingthe model to learn the current focus of attention given visualcues rather than prior spatial location The C3D training processdescribed in [65] employs a 128 times 128 image resize and thena 112 times 112 random crop However the small difference in thetwo resolutions limits the variance of gaze position in ground

truth fixation maps and is not sufficient to avoid the attractiontowards the center of the image For this reason training imagesare resized to 256times 256 before being cropped to 112times 112 Thiscrop policy generates samples that cover less than a quarter ofthe original image thus ensuring a sufficient variety in predictiontargets This comes at the cost of a coarser prediction as cropsget smaller the ratio of pixels in the ground truth covered by gazeincreases leading the model to learn larger mapsIn contrast the second stream feeds the same COARSE modelwith the same images this time resized to 112 times 112 ndash and notcropped The coarse prediction obtained from the COARSE modelis then concatenated with the final frame of the input clip iethe frame corresponding to the final prediction Eventually theconcatenated tensor goes through the REFINE module to obtaina higher resolution prediction of the FoAThe overall two-stream training procedure for a single branch issummarized in Algorithm 1

Training objective Prediction cost can be minimized interms of Kullback-Leibler divergence

DKL(Y Y ) =sumi

Y (i) log

(ε+

Y (i)

ε+ Y (i)

)(3)

where Y is the ground truth distribution Y is the prediction thesummation index i spans across image pixels and ε is a smallconstant that ensures numerical stability2 Since each single FoAbranch computes an error on both the cropped image stream andthe resized image stream the branch loss can be defined as

Lb(XbY) =summ

(DKL(φ(Y m)C(φ(Xm

b ))) +

DKL(Y mR(C(ψ(Xmb )) Xm

b )))

) (4)

2 Please note that DKL inputs are always normalized to be a validprobability distribution despite this may be omitted in notation to improveequations readability

8

Algorithm 1 TRAINING The model is trained in two steps first each branch is trained separately through iterations detailed inprocedure SINGLE_BRANCH_TRAINING_ITERATION then the three branches are fine-tuned altogether as shown by procedureMULTI_BRANCH_FINE-TUNING_ITERATION For clarity we omit from notation i) the subscript b denoting the current domain inall X x and y variables in the single branch iteration and ii) the normalization of the sum of the outputs from each branch in line 13

1 procedure A SINGLE_BRANCH_TRAINING_ITERATION

input domain data X = x1 x2 x16 true attentional map y of last frame x16 of videoclip Xoutput branch loss Lb computed on input sample (X y)

2 Xres larr resize(X (112 112))3 Xcrop ycrop larr get_crop((X y) (112 112))4 ycrop larr COARSE(Xcrop) get coarse prediction on uncentered crop5 y larr REFINE(stack(x16 upsample(COARSE(Xres)))) get refined prediction over whole image6 Lb(XY )larr DKL(ycropycrop) +DKL(yy) compute branch loss as in Eq 4

7 procedure B MULTI_BRANCH_FINE-TUNING_ITERATION

input data X = x1 x2 x16 for all domains true attentional map y of last frame x16 of videoclip Xoutput overall loss L computed on input sample (X y)

8 Xres larr resize(X (112 112))9 Xcrop ycrop larr get_crop((X y) (112 112))

10 for branch b isin RGB flow seg do11 ybcrop larr COARSE(Xbcrop ) as in line 4 of the above procedure12 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres )))) as in line 5 of the above procedure13 L(XY )larr DKL(ycrop

sumb ybcrop ) +DKL(y

sumb yb) compute overall loss as in Eq 5

where C and R denote COARSE and REFINE modules(Xm

b Ym) isin Xb times Y is the m-th training example in the

b-th domain (namely RGB optical flow semantic segmentation)and φ and ψ indicate the crop and the resize functions respectively

Inference step While the presence of the C(φ(Xmb )) stream

is beneficial in training to reduce the spatial bias at test timeonly the R(C(ψ(Xm

b )) Xmb )) stream producing higher quality

prediction is used The outputs of such stream from each branch bare then summed together as explained in the following section

42 Multi-Branch modelAs described at the beginning of this section and depicted inFig 11 the multi-branch model is composed of three iden-tical branches The architecture of each branch has already beendescribed in Sec 41 above Each branch exploits complementaryinformation from a different domain and contributes to the finalprediction accordingly In detail the first branch works in the RGBdomain and processes raw visual data about the scene XRGBThe second branch focuses on motion through the optical flowrepresentation Xflow described in [23] Eventually the last branchtakes as input semantic segmentation probability maps Xseg Forthis last branch the number of input channels depends on thespecific algorithm used to extract the results 19 in our setup (Yu

Algorithm 2 INFERENCE At test time the data extracted fromthe resized videoclip is input to the three branches and their outputis summed and normalized to obtain the final FoA prediction

input data X = x1 x2 x16 for all domainsoutput predicted FoA map y1 Xres larr resize(X (112 112))2 for branch b isin RGB flow seg do3 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres ))))4 y larr

sumb yb

sumi

sumb yb(i)

and Koltun [76]) The three independent predicted FoA maps aresummed and normalized to result in a probability distributionTo allow for larger batch size we choose to bootstrap each branchindependently by training it according to Eq 4 Then the completemulti-branch model which merges the three branches is fine-tuned with the following loss

L(X Y) =summ

(DKL(φ(Y m)

sumb

C(φ(Xmb ))) +

DKL(Y msumb

R(C(ψ(Xmb )) Xm

b )))

)

(5)The algorithm describing the complete inference over themulti-branch model in detailed in Alg 2

5 EXPERIMENTS

In this section we evaluate the performance of the proposedmulti-branch model First we start by comparing our modelagainst some baselines and other methods in literature Followingthe guidelines in [12] for the evaluation phase we rely on Pear-sonrsquos Correlation Coefficient (CC) and KullbackndashLeibler Diver-gence (DKL) measures Moreover we evaluate the InformationGain (IG) [35] measure to assess the quality of a predicted mapP with respect to a ground truth map Y in presence of a strongbias as

IG(P YB) =1

N

sumi

Yi[(log2(ε+ Pi)minus log2(ε+Bi)] (6)

where i is an index spanning all the N pixels in the image B thebias computed as the average training fixation map and ε ensuresnumerical stabilityFurthermore we conduct an ablation study to investigate howdifferent branches affect the final prediction and how their mutualinfluence changes in different scenarios We then study whetherour model captures the attention dynamics observed in Sec 31

9

nalprediction

+

FoAfrom motionow

clip

FoAfrom semanticsseg

clip

FoAfrom colorRGB

clip

Fig 11 The multi-branch model is composed of three different branches each of which has its own set of parameters and their predictions aresummed to obtain the final map Note that in this figure cropped streams are dropped to ease representation but are employed during training (asdiscussed in Sec 42 and depicted in Fig 10

Eventually we assess our model from a human perceptionperspective

Implementation details The three different pathways ofthe multi-branch model (namely FoA from color frommotion and from semantics) have been pre-trained independentlyusing the same cropping policy of Sec 42 and minimizing theobjective function in Eq 4 Each branch has been respectively fedwithbull 16 frames clips in raw RGB color spacebull 16 frames clips with optical flow maps encoded as color

images through the flow field encoding [23]bull 16 frames clips holding semantic segmentation from [76]

encoded as 19 scalar activation maps one per segmentationclass

During individual branch pre-training clips were randomly mir-rored for data augmentation We employ Adam optimizer with pa-rameters as suggested in the original paper [32] with the exceptionof the learning rate that we set to 10minus4 Eventually batch size wasfixed to 32 and each branch was trained until convergence TheDR(eye)VE dataset is split into train validation and test set asfollows sequences 1-38 are used for training sequences 39-74 fortesting The 500 frames in the middle of each training sequenceconstitute the validation setMoreover the complete multi-branch architecture was fine-tuned using the same cropping and data augmentation strategiesminimizing cost function in Eq 5 In this phase batch size was setto 4 due to GPU memory constraints and learning rate value waslowered to 10minus5 Inference time of each branch of our architectureis asymp 30 milliseconds per videoclip on an NVIDIA Titan X

51 Model evaluation

In Tab 3 we report results of our proposal against other state-of-the-art models [4] [15] [46] [59] [72] [73] evaluated both onthe complete test set and on acting subsequences only All thecompetitors with the exception of [46] are bottom-up approachesand mainly rely on appearance and motion discontinuities To testthe effectiveness of deep architectures for saliency prediction wecompare against the Multi-Level Network (MLNet) [15] which

scored favourably in the MIT300 saliency benchmark [11] andthe Recurrent Mixture Density Network (RMDN) [4] whichrepresents the only deep model addressing video saliency WhileMLNet works on images discarding the temporal informationRMDN encodes short sequences in a similar way to our COARSEmodule and then relies on a LSTM architecture to model longterm dependencies and estimates the fixation map in terms of aGMM To favor the comparison both models were re-trained onthe DR(eye)VE datasetResults highlight the superiority of our multi-branch archi-tecture on all test sequences The gap in performance with respectto bottom-up unsupervised approaches [72] [73] is higher and ismotivated by the peculiarity of the attention behavior within thedriving context which calls for a task-oriented training procedureMoreover MLNetrsquos low performance testifies for the need of ac-counting for the temporal correlation between consecutive framesthat distinguishes the tasks of attention prediction in images andvideos Indeed RMDN processes video inputs and outperformsMLNet on both DKL and IG metrics performing comparablyon CC Nonetheless its performance is still limited indeedqualitative results reported in Fig 12 suggest that long termdependencies captured by its recurrent module lead the networktowards the regression of the mean discarding contextual and

TABLE 3Experiments illustrating the superior performance of the

multi-branch model over several baselines and competitors Wereport both the average across the complete test sequences and only

the acting frames

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

Baseline Gaussian 040 216 -049 026 241 003Baseline Mean 051 160 000 022 235 000

Mathe et al [59] 004 330 -208 - - -Wang et al [72] 004 340 -221 - - -Wang et al [73] 011 306 -172 - - -

MLNet [15] 044 200 -088 032 235 -036RMDN [4] 041 177 -006 031 213 031

Palazzi et al [46] 055 148 -021 037 200 020multi-branch 056 140 004 041 180 051

10

Input frame GT multi-branch [46] [4] [15]

Fig 12 Qualitative assessment of the predicted fixation maps From left to right input clip ground truth map our prediction prediction of theprevious version of the model [46] prediction of RMDN [4] and prediction of MLNet [15]

frame-specific variations that would be preferrable to keep Tosupport this intuition we measure the average DKL betweenRMDN predictions and the mean training fixation map (Base-line Mean) resulting in a value of 011 Being lower than thedivergence measured with respect to groundtruth maps this valuehighlights the closer correlation to a central baseline rather thanto groundtruth Eventually we also observe improvements withrespect to our previous proposal [46] that relies on a more com-plex backbone model (also including a deconvolutional module)and processes RGB clips only The gap in performance residesin the greater awareness of our multi-branch architecture ofthe aspects that characterize the driving task as emerged fromthe analysis in Sec 31 The positive performances of our modelare also confirmed when evaluated on the acting partition of thedataset We recall that acting indicates sub-sequences exhibiting asignificant task-driven shift of attention from the center of theimage (Fig 5) Being able to predict the FoA also on actingsub-sequences means that the model captures the strong centeredattention bias but is capable of generalizing when required by thecontextThis is further shown by the comparison against a centeredGaussian baseline (BG) and against the average of all trainingset fixation maps (BM) The former baseline has proven effectiveon many image saliency detection tasks [11] while the latterrepresents a more task-driven version The superior performanceof the multi-branch model wrt baselines highlights thatdespite the attention is often strongly biased towards the vanishingpoint of the road the network is able to deal with sudden task-driven changes in gaze direction

52 Model analysis

In this section we investigate the behavior of our proposed modelunder different landscapes time of day and weather (Sec 521)we study the contribution of each branch to the FoA predictiontask (Sec 522) and we compare the learnt attention dynamicsagainst the one observed in the human data (Sec 523)

12

13

14

15

16

17

18

19

Downt

own

Coun

trysid

e

High

way

Mornin

g

Even

ingNi

ght

Sunn

y

Cloud

yRa

iny

DKL

Multi-branchImage

FlowSegm

________________________Landscape ________________________Time of day ________________________Weather

Fig 13 DKL of the different branches in several conditions (from left toright downtown countryside highway morning evening night sunnycloudy rainy) Underlining highlights difference of aggregation in termsof landscape time of day and weather Please note that lower DKL

indicates better predictions

521 Dependency on driving environment

The DR(eye)VE data has been recorded under varying land-scapes time of day and weather conditions We tested our modelin all such different driving conditions As would be expectedFig 13 shows that the human attention is easier to predict inhighways rather than downtown where the focus can shift towardsmore distractors The model seems more reliable in eveningscenarios rather than morning or night where we observed betterlightning conditions and lack of shadows over-exposure and soon Lastly in rainy conditions we notice that human gaze iseasier to model possibly due to the higher level of awarenessdemanded to the driver and his consequent inability to focus awayfrom vanishing point To support the latter intuition we measuredthe performance of BM baseline (ie the average training fixationmap) grouped for weather condition As expected theDKL valuein rainy weather (153) is significantly lower than the ones forcloudy (161) and sunny weather (175) highlighting that whenrainy the driver is more focused on the road

11

|Σ| = 250times 109 |Σ| = 157times 109 |Σ| = 349times 108 |Σ| = 120times 108 |Σ| = 538times 107

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 14 Model prediction averaged across all test sequences and grouped by driving speed As the speed increases the area of the predicted mapshrinks recalling the trend observed in ground truth maps As in Fig 7 for each map a two dimensional Gaussian is fitted and the determinant ofits covariance matrix Σ is reported as a measure of the spread

road sidewalk buildings traffic signs trees flowerbed sky vehicles people cycles10

-4

10-3

10-2

10-1

100

prob

abilit

y of

fixa

tions

(log

sca

le)

Fig 15 Comparison between ground truth (gray bars) and predictedfixation maps (colored bars) when used to mask semantic segmentationof the scene The probability of fixation (in log-scale) for both groundtruth and model prediction is reported for each semantic class Despiteabsolute errors exist the two bar series agree on the relative importanceof different categories

522 Ablation study

In order to validate the design of the multi-branch model(see Sec 42) here we study the individual contributions of thedifferent branches by disabling one or more of themResults in Tab 4 show that the RGB branch plays a majorrole in FoA prediction The motion stream is also beneficialand provides a slight improvement that becomes clearer in theacting subsequences Indeed optical flow intrinsically captures avariety of peculiar scenarios that are non-trivial to classify whenonly color information is provided eg when the car is still at atraffic light or is turning The semantic stream on the other handprovides very little improvement In particular from Tab 4 and by

TABLE 4The ablation study performed on our multi-branch model I F and S

represent image optical flow and semantic segmentation branchesrespectively

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

I 0554 1415 -0008 0403 1826 0458F 0516 1616 -0137 0368 2010 0349S 0479 1699 -0119 0344 2082 0288

I+F 0558 1399 0033 0410 1799 0510I+S 0554 1413 -0001 0404 1823 0466F+S 0528 1571 -0055 0380 1956 0427

I+F+S 0559 1398 0038 0410 1797 0515

specifically comparing I+F and I+F+S a slight increase in the IGmeasure can be appreciated Nevertheless such improvement hasto be considered negligible when compared to color and motionsuggesting that in presence of efficiency concerns or real-timeconstraints the semantic stream can be discarded with little lossesin performance However we expect the benefit from this branchto increase as more accurate segmentation models will be released

523 Do we capture the attention dynamics

The previous sections validate quantitatively the proposed modelNow we assess its capability to attend like a human driverby comparing its predictions against the analysis performed inSec 31First we report the average predicted fixation map in severalspeed ranges in Fig 14 The conclusions we draw are twofoldi) generally the model succeeds in modeling the behavior ofthe driver at different speeds and ii) as the speed increasesfixation maps exhibit lower variance easing the modeling taskand prediction errors decreaseWe also study how often our model focuses on different semanticcategories in a fashion that recalls the analysis of Sec 31 butemploying our predictions rather than ground truth maps as focusof attention More precisely we normalize each map so thatthe maximum value equals 1 and apply the same thresholdingstrategy described in Sec 31 Likewise for each threshold valuea histogram over class labels is built by accounting all pixelsfalling within the binary map for all test frames This results innine histograms over semantic labels that we merge together byaveraging probabilities belonging to different threshold Fig 15shows the comparison Color bars represent how often the pre-dicted map focuses on a certain category while gray bars depictground truth behavior and are obtained by averaging histogramsin Fig 8 across different thresholds Please note that to highlightdifferences for low populated categories values are reported ona logarithmic scale The plot shows a certain degree of absoluteerror is present for all categories However in a broader sense ourmodel replicates the relative weight of different semantic classeswhile driving as testified by the importance of roads and vehiclesthat still dominate against other categories such as people andcycles that are mostly neglected This correlation is confirmed byKendall rank coefficient which scored 051 when computed onthe two bar series

53 Visual assessment of predicted fixation maps

To further validate the predictions of our model from the humanperception perspective 50 people with at least 3 years of driving

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

2

reactive times and strong contextual constraints Thus very littleinformation is needed to drive if guided by a strong focus ofattention (FoA) on a limited set of targets our model aims atpredicting them

The paper is organized as follows In Sec 2 related worksabout computer vision and gaze prediction are provided to frameour work in the current state-of-the-art scenario Sec 3 describesthe DR(eye)VE dataset and some insights about several attentionpatterns that human drivers exhibit Sec 4 illustrates the proposeddeep network to replicate such human behavior and Sec 5 reportsthe performed experiments

2 RELATED WORK

The way humans favor some entities in the scene along withkey factors guiding eye fixations in presence of a given task (egvisual search) has been extensively studied for decades [66][74] The main difficulty that rises when approaching thesubject is the variety of perspectives under which it can be castIndeed visual attention has been approached by psychologistsneurobiologists and computer scientists making the field highlyinterdisciplinary [20] We are particularly interested in thecomputational perspective in which predicting human attention isoften formalized as an estimation task delivering the probabilityof each point in a given scene to attract the observerrsquos gaze

Attention in images and videos Coherently with psychologicalliterature that identifies two distinct mechanisms guiding humaneye fixations [63] computational models for FoA predictionbranch into two families top-down and bottom-up strategiesFormer approaches aim at highlighting objects and cues that couldbe meaningful in the context of a given task For this reasonsuch methods are also known as task-driven Usually top-downcomputer vision models are built to integrate semantic contextualinformation in the attention prediction process [64] This can beachieved by either merging estimated maps at different levelsof scale and abstraction [24] or including a-priori cues aboutrelevant objects for the task at hand [17] [22] [75] Humanfocus in complex interactive environments (eg while playingvideogames) [9] [49] [50] follows task-driven behaviors as well

Conversely bottom-up models capture salient objects orevents naturally popping out in the image independently of theobserver the undergoing task and other external factors Thistask is widely known in literature as visual saliency predictionIn this context computational models focus on spotting visualdiscontinuities either by clustering features or consideringthe rarity of image regions locally [39] [57] or globally [1][14] [77] For a comprehensive review of visual attentionprediction methods we refer the reader to [7] Recently thesuccess of deep networks involved both task-driven attentionand saliency prediction as models have become more powerfulin both paradigms achieving state-of-the-art results on publicbenchmarks [15] [16] [28] [34] [37]In video attention prediction and saliency estimation are morecomplex with respect to still images since motion heavily affectshuman gaze Some models merge bottom-up saliency with motionmaps either by means of optical flow [79] or feature tracking [78]Other methods enforce temporal dependencies between bottom-upfeatures in successive frames Both supervised [59] [79] andunsupervised [42] [72] [73] feature extraction can be employed

and temporal coherence can be achieved either by conditioning thecurrent prediction on information from previous frames [54] or bycapturing motion smoothness with optical flow [59] [79] Whiledeep video saliency models still lack an interesting work is [4]which relies on a recurrent architecture fed with clip encodings topredict the fixation map by means of a Gaussian Mixture Model(GMM) Nevertheless most methods limit to bottom-up featuresaccounting for just visual discontinuities in terms of textures orcontours Our proposal instead is specifically tailored to thedriving task and fuses the bottom-up information with semanticsand motion elements that have emerged as attention factors fromthe analysis of the DR(eye)VE datasetAttention and driving Prior works addressed the task ofdetecting saliency and attention in the specific context of assisteddriving In such cases however gaze and attentive mechanismshave been mainly studied for some driving sub-tasks onlyoften acquiring gaze maps from on-screen images Bremond etal [58] presented a model that exploits visual saliency witha non-linear SVM classifier for the detection of traffic signsThe validation of this study was performed in a laboratorynon-realistic setting emulating an in-car driving session A morerealistic experiment [10] was then conducted with a larger set oftargets eg including pedestrians and bicyclesDriverrsquos gaze has also been studied in a pre-attention context bymeans of intention prediction relying only on fixation maps [52]The study in [68] inspects the driverrsquos attention at T junctions inparticular towards pedestrians and motorbikes and exploits objectsaliency to avoid the looked-but-failed-to-see effect In absence ofeye tracking systems and reliable gaze data [5] [19] [62] [69]focus on driversrsquo head detecting facial landmarks to predict headorientation Such mechanisms are more robust to varying lightingconditions and occlusions but there is no certainty about theadherence of predictions to the true gaze during the driving task

Datasets Many image saliency datasets have been releasedin the past few years improving the understanding of the humanvisual attention and pushing computational models forward Mostof these datasets include no motion information as saliencyground truth maps are built by aggregating fixations of severalusers within the same still image Usually a Gaussian filteringpost-processing step is employed on recorded data in order tosmooth such fixations and integrate their spatial locations Somedatasets such as the MIT saliency benchmark [11] were labeledthrough an eye tracking system while others like the SALICONdataset [30] relied on users clicking on salient image locationsWe refer the reader to [8] for a comprehensive list of availabledatasets On the contrary datasets addressing human attentionprediction in video still lack Up to now Action in the Eye [41]represents the most important contribution since it consistsin the largest video dataset accompanied by gaze and fixationannotations That information however is collected in the contextof action recognition so it is heavily task-driven A few datasetsaddress directly the study of attention mechanisms while drivingas summarized in Tab 1 However these are mostly restrictedto limited settings and are not publicly available In some ofthem [58] [68] fixation and saliency maps are acquired duringan in-lab simulated driving experience In-lab experiments enableseveral attention drifts that are influenced by external factors(eg monitor distance and others) rather than the primary taskof driving [61] A few in-car datasets exist [10] [52] but wereprecisely tailored to force the driver to fulfill some tasks such

3

Fig 2 Examples taken from a random sequence of DR(eye)VE From left to right frames from the eye tracking glasses with gaze data from theroof-mounted camera temporal aggregated fixation maps (as defined in Sec 3) and overlays between frames and fixation maps

as looking at people or traffic signs Coarse gaze informationis also available in [19] while the external road scene imagesare not acquired We believe that the dataset presented in [52]is among the others the closer to our proposal Yet videosequences are collected from one driver only it is not publiclyavailable Conversely our DR(eye)VE dataset is the first datasetaddressing driverrsquos focus of attention prediction that is madepublicly available Furthermore it includes sequences fromseveral different drivers and presents a high variety of landscapes(ie highway downtown and countryside) lighting and weatherconditions

3 THE DR(EYE)VE PROJECT

In this section we present the DR(eye)VE dataset (Fig 2)the protocol adopted for video registration and annotation theautomatic processing of eye-tracker data and the analysis of thedriverrsquos behavior in different conditions

The dataset The DR(eye)VE dataset consists of 555000frames divided in 74 sequences each of which is 5 minutes longEight different drivers of varying age from 20 to 40 including7 men and a woman took part to the driving experiment thatlasted more than two months Videos were recorded in differentcontexts both in terms of landscape (downtown countrysidehighway) and traffic condition ranging from traffic-free to highlycluttered scenarios They were recorded in diverse weatherconditions (sunny rainy cloudy) and at different hours of the day(both daytime and night) Tab 1 recaps the dataset features andTab 2 compares it with other related proposals DR(eye)VE iscurrently the largest publicly available dataset including gaze anddriving behavior in automotive settings

The Acquisition System The driverrsquos gaze informationwas captured using the commercial SMI ETG 2w Eye TrackingGlasses (ETG) ETG capture attention dynamics also in presenceof head pose changes which occur very often during the task ofdriving While a frontal camera acquires the scene at 720p30fpsusers pupils are tracked at 60Hz Gaze information are providedin terms of eye fixations and saccade movements ETG was

manually calibrated before each sequence for every driverSimultaneously videos from the car perspective were acquiredusing the GARMIN VirbX camera mounted on the car roof(RMC Roof-Mounted Camera) Such sensor captures frames at1080p25fps and includes further information such as GPS dataaccelerometer and gyroscope measurements

Video-gaze registration The dataset has been processed tomove the acquired gaze from the egocentric (ETG) view to thecar (RMC) view The latter features a much wider field of view(FoV) and can contain fixations that are out of the egocentricview For instance this can occur whenever the driver takes apeek at something at the border of this FoV but doesnrsquot move hishead For every sequence the two videos were manually alignedto cope with the difference in sensors framerate Videos were thenregistered frame-by-frame through a homographic transformationthat projects fixation points across views More formally at eachtimestep t the RMC frame ItRMC and the ETG frame ItETGare registered by means of a homography matrix HETGrarrRMC computed by matching SIFT descriptors [38] from one viewto the other (see Fig 3) A further RANSAC [18] procedureensures robustness to outliers While homographic mappingis theoretically sound only across planar views - which is notthe case of outdoor environments - we empirically found thatprojecting an object from one image to another always recoveredthe correct position This makes sense if the distance between theprojected object and the camera is far greater than the distancebetween the object and the projective plane In Sec 13 of thesupplementary material we derive formal bounds to explain thisphenomena

Fixation map computation The pipeline discussed aboveprovides a frame-level annotation of the driverrsquos fixations Incontrast to image saliency experiments [11] there is no clear andindisputable protocol for obtaining continuous maps from rawfixations when acquired in task-driven real-life scenarios Thisis even more evident when fixations are collected in task-drivenreal-life scenarios The main motivation resides in the fact thatobserverrsquos subjectivity cannot be removed by averaging different

4

TABLE 1Summary of the DR(eye)VE dataset characteristics The dataset was designed to embody the most possible diversity in the combination of

different features The reader is referred to either the additional material or to the dataset presentation [2] for details on each sequence

Videos Frames Drivers Weather conditions Lighting Gaze Info Metadata Camera Viewpoint

74 555000 8sunny day raw fixations GPS driver (720p)cloudy evening gaze map car speed car (1080p)rainy night pupil dilation car course

TABLE 2A comparison between DR(eye)VE and other datasets

Dataset Frames Drivers Scenarios Annotations Real-world Public

Pugeault et al [52] 158668 ndash Countryside HighwayDowntown

Gaze MapsDriverrsquos Actions Yes No

Simon et al [58] 40 30 Downtown Gaze Maps No NoUnderwood et al [68] 120 77 Urban Motorway ndash No NoFridman et al [19] 1860761 50 Highway 6 Gaze Location Classes Yes No

DR(eye)VE [2] 555000 8 Countryside HighwayDowntown

Gaze MapsGPS Speed Course Yes Yes

observersrsquo fixations Indeed two different observers cannotexperience the same scene at the same time (eg two driverscannot be at the same time in the same point of the street) Theonly chance to average among different observers would be theadoption of a simulation environment but it has been provedthat the cognitive load in controlled experiments is lower thanin real test scenarios and it effects the true attention mechanismof the observer [55] In our preliminary DR(eye)VE release[2] fixation points were aggregated and smoothed by means ofa temporal sliding window In such a way temporal filteringdiscarded momentary glimpses that contain precious informationabout the driverrsquos attention Following the psychological protocolin [40] and [25] this limitation was overcome in the currentrelease where the new fixation maps were computed withouttemporal smoothing Both [40] and [25] highlight the high degreeof subjectivity of scene scanpaths in short temporal windows (lt 1sec) and suggest to neglect the fixations pop-out order withinsuch windows This mechanism also ameliorates the inhibitionof return phenomenon that may prevent interesting objects to beobserved twice in short temporal intervals [27] [51] leading tothe underestimation of their importanceMore formally the fixation map Ft for a frame at time t is builtby accumulating projected gaze points in a temporal slidingwindow of k = 25 frames centered in t For each time step t+ iin the window where i isin minusk2 minus

k2 + 1 k2 minus 1 k2 gaze

points projections on Ft are estimated through the homographytransformation Ht

t+i that projects points from the image plane atframe t+ i namely pt+i to the image plane in Ft A continuousfixation map is obtained from the projected fixations by centeringon each of them a multivariate Gaussian having a diagonalcovariance matrix Σ (the spatial variance of each variable is set to

Fig 3 Registration between the egocentric and roof-mounted cameraviews by means of SIFT descriptor matching

Fig 4 Resulting fixation map from a 1 second integration (25 frames)The adoption of the max aggregation of equation 1 allows to account inthe final map two brief glances towards traffic lights

σ2s = 200 pixels) and taking the max value along the time axis

Ft(x y) = maxiisin(minus k

2 k2 )N ((x y) |Ht

t+i middot pt+iΣ) (1)

The Gaussian variance has been computed by averaging the ETGspatial acquisition errors on 20 observers looking at calibrationpatterns at different distances from 5 to 15 meters The describedprocess can be appreciated in Fig 4 Eventually each map Ft isnormalized to sum to 1 so that it can be considered a probabilitydistribution of fixation points

Labeling attention drifts Fixation maps exhibit a verystrong central bias This is common in saliency annotations [60]and even more in the context of driving For these reasons thereis a strong unbalance between lots of easy-to-predict scenariosand unfrequent but interesting hard-to-predict events

To enable the evaluation of computational models under suchcircumstances the DR(eye)VE dataset has been extended witha set of further annotations For each video subsequences whoseground truth poorly correlates with the average ground truth ofthat sequence are selected We employ Pearsonrsquos CorrelationCoefficient (CC) and select subsequences with CC lt 03 Thishappens when the attention of the driver focuses far from the

5

(a) Acting - 69 719 (b) Inattentive - 12 282

(c) Error - 22 893 (d) Subjective - 3 166

Fig 5 Examples of the categorization of frames where gaze is farfrom the mean Overall 108 060 frames (sim20 of DR(eye)VE) wereextended with this type of information

vanishing point of the road Examples of such subsequencesare depicted in Fig 5 Several human annotators inspected theselected frames and manually split them into (a) acting (b)inattentive (c) errors and (d) subjective eventsbull errors can happen either due to failures in the measuring tool

(eg in extreme lighting conditions) or in the successive dataprocessing phase (eg SIFT matching)

bull inattentive subsequences occur when the driver focuses hisgaze on objects unrelated to the driving task (eg looking atan advertisement)

bull subjective subsequences describe situations in which theattention is closely related to the individual experience ofthe driver eg a road sign on the side might be an interestingelement to focus for someone that has never been on that roadbefore but might be safely ignored by someone who drivesthat road every day

bull acting subsequences include all the remaining onesActing subsequences are particularly interesting as the deviationof driverrsquos attention from the common central pattern denotes anintention linked to task-specific actions (eg turning changinglanes overtaking ) For these reasons subsequences of thiskind will have a central role in the evaluation of predictive modelsin Sec 5

31 Dataset analysisBy analyzing the dataset frames the very first insight is thepresence of a strong attraction of driverrsquos focus towards thevanishing point of the road that can be appreciated in Fig 6 Thesame phenomenon was observed in previous studies [6] [67] inthe context of visual search tasks We observed indeed that driversoften tend to disregard road signals cars coming from the opposite

(a) (b)

Fig 6 Mean frame (a) and fixation map (b) averaged across the wholesequence 02 highlighting the link between driverrsquos focus and the van-ishing point of the road

direction and pedestrians on sidewalks This is an effect of humanperipheral vision [56] that allows observers to still perceive andinterpret stimuli out of - but sufficiently close to - their focus ofattention (FoA) A driver can therefore achieve a larger area ofattention by focusing on the roadrsquos vanishing point due to thegeometry of the road environment many of the objects worth ofattention are coming from there and have already been perceivedwhen distantMoreover the gaze location tends to drift from this central attrac-

tor when the context changes in terms of car speed and landscapeIndeed [53] suggests that our brain is able to compensate spatiallyor temporally dense information by reducing the visual field sizeIn particular as the car travels at higher speed the temporal densityof information (ie the amount of information that the driver needsto elaborate per unit of time) increases this causes the usefulvisual field of the driver to shrink [53] We also observe thisphenomenon in our experiments as shown in Fig 7DR(eye)VE data also highlight that the driverrsquos gaze is attractedtowards specific semantic categories To reach the above conclu-sion the dataset is analysed by means of the semantic segmenta-tion model in [76] and the distribution of semantic classes withinthe fixation map evaluated More precisely given a segmentedframe and the corresponding fixation map the probability for eachsemantic class to fall within the area of attention is computed asfollows First the fixation map (which is continuous in [0 1]) isnormalized such that the maximum value equals 1 Then nine bi-nary maps are constructed by thresholding such continuous valueslinearly in the interval [0 1] As the threshold moves towards 1(the maximum value) the area of interest shrinks around the realfixation points (since the continuous map is modeled by meansof several Gaussians centered in fixation points see previoussection) For every threshold a histogram over semantic labelswithin the area of interest is built by summing up occurrencescollected from all DR(eye)VE frames Fig 8 displays the resultfor each class the probability of a pixel to fall within the region ofinterest is reported for each threshold value The figure providesinsight about which categories represent the real focus of attentionand which ones tend to fall inside the attention region just byproximity with the formers Object classes that exhibit a positivetrend such as road vehicles and people are the real focus ofthe gaze since the ratio of pixels classified accordingly increaseswhen the observed area shrinks around the fixation point In abroader sense the figure suggests that despite while driving ourfocus is dominated by road and vehicles we often observe specificobjects categories even if they contain little information useful todrive

4 MULTI-BRANCH DEEP ARCHITECTURE FORFOCUS OF ATTENTION PREDICTION

The DR(eye)VE dataset is sufficiently large to allow theconstruction of a deep architecture to model common attentionalpatterns Here we describe our neural network model to predicthuman FoA while driving

Architecture design In the context of high level videoanalysis (eg action recognition and video classification) ithas been shown that a method leveraging single frames canbe outperformed if a sequence of frames is used as inputinstead [31] [65] Temporal dependencies are usually modeledeither by 3D convolutional layers [65] tailored to capture short

6

|Σ| = 249times 109 |Σ| = 203times 109 |Σ| = 861times 108 |Σ| = 102times 109 |Σ| = 928times 108

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 7 As speed gradually increases driverrsquos attention converges towards the vanishing point of the road (a) When the car is approximatelystationary the driver is distracted by many objects in the scene (b-e) As the speed increases the driverrsquos gaze deviates less and less from thevanishing point of the road To measure this effect quantitatively a two-dimensional Gaussian is fitted to approximate the mean map for each speedrange and the determinant of the covariance matrix Σ is reported as an indication of its spread (the determinant equals the product of eigenvalueseach of which measures the spread along a different data dimension) The bar plots illustrate the amount of downtown (red) countryside (green)and highway (blue) frames that concurred to generate the average gaze position for a specific speed range Best viewed on screen

Pro

babi

lity

offix

atio

n

road0

01

02

03

04

sidewalk buildings traffic signs trees flowerbed sky vehicles people0

0005

001

cycles

Fig 8 Proportion of semantic categories that fall within the driverrsquos fixation map when thresholded at increasing values (from left to right) Categoriesexhibiting a positive trend (eg road and vehicles) suggest a real attention focus while a negative trend advocates for an awareness of the objectwhich is only circumstantial See Sec 31 for details

conv3D64

conv3D128

conv3D256

conv3D256

conv2D1

pool3D

pool3D

pool3D

up2D

conv3D

64

conv3D conv3D conv3Dpool3D pool3D pool3D

up2D

128 256 256 1

conv2D

Fig 9 The COARSE module is made of an encoder based on C3D network [65] followed by a bilinear upsampling (bringing representations back tothe resolution of the input image) and a final 2D convolution During feature extraction the temporal axis is lost due to 3D pooling All convolutionallayers are preceded by zero paddings in order keep borders and all kernels have size 3 along all dimensions Pooling layers have size and strideof (1 2 2 4) and (2 2 2 1) along temporal and spatial dimensions respectively All activations are ReLUs

range correlations or by recurrent architectures (eg LSTMGRU) that can model longer term dependencies [3] [47] Ourmodel follows the former approach relying on the assumptionthat a small time window (eg half a second) holds sufficientcontextual information for predicting where the driver wouldfocus in that moment Indeed human drivers can take even lesstime to react to an unexpected stimulus Our architecture takes asequence of 16 consecutive frames (asymp 065s) as input (called clipsfrom now on) and predicts the fixation map for the last frame ofsuch clipMany of the architectural choices made to design the networkcome from insights from the dataset analysis presented in Sec31In particular we rely on the following resultsbull the driversrsquo FoA exhibits consistent patterns suggesting that

it can be reproduced by a computational modelbull the driversrsquo gaze is affected by a strong prior on objects

semantics eg drivers tend to focus on items lying on theroad

bull motion cues like vehicle speed are also key factors thatinfluence gaze

Accordingly the model output merges three branches with identi-cal architecture unshared parameters and different input domains

the RGB image the semantic segmentation and the optical flowfield We call this architecture multi-branch model Followinga bottom-up approach in Sec 41 the building blocks of eachbranch are motivated and described Later in Sec 42 it will beshown how the branches merge into the final model

41 Single FoA branch

Each branch of the multi-branch model is a two-input two-output architecture composed of two intertwined streams The aimof this peculiar setup is to prevent the network from learninga central bias that would otherwise stall the learning in earlytraining stages 1 To this end one of the streams is given as input(output) a severely cropped portion of the original image (groundtruth) ensuring a more uniform distribution of the true gaze andruns through the COARSE module described below Similarly theother stream uses the COARSE module to obtain a rough predictionover the full resized image and then refines it through a stack ofadditional convolutions called REFINE model At test time onlythe output of the REFINE stream is considered Both streams rely

1 For further details the reader can refer to Sec 14 and Sec 15 of thesupplementary material

7

last frame

shared weights

Videoclip

CROP

RESIZE

COARSE REFINE outputinput

448

448

cropped videoclip

112

112

112

112

resized videoclip

Fig 10 A single FoA branch of our prediction architecture The COARSE module (see Fig 9) is applied to both a cropped and a resized version ofthe input tensor which is a videoclip of 16 consecutive frames The cropped input is used during training to augment the data and the variety ofground truth fixation maps The prediction of the resized input is stacked with the last frame of the videoclip and fed to a stack of convolutional layers(refinement module) with the aim of refining the prediction Training is performed end-to-end and weights between COARSE modules are shared Attest time only the refined predictions are used Note that the complete model is composed of three of these branches (see Fig 11) each of whichpredicting visual attention for different inputs (namely image optical flow and semantic segmentation) All activations in the refinement module areLeakyReLU with α = 10minus3 except for the last single channel convolution that features ReLUs Crop and resize streams are highlighted by lightblue and orange arrows respectively

on the COARSE module the convolutional backbone (with sharedweights) which provides the rough estimate of the attentional mapcorresponding to a given clip This component is detailed in Fig 9The COARSE module is based on the C3D architecture [65] thatencodes video dynamics by applying a 3D convolutional kernelon the 4D input tensor As opposed to 2D convolutions that stridealong the width and height dimension of the input tensor a 3Dconvolution also strides along time Formally the j-th feature mapin the i-th layer at position (x y) at time t is computed as

vxytij = bij +summ

Piminus1sump=0

Qiminus1sumq=0

Riminus1sumr=0

wpqrijmvx+py+qt+riminus1m (2)

where m indexes different input feature maps wpqrijm is the valueat the position (p q) at time r of the kernel connected to the m-thfeature map and Pi Qi and Ri are the dimensions of the kernelalong width height and temporal axis respectively bij is the biasfrom layer i to layer jFrom C3D only the most general-purpose features are retainedby removing the last convolutional layer and the fully connectedlayers which are strongly linked to the original action recognitiontask The size of the last pooling layer is also modified in order tocover the remaining temporal dimension entirely This collapsesthe tensor from 4D to 3D making the output independent of timeEventually a bilinear upsampling brings the tensor back to theinput spatial resolution and a 2D convolution merges all featuresinto one channel See Fig 9 for additional details on the COARSEmodule

Training the two streams together The architecture of asingle FoA branch is depicted in Fig 10 During training the firststream feeds the COARSE network with random crops forcingthe model to learn the current focus of attention given visualcues rather than prior spatial location The C3D training processdescribed in [65] employs a 128 times 128 image resize and thena 112 times 112 random crop However the small difference in thetwo resolutions limits the variance of gaze position in ground

truth fixation maps and is not sufficient to avoid the attractiontowards the center of the image For this reason training imagesare resized to 256times 256 before being cropped to 112times 112 Thiscrop policy generates samples that cover less than a quarter ofthe original image thus ensuring a sufficient variety in predictiontargets This comes at the cost of a coarser prediction as cropsget smaller the ratio of pixels in the ground truth covered by gazeincreases leading the model to learn larger mapsIn contrast the second stream feeds the same COARSE modelwith the same images this time resized to 112 times 112 ndash and notcropped The coarse prediction obtained from the COARSE modelis then concatenated with the final frame of the input clip iethe frame corresponding to the final prediction Eventually theconcatenated tensor goes through the REFINE module to obtaina higher resolution prediction of the FoAThe overall two-stream training procedure for a single branch issummarized in Algorithm 1

Training objective Prediction cost can be minimized interms of Kullback-Leibler divergence

DKL(Y Y ) =sumi

Y (i) log

(ε+

Y (i)

ε+ Y (i)

)(3)

where Y is the ground truth distribution Y is the prediction thesummation index i spans across image pixels and ε is a smallconstant that ensures numerical stability2 Since each single FoAbranch computes an error on both the cropped image stream andthe resized image stream the branch loss can be defined as

Lb(XbY) =summ

(DKL(φ(Y m)C(φ(Xm

b ))) +

DKL(Y mR(C(ψ(Xmb )) Xm

b )))

) (4)

2 Please note that DKL inputs are always normalized to be a validprobability distribution despite this may be omitted in notation to improveequations readability

8

Algorithm 1 TRAINING The model is trained in two steps first each branch is trained separately through iterations detailed inprocedure SINGLE_BRANCH_TRAINING_ITERATION then the three branches are fine-tuned altogether as shown by procedureMULTI_BRANCH_FINE-TUNING_ITERATION For clarity we omit from notation i) the subscript b denoting the current domain inall X x and y variables in the single branch iteration and ii) the normalization of the sum of the outputs from each branch in line 13

1 procedure A SINGLE_BRANCH_TRAINING_ITERATION

input domain data X = x1 x2 x16 true attentional map y of last frame x16 of videoclip Xoutput branch loss Lb computed on input sample (X y)

2 Xres larr resize(X (112 112))3 Xcrop ycrop larr get_crop((X y) (112 112))4 ycrop larr COARSE(Xcrop) get coarse prediction on uncentered crop5 y larr REFINE(stack(x16 upsample(COARSE(Xres)))) get refined prediction over whole image6 Lb(XY )larr DKL(ycropycrop) +DKL(yy) compute branch loss as in Eq 4

7 procedure B MULTI_BRANCH_FINE-TUNING_ITERATION

input data X = x1 x2 x16 for all domains true attentional map y of last frame x16 of videoclip Xoutput overall loss L computed on input sample (X y)

8 Xres larr resize(X (112 112))9 Xcrop ycrop larr get_crop((X y) (112 112))

10 for branch b isin RGB flow seg do11 ybcrop larr COARSE(Xbcrop ) as in line 4 of the above procedure12 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres )))) as in line 5 of the above procedure13 L(XY )larr DKL(ycrop

sumb ybcrop ) +DKL(y

sumb yb) compute overall loss as in Eq 5

where C and R denote COARSE and REFINE modules(Xm

b Ym) isin Xb times Y is the m-th training example in the

b-th domain (namely RGB optical flow semantic segmentation)and φ and ψ indicate the crop and the resize functions respectively

Inference step While the presence of the C(φ(Xmb )) stream

is beneficial in training to reduce the spatial bias at test timeonly the R(C(ψ(Xm

b )) Xmb )) stream producing higher quality

prediction is used The outputs of such stream from each branch bare then summed together as explained in the following section

42 Multi-Branch modelAs described at the beginning of this section and depicted inFig 11 the multi-branch model is composed of three iden-tical branches The architecture of each branch has already beendescribed in Sec 41 above Each branch exploits complementaryinformation from a different domain and contributes to the finalprediction accordingly In detail the first branch works in the RGBdomain and processes raw visual data about the scene XRGBThe second branch focuses on motion through the optical flowrepresentation Xflow described in [23] Eventually the last branchtakes as input semantic segmentation probability maps Xseg Forthis last branch the number of input channels depends on thespecific algorithm used to extract the results 19 in our setup (Yu

Algorithm 2 INFERENCE At test time the data extracted fromthe resized videoclip is input to the three branches and their outputis summed and normalized to obtain the final FoA prediction

input data X = x1 x2 x16 for all domainsoutput predicted FoA map y1 Xres larr resize(X (112 112))2 for branch b isin RGB flow seg do3 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres ))))4 y larr

sumb yb

sumi

sumb yb(i)

and Koltun [76]) The three independent predicted FoA maps aresummed and normalized to result in a probability distributionTo allow for larger batch size we choose to bootstrap each branchindependently by training it according to Eq 4 Then the completemulti-branch model which merges the three branches is fine-tuned with the following loss

L(X Y) =summ

(DKL(φ(Y m)

sumb

C(φ(Xmb ))) +

DKL(Y msumb

R(C(ψ(Xmb )) Xm

b )))

)

(5)The algorithm describing the complete inference over themulti-branch model in detailed in Alg 2

5 EXPERIMENTS

In this section we evaluate the performance of the proposedmulti-branch model First we start by comparing our modelagainst some baselines and other methods in literature Followingthe guidelines in [12] for the evaluation phase we rely on Pear-sonrsquos Correlation Coefficient (CC) and KullbackndashLeibler Diver-gence (DKL) measures Moreover we evaluate the InformationGain (IG) [35] measure to assess the quality of a predicted mapP with respect to a ground truth map Y in presence of a strongbias as

IG(P YB) =1

N

sumi

Yi[(log2(ε+ Pi)minus log2(ε+Bi)] (6)

where i is an index spanning all the N pixels in the image B thebias computed as the average training fixation map and ε ensuresnumerical stabilityFurthermore we conduct an ablation study to investigate howdifferent branches affect the final prediction and how their mutualinfluence changes in different scenarios We then study whetherour model captures the attention dynamics observed in Sec 31

9

nalprediction

+

FoAfrom motionow

clip

FoAfrom semanticsseg

clip

FoAfrom colorRGB

clip

Fig 11 The multi-branch model is composed of three different branches each of which has its own set of parameters and their predictions aresummed to obtain the final map Note that in this figure cropped streams are dropped to ease representation but are employed during training (asdiscussed in Sec 42 and depicted in Fig 10

Eventually we assess our model from a human perceptionperspective

Implementation details The three different pathways ofthe multi-branch model (namely FoA from color frommotion and from semantics) have been pre-trained independentlyusing the same cropping policy of Sec 42 and minimizing theobjective function in Eq 4 Each branch has been respectively fedwithbull 16 frames clips in raw RGB color spacebull 16 frames clips with optical flow maps encoded as color

images through the flow field encoding [23]bull 16 frames clips holding semantic segmentation from [76]

encoded as 19 scalar activation maps one per segmentationclass

During individual branch pre-training clips were randomly mir-rored for data augmentation We employ Adam optimizer with pa-rameters as suggested in the original paper [32] with the exceptionof the learning rate that we set to 10minus4 Eventually batch size wasfixed to 32 and each branch was trained until convergence TheDR(eye)VE dataset is split into train validation and test set asfollows sequences 1-38 are used for training sequences 39-74 fortesting The 500 frames in the middle of each training sequenceconstitute the validation setMoreover the complete multi-branch architecture was fine-tuned using the same cropping and data augmentation strategiesminimizing cost function in Eq 5 In this phase batch size was setto 4 due to GPU memory constraints and learning rate value waslowered to 10minus5 Inference time of each branch of our architectureis asymp 30 milliseconds per videoclip on an NVIDIA Titan X

51 Model evaluation

In Tab 3 we report results of our proposal against other state-of-the-art models [4] [15] [46] [59] [72] [73] evaluated both onthe complete test set and on acting subsequences only All thecompetitors with the exception of [46] are bottom-up approachesand mainly rely on appearance and motion discontinuities To testthe effectiveness of deep architectures for saliency prediction wecompare against the Multi-Level Network (MLNet) [15] which

scored favourably in the MIT300 saliency benchmark [11] andthe Recurrent Mixture Density Network (RMDN) [4] whichrepresents the only deep model addressing video saliency WhileMLNet works on images discarding the temporal informationRMDN encodes short sequences in a similar way to our COARSEmodule and then relies on a LSTM architecture to model longterm dependencies and estimates the fixation map in terms of aGMM To favor the comparison both models were re-trained onthe DR(eye)VE datasetResults highlight the superiority of our multi-branch archi-tecture on all test sequences The gap in performance with respectto bottom-up unsupervised approaches [72] [73] is higher and ismotivated by the peculiarity of the attention behavior within thedriving context which calls for a task-oriented training procedureMoreover MLNetrsquos low performance testifies for the need of ac-counting for the temporal correlation between consecutive framesthat distinguishes the tasks of attention prediction in images andvideos Indeed RMDN processes video inputs and outperformsMLNet on both DKL and IG metrics performing comparablyon CC Nonetheless its performance is still limited indeedqualitative results reported in Fig 12 suggest that long termdependencies captured by its recurrent module lead the networktowards the regression of the mean discarding contextual and

TABLE 3Experiments illustrating the superior performance of the

multi-branch model over several baselines and competitors Wereport both the average across the complete test sequences and only

the acting frames

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

Baseline Gaussian 040 216 -049 026 241 003Baseline Mean 051 160 000 022 235 000

Mathe et al [59] 004 330 -208 - - -Wang et al [72] 004 340 -221 - - -Wang et al [73] 011 306 -172 - - -

MLNet [15] 044 200 -088 032 235 -036RMDN [4] 041 177 -006 031 213 031

Palazzi et al [46] 055 148 -021 037 200 020multi-branch 056 140 004 041 180 051

10

Input frame GT multi-branch [46] [4] [15]

Fig 12 Qualitative assessment of the predicted fixation maps From left to right input clip ground truth map our prediction prediction of theprevious version of the model [46] prediction of RMDN [4] and prediction of MLNet [15]

frame-specific variations that would be preferrable to keep Tosupport this intuition we measure the average DKL betweenRMDN predictions and the mean training fixation map (Base-line Mean) resulting in a value of 011 Being lower than thedivergence measured with respect to groundtruth maps this valuehighlights the closer correlation to a central baseline rather thanto groundtruth Eventually we also observe improvements withrespect to our previous proposal [46] that relies on a more com-plex backbone model (also including a deconvolutional module)and processes RGB clips only The gap in performance residesin the greater awareness of our multi-branch architecture ofthe aspects that characterize the driving task as emerged fromthe analysis in Sec 31 The positive performances of our modelare also confirmed when evaluated on the acting partition of thedataset We recall that acting indicates sub-sequences exhibiting asignificant task-driven shift of attention from the center of theimage (Fig 5) Being able to predict the FoA also on actingsub-sequences means that the model captures the strong centeredattention bias but is capable of generalizing when required by thecontextThis is further shown by the comparison against a centeredGaussian baseline (BG) and against the average of all trainingset fixation maps (BM) The former baseline has proven effectiveon many image saliency detection tasks [11] while the latterrepresents a more task-driven version The superior performanceof the multi-branch model wrt baselines highlights thatdespite the attention is often strongly biased towards the vanishingpoint of the road the network is able to deal with sudden task-driven changes in gaze direction

52 Model analysis

In this section we investigate the behavior of our proposed modelunder different landscapes time of day and weather (Sec 521)we study the contribution of each branch to the FoA predictiontask (Sec 522) and we compare the learnt attention dynamicsagainst the one observed in the human data (Sec 523)

12

13

14

15

16

17

18

19

Downt

own

Coun

trysid

e

High

way

Mornin

g

Even

ingNi

ght

Sunn

y

Cloud

yRa

iny

DKL

Multi-branchImage

FlowSegm

________________________Landscape ________________________Time of day ________________________Weather

Fig 13 DKL of the different branches in several conditions (from left toright downtown countryside highway morning evening night sunnycloudy rainy) Underlining highlights difference of aggregation in termsof landscape time of day and weather Please note that lower DKL

indicates better predictions

521 Dependency on driving environment

The DR(eye)VE data has been recorded under varying land-scapes time of day and weather conditions We tested our modelin all such different driving conditions As would be expectedFig 13 shows that the human attention is easier to predict inhighways rather than downtown where the focus can shift towardsmore distractors The model seems more reliable in eveningscenarios rather than morning or night where we observed betterlightning conditions and lack of shadows over-exposure and soon Lastly in rainy conditions we notice that human gaze iseasier to model possibly due to the higher level of awarenessdemanded to the driver and his consequent inability to focus awayfrom vanishing point To support the latter intuition we measuredthe performance of BM baseline (ie the average training fixationmap) grouped for weather condition As expected theDKL valuein rainy weather (153) is significantly lower than the ones forcloudy (161) and sunny weather (175) highlighting that whenrainy the driver is more focused on the road

11

|Σ| = 250times 109 |Σ| = 157times 109 |Σ| = 349times 108 |Σ| = 120times 108 |Σ| = 538times 107

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 14 Model prediction averaged across all test sequences and grouped by driving speed As the speed increases the area of the predicted mapshrinks recalling the trend observed in ground truth maps As in Fig 7 for each map a two dimensional Gaussian is fitted and the determinant ofits covariance matrix Σ is reported as a measure of the spread

road sidewalk buildings traffic signs trees flowerbed sky vehicles people cycles10

-4

10-3

10-2

10-1

100

prob

abilit

y of

fixa

tions

(log

sca

le)

Fig 15 Comparison between ground truth (gray bars) and predictedfixation maps (colored bars) when used to mask semantic segmentationof the scene The probability of fixation (in log-scale) for both groundtruth and model prediction is reported for each semantic class Despiteabsolute errors exist the two bar series agree on the relative importanceof different categories

522 Ablation study

In order to validate the design of the multi-branch model(see Sec 42) here we study the individual contributions of thedifferent branches by disabling one or more of themResults in Tab 4 show that the RGB branch plays a majorrole in FoA prediction The motion stream is also beneficialand provides a slight improvement that becomes clearer in theacting subsequences Indeed optical flow intrinsically captures avariety of peculiar scenarios that are non-trivial to classify whenonly color information is provided eg when the car is still at atraffic light or is turning The semantic stream on the other handprovides very little improvement In particular from Tab 4 and by

TABLE 4The ablation study performed on our multi-branch model I F and S

represent image optical flow and semantic segmentation branchesrespectively

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

I 0554 1415 -0008 0403 1826 0458F 0516 1616 -0137 0368 2010 0349S 0479 1699 -0119 0344 2082 0288

I+F 0558 1399 0033 0410 1799 0510I+S 0554 1413 -0001 0404 1823 0466F+S 0528 1571 -0055 0380 1956 0427

I+F+S 0559 1398 0038 0410 1797 0515

specifically comparing I+F and I+F+S a slight increase in the IGmeasure can be appreciated Nevertheless such improvement hasto be considered negligible when compared to color and motionsuggesting that in presence of efficiency concerns or real-timeconstraints the semantic stream can be discarded with little lossesin performance However we expect the benefit from this branchto increase as more accurate segmentation models will be released

523 Do we capture the attention dynamics

The previous sections validate quantitatively the proposed modelNow we assess its capability to attend like a human driverby comparing its predictions against the analysis performed inSec 31First we report the average predicted fixation map in severalspeed ranges in Fig 14 The conclusions we draw are twofoldi) generally the model succeeds in modeling the behavior ofthe driver at different speeds and ii) as the speed increasesfixation maps exhibit lower variance easing the modeling taskand prediction errors decreaseWe also study how often our model focuses on different semanticcategories in a fashion that recalls the analysis of Sec 31 butemploying our predictions rather than ground truth maps as focusof attention More precisely we normalize each map so thatthe maximum value equals 1 and apply the same thresholdingstrategy described in Sec 31 Likewise for each threshold valuea histogram over class labels is built by accounting all pixelsfalling within the binary map for all test frames This results innine histograms over semantic labels that we merge together byaveraging probabilities belonging to different threshold Fig 15shows the comparison Color bars represent how often the pre-dicted map focuses on a certain category while gray bars depictground truth behavior and are obtained by averaging histogramsin Fig 8 across different thresholds Please note that to highlightdifferences for low populated categories values are reported ona logarithmic scale The plot shows a certain degree of absoluteerror is present for all categories However in a broader sense ourmodel replicates the relative weight of different semantic classeswhile driving as testified by the importance of roads and vehiclesthat still dominate against other categories such as people andcycles that are mostly neglected This correlation is confirmed byKendall rank coefficient which scored 051 when computed onthe two bar series

53 Visual assessment of predicted fixation maps

To further validate the predictions of our model from the humanperception perspective 50 people with at least 3 years of driving

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

3

Fig 2 Examples taken from a random sequence of DR(eye)VE From left to right frames from the eye tracking glasses with gaze data from theroof-mounted camera temporal aggregated fixation maps (as defined in Sec 3) and overlays between frames and fixation maps

as looking at people or traffic signs Coarse gaze informationis also available in [19] while the external road scene imagesare not acquired We believe that the dataset presented in [52]is among the others the closer to our proposal Yet videosequences are collected from one driver only it is not publiclyavailable Conversely our DR(eye)VE dataset is the first datasetaddressing driverrsquos focus of attention prediction that is madepublicly available Furthermore it includes sequences fromseveral different drivers and presents a high variety of landscapes(ie highway downtown and countryside) lighting and weatherconditions

3 THE DR(EYE)VE PROJECT

In this section we present the DR(eye)VE dataset (Fig 2)the protocol adopted for video registration and annotation theautomatic processing of eye-tracker data and the analysis of thedriverrsquos behavior in different conditions

The dataset The DR(eye)VE dataset consists of 555000frames divided in 74 sequences each of which is 5 minutes longEight different drivers of varying age from 20 to 40 including7 men and a woman took part to the driving experiment thatlasted more than two months Videos were recorded in differentcontexts both in terms of landscape (downtown countrysidehighway) and traffic condition ranging from traffic-free to highlycluttered scenarios They were recorded in diverse weatherconditions (sunny rainy cloudy) and at different hours of the day(both daytime and night) Tab 1 recaps the dataset features andTab 2 compares it with other related proposals DR(eye)VE iscurrently the largest publicly available dataset including gaze anddriving behavior in automotive settings

The Acquisition System The driverrsquos gaze informationwas captured using the commercial SMI ETG 2w Eye TrackingGlasses (ETG) ETG capture attention dynamics also in presenceof head pose changes which occur very often during the task ofdriving While a frontal camera acquires the scene at 720p30fpsusers pupils are tracked at 60Hz Gaze information are providedin terms of eye fixations and saccade movements ETG was

manually calibrated before each sequence for every driverSimultaneously videos from the car perspective were acquiredusing the GARMIN VirbX camera mounted on the car roof(RMC Roof-Mounted Camera) Such sensor captures frames at1080p25fps and includes further information such as GPS dataaccelerometer and gyroscope measurements

Video-gaze registration The dataset has been processed tomove the acquired gaze from the egocentric (ETG) view to thecar (RMC) view The latter features a much wider field of view(FoV) and can contain fixations that are out of the egocentricview For instance this can occur whenever the driver takes apeek at something at the border of this FoV but doesnrsquot move hishead For every sequence the two videos were manually alignedto cope with the difference in sensors framerate Videos were thenregistered frame-by-frame through a homographic transformationthat projects fixation points across views More formally at eachtimestep t the RMC frame ItRMC and the ETG frame ItETGare registered by means of a homography matrix HETGrarrRMC computed by matching SIFT descriptors [38] from one viewto the other (see Fig 3) A further RANSAC [18] procedureensures robustness to outliers While homographic mappingis theoretically sound only across planar views - which is notthe case of outdoor environments - we empirically found thatprojecting an object from one image to another always recoveredthe correct position This makes sense if the distance between theprojected object and the camera is far greater than the distancebetween the object and the projective plane In Sec 13 of thesupplementary material we derive formal bounds to explain thisphenomena

Fixation map computation The pipeline discussed aboveprovides a frame-level annotation of the driverrsquos fixations Incontrast to image saliency experiments [11] there is no clear andindisputable protocol for obtaining continuous maps from rawfixations when acquired in task-driven real-life scenarios Thisis even more evident when fixations are collected in task-drivenreal-life scenarios The main motivation resides in the fact thatobserverrsquos subjectivity cannot be removed by averaging different

4

TABLE 1Summary of the DR(eye)VE dataset characteristics The dataset was designed to embody the most possible diversity in the combination of

different features The reader is referred to either the additional material or to the dataset presentation [2] for details on each sequence

Videos Frames Drivers Weather conditions Lighting Gaze Info Metadata Camera Viewpoint

74 555000 8sunny day raw fixations GPS driver (720p)cloudy evening gaze map car speed car (1080p)rainy night pupil dilation car course

TABLE 2A comparison between DR(eye)VE and other datasets

Dataset Frames Drivers Scenarios Annotations Real-world Public

Pugeault et al [52] 158668 ndash Countryside HighwayDowntown

Gaze MapsDriverrsquos Actions Yes No

Simon et al [58] 40 30 Downtown Gaze Maps No NoUnderwood et al [68] 120 77 Urban Motorway ndash No NoFridman et al [19] 1860761 50 Highway 6 Gaze Location Classes Yes No

DR(eye)VE [2] 555000 8 Countryside HighwayDowntown

Gaze MapsGPS Speed Course Yes Yes

observersrsquo fixations Indeed two different observers cannotexperience the same scene at the same time (eg two driverscannot be at the same time in the same point of the street) Theonly chance to average among different observers would be theadoption of a simulation environment but it has been provedthat the cognitive load in controlled experiments is lower thanin real test scenarios and it effects the true attention mechanismof the observer [55] In our preliminary DR(eye)VE release[2] fixation points were aggregated and smoothed by means ofa temporal sliding window In such a way temporal filteringdiscarded momentary glimpses that contain precious informationabout the driverrsquos attention Following the psychological protocolin [40] and [25] this limitation was overcome in the currentrelease where the new fixation maps were computed withouttemporal smoothing Both [40] and [25] highlight the high degreeof subjectivity of scene scanpaths in short temporal windows (lt 1sec) and suggest to neglect the fixations pop-out order withinsuch windows This mechanism also ameliorates the inhibitionof return phenomenon that may prevent interesting objects to beobserved twice in short temporal intervals [27] [51] leading tothe underestimation of their importanceMore formally the fixation map Ft for a frame at time t is builtby accumulating projected gaze points in a temporal slidingwindow of k = 25 frames centered in t For each time step t+ iin the window where i isin minusk2 minus

k2 + 1 k2 minus 1 k2 gaze

points projections on Ft are estimated through the homographytransformation Ht

t+i that projects points from the image plane atframe t+ i namely pt+i to the image plane in Ft A continuousfixation map is obtained from the projected fixations by centeringon each of them a multivariate Gaussian having a diagonalcovariance matrix Σ (the spatial variance of each variable is set to

Fig 3 Registration between the egocentric and roof-mounted cameraviews by means of SIFT descriptor matching

Fig 4 Resulting fixation map from a 1 second integration (25 frames)The adoption of the max aggregation of equation 1 allows to account inthe final map two brief glances towards traffic lights

σ2s = 200 pixels) and taking the max value along the time axis

Ft(x y) = maxiisin(minus k

2 k2 )N ((x y) |Ht

t+i middot pt+iΣ) (1)

The Gaussian variance has been computed by averaging the ETGspatial acquisition errors on 20 observers looking at calibrationpatterns at different distances from 5 to 15 meters The describedprocess can be appreciated in Fig 4 Eventually each map Ft isnormalized to sum to 1 so that it can be considered a probabilitydistribution of fixation points

Labeling attention drifts Fixation maps exhibit a verystrong central bias This is common in saliency annotations [60]and even more in the context of driving For these reasons thereis a strong unbalance between lots of easy-to-predict scenariosand unfrequent but interesting hard-to-predict events

To enable the evaluation of computational models under suchcircumstances the DR(eye)VE dataset has been extended witha set of further annotations For each video subsequences whoseground truth poorly correlates with the average ground truth ofthat sequence are selected We employ Pearsonrsquos CorrelationCoefficient (CC) and select subsequences with CC lt 03 Thishappens when the attention of the driver focuses far from the

5

(a) Acting - 69 719 (b) Inattentive - 12 282

(c) Error - 22 893 (d) Subjective - 3 166

Fig 5 Examples of the categorization of frames where gaze is farfrom the mean Overall 108 060 frames (sim20 of DR(eye)VE) wereextended with this type of information

vanishing point of the road Examples of such subsequencesare depicted in Fig 5 Several human annotators inspected theselected frames and manually split them into (a) acting (b)inattentive (c) errors and (d) subjective eventsbull errors can happen either due to failures in the measuring tool

(eg in extreme lighting conditions) or in the successive dataprocessing phase (eg SIFT matching)

bull inattentive subsequences occur when the driver focuses hisgaze on objects unrelated to the driving task (eg looking atan advertisement)

bull subjective subsequences describe situations in which theattention is closely related to the individual experience ofthe driver eg a road sign on the side might be an interestingelement to focus for someone that has never been on that roadbefore but might be safely ignored by someone who drivesthat road every day

bull acting subsequences include all the remaining onesActing subsequences are particularly interesting as the deviationof driverrsquos attention from the common central pattern denotes anintention linked to task-specific actions (eg turning changinglanes overtaking ) For these reasons subsequences of thiskind will have a central role in the evaluation of predictive modelsin Sec 5

31 Dataset analysisBy analyzing the dataset frames the very first insight is thepresence of a strong attraction of driverrsquos focus towards thevanishing point of the road that can be appreciated in Fig 6 Thesame phenomenon was observed in previous studies [6] [67] inthe context of visual search tasks We observed indeed that driversoften tend to disregard road signals cars coming from the opposite

(a) (b)

Fig 6 Mean frame (a) and fixation map (b) averaged across the wholesequence 02 highlighting the link between driverrsquos focus and the van-ishing point of the road

direction and pedestrians on sidewalks This is an effect of humanperipheral vision [56] that allows observers to still perceive andinterpret stimuli out of - but sufficiently close to - their focus ofattention (FoA) A driver can therefore achieve a larger area ofattention by focusing on the roadrsquos vanishing point due to thegeometry of the road environment many of the objects worth ofattention are coming from there and have already been perceivedwhen distantMoreover the gaze location tends to drift from this central attrac-

tor when the context changes in terms of car speed and landscapeIndeed [53] suggests that our brain is able to compensate spatiallyor temporally dense information by reducing the visual field sizeIn particular as the car travels at higher speed the temporal densityof information (ie the amount of information that the driver needsto elaborate per unit of time) increases this causes the usefulvisual field of the driver to shrink [53] We also observe thisphenomenon in our experiments as shown in Fig 7DR(eye)VE data also highlight that the driverrsquos gaze is attractedtowards specific semantic categories To reach the above conclu-sion the dataset is analysed by means of the semantic segmenta-tion model in [76] and the distribution of semantic classes withinthe fixation map evaluated More precisely given a segmentedframe and the corresponding fixation map the probability for eachsemantic class to fall within the area of attention is computed asfollows First the fixation map (which is continuous in [0 1]) isnormalized such that the maximum value equals 1 Then nine bi-nary maps are constructed by thresholding such continuous valueslinearly in the interval [0 1] As the threshold moves towards 1(the maximum value) the area of interest shrinks around the realfixation points (since the continuous map is modeled by meansof several Gaussians centered in fixation points see previoussection) For every threshold a histogram over semantic labelswithin the area of interest is built by summing up occurrencescollected from all DR(eye)VE frames Fig 8 displays the resultfor each class the probability of a pixel to fall within the region ofinterest is reported for each threshold value The figure providesinsight about which categories represent the real focus of attentionand which ones tend to fall inside the attention region just byproximity with the formers Object classes that exhibit a positivetrend such as road vehicles and people are the real focus ofthe gaze since the ratio of pixels classified accordingly increaseswhen the observed area shrinks around the fixation point In abroader sense the figure suggests that despite while driving ourfocus is dominated by road and vehicles we often observe specificobjects categories even if they contain little information useful todrive

4 MULTI-BRANCH DEEP ARCHITECTURE FORFOCUS OF ATTENTION PREDICTION

The DR(eye)VE dataset is sufficiently large to allow theconstruction of a deep architecture to model common attentionalpatterns Here we describe our neural network model to predicthuman FoA while driving

Architecture design In the context of high level videoanalysis (eg action recognition and video classification) ithas been shown that a method leveraging single frames canbe outperformed if a sequence of frames is used as inputinstead [31] [65] Temporal dependencies are usually modeledeither by 3D convolutional layers [65] tailored to capture short

6

|Σ| = 249times 109 |Σ| = 203times 109 |Σ| = 861times 108 |Σ| = 102times 109 |Σ| = 928times 108

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 7 As speed gradually increases driverrsquos attention converges towards the vanishing point of the road (a) When the car is approximatelystationary the driver is distracted by many objects in the scene (b-e) As the speed increases the driverrsquos gaze deviates less and less from thevanishing point of the road To measure this effect quantitatively a two-dimensional Gaussian is fitted to approximate the mean map for each speedrange and the determinant of the covariance matrix Σ is reported as an indication of its spread (the determinant equals the product of eigenvalueseach of which measures the spread along a different data dimension) The bar plots illustrate the amount of downtown (red) countryside (green)and highway (blue) frames that concurred to generate the average gaze position for a specific speed range Best viewed on screen

Pro

babi

lity

offix

atio

n

road0

01

02

03

04

sidewalk buildings traffic signs trees flowerbed sky vehicles people0

0005

001

cycles

Fig 8 Proportion of semantic categories that fall within the driverrsquos fixation map when thresholded at increasing values (from left to right) Categoriesexhibiting a positive trend (eg road and vehicles) suggest a real attention focus while a negative trend advocates for an awareness of the objectwhich is only circumstantial See Sec 31 for details

conv3D64

conv3D128

conv3D256

conv3D256

conv2D1

pool3D

pool3D

pool3D

up2D

conv3D

64

conv3D conv3D conv3Dpool3D pool3D pool3D

up2D

128 256 256 1

conv2D

Fig 9 The COARSE module is made of an encoder based on C3D network [65] followed by a bilinear upsampling (bringing representations back tothe resolution of the input image) and a final 2D convolution During feature extraction the temporal axis is lost due to 3D pooling All convolutionallayers are preceded by zero paddings in order keep borders and all kernels have size 3 along all dimensions Pooling layers have size and strideof (1 2 2 4) and (2 2 2 1) along temporal and spatial dimensions respectively All activations are ReLUs

range correlations or by recurrent architectures (eg LSTMGRU) that can model longer term dependencies [3] [47] Ourmodel follows the former approach relying on the assumptionthat a small time window (eg half a second) holds sufficientcontextual information for predicting where the driver wouldfocus in that moment Indeed human drivers can take even lesstime to react to an unexpected stimulus Our architecture takes asequence of 16 consecutive frames (asymp 065s) as input (called clipsfrom now on) and predicts the fixation map for the last frame ofsuch clipMany of the architectural choices made to design the networkcome from insights from the dataset analysis presented in Sec31In particular we rely on the following resultsbull the driversrsquo FoA exhibits consistent patterns suggesting that

it can be reproduced by a computational modelbull the driversrsquo gaze is affected by a strong prior on objects

semantics eg drivers tend to focus on items lying on theroad

bull motion cues like vehicle speed are also key factors thatinfluence gaze

Accordingly the model output merges three branches with identi-cal architecture unshared parameters and different input domains

the RGB image the semantic segmentation and the optical flowfield We call this architecture multi-branch model Followinga bottom-up approach in Sec 41 the building blocks of eachbranch are motivated and described Later in Sec 42 it will beshown how the branches merge into the final model

41 Single FoA branch

Each branch of the multi-branch model is a two-input two-output architecture composed of two intertwined streams The aimof this peculiar setup is to prevent the network from learninga central bias that would otherwise stall the learning in earlytraining stages 1 To this end one of the streams is given as input(output) a severely cropped portion of the original image (groundtruth) ensuring a more uniform distribution of the true gaze andruns through the COARSE module described below Similarly theother stream uses the COARSE module to obtain a rough predictionover the full resized image and then refines it through a stack ofadditional convolutions called REFINE model At test time onlythe output of the REFINE stream is considered Both streams rely

1 For further details the reader can refer to Sec 14 and Sec 15 of thesupplementary material

7

last frame

shared weights

Videoclip

CROP

RESIZE

COARSE REFINE outputinput

448

448

cropped videoclip

112

112

112

112

resized videoclip

Fig 10 A single FoA branch of our prediction architecture The COARSE module (see Fig 9) is applied to both a cropped and a resized version ofthe input tensor which is a videoclip of 16 consecutive frames The cropped input is used during training to augment the data and the variety ofground truth fixation maps The prediction of the resized input is stacked with the last frame of the videoclip and fed to a stack of convolutional layers(refinement module) with the aim of refining the prediction Training is performed end-to-end and weights between COARSE modules are shared Attest time only the refined predictions are used Note that the complete model is composed of three of these branches (see Fig 11) each of whichpredicting visual attention for different inputs (namely image optical flow and semantic segmentation) All activations in the refinement module areLeakyReLU with α = 10minus3 except for the last single channel convolution that features ReLUs Crop and resize streams are highlighted by lightblue and orange arrows respectively

on the COARSE module the convolutional backbone (with sharedweights) which provides the rough estimate of the attentional mapcorresponding to a given clip This component is detailed in Fig 9The COARSE module is based on the C3D architecture [65] thatencodes video dynamics by applying a 3D convolutional kernelon the 4D input tensor As opposed to 2D convolutions that stridealong the width and height dimension of the input tensor a 3Dconvolution also strides along time Formally the j-th feature mapin the i-th layer at position (x y) at time t is computed as

vxytij = bij +summ

Piminus1sump=0

Qiminus1sumq=0

Riminus1sumr=0

wpqrijmvx+py+qt+riminus1m (2)

where m indexes different input feature maps wpqrijm is the valueat the position (p q) at time r of the kernel connected to the m-thfeature map and Pi Qi and Ri are the dimensions of the kernelalong width height and temporal axis respectively bij is the biasfrom layer i to layer jFrom C3D only the most general-purpose features are retainedby removing the last convolutional layer and the fully connectedlayers which are strongly linked to the original action recognitiontask The size of the last pooling layer is also modified in order tocover the remaining temporal dimension entirely This collapsesthe tensor from 4D to 3D making the output independent of timeEventually a bilinear upsampling brings the tensor back to theinput spatial resolution and a 2D convolution merges all featuresinto one channel See Fig 9 for additional details on the COARSEmodule

Training the two streams together The architecture of asingle FoA branch is depicted in Fig 10 During training the firststream feeds the COARSE network with random crops forcingthe model to learn the current focus of attention given visualcues rather than prior spatial location The C3D training processdescribed in [65] employs a 128 times 128 image resize and thena 112 times 112 random crop However the small difference in thetwo resolutions limits the variance of gaze position in ground

truth fixation maps and is not sufficient to avoid the attractiontowards the center of the image For this reason training imagesare resized to 256times 256 before being cropped to 112times 112 Thiscrop policy generates samples that cover less than a quarter ofthe original image thus ensuring a sufficient variety in predictiontargets This comes at the cost of a coarser prediction as cropsget smaller the ratio of pixels in the ground truth covered by gazeincreases leading the model to learn larger mapsIn contrast the second stream feeds the same COARSE modelwith the same images this time resized to 112 times 112 ndash and notcropped The coarse prediction obtained from the COARSE modelis then concatenated with the final frame of the input clip iethe frame corresponding to the final prediction Eventually theconcatenated tensor goes through the REFINE module to obtaina higher resolution prediction of the FoAThe overall two-stream training procedure for a single branch issummarized in Algorithm 1

Training objective Prediction cost can be minimized interms of Kullback-Leibler divergence

DKL(Y Y ) =sumi

Y (i) log

(ε+

Y (i)

ε+ Y (i)

)(3)

where Y is the ground truth distribution Y is the prediction thesummation index i spans across image pixels and ε is a smallconstant that ensures numerical stability2 Since each single FoAbranch computes an error on both the cropped image stream andthe resized image stream the branch loss can be defined as

Lb(XbY) =summ

(DKL(φ(Y m)C(φ(Xm

b ))) +

DKL(Y mR(C(ψ(Xmb )) Xm

b )))

) (4)

2 Please note that DKL inputs are always normalized to be a validprobability distribution despite this may be omitted in notation to improveequations readability

8

Algorithm 1 TRAINING The model is trained in two steps first each branch is trained separately through iterations detailed inprocedure SINGLE_BRANCH_TRAINING_ITERATION then the three branches are fine-tuned altogether as shown by procedureMULTI_BRANCH_FINE-TUNING_ITERATION For clarity we omit from notation i) the subscript b denoting the current domain inall X x and y variables in the single branch iteration and ii) the normalization of the sum of the outputs from each branch in line 13

1 procedure A SINGLE_BRANCH_TRAINING_ITERATION

input domain data X = x1 x2 x16 true attentional map y of last frame x16 of videoclip Xoutput branch loss Lb computed on input sample (X y)

2 Xres larr resize(X (112 112))3 Xcrop ycrop larr get_crop((X y) (112 112))4 ycrop larr COARSE(Xcrop) get coarse prediction on uncentered crop5 y larr REFINE(stack(x16 upsample(COARSE(Xres)))) get refined prediction over whole image6 Lb(XY )larr DKL(ycropycrop) +DKL(yy) compute branch loss as in Eq 4

7 procedure B MULTI_BRANCH_FINE-TUNING_ITERATION

input data X = x1 x2 x16 for all domains true attentional map y of last frame x16 of videoclip Xoutput overall loss L computed on input sample (X y)

8 Xres larr resize(X (112 112))9 Xcrop ycrop larr get_crop((X y) (112 112))

10 for branch b isin RGB flow seg do11 ybcrop larr COARSE(Xbcrop ) as in line 4 of the above procedure12 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres )))) as in line 5 of the above procedure13 L(XY )larr DKL(ycrop

sumb ybcrop ) +DKL(y

sumb yb) compute overall loss as in Eq 5

where C and R denote COARSE and REFINE modules(Xm

b Ym) isin Xb times Y is the m-th training example in the

b-th domain (namely RGB optical flow semantic segmentation)and φ and ψ indicate the crop and the resize functions respectively

Inference step While the presence of the C(φ(Xmb )) stream

is beneficial in training to reduce the spatial bias at test timeonly the R(C(ψ(Xm

b )) Xmb )) stream producing higher quality

prediction is used The outputs of such stream from each branch bare then summed together as explained in the following section

42 Multi-Branch modelAs described at the beginning of this section and depicted inFig 11 the multi-branch model is composed of three iden-tical branches The architecture of each branch has already beendescribed in Sec 41 above Each branch exploits complementaryinformation from a different domain and contributes to the finalprediction accordingly In detail the first branch works in the RGBdomain and processes raw visual data about the scene XRGBThe second branch focuses on motion through the optical flowrepresentation Xflow described in [23] Eventually the last branchtakes as input semantic segmentation probability maps Xseg Forthis last branch the number of input channels depends on thespecific algorithm used to extract the results 19 in our setup (Yu

Algorithm 2 INFERENCE At test time the data extracted fromthe resized videoclip is input to the three branches and their outputis summed and normalized to obtain the final FoA prediction

input data X = x1 x2 x16 for all domainsoutput predicted FoA map y1 Xres larr resize(X (112 112))2 for branch b isin RGB flow seg do3 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres ))))4 y larr

sumb yb

sumi

sumb yb(i)

and Koltun [76]) The three independent predicted FoA maps aresummed and normalized to result in a probability distributionTo allow for larger batch size we choose to bootstrap each branchindependently by training it according to Eq 4 Then the completemulti-branch model which merges the three branches is fine-tuned with the following loss

L(X Y) =summ

(DKL(φ(Y m)

sumb

C(φ(Xmb ))) +

DKL(Y msumb

R(C(ψ(Xmb )) Xm

b )))

)

(5)The algorithm describing the complete inference over themulti-branch model in detailed in Alg 2

5 EXPERIMENTS

In this section we evaluate the performance of the proposedmulti-branch model First we start by comparing our modelagainst some baselines and other methods in literature Followingthe guidelines in [12] for the evaluation phase we rely on Pear-sonrsquos Correlation Coefficient (CC) and KullbackndashLeibler Diver-gence (DKL) measures Moreover we evaluate the InformationGain (IG) [35] measure to assess the quality of a predicted mapP with respect to a ground truth map Y in presence of a strongbias as

IG(P YB) =1

N

sumi

Yi[(log2(ε+ Pi)minus log2(ε+Bi)] (6)

where i is an index spanning all the N pixels in the image B thebias computed as the average training fixation map and ε ensuresnumerical stabilityFurthermore we conduct an ablation study to investigate howdifferent branches affect the final prediction and how their mutualinfluence changes in different scenarios We then study whetherour model captures the attention dynamics observed in Sec 31

9

nalprediction

+

FoAfrom motionow

clip

FoAfrom semanticsseg

clip

FoAfrom colorRGB

clip

Fig 11 The multi-branch model is composed of three different branches each of which has its own set of parameters and their predictions aresummed to obtain the final map Note that in this figure cropped streams are dropped to ease representation but are employed during training (asdiscussed in Sec 42 and depicted in Fig 10

Eventually we assess our model from a human perceptionperspective

Implementation details The three different pathways ofthe multi-branch model (namely FoA from color frommotion and from semantics) have been pre-trained independentlyusing the same cropping policy of Sec 42 and minimizing theobjective function in Eq 4 Each branch has been respectively fedwithbull 16 frames clips in raw RGB color spacebull 16 frames clips with optical flow maps encoded as color

images through the flow field encoding [23]bull 16 frames clips holding semantic segmentation from [76]

encoded as 19 scalar activation maps one per segmentationclass

During individual branch pre-training clips were randomly mir-rored for data augmentation We employ Adam optimizer with pa-rameters as suggested in the original paper [32] with the exceptionof the learning rate that we set to 10minus4 Eventually batch size wasfixed to 32 and each branch was trained until convergence TheDR(eye)VE dataset is split into train validation and test set asfollows sequences 1-38 are used for training sequences 39-74 fortesting The 500 frames in the middle of each training sequenceconstitute the validation setMoreover the complete multi-branch architecture was fine-tuned using the same cropping and data augmentation strategiesminimizing cost function in Eq 5 In this phase batch size was setto 4 due to GPU memory constraints and learning rate value waslowered to 10minus5 Inference time of each branch of our architectureis asymp 30 milliseconds per videoclip on an NVIDIA Titan X

51 Model evaluation

In Tab 3 we report results of our proposal against other state-of-the-art models [4] [15] [46] [59] [72] [73] evaluated both onthe complete test set and on acting subsequences only All thecompetitors with the exception of [46] are bottom-up approachesand mainly rely on appearance and motion discontinuities To testthe effectiveness of deep architectures for saliency prediction wecompare against the Multi-Level Network (MLNet) [15] which

scored favourably in the MIT300 saliency benchmark [11] andthe Recurrent Mixture Density Network (RMDN) [4] whichrepresents the only deep model addressing video saliency WhileMLNet works on images discarding the temporal informationRMDN encodes short sequences in a similar way to our COARSEmodule and then relies on a LSTM architecture to model longterm dependencies and estimates the fixation map in terms of aGMM To favor the comparison both models were re-trained onthe DR(eye)VE datasetResults highlight the superiority of our multi-branch archi-tecture on all test sequences The gap in performance with respectto bottom-up unsupervised approaches [72] [73] is higher and ismotivated by the peculiarity of the attention behavior within thedriving context which calls for a task-oriented training procedureMoreover MLNetrsquos low performance testifies for the need of ac-counting for the temporal correlation between consecutive framesthat distinguishes the tasks of attention prediction in images andvideos Indeed RMDN processes video inputs and outperformsMLNet on both DKL and IG metrics performing comparablyon CC Nonetheless its performance is still limited indeedqualitative results reported in Fig 12 suggest that long termdependencies captured by its recurrent module lead the networktowards the regression of the mean discarding contextual and

TABLE 3Experiments illustrating the superior performance of the

multi-branch model over several baselines and competitors Wereport both the average across the complete test sequences and only

the acting frames

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

Baseline Gaussian 040 216 -049 026 241 003Baseline Mean 051 160 000 022 235 000

Mathe et al [59] 004 330 -208 - - -Wang et al [72] 004 340 -221 - - -Wang et al [73] 011 306 -172 - - -

MLNet [15] 044 200 -088 032 235 -036RMDN [4] 041 177 -006 031 213 031

Palazzi et al [46] 055 148 -021 037 200 020multi-branch 056 140 004 041 180 051

10

Input frame GT multi-branch [46] [4] [15]

Fig 12 Qualitative assessment of the predicted fixation maps From left to right input clip ground truth map our prediction prediction of theprevious version of the model [46] prediction of RMDN [4] and prediction of MLNet [15]

frame-specific variations that would be preferrable to keep Tosupport this intuition we measure the average DKL betweenRMDN predictions and the mean training fixation map (Base-line Mean) resulting in a value of 011 Being lower than thedivergence measured with respect to groundtruth maps this valuehighlights the closer correlation to a central baseline rather thanto groundtruth Eventually we also observe improvements withrespect to our previous proposal [46] that relies on a more com-plex backbone model (also including a deconvolutional module)and processes RGB clips only The gap in performance residesin the greater awareness of our multi-branch architecture ofthe aspects that characterize the driving task as emerged fromthe analysis in Sec 31 The positive performances of our modelare also confirmed when evaluated on the acting partition of thedataset We recall that acting indicates sub-sequences exhibiting asignificant task-driven shift of attention from the center of theimage (Fig 5) Being able to predict the FoA also on actingsub-sequences means that the model captures the strong centeredattention bias but is capable of generalizing when required by thecontextThis is further shown by the comparison against a centeredGaussian baseline (BG) and against the average of all trainingset fixation maps (BM) The former baseline has proven effectiveon many image saliency detection tasks [11] while the latterrepresents a more task-driven version The superior performanceof the multi-branch model wrt baselines highlights thatdespite the attention is often strongly biased towards the vanishingpoint of the road the network is able to deal with sudden task-driven changes in gaze direction

52 Model analysis

In this section we investigate the behavior of our proposed modelunder different landscapes time of day and weather (Sec 521)we study the contribution of each branch to the FoA predictiontask (Sec 522) and we compare the learnt attention dynamicsagainst the one observed in the human data (Sec 523)

12

13

14

15

16

17

18

19

Downt

own

Coun

trysid

e

High

way

Mornin

g

Even

ingNi

ght

Sunn

y

Cloud

yRa

iny

DKL

Multi-branchImage

FlowSegm

________________________Landscape ________________________Time of day ________________________Weather

Fig 13 DKL of the different branches in several conditions (from left toright downtown countryside highway morning evening night sunnycloudy rainy) Underlining highlights difference of aggregation in termsof landscape time of day and weather Please note that lower DKL

indicates better predictions

521 Dependency on driving environment

The DR(eye)VE data has been recorded under varying land-scapes time of day and weather conditions We tested our modelin all such different driving conditions As would be expectedFig 13 shows that the human attention is easier to predict inhighways rather than downtown where the focus can shift towardsmore distractors The model seems more reliable in eveningscenarios rather than morning or night where we observed betterlightning conditions and lack of shadows over-exposure and soon Lastly in rainy conditions we notice that human gaze iseasier to model possibly due to the higher level of awarenessdemanded to the driver and his consequent inability to focus awayfrom vanishing point To support the latter intuition we measuredthe performance of BM baseline (ie the average training fixationmap) grouped for weather condition As expected theDKL valuein rainy weather (153) is significantly lower than the ones forcloudy (161) and sunny weather (175) highlighting that whenrainy the driver is more focused on the road

11

|Σ| = 250times 109 |Σ| = 157times 109 |Σ| = 349times 108 |Σ| = 120times 108 |Σ| = 538times 107

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 14 Model prediction averaged across all test sequences and grouped by driving speed As the speed increases the area of the predicted mapshrinks recalling the trend observed in ground truth maps As in Fig 7 for each map a two dimensional Gaussian is fitted and the determinant ofits covariance matrix Σ is reported as a measure of the spread

road sidewalk buildings traffic signs trees flowerbed sky vehicles people cycles10

-4

10-3

10-2

10-1

100

prob

abilit

y of

fixa

tions

(log

sca

le)

Fig 15 Comparison between ground truth (gray bars) and predictedfixation maps (colored bars) when used to mask semantic segmentationof the scene The probability of fixation (in log-scale) for both groundtruth and model prediction is reported for each semantic class Despiteabsolute errors exist the two bar series agree on the relative importanceof different categories

522 Ablation study

In order to validate the design of the multi-branch model(see Sec 42) here we study the individual contributions of thedifferent branches by disabling one or more of themResults in Tab 4 show that the RGB branch plays a majorrole in FoA prediction The motion stream is also beneficialand provides a slight improvement that becomes clearer in theacting subsequences Indeed optical flow intrinsically captures avariety of peculiar scenarios that are non-trivial to classify whenonly color information is provided eg when the car is still at atraffic light or is turning The semantic stream on the other handprovides very little improvement In particular from Tab 4 and by

TABLE 4The ablation study performed on our multi-branch model I F and S

represent image optical flow and semantic segmentation branchesrespectively

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

I 0554 1415 -0008 0403 1826 0458F 0516 1616 -0137 0368 2010 0349S 0479 1699 -0119 0344 2082 0288

I+F 0558 1399 0033 0410 1799 0510I+S 0554 1413 -0001 0404 1823 0466F+S 0528 1571 -0055 0380 1956 0427

I+F+S 0559 1398 0038 0410 1797 0515

specifically comparing I+F and I+F+S a slight increase in the IGmeasure can be appreciated Nevertheless such improvement hasto be considered negligible when compared to color and motionsuggesting that in presence of efficiency concerns or real-timeconstraints the semantic stream can be discarded with little lossesin performance However we expect the benefit from this branchto increase as more accurate segmentation models will be released

523 Do we capture the attention dynamics

The previous sections validate quantitatively the proposed modelNow we assess its capability to attend like a human driverby comparing its predictions against the analysis performed inSec 31First we report the average predicted fixation map in severalspeed ranges in Fig 14 The conclusions we draw are twofoldi) generally the model succeeds in modeling the behavior ofthe driver at different speeds and ii) as the speed increasesfixation maps exhibit lower variance easing the modeling taskand prediction errors decreaseWe also study how often our model focuses on different semanticcategories in a fashion that recalls the analysis of Sec 31 butemploying our predictions rather than ground truth maps as focusof attention More precisely we normalize each map so thatthe maximum value equals 1 and apply the same thresholdingstrategy described in Sec 31 Likewise for each threshold valuea histogram over class labels is built by accounting all pixelsfalling within the binary map for all test frames This results innine histograms over semantic labels that we merge together byaveraging probabilities belonging to different threshold Fig 15shows the comparison Color bars represent how often the pre-dicted map focuses on a certain category while gray bars depictground truth behavior and are obtained by averaging histogramsin Fig 8 across different thresholds Please note that to highlightdifferences for low populated categories values are reported ona logarithmic scale The plot shows a certain degree of absoluteerror is present for all categories However in a broader sense ourmodel replicates the relative weight of different semantic classeswhile driving as testified by the importance of roads and vehiclesthat still dominate against other categories such as people andcycles that are mostly neglected This correlation is confirmed byKendall rank coefficient which scored 051 when computed onthe two bar series

53 Visual assessment of predicted fixation maps

To further validate the predictions of our model from the humanperception perspective 50 people with at least 3 years of driving

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

4

TABLE 1Summary of the DR(eye)VE dataset characteristics The dataset was designed to embody the most possible diversity in the combination of

different features The reader is referred to either the additional material or to the dataset presentation [2] for details on each sequence

Videos Frames Drivers Weather conditions Lighting Gaze Info Metadata Camera Viewpoint

74 555000 8sunny day raw fixations GPS driver (720p)cloudy evening gaze map car speed car (1080p)rainy night pupil dilation car course

TABLE 2A comparison between DR(eye)VE and other datasets

Dataset Frames Drivers Scenarios Annotations Real-world Public

Pugeault et al [52] 158668 ndash Countryside HighwayDowntown

Gaze MapsDriverrsquos Actions Yes No

Simon et al [58] 40 30 Downtown Gaze Maps No NoUnderwood et al [68] 120 77 Urban Motorway ndash No NoFridman et al [19] 1860761 50 Highway 6 Gaze Location Classes Yes No

DR(eye)VE [2] 555000 8 Countryside HighwayDowntown

Gaze MapsGPS Speed Course Yes Yes

observersrsquo fixations Indeed two different observers cannotexperience the same scene at the same time (eg two driverscannot be at the same time in the same point of the street) Theonly chance to average among different observers would be theadoption of a simulation environment but it has been provedthat the cognitive load in controlled experiments is lower thanin real test scenarios and it effects the true attention mechanismof the observer [55] In our preliminary DR(eye)VE release[2] fixation points were aggregated and smoothed by means ofa temporal sliding window In such a way temporal filteringdiscarded momentary glimpses that contain precious informationabout the driverrsquos attention Following the psychological protocolin [40] and [25] this limitation was overcome in the currentrelease where the new fixation maps were computed withouttemporal smoothing Both [40] and [25] highlight the high degreeof subjectivity of scene scanpaths in short temporal windows (lt 1sec) and suggest to neglect the fixations pop-out order withinsuch windows This mechanism also ameliorates the inhibitionof return phenomenon that may prevent interesting objects to beobserved twice in short temporal intervals [27] [51] leading tothe underestimation of their importanceMore formally the fixation map Ft for a frame at time t is builtby accumulating projected gaze points in a temporal slidingwindow of k = 25 frames centered in t For each time step t+ iin the window where i isin minusk2 minus

k2 + 1 k2 minus 1 k2 gaze

points projections on Ft are estimated through the homographytransformation Ht

t+i that projects points from the image plane atframe t+ i namely pt+i to the image plane in Ft A continuousfixation map is obtained from the projected fixations by centeringon each of them a multivariate Gaussian having a diagonalcovariance matrix Σ (the spatial variance of each variable is set to

Fig 3 Registration between the egocentric and roof-mounted cameraviews by means of SIFT descriptor matching

Fig 4 Resulting fixation map from a 1 second integration (25 frames)The adoption of the max aggregation of equation 1 allows to account inthe final map two brief glances towards traffic lights

σ2s = 200 pixels) and taking the max value along the time axis

Ft(x y) = maxiisin(minus k

2 k2 )N ((x y) |Ht

t+i middot pt+iΣ) (1)

The Gaussian variance has been computed by averaging the ETGspatial acquisition errors on 20 observers looking at calibrationpatterns at different distances from 5 to 15 meters The describedprocess can be appreciated in Fig 4 Eventually each map Ft isnormalized to sum to 1 so that it can be considered a probabilitydistribution of fixation points

Labeling attention drifts Fixation maps exhibit a verystrong central bias This is common in saliency annotations [60]and even more in the context of driving For these reasons thereis a strong unbalance between lots of easy-to-predict scenariosand unfrequent but interesting hard-to-predict events

To enable the evaluation of computational models under suchcircumstances the DR(eye)VE dataset has been extended witha set of further annotations For each video subsequences whoseground truth poorly correlates with the average ground truth ofthat sequence are selected We employ Pearsonrsquos CorrelationCoefficient (CC) and select subsequences with CC lt 03 Thishappens when the attention of the driver focuses far from the

5

(a) Acting - 69 719 (b) Inattentive - 12 282

(c) Error - 22 893 (d) Subjective - 3 166

Fig 5 Examples of the categorization of frames where gaze is farfrom the mean Overall 108 060 frames (sim20 of DR(eye)VE) wereextended with this type of information

vanishing point of the road Examples of such subsequencesare depicted in Fig 5 Several human annotators inspected theselected frames and manually split them into (a) acting (b)inattentive (c) errors and (d) subjective eventsbull errors can happen either due to failures in the measuring tool

(eg in extreme lighting conditions) or in the successive dataprocessing phase (eg SIFT matching)

bull inattentive subsequences occur when the driver focuses hisgaze on objects unrelated to the driving task (eg looking atan advertisement)

bull subjective subsequences describe situations in which theattention is closely related to the individual experience ofthe driver eg a road sign on the side might be an interestingelement to focus for someone that has never been on that roadbefore but might be safely ignored by someone who drivesthat road every day

bull acting subsequences include all the remaining onesActing subsequences are particularly interesting as the deviationof driverrsquos attention from the common central pattern denotes anintention linked to task-specific actions (eg turning changinglanes overtaking ) For these reasons subsequences of thiskind will have a central role in the evaluation of predictive modelsin Sec 5

31 Dataset analysisBy analyzing the dataset frames the very first insight is thepresence of a strong attraction of driverrsquos focus towards thevanishing point of the road that can be appreciated in Fig 6 Thesame phenomenon was observed in previous studies [6] [67] inthe context of visual search tasks We observed indeed that driversoften tend to disregard road signals cars coming from the opposite

(a) (b)

Fig 6 Mean frame (a) and fixation map (b) averaged across the wholesequence 02 highlighting the link between driverrsquos focus and the van-ishing point of the road

direction and pedestrians on sidewalks This is an effect of humanperipheral vision [56] that allows observers to still perceive andinterpret stimuli out of - but sufficiently close to - their focus ofattention (FoA) A driver can therefore achieve a larger area ofattention by focusing on the roadrsquos vanishing point due to thegeometry of the road environment many of the objects worth ofattention are coming from there and have already been perceivedwhen distantMoreover the gaze location tends to drift from this central attrac-

tor when the context changes in terms of car speed and landscapeIndeed [53] suggests that our brain is able to compensate spatiallyor temporally dense information by reducing the visual field sizeIn particular as the car travels at higher speed the temporal densityof information (ie the amount of information that the driver needsto elaborate per unit of time) increases this causes the usefulvisual field of the driver to shrink [53] We also observe thisphenomenon in our experiments as shown in Fig 7DR(eye)VE data also highlight that the driverrsquos gaze is attractedtowards specific semantic categories To reach the above conclu-sion the dataset is analysed by means of the semantic segmenta-tion model in [76] and the distribution of semantic classes withinthe fixation map evaluated More precisely given a segmentedframe and the corresponding fixation map the probability for eachsemantic class to fall within the area of attention is computed asfollows First the fixation map (which is continuous in [0 1]) isnormalized such that the maximum value equals 1 Then nine bi-nary maps are constructed by thresholding such continuous valueslinearly in the interval [0 1] As the threshold moves towards 1(the maximum value) the area of interest shrinks around the realfixation points (since the continuous map is modeled by meansof several Gaussians centered in fixation points see previoussection) For every threshold a histogram over semantic labelswithin the area of interest is built by summing up occurrencescollected from all DR(eye)VE frames Fig 8 displays the resultfor each class the probability of a pixel to fall within the region ofinterest is reported for each threshold value The figure providesinsight about which categories represent the real focus of attentionand which ones tend to fall inside the attention region just byproximity with the formers Object classes that exhibit a positivetrend such as road vehicles and people are the real focus ofthe gaze since the ratio of pixels classified accordingly increaseswhen the observed area shrinks around the fixation point In abroader sense the figure suggests that despite while driving ourfocus is dominated by road and vehicles we often observe specificobjects categories even if they contain little information useful todrive

4 MULTI-BRANCH DEEP ARCHITECTURE FORFOCUS OF ATTENTION PREDICTION

The DR(eye)VE dataset is sufficiently large to allow theconstruction of a deep architecture to model common attentionalpatterns Here we describe our neural network model to predicthuman FoA while driving

Architecture design In the context of high level videoanalysis (eg action recognition and video classification) ithas been shown that a method leveraging single frames canbe outperformed if a sequence of frames is used as inputinstead [31] [65] Temporal dependencies are usually modeledeither by 3D convolutional layers [65] tailored to capture short

6

|Σ| = 249times 109 |Σ| = 203times 109 |Σ| = 861times 108 |Σ| = 102times 109 |Σ| = 928times 108

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 7 As speed gradually increases driverrsquos attention converges towards the vanishing point of the road (a) When the car is approximatelystationary the driver is distracted by many objects in the scene (b-e) As the speed increases the driverrsquos gaze deviates less and less from thevanishing point of the road To measure this effect quantitatively a two-dimensional Gaussian is fitted to approximate the mean map for each speedrange and the determinant of the covariance matrix Σ is reported as an indication of its spread (the determinant equals the product of eigenvalueseach of which measures the spread along a different data dimension) The bar plots illustrate the amount of downtown (red) countryside (green)and highway (blue) frames that concurred to generate the average gaze position for a specific speed range Best viewed on screen

Pro

babi

lity

offix

atio

n

road0

01

02

03

04

sidewalk buildings traffic signs trees flowerbed sky vehicles people0

0005

001

cycles

Fig 8 Proportion of semantic categories that fall within the driverrsquos fixation map when thresholded at increasing values (from left to right) Categoriesexhibiting a positive trend (eg road and vehicles) suggest a real attention focus while a negative trend advocates for an awareness of the objectwhich is only circumstantial See Sec 31 for details

conv3D64

conv3D128

conv3D256

conv3D256

conv2D1

pool3D

pool3D

pool3D

up2D

conv3D

64

conv3D conv3D conv3Dpool3D pool3D pool3D

up2D

128 256 256 1

conv2D

Fig 9 The COARSE module is made of an encoder based on C3D network [65] followed by a bilinear upsampling (bringing representations back tothe resolution of the input image) and a final 2D convolution During feature extraction the temporal axis is lost due to 3D pooling All convolutionallayers are preceded by zero paddings in order keep borders and all kernels have size 3 along all dimensions Pooling layers have size and strideof (1 2 2 4) and (2 2 2 1) along temporal and spatial dimensions respectively All activations are ReLUs

range correlations or by recurrent architectures (eg LSTMGRU) that can model longer term dependencies [3] [47] Ourmodel follows the former approach relying on the assumptionthat a small time window (eg half a second) holds sufficientcontextual information for predicting where the driver wouldfocus in that moment Indeed human drivers can take even lesstime to react to an unexpected stimulus Our architecture takes asequence of 16 consecutive frames (asymp 065s) as input (called clipsfrom now on) and predicts the fixation map for the last frame ofsuch clipMany of the architectural choices made to design the networkcome from insights from the dataset analysis presented in Sec31In particular we rely on the following resultsbull the driversrsquo FoA exhibits consistent patterns suggesting that

it can be reproduced by a computational modelbull the driversrsquo gaze is affected by a strong prior on objects

semantics eg drivers tend to focus on items lying on theroad

bull motion cues like vehicle speed are also key factors thatinfluence gaze

Accordingly the model output merges three branches with identi-cal architecture unshared parameters and different input domains

the RGB image the semantic segmentation and the optical flowfield We call this architecture multi-branch model Followinga bottom-up approach in Sec 41 the building blocks of eachbranch are motivated and described Later in Sec 42 it will beshown how the branches merge into the final model

41 Single FoA branch

Each branch of the multi-branch model is a two-input two-output architecture composed of two intertwined streams The aimof this peculiar setup is to prevent the network from learninga central bias that would otherwise stall the learning in earlytraining stages 1 To this end one of the streams is given as input(output) a severely cropped portion of the original image (groundtruth) ensuring a more uniform distribution of the true gaze andruns through the COARSE module described below Similarly theother stream uses the COARSE module to obtain a rough predictionover the full resized image and then refines it through a stack ofadditional convolutions called REFINE model At test time onlythe output of the REFINE stream is considered Both streams rely

1 For further details the reader can refer to Sec 14 and Sec 15 of thesupplementary material

7

last frame

shared weights

Videoclip

CROP

RESIZE

COARSE REFINE outputinput

448

448

cropped videoclip

112

112

112

112

resized videoclip

Fig 10 A single FoA branch of our prediction architecture The COARSE module (see Fig 9) is applied to both a cropped and a resized version ofthe input tensor which is a videoclip of 16 consecutive frames The cropped input is used during training to augment the data and the variety ofground truth fixation maps The prediction of the resized input is stacked with the last frame of the videoclip and fed to a stack of convolutional layers(refinement module) with the aim of refining the prediction Training is performed end-to-end and weights between COARSE modules are shared Attest time only the refined predictions are used Note that the complete model is composed of three of these branches (see Fig 11) each of whichpredicting visual attention for different inputs (namely image optical flow and semantic segmentation) All activations in the refinement module areLeakyReLU with α = 10minus3 except for the last single channel convolution that features ReLUs Crop and resize streams are highlighted by lightblue and orange arrows respectively

on the COARSE module the convolutional backbone (with sharedweights) which provides the rough estimate of the attentional mapcorresponding to a given clip This component is detailed in Fig 9The COARSE module is based on the C3D architecture [65] thatencodes video dynamics by applying a 3D convolutional kernelon the 4D input tensor As opposed to 2D convolutions that stridealong the width and height dimension of the input tensor a 3Dconvolution also strides along time Formally the j-th feature mapin the i-th layer at position (x y) at time t is computed as

vxytij = bij +summ

Piminus1sump=0

Qiminus1sumq=0

Riminus1sumr=0

wpqrijmvx+py+qt+riminus1m (2)

where m indexes different input feature maps wpqrijm is the valueat the position (p q) at time r of the kernel connected to the m-thfeature map and Pi Qi and Ri are the dimensions of the kernelalong width height and temporal axis respectively bij is the biasfrom layer i to layer jFrom C3D only the most general-purpose features are retainedby removing the last convolutional layer and the fully connectedlayers which are strongly linked to the original action recognitiontask The size of the last pooling layer is also modified in order tocover the remaining temporal dimension entirely This collapsesthe tensor from 4D to 3D making the output independent of timeEventually a bilinear upsampling brings the tensor back to theinput spatial resolution and a 2D convolution merges all featuresinto one channel See Fig 9 for additional details on the COARSEmodule

Training the two streams together The architecture of asingle FoA branch is depicted in Fig 10 During training the firststream feeds the COARSE network with random crops forcingthe model to learn the current focus of attention given visualcues rather than prior spatial location The C3D training processdescribed in [65] employs a 128 times 128 image resize and thena 112 times 112 random crop However the small difference in thetwo resolutions limits the variance of gaze position in ground

truth fixation maps and is not sufficient to avoid the attractiontowards the center of the image For this reason training imagesare resized to 256times 256 before being cropped to 112times 112 Thiscrop policy generates samples that cover less than a quarter ofthe original image thus ensuring a sufficient variety in predictiontargets This comes at the cost of a coarser prediction as cropsget smaller the ratio of pixels in the ground truth covered by gazeincreases leading the model to learn larger mapsIn contrast the second stream feeds the same COARSE modelwith the same images this time resized to 112 times 112 ndash and notcropped The coarse prediction obtained from the COARSE modelis then concatenated with the final frame of the input clip iethe frame corresponding to the final prediction Eventually theconcatenated tensor goes through the REFINE module to obtaina higher resolution prediction of the FoAThe overall two-stream training procedure for a single branch issummarized in Algorithm 1

Training objective Prediction cost can be minimized interms of Kullback-Leibler divergence

DKL(Y Y ) =sumi

Y (i) log

(ε+

Y (i)

ε+ Y (i)

)(3)

where Y is the ground truth distribution Y is the prediction thesummation index i spans across image pixels and ε is a smallconstant that ensures numerical stability2 Since each single FoAbranch computes an error on both the cropped image stream andthe resized image stream the branch loss can be defined as

Lb(XbY) =summ

(DKL(φ(Y m)C(φ(Xm

b ))) +

DKL(Y mR(C(ψ(Xmb )) Xm

b )))

) (4)

2 Please note that DKL inputs are always normalized to be a validprobability distribution despite this may be omitted in notation to improveequations readability

8

Algorithm 1 TRAINING The model is trained in two steps first each branch is trained separately through iterations detailed inprocedure SINGLE_BRANCH_TRAINING_ITERATION then the three branches are fine-tuned altogether as shown by procedureMULTI_BRANCH_FINE-TUNING_ITERATION For clarity we omit from notation i) the subscript b denoting the current domain inall X x and y variables in the single branch iteration and ii) the normalization of the sum of the outputs from each branch in line 13

1 procedure A SINGLE_BRANCH_TRAINING_ITERATION

input domain data X = x1 x2 x16 true attentional map y of last frame x16 of videoclip Xoutput branch loss Lb computed on input sample (X y)

2 Xres larr resize(X (112 112))3 Xcrop ycrop larr get_crop((X y) (112 112))4 ycrop larr COARSE(Xcrop) get coarse prediction on uncentered crop5 y larr REFINE(stack(x16 upsample(COARSE(Xres)))) get refined prediction over whole image6 Lb(XY )larr DKL(ycropycrop) +DKL(yy) compute branch loss as in Eq 4

7 procedure B MULTI_BRANCH_FINE-TUNING_ITERATION

input data X = x1 x2 x16 for all domains true attentional map y of last frame x16 of videoclip Xoutput overall loss L computed on input sample (X y)

8 Xres larr resize(X (112 112))9 Xcrop ycrop larr get_crop((X y) (112 112))

10 for branch b isin RGB flow seg do11 ybcrop larr COARSE(Xbcrop ) as in line 4 of the above procedure12 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres )))) as in line 5 of the above procedure13 L(XY )larr DKL(ycrop

sumb ybcrop ) +DKL(y

sumb yb) compute overall loss as in Eq 5

where C and R denote COARSE and REFINE modules(Xm

b Ym) isin Xb times Y is the m-th training example in the

b-th domain (namely RGB optical flow semantic segmentation)and φ and ψ indicate the crop and the resize functions respectively

Inference step While the presence of the C(φ(Xmb )) stream

is beneficial in training to reduce the spatial bias at test timeonly the R(C(ψ(Xm

b )) Xmb )) stream producing higher quality

prediction is used The outputs of such stream from each branch bare then summed together as explained in the following section

42 Multi-Branch modelAs described at the beginning of this section and depicted inFig 11 the multi-branch model is composed of three iden-tical branches The architecture of each branch has already beendescribed in Sec 41 above Each branch exploits complementaryinformation from a different domain and contributes to the finalprediction accordingly In detail the first branch works in the RGBdomain and processes raw visual data about the scene XRGBThe second branch focuses on motion through the optical flowrepresentation Xflow described in [23] Eventually the last branchtakes as input semantic segmentation probability maps Xseg Forthis last branch the number of input channels depends on thespecific algorithm used to extract the results 19 in our setup (Yu

Algorithm 2 INFERENCE At test time the data extracted fromthe resized videoclip is input to the three branches and their outputis summed and normalized to obtain the final FoA prediction

input data X = x1 x2 x16 for all domainsoutput predicted FoA map y1 Xres larr resize(X (112 112))2 for branch b isin RGB flow seg do3 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres ))))4 y larr

sumb yb

sumi

sumb yb(i)

and Koltun [76]) The three independent predicted FoA maps aresummed and normalized to result in a probability distributionTo allow for larger batch size we choose to bootstrap each branchindependently by training it according to Eq 4 Then the completemulti-branch model which merges the three branches is fine-tuned with the following loss

L(X Y) =summ

(DKL(φ(Y m)

sumb

C(φ(Xmb ))) +

DKL(Y msumb

R(C(ψ(Xmb )) Xm

b )))

)

(5)The algorithm describing the complete inference over themulti-branch model in detailed in Alg 2

5 EXPERIMENTS

In this section we evaluate the performance of the proposedmulti-branch model First we start by comparing our modelagainst some baselines and other methods in literature Followingthe guidelines in [12] for the evaluation phase we rely on Pear-sonrsquos Correlation Coefficient (CC) and KullbackndashLeibler Diver-gence (DKL) measures Moreover we evaluate the InformationGain (IG) [35] measure to assess the quality of a predicted mapP with respect to a ground truth map Y in presence of a strongbias as

IG(P YB) =1

N

sumi

Yi[(log2(ε+ Pi)minus log2(ε+Bi)] (6)

where i is an index spanning all the N pixels in the image B thebias computed as the average training fixation map and ε ensuresnumerical stabilityFurthermore we conduct an ablation study to investigate howdifferent branches affect the final prediction and how their mutualinfluence changes in different scenarios We then study whetherour model captures the attention dynamics observed in Sec 31

9

nalprediction

+

FoAfrom motionow

clip

FoAfrom semanticsseg

clip

FoAfrom colorRGB

clip

Fig 11 The multi-branch model is composed of three different branches each of which has its own set of parameters and their predictions aresummed to obtain the final map Note that in this figure cropped streams are dropped to ease representation but are employed during training (asdiscussed in Sec 42 and depicted in Fig 10

Eventually we assess our model from a human perceptionperspective

Implementation details The three different pathways ofthe multi-branch model (namely FoA from color frommotion and from semantics) have been pre-trained independentlyusing the same cropping policy of Sec 42 and minimizing theobjective function in Eq 4 Each branch has been respectively fedwithbull 16 frames clips in raw RGB color spacebull 16 frames clips with optical flow maps encoded as color

images through the flow field encoding [23]bull 16 frames clips holding semantic segmentation from [76]

encoded as 19 scalar activation maps one per segmentationclass

During individual branch pre-training clips were randomly mir-rored for data augmentation We employ Adam optimizer with pa-rameters as suggested in the original paper [32] with the exceptionof the learning rate that we set to 10minus4 Eventually batch size wasfixed to 32 and each branch was trained until convergence TheDR(eye)VE dataset is split into train validation and test set asfollows sequences 1-38 are used for training sequences 39-74 fortesting The 500 frames in the middle of each training sequenceconstitute the validation setMoreover the complete multi-branch architecture was fine-tuned using the same cropping and data augmentation strategiesminimizing cost function in Eq 5 In this phase batch size was setto 4 due to GPU memory constraints and learning rate value waslowered to 10minus5 Inference time of each branch of our architectureis asymp 30 milliseconds per videoclip on an NVIDIA Titan X

51 Model evaluation

In Tab 3 we report results of our proposal against other state-of-the-art models [4] [15] [46] [59] [72] [73] evaluated both onthe complete test set and on acting subsequences only All thecompetitors with the exception of [46] are bottom-up approachesand mainly rely on appearance and motion discontinuities To testthe effectiveness of deep architectures for saliency prediction wecompare against the Multi-Level Network (MLNet) [15] which

scored favourably in the MIT300 saliency benchmark [11] andthe Recurrent Mixture Density Network (RMDN) [4] whichrepresents the only deep model addressing video saliency WhileMLNet works on images discarding the temporal informationRMDN encodes short sequences in a similar way to our COARSEmodule and then relies on a LSTM architecture to model longterm dependencies and estimates the fixation map in terms of aGMM To favor the comparison both models were re-trained onthe DR(eye)VE datasetResults highlight the superiority of our multi-branch archi-tecture on all test sequences The gap in performance with respectto bottom-up unsupervised approaches [72] [73] is higher and ismotivated by the peculiarity of the attention behavior within thedriving context which calls for a task-oriented training procedureMoreover MLNetrsquos low performance testifies for the need of ac-counting for the temporal correlation between consecutive framesthat distinguishes the tasks of attention prediction in images andvideos Indeed RMDN processes video inputs and outperformsMLNet on both DKL and IG metrics performing comparablyon CC Nonetheless its performance is still limited indeedqualitative results reported in Fig 12 suggest that long termdependencies captured by its recurrent module lead the networktowards the regression of the mean discarding contextual and

TABLE 3Experiments illustrating the superior performance of the

multi-branch model over several baselines and competitors Wereport both the average across the complete test sequences and only

the acting frames

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

Baseline Gaussian 040 216 -049 026 241 003Baseline Mean 051 160 000 022 235 000

Mathe et al [59] 004 330 -208 - - -Wang et al [72] 004 340 -221 - - -Wang et al [73] 011 306 -172 - - -

MLNet [15] 044 200 -088 032 235 -036RMDN [4] 041 177 -006 031 213 031

Palazzi et al [46] 055 148 -021 037 200 020multi-branch 056 140 004 041 180 051

10

Input frame GT multi-branch [46] [4] [15]

Fig 12 Qualitative assessment of the predicted fixation maps From left to right input clip ground truth map our prediction prediction of theprevious version of the model [46] prediction of RMDN [4] and prediction of MLNet [15]

frame-specific variations that would be preferrable to keep Tosupport this intuition we measure the average DKL betweenRMDN predictions and the mean training fixation map (Base-line Mean) resulting in a value of 011 Being lower than thedivergence measured with respect to groundtruth maps this valuehighlights the closer correlation to a central baseline rather thanto groundtruth Eventually we also observe improvements withrespect to our previous proposal [46] that relies on a more com-plex backbone model (also including a deconvolutional module)and processes RGB clips only The gap in performance residesin the greater awareness of our multi-branch architecture ofthe aspects that characterize the driving task as emerged fromthe analysis in Sec 31 The positive performances of our modelare also confirmed when evaluated on the acting partition of thedataset We recall that acting indicates sub-sequences exhibiting asignificant task-driven shift of attention from the center of theimage (Fig 5) Being able to predict the FoA also on actingsub-sequences means that the model captures the strong centeredattention bias but is capable of generalizing when required by thecontextThis is further shown by the comparison against a centeredGaussian baseline (BG) and against the average of all trainingset fixation maps (BM) The former baseline has proven effectiveon many image saliency detection tasks [11] while the latterrepresents a more task-driven version The superior performanceof the multi-branch model wrt baselines highlights thatdespite the attention is often strongly biased towards the vanishingpoint of the road the network is able to deal with sudden task-driven changes in gaze direction

52 Model analysis

In this section we investigate the behavior of our proposed modelunder different landscapes time of day and weather (Sec 521)we study the contribution of each branch to the FoA predictiontask (Sec 522) and we compare the learnt attention dynamicsagainst the one observed in the human data (Sec 523)

12

13

14

15

16

17

18

19

Downt

own

Coun

trysid

e

High

way

Mornin

g

Even

ingNi

ght

Sunn

y

Cloud

yRa

iny

DKL

Multi-branchImage

FlowSegm

________________________Landscape ________________________Time of day ________________________Weather

Fig 13 DKL of the different branches in several conditions (from left toright downtown countryside highway morning evening night sunnycloudy rainy) Underlining highlights difference of aggregation in termsof landscape time of day and weather Please note that lower DKL

indicates better predictions

521 Dependency on driving environment

The DR(eye)VE data has been recorded under varying land-scapes time of day and weather conditions We tested our modelin all such different driving conditions As would be expectedFig 13 shows that the human attention is easier to predict inhighways rather than downtown where the focus can shift towardsmore distractors The model seems more reliable in eveningscenarios rather than morning or night where we observed betterlightning conditions and lack of shadows over-exposure and soon Lastly in rainy conditions we notice that human gaze iseasier to model possibly due to the higher level of awarenessdemanded to the driver and his consequent inability to focus awayfrom vanishing point To support the latter intuition we measuredthe performance of BM baseline (ie the average training fixationmap) grouped for weather condition As expected theDKL valuein rainy weather (153) is significantly lower than the ones forcloudy (161) and sunny weather (175) highlighting that whenrainy the driver is more focused on the road

11

|Σ| = 250times 109 |Σ| = 157times 109 |Σ| = 349times 108 |Σ| = 120times 108 |Σ| = 538times 107

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 14 Model prediction averaged across all test sequences and grouped by driving speed As the speed increases the area of the predicted mapshrinks recalling the trend observed in ground truth maps As in Fig 7 for each map a two dimensional Gaussian is fitted and the determinant ofits covariance matrix Σ is reported as a measure of the spread

road sidewalk buildings traffic signs trees flowerbed sky vehicles people cycles10

-4

10-3

10-2

10-1

100

prob

abilit

y of

fixa

tions

(log

sca

le)

Fig 15 Comparison between ground truth (gray bars) and predictedfixation maps (colored bars) when used to mask semantic segmentationof the scene The probability of fixation (in log-scale) for both groundtruth and model prediction is reported for each semantic class Despiteabsolute errors exist the two bar series agree on the relative importanceof different categories

522 Ablation study

In order to validate the design of the multi-branch model(see Sec 42) here we study the individual contributions of thedifferent branches by disabling one or more of themResults in Tab 4 show that the RGB branch plays a majorrole in FoA prediction The motion stream is also beneficialand provides a slight improvement that becomes clearer in theacting subsequences Indeed optical flow intrinsically captures avariety of peculiar scenarios that are non-trivial to classify whenonly color information is provided eg when the car is still at atraffic light or is turning The semantic stream on the other handprovides very little improvement In particular from Tab 4 and by

TABLE 4The ablation study performed on our multi-branch model I F and S

represent image optical flow and semantic segmentation branchesrespectively

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

I 0554 1415 -0008 0403 1826 0458F 0516 1616 -0137 0368 2010 0349S 0479 1699 -0119 0344 2082 0288

I+F 0558 1399 0033 0410 1799 0510I+S 0554 1413 -0001 0404 1823 0466F+S 0528 1571 -0055 0380 1956 0427

I+F+S 0559 1398 0038 0410 1797 0515

specifically comparing I+F and I+F+S a slight increase in the IGmeasure can be appreciated Nevertheless such improvement hasto be considered negligible when compared to color and motionsuggesting that in presence of efficiency concerns or real-timeconstraints the semantic stream can be discarded with little lossesin performance However we expect the benefit from this branchto increase as more accurate segmentation models will be released

523 Do we capture the attention dynamics

The previous sections validate quantitatively the proposed modelNow we assess its capability to attend like a human driverby comparing its predictions against the analysis performed inSec 31First we report the average predicted fixation map in severalspeed ranges in Fig 14 The conclusions we draw are twofoldi) generally the model succeeds in modeling the behavior ofthe driver at different speeds and ii) as the speed increasesfixation maps exhibit lower variance easing the modeling taskand prediction errors decreaseWe also study how often our model focuses on different semanticcategories in a fashion that recalls the analysis of Sec 31 butemploying our predictions rather than ground truth maps as focusof attention More precisely we normalize each map so thatthe maximum value equals 1 and apply the same thresholdingstrategy described in Sec 31 Likewise for each threshold valuea histogram over class labels is built by accounting all pixelsfalling within the binary map for all test frames This results innine histograms over semantic labels that we merge together byaveraging probabilities belonging to different threshold Fig 15shows the comparison Color bars represent how often the pre-dicted map focuses on a certain category while gray bars depictground truth behavior and are obtained by averaging histogramsin Fig 8 across different thresholds Please note that to highlightdifferences for low populated categories values are reported ona logarithmic scale The plot shows a certain degree of absoluteerror is present for all categories However in a broader sense ourmodel replicates the relative weight of different semantic classeswhile driving as testified by the importance of roads and vehiclesthat still dominate against other categories such as people andcycles that are mostly neglected This correlation is confirmed byKendall rank coefficient which scored 051 when computed onthe two bar series

53 Visual assessment of predicted fixation maps

To further validate the predictions of our model from the humanperception perspective 50 people with at least 3 years of driving

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

5

(a) Acting - 69 719 (b) Inattentive - 12 282

(c) Error - 22 893 (d) Subjective - 3 166

Fig 5 Examples of the categorization of frames where gaze is farfrom the mean Overall 108 060 frames (sim20 of DR(eye)VE) wereextended with this type of information

vanishing point of the road Examples of such subsequencesare depicted in Fig 5 Several human annotators inspected theselected frames and manually split them into (a) acting (b)inattentive (c) errors and (d) subjective eventsbull errors can happen either due to failures in the measuring tool

(eg in extreme lighting conditions) or in the successive dataprocessing phase (eg SIFT matching)

bull inattentive subsequences occur when the driver focuses hisgaze on objects unrelated to the driving task (eg looking atan advertisement)

bull subjective subsequences describe situations in which theattention is closely related to the individual experience ofthe driver eg a road sign on the side might be an interestingelement to focus for someone that has never been on that roadbefore but might be safely ignored by someone who drivesthat road every day

bull acting subsequences include all the remaining onesActing subsequences are particularly interesting as the deviationof driverrsquos attention from the common central pattern denotes anintention linked to task-specific actions (eg turning changinglanes overtaking ) For these reasons subsequences of thiskind will have a central role in the evaluation of predictive modelsin Sec 5

31 Dataset analysisBy analyzing the dataset frames the very first insight is thepresence of a strong attraction of driverrsquos focus towards thevanishing point of the road that can be appreciated in Fig 6 Thesame phenomenon was observed in previous studies [6] [67] inthe context of visual search tasks We observed indeed that driversoften tend to disregard road signals cars coming from the opposite

(a) (b)

Fig 6 Mean frame (a) and fixation map (b) averaged across the wholesequence 02 highlighting the link between driverrsquos focus and the van-ishing point of the road

direction and pedestrians on sidewalks This is an effect of humanperipheral vision [56] that allows observers to still perceive andinterpret stimuli out of - but sufficiently close to - their focus ofattention (FoA) A driver can therefore achieve a larger area ofattention by focusing on the roadrsquos vanishing point due to thegeometry of the road environment many of the objects worth ofattention are coming from there and have already been perceivedwhen distantMoreover the gaze location tends to drift from this central attrac-

tor when the context changes in terms of car speed and landscapeIndeed [53] suggests that our brain is able to compensate spatiallyor temporally dense information by reducing the visual field sizeIn particular as the car travels at higher speed the temporal densityof information (ie the amount of information that the driver needsto elaborate per unit of time) increases this causes the usefulvisual field of the driver to shrink [53] We also observe thisphenomenon in our experiments as shown in Fig 7DR(eye)VE data also highlight that the driverrsquos gaze is attractedtowards specific semantic categories To reach the above conclu-sion the dataset is analysed by means of the semantic segmenta-tion model in [76] and the distribution of semantic classes withinthe fixation map evaluated More precisely given a segmentedframe and the corresponding fixation map the probability for eachsemantic class to fall within the area of attention is computed asfollows First the fixation map (which is continuous in [0 1]) isnormalized such that the maximum value equals 1 Then nine bi-nary maps are constructed by thresholding such continuous valueslinearly in the interval [0 1] As the threshold moves towards 1(the maximum value) the area of interest shrinks around the realfixation points (since the continuous map is modeled by meansof several Gaussians centered in fixation points see previoussection) For every threshold a histogram over semantic labelswithin the area of interest is built by summing up occurrencescollected from all DR(eye)VE frames Fig 8 displays the resultfor each class the probability of a pixel to fall within the region ofinterest is reported for each threshold value The figure providesinsight about which categories represent the real focus of attentionand which ones tend to fall inside the attention region just byproximity with the formers Object classes that exhibit a positivetrend such as road vehicles and people are the real focus ofthe gaze since the ratio of pixels classified accordingly increaseswhen the observed area shrinks around the fixation point In abroader sense the figure suggests that despite while driving ourfocus is dominated by road and vehicles we often observe specificobjects categories even if they contain little information useful todrive

4 MULTI-BRANCH DEEP ARCHITECTURE FORFOCUS OF ATTENTION PREDICTION

The DR(eye)VE dataset is sufficiently large to allow theconstruction of a deep architecture to model common attentionalpatterns Here we describe our neural network model to predicthuman FoA while driving

Architecture design In the context of high level videoanalysis (eg action recognition and video classification) ithas been shown that a method leveraging single frames canbe outperformed if a sequence of frames is used as inputinstead [31] [65] Temporal dependencies are usually modeledeither by 3D convolutional layers [65] tailored to capture short

6

|Σ| = 249times 109 |Σ| = 203times 109 |Σ| = 861times 108 |Σ| = 102times 109 |Σ| = 928times 108

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 7 As speed gradually increases driverrsquos attention converges towards the vanishing point of the road (a) When the car is approximatelystationary the driver is distracted by many objects in the scene (b-e) As the speed increases the driverrsquos gaze deviates less and less from thevanishing point of the road To measure this effect quantitatively a two-dimensional Gaussian is fitted to approximate the mean map for each speedrange and the determinant of the covariance matrix Σ is reported as an indication of its spread (the determinant equals the product of eigenvalueseach of which measures the spread along a different data dimension) The bar plots illustrate the amount of downtown (red) countryside (green)and highway (blue) frames that concurred to generate the average gaze position for a specific speed range Best viewed on screen

Pro

babi

lity

offix

atio

n

road0

01

02

03

04

sidewalk buildings traffic signs trees flowerbed sky vehicles people0

0005

001

cycles

Fig 8 Proportion of semantic categories that fall within the driverrsquos fixation map when thresholded at increasing values (from left to right) Categoriesexhibiting a positive trend (eg road and vehicles) suggest a real attention focus while a negative trend advocates for an awareness of the objectwhich is only circumstantial See Sec 31 for details

conv3D64

conv3D128

conv3D256

conv3D256

conv2D1

pool3D

pool3D

pool3D

up2D

conv3D

64

conv3D conv3D conv3Dpool3D pool3D pool3D

up2D

128 256 256 1

conv2D

Fig 9 The COARSE module is made of an encoder based on C3D network [65] followed by a bilinear upsampling (bringing representations back tothe resolution of the input image) and a final 2D convolution During feature extraction the temporal axis is lost due to 3D pooling All convolutionallayers are preceded by zero paddings in order keep borders and all kernels have size 3 along all dimensions Pooling layers have size and strideof (1 2 2 4) and (2 2 2 1) along temporal and spatial dimensions respectively All activations are ReLUs

range correlations or by recurrent architectures (eg LSTMGRU) that can model longer term dependencies [3] [47] Ourmodel follows the former approach relying on the assumptionthat a small time window (eg half a second) holds sufficientcontextual information for predicting where the driver wouldfocus in that moment Indeed human drivers can take even lesstime to react to an unexpected stimulus Our architecture takes asequence of 16 consecutive frames (asymp 065s) as input (called clipsfrom now on) and predicts the fixation map for the last frame ofsuch clipMany of the architectural choices made to design the networkcome from insights from the dataset analysis presented in Sec31In particular we rely on the following resultsbull the driversrsquo FoA exhibits consistent patterns suggesting that

it can be reproduced by a computational modelbull the driversrsquo gaze is affected by a strong prior on objects

semantics eg drivers tend to focus on items lying on theroad

bull motion cues like vehicle speed are also key factors thatinfluence gaze

Accordingly the model output merges three branches with identi-cal architecture unshared parameters and different input domains

the RGB image the semantic segmentation and the optical flowfield We call this architecture multi-branch model Followinga bottom-up approach in Sec 41 the building blocks of eachbranch are motivated and described Later in Sec 42 it will beshown how the branches merge into the final model

41 Single FoA branch

Each branch of the multi-branch model is a two-input two-output architecture composed of two intertwined streams The aimof this peculiar setup is to prevent the network from learninga central bias that would otherwise stall the learning in earlytraining stages 1 To this end one of the streams is given as input(output) a severely cropped portion of the original image (groundtruth) ensuring a more uniform distribution of the true gaze andruns through the COARSE module described below Similarly theother stream uses the COARSE module to obtain a rough predictionover the full resized image and then refines it through a stack ofadditional convolutions called REFINE model At test time onlythe output of the REFINE stream is considered Both streams rely

1 For further details the reader can refer to Sec 14 and Sec 15 of thesupplementary material

7

last frame

shared weights

Videoclip

CROP

RESIZE

COARSE REFINE outputinput

448

448

cropped videoclip

112

112

112

112

resized videoclip

Fig 10 A single FoA branch of our prediction architecture The COARSE module (see Fig 9) is applied to both a cropped and a resized version ofthe input tensor which is a videoclip of 16 consecutive frames The cropped input is used during training to augment the data and the variety ofground truth fixation maps The prediction of the resized input is stacked with the last frame of the videoclip and fed to a stack of convolutional layers(refinement module) with the aim of refining the prediction Training is performed end-to-end and weights between COARSE modules are shared Attest time only the refined predictions are used Note that the complete model is composed of three of these branches (see Fig 11) each of whichpredicting visual attention for different inputs (namely image optical flow and semantic segmentation) All activations in the refinement module areLeakyReLU with α = 10minus3 except for the last single channel convolution that features ReLUs Crop and resize streams are highlighted by lightblue and orange arrows respectively

on the COARSE module the convolutional backbone (with sharedweights) which provides the rough estimate of the attentional mapcorresponding to a given clip This component is detailed in Fig 9The COARSE module is based on the C3D architecture [65] thatencodes video dynamics by applying a 3D convolutional kernelon the 4D input tensor As opposed to 2D convolutions that stridealong the width and height dimension of the input tensor a 3Dconvolution also strides along time Formally the j-th feature mapin the i-th layer at position (x y) at time t is computed as

vxytij = bij +summ

Piminus1sump=0

Qiminus1sumq=0

Riminus1sumr=0

wpqrijmvx+py+qt+riminus1m (2)

where m indexes different input feature maps wpqrijm is the valueat the position (p q) at time r of the kernel connected to the m-thfeature map and Pi Qi and Ri are the dimensions of the kernelalong width height and temporal axis respectively bij is the biasfrom layer i to layer jFrom C3D only the most general-purpose features are retainedby removing the last convolutional layer and the fully connectedlayers which are strongly linked to the original action recognitiontask The size of the last pooling layer is also modified in order tocover the remaining temporal dimension entirely This collapsesthe tensor from 4D to 3D making the output independent of timeEventually a bilinear upsampling brings the tensor back to theinput spatial resolution and a 2D convolution merges all featuresinto one channel See Fig 9 for additional details on the COARSEmodule

Training the two streams together The architecture of asingle FoA branch is depicted in Fig 10 During training the firststream feeds the COARSE network with random crops forcingthe model to learn the current focus of attention given visualcues rather than prior spatial location The C3D training processdescribed in [65] employs a 128 times 128 image resize and thena 112 times 112 random crop However the small difference in thetwo resolutions limits the variance of gaze position in ground

truth fixation maps and is not sufficient to avoid the attractiontowards the center of the image For this reason training imagesare resized to 256times 256 before being cropped to 112times 112 Thiscrop policy generates samples that cover less than a quarter ofthe original image thus ensuring a sufficient variety in predictiontargets This comes at the cost of a coarser prediction as cropsget smaller the ratio of pixels in the ground truth covered by gazeincreases leading the model to learn larger mapsIn contrast the second stream feeds the same COARSE modelwith the same images this time resized to 112 times 112 ndash and notcropped The coarse prediction obtained from the COARSE modelis then concatenated with the final frame of the input clip iethe frame corresponding to the final prediction Eventually theconcatenated tensor goes through the REFINE module to obtaina higher resolution prediction of the FoAThe overall two-stream training procedure for a single branch issummarized in Algorithm 1

Training objective Prediction cost can be minimized interms of Kullback-Leibler divergence

DKL(Y Y ) =sumi

Y (i) log

(ε+

Y (i)

ε+ Y (i)

)(3)

where Y is the ground truth distribution Y is the prediction thesummation index i spans across image pixels and ε is a smallconstant that ensures numerical stability2 Since each single FoAbranch computes an error on both the cropped image stream andthe resized image stream the branch loss can be defined as

Lb(XbY) =summ

(DKL(φ(Y m)C(φ(Xm

b ))) +

DKL(Y mR(C(ψ(Xmb )) Xm

b )))

) (4)

2 Please note that DKL inputs are always normalized to be a validprobability distribution despite this may be omitted in notation to improveequations readability

8

Algorithm 1 TRAINING The model is trained in two steps first each branch is trained separately through iterations detailed inprocedure SINGLE_BRANCH_TRAINING_ITERATION then the three branches are fine-tuned altogether as shown by procedureMULTI_BRANCH_FINE-TUNING_ITERATION For clarity we omit from notation i) the subscript b denoting the current domain inall X x and y variables in the single branch iteration and ii) the normalization of the sum of the outputs from each branch in line 13

1 procedure A SINGLE_BRANCH_TRAINING_ITERATION

input domain data X = x1 x2 x16 true attentional map y of last frame x16 of videoclip Xoutput branch loss Lb computed on input sample (X y)

2 Xres larr resize(X (112 112))3 Xcrop ycrop larr get_crop((X y) (112 112))4 ycrop larr COARSE(Xcrop) get coarse prediction on uncentered crop5 y larr REFINE(stack(x16 upsample(COARSE(Xres)))) get refined prediction over whole image6 Lb(XY )larr DKL(ycropycrop) +DKL(yy) compute branch loss as in Eq 4

7 procedure B MULTI_BRANCH_FINE-TUNING_ITERATION

input data X = x1 x2 x16 for all domains true attentional map y of last frame x16 of videoclip Xoutput overall loss L computed on input sample (X y)

8 Xres larr resize(X (112 112))9 Xcrop ycrop larr get_crop((X y) (112 112))

10 for branch b isin RGB flow seg do11 ybcrop larr COARSE(Xbcrop ) as in line 4 of the above procedure12 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres )))) as in line 5 of the above procedure13 L(XY )larr DKL(ycrop

sumb ybcrop ) +DKL(y

sumb yb) compute overall loss as in Eq 5

where C and R denote COARSE and REFINE modules(Xm

b Ym) isin Xb times Y is the m-th training example in the

b-th domain (namely RGB optical flow semantic segmentation)and φ and ψ indicate the crop and the resize functions respectively

Inference step While the presence of the C(φ(Xmb )) stream

is beneficial in training to reduce the spatial bias at test timeonly the R(C(ψ(Xm

b )) Xmb )) stream producing higher quality

prediction is used The outputs of such stream from each branch bare then summed together as explained in the following section

42 Multi-Branch modelAs described at the beginning of this section and depicted inFig 11 the multi-branch model is composed of three iden-tical branches The architecture of each branch has already beendescribed in Sec 41 above Each branch exploits complementaryinformation from a different domain and contributes to the finalprediction accordingly In detail the first branch works in the RGBdomain and processes raw visual data about the scene XRGBThe second branch focuses on motion through the optical flowrepresentation Xflow described in [23] Eventually the last branchtakes as input semantic segmentation probability maps Xseg Forthis last branch the number of input channels depends on thespecific algorithm used to extract the results 19 in our setup (Yu

Algorithm 2 INFERENCE At test time the data extracted fromthe resized videoclip is input to the three branches and their outputis summed and normalized to obtain the final FoA prediction

input data X = x1 x2 x16 for all domainsoutput predicted FoA map y1 Xres larr resize(X (112 112))2 for branch b isin RGB flow seg do3 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres ))))4 y larr

sumb yb

sumi

sumb yb(i)

and Koltun [76]) The three independent predicted FoA maps aresummed and normalized to result in a probability distributionTo allow for larger batch size we choose to bootstrap each branchindependently by training it according to Eq 4 Then the completemulti-branch model which merges the three branches is fine-tuned with the following loss

L(X Y) =summ

(DKL(φ(Y m)

sumb

C(φ(Xmb ))) +

DKL(Y msumb

R(C(ψ(Xmb )) Xm

b )))

)

(5)The algorithm describing the complete inference over themulti-branch model in detailed in Alg 2

5 EXPERIMENTS

In this section we evaluate the performance of the proposedmulti-branch model First we start by comparing our modelagainst some baselines and other methods in literature Followingthe guidelines in [12] for the evaluation phase we rely on Pear-sonrsquos Correlation Coefficient (CC) and KullbackndashLeibler Diver-gence (DKL) measures Moreover we evaluate the InformationGain (IG) [35] measure to assess the quality of a predicted mapP with respect to a ground truth map Y in presence of a strongbias as

IG(P YB) =1

N

sumi

Yi[(log2(ε+ Pi)minus log2(ε+Bi)] (6)

where i is an index spanning all the N pixels in the image B thebias computed as the average training fixation map and ε ensuresnumerical stabilityFurthermore we conduct an ablation study to investigate howdifferent branches affect the final prediction and how their mutualinfluence changes in different scenarios We then study whetherour model captures the attention dynamics observed in Sec 31

9

nalprediction

+

FoAfrom motionow

clip

FoAfrom semanticsseg

clip

FoAfrom colorRGB

clip

Fig 11 The multi-branch model is composed of three different branches each of which has its own set of parameters and their predictions aresummed to obtain the final map Note that in this figure cropped streams are dropped to ease representation but are employed during training (asdiscussed in Sec 42 and depicted in Fig 10

Eventually we assess our model from a human perceptionperspective

Implementation details The three different pathways ofthe multi-branch model (namely FoA from color frommotion and from semantics) have been pre-trained independentlyusing the same cropping policy of Sec 42 and minimizing theobjective function in Eq 4 Each branch has been respectively fedwithbull 16 frames clips in raw RGB color spacebull 16 frames clips with optical flow maps encoded as color

images through the flow field encoding [23]bull 16 frames clips holding semantic segmentation from [76]

encoded as 19 scalar activation maps one per segmentationclass

During individual branch pre-training clips were randomly mir-rored for data augmentation We employ Adam optimizer with pa-rameters as suggested in the original paper [32] with the exceptionof the learning rate that we set to 10minus4 Eventually batch size wasfixed to 32 and each branch was trained until convergence TheDR(eye)VE dataset is split into train validation and test set asfollows sequences 1-38 are used for training sequences 39-74 fortesting The 500 frames in the middle of each training sequenceconstitute the validation setMoreover the complete multi-branch architecture was fine-tuned using the same cropping and data augmentation strategiesminimizing cost function in Eq 5 In this phase batch size was setto 4 due to GPU memory constraints and learning rate value waslowered to 10minus5 Inference time of each branch of our architectureis asymp 30 milliseconds per videoclip on an NVIDIA Titan X

51 Model evaluation

In Tab 3 we report results of our proposal against other state-of-the-art models [4] [15] [46] [59] [72] [73] evaluated both onthe complete test set and on acting subsequences only All thecompetitors with the exception of [46] are bottom-up approachesand mainly rely on appearance and motion discontinuities To testthe effectiveness of deep architectures for saliency prediction wecompare against the Multi-Level Network (MLNet) [15] which

scored favourably in the MIT300 saliency benchmark [11] andthe Recurrent Mixture Density Network (RMDN) [4] whichrepresents the only deep model addressing video saliency WhileMLNet works on images discarding the temporal informationRMDN encodes short sequences in a similar way to our COARSEmodule and then relies on a LSTM architecture to model longterm dependencies and estimates the fixation map in terms of aGMM To favor the comparison both models were re-trained onthe DR(eye)VE datasetResults highlight the superiority of our multi-branch archi-tecture on all test sequences The gap in performance with respectto bottom-up unsupervised approaches [72] [73] is higher and ismotivated by the peculiarity of the attention behavior within thedriving context which calls for a task-oriented training procedureMoreover MLNetrsquos low performance testifies for the need of ac-counting for the temporal correlation between consecutive framesthat distinguishes the tasks of attention prediction in images andvideos Indeed RMDN processes video inputs and outperformsMLNet on both DKL and IG metrics performing comparablyon CC Nonetheless its performance is still limited indeedqualitative results reported in Fig 12 suggest that long termdependencies captured by its recurrent module lead the networktowards the regression of the mean discarding contextual and

TABLE 3Experiments illustrating the superior performance of the

multi-branch model over several baselines and competitors Wereport both the average across the complete test sequences and only

the acting frames

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

Baseline Gaussian 040 216 -049 026 241 003Baseline Mean 051 160 000 022 235 000

Mathe et al [59] 004 330 -208 - - -Wang et al [72] 004 340 -221 - - -Wang et al [73] 011 306 -172 - - -

MLNet [15] 044 200 -088 032 235 -036RMDN [4] 041 177 -006 031 213 031

Palazzi et al [46] 055 148 -021 037 200 020multi-branch 056 140 004 041 180 051

10

Input frame GT multi-branch [46] [4] [15]

Fig 12 Qualitative assessment of the predicted fixation maps From left to right input clip ground truth map our prediction prediction of theprevious version of the model [46] prediction of RMDN [4] and prediction of MLNet [15]

frame-specific variations that would be preferrable to keep Tosupport this intuition we measure the average DKL betweenRMDN predictions and the mean training fixation map (Base-line Mean) resulting in a value of 011 Being lower than thedivergence measured with respect to groundtruth maps this valuehighlights the closer correlation to a central baseline rather thanto groundtruth Eventually we also observe improvements withrespect to our previous proposal [46] that relies on a more com-plex backbone model (also including a deconvolutional module)and processes RGB clips only The gap in performance residesin the greater awareness of our multi-branch architecture ofthe aspects that characterize the driving task as emerged fromthe analysis in Sec 31 The positive performances of our modelare also confirmed when evaluated on the acting partition of thedataset We recall that acting indicates sub-sequences exhibiting asignificant task-driven shift of attention from the center of theimage (Fig 5) Being able to predict the FoA also on actingsub-sequences means that the model captures the strong centeredattention bias but is capable of generalizing when required by thecontextThis is further shown by the comparison against a centeredGaussian baseline (BG) and against the average of all trainingset fixation maps (BM) The former baseline has proven effectiveon many image saliency detection tasks [11] while the latterrepresents a more task-driven version The superior performanceof the multi-branch model wrt baselines highlights thatdespite the attention is often strongly biased towards the vanishingpoint of the road the network is able to deal with sudden task-driven changes in gaze direction

52 Model analysis

In this section we investigate the behavior of our proposed modelunder different landscapes time of day and weather (Sec 521)we study the contribution of each branch to the FoA predictiontask (Sec 522) and we compare the learnt attention dynamicsagainst the one observed in the human data (Sec 523)

12

13

14

15

16

17

18

19

Downt

own

Coun

trysid

e

High

way

Mornin

g

Even

ingNi

ght

Sunn

y

Cloud

yRa

iny

DKL

Multi-branchImage

FlowSegm

________________________Landscape ________________________Time of day ________________________Weather

Fig 13 DKL of the different branches in several conditions (from left toright downtown countryside highway morning evening night sunnycloudy rainy) Underlining highlights difference of aggregation in termsof landscape time of day and weather Please note that lower DKL

indicates better predictions

521 Dependency on driving environment

The DR(eye)VE data has been recorded under varying land-scapes time of day and weather conditions We tested our modelin all such different driving conditions As would be expectedFig 13 shows that the human attention is easier to predict inhighways rather than downtown where the focus can shift towardsmore distractors The model seems more reliable in eveningscenarios rather than morning or night where we observed betterlightning conditions and lack of shadows over-exposure and soon Lastly in rainy conditions we notice that human gaze iseasier to model possibly due to the higher level of awarenessdemanded to the driver and his consequent inability to focus awayfrom vanishing point To support the latter intuition we measuredthe performance of BM baseline (ie the average training fixationmap) grouped for weather condition As expected theDKL valuein rainy weather (153) is significantly lower than the ones forcloudy (161) and sunny weather (175) highlighting that whenrainy the driver is more focused on the road

11

|Σ| = 250times 109 |Σ| = 157times 109 |Σ| = 349times 108 |Σ| = 120times 108 |Σ| = 538times 107

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 14 Model prediction averaged across all test sequences and grouped by driving speed As the speed increases the area of the predicted mapshrinks recalling the trend observed in ground truth maps As in Fig 7 for each map a two dimensional Gaussian is fitted and the determinant ofits covariance matrix Σ is reported as a measure of the spread

road sidewalk buildings traffic signs trees flowerbed sky vehicles people cycles10

-4

10-3

10-2

10-1

100

prob

abilit

y of

fixa

tions

(log

sca

le)

Fig 15 Comparison between ground truth (gray bars) and predictedfixation maps (colored bars) when used to mask semantic segmentationof the scene The probability of fixation (in log-scale) for both groundtruth and model prediction is reported for each semantic class Despiteabsolute errors exist the two bar series agree on the relative importanceof different categories

522 Ablation study

In order to validate the design of the multi-branch model(see Sec 42) here we study the individual contributions of thedifferent branches by disabling one or more of themResults in Tab 4 show that the RGB branch plays a majorrole in FoA prediction The motion stream is also beneficialand provides a slight improvement that becomes clearer in theacting subsequences Indeed optical flow intrinsically captures avariety of peculiar scenarios that are non-trivial to classify whenonly color information is provided eg when the car is still at atraffic light or is turning The semantic stream on the other handprovides very little improvement In particular from Tab 4 and by

TABLE 4The ablation study performed on our multi-branch model I F and S

represent image optical flow and semantic segmentation branchesrespectively

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

I 0554 1415 -0008 0403 1826 0458F 0516 1616 -0137 0368 2010 0349S 0479 1699 -0119 0344 2082 0288

I+F 0558 1399 0033 0410 1799 0510I+S 0554 1413 -0001 0404 1823 0466F+S 0528 1571 -0055 0380 1956 0427

I+F+S 0559 1398 0038 0410 1797 0515

specifically comparing I+F and I+F+S a slight increase in the IGmeasure can be appreciated Nevertheless such improvement hasto be considered negligible when compared to color and motionsuggesting that in presence of efficiency concerns or real-timeconstraints the semantic stream can be discarded with little lossesin performance However we expect the benefit from this branchto increase as more accurate segmentation models will be released

523 Do we capture the attention dynamics

The previous sections validate quantitatively the proposed modelNow we assess its capability to attend like a human driverby comparing its predictions against the analysis performed inSec 31First we report the average predicted fixation map in severalspeed ranges in Fig 14 The conclusions we draw are twofoldi) generally the model succeeds in modeling the behavior ofthe driver at different speeds and ii) as the speed increasesfixation maps exhibit lower variance easing the modeling taskand prediction errors decreaseWe also study how often our model focuses on different semanticcategories in a fashion that recalls the analysis of Sec 31 butemploying our predictions rather than ground truth maps as focusof attention More precisely we normalize each map so thatthe maximum value equals 1 and apply the same thresholdingstrategy described in Sec 31 Likewise for each threshold valuea histogram over class labels is built by accounting all pixelsfalling within the binary map for all test frames This results innine histograms over semantic labels that we merge together byaveraging probabilities belonging to different threshold Fig 15shows the comparison Color bars represent how often the pre-dicted map focuses on a certain category while gray bars depictground truth behavior and are obtained by averaging histogramsin Fig 8 across different thresholds Please note that to highlightdifferences for low populated categories values are reported ona logarithmic scale The plot shows a certain degree of absoluteerror is present for all categories However in a broader sense ourmodel replicates the relative weight of different semantic classeswhile driving as testified by the importance of roads and vehiclesthat still dominate against other categories such as people andcycles that are mostly neglected This correlation is confirmed byKendall rank coefficient which scored 051 when computed onthe two bar series

53 Visual assessment of predicted fixation maps

To further validate the predictions of our model from the humanperception perspective 50 people with at least 3 years of driving

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

6

|Σ| = 249times 109 |Σ| = 203times 109 |Σ| = 861times 108 |Σ| = 102times 109 |Σ| = 928times 108

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 7 As speed gradually increases driverrsquos attention converges towards the vanishing point of the road (a) When the car is approximatelystationary the driver is distracted by many objects in the scene (b-e) As the speed increases the driverrsquos gaze deviates less and less from thevanishing point of the road To measure this effect quantitatively a two-dimensional Gaussian is fitted to approximate the mean map for each speedrange and the determinant of the covariance matrix Σ is reported as an indication of its spread (the determinant equals the product of eigenvalueseach of which measures the spread along a different data dimension) The bar plots illustrate the amount of downtown (red) countryside (green)and highway (blue) frames that concurred to generate the average gaze position for a specific speed range Best viewed on screen

Pro

babi

lity

offix

atio

n

road0

01

02

03

04

sidewalk buildings traffic signs trees flowerbed sky vehicles people0

0005

001

cycles

Fig 8 Proportion of semantic categories that fall within the driverrsquos fixation map when thresholded at increasing values (from left to right) Categoriesexhibiting a positive trend (eg road and vehicles) suggest a real attention focus while a negative trend advocates for an awareness of the objectwhich is only circumstantial See Sec 31 for details

conv3D64

conv3D128

conv3D256

conv3D256

conv2D1

pool3D

pool3D

pool3D

up2D

conv3D

64

conv3D conv3D conv3Dpool3D pool3D pool3D

up2D

128 256 256 1

conv2D

Fig 9 The COARSE module is made of an encoder based on C3D network [65] followed by a bilinear upsampling (bringing representations back tothe resolution of the input image) and a final 2D convolution During feature extraction the temporal axis is lost due to 3D pooling All convolutionallayers are preceded by zero paddings in order keep borders and all kernels have size 3 along all dimensions Pooling layers have size and strideof (1 2 2 4) and (2 2 2 1) along temporal and spatial dimensions respectively All activations are ReLUs

range correlations or by recurrent architectures (eg LSTMGRU) that can model longer term dependencies [3] [47] Ourmodel follows the former approach relying on the assumptionthat a small time window (eg half a second) holds sufficientcontextual information for predicting where the driver wouldfocus in that moment Indeed human drivers can take even lesstime to react to an unexpected stimulus Our architecture takes asequence of 16 consecutive frames (asymp 065s) as input (called clipsfrom now on) and predicts the fixation map for the last frame ofsuch clipMany of the architectural choices made to design the networkcome from insights from the dataset analysis presented in Sec31In particular we rely on the following resultsbull the driversrsquo FoA exhibits consistent patterns suggesting that

it can be reproduced by a computational modelbull the driversrsquo gaze is affected by a strong prior on objects

semantics eg drivers tend to focus on items lying on theroad

bull motion cues like vehicle speed are also key factors thatinfluence gaze

Accordingly the model output merges three branches with identi-cal architecture unshared parameters and different input domains

the RGB image the semantic segmentation and the optical flowfield We call this architecture multi-branch model Followinga bottom-up approach in Sec 41 the building blocks of eachbranch are motivated and described Later in Sec 42 it will beshown how the branches merge into the final model

41 Single FoA branch

Each branch of the multi-branch model is a two-input two-output architecture composed of two intertwined streams The aimof this peculiar setup is to prevent the network from learninga central bias that would otherwise stall the learning in earlytraining stages 1 To this end one of the streams is given as input(output) a severely cropped portion of the original image (groundtruth) ensuring a more uniform distribution of the true gaze andruns through the COARSE module described below Similarly theother stream uses the COARSE module to obtain a rough predictionover the full resized image and then refines it through a stack ofadditional convolutions called REFINE model At test time onlythe output of the REFINE stream is considered Both streams rely

1 For further details the reader can refer to Sec 14 and Sec 15 of thesupplementary material

7

last frame

shared weights

Videoclip

CROP

RESIZE

COARSE REFINE outputinput

448

448

cropped videoclip

112

112

112

112

resized videoclip

Fig 10 A single FoA branch of our prediction architecture The COARSE module (see Fig 9) is applied to both a cropped and a resized version ofthe input tensor which is a videoclip of 16 consecutive frames The cropped input is used during training to augment the data and the variety ofground truth fixation maps The prediction of the resized input is stacked with the last frame of the videoclip and fed to a stack of convolutional layers(refinement module) with the aim of refining the prediction Training is performed end-to-end and weights between COARSE modules are shared Attest time only the refined predictions are used Note that the complete model is composed of three of these branches (see Fig 11) each of whichpredicting visual attention for different inputs (namely image optical flow and semantic segmentation) All activations in the refinement module areLeakyReLU with α = 10minus3 except for the last single channel convolution that features ReLUs Crop and resize streams are highlighted by lightblue and orange arrows respectively

on the COARSE module the convolutional backbone (with sharedweights) which provides the rough estimate of the attentional mapcorresponding to a given clip This component is detailed in Fig 9The COARSE module is based on the C3D architecture [65] thatencodes video dynamics by applying a 3D convolutional kernelon the 4D input tensor As opposed to 2D convolutions that stridealong the width and height dimension of the input tensor a 3Dconvolution also strides along time Formally the j-th feature mapin the i-th layer at position (x y) at time t is computed as

vxytij = bij +summ

Piminus1sump=0

Qiminus1sumq=0

Riminus1sumr=0

wpqrijmvx+py+qt+riminus1m (2)

where m indexes different input feature maps wpqrijm is the valueat the position (p q) at time r of the kernel connected to the m-thfeature map and Pi Qi and Ri are the dimensions of the kernelalong width height and temporal axis respectively bij is the biasfrom layer i to layer jFrom C3D only the most general-purpose features are retainedby removing the last convolutional layer and the fully connectedlayers which are strongly linked to the original action recognitiontask The size of the last pooling layer is also modified in order tocover the remaining temporal dimension entirely This collapsesthe tensor from 4D to 3D making the output independent of timeEventually a bilinear upsampling brings the tensor back to theinput spatial resolution and a 2D convolution merges all featuresinto one channel See Fig 9 for additional details on the COARSEmodule

Training the two streams together The architecture of asingle FoA branch is depicted in Fig 10 During training the firststream feeds the COARSE network with random crops forcingthe model to learn the current focus of attention given visualcues rather than prior spatial location The C3D training processdescribed in [65] employs a 128 times 128 image resize and thena 112 times 112 random crop However the small difference in thetwo resolutions limits the variance of gaze position in ground

truth fixation maps and is not sufficient to avoid the attractiontowards the center of the image For this reason training imagesare resized to 256times 256 before being cropped to 112times 112 Thiscrop policy generates samples that cover less than a quarter ofthe original image thus ensuring a sufficient variety in predictiontargets This comes at the cost of a coarser prediction as cropsget smaller the ratio of pixels in the ground truth covered by gazeincreases leading the model to learn larger mapsIn contrast the second stream feeds the same COARSE modelwith the same images this time resized to 112 times 112 ndash and notcropped The coarse prediction obtained from the COARSE modelis then concatenated with the final frame of the input clip iethe frame corresponding to the final prediction Eventually theconcatenated tensor goes through the REFINE module to obtaina higher resolution prediction of the FoAThe overall two-stream training procedure for a single branch issummarized in Algorithm 1

Training objective Prediction cost can be minimized interms of Kullback-Leibler divergence

DKL(Y Y ) =sumi

Y (i) log

(ε+

Y (i)

ε+ Y (i)

)(3)

where Y is the ground truth distribution Y is the prediction thesummation index i spans across image pixels and ε is a smallconstant that ensures numerical stability2 Since each single FoAbranch computes an error on both the cropped image stream andthe resized image stream the branch loss can be defined as

Lb(XbY) =summ

(DKL(φ(Y m)C(φ(Xm

b ))) +

DKL(Y mR(C(ψ(Xmb )) Xm

b )))

) (4)

2 Please note that DKL inputs are always normalized to be a validprobability distribution despite this may be omitted in notation to improveequations readability

8

Algorithm 1 TRAINING The model is trained in two steps first each branch is trained separately through iterations detailed inprocedure SINGLE_BRANCH_TRAINING_ITERATION then the three branches are fine-tuned altogether as shown by procedureMULTI_BRANCH_FINE-TUNING_ITERATION For clarity we omit from notation i) the subscript b denoting the current domain inall X x and y variables in the single branch iteration and ii) the normalization of the sum of the outputs from each branch in line 13

1 procedure A SINGLE_BRANCH_TRAINING_ITERATION

input domain data X = x1 x2 x16 true attentional map y of last frame x16 of videoclip Xoutput branch loss Lb computed on input sample (X y)

2 Xres larr resize(X (112 112))3 Xcrop ycrop larr get_crop((X y) (112 112))4 ycrop larr COARSE(Xcrop) get coarse prediction on uncentered crop5 y larr REFINE(stack(x16 upsample(COARSE(Xres)))) get refined prediction over whole image6 Lb(XY )larr DKL(ycropycrop) +DKL(yy) compute branch loss as in Eq 4

7 procedure B MULTI_BRANCH_FINE-TUNING_ITERATION

input data X = x1 x2 x16 for all domains true attentional map y of last frame x16 of videoclip Xoutput overall loss L computed on input sample (X y)

8 Xres larr resize(X (112 112))9 Xcrop ycrop larr get_crop((X y) (112 112))

10 for branch b isin RGB flow seg do11 ybcrop larr COARSE(Xbcrop ) as in line 4 of the above procedure12 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres )))) as in line 5 of the above procedure13 L(XY )larr DKL(ycrop

sumb ybcrop ) +DKL(y

sumb yb) compute overall loss as in Eq 5

where C and R denote COARSE and REFINE modules(Xm

b Ym) isin Xb times Y is the m-th training example in the

b-th domain (namely RGB optical flow semantic segmentation)and φ and ψ indicate the crop and the resize functions respectively

Inference step While the presence of the C(φ(Xmb )) stream

is beneficial in training to reduce the spatial bias at test timeonly the R(C(ψ(Xm

b )) Xmb )) stream producing higher quality

prediction is used The outputs of such stream from each branch bare then summed together as explained in the following section

42 Multi-Branch modelAs described at the beginning of this section and depicted inFig 11 the multi-branch model is composed of three iden-tical branches The architecture of each branch has already beendescribed in Sec 41 above Each branch exploits complementaryinformation from a different domain and contributes to the finalprediction accordingly In detail the first branch works in the RGBdomain and processes raw visual data about the scene XRGBThe second branch focuses on motion through the optical flowrepresentation Xflow described in [23] Eventually the last branchtakes as input semantic segmentation probability maps Xseg Forthis last branch the number of input channels depends on thespecific algorithm used to extract the results 19 in our setup (Yu

Algorithm 2 INFERENCE At test time the data extracted fromthe resized videoclip is input to the three branches and their outputis summed and normalized to obtain the final FoA prediction

input data X = x1 x2 x16 for all domainsoutput predicted FoA map y1 Xres larr resize(X (112 112))2 for branch b isin RGB flow seg do3 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres ))))4 y larr

sumb yb

sumi

sumb yb(i)

and Koltun [76]) The three independent predicted FoA maps aresummed and normalized to result in a probability distributionTo allow for larger batch size we choose to bootstrap each branchindependently by training it according to Eq 4 Then the completemulti-branch model which merges the three branches is fine-tuned with the following loss

L(X Y) =summ

(DKL(φ(Y m)

sumb

C(φ(Xmb ))) +

DKL(Y msumb

R(C(ψ(Xmb )) Xm

b )))

)

(5)The algorithm describing the complete inference over themulti-branch model in detailed in Alg 2

5 EXPERIMENTS

In this section we evaluate the performance of the proposedmulti-branch model First we start by comparing our modelagainst some baselines and other methods in literature Followingthe guidelines in [12] for the evaluation phase we rely on Pear-sonrsquos Correlation Coefficient (CC) and KullbackndashLeibler Diver-gence (DKL) measures Moreover we evaluate the InformationGain (IG) [35] measure to assess the quality of a predicted mapP with respect to a ground truth map Y in presence of a strongbias as

IG(P YB) =1

N

sumi

Yi[(log2(ε+ Pi)minus log2(ε+Bi)] (6)

where i is an index spanning all the N pixels in the image B thebias computed as the average training fixation map and ε ensuresnumerical stabilityFurthermore we conduct an ablation study to investigate howdifferent branches affect the final prediction and how their mutualinfluence changes in different scenarios We then study whetherour model captures the attention dynamics observed in Sec 31

9

nalprediction

+

FoAfrom motionow

clip

FoAfrom semanticsseg

clip

FoAfrom colorRGB

clip

Fig 11 The multi-branch model is composed of three different branches each of which has its own set of parameters and their predictions aresummed to obtain the final map Note that in this figure cropped streams are dropped to ease representation but are employed during training (asdiscussed in Sec 42 and depicted in Fig 10

Eventually we assess our model from a human perceptionperspective

Implementation details The three different pathways ofthe multi-branch model (namely FoA from color frommotion and from semantics) have been pre-trained independentlyusing the same cropping policy of Sec 42 and minimizing theobjective function in Eq 4 Each branch has been respectively fedwithbull 16 frames clips in raw RGB color spacebull 16 frames clips with optical flow maps encoded as color

images through the flow field encoding [23]bull 16 frames clips holding semantic segmentation from [76]

encoded as 19 scalar activation maps one per segmentationclass

During individual branch pre-training clips were randomly mir-rored for data augmentation We employ Adam optimizer with pa-rameters as suggested in the original paper [32] with the exceptionof the learning rate that we set to 10minus4 Eventually batch size wasfixed to 32 and each branch was trained until convergence TheDR(eye)VE dataset is split into train validation and test set asfollows sequences 1-38 are used for training sequences 39-74 fortesting The 500 frames in the middle of each training sequenceconstitute the validation setMoreover the complete multi-branch architecture was fine-tuned using the same cropping and data augmentation strategiesminimizing cost function in Eq 5 In this phase batch size was setto 4 due to GPU memory constraints and learning rate value waslowered to 10minus5 Inference time of each branch of our architectureis asymp 30 milliseconds per videoclip on an NVIDIA Titan X

51 Model evaluation

In Tab 3 we report results of our proposal against other state-of-the-art models [4] [15] [46] [59] [72] [73] evaluated both onthe complete test set and on acting subsequences only All thecompetitors with the exception of [46] are bottom-up approachesand mainly rely on appearance and motion discontinuities To testthe effectiveness of deep architectures for saliency prediction wecompare against the Multi-Level Network (MLNet) [15] which

scored favourably in the MIT300 saliency benchmark [11] andthe Recurrent Mixture Density Network (RMDN) [4] whichrepresents the only deep model addressing video saliency WhileMLNet works on images discarding the temporal informationRMDN encodes short sequences in a similar way to our COARSEmodule and then relies on a LSTM architecture to model longterm dependencies and estimates the fixation map in terms of aGMM To favor the comparison both models were re-trained onthe DR(eye)VE datasetResults highlight the superiority of our multi-branch archi-tecture on all test sequences The gap in performance with respectto bottom-up unsupervised approaches [72] [73] is higher and ismotivated by the peculiarity of the attention behavior within thedriving context which calls for a task-oriented training procedureMoreover MLNetrsquos low performance testifies for the need of ac-counting for the temporal correlation between consecutive framesthat distinguishes the tasks of attention prediction in images andvideos Indeed RMDN processes video inputs and outperformsMLNet on both DKL and IG metrics performing comparablyon CC Nonetheless its performance is still limited indeedqualitative results reported in Fig 12 suggest that long termdependencies captured by its recurrent module lead the networktowards the regression of the mean discarding contextual and

TABLE 3Experiments illustrating the superior performance of the

multi-branch model over several baselines and competitors Wereport both the average across the complete test sequences and only

the acting frames

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

Baseline Gaussian 040 216 -049 026 241 003Baseline Mean 051 160 000 022 235 000

Mathe et al [59] 004 330 -208 - - -Wang et al [72] 004 340 -221 - - -Wang et al [73] 011 306 -172 - - -

MLNet [15] 044 200 -088 032 235 -036RMDN [4] 041 177 -006 031 213 031

Palazzi et al [46] 055 148 -021 037 200 020multi-branch 056 140 004 041 180 051

10

Input frame GT multi-branch [46] [4] [15]

Fig 12 Qualitative assessment of the predicted fixation maps From left to right input clip ground truth map our prediction prediction of theprevious version of the model [46] prediction of RMDN [4] and prediction of MLNet [15]

frame-specific variations that would be preferrable to keep Tosupport this intuition we measure the average DKL betweenRMDN predictions and the mean training fixation map (Base-line Mean) resulting in a value of 011 Being lower than thedivergence measured with respect to groundtruth maps this valuehighlights the closer correlation to a central baseline rather thanto groundtruth Eventually we also observe improvements withrespect to our previous proposal [46] that relies on a more com-plex backbone model (also including a deconvolutional module)and processes RGB clips only The gap in performance residesin the greater awareness of our multi-branch architecture ofthe aspects that characterize the driving task as emerged fromthe analysis in Sec 31 The positive performances of our modelare also confirmed when evaluated on the acting partition of thedataset We recall that acting indicates sub-sequences exhibiting asignificant task-driven shift of attention from the center of theimage (Fig 5) Being able to predict the FoA also on actingsub-sequences means that the model captures the strong centeredattention bias but is capable of generalizing when required by thecontextThis is further shown by the comparison against a centeredGaussian baseline (BG) and against the average of all trainingset fixation maps (BM) The former baseline has proven effectiveon many image saliency detection tasks [11] while the latterrepresents a more task-driven version The superior performanceof the multi-branch model wrt baselines highlights thatdespite the attention is often strongly biased towards the vanishingpoint of the road the network is able to deal with sudden task-driven changes in gaze direction

52 Model analysis

In this section we investigate the behavior of our proposed modelunder different landscapes time of day and weather (Sec 521)we study the contribution of each branch to the FoA predictiontask (Sec 522) and we compare the learnt attention dynamicsagainst the one observed in the human data (Sec 523)

12

13

14

15

16

17

18

19

Downt

own

Coun

trysid

e

High

way

Mornin

g

Even

ingNi

ght

Sunn

y

Cloud

yRa

iny

DKL

Multi-branchImage

FlowSegm

________________________Landscape ________________________Time of day ________________________Weather

Fig 13 DKL of the different branches in several conditions (from left toright downtown countryside highway morning evening night sunnycloudy rainy) Underlining highlights difference of aggregation in termsof landscape time of day and weather Please note that lower DKL

indicates better predictions

521 Dependency on driving environment

The DR(eye)VE data has been recorded under varying land-scapes time of day and weather conditions We tested our modelin all such different driving conditions As would be expectedFig 13 shows that the human attention is easier to predict inhighways rather than downtown where the focus can shift towardsmore distractors The model seems more reliable in eveningscenarios rather than morning or night where we observed betterlightning conditions and lack of shadows over-exposure and soon Lastly in rainy conditions we notice that human gaze iseasier to model possibly due to the higher level of awarenessdemanded to the driver and his consequent inability to focus awayfrom vanishing point To support the latter intuition we measuredthe performance of BM baseline (ie the average training fixationmap) grouped for weather condition As expected theDKL valuein rainy weather (153) is significantly lower than the ones forcloudy (161) and sunny weather (175) highlighting that whenrainy the driver is more focused on the road

11

|Σ| = 250times 109 |Σ| = 157times 109 |Σ| = 349times 108 |Σ| = 120times 108 |Σ| = 538times 107

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 14 Model prediction averaged across all test sequences and grouped by driving speed As the speed increases the area of the predicted mapshrinks recalling the trend observed in ground truth maps As in Fig 7 for each map a two dimensional Gaussian is fitted and the determinant ofits covariance matrix Σ is reported as a measure of the spread

road sidewalk buildings traffic signs trees flowerbed sky vehicles people cycles10

-4

10-3

10-2

10-1

100

prob

abilit

y of

fixa

tions

(log

sca

le)

Fig 15 Comparison between ground truth (gray bars) and predictedfixation maps (colored bars) when used to mask semantic segmentationof the scene The probability of fixation (in log-scale) for both groundtruth and model prediction is reported for each semantic class Despiteabsolute errors exist the two bar series agree on the relative importanceof different categories

522 Ablation study

In order to validate the design of the multi-branch model(see Sec 42) here we study the individual contributions of thedifferent branches by disabling one or more of themResults in Tab 4 show that the RGB branch plays a majorrole in FoA prediction The motion stream is also beneficialand provides a slight improvement that becomes clearer in theacting subsequences Indeed optical flow intrinsically captures avariety of peculiar scenarios that are non-trivial to classify whenonly color information is provided eg when the car is still at atraffic light or is turning The semantic stream on the other handprovides very little improvement In particular from Tab 4 and by

TABLE 4The ablation study performed on our multi-branch model I F and S

represent image optical flow and semantic segmentation branchesrespectively

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

I 0554 1415 -0008 0403 1826 0458F 0516 1616 -0137 0368 2010 0349S 0479 1699 -0119 0344 2082 0288

I+F 0558 1399 0033 0410 1799 0510I+S 0554 1413 -0001 0404 1823 0466F+S 0528 1571 -0055 0380 1956 0427

I+F+S 0559 1398 0038 0410 1797 0515

specifically comparing I+F and I+F+S a slight increase in the IGmeasure can be appreciated Nevertheless such improvement hasto be considered negligible when compared to color and motionsuggesting that in presence of efficiency concerns or real-timeconstraints the semantic stream can be discarded with little lossesin performance However we expect the benefit from this branchto increase as more accurate segmentation models will be released

523 Do we capture the attention dynamics

The previous sections validate quantitatively the proposed modelNow we assess its capability to attend like a human driverby comparing its predictions against the analysis performed inSec 31First we report the average predicted fixation map in severalspeed ranges in Fig 14 The conclusions we draw are twofoldi) generally the model succeeds in modeling the behavior ofthe driver at different speeds and ii) as the speed increasesfixation maps exhibit lower variance easing the modeling taskand prediction errors decreaseWe also study how often our model focuses on different semanticcategories in a fashion that recalls the analysis of Sec 31 butemploying our predictions rather than ground truth maps as focusof attention More precisely we normalize each map so thatthe maximum value equals 1 and apply the same thresholdingstrategy described in Sec 31 Likewise for each threshold valuea histogram over class labels is built by accounting all pixelsfalling within the binary map for all test frames This results innine histograms over semantic labels that we merge together byaveraging probabilities belonging to different threshold Fig 15shows the comparison Color bars represent how often the pre-dicted map focuses on a certain category while gray bars depictground truth behavior and are obtained by averaging histogramsin Fig 8 across different thresholds Please note that to highlightdifferences for low populated categories values are reported ona logarithmic scale The plot shows a certain degree of absoluteerror is present for all categories However in a broader sense ourmodel replicates the relative weight of different semantic classeswhile driving as testified by the importance of roads and vehiclesthat still dominate against other categories such as people andcycles that are mostly neglected This correlation is confirmed byKendall rank coefficient which scored 051 when computed onthe two bar series

53 Visual assessment of predicted fixation maps

To further validate the predictions of our model from the humanperception perspective 50 people with at least 3 years of driving

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

7

last frame

shared weights

Videoclip

CROP

RESIZE

COARSE REFINE outputinput

448

448

cropped videoclip

112

112

112

112

resized videoclip

Fig 10 A single FoA branch of our prediction architecture The COARSE module (see Fig 9) is applied to both a cropped and a resized version ofthe input tensor which is a videoclip of 16 consecutive frames The cropped input is used during training to augment the data and the variety ofground truth fixation maps The prediction of the resized input is stacked with the last frame of the videoclip and fed to a stack of convolutional layers(refinement module) with the aim of refining the prediction Training is performed end-to-end and weights between COARSE modules are shared Attest time only the refined predictions are used Note that the complete model is composed of three of these branches (see Fig 11) each of whichpredicting visual attention for different inputs (namely image optical flow and semantic segmentation) All activations in the refinement module areLeakyReLU with α = 10minus3 except for the last single channel convolution that features ReLUs Crop and resize streams are highlighted by lightblue and orange arrows respectively

on the COARSE module the convolutional backbone (with sharedweights) which provides the rough estimate of the attentional mapcorresponding to a given clip This component is detailed in Fig 9The COARSE module is based on the C3D architecture [65] thatencodes video dynamics by applying a 3D convolutional kernelon the 4D input tensor As opposed to 2D convolutions that stridealong the width and height dimension of the input tensor a 3Dconvolution also strides along time Formally the j-th feature mapin the i-th layer at position (x y) at time t is computed as

vxytij = bij +summ

Piminus1sump=0

Qiminus1sumq=0

Riminus1sumr=0

wpqrijmvx+py+qt+riminus1m (2)

where m indexes different input feature maps wpqrijm is the valueat the position (p q) at time r of the kernel connected to the m-thfeature map and Pi Qi and Ri are the dimensions of the kernelalong width height and temporal axis respectively bij is the biasfrom layer i to layer jFrom C3D only the most general-purpose features are retainedby removing the last convolutional layer and the fully connectedlayers which are strongly linked to the original action recognitiontask The size of the last pooling layer is also modified in order tocover the remaining temporal dimension entirely This collapsesthe tensor from 4D to 3D making the output independent of timeEventually a bilinear upsampling brings the tensor back to theinput spatial resolution and a 2D convolution merges all featuresinto one channel See Fig 9 for additional details on the COARSEmodule

Training the two streams together The architecture of asingle FoA branch is depicted in Fig 10 During training the firststream feeds the COARSE network with random crops forcingthe model to learn the current focus of attention given visualcues rather than prior spatial location The C3D training processdescribed in [65] employs a 128 times 128 image resize and thena 112 times 112 random crop However the small difference in thetwo resolutions limits the variance of gaze position in ground

truth fixation maps and is not sufficient to avoid the attractiontowards the center of the image For this reason training imagesare resized to 256times 256 before being cropped to 112times 112 Thiscrop policy generates samples that cover less than a quarter ofthe original image thus ensuring a sufficient variety in predictiontargets This comes at the cost of a coarser prediction as cropsget smaller the ratio of pixels in the ground truth covered by gazeincreases leading the model to learn larger mapsIn contrast the second stream feeds the same COARSE modelwith the same images this time resized to 112 times 112 ndash and notcropped The coarse prediction obtained from the COARSE modelis then concatenated with the final frame of the input clip iethe frame corresponding to the final prediction Eventually theconcatenated tensor goes through the REFINE module to obtaina higher resolution prediction of the FoAThe overall two-stream training procedure for a single branch issummarized in Algorithm 1

Training objective Prediction cost can be minimized interms of Kullback-Leibler divergence

DKL(Y Y ) =sumi

Y (i) log

(ε+

Y (i)

ε+ Y (i)

)(3)

where Y is the ground truth distribution Y is the prediction thesummation index i spans across image pixels and ε is a smallconstant that ensures numerical stability2 Since each single FoAbranch computes an error on both the cropped image stream andthe resized image stream the branch loss can be defined as

Lb(XbY) =summ

(DKL(φ(Y m)C(φ(Xm

b ))) +

DKL(Y mR(C(ψ(Xmb )) Xm

b )))

) (4)

2 Please note that DKL inputs are always normalized to be a validprobability distribution despite this may be omitted in notation to improveequations readability

8

Algorithm 1 TRAINING The model is trained in two steps first each branch is trained separately through iterations detailed inprocedure SINGLE_BRANCH_TRAINING_ITERATION then the three branches are fine-tuned altogether as shown by procedureMULTI_BRANCH_FINE-TUNING_ITERATION For clarity we omit from notation i) the subscript b denoting the current domain inall X x and y variables in the single branch iteration and ii) the normalization of the sum of the outputs from each branch in line 13

1 procedure A SINGLE_BRANCH_TRAINING_ITERATION

input domain data X = x1 x2 x16 true attentional map y of last frame x16 of videoclip Xoutput branch loss Lb computed on input sample (X y)

2 Xres larr resize(X (112 112))3 Xcrop ycrop larr get_crop((X y) (112 112))4 ycrop larr COARSE(Xcrop) get coarse prediction on uncentered crop5 y larr REFINE(stack(x16 upsample(COARSE(Xres)))) get refined prediction over whole image6 Lb(XY )larr DKL(ycropycrop) +DKL(yy) compute branch loss as in Eq 4

7 procedure B MULTI_BRANCH_FINE-TUNING_ITERATION

input data X = x1 x2 x16 for all domains true attentional map y of last frame x16 of videoclip Xoutput overall loss L computed on input sample (X y)

8 Xres larr resize(X (112 112))9 Xcrop ycrop larr get_crop((X y) (112 112))

10 for branch b isin RGB flow seg do11 ybcrop larr COARSE(Xbcrop ) as in line 4 of the above procedure12 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres )))) as in line 5 of the above procedure13 L(XY )larr DKL(ycrop

sumb ybcrop ) +DKL(y

sumb yb) compute overall loss as in Eq 5

where C and R denote COARSE and REFINE modules(Xm

b Ym) isin Xb times Y is the m-th training example in the

b-th domain (namely RGB optical flow semantic segmentation)and φ and ψ indicate the crop and the resize functions respectively

Inference step While the presence of the C(φ(Xmb )) stream

is beneficial in training to reduce the spatial bias at test timeonly the R(C(ψ(Xm

b )) Xmb )) stream producing higher quality

prediction is used The outputs of such stream from each branch bare then summed together as explained in the following section

42 Multi-Branch modelAs described at the beginning of this section and depicted inFig 11 the multi-branch model is composed of three iden-tical branches The architecture of each branch has already beendescribed in Sec 41 above Each branch exploits complementaryinformation from a different domain and contributes to the finalprediction accordingly In detail the first branch works in the RGBdomain and processes raw visual data about the scene XRGBThe second branch focuses on motion through the optical flowrepresentation Xflow described in [23] Eventually the last branchtakes as input semantic segmentation probability maps Xseg Forthis last branch the number of input channels depends on thespecific algorithm used to extract the results 19 in our setup (Yu

Algorithm 2 INFERENCE At test time the data extracted fromthe resized videoclip is input to the three branches and their outputis summed and normalized to obtain the final FoA prediction

input data X = x1 x2 x16 for all domainsoutput predicted FoA map y1 Xres larr resize(X (112 112))2 for branch b isin RGB flow seg do3 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres ))))4 y larr

sumb yb

sumi

sumb yb(i)

and Koltun [76]) The three independent predicted FoA maps aresummed and normalized to result in a probability distributionTo allow for larger batch size we choose to bootstrap each branchindependently by training it according to Eq 4 Then the completemulti-branch model which merges the three branches is fine-tuned with the following loss

L(X Y) =summ

(DKL(φ(Y m)

sumb

C(φ(Xmb ))) +

DKL(Y msumb

R(C(ψ(Xmb )) Xm

b )))

)

(5)The algorithm describing the complete inference over themulti-branch model in detailed in Alg 2

5 EXPERIMENTS

In this section we evaluate the performance of the proposedmulti-branch model First we start by comparing our modelagainst some baselines and other methods in literature Followingthe guidelines in [12] for the evaluation phase we rely on Pear-sonrsquos Correlation Coefficient (CC) and KullbackndashLeibler Diver-gence (DKL) measures Moreover we evaluate the InformationGain (IG) [35] measure to assess the quality of a predicted mapP with respect to a ground truth map Y in presence of a strongbias as

IG(P YB) =1

N

sumi

Yi[(log2(ε+ Pi)minus log2(ε+Bi)] (6)

where i is an index spanning all the N pixels in the image B thebias computed as the average training fixation map and ε ensuresnumerical stabilityFurthermore we conduct an ablation study to investigate howdifferent branches affect the final prediction and how their mutualinfluence changes in different scenarios We then study whetherour model captures the attention dynamics observed in Sec 31

9

nalprediction

+

FoAfrom motionow

clip

FoAfrom semanticsseg

clip

FoAfrom colorRGB

clip

Fig 11 The multi-branch model is composed of three different branches each of which has its own set of parameters and their predictions aresummed to obtain the final map Note that in this figure cropped streams are dropped to ease representation but are employed during training (asdiscussed in Sec 42 and depicted in Fig 10

Eventually we assess our model from a human perceptionperspective

Implementation details The three different pathways ofthe multi-branch model (namely FoA from color frommotion and from semantics) have been pre-trained independentlyusing the same cropping policy of Sec 42 and minimizing theobjective function in Eq 4 Each branch has been respectively fedwithbull 16 frames clips in raw RGB color spacebull 16 frames clips with optical flow maps encoded as color

images through the flow field encoding [23]bull 16 frames clips holding semantic segmentation from [76]

encoded as 19 scalar activation maps one per segmentationclass

During individual branch pre-training clips were randomly mir-rored for data augmentation We employ Adam optimizer with pa-rameters as suggested in the original paper [32] with the exceptionof the learning rate that we set to 10minus4 Eventually batch size wasfixed to 32 and each branch was trained until convergence TheDR(eye)VE dataset is split into train validation and test set asfollows sequences 1-38 are used for training sequences 39-74 fortesting The 500 frames in the middle of each training sequenceconstitute the validation setMoreover the complete multi-branch architecture was fine-tuned using the same cropping and data augmentation strategiesminimizing cost function in Eq 5 In this phase batch size was setto 4 due to GPU memory constraints and learning rate value waslowered to 10minus5 Inference time of each branch of our architectureis asymp 30 milliseconds per videoclip on an NVIDIA Titan X

51 Model evaluation

In Tab 3 we report results of our proposal against other state-of-the-art models [4] [15] [46] [59] [72] [73] evaluated both onthe complete test set and on acting subsequences only All thecompetitors with the exception of [46] are bottom-up approachesand mainly rely on appearance and motion discontinuities To testthe effectiveness of deep architectures for saliency prediction wecompare against the Multi-Level Network (MLNet) [15] which

scored favourably in the MIT300 saliency benchmark [11] andthe Recurrent Mixture Density Network (RMDN) [4] whichrepresents the only deep model addressing video saliency WhileMLNet works on images discarding the temporal informationRMDN encodes short sequences in a similar way to our COARSEmodule and then relies on a LSTM architecture to model longterm dependencies and estimates the fixation map in terms of aGMM To favor the comparison both models were re-trained onthe DR(eye)VE datasetResults highlight the superiority of our multi-branch archi-tecture on all test sequences The gap in performance with respectto bottom-up unsupervised approaches [72] [73] is higher and ismotivated by the peculiarity of the attention behavior within thedriving context which calls for a task-oriented training procedureMoreover MLNetrsquos low performance testifies for the need of ac-counting for the temporal correlation between consecutive framesthat distinguishes the tasks of attention prediction in images andvideos Indeed RMDN processes video inputs and outperformsMLNet on both DKL and IG metrics performing comparablyon CC Nonetheless its performance is still limited indeedqualitative results reported in Fig 12 suggest that long termdependencies captured by its recurrent module lead the networktowards the regression of the mean discarding contextual and

TABLE 3Experiments illustrating the superior performance of the

multi-branch model over several baselines and competitors Wereport both the average across the complete test sequences and only

the acting frames

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

Baseline Gaussian 040 216 -049 026 241 003Baseline Mean 051 160 000 022 235 000

Mathe et al [59] 004 330 -208 - - -Wang et al [72] 004 340 -221 - - -Wang et al [73] 011 306 -172 - - -

MLNet [15] 044 200 -088 032 235 -036RMDN [4] 041 177 -006 031 213 031

Palazzi et al [46] 055 148 -021 037 200 020multi-branch 056 140 004 041 180 051

10

Input frame GT multi-branch [46] [4] [15]

Fig 12 Qualitative assessment of the predicted fixation maps From left to right input clip ground truth map our prediction prediction of theprevious version of the model [46] prediction of RMDN [4] and prediction of MLNet [15]

frame-specific variations that would be preferrable to keep Tosupport this intuition we measure the average DKL betweenRMDN predictions and the mean training fixation map (Base-line Mean) resulting in a value of 011 Being lower than thedivergence measured with respect to groundtruth maps this valuehighlights the closer correlation to a central baseline rather thanto groundtruth Eventually we also observe improvements withrespect to our previous proposal [46] that relies on a more com-plex backbone model (also including a deconvolutional module)and processes RGB clips only The gap in performance residesin the greater awareness of our multi-branch architecture ofthe aspects that characterize the driving task as emerged fromthe analysis in Sec 31 The positive performances of our modelare also confirmed when evaluated on the acting partition of thedataset We recall that acting indicates sub-sequences exhibiting asignificant task-driven shift of attention from the center of theimage (Fig 5) Being able to predict the FoA also on actingsub-sequences means that the model captures the strong centeredattention bias but is capable of generalizing when required by thecontextThis is further shown by the comparison against a centeredGaussian baseline (BG) and against the average of all trainingset fixation maps (BM) The former baseline has proven effectiveon many image saliency detection tasks [11] while the latterrepresents a more task-driven version The superior performanceof the multi-branch model wrt baselines highlights thatdespite the attention is often strongly biased towards the vanishingpoint of the road the network is able to deal with sudden task-driven changes in gaze direction

52 Model analysis

In this section we investigate the behavior of our proposed modelunder different landscapes time of day and weather (Sec 521)we study the contribution of each branch to the FoA predictiontask (Sec 522) and we compare the learnt attention dynamicsagainst the one observed in the human data (Sec 523)

12

13

14

15

16

17

18

19

Downt

own

Coun

trysid

e

High

way

Mornin

g

Even

ingNi

ght

Sunn

y

Cloud

yRa

iny

DKL

Multi-branchImage

FlowSegm

________________________Landscape ________________________Time of day ________________________Weather

Fig 13 DKL of the different branches in several conditions (from left toright downtown countryside highway morning evening night sunnycloudy rainy) Underlining highlights difference of aggregation in termsof landscape time of day and weather Please note that lower DKL

indicates better predictions

521 Dependency on driving environment

The DR(eye)VE data has been recorded under varying land-scapes time of day and weather conditions We tested our modelin all such different driving conditions As would be expectedFig 13 shows that the human attention is easier to predict inhighways rather than downtown where the focus can shift towardsmore distractors The model seems more reliable in eveningscenarios rather than morning or night where we observed betterlightning conditions and lack of shadows over-exposure and soon Lastly in rainy conditions we notice that human gaze iseasier to model possibly due to the higher level of awarenessdemanded to the driver and his consequent inability to focus awayfrom vanishing point To support the latter intuition we measuredthe performance of BM baseline (ie the average training fixationmap) grouped for weather condition As expected theDKL valuein rainy weather (153) is significantly lower than the ones forcloudy (161) and sunny weather (175) highlighting that whenrainy the driver is more focused on the road

11

|Σ| = 250times 109 |Σ| = 157times 109 |Σ| = 349times 108 |Σ| = 120times 108 |Σ| = 538times 107

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 14 Model prediction averaged across all test sequences and grouped by driving speed As the speed increases the area of the predicted mapshrinks recalling the trend observed in ground truth maps As in Fig 7 for each map a two dimensional Gaussian is fitted and the determinant ofits covariance matrix Σ is reported as a measure of the spread

road sidewalk buildings traffic signs trees flowerbed sky vehicles people cycles10

-4

10-3

10-2

10-1

100

prob

abilit

y of

fixa

tions

(log

sca

le)

Fig 15 Comparison between ground truth (gray bars) and predictedfixation maps (colored bars) when used to mask semantic segmentationof the scene The probability of fixation (in log-scale) for both groundtruth and model prediction is reported for each semantic class Despiteabsolute errors exist the two bar series agree on the relative importanceof different categories

522 Ablation study

In order to validate the design of the multi-branch model(see Sec 42) here we study the individual contributions of thedifferent branches by disabling one or more of themResults in Tab 4 show that the RGB branch plays a majorrole in FoA prediction The motion stream is also beneficialand provides a slight improvement that becomes clearer in theacting subsequences Indeed optical flow intrinsically captures avariety of peculiar scenarios that are non-trivial to classify whenonly color information is provided eg when the car is still at atraffic light or is turning The semantic stream on the other handprovides very little improvement In particular from Tab 4 and by

TABLE 4The ablation study performed on our multi-branch model I F and S

represent image optical flow and semantic segmentation branchesrespectively

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

I 0554 1415 -0008 0403 1826 0458F 0516 1616 -0137 0368 2010 0349S 0479 1699 -0119 0344 2082 0288

I+F 0558 1399 0033 0410 1799 0510I+S 0554 1413 -0001 0404 1823 0466F+S 0528 1571 -0055 0380 1956 0427

I+F+S 0559 1398 0038 0410 1797 0515

specifically comparing I+F and I+F+S a slight increase in the IGmeasure can be appreciated Nevertheless such improvement hasto be considered negligible when compared to color and motionsuggesting that in presence of efficiency concerns or real-timeconstraints the semantic stream can be discarded with little lossesin performance However we expect the benefit from this branchto increase as more accurate segmentation models will be released

523 Do we capture the attention dynamics

The previous sections validate quantitatively the proposed modelNow we assess its capability to attend like a human driverby comparing its predictions against the analysis performed inSec 31First we report the average predicted fixation map in severalspeed ranges in Fig 14 The conclusions we draw are twofoldi) generally the model succeeds in modeling the behavior ofthe driver at different speeds and ii) as the speed increasesfixation maps exhibit lower variance easing the modeling taskand prediction errors decreaseWe also study how often our model focuses on different semanticcategories in a fashion that recalls the analysis of Sec 31 butemploying our predictions rather than ground truth maps as focusof attention More precisely we normalize each map so thatthe maximum value equals 1 and apply the same thresholdingstrategy described in Sec 31 Likewise for each threshold valuea histogram over class labels is built by accounting all pixelsfalling within the binary map for all test frames This results innine histograms over semantic labels that we merge together byaveraging probabilities belonging to different threshold Fig 15shows the comparison Color bars represent how often the pre-dicted map focuses on a certain category while gray bars depictground truth behavior and are obtained by averaging histogramsin Fig 8 across different thresholds Please note that to highlightdifferences for low populated categories values are reported ona logarithmic scale The plot shows a certain degree of absoluteerror is present for all categories However in a broader sense ourmodel replicates the relative weight of different semantic classeswhile driving as testified by the importance of roads and vehiclesthat still dominate against other categories such as people andcycles that are mostly neglected This correlation is confirmed byKendall rank coefficient which scored 051 when computed onthe two bar series

53 Visual assessment of predicted fixation maps

To further validate the predictions of our model from the humanperception perspective 50 people with at least 3 years of driving

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

8

Algorithm 1 TRAINING The model is trained in two steps first each branch is trained separately through iterations detailed inprocedure SINGLE_BRANCH_TRAINING_ITERATION then the three branches are fine-tuned altogether as shown by procedureMULTI_BRANCH_FINE-TUNING_ITERATION For clarity we omit from notation i) the subscript b denoting the current domain inall X x and y variables in the single branch iteration and ii) the normalization of the sum of the outputs from each branch in line 13

1 procedure A SINGLE_BRANCH_TRAINING_ITERATION

input domain data X = x1 x2 x16 true attentional map y of last frame x16 of videoclip Xoutput branch loss Lb computed on input sample (X y)

2 Xres larr resize(X (112 112))3 Xcrop ycrop larr get_crop((X y) (112 112))4 ycrop larr COARSE(Xcrop) get coarse prediction on uncentered crop5 y larr REFINE(stack(x16 upsample(COARSE(Xres)))) get refined prediction over whole image6 Lb(XY )larr DKL(ycropycrop) +DKL(yy) compute branch loss as in Eq 4

7 procedure B MULTI_BRANCH_FINE-TUNING_ITERATION

input data X = x1 x2 x16 for all domains true attentional map y of last frame x16 of videoclip Xoutput overall loss L computed on input sample (X y)

8 Xres larr resize(X (112 112))9 Xcrop ycrop larr get_crop((X y) (112 112))

10 for branch b isin RGB flow seg do11 ybcrop larr COARSE(Xbcrop ) as in line 4 of the above procedure12 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres )))) as in line 5 of the above procedure13 L(XY )larr DKL(ycrop

sumb ybcrop ) +DKL(y

sumb yb) compute overall loss as in Eq 5

where C and R denote COARSE and REFINE modules(Xm

b Ym) isin Xb times Y is the m-th training example in the

b-th domain (namely RGB optical flow semantic segmentation)and φ and ψ indicate the crop and the resize functions respectively

Inference step While the presence of the C(φ(Xmb )) stream

is beneficial in training to reduce the spatial bias at test timeonly the R(C(ψ(Xm

b )) Xmb )) stream producing higher quality

prediction is used The outputs of such stream from each branch bare then summed together as explained in the following section

42 Multi-Branch modelAs described at the beginning of this section and depicted inFig 11 the multi-branch model is composed of three iden-tical branches The architecture of each branch has already beendescribed in Sec 41 above Each branch exploits complementaryinformation from a different domain and contributes to the finalprediction accordingly In detail the first branch works in the RGBdomain and processes raw visual data about the scene XRGBThe second branch focuses on motion through the optical flowrepresentation Xflow described in [23] Eventually the last branchtakes as input semantic segmentation probability maps Xseg Forthis last branch the number of input channels depends on thespecific algorithm used to extract the results 19 in our setup (Yu

Algorithm 2 INFERENCE At test time the data extracted fromthe resized videoclip is input to the three branches and their outputis summed and normalized to obtain the final FoA prediction

input data X = x1 x2 x16 for all domainsoutput predicted FoA map y1 Xres larr resize(X (112 112))2 for branch b isin RGB flow seg do3 yb larr REFINE(stack(xb16 upsample(COARSE(Xbres ))))4 y larr

sumb yb

sumi

sumb yb(i)

and Koltun [76]) The three independent predicted FoA maps aresummed and normalized to result in a probability distributionTo allow for larger batch size we choose to bootstrap each branchindependently by training it according to Eq 4 Then the completemulti-branch model which merges the three branches is fine-tuned with the following loss

L(X Y) =summ

(DKL(φ(Y m)

sumb

C(φ(Xmb ))) +

DKL(Y msumb

R(C(ψ(Xmb )) Xm

b )))

)

(5)The algorithm describing the complete inference over themulti-branch model in detailed in Alg 2

5 EXPERIMENTS

In this section we evaluate the performance of the proposedmulti-branch model First we start by comparing our modelagainst some baselines and other methods in literature Followingthe guidelines in [12] for the evaluation phase we rely on Pear-sonrsquos Correlation Coefficient (CC) and KullbackndashLeibler Diver-gence (DKL) measures Moreover we evaluate the InformationGain (IG) [35] measure to assess the quality of a predicted mapP with respect to a ground truth map Y in presence of a strongbias as

IG(P YB) =1

N

sumi

Yi[(log2(ε+ Pi)minus log2(ε+Bi)] (6)

where i is an index spanning all the N pixels in the image B thebias computed as the average training fixation map and ε ensuresnumerical stabilityFurthermore we conduct an ablation study to investigate howdifferent branches affect the final prediction and how their mutualinfluence changes in different scenarios We then study whetherour model captures the attention dynamics observed in Sec 31

9

nalprediction

+

FoAfrom motionow

clip

FoAfrom semanticsseg

clip

FoAfrom colorRGB

clip

Fig 11 The multi-branch model is composed of three different branches each of which has its own set of parameters and their predictions aresummed to obtain the final map Note that in this figure cropped streams are dropped to ease representation but are employed during training (asdiscussed in Sec 42 and depicted in Fig 10

Eventually we assess our model from a human perceptionperspective

Implementation details The three different pathways ofthe multi-branch model (namely FoA from color frommotion and from semantics) have been pre-trained independentlyusing the same cropping policy of Sec 42 and minimizing theobjective function in Eq 4 Each branch has been respectively fedwithbull 16 frames clips in raw RGB color spacebull 16 frames clips with optical flow maps encoded as color

images through the flow field encoding [23]bull 16 frames clips holding semantic segmentation from [76]

encoded as 19 scalar activation maps one per segmentationclass

During individual branch pre-training clips were randomly mir-rored for data augmentation We employ Adam optimizer with pa-rameters as suggested in the original paper [32] with the exceptionof the learning rate that we set to 10minus4 Eventually batch size wasfixed to 32 and each branch was trained until convergence TheDR(eye)VE dataset is split into train validation and test set asfollows sequences 1-38 are used for training sequences 39-74 fortesting The 500 frames in the middle of each training sequenceconstitute the validation setMoreover the complete multi-branch architecture was fine-tuned using the same cropping and data augmentation strategiesminimizing cost function in Eq 5 In this phase batch size was setto 4 due to GPU memory constraints and learning rate value waslowered to 10minus5 Inference time of each branch of our architectureis asymp 30 milliseconds per videoclip on an NVIDIA Titan X

51 Model evaluation

In Tab 3 we report results of our proposal against other state-of-the-art models [4] [15] [46] [59] [72] [73] evaluated both onthe complete test set and on acting subsequences only All thecompetitors with the exception of [46] are bottom-up approachesand mainly rely on appearance and motion discontinuities To testthe effectiveness of deep architectures for saliency prediction wecompare against the Multi-Level Network (MLNet) [15] which

scored favourably in the MIT300 saliency benchmark [11] andthe Recurrent Mixture Density Network (RMDN) [4] whichrepresents the only deep model addressing video saliency WhileMLNet works on images discarding the temporal informationRMDN encodes short sequences in a similar way to our COARSEmodule and then relies on a LSTM architecture to model longterm dependencies and estimates the fixation map in terms of aGMM To favor the comparison both models were re-trained onthe DR(eye)VE datasetResults highlight the superiority of our multi-branch archi-tecture on all test sequences The gap in performance with respectto bottom-up unsupervised approaches [72] [73] is higher and ismotivated by the peculiarity of the attention behavior within thedriving context which calls for a task-oriented training procedureMoreover MLNetrsquos low performance testifies for the need of ac-counting for the temporal correlation between consecutive framesthat distinguishes the tasks of attention prediction in images andvideos Indeed RMDN processes video inputs and outperformsMLNet on both DKL and IG metrics performing comparablyon CC Nonetheless its performance is still limited indeedqualitative results reported in Fig 12 suggest that long termdependencies captured by its recurrent module lead the networktowards the regression of the mean discarding contextual and

TABLE 3Experiments illustrating the superior performance of the

multi-branch model over several baselines and competitors Wereport both the average across the complete test sequences and only

the acting frames

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

Baseline Gaussian 040 216 -049 026 241 003Baseline Mean 051 160 000 022 235 000

Mathe et al [59] 004 330 -208 - - -Wang et al [72] 004 340 -221 - - -Wang et al [73] 011 306 -172 - - -

MLNet [15] 044 200 -088 032 235 -036RMDN [4] 041 177 -006 031 213 031

Palazzi et al [46] 055 148 -021 037 200 020multi-branch 056 140 004 041 180 051

10

Input frame GT multi-branch [46] [4] [15]

Fig 12 Qualitative assessment of the predicted fixation maps From left to right input clip ground truth map our prediction prediction of theprevious version of the model [46] prediction of RMDN [4] and prediction of MLNet [15]

frame-specific variations that would be preferrable to keep Tosupport this intuition we measure the average DKL betweenRMDN predictions and the mean training fixation map (Base-line Mean) resulting in a value of 011 Being lower than thedivergence measured with respect to groundtruth maps this valuehighlights the closer correlation to a central baseline rather thanto groundtruth Eventually we also observe improvements withrespect to our previous proposal [46] that relies on a more com-plex backbone model (also including a deconvolutional module)and processes RGB clips only The gap in performance residesin the greater awareness of our multi-branch architecture ofthe aspects that characterize the driving task as emerged fromthe analysis in Sec 31 The positive performances of our modelare also confirmed when evaluated on the acting partition of thedataset We recall that acting indicates sub-sequences exhibiting asignificant task-driven shift of attention from the center of theimage (Fig 5) Being able to predict the FoA also on actingsub-sequences means that the model captures the strong centeredattention bias but is capable of generalizing when required by thecontextThis is further shown by the comparison against a centeredGaussian baseline (BG) and against the average of all trainingset fixation maps (BM) The former baseline has proven effectiveon many image saliency detection tasks [11] while the latterrepresents a more task-driven version The superior performanceof the multi-branch model wrt baselines highlights thatdespite the attention is often strongly biased towards the vanishingpoint of the road the network is able to deal with sudden task-driven changes in gaze direction

52 Model analysis

In this section we investigate the behavior of our proposed modelunder different landscapes time of day and weather (Sec 521)we study the contribution of each branch to the FoA predictiontask (Sec 522) and we compare the learnt attention dynamicsagainst the one observed in the human data (Sec 523)

12

13

14

15

16

17

18

19

Downt

own

Coun

trysid

e

High

way

Mornin

g

Even

ingNi

ght

Sunn

y

Cloud

yRa

iny

DKL

Multi-branchImage

FlowSegm

________________________Landscape ________________________Time of day ________________________Weather

Fig 13 DKL of the different branches in several conditions (from left toright downtown countryside highway morning evening night sunnycloudy rainy) Underlining highlights difference of aggregation in termsof landscape time of day and weather Please note that lower DKL

indicates better predictions

521 Dependency on driving environment

The DR(eye)VE data has been recorded under varying land-scapes time of day and weather conditions We tested our modelin all such different driving conditions As would be expectedFig 13 shows that the human attention is easier to predict inhighways rather than downtown where the focus can shift towardsmore distractors The model seems more reliable in eveningscenarios rather than morning or night where we observed betterlightning conditions and lack of shadows over-exposure and soon Lastly in rainy conditions we notice that human gaze iseasier to model possibly due to the higher level of awarenessdemanded to the driver and his consequent inability to focus awayfrom vanishing point To support the latter intuition we measuredthe performance of BM baseline (ie the average training fixationmap) grouped for weather condition As expected theDKL valuein rainy weather (153) is significantly lower than the ones forcloudy (161) and sunny weather (175) highlighting that whenrainy the driver is more focused on the road

11

|Σ| = 250times 109 |Σ| = 157times 109 |Σ| = 349times 108 |Σ| = 120times 108 |Σ| = 538times 107

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 14 Model prediction averaged across all test sequences and grouped by driving speed As the speed increases the area of the predicted mapshrinks recalling the trend observed in ground truth maps As in Fig 7 for each map a two dimensional Gaussian is fitted and the determinant ofits covariance matrix Σ is reported as a measure of the spread

road sidewalk buildings traffic signs trees flowerbed sky vehicles people cycles10

-4

10-3

10-2

10-1

100

prob

abilit

y of

fixa

tions

(log

sca

le)

Fig 15 Comparison between ground truth (gray bars) and predictedfixation maps (colored bars) when used to mask semantic segmentationof the scene The probability of fixation (in log-scale) for both groundtruth and model prediction is reported for each semantic class Despiteabsolute errors exist the two bar series agree on the relative importanceof different categories

522 Ablation study

In order to validate the design of the multi-branch model(see Sec 42) here we study the individual contributions of thedifferent branches by disabling one or more of themResults in Tab 4 show that the RGB branch plays a majorrole in FoA prediction The motion stream is also beneficialand provides a slight improvement that becomes clearer in theacting subsequences Indeed optical flow intrinsically captures avariety of peculiar scenarios that are non-trivial to classify whenonly color information is provided eg when the car is still at atraffic light or is turning The semantic stream on the other handprovides very little improvement In particular from Tab 4 and by

TABLE 4The ablation study performed on our multi-branch model I F and S

represent image optical flow and semantic segmentation branchesrespectively

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

I 0554 1415 -0008 0403 1826 0458F 0516 1616 -0137 0368 2010 0349S 0479 1699 -0119 0344 2082 0288

I+F 0558 1399 0033 0410 1799 0510I+S 0554 1413 -0001 0404 1823 0466F+S 0528 1571 -0055 0380 1956 0427

I+F+S 0559 1398 0038 0410 1797 0515

specifically comparing I+F and I+F+S a slight increase in the IGmeasure can be appreciated Nevertheless such improvement hasto be considered negligible when compared to color and motionsuggesting that in presence of efficiency concerns or real-timeconstraints the semantic stream can be discarded with little lossesin performance However we expect the benefit from this branchto increase as more accurate segmentation models will be released

523 Do we capture the attention dynamics

The previous sections validate quantitatively the proposed modelNow we assess its capability to attend like a human driverby comparing its predictions against the analysis performed inSec 31First we report the average predicted fixation map in severalspeed ranges in Fig 14 The conclusions we draw are twofoldi) generally the model succeeds in modeling the behavior ofthe driver at different speeds and ii) as the speed increasesfixation maps exhibit lower variance easing the modeling taskand prediction errors decreaseWe also study how often our model focuses on different semanticcategories in a fashion that recalls the analysis of Sec 31 butemploying our predictions rather than ground truth maps as focusof attention More precisely we normalize each map so thatthe maximum value equals 1 and apply the same thresholdingstrategy described in Sec 31 Likewise for each threshold valuea histogram over class labels is built by accounting all pixelsfalling within the binary map for all test frames This results innine histograms over semantic labels that we merge together byaveraging probabilities belonging to different threshold Fig 15shows the comparison Color bars represent how often the pre-dicted map focuses on a certain category while gray bars depictground truth behavior and are obtained by averaging histogramsin Fig 8 across different thresholds Please note that to highlightdifferences for low populated categories values are reported ona logarithmic scale The plot shows a certain degree of absoluteerror is present for all categories However in a broader sense ourmodel replicates the relative weight of different semantic classeswhile driving as testified by the importance of roads and vehiclesthat still dominate against other categories such as people andcycles that are mostly neglected This correlation is confirmed byKendall rank coefficient which scored 051 when computed onthe two bar series

53 Visual assessment of predicted fixation maps

To further validate the predictions of our model from the humanperception perspective 50 people with at least 3 years of driving

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

9

nalprediction

+

FoAfrom motionow

clip

FoAfrom semanticsseg

clip

FoAfrom colorRGB

clip

Fig 11 The multi-branch model is composed of three different branches each of which has its own set of parameters and their predictions aresummed to obtain the final map Note that in this figure cropped streams are dropped to ease representation but are employed during training (asdiscussed in Sec 42 and depicted in Fig 10

Eventually we assess our model from a human perceptionperspective

Implementation details The three different pathways ofthe multi-branch model (namely FoA from color frommotion and from semantics) have been pre-trained independentlyusing the same cropping policy of Sec 42 and minimizing theobjective function in Eq 4 Each branch has been respectively fedwithbull 16 frames clips in raw RGB color spacebull 16 frames clips with optical flow maps encoded as color

images through the flow field encoding [23]bull 16 frames clips holding semantic segmentation from [76]

encoded as 19 scalar activation maps one per segmentationclass

During individual branch pre-training clips were randomly mir-rored for data augmentation We employ Adam optimizer with pa-rameters as suggested in the original paper [32] with the exceptionof the learning rate that we set to 10minus4 Eventually batch size wasfixed to 32 and each branch was trained until convergence TheDR(eye)VE dataset is split into train validation and test set asfollows sequences 1-38 are used for training sequences 39-74 fortesting The 500 frames in the middle of each training sequenceconstitute the validation setMoreover the complete multi-branch architecture was fine-tuned using the same cropping and data augmentation strategiesminimizing cost function in Eq 5 In this phase batch size was setto 4 due to GPU memory constraints and learning rate value waslowered to 10minus5 Inference time of each branch of our architectureis asymp 30 milliseconds per videoclip on an NVIDIA Titan X

51 Model evaluation

In Tab 3 we report results of our proposal against other state-of-the-art models [4] [15] [46] [59] [72] [73] evaluated both onthe complete test set and on acting subsequences only All thecompetitors with the exception of [46] are bottom-up approachesand mainly rely on appearance and motion discontinuities To testthe effectiveness of deep architectures for saliency prediction wecompare against the Multi-Level Network (MLNet) [15] which

scored favourably in the MIT300 saliency benchmark [11] andthe Recurrent Mixture Density Network (RMDN) [4] whichrepresents the only deep model addressing video saliency WhileMLNet works on images discarding the temporal informationRMDN encodes short sequences in a similar way to our COARSEmodule and then relies on a LSTM architecture to model longterm dependencies and estimates the fixation map in terms of aGMM To favor the comparison both models were re-trained onthe DR(eye)VE datasetResults highlight the superiority of our multi-branch archi-tecture on all test sequences The gap in performance with respectto bottom-up unsupervised approaches [72] [73] is higher and ismotivated by the peculiarity of the attention behavior within thedriving context which calls for a task-oriented training procedureMoreover MLNetrsquos low performance testifies for the need of ac-counting for the temporal correlation between consecutive framesthat distinguishes the tasks of attention prediction in images andvideos Indeed RMDN processes video inputs and outperformsMLNet on both DKL and IG metrics performing comparablyon CC Nonetheless its performance is still limited indeedqualitative results reported in Fig 12 suggest that long termdependencies captured by its recurrent module lead the networktowards the regression of the mean discarding contextual and

TABLE 3Experiments illustrating the superior performance of the

multi-branch model over several baselines and competitors Wereport both the average across the complete test sequences and only

the acting frames

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

Baseline Gaussian 040 216 -049 026 241 003Baseline Mean 051 160 000 022 235 000

Mathe et al [59] 004 330 -208 - - -Wang et al [72] 004 340 -221 - - -Wang et al [73] 011 306 -172 - - -

MLNet [15] 044 200 -088 032 235 -036RMDN [4] 041 177 -006 031 213 031

Palazzi et al [46] 055 148 -021 037 200 020multi-branch 056 140 004 041 180 051

10

Input frame GT multi-branch [46] [4] [15]

Fig 12 Qualitative assessment of the predicted fixation maps From left to right input clip ground truth map our prediction prediction of theprevious version of the model [46] prediction of RMDN [4] and prediction of MLNet [15]

frame-specific variations that would be preferrable to keep Tosupport this intuition we measure the average DKL betweenRMDN predictions and the mean training fixation map (Base-line Mean) resulting in a value of 011 Being lower than thedivergence measured with respect to groundtruth maps this valuehighlights the closer correlation to a central baseline rather thanto groundtruth Eventually we also observe improvements withrespect to our previous proposal [46] that relies on a more com-plex backbone model (also including a deconvolutional module)and processes RGB clips only The gap in performance residesin the greater awareness of our multi-branch architecture ofthe aspects that characterize the driving task as emerged fromthe analysis in Sec 31 The positive performances of our modelare also confirmed when evaluated on the acting partition of thedataset We recall that acting indicates sub-sequences exhibiting asignificant task-driven shift of attention from the center of theimage (Fig 5) Being able to predict the FoA also on actingsub-sequences means that the model captures the strong centeredattention bias but is capable of generalizing when required by thecontextThis is further shown by the comparison against a centeredGaussian baseline (BG) and against the average of all trainingset fixation maps (BM) The former baseline has proven effectiveon many image saliency detection tasks [11] while the latterrepresents a more task-driven version The superior performanceof the multi-branch model wrt baselines highlights thatdespite the attention is often strongly biased towards the vanishingpoint of the road the network is able to deal with sudden task-driven changes in gaze direction

52 Model analysis

In this section we investigate the behavior of our proposed modelunder different landscapes time of day and weather (Sec 521)we study the contribution of each branch to the FoA predictiontask (Sec 522) and we compare the learnt attention dynamicsagainst the one observed in the human data (Sec 523)

12

13

14

15

16

17

18

19

Downt

own

Coun

trysid

e

High

way

Mornin

g

Even

ingNi

ght

Sunn

y

Cloud

yRa

iny

DKL

Multi-branchImage

FlowSegm

________________________Landscape ________________________Time of day ________________________Weather

Fig 13 DKL of the different branches in several conditions (from left toright downtown countryside highway morning evening night sunnycloudy rainy) Underlining highlights difference of aggregation in termsof landscape time of day and weather Please note that lower DKL

indicates better predictions

521 Dependency on driving environment

The DR(eye)VE data has been recorded under varying land-scapes time of day and weather conditions We tested our modelin all such different driving conditions As would be expectedFig 13 shows that the human attention is easier to predict inhighways rather than downtown where the focus can shift towardsmore distractors The model seems more reliable in eveningscenarios rather than morning or night where we observed betterlightning conditions and lack of shadows over-exposure and soon Lastly in rainy conditions we notice that human gaze iseasier to model possibly due to the higher level of awarenessdemanded to the driver and his consequent inability to focus awayfrom vanishing point To support the latter intuition we measuredthe performance of BM baseline (ie the average training fixationmap) grouped for weather condition As expected theDKL valuein rainy weather (153) is significantly lower than the ones forcloudy (161) and sunny weather (175) highlighting that whenrainy the driver is more focused on the road

11

|Σ| = 250times 109 |Σ| = 157times 109 |Σ| = 349times 108 |Σ| = 120times 108 |Σ| = 538times 107

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 14 Model prediction averaged across all test sequences and grouped by driving speed As the speed increases the area of the predicted mapshrinks recalling the trend observed in ground truth maps As in Fig 7 for each map a two dimensional Gaussian is fitted and the determinant ofits covariance matrix Σ is reported as a measure of the spread

road sidewalk buildings traffic signs trees flowerbed sky vehicles people cycles10

-4

10-3

10-2

10-1

100

prob

abilit

y of

fixa

tions

(log

sca

le)

Fig 15 Comparison between ground truth (gray bars) and predictedfixation maps (colored bars) when used to mask semantic segmentationof the scene The probability of fixation (in log-scale) for both groundtruth and model prediction is reported for each semantic class Despiteabsolute errors exist the two bar series agree on the relative importanceof different categories

522 Ablation study

In order to validate the design of the multi-branch model(see Sec 42) here we study the individual contributions of thedifferent branches by disabling one or more of themResults in Tab 4 show that the RGB branch plays a majorrole in FoA prediction The motion stream is also beneficialand provides a slight improvement that becomes clearer in theacting subsequences Indeed optical flow intrinsically captures avariety of peculiar scenarios that are non-trivial to classify whenonly color information is provided eg when the car is still at atraffic light or is turning The semantic stream on the other handprovides very little improvement In particular from Tab 4 and by

TABLE 4The ablation study performed on our multi-branch model I F and S

represent image optical flow and semantic segmentation branchesrespectively

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

I 0554 1415 -0008 0403 1826 0458F 0516 1616 -0137 0368 2010 0349S 0479 1699 -0119 0344 2082 0288

I+F 0558 1399 0033 0410 1799 0510I+S 0554 1413 -0001 0404 1823 0466F+S 0528 1571 -0055 0380 1956 0427

I+F+S 0559 1398 0038 0410 1797 0515

specifically comparing I+F and I+F+S a slight increase in the IGmeasure can be appreciated Nevertheless such improvement hasto be considered negligible when compared to color and motionsuggesting that in presence of efficiency concerns or real-timeconstraints the semantic stream can be discarded with little lossesin performance However we expect the benefit from this branchto increase as more accurate segmentation models will be released

523 Do we capture the attention dynamics

The previous sections validate quantitatively the proposed modelNow we assess its capability to attend like a human driverby comparing its predictions against the analysis performed inSec 31First we report the average predicted fixation map in severalspeed ranges in Fig 14 The conclusions we draw are twofoldi) generally the model succeeds in modeling the behavior ofthe driver at different speeds and ii) as the speed increasesfixation maps exhibit lower variance easing the modeling taskand prediction errors decreaseWe also study how often our model focuses on different semanticcategories in a fashion that recalls the analysis of Sec 31 butemploying our predictions rather than ground truth maps as focusof attention More precisely we normalize each map so thatthe maximum value equals 1 and apply the same thresholdingstrategy described in Sec 31 Likewise for each threshold valuea histogram over class labels is built by accounting all pixelsfalling within the binary map for all test frames This results innine histograms over semantic labels that we merge together byaveraging probabilities belonging to different threshold Fig 15shows the comparison Color bars represent how often the pre-dicted map focuses on a certain category while gray bars depictground truth behavior and are obtained by averaging histogramsin Fig 8 across different thresholds Please note that to highlightdifferences for low populated categories values are reported ona logarithmic scale The plot shows a certain degree of absoluteerror is present for all categories However in a broader sense ourmodel replicates the relative weight of different semantic classeswhile driving as testified by the importance of roads and vehiclesthat still dominate against other categories such as people andcycles that are mostly neglected This correlation is confirmed byKendall rank coefficient which scored 051 when computed onthe two bar series

53 Visual assessment of predicted fixation maps

To further validate the predictions of our model from the humanperception perspective 50 people with at least 3 years of driving

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

10

Input frame GT multi-branch [46] [4] [15]

Fig 12 Qualitative assessment of the predicted fixation maps From left to right input clip ground truth map our prediction prediction of theprevious version of the model [46] prediction of RMDN [4] and prediction of MLNet [15]

frame-specific variations that would be preferrable to keep Tosupport this intuition we measure the average DKL betweenRMDN predictions and the mean training fixation map (Base-line Mean) resulting in a value of 011 Being lower than thedivergence measured with respect to groundtruth maps this valuehighlights the closer correlation to a central baseline rather thanto groundtruth Eventually we also observe improvements withrespect to our previous proposal [46] that relies on a more com-plex backbone model (also including a deconvolutional module)and processes RGB clips only The gap in performance residesin the greater awareness of our multi-branch architecture ofthe aspects that characterize the driving task as emerged fromthe analysis in Sec 31 The positive performances of our modelare also confirmed when evaluated on the acting partition of thedataset We recall that acting indicates sub-sequences exhibiting asignificant task-driven shift of attention from the center of theimage (Fig 5) Being able to predict the FoA also on actingsub-sequences means that the model captures the strong centeredattention bias but is capable of generalizing when required by thecontextThis is further shown by the comparison against a centeredGaussian baseline (BG) and against the average of all trainingset fixation maps (BM) The former baseline has proven effectiveon many image saliency detection tasks [11] while the latterrepresents a more task-driven version The superior performanceof the multi-branch model wrt baselines highlights thatdespite the attention is often strongly biased towards the vanishingpoint of the road the network is able to deal with sudden task-driven changes in gaze direction

52 Model analysis

In this section we investigate the behavior of our proposed modelunder different landscapes time of day and weather (Sec 521)we study the contribution of each branch to the FoA predictiontask (Sec 522) and we compare the learnt attention dynamicsagainst the one observed in the human data (Sec 523)

12

13

14

15

16

17

18

19

Downt

own

Coun

trysid

e

High

way

Mornin

g

Even

ingNi

ght

Sunn

y

Cloud

yRa

iny

DKL

Multi-branchImage

FlowSegm

________________________Landscape ________________________Time of day ________________________Weather

Fig 13 DKL of the different branches in several conditions (from left toright downtown countryside highway morning evening night sunnycloudy rainy) Underlining highlights difference of aggregation in termsof landscape time of day and weather Please note that lower DKL

indicates better predictions

521 Dependency on driving environment

The DR(eye)VE data has been recorded under varying land-scapes time of day and weather conditions We tested our modelin all such different driving conditions As would be expectedFig 13 shows that the human attention is easier to predict inhighways rather than downtown where the focus can shift towardsmore distractors The model seems more reliable in eveningscenarios rather than morning or night where we observed betterlightning conditions and lack of shadows over-exposure and soon Lastly in rainy conditions we notice that human gaze iseasier to model possibly due to the higher level of awarenessdemanded to the driver and his consequent inability to focus awayfrom vanishing point To support the latter intuition we measuredthe performance of BM baseline (ie the average training fixationmap) grouped for weather condition As expected theDKL valuein rainy weather (153) is significantly lower than the ones forcloudy (161) and sunny weather (175) highlighting that whenrainy the driver is more focused on the road

11

|Σ| = 250times 109 |Σ| = 157times 109 |Σ| = 349times 108 |Σ| = 120times 108 |Σ| = 538times 107

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 14 Model prediction averaged across all test sequences and grouped by driving speed As the speed increases the area of the predicted mapshrinks recalling the trend observed in ground truth maps As in Fig 7 for each map a two dimensional Gaussian is fitted and the determinant ofits covariance matrix Σ is reported as a measure of the spread

road sidewalk buildings traffic signs trees flowerbed sky vehicles people cycles10

-4

10-3

10-2

10-1

100

prob

abilit

y of

fixa

tions

(log

sca

le)

Fig 15 Comparison between ground truth (gray bars) and predictedfixation maps (colored bars) when used to mask semantic segmentationof the scene The probability of fixation (in log-scale) for both groundtruth and model prediction is reported for each semantic class Despiteabsolute errors exist the two bar series agree on the relative importanceof different categories

522 Ablation study

In order to validate the design of the multi-branch model(see Sec 42) here we study the individual contributions of thedifferent branches by disabling one or more of themResults in Tab 4 show that the RGB branch plays a majorrole in FoA prediction The motion stream is also beneficialand provides a slight improvement that becomes clearer in theacting subsequences Indeed optical flow intrinsically captures avariety of peculiar scenarios that are non-trivial to classify whenonly color information is provided eg when the car is still at atraffic light or is turning The semantic stream on the other handprovides very little improvement In particular from Tab 4 and by

TABLE 4The ablation study performed on our multi-branch model I F and S

represent image optical flow and semantic segmentation branchesrespectively

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

I 0554 1415 -0008 0403 1826 0458F 0516 1616 -0137 0368 2010 0349S 0479 1699 -0119 0344 2082 0288

I+F 0558 1399 0033 0410 1799 0510I+S 0554 1413 -0001 0404 1823 0466F+S 0528 1571 -0055 0380 1956 0427

I+F+S 0559 1398 0038 0410 1797 0515

specifically comparing I+F and I+F+S a slight increase in the IGmeasure can be appreciated Nevertheless such improvement hasto be considered negligible when compared to color and motionsuggesting that in presence of efficiency concerns or real-timeconstraints the semantic stream can be discarded with little lossesin performance However we expect the benefit from this branchto increase as more accurate segmentation models will be released

523 Do we capture the attention dynamics

The previous sections validate quantitatively the proposed modelNow we assess its capability to attend like a human driverby comparing its predictions against the analysis performed inSec 31First we report the average predicted fixation map in severalspeed ranges in Fig 14 The conclusions we draw are twofoldi) generally the model succeeds in modeling the behavior ofthe driver at different speeds and ii) as the speed increasesfixation maps exhibit lower variance easing the modeling taskand prediction errors decreaseWe also study how often our model focuses on different semanticcategories in a fashion that recalls the analysis of Sec 31 butemploying our predictions rather than ground truth maps as focusof attention More precisely we normalize each map so thatthe maximum value equals 1 and apply the same thresholdingstrategy described in Sec 31 Likewise for each threshold valuea histogram over class labels is built by accounting all pixelsfalling within the binary map for all test frames This results innine histograms over semantic labels that we merge together byaveraging probabilities belonging to different threshold Fig 15shows the comparison Color bars represent how often the pre-dicted map focuses on a certain category while gray bars depictground truth behavior and are obtained by averaging histogramsin Fig 8 across different thresholds Please note that to highlightdifferences for low populated categories values are reported ona logarithmic scale The plot shows a certain degree of absoluteerror is present for all categories However in a broader sense ourmodel replicates the relative weight of different semantic classeswhile driving as testified by the importance of roads and vehiclesthat still dominate against other categories such as people andcycles that are mostly neglected This correlation is confirmed byKendall rank coefficient which scored 051 when computed onthe two bar series

53 Visual assessment of predicted fixation maps

To further validate the predictions of our model from the humanperception perspective 50 people with at least 3 years of driving

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

11

|Σ| = 250times 109 |Σ| = 157times 109 |Σ| = 349times 108 |Σ| = 120times 108 |Σ| = 538times 107

(a) 0 le kmh le 10 (b) 10 le kmh le 30 (c) 30 le kmh le 50 (d) 50 le kmh le 70 (e) 70 le kmh

Fig 14 Model prediction averaged across all test sequences and grouped by driving speed As the speed increases the area of the predicted mapshrinks recalling the trend observed in ground truth maps As in Fig 7 for each map a two dimensional Gaussian is fitted and the determinant ofits covariance matrix Σ is reported as a measure of the spread

road sidewalk buildings traffic signs trees flowerbed sky vehicles people cycles10

-4

10-3

10-2

10-1

100

prob

abilit

y of

fixa

tions

(log

sca

le)

Fig 15 Comparison between ground truth (gray bars) and predictedfixation maps (colored bars) when used to mask semantic segmentationof the scene The probability of fixation (in log-scale) for both groundtruth and model prediction is reported for each semantic class Despiteabsolute errors exist the two bar series agree on the relative importanceof different categories

522 Ablation study

In order to validate the design of the multi-branch model(see Sec 42) here we study the individual contributions of thedifferent branches by disabling one or more of themResults in Tab 4 show that the RGB branch plays a majorrole in FoA prediction The motion stream is also beneficialand provides a slight improvement that becomes clearer in theacting subsequences Indeed optical flow intrinsically captures avariety of peculiar scenarios that are non-trivial to classify whenonly color information is provided eg when the car is still at atraffic light or is turning The semantic stream on the other handprovides very little improvement In particular from Tab 4 and by

TABLE 4The ablation study performed on our multi-branch model I F and S

represent image optical flow and semantic segmentation branchesrespectively

Test sequences Acting subsequencesCC DKL IG CC DKL IGuarr darr uarr uarr darr uarr

I 0554 1415 -0008 0403 1826 0458F 0516 1616 -0137 0368 2010 0349S 0479 1699 -0119 0344 2082 0288

I+F 0558 1399 0033 0410 1799 0510I+S 0554 1413 -0001 0404 1823 0466F+S 0528 1571 -0055 0380 1956 0427

I+F+S 0559 1398 0038 0410 1797 0515

specifically comparing I+F and I+F+S a slight increase in the IGmeasure can be appreciated Nevertheless such improvement hasto be considered negligible when compared to color and motionsuggesting that in presence of efficiency concerns or real-timeconstraints the semantic stream can be discarded with little lossesin performance However we expect the benefit from this branchto increase as more accurate segmentation models will be released

523 Do we capture the attention dynamics

The previous sections validate quantitatively the proposed modelNow we assess its capability to attend like a human driverby comparing its predictions against the analysis performed inSec 31First we report the average predicted fixation map in severalspeed ranges in Fig 14 The conclusions we draw are twofoldi) generally the model succeeds in modeling the behavior ofthe driver at different speeds and ii) as the speed increasesfixation maps exhibit lower variance easing the modeling taskand prediction errors decreaseWe also study how often our model focuses on different semanticcategories in a fashion that recalls the analysis of Sec 31 butemploying our predictions rather than ground truth maps as focusof attention More precisely we normalize each map so thatthe maximum value equals 1 and apply the same thresholdingstrategy described in Sec 31 Likewise for each threshold valuea histogram over class labels is built by accounting all pixelsfalling within the binary map for all test frames This results innine histograms over semantic labels that we merge together byaveraging probabilities belonging to different threshold Fig 15shows the comparison Color bars represent how often the pre-dicted map focuses on a certain category while gray bars depictground truth behavior and are obtained by averaging histogramsin Fig 8 across different thresholds Please note that to highlightdifferences for low populated categories values are reported ona logarithmic scale The plot shows a certain degree of absoluteerror is present for all categories However in a broader sense ourmodel replicates the relative weight of different semantic classeswhile driving as testified by the importance of roads and vehiclesthat still dominate against other categories such as people andcycles that are mostly neglected This correlation is confirmed byKendall rank coefficient which scored 051 when computed onthe two bar series

53 Visual assessment of predicted fixation maps

To further validate the predictions of our model from the humanperception perspective 50 people with at least 3 years of driving

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

12

Fig 16 The figure depicts a videoclip frame that underwent the foveationprocess The attentional map (above) is employed to blur the framein a way that approximates the foveal vision of the driver [48] In thefoveated frame (below) it can be appreciated how the ratio of high-levelinformation smoothly degrades getting farther from fixation points

experience were asked to participate in a visual assessment3 Firsta pool of 400 videoclips (40 seconds long) is sampled from theDR(eye)VE dataset Sampling is weighted such that resultingvideoclips are evenly distributed among different scenarios weath-ers drivers and daylight conditions Also half of these videoclipscontain sub-sequences that were previously annotated as actingTo approximate as realistically as possible the visual field ofattention of the driver sampled videoclips are pre-processedfollowing the procedure in [71] As in [71] we leverage the SpaceVariant Imaging Toolbox [48] to implement this phase setting theparameter that halves the spatial resolution every 23 to mirrorhuman vision [36] [71] The resulting videoclip preserves detailsnear to the fixation points in each frame whereas the rest of thescene gets more and more blurred getting farther from fixationsuntil only low-frequency contextual information survive Coher-ently with [71] we refer to this process as foveation (in analogywith human foveal vision) Thus pre-processed videoclips will becalled foveated videoclips from now on To appreciate the effectof this step the reader is referred to Fig 16Foveated videoclips were created by randomly selecting one ofthe following three fixation maps the ground truth fixation map (Gvideoclips) the fixation map predicted by our model (P videoclips)or the average fixation map in the DR(eye)VE training set (Cvideoclips) The latter central baseline allows to take into accountthe potential preference for a stable attentional map (ie lack ofswitching of focus) Further details about the creation of foveatedvideoclips are reported in Sec 8 of the supplementary material

Each participant was asked to watch five randomly sampledfoveated videoclips After each videoclip he answered the fol-lowing questionbull Would you say the observed attention behavior comes from a

human driver (yesno)Each of the 50 participant evaluates five foveated videoclips for atotal of 250 examplesThe confusion matrix of provided answers is reported in Fig 17

3 These were students (11 females 39 males) of age between 21 and 26(micro = 234 σ = 16) recruited at our University on a voluntary basis throughan online form

Prediction

Pred

ictio

n

Human

Hum

an

Predicted Label

True

Lab

el

021

030

TP025

FN024

FP021

TN030

Confusion Matrix

Fig 17 The confusion matrix reports the results of participantsrsquo guesseson the source of fixation maps Overall accuracy is about 55 which isfairly close to random chance

Participants were not particularly good at discriminating betweenhumanrsquos gaze and model generated maps scoring about the 55of accuracy which is comparable to random guessing this suggestsour model is capable of producing plausible attentional patternsthat resemble a proper driving behavior to a human observer

6 CONCLUSIONS

This paper presents a study of human attention dynamics un-derpinning the driving experience Our main contribution is amulti-branch deep network capable of capturing such factorsand replicating the driverrsquos focus of attention from raw videosequences The design of our model has been guided by a prioranalysis highlighting i) the existence of common gaze patternsacross drivers and different scenarios and ii) a consistent re-lation between changes in speed lightning conditions weatherand landscape and changes in the driverrsquos focus of attentionExperiments with the proposed architecture and related trainingstrategies yielded state-of-the-art results To our knowledge ourmodel is the first able to predict human attention in real-worlddriving sequences As the model only input are car-centric videosit might be integrated with already adopted ADAS technologies

ACKNOWLEDGMENTSWe acknowledge the CINECA award under the ISCRA initiativefor the availability of high performance computing resourcesand support We also gratefully acknowledge the support ofFacebook Artificial Intelligence Research and Panasonic Sili-con Valley Lab for the donation of GPUs used for this re-search

REFERENCES

[1] R Achanta S Hemami F Estrada and S Susstrunk Frequency-tunedsalient region detection In Computer Vision and Pattern Recognition2009 CVPR 2009 IEEE Conference on June 2009 2

[2] S Alletto A Palazzi F Solera S Calderara and R CucchiaraDr(Eye)Ve A dataset for attention-based tasks with applications toautonomous and assisted driving In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops 2016 4

[3] L Baraldi C Grana and R Cucchiara Hierarchical boundary-awareneural encoder for video captioning In Proceedings of the IEEEInternational Conference on Computer Vision 2017 6

[4] L Bazzani H Larochelle and L Torresani Recurrent mixture densitynetwork for spatiotemporal visual attention In International Conferenceon Learning Representations (ICLR) 2017 2 9 10

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

13

[5] G Borghi M Venturelli R Vezzani and R Cucchiara Poseidon Face-from-depth for driver pose estimation In CVPR 2017 2

[6] A Borji M Feng and H Lu Vanishing point attracts gaze in free-viewing and visual search tasks Journal of vision 16(14) 2016 5

[7] A Borji and L Itti State-of-the-art in visual attention modeling IEEEtransactions on pattern analysis and machine intelligence 35(1) 20132

[8] A Borji and L Itti Cat2000 A large scale fixation dataset for boostingsaliency research CVPR 2015 workshop on Future of Datasets 20152

[9] A Borji D N Sihite and L Itti Whatwhere to look next modelingtop-down visual attention in complex interactive environments IEEETransactions on Systems Man and Cybernetics Systems 44(5) 2014 2

[10] R Breacutemond J-M Auberlet V Cavallo L Deacutesireacute V Faure S Lemon-nier R Lobjois and J-P Tarel Where we look when we drive Amultidisciplinary approach In Proceedings of Transport Research Arena(TRArsquo14) Paris France 2014 2

[11] Z Bylinskii T Judd A Borji L Itti F Durand A Oliva andA Torralba Mit saliency benchmark 2015 1 2 3 9 10

[12] Z Bylinskii T Judd A Oliva A Torralba and F Durand What dodifferent evaluation metrics tell us about saliency models arXiv preprintarXiv160403605 2016 8

[13] X Chen K Kundu Y Zhu A Berneshawi H Ma S Fidler andR Urtasun 3d object proposals for accurate object class detection InNIPS 2015 1

[14] M M Cheng N J Mitra X Huang P H S Torr and S M Hu Globalcontrast based salient region detection IEEE Transactions on PatternAnalysis and Machine Intelligence 37(3) March 2015 2

[15] M Cornia L Baraldi G Serra and R Cucchiara A Deep Multi-LevelNetwork for Saliency Prediction In International Conference on PatternRecognition (ICPR) 2016 2 9 10

[16] M Cornia L Baraldi G Serra and R Cucchiara Predicting humaneye fixations via an lstm-based saliency attentive model arXiv preprintarXiv161109571 2016 2

[17] L Elazary and L Itti A bayesian model for efficient visual search andrecognition Vision Research 50(14) 2010 2

[18] M A Fischler and R C Bolles Random sample consensus a paradigmfor model fitting with applications to image analysis and automatedcartography Communications of the ACM 24(6) 1981 3

[19] L Fridman P Langhans J Lee and B Reimer Driver gaze regionestimation without use of eye movement IEEE Intelligent Systems 31(3)2016 2 3 4

[20] S Frintrop E Rome and H I Christensen Computational visualattention systems and their cognitive foundations A survey ACMTransactions on Applied Perception (TAP) 7(1) 2010 2

[21] B Froumlhlich M Enzweiler and U Franke Will this car change the lane-turn signal recognition in the frequency domain In 2014 IEEE IntelligentVehicles Symposium Proceedings IEEE 2014 1

[22] D Gao V Mahadevan and N Vasconcelos On the plausibility of thediscriminant centersurround hypothesis for visual saliency Journal ofVision 2008 2

[23] T Gkamas and C Nikou Guiding optical flow estimation usingsuperpixels In Digital Signal Processing (DSP) 2011 17th InternationalConference on IEEE 2011 8 9

[24] S Goferman L Zelnik-Manor and A Tal Context-aware saliencydetection IEEE Trans Pattern Anal Mach Intell 34(10) Oct 2012 2

[25] R Groner F Walder and M Groner Looking at faces Local and globalaspects of scanpaths Advances in Psychology 22 1984 4

[26] S Grubmuumlller J Plihal and P Nedoma Automated Driving from theView of Technical Standards Springer International Publishing Cham2017 1

[27] J M Henderson Human gaze control during real-world scene percep-tion Trends in cognitive sciences 7(11) 2003 4

[28] X Huang C Shen X Boix and Q Zhao Salicon Reducing thesemantic gap in saliency prediction by adapting deep neural networks In2015 IEEE International Conference on Computer Vision (ICCV) Dec2015 2

[29] A Jain H S Koppula B Raghavan S Soh and A Saxena Car thatknows before you do Anticipating maneuvers via learning temporaldriving models In Proceedings of the IEEE International Conferenceon Computer Vision 2015 1

[30] M Jiang S Huang J Duan and Q Zhao Salicon Saliency in contextIn The IEEE Conference on Computer Vision and Pattern Recognition(CVPR) June 2015 1 2

[31] A Karpathy G Toderici S Shetty T Leung R Sukthankar and L Fei-Fei Large-scale video classification with convolutional neural networksIn Proceedings of the IEEE conference on Computer Vision and PatternRecognition 2014 5

[32] D Kingma and J Ba Adam A method for stochastic optimization InInternational Conference on Learning Representations (ICLR) 2015 9

[33] P Kumar M Perrollaz S Lefevre and C Laugier Learning-based

approach for online lane change intention prediction In IntelligentVehicles Symposium (IV) 2013 IEEE IEEE 2013 1

[34] M Kuumlmmerer L Theis and M Bethge Deep gaze I boosting saliencyprediction with feature maps trained on imagenet In InternationalConference on Learning Representations Workshops (ICLRW) 2015 2

[35] M Kuumlmmerer T S Wallis and M Bethge Information-theoreticmodel comparison unifies saliency metrics Proceedings of the NationalAcademy of Sciences 112(52) 2015 8

[36] A M Larson and L C Loschky The contributions of central versusperipheral vision to scene gist recognition Journal of Vision 9(10)2009 12

[37] N Liu J Han D Zhang S Wen and T Liu Predicting eye fixationsusing convolutional neural networks In Computer Vision and PatternRecognition (CVPR) 2015 IEEE Conference on June 2015 2

[38] D G Lowe Object recognition from local scale-invariant features InComputer vision 1999 The proceedings of the seventh IEEE interna-tional conference on volume 2 Ieee 1999 3

[39] Y-F Ma and H-J Zhang Contrast-based image attention analysis byusing fuzzy growing In Proceedings of the Eleventh ACM InternationalConference on Multimedia MULTIMEDIA rsquo03 New York NY USA2003 ACM 2

[40] S Mannan K Ruddock and D Wooding Fixation sequences madeduring visual examination of briefly presented 2d images Spatial vision11(2) 1997 4

[41] S Mathe and C Sminchisescu Actions in the eye Dynamic gazedatasets and learnt saliency models for visual recognition IEEE Trans-actions on Pattern Analysis and Machine Intelligence 37(7) July 20152

[42] T Mauthner H Possegger G Waltner and H Bischof Encoding basedsaliency detection for videos and images In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition 2015 2

[43] B Morris A Doshi and M Trivedi Lane change intent prediction fordriver assistance On-road design and evaluation In Intelligent VehiclesSymposium (IV) 2011 IEEE IEEE 2011 1

[44] A Mousavian D Anguelov J Flynn and J Kosecka 3d bounding boxestimation using deep learning and geometry In CVPR 2017 1

[45] D Nilsson and C Sminchisescu Semantic video segmentation by gatedrecurrent flow propagation arXiv preprint arXiv161208871 2016 1

[46] A Palazzi F Solera S Calderara S Alletto and R Cucchiara Whereshould you attend while driving In IEEE Intelligent Vehicles SymposiumProceedings 2017 9 10

[47] P Pan Z Xu Y Yang F Wu and Y Zhuang Hierarchical recurrentneural encoder for video representation with application to captioningIn Proceedings of the IEEE Conference on Computer Vision and PatternRecognition 2016 6

[48] J S Perry and W S Geisler Gaze-contingent real-time simulationof arbitrary visual fields In Human vision and electronic imagingvolume 57 2002 12 15

[49] R J Peters and L Itti Beyond bottom-up Incorporating task-dependentinfluences into a computational model of spatial attention In ComputerVision and Pattern Recognition 2007 CVPRrsquo07 IEEE Conference onIEEE 2007 2

[50] R J Peters and L Itti Applying computational tools to predict gazedirection in interactive visual environments ACM Transactions onApplied Perception (TAP) 5(2) 2008 2

[51] M I Posner R D Rafal L S Choate and J Vaughan Inhibitionof return Neural basis and function Cognitive neuropsychology 2(3)1985 4

[52] N Pugeault and R Bowden How much of driving is preattentive IEEETransactions on Vehicular Technology 64(12) Dec 2015 2 3 4

[53] J Rogeacute T Peacutebayle E Lambilliotte F Spitzenstetter D Giselbrecht andA Muzet Influence of age speed and duration of monotonous drivingtask in traffic on the driveracircAZs useful visual field Vision research44(23) 2004 5

[54] D Rudoy D B Goldman E Shechtman and L Zelnik-Manor Learn-ing video saliency from human gaze using candidate selection InProceedings of the IEEE Conference on Computer Vision and PatternRecognition 2013 2

[55] R RukAringaAumlUnas J Back P Curzon and A Blandford Formal mod-elling of salience and cognitive load Electronic Notes in TheoreticalComputer Science 208 2008 4

[56] J Sardegna S Shelly and S Steidl The encyclopedia of blindness andvision impairment Infobase Publishing 2002 5

[57] B SchAtildeulkopf J Platt and T Hofmann Graph-Based Visual SaliencyMIT Press 2007 2

[58] L Simon J P Tarel and R Bremond Alerting the drivers about roadsigns with poor visual saliency In Intelligent Vehicles Symposium 2009IEEE June 2009 2 4

[59] C S Stefan Mathe Actions in the eye Dynamic gaze datasets and learntsaliency models for visual recognition IEEE Transactions on Pattern

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

14

Analysis and Machine Intelligence 37 2015 2 9[60] B W Tatler The central fixation bias in scene viewing Selecting

an optimal viewing position independently of motor biases and imagefeature distributions Journal of vision 7(14) 2007 4

[61] B W Tatler M M Hayhoe M F Land and D H Ballard Eye guidancein natural vision Reinterpreting salience Journal of Vision 11(5) May2011 1 2

[62] A Tawari and M M Trivedi Robust and continuous estimation of drivergaze zone by dynamic analysis of multiple face videos In IntelligentVehicles Symposium Proceedings 2014 IEEE June 2014 2

[63] J Theeuwes Topndashdown and bottomndashup control of visual selection Actapsychologica 135(2) 2010 2

[64] A Torralba A Oliva M S Castelhano and J M Henderson Contextualguidance of eye movements and attention in real-world scenes the roleof global features in object search Psychological review 113(4) 20062

[65] D Tran L Bourdev R Fergus L Torresani and M Paluri Learningspatiotemporal features with 3d convolutional networks In 2015 IEEEInternational Conference on Computer Vision (ICCV) Dec 2015 5 6 7

[66] A M Treisman and G Gelade A feature-integration theory of attentionCognitive psychology 12(1) 1980 2

[67] Y Ueda Y Kamakura and J Saiki Eye movements converge onvanishing points during visual search Japanese Psychological Research59(2) 2017 5

[68] G Underwood K Humphrey and E van Loon Decisions about objectsin real-world scenes are influenced by visual saliency before and duringtheir inspection Vision Research 51(18) 2011 1 2 4

[69] F Vicente Z Huang X Xiong F D la Torre W Zhang and D LeviDriver gaze tracking and eyes off the road detection system IEEETransactions on Intelligent Transportation Systems 16(4) Aug 20152

[70] P Wang P Chen Y Yuan D Liu Z Huang X Hou and G CottrellUnderstanding convolution for semantic segmentation arXiv preprintarXiv170208502 2017 1

[71] P Wang and G W Cottrell Central and peripheral vision for scenerecognition A neurocomputational modeling explorationwang amp cottrellJournal of Vision 17(4) 2017 12

[72] W Wang J Shen and F Porikli Saliency-aware geodesic video objectsegmentation In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition 2015 2 9

[73] W Wang J Shen and L Shao Consistent video saliency using localgradient flow optimization and global refinement IEEE Transactions onImage Processing 24(11) 2015 2 9

[74] J M Wolfe Visual search Attention 1 1998 2[75] J M Wolfe K R Cave and S L Franzel Guided search an

alternative to the feature integration model for visual search Journalof Experimental Psychology Human Perception amp Performance 19892

[76] F Yu and V Koltun Multi-scale context aggregation by dilated convolu-tions In ICLR 2016 1 5 8 9

[77] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[78] Y Zhai and M Shah Visual attention detection in video sequencesusing spatiotemporal cues In Proceedings of the 14th ACM InternationalConference on Multimedia MM rsquo06 New York NY USA 2006 ACM2

[79] S-h Zhong Y Liu F Ren J Zhang and T Ren Video saliencydetection via dynamic consistent spatio-temporal attention modelling InAAAI 2013 2

Andrea Palazzi received the masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently PhD candidate within the ImageLab groupin Modena researching on computer vision anddeep learning for automotive applications

Davide Abati received the masterrsquos degree incomputer engineering from the University ofModena and Reggio Emilia in 2015 He is cur-rently working toward the PhD degree withinthe ImageLab group in Modena researching oncomputer vision and deep learning for image andvideo understanding

Simone Calderara received a computer engi-neering masterrsquos degree in 2005 and the PhDdegree in 2009 from the University of Modenaand Reggio Emilia where he is currently anassistant professor within the Imagelab groupHis current research interests include computervision and machine learning applied to humanbehavior analysis visual tracking in crowdedscenarios and time series analysis for forensicapplications He is a member of the IEEE

Francesco Solera obtained a masterrsquos degreein computer engineering from the University ofModena and Reggio Emilia in 2013 and a PhDdegree in 2017 His research mainly addressesapplied machine learning and social computervision

Rita Cucchiara received the masterrsquos degreein Electronic Engineering and the PhD degreein Computer Engineering from the University ofBologna Italy in 1989 and 1992 respectivelySince 2005 she is a full professor at the Univer-sity of Modena and Reggio Emilia Italy whereshe heads the ImageLab group and is Direc-tor of the SOFTECH-ICT research center Sheis currently President of the Italian Associationof Pattern Recognition (GIRPR) affilaited withIAPR She published more than 300 papers on

pattern recognition computer vision and multimedia and in particularin human analysis HBU and egocentric-vision The research carriedout spans on different application fields such as video-surveillanceautomotive and multimedia big data annotation Corrently she is AE ofIEEE Transactions on Multimedia and serves in the Governing Board ofIAPR and in the Advisory Board of the CVF

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

15

Supplementary MaterialHere we provide additional material useful to the understandingof the paper Additional multimedia are available at httpsndrplzgithubiodreyeve

7 DR(EYE)VE DATASET DESIGN

The following table reports the design the DR(eye)VE datasetThe dataset is composed of 74 sequences of 5 minutes eachrecorded under a variety of driving conditions Experimentaldesign played a crucial role in preparing the dataset to rule outspurious correlation between driver weather traffic daytime andscenario Here we report the details for each sequence

8 VISUAL ASSESSMENT DETAILS

The aim of this section is to provide additional details on theimplementation of visual assessment presented in Sec 53 of thepaper Please note that additional videos regarding this sectioncan be found together with other supplementary multimedia athttpsndrplzgithubiodreyeve Eventually the reader is referredto httpsgithubcomndrplzdreyeve for the code used to createfoveated videos for visual assessment

Space Variant Imaging SystemSpace Variant Imaging System (SVIS) is a MATLAB toolboxthat allows to foveate images in real-time [48] which has beenused in a large number of scientific works to approximate humanfoveal vision since its introduction in 2002 In this frame theterm foveated imaging refers to the creation and display of staticor video imagery where the resolution varies across the imageIn analogy to human foveal vision the highest resolution regionis called the foveation region In a video the location of thefoveation region can obviously change dynamically It is alsopossible to have more than one foveation region in each image

The foveation process is implemented in the SVIS toolboxas follows first the the input image is repeatedly low-passedfiltered and down-sampled to half of the current resolution by aFoveation Encoder In this way a low-pass pyramid of images isobtained Then a foveation pyramid is created selecting regionsfrom different resolutions proportionally to the distance fromthe foveation point Concretely the foveation region will be atthe highest resolution first ring around the foveation region willbe taken from half-resolution image and so on Eventually aFoveation Decoder up-sample interpolate and blend each layer inthe foveation pyramid to create the output foveated image

The software is open-source and publicly available here httpsvicpsutexasedusoftwareshtml The interested reader is referred tothe SVIS website for further details

Videoclip FoveationFrom fixation maps back to fixations The SVIS toolbox allowsto foveate images starting from a list of (x y) coordinates whichrepresent the foveation points in the given image (please seeFig 18 for details) However we do not have this information asin our work we deal with continuous attentional maps rather than

TABLE 5DR(eye)VE train set details for each sequence

Sequence Daytime Weather Landscape Driver Set01 Evening Sunny Countryside D8 Train Set02 Morning Cloudy Highway D2 Train Set03 Evening Sunny Highway D3 Train Set04 Night Sunny Downtown D2 Train Set05 Morning Cloudy Countryside D7 Train Set06 Morning Sunny Downtown D7 Train Set07 Evening Rainy Downtown D3 Train Set08 Evening Sunny Countryside D1 Train Set09 Night Sunny Highway D1 Train Set10 Evening Rainy Downtown D2 Train Set11 Evening Cloudy Downtown D5 Train Set12 Evening Rainy Downtown D1 Train Set13 Night Rainy Downtown D4 Train Set14 Morning Rainy Highway D6 Train Set15 Evening Sunny Countryside D5 Train Set16 Night Cloudy Downtown D7 Train Set17 Evening Rainy Countryside D4 Train Set18 Night Sunny Downtown D1 Train Set19 Night Sunny Downtown D6 Train Set20 Evening Sunny Countryside D2 Train Set21 Night Cloudy Countryside D3 Train Set22 Morning Rainy Countryside D7 Train Set23 Morning Sunny Countryside D5 Train Set24 Night Rainy Countryside D6 Train Set25 Morning Sunny Highway D4 Train Set26 Morning Rainy Downtown D5 Train Set27 Evening Rainy Downtown D6 Train Set28 Night Cloudy Highway D5 Train Set29 Night Cloudy Countryside D8 Train Set30 Evening Cloudy Highway D7 Train Set31 Morning Rainy Highway D8 Train Set32 Morning Rainy Highway D1 Train Set33 Evening Cloudy Highway D4 Train Set34 Morning Sunny Countryside D3 Train Set35 Morning Cloudy Downtown D3 Train Set36 Evening Cloudy Countryside D1 Train Set37 Morning Rainy Highway D8 Train Set

discrete points of fixations To be able to use the same softwareAPI we need to regress from the attentional map (either true orpredicted) a list of approximated yet plausible fixation locationsTo this aim we simply extract the 25 points with highest valuein the attentional map This is justified by the fact that in thephase of dataset creation the ground truth fixation map Ft for aframe at time t is built by accumulating projected gaze points ina temporal sliding window of k = 25 frames centered in t (seeSec 3 of the paper) The output of this phase is thus a fixationmap we can use as input for the SVIS toolbox

Taking the blurred-deblurred ratio into account To the visualassessment purposes keeping track the amount of blur that avideoclip has undergone is also relevant Indeed a certain videomay give rise to higher perceived safety only because a moredelicate blur allows the subject to see a clearer picture of thedriving scene In order to consider this phenomenon we do thefollowingGiven an input image I isin Rhwc the output of the FoveationEncoder is a resolution map Rmap isin Rhw1 taking value inrange [0 255] as depicted in Fig 18 (b) Each value indicatesthe resolution that a certain pixel will have in the foveated imageafter decoding where 0 and 255 indicates minimum and maximumresolution respectively

16

(a) (b) (c)

Fig 18 Foveation process using SVIS software is depicted here Starting from one or more fixation points in a given frame (a) a smooth resolutionmap is built (b) Image locations with higher values in the resolution map will undergo less blur in the output image (c)

TABLE 6DR(eye)VE test set details for each sequence

Sequence Daytime Weather Landscape Driver Set38 Night Sunny Downtown D8 Test Set39 Night Rainy Downtown D4 Test Set40 Morning Sunny Downtown D1 Test Set41 Night Sunny Highway D1 Test Set42 Evening Cloudy Highway D1 Test Set43 Night Cloudy Countryside D2 Test Set44 Morning Rainy Countryside D1 Test Set45 Evening Sunny Countryside D4 Test Set46 Evening Rainy Countryside D5 Test Set47 Morning Rainy Downtown D7 Test Set48 Morning Cloudy Countryside D8 Test Set49 Morning Cloudy Highway D3 Test Set50 Morning Rainy Highway D2 Test Set51 Night Sunny Downtown D3 Test Set52 Evening Sunny Highway D7 Test Set53 Evening Cloudy Downtown D7 Test Set54 Night Cloudy Highway D8 Test Set55 Morning Sunny Countryside D6 Test Set56 Night Rainy Countryside D6 Test Set57 Evening Sunny Highway D5 Test Set58 Night Cloudy Downtown D4 Test Set59 Morning Cloudy Highway D7 Test Set60 Morning Cloudy Downtown D5 Test Set61 Night Sunny Downtown D5 Test Set62 Night Cloudy Countryside D6 Test Set63 Morning Rainy Countryside D8 Test Set64 Evening Cloudy Downtown D8 Test Set65 Morning Sunny Downtown D2 Test Set66 Evening Sunny Highway D6 Test Set67 Evening Cloudy Countryside D3 Test Set68 Morning Cloudy Countryside D4 Test Set69 Evening Rainy Highway D2 Test Set70 Morning Rainy Downtown D3 Test Set71 Night Cloudy Highway D6 Test Set72 Evening Cloudy Downtown D2 Test Set73 Night Sunny Countryside D7 Test Set74 Morning Rainy Downtown D4 Test Set

For each video v we measure video average resolution afterfoveation as follows

vres =1

N

Nsumf=1

sumi

Rmap(i f)

where N is the number of frames in the video (1000 in our setting)and Rmap(i f) denotes the ith pixel of the resolution mapcorresponding to the f th frame of the input video The higher thevalue of vres the more information is preserved in the foveationprocess Due to the sparser location of fixations in ground truth

attentional maps these result in much less blurred videoclipsIndeed videos foveated with model predicted attentional mapshave in average only the 38 of the resolution wrt videosfoveated starting from ground truth attentional maps Despite thisbias model predicted foveated videos still gave rise to higherperceived safety to assessment participants

9 PERCEIVED SAFETY ASSESSMENT

The assessment of predicted fixation maps described in Sec 53has also been carried out for validating the model in terms ofperceived safety Indeed partecipants were also asked to answerthe following questionbull If you were sitting in the same car of the driver whose

attention behavior you just observed how safe would youfeel (rate from 1 to 5)

The aim of the question is to measure the comfort level of theobserver during a driving experience when suggested to focusat specific locations in the scene The underlying assumption isthat the observer is more likely to feel safe if he agrees thatthe suggested focus is lighting up the right portion of the scenethat is what he thinks it is worth looking in the current drivingscene Conversely if the observer wishes to focus at some specificlocation but he cannot retrieve details there he is going to feeluncomfortableThe answers provided by subjects summarized in Fig 19 indicatethat perceived safety for videoclips foveated using the attentionalmaps predicted by the model is generally higher than for the onesfoveated using either human or central baseline maps Nonethelessthe central bias baseline proves to be extremely competitive inparticular in non-acting videoclips in which it scores similarly tothe model prediction It is worth noticing that in this latter caseboth kind of automatic predictions outperform human ground truthby a significant margin (Fig 19b) Conversely when we consideronly the foveated videoclips containing acting subsequences thehuman ground truth is perceived as much safer than the baselinedespite still scores worse than our model prediction (Fig 19c)These results hold despite due to the localization of the fixationsthe average resolution of the predicted maps is only the 38of the resolution of ground truth maps (ie videos foveatedusing prediction map feature much less information) We didnot measure significant difference in perceived safety across thedifferent drivers in the dataset (σ2 = 009) We report in Fig 20the composition of each score in terms of answers to the othervisual assessment question (ldquoWould you say the observed attention

17

(a) (b) (c)

Fig 19 Distributions of safeness scores for different map sources namely Model Prediction Center Baseline and Human Groundtruth Consideringthe score distribution over all foveated videoclips (a) the three distributions may look similar even though the model prediction still scores slightlybetter However when considering only the foveated videos contaning acting subsequences (b) the model prediction significantly outperformsboth center baseline and human groundtruth Conversely when the videoclips did not contain acting subsequences (ie the car was mostly goingstraight) the fixation map from human driver is the one perceived as less safe while both model prediction and center baseline perform similarly

Score Composition

1 2 3 4 50

1TP

TN

FP

FN

Safeness Score

Prob

abili

ty

Fig 20 The stacked bar graph represents the ratio of TP TN FP and FNcomposing each score The increasing score of FP ndash participants falselythought the attentional map came from a human driver ndash highlightsthat participants were tricked into believing that safer clips came fromhumans

behavior comes from a human driver (yesno)rdquo) This analysisaims to measure participantsrsquo bias towards human driving abilityIndeed increasing trend of false positives towards higher scoressuggests that participants were tricked into believing that ldquosaferrdquoclips came from humans The reader is referred to Fig 20 forfurther details

10 DO SUBTASKS HELP IN FOA PREDICTION

The driving task is inherently composed of many subtasks suchas turning or merging in traffic looking for parking and soon While such fine-grained subtasks are hard to discover (andprobably to emerge during learning) due to scarcity here weshow how the proposed model has been able to leverage on morecommon subtask to get to the final prediction These subtasks areturning leftright going straight being still We gathered automaticannotation through GPS information released with the datasetWe then train a linear SVM classifier to distinguish the above4 different actions starting from the activations of the last layerof multi-path model unrolled in a feature vector The SVMclassifier scores a 90 of accuracy on the test set (5000 uniformelysampled videoclips) supporting the fact that network activationsare highly discriminative for distinguishing the different drivingsubtasks Please refer to Fig 21 for further details Code to repli-cate this result is available at httpsgithubcomndrplzdreyevealong with the code of all other experiments in the paper

Fig 21 Confusion matrix for SVM classifier trained to distinguish drivingactions from network activations The accuracy is generally high whichcorroborates the assumption that the model benefits from learning aninternal representation of the different driving sub-tasks

11 SEGMENTATION

In this section we report exemplar cases that particularly benefitfrom the segmentation branch In Fig 22 we can appreciate thatamong the three branches only the semantic one captures the realgaze that is focused on traffic lights and street signs

12 ABLATION STUDY

In Fig 23 we showcase several examples depicting the contribu-tion of each branch of the multi-branch model in predictingthe visual focus of attention of the driver As expected the RGBbranch is the one that more heavily influences the overall networkoutput

13 ERROR ANALYSIS FOR NON-PLANAR HOMO-GRAPHIC PROJECTION

A homography H is a projective transformation from a plane Ato another plane B such that the collinearity property is preservedduring the mapping In real world applications the homographymatrix H is often computed through an overdetermined set ofimage coordinates lying on the same implicit plane aligning

17

(a) (b) (c)

Fig 19 Distributions of safeness scores for different map sources namely Model Prediction Center Baseline and Human Groundtruth Consideringthe score distribution over all foveated videoclips (a) the three distributions may look similar even though the model prediction still scores slightlybetter However when considering only the foveated videos contaning acting subsequences (b) the model prediction significantly outperformsboth center baseline and human groundtruth Conversely when the videoclips did not contain acting subsequences (ie the car was mostly goingstraight) the fixation map from human driver is the one perceived as less safe while both model prediction and center baseline perform similarly

Score Composition

1 2 3 4 50

1TP

TN

FP

FN

Safeness Score

Prob

abili

ty

Fig 20 The stacked bar graph represents the ratio of TP TN FP and FNcomposing each score The increasing score of FP ndash participants falselythought the attentional map came from a human driver ndash highlightsthat participants were tricked into believing that safer clips came fromhumans

behavior comes from a human driver (yesno)rdquo) This analysisaims to measure participantsrsquo bias towards human driving abilityIndeed increasing trend of false positives towards higher scoressuggests that participants were tricked into believing that ldquosaferrdquoclips came from humans The reader is referred to Fig 20 forfurther details

10 DO SUBTASKS HELP IN FOA PREDICTION

The driving task is inherently composed of many subtasks suchas turning or merging in traffic looking for parking and soon While such fine-grained subtasks are hard to discover (andprobably to emerge during learning) due to scarcity here weshow how the proposed model has been able to leverage on morecommon subtask to get to the final prediction These subtasks areturning leftright going straight being still We gathered automaticannotation through GPS information released with the datasetWe then train a linear SVM classifier to distinguish the above4 different actions starting from the activations of the last layerof multi-path model unrolled in a feature vector The SVMclassifier scores a 90 of accuracy on the test set (5000 uniformelysampled videoclips) supporting the fact that network activationsare highly discriminative for distinguishing the different drivingsubtasks Please refer to Fig 21 for further details Code to repli-cate this result is available at httpsgithubcomndrplzdreyevealong with the code of all other experiments in the paper

Fig 21 Confusion matrix for SVM classifier trained to distinguish drivingactions from network activations The accuracy is generally high whichcorroborates the assumption that the model benefits from learning aninternal representation of the different driving sub-tasks

11 SEGMENTATION

In this section we report exemplar cases that particularly benefitfrom the segmentation branch In Fig 22 we can appreciate thatamong the three branches only the semantic one captures the realgaze that is focused on traffic lights and street signs

12 ABLATION STUDY

In Fig 23 we showcase several examples depicting the contribu-tion of each branch of the multi-branch model in predictingthe visual focus of attention of the driver As expected the RGBbranch is the one that more heavily influences the overall networkoutput

13 ERROR ANALYSIS FOR NON-PLANAR HOMO-GRAPHIC PROJECTION

A homography H is a projective transformation from a plane Ato another plane B such that the collinearity property is preservedduring the mapping In real world applications the homographymatrix H is often computed through an overdetermined set ofimage coordinates lying on the same implicit plane aligning

18

RGB flow segmentation gt

Fig 22 Some examples of the beneficial effect of the semantic segmentation branch In the two cases depicted here the car is stopped at acrossroad While the RGB branch remains biased towards the road vanishing point and the optical flow branch focuses on moving objects thesemantic branch tends to highlight traffic lights and signals coherently with the human behavior

Input frame GT multi-branch RGB FLOW SEG

Fig 23 Example cases that qualitatively show how each branch contribute to the final prediction Best viewed on screen

points on the plane in one image with points on the plane inthe other image If the input set of points is approximately lyingon the true implicit plane then H can be efficiently recoveredthrough least square projection minimization

Once the transformation H has been either defined or approx-imated from data to map an image point x from the first image tothe respective point Hx in the second image the basic assumptionis that x actually lies on the implicit plane In practice thisassumption is widely violated in real world applications when theprocess of mapping is automated and the content of the mappingis not known a-priori

131 The geometry of the problemIn Fig 24 we show the generic setting of two cameras capturingthe same 3D plane To construct an erroneous case study weput a cylinder on top of the plane Points on the implicit 3Dworld plane can be consistently mapped across views with anhomography transformation and retain their original semantic Asan example the point x1 is the center of the cylinder base bothin world coordinates and across different views Conversely thepoint x2 on the top of the cylinder cannot be consistently mapped

from one view to the other To see why suppose we want to mapx2 from view B to view A Since the homography assumes x2

to also be on the implicit plane its inferred 3D position is farfrom the true top of the cylinder and is depicted with the leftmostempty circle in Fig 24 When this point gets reprojected to viewA its image coordinates are unaligned with the correct position ofthe cylinder top in that image We call this offset the reprojectionerror on plane A or eA Analogously a reprojection error onplane B could be computed with an homographic projection ofpoint x2 from view A to view B

The reprojection error is useful to measure the perceptualmisalignment of projected points with their intended locations butdue to the (re)projections involved is not an easy tool to work withMoreover the very same point can produce different reprojectionerrors when measured on A and on B A related error also arisingin this setting is the metric error eW or the displacement inworld space of the projected image points at the intersection withthe implicit plane This measure of error is of particular interestbecause it is view-independent does not depend on the rotation ofthe cameras with respect to the plane and is zero if and only if the

19

B

(a) (b)

Fig 24 (a) Two image planes capture a 3D scene from different viewpoints and (b) a use case of the bound derived below

reprojection error is also zero

132 Computing the metric errorSince the metric error does not depend on the mutual rotationof the plane with the camera views we can simplify Fig 24 byretaining only the optical centers A and B from all cameras andby setting without loss of generality the reference system on theprojection of the 3D point on the plane This second step is usefulto factor out the rotation of the world plane which is unknownin the general setting The only assumption we make is that thenon-planar point x2 can be seen from both camera views Thissimplification is depicted in Fig 25(a) where we have also namedseveral important quantities such as the distance h of x2 from theplane

In Fig 25(a) the metric error can be computed as the magni-tude of the difference between the two vectors relating points xa2and xb2 to the origin

ew = xa2 minus xb2 (7)

The aforementioned points are at the intersection of the linesconnecting the optical center of the cameras with the 3D point x2

and the implicit plane An easy way to get such points is throughtheir magnitude and orientation As an example consider the pointxa2 Starting from xa2 the following two similar triangles can bebuilt

∆xa2paA sim ∆xa2Ox2 (8)

Since they are similar ie they share the same shape we canmeasure the distance of xa2 from the origin More formally

Lapa+ xa2

=h

xa2 (9)

from which we can recover

xa2 =hpaLa minus h

(10)

The orientation of the xa2 vector can be obtained directly from theorientation of the pa vector which is known and equal to

rdquo

xa2 = minus rdquopa = minus papa

(11)

Eventually with the magnitude and orientation in place we canlocate the vector pointing to xa2

xa2 = xa2 rdquo

xa2 = minus h

La minus hpa (12)

Similarly xb2 can also be computed The metric error can thus bedescribed by the following relation

ew = h

(pb

Lb minus hminus paLa minus h

) (13)

The error ew is a vector but a convenient scalar can be obtainedby using the preferred norm

133 Computing the error on a camera reference sys-temWhen the plane inducing the homography remains unknownthe bound and the error estimation from the previous sectioncannot be directly applied A more general case is obtained if thereference system is set off the plane and in particular on oneof the cameras The new geometry of the problem is shown inFig 25(b) where the reference system is placed on camera AIn this setting the metric error is a function of four independentquantities (highlighted in red in the figure) i) the point x2 ii) thedistance of such point from the inducing plane h iii) the planenormal rdquon and iv) the distance between the cameras v which isalso equal to the position of camera B

To this end starting from Eq (13) we are interested in expressingpb pa Lb and La in terms of this new reference system Sincepa is the projection of A on the plane it can also be defined as

pa = Aminus (Aminus pk) rdquon otimes rdquon = A+ α rdquon otimes rdquon (14)

where rdquon is the plane normal pk is an arbitrary point on theplane that we set to x2 minus h otimes rdquon ie the projection of x2 on theplane To ease the readability of the following equations α =minus(Aminus x2 minus hotimes rdquon) Now if v describes the distance from A toB we have

pb = A+ v minus (A+ v minus pk) rdquon otimes rdquon

= A+ α rdquon otimes rdquon + v minus v rdquon otimes rdquon (15)

Through similar reasoning La and Lb are also rewritten asfollows

La = Aminus pa = minusα rdquon otimes rdquon

Lb = B minus pb = minusα rdquon otimes rdquon + v rdquon otimes rdquon (16)

Eventually by substituting Eq (14)-(16) in Eq (13) and by fixingthe origin on the location of camera A so that A = (0 0 0)T wehave

ew =hotimes v

(x2 minus v) rdquon

(rdquon otimes rdquon(1minus 1

1minus hhminusx2

rdquon

)minus I) (17)

20

119859119859

119858

119847119847

x2

h

γ

hh

θ

(a) (b) (c)

Fig 25 (a) By aligning the reference system with the plane normal centered on the projection of the non-planar point onto the plane the metric erroris the magnitude of the difference between the two vectors ~xa

2 and ~xb2 The red lines help to highlight the similarity of inner and outer triangles having

xax as a vertex (b) The geometry of the problem when the reference system is placed off the plane in an arbitrary position (gray) or specifically on

one of the camera (black) (c) The simplified setting in which we consider the projection of the metric error ew on the camera plane of A

Notably the vector v and the scalar h both appear as multiplicativefactors in Eq (17) so that if any of them goes to zero then themagnitude of the metric error ew also goes to zeroIf we assume that h 6= 0 we can go one step further and obtaina formulation were x2 and v are always divided by h suggestingthat what really matters is not the absolute position of x2 orcamera B with respect to camera A but rather how many timesfurther x2 and camera B are from A than x2 from the plane Suchrelation is made explicit below

ew =hvh

(x2h otimes rdquox2 minus vh otimes

rdquov ) rdquon︸ ︷︷ ︸Q

otimes (18)

rdquov

(rdquon otimes rdquon (1minus 1

1minus 1

1minus x2h otimes rdquox 2

rdquon

)

︸ ︷︷ ︸Z

minusI) (19)

134 Working towards a bound

Let M = x2v| cos θ| being θ the angle between rdquox2 andrdquon and let β be the angle between rdquov and rdquon Then Q can berewritten as Q = (M minuscosβ)minus1 Note that under the assumptionthat M ge 2 Q le 1 always holds Indeed for M ge 2 to holdwe need to require x2 ge 2v| cos θ| Next consider thescalar Z it is easy to verify that if |x2h otimes rdquox2

rdquon | gt 1 then|Z| le 1 Since both rdquox2 and rdquon are versors the magnitude of theirdot product is at most one It follows that |Z| lt 1 if and only ifx2 gt h Now we are left with a versor rdquov that multiplies thedifference of two matrices If we compute such product we obtaina new vector with magnitude less or equal to one rdquov rdquon otimes rdquon andthe versor rdquov The difference of such vectors is at most 2 Summingup all the presented considerations we have that the magnitude ofthe error is bounded as follows

Observation 1If x2 ge 2v| cos θ| and x2 gt h thenew le 2h

We now aim to derive a projection error bound from theabove presented metric error bound In order to do so we need

to introduce the focal length of the camera f For simplicity wersquollassume that f = fx = fy First we simplify our setting withoutloosing the upper bound constraint To do so we consider theworst case scenario in which the mutual position of the plane andthe camera maximizes the projected errorbull the plane rotation is so that rdquon zbull the error segment is just in front of the camerabull the plane rotation along the z axis is such that the parallel

component of the error wrt the x axis is zero (this allowsus to express the segment end points with simple coordinateswithout loosing generality)

bull the camera A falls on the middle point of the error segmentIn the simplified scenario depicted in Fig 25(c) the projectionof the error is maximized In this case the two points we wantto project are p1 = [0minush γ] and p2 = [0 h γ] (we considerthe case in which ||ew|| = 2h see Observation 1) where γ isthe distance of the camera from the plane Considering the focallength f of camera A p1 and p2 are projected as follows

KAp1 =

f 0 0 00 f 0 00 0 1 0

0minushγ

=

0minusfhγ

rarr [0

minus fhγ

](20)

KAp2 =

f 0 0 00 f 0 00 0 1 0

0hγ

=

0fhγ

rarr [0fhγ

](21)

Thus the magnitude of the projection ea of the metric errorew is bounded by 2fh

γ Now we notice that γ = h+ x2

rdquon = h+ x2cos(θ) so

ea le2fh

γ=

2fh

h+ x2cos(θ)=

2f

1 + x2h cos(θ)

(22)

Notably the right term of the equation is maximized whencos(θ) = 0 (since when cos(θ) lt 0 the point is behind thecamera which is impossible in our setting) Thus we obtain thatea le 2f

Fig 24(b) shows a use case of the bound in Eq 22 It shows valuesof θ up to pi2 where the presented bound simplifies to ea le2f (dashed black line) In practice if we require i) θ le π3 andii) that the camera-object distance x2 is at least three times the

21

800 600 400 200 0 200 400 600 800Shift (px)

0

1

2

3

4

5

6

7

DKL

Central cropRandom crop

Fig 26 Performances of the multi-branch model trained with ran-dom and central crop policies in presence of horizontal shifts in inputclips Colored background regions highlight the area within the standarddeviation of the measured divergence Recall that a lower value of DKL

means a more accurate prediction

plane-object distance h and if we let f = 350px then the erroris always lower than 200px which translate to a precision up to20 of an image at 1080p resolution

14 THE EFFECT OF RANDOM CROPPING

In order to advocate for the peculiar training strategy illustratedin Sec 41 involving two streams processing both resized clipsand randomly cropped clips we perform an additional experimentas follows We first re-train our multi-branch architecturefollowing the same procedures explained in the main paper exceptfor the cropping of input clips and groundtruth maps which isalways central rather than random At test time we shift each inputclip in the range [minus800 800] pixels (negative and positive shiftsindicate left and right translations respectively) After the trans-lation we apply mirroring to fill borders on the opposite side ofthe shift direction We perform the same operation on groundtruthmaps and report the mean DKL of the multi-branch modelwhen trained with random and central crops as a function of thetranslation size in Fig 26 The figure highlights how randomcropping consistently outperforms central cropping Importantlythe gap in performance increases with amount of shift appliedfrom a 2786 relative DKL difference when no translation isperformed to 25813 and 21617 increases for minus800 px and800 px translations suggesting the model trained with centralcrops is not robust to positional shifts

15 THE EFFECT OF PADDED CONVOLUTIONS INLEARNING A CENTRAL BIAS

Learning a globally localized solution seems theoretically impos-sible when using a fully convolutional architecture Indeed convo-lutional kernels are applied uniformly along all spatial dimensionof the input feature map Conversely a globally localized solutionrequires knowing where kernels are applied during convolutionsWe argue that a convolutional element can know its absolute posi-tion if there are latent statistics contained in the activations of theprevious layer In what follows we show how the common habitof padding feature maps before feeding them to convolutional

layers in order to maintain borders is an underestimated source ofspatially localized statistics Indeed padded values are always con-stant and unrelated to the input feature map Thus a convolutionaloperation depending on its receptive field can localize itself in thefeature map by looking for statistics biased by the padding valuesTo validate this claim we design a toy experiment in which a fullyconvolutional neural network is tasked to regress a white centralsquare on a black background when provided with a uniformor a noisy input map (in both cases the target is independentfrom the input) We position the square (bias element) at thecenter of the output as it is the furthest position from borders iewhere the bias originates We perform the experiment with severalnetworks featuring the same number of parameters yet differentreceptive fields4 Moreover to advocate for the random croppingstrategy employed in the training phase of our network (recallthat it was introduced to prevent a saliency-branch to regress acentral bias) we repeat each experiment employing such strategyduring training All models were trained to minimize the meansquared error between target maps and predicted maps by meansof Adam optimizer5 The outcome of such experiments in terms ofregressed prediction maps and loss function value at convergenceare reported in Fig 27 and Fig 28 As shown by the top-mostregion of both figures despite the uncorrelation between inputand target all models can learn a central biased map Moreoverthe receptive field plays a crucial role as it controls the amount ofpixels able to ldquolocalize themselvesrdquo within the predicted map Asthe receptive field of the network increases the responsive areashrinks to the groundtruth area and loss value lowers reachingzero For an intuition of why this is the case we refer the readerto Fig 29 Conversely as clearly emerges from the bottom-mostregion of both figures random cropping prevents the model toregress a biased map regardless the receptive field of the networkThe reason underlying this phenomenon is that padding is appliedafter the crop so its relative position with respect to the targetdepends on the crop location which is random

16 ON FIG 8We represent in Fig 30 the process employed for building thehistogram in Fig8 (in the paper) Given a segmentation map of aframe and the corresponding ground-truth fixation map we collectpixel classes within the area of fixation thresholded at differentlevels as the threshold increases the area shrinks to the realfixation point A better visualization of the process can be foundat httpsndrplzgithubiodreyeve

17 FAILURE CASES

In Fig 31 we report several clips in which our architecture fails tocapture the groundtruth human attention

4 a simple way to decrease the receptive field without changing the numberof parameters is to replace two convolutional layers featuring C outputchannels into a single one featuring 2C output channels

5 the code to reproduce this experiment is publicly released at httpsgithubcomDavideAcan_i_learn_central_bias

19

B

(a) (b)

Fig 24 (a) Two image planes capture a 3D scene from different viewpoints and (b) a use case of the bound derived below

reprojection error is also zero

132 Computing the metric errorSince the metric error does not depend on the mutual rotationof the plane with the camera views we can simplify Fig 24 byretaining only the optical centers A and B from all cameras andby setting without loss of generality the reference system on theprojection of the 3D point on the plane This second step is usefulto factor out the rotation of the world plane which is unknownin the general setting The only assumption we make is that thenon-planar point x2 can be seen from both camera views Thissimplification is depicted in Fig 25(a) where we have also namedseveral important quantities such as the distance h of x2 from theplane

In Fig 25(a) the metric error can be computed as the magni-tude of the difference between the two vectors relating points xa2and xb2 to the origin

ew = xa2 minus xb2 (7)

The aforementioned points are at the intersection of the linesconnecting the optical center of the cameras with the 3D point x2

and the implicit plane An easy way to get such points is throughtheir magnitude and orientation As an example consider the pointxa2 Starting from xa2 the following two similar triangles can bebuilt

∆xa2paA sim ∆xa2Ox2 (8)

Since they are similar ie they share the same shape we canmeasure the distance of xa2 from the origin More formally

Lapa+ xa2

=h

xa2 (9)

from which we can recover

xa2 =hpaLa minus h

(10)

The orientation of the xa2 vector can be obtained directly from theorientation of the pa vector which is known and equal to

rdquo

xa2 = minus rdquopa = minus papa

(11)

Eventually with the magnitude and orientation in place we canlocate the vector pointing to xa2

xa2 = xa2 rdquo

xa2 = minus h

La minus hpa (12)

Similarly xb2 can also be computed The metric error can thus bedescribed by the following relation

ew = h

(pb

Lb minus hminus paLa minus h

) (13)

The error ew is a vector but a convenient scalar can be obtainedby using the preferred norm

133 Computing the error on a camera reference sys-temWhen the plane inducing the homography remains unknownthe bound and the error estimation from the previous sectioncannot be directly applied A more general case is obtained if thereference system is set off the plane and in particular on oneof the cameras The new geometry of the problem is shown inFig 25(b) where the reference system is placed on camera AIn this setting the metric error is a function of four independentquantities (highlighted in red in the figure) i) the point x2 ii) thedistance of such point from the inducing plane h iii) the planenormal rdquon and iv) the distance between the cameras v which isalso equal to the position of camera B

To this end starting from Eq (13) we are interested in expressingpb pa Lb and La in terms of this new reference system Sincepa is the projection of A on the plane it can also be defined as

pa = Aminus (Aminus pk) rdquon otimes rdquon = A+ α rdquon otimes rdquon (14)

where rdquon is the plane normal pk is an arbitrary point on theplane that we set to x2 minus h otimes rdquon ie the projection of x2 on theplane To ease the readability of the following equations α =minus(Aminus x2 minus hotimes rdquon) Now if v describes the distance from A toB we have

pb = A+ v minus (A+ v minus pk) rdquon otimes rdquon

= A+ α rdquon otimes rdquon + v minus v rdquon otimes rdquon (15)

Through similar reasoning La and Lb are also rewritten asfollows

La = Aminus pa = minusα rdquon otimes rdquon

Lb = B minus pb = minusα rdquon otimes rdquon + v rdquon otimes rdquon (16)

Eventually by substituting Eq (14)-(16) in Eq (13) and by fixingthe origin on the location of camera A so that A = (0 0 0)T wehave

ew =hotimes v

(x2 minus v) rdquon

(rdquon otimes rdquon(1minus 1

1minus hhminusx2

rdquon

)minus I) (17)

20

119859119859

119858

119847119847

x2

h

γ

hh

θ

(a) (b) (c)

Fig 25 (a) By aligning the reference system with the plane normal centered on the projection of the non-planar point onto the plane the metric erroris the magnitude of the difference between the two vectors ~xa

2 and ~xb2 The red lines help to highlight the similarity of inner and outer triangles having

xax as a vertex (b) The geometry of the problem when the reference system is placed off the plane in an arbitrary position (gray) or specifically on

one of the camera (black) (c) The simplified setting in which we consider the projection of the metric error ew on the camera plane of A

Notably the vector v and the scalar h both appear as multiplicativefactors in Eq (17) so that if any of them goes to zero then themagnitude of the metric error ew also goes to zeroIf we assume that h 6= 0 we can go one step further and obtaina formulation were x2 and v are always divided by h suggestingthat what really matters is not the absolute position of x2 orcamera B with respect to camera A but rather how many timesfurther x2 and camera B are from A than x2 from the plane Suchrelation is made explicit below

ew =hvh

(x2h otimes rdquox2 minus vh otimes

rdquov ) rdquon︸ ︷︷ ︸Q

otimes (18)

rdquov

(rdquon otimes rdquon (1minus 1

1minus 1

1minus x2h otimes rdquox 2

rdquon

)

︸ ︷︷ ︸Z

minusI) (19)

134 Working towards a bound

Let M = x2v| cos θ| being θ the angle between rdquox2 andrdquon and let β be the angle between rdquov and rdquon Then Q can berewritten as Q = (M minuscosβ)minus1 Note that under the assumptionthat M ge 2 Q le 1 always holds Indeed for M ge 2 to holdwe need to require x2 ge 2v| cos θ| Next consider thescalar Z it is easy to verify that if |x2h otimes rdquox2

rdquon | gt 1 then|Z| le 1 Since both rdquox2 and rdquon are versors the magnitude of theirdot product is at most one It follows that |Z| lt 1 if and only ifx2 gt h Now we are left with a versor rdquov that multiplies thedifference of two matrices If we compute such product we obtaina new vector with magnitude less or equal to one rdquov rdquon otimes rdquon andthe versor rdquov The difference of such vectors is at most 2 Summingup all the presented considerations we have that the magnitude ofthe error is bounded as follows

Observation 1If x2 ge 2v| cos θ| and x2 gt h thenew le 2h

We now aim to derive a projection error bound from theabove presented metric error bound In order to do so we need

to introduce the focal length of the camera f For simplicity wersquollassume that f = fx = fy First we simplify our setting withoutloosing the upper bound constraint To do so we consider theworst case scenario in which the mutual position of the plane andthe camera maximizes the projected errorbull the plane rotation is so that rdquon zbull the error segment is just in front of the camerabull the plane rotation along the z axis is such that the parallel

component of the error wrt the x axis is zero (this allowsus to express the segment end points with simple coordinateswithout loosing generality)

bull the camera A falls on the middle point of the error segmentIn the simplified scenario depicted in Fig 25(c) the projectionof the error is maximized In this case the two points we wantto project are p1 = [0minush γ] and p2 = [0 h γ] (we considerthe case in which ||ew|| = 2h see Observation 1) where γ isthe distance of the camera from the plane Considering the focallength f of camera A p1 and p2 are projected as follows

KAp1 =

f 0 0 00 f 0 00 0 1 0

0minushγ

=

0minusfhγ

rarr [0

minus fhγ

](20)

KAp2 =

f 0 0 00 f 0 00 0 1 0

0hγ

=

0fhγ

rarr [0fhγ

](21)

Thus the magnitude of the projection ea of the metric errorew is bounded by 2fh

γ Now we notice that γ = h+ x2

rdquon = h+ x2cos(θ) so

ea le2fh

γ=

2fh

h+ x2cos(θ)=

2f

1 + x2h cos(θ)

(22)

Notably the right term of the equation is maximized whencos(θ) = 0 (since when cos(θ) lt 0 the point is behind thecamera which is impossible in our setting) Thus we obtain thatea le 2f

Fig 24(b) shows a use case of the bound in Eq 22 It shows valuesof θ up to pi2 where the presented bound simplifies to ea le2f (dashed black line) In practice if we require i) θ le π3 andii) that the camera-object distance x2 is at least three times the

21

800 600 400 200 0 200 400 600 800Shift (px)

0

1

2

3

4

5

6

7

DKL

Central cropRandom crop

Fig 26 Performances of the multi-branch model trained with ran-dom and central crop policies in presence of horizontal shifts in inputclips Colored background regions highlight the area within the standarddeviation of the measured divergence Recall that a lower value of DKL

means a more accurate prediction

plane-object distance h and if we let f = 350px then the erroris always lower than 200px which translate to a precision up to20 of an image at 1080p resolution

14 THE EFFECT OF RANDOM CROPPING

In order to advocate for the peculiar training strategy illustratedin Sec 41 involving two streams processing both resized clipsand randomly cropped clips we perform an additional experimentas follows We first re-train our multi-branch architecturefollowing the same procedures explained in the main paper exceptfor the cropping of input clips and groundtruth maps which isalways central rather than random At test time we shift each inputclip in the range [minus800 800] pixels (negative and positive shiftsindicate left and right translations respectively) After the trans-lation we apply mirroring to fill borders on the opposite side ofthe shift direction We perform the same operation on groundtruthmaps and report the mean DKL of the multi-branch modelwhen trained with random and central crops as a function of thetranslation size in Fig 26 The figure highlights how randomcropping consistently outperforms central cropping Importantlythe gap in performance increases with amount of shift appliedfrom a 2786 relative DKL difference when no translation isperformed to 25813 and 21617 increases for minus800 px and800 px translations suggesting the model trained with centralcrops is not robust to positional shifts

15 THE EFFECT OF PADDED CONVOLUTIONS INLEARNING A CENTRAL BIAS

Learning a globally localized solution seems theoretically impos-sible when using a fully convolutional architecture Indeed convo-lutional kernels are applied uniformly along all spatial dimensionof the input feature map Conversely a globally localized solutionrequires knowing where kernels are applied during convolutionsWe argue that a convolutional element can know its absolute posi-tion if there are latent statistics contained in the activations of theprevious layer In what follows we show how the common habitof padding feature maps before feeding them to convolutional

layers in order to maintain borders is an underestimated source ofspatially localized statistics Indeed padded values are always con-stant and unrelated to the input feature map Thus a convolutionaloperation depending on its receptive field can localize itself in thefeature map by looking for statistics biased by the padding valuesTo validate this claim we design a toy experiment in which a fullyconvolutional neural network is tasked to regress a white centralsquare on a black background when provided with a uniformor a noisy input map (in both cases the target is independentfrom the input) We position the square (bias element) at thecenter of the output as it is the furthest position from borders iewhere the bias originates We perform the experiment with severalnetworks featuring the same number of parameters yet differentreceptive fields4 Moreover to advocate for the random croppingstrategy employed in the training phase of our network (recallthat it was introduced to prevent a saliency-branch to regress acentral bias) we repeat each experiment employing such strategyduring training All models were trained to minimize the meansquared error between target maps and predicted maps by meansof Adam optimizer5 The outcome of such experiments in terms ofregressed prediction maps and loss function value at convergenceare reported in Fig 27 and Fig 28 As shown by the top-mostregion of both figures despite the uncorrelation between inputand target all models can learn a central biased map Moreoverthe receptive field plays a crucial role as it controls the amount ofpixels able to ldquolocalize themselvesrdquo within the predicted map Asthe receptive field of the network increases the responsive areashrinks to the groundtruth area and loss value lowers reachingzero For an intuition of why this is the case we refer the readerto Fig 29 Conversely as clearly emerges from the bottom-mostregion of both figures random cropping prevents the model toregress a biased map regardless the receptive field of the networkThe reason underlying this phenomenon is that padding is appliedafter the crop so its relative position with respect to the targetdepends on the crop location which is random

16 ON FIG 8We represent in Fig 30 the process employed for building thehistogram in Fig8 (in the paper) Given a segmentation map of aframe and the corresponding ground-truth fixation map we collectpixel classes within the area of fixation thresholded at differentlevels as the threshold increases the area shrinks to the realfixation point A better visualization of the process can be foundat httpsndrplzgithubiodreyeve

17 FAILURE CASES

In Fig 31 we report several clips in which our architecture fails tocapture the groundtruth human attention

4 a simple way to decrease the receptive field without changing the numberof parameters is to replace two convolutional layers featuring C outputchannels into a single one featuring 2C output channels

5 the code to reproduce this experiment is publicly released at httpsgithubcomDavideAcan_i_learn_central_bias

20

119859119859

119858

119847119847

x2

h

γ

hh

θ

(a) (b) (c)

Fig 25 (a) By aligning the reference system with the plane normal centered on the projection of the non-planar point onto the plane the metric erroris the magnitude of the difference between the two vectors ~xa

2 and ~xb2 The red lines help to highlight the similarity of inner and outer triangles having

xax as a vertex (b) The geometry of the problem when the reference system is placed off the plane in an arbitrary position (gray) or specifically on

one of the camera (black) (c) The simplified setting in which we consider the projection of the metric error ew on the camera plane of A

Notably the vector v and the scalar h both appear as multiplicativefactors in Eq (17) so that if any of them goes to zero then themagnitude of the metric error ew also goes to zeroIf we assume that h 6= 0 we can go one step further and obtaina formulation were x2 and v are always divided by h suggestingthat what really matters is not the absolute position of x2 orcamera B with respect to camera A but rather how many timesfurther x2 and camera B are from A than x2 from the plane Suchrelation is made explicit below

ew =hvh

(x2h otimes rdquox2 minus vh otimes

rdquov ) rdquon︸ ︷︷ ︸Q

otimes (18)

rdquov

(rdquon otimes rdquon (1minus 1

1minus 1

1minus x2h otimes rdquox 2

rdquon

)

︸ ︷︷ ︸Z

minusI) (19)

134 Working towards a bound

Let M = x2v| cos θ| being θ the angle between rdquox2 andrdquon and let β be the angle between rdquov and rdquon Then Q can berewritten as Q = (M minuscosβ)minus1 Note that under the assumptionthat M ge 2 Q le 1 always holds Indeed for M ge 2 to holdwe need to require x2 ge 2v| cos θ| Next consider thescalar Z it is easy to verify that if |x2h otimes rdquox2

rdquon | gt 1 then|Z| le 1 Since both rdquox2 and rdquon are versors the magnitude of theirdot product is at most one It follows that |Z| lt 1 if and only ifx2 gt h Now we are left with a versor rdquov that multiplies thedifference of two matrices If we compute such product we obtaina new vector with magnitude less or equal to one rdquov rdquon otimes rdquon andthe versor rdquov The difference of such vectors is at most 2 Summingup all the presented considerations we have that the magnitude ofthe error is bounded as follows

Observation 1If x2 ge 2v| cos θ| and x2 gt h thenew le 2h

We now aim to derive a projection error bound from theabove presented metric error bound In order to do so we need

to introduce the focal length of the camera f For simplicity wersquollassume that f = fx = fy First we simplify our setting withoutloosing the upper bound constraint To do so we consider theworst case scenario in which the mutual position of the plane andthe camera maximizes the projected errorbull the plane rotation is so that rdquon zbull the error segment is just in front of the camerabull the plane rotation along the z axis is such that the parallel

component of the error wrt the x axis is zero (this allowsus to express the segment end points with simple coordinateswithout loosing generality)

bull the camera A falls on the middle point of the error segmentIn the simplified scenario depicted in Fig 25(c) the projectionof the error is maximized In this case the two points we wantto project are p1 = [0minush γ] and p2 = [0 h γ] (we considerthe case in which ||ew|| = 2h see Observation 1) where γ isthe distance of the camera from the plane Considering the focallength f of camera A p1 and p2 are projected as follows

KAp1 =

f 0 0 00 f 0 00 0 1 0

0minushγ

=

0minusfhγ

rarr [0

minus fhγ

](20)

KAp2 =

f 0 0 00 f 0 00 0 1 0

0hγ

=

0fhγ

rarr [0fhγ

](21)

Thus the magnitude of the projection ea of the metric errorew is bounded by 2fh

γ Now we notice that γ = h+ x2

rdquon = h+ x2cos(θ) so

ea le2fh

γ=

2fh

h+ x2cos(θ)=

2f

1 + x2h cos(θ)

(22)

Notably the right term of the equation is maximized whencos(θ) = 0 (since when cos(θ) lt 0 the point is behind thecamera which is impossible in our setting) Thus we obtain thatea le 2f

Fig 24(b) shows a use case of the bound in Eq 22 It shows valuesof θ up to pi2 where the presented bound simplifies to ea le2f (dashed black line) In practice if we require i) θ le π3 andii) that the camera-object distance x2 is at least three times the

21

800 600 400 200 0 200 400 600 800Shift (px)

0

1

2

3

4

5

6

7

DKL

Central cropRandom crop

Fig 26 Performances of the multi-branch model trained with ran-dom and central crop policies in presence of horizontal shifts in inputclips Colored background regions highlight the area within the standarddeviation of the measured divergence Recall that a lower value of DKL

means a more accurate prediction

plane-object distance h and if we let f = 350px then the erroris always lower than 200px which translate to a precision up to20 of an image at 1080p resolution

14 THE EFFECT OF RANDOM CROPPING

In order to advocate for the peculiar training strategy illustratedin Sec 41 involving two streams processing both resized clipsand randomly cropped clips we perform an additional experimentas follows We first re-train our multi-branch architecturefollowing the same procedures explained in the main paper exceptfor the cropping of input clips and groundtruth maps which isalways central rather than random At test time we shift each inputclip in the range [minus800 800] pixels (negative and positive shiftsindicate left and right translations respectively) After the trans-lation we apply mirroring to fill borders on the opposite side ofthe shift direction We perform the same operation on groundtruthmaps and report the mean DKL of the multi-branch modelwhen trained with random and central crops as a function of thetranslation size in Fig 26 The figure highlights how randomcropping consistently outperforms central cropping Importantlythe gap in performance increases with amount of shift appliedfrom a 2786 relative DKL difference when no translation isperformed to 25813 and 21617 increases for minus800 px and800 px translations suggesting the model trained with centralcrops is not robust to positional shifts

15 THE EFFECT OF PADDED CONVOLUTIONS INLEARNING A CENTRAL BIAS

Learning a globally localized solution seems theoretically impos-sible when using a fully convolutional architecture Indeed convo-lutional kernels are applied uniformly along all spatial dimensionof the input feature map Conversely a globally localized solutionrequires knowing where kernels are applied during convolutionsWe argue that a convolutional element can know its absolute posi-tion if there are latent statistics contained in the activations of theprevious layer In what follows we show how the common habitof padding feature maps before feeding them to convolutional

layers in order to maintain borders is an underestimated source ofspatially localized statistics Indeed padded values are always con-stant and unrelated to the input feature map Thus a convolutionaloperation depending on its receptive field can localize itself in thefeature map by looking for statistics biased by the padding valuesTo validate this claim we design a toy experiment in which a fullyconvolutional neural network is tasked to regress a white centralsquare on a black background when provided with a uniformor a noisy input map (in both cases the target is independentfrom the input) We position the square (bias element) at thecenter of the output as it is the furthest position from borders iewhere the bias originates We perform the experiment with severalnetworks featuring the same number of parameters yet differentreceptive fields4 Moreover to advocate for the random croppingstrategy employed in the training phase of our network (recallthat it was introduced to prevent a saliency-branch to regress acentral bias) we repeat each experiment employing such strategyduring training All models were trained to minimize the meansquared error between target maps and predicted maps by meansof Adam optimizer5 The outcome of such experiments in terms ofregressed prediction maps and loss function value at convergenceare reported in Fig 27 and Fig 28 As shown by the top-mostregion of both figures despite the uncorrelation between inputand target all models can learn a central biased map Moreoverthe receptive field plays a crucial role as it controls the amount ofpixels able to ldquolocalize themselvesrdquo within the predicted map Asthe receptive field of the network increases the responsive areashrinks to the groundtruth area and loss value lowers reachingzero For an intuition of why this is the case we refer the readerto Fig 29 Conversely as clearly emerges from the bottom-mostregion of both figures random cropping prevents the model toregress a biased map regardless the receptive field of the networkThe reason underlying this phenomenon is that padding is appliedafter the crop so its relative position with respect to the targetdepends on the crop location which is random

16 ON FIG 8We represent in Fig 30 the process employed for building thehistogram in Fig8 (in the paper) Given a segmentation map of aframe and the corresponding ground-truth fixation map we collectpixel classes within the area of fixation thresholded at differentlevels as the threshold increases the area shrinks to the realfixation point A better visualization of the process can be foundat httpsndrplzgithubiodreyeve

17 FAILURE CASES

In Fig 31 we report several clips in which our architecture fails tocapture the groundtruth human attention

4 a simple way to decrease the receptive field without changing the numberof parameters is to replace two convolutional layers featuring C outputchannels into a single one featuring 2C output channels

5 the code to reproduce this experiment is publicly released at httpsgithubcomDavideAcan_i_learn_central_bias

21

800 600 400 200 0 200 400 600 800Shift (px)

0

1

2

3

4

5

6

7

DKL

Central cropRandom crop

Fig 26 Performances of the multi-branch model trained with ran-dom and central crop policies in presence of horizontal shifts in inputclips Colored background regions highlight the area within the standarddeviation of the measured divergence Recall that a lower value of DKL

means a more accurate prediction

plane-object distance h and if we let f = 350px then the erroris always lower than 200px which translate to a precision up to20 of an image at 1080p resolution

14 THE EFFECT OF RANDOM CROPPING

In order to advocate for the peculiar training strategy illustratedin Sec 41 involving two streams processing both resized clipsand randomly cropped clips we perform an additional experimentas follows We first re-train our multi-branch architecturefollowing the same procedures explained in the main paper exceptfor the cropping of input clips and groundtruth maps which isalways central rather than random At test time we shift each inputclip in the range [minus800 800] pixels (negative and positive shiftsindicate left and right translations respectively) After the trans-lation we apply mirroring to fill borders on the opposite side ofthe shift direction We perform the same operation on groundtruthmaps and report the mean DKL of the multi-branch modelwhen trained with random and central crops as a function of thetranslation size in Fig 26 The figure highlights how randomcropping consistently outperforms central cropping Importantlythe gap in performance increases with amount of shift appliedfrom a 2786 relative DKL difference when no translation isperformed to 25813 and 21617 increases for minus800 px and800 px translations suggesting the model trained with centralcrops is not robust to positional shifts

15 THE EFFECT OF PADDED CONVOLUTIONS INLEARNING A CENTRAL BIAS

Learning a globally localized solution seems theoretically impos-sible when using a fully convolutional architecture Indeed convo-lutional kernels are applied uniformly along all spatial dimensionof the input feature map Conversely a globally localized solutionrequires knowing where kernels are applied during convolutionsWe argue that a convolutional element can know its absolute posi-tion if there are latent statistics contained in the activations of theprevious layer In what follows we show how the common habitof padding feature maps before feeding them to convolutional

layers in order to maintain borders is an underestimated source ofspatially localized statistics Indeed padded values are always con-stant and unrelated to the input feature map Thus a convolutionaloperation depending on its receptive field can localize itself in thefeature map by looking for statistics biased by the padding valuesTo validate this claim we design a toy experiment in which a fullyconvolutional neural network is tasked to regress a white centralsquare on a black background when provided with a uniformor a noisy input map (in both cases the target is independentfrom the input) We position the square (bias element) at thecenter of the output as it is the furthest position from borders iewhere the bias originates We perform the experiment with severalnetworks featuring the same number of parameters yet differentreceptive fields4 Moreover to advocate for the random croppingstrategy employed in the training phase of our network (recallthat it was introduced to prevent a saliency-branch to regress acentral bias) we repeat each experiment employing such strategyduring training All models were trained to minimize the meansquared error between target maps and predicted maps by meansof Adam optimizer5 The outcome of such experiments in terms ofregressed prediction maps and loss function value at convergenceare reported in Fig 27 and Fig 28 As shown by the top-mostregion of both figures despite the uncorrelation between inputand target all models can learn a central biased map Moreoverthe receptive field plays a crucial role as it controls the amount ofpixels able to ldquolocalize themselvesrdquo within the predicted map Asthe receptive field of the network increases the responsive areashrinks to the groundtruth area and loss value lowers reachingzero For an intuition of why this is the case we refer the readerto Fig 29 Conversely as clearly emerges from the bottom-mostregion of both figures random cropping prevents the model toregress a biased map regardless the receptive field of the networkThe reason underlying this phenomenon is that padding is appliedafter the crop so its relative position with respect to the targetdepends on the crop location which is random

16 ON FIG 8We represent in Fig 30 the process employed for building thehistogram in Fig8 (in the paper) Given a segmentation map of aframe and the corresponding ground-truth fixation map we collectpixel classes within the area of fixation thresholded at differentlevels as the threshold increases the area shrinks to the realfixation point A better visualization of the process can be foundat httpsndrplzgithubiodreyeve

17 FAILURE CASES

In Fig 31 we report several clips in which our architecture fails tocapture the groundtruth human attention

4 a simple way to decrease the receptive field without changing the numberof parameters is to replace two convolutional layers featuring C outputchannels into a single one featuring 2C output channels

5 the code to reproduce this experiment is publicly released at httpsgithubcomDavideAcan_i_learn_central_bias

22

TRAINING WITHOUT RANDOM CROPPING

Input Target RF=114 RF=106 RF=98

0

0002

0004

0006

0008

001

0012

0014

0016

0 500 1000 1500 2000

Loss

fun

ctio

n va

lue

Batch

Input uniform

RF=98RF=106RF=114

TRAINING WITH RANDOM CROPPING

Input Target RF=114 RF=106 RF=98

0

001

002

003

004

005

006

0 500 1000 1500 2000

Loss

fun

ctio

n va

lue

Batch

Input uniform

RF=98RF=106RF=114

Fig 27 To show the beneficial effect of random cropping in preventing a convolutional network to learn a biased map we train several models toregress a fixed map from uniform input images We argue that in presence of padded convolutions and big receptive fields the relative locationof the groundtruth map with respect to image borders is fixed and proper kernels can be learned to localize the output map We report outputsolutions and loss functions for different receptive fields As the receptive field grows the portion of the image accessing borders grows and thesolution improves (reaching lower loss values) Please note that in this setting the input image is 128x128 while the output map is 32x32 andthe central bias is 4x4 Therefore the minimum receptive field required to solve the task is 2 lowast (322 minus 42) lowast (12832) = 112 Conversely whentrained by randomly cropping images and groundtruth maps the reliable statistic (in this case the relative location of the map with respect to imageborders) ceases to exist making the training process hopeless

23

TRAINING WITHOUT RANDOM CROPPING

Input Target RF=114 RF=106 RF=98

0

0002

0004

0006

0008

001

0012

0014

0016

0 500 1000 1500 2000

Loss

fun

ctio

n va

lue

Batch

Input noise

RF=98RF=106RF=114

TRAINING WITH RANDOM CROPPING

Input Target RF=114 RF=106 RF=98

0

001

002

003

004

005

006

007

0 500 1000 1500 2000

Loss

fun

ctio

n va

lue

Batch

Input noise

RF=98RF=106RF=114

Fig 28 This figure illustrates the same content as Fig 27 but reports output solutions and loss functions obtained from noisy inputs See Fig 27for further details

Fig 29 Importance of the receptive field in the task of regressing a central bias exploiting padding the receptive field of the red pixel in the outputmap has no access to padding statistics so it will be activated On the contrary the green pixelrsquos receptive field exceeds image borders in thiscase padding is a reliable anchor to break the uniformity of feature maps

24

image segmentation

different fixation map thresholds

Fig 30 Representation of the process employed to count class occurrences to build Fig 8 See text and paper for more details

25

Input frame GT multi-branch

Fig 31 Some failure cases of our multi-branch architecture

23

TRAINING WITHOUT RANDOM CROPPING

Input Target RF=114 RF=106 RF=98

0

0002

0004

0006

0008

001

0012

0014

0016

0 500 1000 1500 2000

Loss

fun

ctio

n va

lue

Batch

Input noise

RF=98RF=106RF=114

TRAINING WITH RANDOM CROPPING

Input Target RF=114 RF=106 RF=98

0

001

002

003

004

005

006

007

0 500 1000 1500 2000

Loss

fun

ctio

n va

lue

Batch

Input noise

RF=98RF=106RF=114

Fig 28 This figure illustrates the same content as Fig 27 but reports output solutions and loss functions obtained from noisy inputs See Fig 27for further details

Fig 29 Importance of the receptive field in the task of regressing a central bias exploiting padding the receptive field of the red pixel in the outputmap has no access to padding statistics so it will be activated On the contrary the green pixelrsquos receptive field exceeds image borders in thiscase padding is a reliable anchor to break the uniformity of feature maps

24

image segmentation

different fixation map thresholds

Fig 30 Representation of the process employed to count class occurrences to build Fig 8 See text and paper for more details

25

Input frame GT multi-branch

Fig 31 Some failure cases of our multi-branch architecture

24

image segmentation

different fixation map thresholds

Fig 30 Representation of the process employed to count class occurrences to build Fig 8 See text and paper for more details

25

Input frame GT multi-branch

Fig 31 Some failure cases of our multi-branch architecture

25

Input frame GT multi-branch

Fig 31 Some failure cases of our multi-branch architecture