3D gaze cursor: continuous calibration and end-point grasp control ofrobotic actuators

Pablo M. Tostado1, William W. Abbott1 and A. Aldo Faisal1,2,3, Member IEEE

Abstract— Eye movements are closely related to motor ac-tions, and hence can be used to infer motor intentions. Ad-ditionally, eye movements are in some cases the only meansof communication and interaction with the environment forparalysed and impaired patients with severe motor deficiencies.Despite this, eye-tracking technology still has a very limiteduse as a human-robot control interface and its applicability ishighly restricted to 2D simple tasks that operate on screen basedinterfaces and do not suffice for natural physical interactionwith the environment. We propose that decoding the gazeposition in 3D space rather than in 2D results into a muchricher "spatial cursor" signal that allows users to performeveryday tasks such as grasping and moving objects via gaze-based robotic teleoperation. Eye tracking in 3D calibration isusually slow – we demonstrate here that by using a full 3Dtrajectory for system calibration generated by a robotic armrather than a simple grid of discrete points, gaze calibration inthe 3 dimensions can be successfully achieved in short time andwith high accuracy. We perform the non-linear regression fromeye-image to 3D-end point using Gaussian Process regressors,which allows us to handle uncertainty in end-point estimatesgracefully. Our telerobotic system uses a multi-joint robot armwith a gripper and is integrated with our in-house "GT3D"binocular eye tracker. This prototype system has been evaluatedand assessed in a test environment with 7 users, yielding gaze-estimation errors of less than 1cm in the horizontal, vertical anddepth dimensions, and less than 2cm in the overall 3D Euclideanspace. Users reported intuitive, low-cognitive load, control of thesystem right from their first trial and were straightaway ableto simply look at an object and command through a wink to"grasp this" object with the robot gripper.

I. INTRODUCTION

The needs for assistive robotics and robotic rehabilitationhas been driven by an ageing population with increasinglyhigher number of individuals that depend on others toperform daily-life activities due to movement disabilities.Moreover, paralysed patients and amputees have driven thedevelopment of brain-machine interfaces and neuroprosthet-ics. Accurate accurate control is a state-of-the-art problemof distinguished importance in the field of bioengineering.For years, Brain Machine Interfaces (BMIs) and electromyo-graphic (EMG) signals have been the most commonly usedapproaches as interactive pathways from the user ( to anexternal device such as a prosthesis [1], [2]. However, theimplementation of EMG-technologies can be unfeasible inthose cases in which the patient suffers from severe multi-ple disabilities, and brain computer interfaces are generallyextremely invasive when fast, accurate control is sought and

Brain & Behaviour Lab - 1Dept. of Bioengineering & 2Dept. of Com-puting, Imperial College London, South Kensington Campus, SW7 2AZ,London, UK,3MRC Clinical Sciences Centre. Address for correspondence:aldo.faisal@imperial.ac.uk

require considerable setup and training time – even for EEGbase systems. In addition, the clinical and operational costsof such currently-emerging technologies often go beyond themeans of potential users. Several different approaches havebeen attempted to circumvent these problems. In the recentyears, eye-tracking systems have been suggested as a reliablenon-invasive alternative to BMI and have been extensivelydeveloped as a cheaper and more compact interface betweenhumans and robots[3]. The idea behind the use of eyemovements as a control signal relies on the fact that our gazeis proactive and directly correlated to action intentions andcognitive control [4], [5]. Furthermore, function the ocular-motor system control is usually retained even in the mostsevere cases of paralysis related to muscular dystrophies orbrain stroke, and movements of the eyes are preserved muchlonger then skeletal movement in neurodegenerative diseasesaffecting the motor system, such as Multiple Sclerosis,Parkinson’s, or ALS [6].

2D Eye-tracking is a long established technology that hasbeen extensively used as a control interface in academic andindustrial human-computer interface research. However, theuse of 3D eye movements for control of gaze-based roboticinteraction systems is not a straightforward process. Oneof the main issues limiting the direct implementation ofeye-tracking systems is the difficulty to distinguish betweenintentional visual fixations and unintentional environmentscrutiny, known as the Midas touch problem. Current ap-proaches to this problem usually rely on differences ingaze-dwell times between natural fixations and saccades [7].Yet, the use of this technology is significantly limited bythe expensive cost of high performance commercial eye-tracking systems. Efforts have been made focusing on theenrichment of the signal interpretation and cheapening ofeye-tracking devices. We have utilised an ultra low-costeye-tracker "GT3D" custom built by the integration of twocommercial Playstation3 cameras (10USD each) and twoInfrared (IR) LEDs that yield pupil detection at high speedframe rates (up to 120 Hz) developed by us [3]. We haveresolved the Midas touch problem here by using " Winks" foropening/closing of the gripper and "long fixations" to triggermovement to the gaze-end point – effectively proactivelymoving the arm to objects we examine for a long time.

In cases were neurodegenerative diseases affect the capac-ity of communication, eye-tracking technology is suitable ascommunication tool via computers. In some cases, detectionof fixations or blink patterns allow the user to operateinteractive games or spelling programs [8]. Eye movementsequences including winks and blinks can be composed into

"gaze gestures" are translated into commands. 2D eye track-ing has therefore been used in human-computer interactivesystems focused on the control of machines such as poweredwheelchairs substituting current control mechanisms, forinstance the joystick. Commands are displayed on a screen,and the direction of the eye gaze is decoded in real timein order to implement the desired instruction [9], [10]. Thepotential of a computer decoding natural eye movements(the same that occur during mouse-based computer use) tooperate arcade video games has been shown in field trialswith hundreds of users being able to play (and score) inunder 15 seconds from first use of the system [11]. Whileable-bodied users can seamlessly interact with the physicalworld without thinking about it, movement impaired usershave to learn to operate a gaze-based user interface to controla device – which is unnatural in their interaction, causesadditional cognitive load, and divert (literally) their visualattention from the task at hand. The challenge lies in theuser interface, when we manipulate with the physical worldour gaze follows naturally our movement intentions andeven our motor imagery (thinking about grasping a glasscan trigger the same eye movements as actually graspingit). Thus, ideally we want to remove screen-based userinterfaces and enable users to interact more naturally bydirectly looking at the objects they wish to manipulate with arobot. We previously demonstrated a gaze decoding solutiondeployed on an electrical wheelchair to freely drive thewheelchair (without having to look at a computer display orother user interface) by correlating naturally occurring eye-movements during joystick-based driving into decoders fordriving by eye-tracking [12]. Reaching, grasping and objectmanipulation are however more complex than driving, asthey require 3D interactions. Thus, the main restriction thatprevents binocular eye-tracking systems from being morewidely used in fields such as teleoperation and prosthetics hasbeen the difficulty to implement accurate 3D gaze estimation.

One of the fundamental challenges preventing this ex-pected upgrading of eye-tracking systems is related to theproblematic calibration of the system for individual users

in the depth dimension. Some algorithms have approachedthe inference of 3D gaze position in virtual environments.Most of the studies aiming to acquire a depth directionalgaze position examine the relationship between the inter-distance of the centre of both pupils and the depth of thegaze fixation [13], [14] and fit a mechanical model of the eyeto the data. Nonetheless, the obtained accuracy in the depthdirection is comparatively low. One of the only 3D gazeestimation systems implemented in a real-world environmentuses the combined geometry of the eyeball and the eye-tracking cameras setup to estimate the gaze vectors that linkthe fovea and the center of the tracked pupil of both eyes.The 3D gaze fixating point is inferred from the calculation ofthe closest intersection point of the aforementioned vectors inthe 3D space. The overall determined accuracy of the systemwas 3.93 cm in Euclidean space [15]. Alternative solutions tosolve the insufficient accuracy in the depth dimension haveattempted to use 2D gaze control while performing planeselection independently. A recent approach combined 2Deye-tracking control with an imaginary-based BMIs for depthplane selection, yielding a pseudo-3D interface. Assessmentof the interface for 3D interaction was performed by combin-ing the eye-tracker system with an arm reaching mode as twoindependent navigation methods, which were independentlyselected by imaginary-based BMI electroencephalography[16].

Here, we present a novel calibration approach that aimsto exploit the huge potential of 3D gaze estimation in thefield of robotic teleoperation and prosthetics control. Acontinuous trajectory in 3D space generated by the robotrather than a discrete set of points in a given 2D plane isused for calibration. This richer and continuous data set canbe actually obtained faster and more naturally then screen-based interactions, by simply asking the user to track therobot end-point marker. Our integrated system can reliablyestimate gaze fixations with high and near equal accuracyin all three spatial coordinates, and can be used to operateour robot arm including grasping of small objects with thegripper.

Fig. 1. High-resolution eye tracking enables tracking of a user 3D gaze target location (red arrow). Two type of gaze-gesture commands ("wink" and"fixations") enable intuitive and seamless operation of a 5DOF robotic gripper arm. Inset: IR camera images of user’s left and right eye cameras andtracking ellipse of the pupils (blue circles).

II. METHODS

Our integrated system (see Fig. 1) consists of two eye-tracking cameras, an OptiTrack motion tracking system de-signed and a WidowX robot arm. The streaming of data fromand to each individual piece of software is synchronized byan integrated C++ interface.

a) WidowX robotic arm: This low-end robotic armoperated in this system is the WidowX Reactor RobotArm (TrossenRobotics, Downers Grove IL). The robot armpossesses 5 DOF granted by 5 different servos, includingthe ability to open and close a parallel gripper with 500gholding strength. Communication to and from the servos ofthe WidowX is enabled by an ArbotiX micro-controller thatalso performs the inverse kinematic calculations for a giventarget position. This communication is integrated in our C++interface, from which the endpoint position is computed. Thecurrent specifications of this robot arm describe a maximumradius reach of 40 cm.

b) GT3D binocular eye-tracker: The GT3D binoculareye-tracker utilised in our system was developed by Abbott& Faisal, [3]. The eye-tracker is built from two commercialPlaystation 3 cameras yielding a resolution of 320 x 240pixels at a maximum frame-rate of 120Hz. Pupil recognitionis achieved by the inclusion of two infrared (IR) lightLED sources and modifying the cameras’ filter to allow IRlight flow. This adjustment optimises the imaging conditionseasing pupil detection by increasing the contrast betweenthe pupil and the iris. Common image-processing methods(OpenCV) are implemented to generate a binary image of theeye from the greyscale recorded image by threshold imagesegmentation in each video frame. Subsequently, erosion anddilation morphological operations are performed over thebinary image preceding a connecting component labellingand a shape-based filter leading to pupil classification. Anellipse is fitted to the classified pupil using a least-squaresregression algorithm that allows extraction of the center-of-

Fig. 2. End-effector’s position commands (red dots) and continuoustrajectory performed by the robot (connecting lines). High data throughputfor eye-tracker 3D calibration is achieved by using continuous movementof the robot arm in a three dimensional space.

pupil coordinates. Further zoom and threshold settings areaccessible during the eye-tracker setup. The hardware costsfor the system are at 20 USD.

c) Motion tracking and synchronisation: The noveltyof our system relies on the form of calibration of the eye-tracker. This process is performed using the robot arm itselfto define a continuous set of calibration points in 3D spacegiven by a pre-defined trajectory. During robot movementacross this trajectory, the user must look at the endpointposition of the gripper in a non-stopping fashion. Normallythe inverse kinematics of a robot arms provides a highresolution end-point position. These 3D positions need tobe synchronised with the eye movement information fromthe eye tracker. This proved challenging with the WidowXarm as feedback signals were transmitted a serial-USB com-munication back to our computer – which caused unreliabletime synchronisation (>300 ms variability). Therefore, wemonitored during calibration the endpoint position of thegripper by using an OptiTrak optical motion capture system(Natural Point Inc., Corvallis, OR) and recorded continuouslythe position of the system. A synchronisation algorithm wasrun to match the simultaneous acquisition of data fromboth systems at every time step. Both the center-of-pupilsdata and the continuous spatial position of the robot arerecorded and saved respectively for posterior calibration. Themotion tracking system requires a single-time self-calibrationafter being setup. OptiTrack technology can resolve thelocation of markers with millimetre accuracy. A restrictedworking volume of 40cm x 30cm x 20cm (width x heightx depth) to operate the robot arm was defined. Fig.2 showsa hypothetical trajectory followed by the robot for 3D usercalibration.

A. Gaussian Processes Regression for calibration

The non-linear regression algorithm used to infer thecorrelation between the pupil-centers and the gaze pointof the user is the core part of our continuous eye-trackercalibration. We used Gaussian Processes (GP) Regression,which is a non-parametric Bayesian supervised learningalgorithm [17]. GPs are a family of statistical distributionscharacterised by the association of every data point in theinput space with a normally distributed random variable.Moreover, any collection of n input random variables istreated as a n-dimensional normal distribution, and the result-ing distribution of a GP is the joint distribution of all inputrandom variables. Henceforth, GP may be understood asan infinite-dimensional generalization of multivariate normaldistributions.

The advantage of this approach is that we do not makeassumptions about the eye geometry. Thus, we can performmodel-free gaze end-point calibration, making the systemautomatically also suitable for users with strabism or othereye alignment defects – as long as the eyes show repeatabilityin how they are looking at a target . This is possible becausewe collect more data from the user (continuous trajectoriesinstead of discrete points) that are more accurate (usingsmooth pursuit eye movements tracking an object instead

0 10 20 30 40 50

0

10

20

30

40

50P

red

icte

d X

-po

sit

ion

[c

m]

X-Dimension GP Regression

Real X-position [cm]

Real vs Target X-positionConfident Interval

-30 -20 -10 0 10

-30

-20

-10

0

10

Pre

dic

ted

Y-p

os

itio

n [

cm

]

Y-Dimension GP Regression

Real Y-position [cm]

-5 0 5 10 15 20

-5

0

5

10

15

20

Pre

dic

ted

Z-p

os

itio

n [

cm

]

Z-Dimension GP Regression

Real Z-position [cm]Fig. 3. Calibration accuracy from robot end-point to pupil data. Evaluation of the GP regression based calibration for each spatial actuator coordinate ontest data for one representative user. The GP confidence 95% interval of the predicted value is shown for each data point in X, Y and Z coordinates.

of saccades between jumping calibration targets), but alsobecause GP based approaches allows us to perform non-linear regression in a sdata efficient regime as they operatewith outliers and can track associated uncertainty whenreconstructing gaze, which becomes a crucial aspect in thesafety aspects of robotic operation The key distinction be-tween this regression estimator and other non-linear functionapproximators is that the resulting output from given setof arguments is considered to be a stochastic process. Thismeans that the output is randomly drawn from a joint normaldistribution of potential calibration functions consistent withthe data. Additionally, it is important to realise that GP do notexplicitly estimate functions, but rather evaluates correlationsbetween locations where data was collected.

We have a stochastic process t : X → R where all(t(x1), · · · , t(xn)) are normally distributed. Like any normaldistribution a GP has a mean and a covariance defined forits potentially infinite domain. The correlation between theoutputs of a GP is given by the corresponding covariancefunction, which is at the same time determined by a certaincovariance kernel. For a certain unknown model, a set ofoutput samples obtained from given chosen inputs is assumedto be the result of the normal distribution of the expectedmean and covariance: f1

...fn

∼ N

µ1

...µn

,K11 · · · K1n

.... . .

...Kn1 · · · Knn

The covariance function is determined by the covari-ance kernel, which evaluates a presumed relation betweenobserved variables. Generally, any kernel assumes highercorrelation between variables closer to each other in space,and lower correlation otherwise. Furthermore, for a smallvariation in the input arguments, a small variation in theestimation is expected. We used one of the most simplekernel functions, the Squared Exponential Kernel:

Ki j = e−||xi−x j ||,Ki j =

{0, if ||xi− x j|| → ∞

1, if xi ≡ x j

Then in a GP, for a given random input argument x∗, theestimated output is similarly assumed to be drawn from a

normal distribution as follows:

f1...fnf∗

∼ N

µ1...

µnµ∗

,

K︷ ︸︸ ︷K11 · · · K1n...

. . ....

Kn1 · · · Knn

K*︷ ︸︸ ︷K1∗· · ·Kn∗

[K∗1 · · · K∗n

][K∗∗]︸ ︷︷ ︸

K**

From this assumption, the mean and standard deviation of

the estimation is:E(f∗) = µ(x∗)+KT

∗K−1(f−µ(x))

σ∗ =−KT

∗K−1K∗+K∗∗GP Regressors were trained with eye-parameters for each

of the spatial coordinates of the robot’s working environmentat different positions and used to estimate the gaze fixationpoint for new eye-data. The input was 4 dimensional (pupilX-left, pupilY-left, pupilX-right, pupilY-right) and the mappingwas (constrained by the GP formalism) to 1-dimension, sowe trained 3 independent regressors, one for each 3D coor-dinate. This could be further refined, e.g. by implementinga joint prior distribution of joint locations.

III. RESULTS

The Matlab Gaussian Processes Regression for MachineLearning (GPML) toolbox by Rasmussen & Nickisch wasused for implementation of our GP model [18]. The numberof calibration points and hence the calibration accuracy isdependent on the calibration process explained above. Thenumber of acquired points usually ranges between 500 and1500 points, depending on the trajectory selected and theperformance of the user during the procedure. The GP modelselected for the system was run with settings (1) LinearMean, (2) Matérn covariance, (3) Gaussian Likelihood and(4) Exact Inference method. The training method retrievesthe optimal intrinsic constant hyperparameters that definethe chosen functions to accurately describe the model’sbehaviour. Once these hyperparameters are defined, simpleregression may be applied to infer the gaze fixating pointfrom new eye-data.

After calibration, the user is in free control of the robotarm in 3D space within the working volume. The systemdeveloped allows for two deterministic, non-overlapping

Fig. 4. Histogram of the Euclidean Error of the inference procedure fortesting data after calibration for the same representative user as in Fig. 3.

commands: motion command and opening/closing of therobot gripper. A motion commands is performed by anextended fixation on a target point in the 3D workingvolume. The Midas touch problem is solved by determining athreshold in the eye-data (pupils’ position) standard deviationfor fixations, together with a distinction between differentgaze dwell-times associated with voluntary fixations andnaive inspection of the environment. The determination ofthe threshold for the variance in the eye-data to recognize afixation was performed by recording the eye pupil-centers inthe eye-tracking cameras of five different subjects completingextended voluntary fixations (over 2 seconds) at three differ-ent target points in space. The threshold in the variance of thecenter-of-pupils detected was 15 pixels, and an empiricallychosen minimum dwell-time of 250ms is required for motora command to be executed during online control. This choiceof parameters led to natural control of the robot movements.Whereas mere fixation detection is used to trigger movementof the robot end-point to towards the gaze- target, ’wink’ ofthe preferred eye for the user is used to send the commandto open/close the gripper successively. The analysis of theregression accuracy shows Euclidean errors (across x,y,zdimensions) in the predicted gaze position lower than 2cmon average for all subjects tested. Fig.3 shows the actualposition of the robot versus the predicted position from eye-data for the X, Y and Z coordinates respectively, as well asthe confident intervals of the prediction for a sample subject.A histogram of the Euclidean Error in the predicted positionfor the same subject is shown in Fig. 4.

Assessment of the accuracy of the calibration algorithmis performed by evaluating the prediction of the 3D positionof the robot from the corresponding pupils’ data for sevendifferent subjects. 70% of the recorded data was used totrain the Gaussian Process Regression. All the analyseswere performed over 30% of the data (testing data), whichcorresponded at least to 200 data points for each subject.Performance of the estimation was independently analysedby projection on each individual dimension. The overall

TABLE I3D CALIBRATION OF GP REGRESSION ON 7 SUBJECTS

Mean absolute error(cm) Standard deviation(cm)

x 0.6 1.0y 0.6 1.1z 0.8 1.3

Euclidean 1.6 1.7

prediction accuracy is assessed as the 3D Euclidean distancebetween the actual and predicted point. The mean absoluteerror and standard deviation of this analysis across subjectsis shown in Table I.

IV. DISCUSSION

We have developed and demonstrated continuous controlof a robotic arm and gripper operation in 3D space using3D eye-tracking technology. We have taken gaze interactionfrom 2D to 3D by using a continuous trajectory for calibra-tion rather than a discrete grid of points in a 2D plane. Thisnovel procedure allows for a very fast high calibration datathroughput (up to 600 points per minute), while keeping thecalibration time very similar to 2D calibration processes thatoften use a 3× 3 set of points. This high data throughputenables Gaussian Processes Regression to be applied toestimate in a model-free manner 3D gaze end-point . Theuse of GP regression yielded the lowest estimation Euclideanerror reported by any 3D gaze-controlled system up todate, operating below 2cm. Moreover, outliers of larger than4cm are very rare (see Fig. 4). Crucially, these outlierscoincided with uncertainty reported by the GP (e.g. outlier vsGP 95% confidence interval, see Fig. 3). Thus, when highreliability end-point precision is necessary, e.g. for safetyconsiderations during robot operation, come into play, wecan use the explicit representation of uncertainty by the GPregressor to detect and trigger an appropriate action.

One of the most important limitations preventing a wideruse of eye-tracking technology as a robot control interface isits deficient performance in the interaction with the real 3Dworld. Our system shows promising improvement towardsfluent eye-tracking control in 3 dimensions. It allows forfree direct interaction with the environment incorporatingthe possibility of grasping and moving objects around. Anaccurate gaze-estimation in the 3D domain supposes a directapplication to the field of prosthetics. The setup and calibra-tion of our system can be performed by the user and takesonly a few minutes from the user sitting down at the robot,calibration to direct control of reaching and grasping. This isa rather short time compared to technology like EEG used forneuroprosthetics control, which needs protracted calibrationprocedures (usually longer than 30 minutes) and expertuser assistance [19] or invasive brain-machine interfaces thatrequire (after surgery and rehabilitation on the time scale ofmonths) hour-long calibration before being able to controlrobotic actuators.

Our subjects reported natural control of the system throughthe use of detecting long natural fixations to encode move-ment commands, avoiding the cumbersome procedure oflearning gaze gestures. Moreover, user visual attention wasalways on the scene of their interactions, and not distractedby a screen-based user interface. Binocular eye trackingallowed us not only to perform 3D gaze measurement, butalso to add a further dimension. We used voluntary ’winks’rather than natural occurring blinks and to include extraaction selection features to our system: here, opening andclosing of the robot gripper. New commands could be easily

implemented including the detection of left vs right ’winks’independently or sequences of them, that may potentially addcomplexity and applicability towards prosthetic control.

In addition, the effective integration of the different soft-ware and algorithms into a single common interface facili-tates the portability of the system, that could be potentiallyutilised to actuate other operational devices such as an eye-driven vehicle or drone.

A potential issue affecting the calibration procedure andposterior control of the actuator is the user. Although strictconstraints (such as simultaneous detection of both pupils) are built-in during the data acquisition process in orderto consider a certain data point for calibration, the activeparticipation of the subject during the process is of majorimportance. This may preclude safe use by patients withmore advanced motor deficiencies in executive control orautotistic disorders. Additionally, the system as it is doescurrently not perform head- tracking and thus assumes a fixedposition of the user’s head, which may introduce calibrationerrors if movement occurs (such as smooth movementsoccurring during smooth pursuit). This could be solvedby 1. monitoring the user’s head position and accountingfor displacements to achieve increased precision and usercomfort and 2. by using a head-mounted eye-tracker (eye-tracking glasses).

In the system herein described, we used optical motiontrackers in order to achieve high-temporal accuracy in thesynchronisation gripper’s position monitoring during calibra-tion and reduce the overall position-prediction error duringcontrol. An alternative approach intended to estimate therobot’s trajectory from the end-effector’s position commands(robot’s forward kinematics) was attempted. However, thesignal feedback from our amateur-grade ArbotiX micro-controller timing was not accurate enough to allow goodsynchronisation with the eye-tracker (>300 ms synchroni-sation mismatch). Furthermore, the by design asynchronouscommands sent to the servos in sequence compounded thea noisy movement estimation. Nonetheless, we are confidentthat time synchronisation with more professional roboticand prosthetic actuator will enable removing the need ofadditional motion tracking hardware.

In conclusion, we are exploiting that eye movements arehighly correlated to our motor actions. They are a key factorin human motor planning and help us to direct and coordinatelimb movements. Consequently, gaze-tracking becomes arich source of information of user intention that may betaken advantage of. We have proven that accurate 3D gazetracking is possible in a fast and accurate manner. Thistechnology bears a tremendous potential as a signal forrobotic teleoperation. Eye-movements are retained in themajority of motor disabilities enhances the applicability ofeye-tracking technology in the field of brain-robot interfacesfor the paralysed and in prosthetics. In contrast with thelong training periods for BMI-based control, our systemrequires only a short calibration time and virtually no trainingfrom the user. We used our low-cost GT3D eyetracker, andwith the rise of more low-cost, high-precision behaviour-

based neuro technology [20] we expect that at least thesensor side can be further augmented at the same lowequipment cost and even take the technology outdoors. Oursystem allows for straightforward control of a robot arm thatpermits the user to grab and move objects and manipulatethe environment with the mere use of their eyes. Moreover,the high spatial accuracy and low latency (<100 ms) ofour closed-loop system enables natural feeling interactions.This makes our 3D gaze-controlled robot system a vividcompetitor to more conventional Brain-Machine Interfacesand screen-based assistive devices. The ability to freely lookat the scene of our interaction and control the robot enpassant enables novel ways of achieving embodiment inrobotic solution that restore or augment human function.

REFERENCES

[1] J. M. Carmena, “Advances in neuroprosthetic learning and control,”PLoS Biol., vol. 11, no. 5, p. e1001561, 2013.

[2] P. G. Patil and D. A. Turner, “The development of brain-machineinterface neuroprosthetic devices,” Neurotherapeutics, vol. 5, no. 1,pp. 137–146, 2008.

[3] W. Abbott and A. Faisal, “Ultra-low-cost 3d gaze estimation: anintuitive high information throughput compliment to direct brain–machine interfaces,” J. Neural Eng., vol. 9, no. 4, p. 046016, 2012.

[4] M. Land et al., “The roles of vision and eye movements in the controlof activities of daily living,” Perception-London, vol. 28, no. 11, pp.1311–1328, 1999.

[5] M. Hayhoe and D. Ballard, “Eye movements in natural behavior,”Trends in Cog Sci, vol. 9, no. 4, pp. 188–194, 2005.

[6] P. Cipresso et al., “The combined use of brain computer interfaceand eye-tracking technology for cognitive assessment in amyotrophiclateral sclerosis,” IEEE Pervasive Health, no. 5, pp. 320–324, 2011.

[7] R. J. Jacob, “What you look at is what you get: eye movement-basedinteraction techniques,” in Proceedings of the SIGCHI conf on Humanfactors in Comp Sys. ACM, 1990, pp. 11–18.

[8] M. B. Friedman et al., “Patent: Eye-tracker communication system,”Mar. 3 1987.

[9] C.-S. Lin et al., “Powered wheelchair controlled by eye-trackingsystem,” Optica Applicata, vol. 36, no. 2/3, p. 401, 2006.

[10] E. Wästlund et al., “What you see is where you go: testing agaze-driven power wheelchair for individuals with severe multipledisabilities,” in Proc Symp. on Eye-Tracking Res. & App. ACM,2010, pp. 133–136.

[11] W. W. Abbott, A. Zucconi, and A. Faisal, “Large-field study of ultralow-cost, non-invasive task level bmi,” 2013, pp. 97–100.

[12] S. I. Ktena, W. Abbott, and A. A. Faisal, “A virtual reality platform forsafe evaluation and training of natural gaze-based wheelchair driving,”IEEE Neural Engineering (NER), vol. 7, pp. 236–239, 2015.

[13] K. Essig, M. Pomplun, and H. Ritter, “Application of a novel neuralapproach to 3d gaze tracking: Vergence eye-movements in autostere-ograms,” in Proc. 26th Ann Meet Cog. Sci. Soc, 2004, pp. 357–362.

[14] J. Ki and Y.-M. Kwon, “3d gaze estimation and interaction,” in IEEE3DTV Conf., 2008, pp. 373–376.

[15] C. Hennessey and P. Lawrence, “Noncontact binocular eye-gaze track-ing for point-of-gaze estimation in three dimensions,” IEEE TransBiomed Eng, vol. 56, no. 3, pp. 790–799, 2009.

[16] E. C. Lee et al., “A brain–computer interface method combined witheye tracking for 3d interaction,” J Neurosci Meth, vol. 190, no. 2, pp.289–298, 2010.

[17] C. E. Rasmussen, Gaussian processes for machine learning. MITPress, 2006.

[18] C. E. Rasmussen and H. Nickisch, “Gaussian processes for machinelearning (gpml) toolbox,” J. Machine Learning Res, vol. 11, pp. 3011–3015, 2010.

[19] F. Lotte and C. Guan, “Learning from other subjects helps reducingbrain-computer interface calibration time,” in IEEE Intl. Conf. on Acc.Speech & Sig. Proc. (ICASSP), 2010, pp. 614–617.

[20] C. Gavriel and A. A. Faisal, “Wireless kinematic body sensor networkfor low-cost neurotechnology applications in-the-wild,” IEEE Neural

Engineering (NER), vol. 6, pp. 1279–1282, 2013.