Pupil: An Open Source Platform forPervasive Eye Tracking and MobileGaze-based Interaction

Moritz KassnerPupil Labs UGBerlin, Germanymoritz@pupil-labs.com

William PateraPupil Labs UGBerlin, Germanywill@pupil-labs.com

Andreas BullingMax Planck Institute forInformaticsPerceptual User InterfacesGroupSaarbrucken, Germanyandreas.bulling@acm.org

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bearthis notice and the full citation on the first page. Copyrights for componentsof this work owned by others than the author(s) must be honored. Abstractingwith credit is permitted. To copy otherwise, or republish, to post on servers orto redistribute to lists, requires prior specific permission and/or a fee. Requestpermissions from permissions@acm.org. UbiComp ’14, September 13 - 172014, Seattle, WA, USA Copyright is held by the owner/author(s).Publication rights licensed to ACM. ACM 978-1-4503-3047-3/14/09$ 15.00.http://dx.doi.org/10.1145/2638728.2641695

AbstractIn this paper we present Pupil – an accessible, affordable,and extensible open source platform for pervasive eyetracking and gaze-based interaction. Pupil comprises 1) alight-weight eye tracking headset, 2) an open sourcesoftware framework for mobile eye tracking, as well as 3) agraphical user interface to playback and visualize videoand gaze data. Pupil features high-resolution scene andeye cameras for monocular and binocular gaze estimation.The software and GUI are platform-independent andinclude state-of-the-art algorithms for real-time pupildetection and tracking, calibration, and accurate gazeestimation. Results of a performance evaluation show thatPupil can provide an average gaze estimation accuracy of0.6 degree of visual angle (0.08 degree precision) with aprocessing pipeline latency of only 0.045 seconds.

Author KeywordsEye Movement; Mobile Eye Tracking; WearableComputing; Gaze-based Interaction

ACM Classification KeywordsC.3 [Special-purpose and application-based systems]:Real-time and embedded systems.; H.5 [InformationInterfaces and Presentation]: User Interfaces.

1151

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

IntroductionRecent advances in head-mounted eye tracking andautomated eye movement analysis point the way towardunobtrusive eye-based human-computer interfaces that arepervasively usable in everyday life. This new paradigm iscalled pervasive eye tracking - continuous eye monitoringand analysis [2]. The ability to track and analyze eyemovements anywhere and at any time will enable newresearch to develop and understand visual behavior andeye-based interaction in daily life settings.

Commercially available head-mounted eye trackingsystems are robust and provide useful features tocustomers in industry and research, such as for marketingstudies, website analytics, or research studies. However,commercial systems are expensive, therefore typically usedby specialized user groups, and rely on closed sourcehardware and software. This limits the potential scale andapplication areas of eye tracking to expert users andinhibits user-driven development, customization, andextension. Open source software (OSS) eye trackers haveemerged as low cost alternatives to commercial eyetracking systems using consumer digital camera sensorsand open source computer vision softwarelibraries [1, 8, 13, 9, 10, 11, 4, 16]. The OSS routeenables users to rapidly develop and modify hardware andsoftware based on experimental findings [2].

We argue that affordability does not necessarily align withaccessibility. In this paper we define accessible eyetracking platforms to have the following qualities: opensource components, modular hardware and softwaredesign, comprehensive documentation, user support,affordable price, and flexibility for future changes.Towards this goal we describe Pupil, a mobile eye trackingheadset and an open source software platform as an

Figure 1: Front rendering of the Pupil Pro headset (rev 20)showing the frame, tiltable scene camera and adjustable eyecamera.

accessible, affordable, and extensible tool for pervasive eyetracking research. We explain the design motivation,provide an in depth technical description of both hardwareand software, and provide an analysis of accuracy andperformance of the system.

System OverviewPupil is a mobile eye tracking headset with one scenecamera and one infrared (IR) spectrum eye camera fordark pupil detection. Both cameras connect to a laptop,desktop, or mobile computer platform via high speed USB2.0. The camera video streams are read using PupilCapture software for real-time pupil detection, gazemapping, recording, and other functions.

1152

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

System Design ObjectivesWe made several design decisions while satisfying anumber of factors to balance ergonomic constraints withperformance. Pupil leverages the rapid development cycleand scaling effects of consumer electronics - USB camerasand consumer computing hardware - instead of usingcustom cameras and computing solutions. Pupil headsetsare fabricated using Selective Laser Sintering (SLS)instead of established fabrication methods like injectionmolding. This rapid fabrication process accommodatesfrequent design changes, comparable to the continuousdevelopment of Pupil software. We employed modulardesign principles in both hardware and software to enablemodifications by users. Pupil software is open source andstrives to build and support a community of eye trackingresearchers and developers.

Figure 2: Renderingdemonstrating the range ofmotion of the world camera(scene camera) mount andconnection to the frame.

Figure 3: Renderingdemonstrating the range ofmotion of the eye camera mountand connection to the frame.

Pupil Headset Design and HardwareFactors that influence the headset design are: mobility,modularity and customization, minimizing visualobstruction and weight, accommodation of various facialgeometries, minimization of headset movement due toslippage and deformation, durability, and wear comfort.

HeadsetThe Pupil headset is made up of three modules: frame,scene camera mount, and eye camera mount.

FrameThe frame was designed based on a 3D scan of a humanhead and iteratively refined to accommodate physiologicalvariations between users. We developed a novel designprocess where we apply specific forces to deform theheadset model using Finite Element Analysis (FEA).

We use Finite Element Analysis (FEA) to apply forcesthat push the earpieces of the frame together. We print

the resulting deformed geometry from the simulation.When the user puts on the headset, the earpieces of theframe are pushed apart, and the entire frame flexes open.This process ensures that cameras align as designed whenthe headset is worn and results in a comfortable, snugfitting, and lightweight (9g) frame.

Camera MountsCamera mount geometries are hosted in a Git repository.By releasing the mount geometry we automaticallydocument the interface, allowing users to develop theirown mounts for cameras of their choice. The scenecamera mount connects to the frame with a snap fittoothed ratcheting system that allows for radialadjustment within the users vertical field of vision (FOV)along a transverse axis within a 90 degree range. The eyecamera mount is an articulated adjustable armaccommodating variations in users eyes and facegeometries. The camera mount attaches to the framealong a snap fit sliding joint. The eye camera orbits on aball joint that can be fixed by tightening a single screw.

CamerasPupil uses USB cameras that comply with the UVCstandard. Other UVC compliant cameras can be usedwith the system as desired by the user. The Pupil headsetcan be used with other software that supports the UVCinterface. Pupil can be easily extended to use two eyecameras for binocular setups and more scene cameras.

Computing DevicePupil works in conjunction with standard multipurposecomputers: laptop, desktop, or tablet. Designing for usersupplied recording and processing hardware introduces asource for compatibility issues and requires more setupeffort for both users and developers. However, enablingthe user to pair the headset with their own computing

1153

WORKSHOP: PETMEI

Figure 4: Screen capture of Pupil Capture. Left) World window displays real-time video feed of the user’s FOV from the scene cameraalong with GUI controls for the world camera and plugins. Red circle is the gaze position of the user. Top Right) Eye window displaysreal-time video feed of the user’s eye. Bottom Right) Plugin window plugins can also spawn their own windows. Shown here is a pluginto visualize the pupil detection algorithm.

platform makes Pupil a multipurpose eye tracking andanalysis tool. Pupil is deployable for lightweight mobileuse as well as more specialized applications like: streamingover networks, geotagging, multi-user synchronization;and computationally intensive applications like real time3D reconstruction and localization.

Pupil SoftwarePupil software is divided into two main parts, PupilCapture and Pupil Player. Pupil Capture runs in real-timeto capture and process images from the two (or more)camera video streams. Pupil Player is used to playbackand visualize video and gaze data recorded with PupilCapture. Source code is written in Python and C. Pupil

software can be run from source on Linux, MacOS, andWindows or executed as a bundled double clickapplication on Linux and MacOS.

Pupil Detection AlgorithmThe pupil detection algorithm locates the dark pupil in theIR illuminated eye camera image (see Figure 6). In mostenvironments, the algorithm is robust against reflectionsin the pupil area. Furthermore, the algorithm does notdepend on the corneal reflection for detection, and workswith users who wear contact lenses and eyeglasses.

The eye camera image is first converted to grayscale. Theinitial region estimation of the pupil is found via the

1154

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

strongest response for a center-surround feature asproposed by Swirski et al. [15] within the image.

Fig 6-1) We use Canny [3] to find contours in the imageand filter edges based on neighboring pixel intensity. Fig6-2) We then look for darker areas (blue region) using auser-defined offset of the lowest spike in the histogram ofpixel intensities in the eye image. Fig 6-3) We filterremaining edges to exclude those stemming from spectralreflections - yellow region. Remaining edges are extractedinto into contours using connected components [14]. Fig6-4) Contours are filtered and split into sub-contoursbased on criteria of curvature continuity. Fig 6-5)Candidate pupil ellipses are formed using ellipse fitting [5]onto a subset of the contours looking for good fits in aleast square sense, major radii within a user defined range,and a few additional criteria. A combinatorial search looksfor contours that can be added as support to thecandidate ellipses. The results are evaluated based on theellipse fit of the supporting edges and the ratio ofsupporting edge length and ellipse circumference (usingRamanujans second approximation [7]). We call this ratio“confidence”. Fig 6-6) If the best results confidence isabove a threshold the algorithm reports this candidateellipse as the ellipse defining the contour of the pupil.Otherwise the algorithm reports that no pupil was found.

Error threshold (pixels)0 2 4 6 8 100

20

40

60

80

100

Detection rate(%)

Error threshold (pixels)0 2 4 6 8 100

20

40

60

80

100

Detection rate(%)

Świrski et al.Pupil

1

ITU Starburst

Pupil w/o ‘p1 right’Pupil full data set

2

Figure 5: 1) Benchmarkcomparison 4 pupil detectionalgorithms. (Figure data foralgorithms other than Pupil usedwith permission of Swirski.) 2)Pupil’s detection algorithmcomparison between fullbenchmark dataset and subset.

Figure 5 shows a performance comparison between Pupil’spupil detection algorithm, the stock algorithm proposed bySwirski et al., the ITU gaze tracker, and Starburst on thebenchmark dataset by Swirski et al. [15]. As error measurewe used the Hausdorff distance between the detected andhand-labeled pupil ellipses [15]. We additionallyconducted a test excluding the dataset p1-right, thatcontains eye images recorded at the most extreme angles,as those do not occur using Pupil hardware.

1 2 3

654

Figure 6: Visualization of pupil detection algorithm.

Fig 5-2 is a more accurate demonstration of how Pupilperforms in real life conditions. Pupil achieves a detectionrate of 80% with an error threshold of 2 pixels; at 5 pixelserror detection rate increases to 90%.

Gaze MappingMapping pupil positions from eye to scene space isimplemented with a transfer function consisting of twobivariate polynomials of adjustable degree. The userspecific polynomial parameters are obtained by runningone of the calibration routines using screen markers,manual markers or natural features. Calibration andmapping functions are abstracted and the underlyingmodels can easily be modified and replaced if needed.

Surface DetectionPupil Capture can detect planar reference surfaces in thescene using markers. There are 64 unique markers. Aminimum of 1 marker is required to define a surface. Gazepositions are mapped into the reference surface coordinate

1155

WORKSHOP: PETMEI

system using homographic transformations between thescene camera plane and reference surface plane.

RecordingPupil Capture can record the scene and eye videos,associated frame timestamps, detected pupil and gazeposition data, and additional user activated plugin data.

StreamingPupil Capture can send real-time gaze and pupilinformation as well as plugin generated data via ZMQ toother applications and network enabled devices. Please seefurther documentation on our wiki: https://github.com/

pupil-labs/pupil/wiki/Pupil-Capture#pupil-server

Plugin StructureEven a standard feature, like recording a video, isabstracted as a plugin. This level of abstraction andmodularity allows for users to develop their own tools evenfor low level functionality. In Pupil Capture plugins can belaunched in either world or eye processes. The modularstructure makes it easy to for users to test out differentmethods at runtime and for developers to extend softwarewithout breaking existing functionality. Plugins have theability to create their own GUI within the process window,access to shared memory like gaze data, and spawn theirown windows. Please see our wiki for furtherdocumentation on developing plugins: https:

//github.com/pupil-labs/pupil/wiki/Plugin-Guide

Performance EvaluationSpatial Accuracy and PrecisionAs performance metrics for spatial accuracy and precisionof the system we employ the metrics defined in COGAINEye tracker accuracy terms and definitions [12]

“Accuracy was calculated as the average angular offset

(distance) (in degrees of visual angle) between fixationslocations and the corresponding locations of the fixationtargets.”

“Precision was calculated as the Root Mean Square(RMS) of the angular distance (in degrees of visual angle)between successive samples to xi, yi to xi+1,yi+1.”

We evaluated the performance in an indoor environmentusing the following procedure: A subject wearing the PupilPro eye tracker sat approximately 0.5m away from a 27inch computer monitor. The eye tracker was calibratedusing the standard 9-point screen marker based calibrationroutine in full screen mode. After calibration the subjectwas asked to fixate on a marker on the monitor. Themarker sampled 10 random positions for 1.5 seconds andthen revisited each of the 9 calibration positions. Whilethe marker was displayed, gaze samples and the positionof the marker detected in the scene camera were recorded.The data was then correlated into gaze point and markerpoint pairs based on minimal temporal distance. Gazepoint marker point pairs with a spatial distance of morethan 5 degrees angular distance were discarded as outliersas they do not correlate to system error but human error(no fixation, blink) [6].

Accuracy and precision were calculated in scene camerapixel space and converted into degrees of visual anglebased on camera intrinsics.

This procedure was repeated with eight different subjectsto reflect a more diverse pool of physiologies. The testwas conducted with a Pupil Pro eye tracker revision 021.The test used Pupil Capture software, version 0.3.8running on Mac OS. The test routine is part of PupilCapture releases, starting with version 0.3.8 and availableto all users.

1156

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Figure 7: Results of accuracy test for one user. Marker targetin green. Gaze point in red. Correspondence error in orange.Notice sample point with big error - during this sample period,the subject was not looking at the marker at the beginning ofthe test.

ResultsUnder ideal conditions 0.6 degrees of accuracy and 0.08degrees of precision are achieved.

Temporal Accuracy, Latency and PrecisionA second critical aspect are the temporal characteristics ofthe pupil eye tracking system.

Stream SynchronizationTimestamping is crucial for stream synchronizationbecause data is obtained from two independent freerunning video sources. Additionally timestamps are usedfor correlation with additional external experiment orsensor data. Thus we strive to obtain a timestamp that is

closest to the timepoint of data collection (in our casecamera sensor exposure).

Pupil capture has two image timestamp implementations:

When the imaging camera is known to produce validhardware timestamps, Pupil Capture uses these hardwareimage timestamps. These timestamps have high precisionand accuracy as they are taken at the beginning of sensorexposure by the camera and transmitted along with theimage data. The hardware timestamp accuracy exceedsour measurement capabilities. The variation of exposuretimes (jitter) reported by the hardware timestamps wemeasure by calculating the standard deviation of 1400successively take frame times. It is 0.0004s for the worldcamera and 0.0001s for the eye camera.

Hardware timestamping is implemented in the Linuxversion of Pupil Capture and supported by both camerasof the Pupil Pro headset revision 020 and up. When PupilCapture runs on an OS without timestamp video driversupport or does not recognize the attached cameras asverified hardware timestamp sources, Pupil Capture usessoftware timestamps as a fallback. The timestamp istaken when the driver makes the image frame available tothe software. Software timestamps are by nature of worseaccuracy and precision, as they are taken after exposure,readout, image transfer, and decompression of the imageon the host side. These steps take an indeterministicamount of time, which makes it impossible to accuratelyestimate time of exposure. Accuracy and precision dependon the camera and video capture driver.

System LatencyFor real-time applications the full latency of Pupil is ofgreat concern. We obtain processing times of functionalgroups in the Pupil hardware and software pipeline by

1157

WORKSHOP: PETMEI

frame time 0.033

frame time 0.033

frame timeoffset

exposure, transport, image formation

pupil detection

write result to queue

read queue, gaze mapping

marker detection, broadcast via 0mq

0.119

0.038

Queue

0.0002

0.0002 0.0045

0.0063

2

1

start of sensor exposure

Figure 8: System latency diagram shows 1) eye process and 2) scene processes and times between sensor exposure and key points inthe processing pipeline. Gray bars represent video streams of each process with vertical divisions as individual image frames within thestreams. Color bars for key functions annotated with processing times.

calculating the time between sensor exposure and keypoints in the workflow. These temporal factors in signaldelay are presented in Figure 8. All measurements weretaken using Pupil Pro headset rev022 connected to aLenovo X201 laptop with an Intel Core i7 620M CPUrunning Ubuntu 12.04. It should be noted that inreal-time applications synchronicity of data is sacrificedfor recency of data. The eye process pipeline is about 1/3the latency of the world process pipeline. Therefore, wechoose to broadcast the eye information as soon as itbecomes available instead of waiting for the temporallyclosest scene image to become available.

Using the most recent data only makes sense for real-timeapplications. No sacrifices are made for any offlinecorrelation of data employed by calibration, testing, orplayback of recorded data. Furthermore, this approach

does not prevent the user from obtaining accuratetemporal correlation in real-time or offline applications.With this approach we can characterize the systemlatency for the eye pipeline, world pipeline separately:

• Total latency of the eye processing pipeline fromstart of sensor exposure to availability of pupilposition: 0.045s (measured across 1400 sampleswith a standard deviation of 0.003 s)

• Total latency of the world pipeline including the eyemeasurement from start of sensor exposure tobroadcast of pupil, gaze, and reference surface datavia network: 0.124 s (measured across 1200 sampleswith a standard deviation of 0.005 s)

1158

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA

Minimum Hardware RequirementsThe Pupil eye tracking system works with a traditionalmultipurpose computer - laptop, desktop, or tablet. It istherefore important to determine the minimum hardwarespecifications required to run Pupil Capture software inreal-time. We tested the performance using a 11 inchMacbook Air (2010 model) with 2 GB of RAM and anIntel Core2Duo SU9400 dual core CPU. The softwareversion used was Pupil Capture v0.3.8. The OS used onthe machine specified above was Ubuntu 13.10.

Our performance test demonstrates that the system’sCPU load never went above 90 percent, using the abovehardware running Pupil Capture in recording mode, pupildetection at 30 fps and the world camera capture at 24fps. The hardware setup was selected because it representsa portable computing platform with limited computingpower. Any computer with a Intel “i” series processor orequivalent will have sufficient CPU resources, whencomparing CPU benchmarks. Pupil Capture relies onseveral libraries to do video decompression/compression,image analysis, and display with platform specificimplementations and efficiencies. Therefore, our test usinga Linux distribution can not be generalized to Windows orMacOS. We found that requirements were similar forMacOS 10.8 and above. We did not establish Windowshardware requirements at the time of writing.

ConclusionIn order to further advance in eye tracking and to supportthe pervasive eye tracking paradigm, we will requireaccessible, affordable, and extensible eye tracking tools.We have developed Pupil as a contribution to the eyetracking research community. Pupil is already used in awide range of disciplines and has developed a communityof researchers and developers. Current limitations to the

system are parallax error and tracking robustness in IRrich environments. Both Pupil software and hardware areunder active development. Future developments will focuson hardware and software in parallel. The next big stepsplanned for Pupil are to improve mobility, implementreal-time pose tracking and scene mapping, simplify userexperience, and improve pupil tracking.

Links1. Pupil mobile eye tracking headset:

http://pupil-labs.com/pupil

2. Pupil open source code repository:http://github.com/pupil-labs/pupil

3. Pupil User Guides:https://github.com/pupil-labs/pupil/wiki/

4. Pupil user group forum: http://groups.google.

com/forum/#!forum/pupil-discuss

5. Pupil open source headset mount repositoryhttps://code.google.com/p/pupil/source/browse/

?repo=hardware

6. Pupil on arxiv.org http://arxiv.org/abs/1405.0006

References[1] Babcock, J. S., and Pelz, J. B. Building a

lightweight eyetracking headgear. In Proceedings ofthe 2004 symposium on Eye tracking research &applications, ACM (2004), 109–114.

[2] Bulling, A., and Gellersen, H. Toward MobileEye-Based Human-Computer Interaction. IEEEPervasive Computing 9, 4 (2010), 8–12.

[3] Canny, J. A computational approach to edgedetection. Pattern Analysis and Machine Intelligence,IEEE Transactions on, 6 (1986), 679–698.

[4] Ferhat, O., Vilarino, F., and Sanchez, F. J. A cheapportable eye-tracker solution for common setups.

1159

WORKSHOP: PETMEI

Journal of Eye Movement Research 7, 3 (2014),2:1–10.

[5] Fitzgibbon, A. W., Fisher, R. B., et al. A buyer’sguide to conic fitting. DAI Research paper (1996).

[6] Global, T. Accuracy and precision test method forremote eye trackers, 2011.

[7] Hardy, G., Seshu Aiyar, P., and Wilson, B. Collectedpapers of srinivasa ramanujan. Cambridge, London(1927).

[8] Li, D., Babcock, J., and Parkhurst, D. J. openeyes:a low-cost head-mounted eye-tracking solution. InProceedings of the 2006 symposium on Eye trackingresearch & applications, ACM (2006), 95–100.

[9] Lukander, K., Jagadeesan, S., Chi, H., and Muller,K. Omg!: A new robust, wearable and affordableopen source mobile gaze tracker. In Proceedings ofthe 15th international conference onHuman-computer interaction with mobile devices andservices, ACM (2013), 408–411.

[10] Mantiuk, R., Kowalik, M., Nowosielski, A., andBazyluk, B. Do-it-yourself eye tracker: Low-costpupil-based eye tracker for computer graphicsapplications. In Advances in Multimedia Modeling.Springer, 2012, 115–125.

[11] Mardanbegi, D. Haytham gaze tracker: Diy hardwareand open source software, March 2013.

[12] Mulvey, F. Woking copy of definitions andterminology for eye tracker accuracy and precision,2010.

[13] Ryan, W. J., Duchowski, A. T., and Birchfield, S. T.Limbus/pupil switching for wearable eye trackingunder variable lighting conditions. In Proceedings ofthe 2008 symposium on Eye tracking research &applications, ACM (2008), 61–64.

[14] Suzuki, S., et al. Topological structural analysis ofdigitized binary images by border following.Computer Vision, Graphics, and Image Processing30, 1 (1985), 32–46.

[15] Swirski, L., Bulling, A., and Dodgson, N. Robustreal-time pupil tracking in highly off-axis images. InProceedings of the Symposium on Eye TrackingResearch and Applications, ACM (2012), 173–176.

[16] Topal, C., Gerek, O. N., and Dogan, A. Ahead-mounted sensor-based eye tracking device: Eyetouch system. In Proceedings of the 2008Symposium on Eye Tracking Research &Applications, ETRA ’08, ACM (New York, NY, USA,2008), 87–90.

1160

UBICOMP '14 ADJUNCT, SEPTEMBER 13 - 17, 2014, SEATTLE, WA, USA