Real-time Gaze-controlled Digital Page Turning System

Yao ChenElectrical Engineering

Stanford Univeristy

yaochen1@stanford.edu

Zhenzhi XiaElectrical Engineering

Stanford University

zxia3@stanford.edu

Aida ZhumabekovaSCPD

Stanford University

adazoom@stanford.edu

Abstract

In this paper we will discuss how we are using imageprocessing to enable real time gaze tracking to use that asa signal to turn the page of a digitally uploaded note book.This project was inspired by the everyday problem manymusicians face with not being able to simultaneously playan instrument and turn the pages of the music sheet. We uti-lize various image processing and computational imagingtechniques we learned in class to showcase how they can beleveraged to solve real world problems with high accuracyand speed.

1. IntroductionFor musicians, turning music sheets by hand while play-

ing an instrument has always posed an unpleasant chal-lenge. The musician often has to stop playing in the middleof the composition to turn the page, therefore a human pageturner is commonly being used during a performance. How-ever, during the personal practicing time musicians are leftto deal with this problem on their own. A pianist for ex-ample, will usually either sacrifice several notes at the endof each page or spend considerable amount of time to prac-tice turning the page as quickly as possible so that the musicflow is not affected. Although, there is a fair amount of digi-tal page turning applications available on the market that usebluetooth connected foot pedals/other physical attachmentsor apps that use music note detection and turn the pages ac-cordingly, such devices still do not provide musicians withenough autonomy because they may require extra foot mo-tion while a pianist needs to pedal the music or, which is themore common downfall, the note detection algorithm willturn the page way too early causing a lot of frustration formusicians. Therefore, we would like to provide a more ro-bust page turning solution to free musicians hands and feetand let them devote wholeheartedly to playing music duringtheir practice sessions and even performance.

Our proposed solution is a program that tracks user’spupil movement and turns pages automatically when the

user signals the intent to turn the page by looking at oneof the corners of the page to turn and intentionally blinking.This allows users to have a free range of hand/feet motionsand to turn pages with an extraordinary ease and speed asthe natural pupil movement is tremendously fast and outper-forms the speed of moving a hand/foot/producing a soundcue to turn.

Since pupil tracking is a notoriously hard problem tosolve as was shown in the Virtual Reality/Augmented Re-ality industry, our program will have to rely on a special-ized hardware that is widely accepted as the go-to solutionfor all sorts of eye activity tracking. We will further discussthe specifics of our chosen hardware device later in this pa-per. We also considered using a webcam as our medium forthis solution because many music players today heavily relyon the electronic music sheets and almost every electronicdevice has a webcam attached, however this forced us torely on a third-party face tracking library which was not al-ways producing robust output and therefore would tamperour performance in some cases. Again, we will discuss thespecifics of the issues later in the paper.

2. Related WorkFor Human-Computer-Interaction purpose, a page turn-

ing system usually consists of a real-time image acquisitionsystem incorporated by some type of the electronic device,an eye tracking system, and a feedback system for pageturning software. The core of the page turning system isto track the page turners eye position e.g. gaze, distanceand its orientation in real time. Tremendous work has beendone in this field [9, 3, 8, 11]. Traditional approach de-tects object position based on the intensity or color distri-bution. Eigen-face detection[11] is one of the methods todo so, and it could easily be extended into the eigen-eye de-tection. However, training such classifier actually requiresa large amount of data due to eye shapes, light conditions,etc. To ease the computational burden of pre-processing thetraining data, Zhiwei Zhu et al. proposed a methodologyfor real-time eye tracking under various lighting conditionsby combining the bright-pupil effect and the conventional

object detection techniques. Another interesting approachwas used by Morimoto et al., who took advantage of pupil-corneal reflection technique for gaze tracking.

3. MethodWe take a video stream from our specialized device and

feed it into our pupil detection script. The image process-ing script is written in Python and uses a range of differentimage manipulation we have learned in class. First, we seg-ment out the pupil from the rest of the eye, this allows usto obtain the gaze direction by calculating the shift betweenthe geometric center and pupil center. We detect when thegaze is directed at either of the edges of the digital notebookthat signals the page to turn. Since our notebook is hostedon a server, when the page turning signal is detected, wesend the real time event to the server through web sock-ets. The server then translates the python response intoJavaScript and sends the command to turn to the next orprevious page in our HTML music sheet. We will discussimage processing steps in greater detail later in the paper.

Figure 1. processing pipeline

3.1. Hardware and Data: Pupil Labs

For the real time data capture, we utilize the lightweightand unobtrusive eye tracker headset provided from PupilLabs[2] to capture the raw video frames. The 3D printedframe holds research grade cameras that are able to recordusers’ field of view and eye movements. The cameras are

suspended right under the user’s eye and can be adjusted tocapture the entire eye region in high quality and resolution.For our purposes we mainly rely on monocular eye move-ment data due to the higher efficiency of streaming.

Figure 2. Pupil Labs eye tracker and example scene of capturingvideo data.

The raw image is given Figure 3. Here, the user can beseen to indicate right, center and left gazes in the order ofmentioning. We continuously obtain raw images of user’seye, and then overlay the detected pupil and gaze informa-tion on the image during the display stage.

Figure 3. Example raw video frame captured by Pupil Labs Cam-era

3.2. Algorithm

This section is devoted to the detailed description of thepupil detection and gaze detection algorithm specially forour digital page turning system.

3.2.1 Pupil Detection

We proposed two algorithms pipelines for pupil detection.In the first algorithm, we detect the pupil based on fast ra-dial symmetry transform (FRST). The pipeline of the firstalgorithm is shown as follows in Figure 4:

Figure 4. Pupil detection image processing pipeline 1 using FRST

For clarity, we describe the FRST algorithms as follows[7]. In FRST algorithm, pupil center estimation is obtainedon the detection of the point presenting the highest radialsymmetry in the image. First, we start with calculatingthe horizontal and vertical gradient of the image. Thenwe apply a empirical threshold to filter out the points withleast significant gradient with small magnitude to reducethe number of pixels to be computed in the transform. Thetransform is calculated for a specific radius n. For each sig-nificant point in gradient map, we define the affected pixelpoints q as the points located at a distance n from point pand to which the gradient vector in p points at, as follows:

q = p+ d g(p)

‖g(p)‖∗ nc (1)

Note that points with positive gradients are related to thedirection from dark regions to bright regions. Therefore,the affected points mainly reside in bright regions. Sincethe pupil center contains a bright glare from reflection un-der lighting conditions, FRST could be applied to detect thebright zone in the pupil and thus estimate the pupil center.For each radius n, an orientation projection On and a mag-nitude projection image Mn are created using the affectedpixels q in the following ways:

On(q) = On(q) + 1

Mn(q) = Mn(q) + ‖g(p)‖(2)

On(q) is the number of affected pixels, and Mn(q) givesthe contribution of each affected pixels in the radius win-dow n based on the magnitude of the gradient. We thencombine both matrices and convolving them with a Gaus-sian smoothing mask G ∼ N(µ, σ), to find the contributionof the radial symmetry of the radius n. α denotes the radialstrictness parameter. High α values eliminate non-radiallysymmetric features, while choosing a low α value includesnon-circular symmetry points of interest. Here we use theempirical value where α = 2.

Sn = (Mn ·Oαn) ∗G (3)

After obtaining the radial symmetry map from fast radialsymmetry transform (FRST), we then apply Otsu’s methodto binarize the radial symmetry map and find the largestconnected components on the radial symmetry map as thepupil region. The centroid of the largest connected compo-nent is calculated as the pupil center.

Figure 8 shows the output of different modules in thefirst image processing pipeline for pupil detection:

The second algorithm takes advantage of dynamicthresholding and contour recognition to locate the pupilcenter. It incorporates the following fives modules: bilateralfiltering, dynamic thresholding, morphological transforma-tion, pupil contour extraction and central moment calcula-

Figure 5. Results from image processing pipeline 1 using thresholdand contour

tion. Figure 6 shows the flowchart of the second imageprocessing pipeline for pupil detection.

Figure 6. Pupil detection image processing pipeline 2 using thresh-old and contour

In order to make the proposed algorithm robust underdifferent light condition, we introduce the dynamic thresh-old to extract the pupil region. Note that pupil region is thedarkest region in the eye regardless of the color of iris. Evenif the iris color is dark, it would not affect the pupil centerlocalization since the iris is moving together with the pupil.

We start finding the dynamic threshold by convertingthe eye image from RGB to gray-scale. Next, a dynamicthreshold is found by using adjusted peak and valley method[4]. Finally, a binary image is obtained with this dynamicthreshold. The histogram of the gray image is calculated inFigure 7 , let g be gray levels, N(g) be number of pixelsin g gray level. Adjusted peak and valley method could bedescribed as follows.

First, we find the all the peaks on the histogram of thegray-scale image, g is a peak if

N(g) > N(g ±∆g)),∆g = 1, ...255 (4)

We store all the peak in Pg . Then find the peak withsmallest level gmin, which is then used as the threshold toextract the pupil region. The resulting binary pupil imagelooks hollow and noisy, thus we apply the morphological

Figure 7. Example histogram of gray video frame

operations, dilation and erosion to fills the holes and re-move the noise. After recovering the shape of the pupil,contours are detected from boundary of the binary eye im-age. Next, based on the number of elements in these con-tours, we choose contours that have this number higher thana certain amount. These contours contain the pupils bound-ary. We then calculate the central moment of the chosencontour regionµpq as follows:

Mpq =∑

x,y∈Regionxp · yq

µpq =∑

x,y∈Region(x− x)p · (y − y)q

x =M10

M00, y =

M01

M00

(5)

The central moment of pupil contour is used as the pupilcenter. Last but not least, we fitted an ellipse to the chosento emphasize the detected pupil shape.

Figure 8 shows the output of different modules in thesecond image processing pipeline for pupil detection:

Figure 8. Results from image processing pipeline 2 using thresholdand contour

3.3. Gaze Detection

As for gaze detection, we start a new algorithm pipelineby applying the morphological edge detector on a raw video

frame to obtain the binary eye contour. Then, corners frombinary image are extracted by using the Harris corner detec-tor [5]. Next, corners that belong to the eye corners areas areselected based on geometric structures eye. After that, thepositions of two eye corners (xl, yl), (xr, yr) are detected.Note that for this algorithm to work, we need to make sureboth ends of the eye appears on the image. Then we cal-culate the arithmetic mean based on the eye corners coor-dinates (Ox, Oy) to obtain position of the geometric centerof the eye. Based on the geometric structure of the eye inFigure 9 , the gaze direction is calculated as follows

Gr =Px

2 ·Ox(6)

Gs =

left, (Gr < 0.45)right, (Gr > 0.56)center, (0.45 < Gr < 0.56)blink, (Gr = None)

(7)

Figure 9. Geometric structure of Eye

Figure 10 shows the output of different modules in thesecond image processing pipeline for pupil detection:

Figure 10. Results from image processing pipeline 3 for gaze de-tection

Gr denotes the ratio of between horizontal coordinatesof the geometric center Ox and pupil center Px. Gs is theresulting gaze state. Note that blink state is obtained hereif we could not detect any pupil contours. Adding blinksupport would make the algorithm more robust, we will il-lustrate the reason in page turning system section.

3.4. Software: Page Turning System Design

First of all, we initialize the center gaze by having theuser focus on the center dot of the screen. In order to easilytrigger the page flipping event, our initial method was tospecify four hot corners as shown in Figure 11. Whenthe gaze is directed to one of these hot corners, the serverwill receive a real time event through web socket and thentranslates this python response into JavaScript and thuscommand the digital music sheet to turn to the previous ornext page depending on which hot corner is detected. Inaddition, the page will only flip when the current gaze isdifferent from the last preserved gaze in order to preventthe duplicated triggering.

Figure 11. illustration of center gaze state and hot corners trigger-ing

In the later development and testing stage, however, inorder to make the page turning algorithm more robust, asshown in Figure 12, we extended these four hot cornersmethod into two threshold regions along with the additionof blinking detection. Now every time before the users startusing the system, they will follow the red dot first, and thentwo red edge lines so that their center gaze, left and rightthreshold gaze states will be recorded down. The compu-tation of gaze shift is constantly running. The page turningmechanism will be only fulfilled when the gaze is directedto the space out of the red lines and a blink is detected. Theblink detection is based on whether our algorithm cannotobtain the pupil contour within three frames. The two redcalibration lines are able to be re-positioned by user by ad-justing offset parameters. This is important in our designin the way that it avoids accidental flipping when users fol-lows the notes on the digital sheet and blinks without in-tention to turn the page. We also add a four-frame wait todistinguish the page turning signaling blink from the natu-ral blink. This waiting time proved to be the best solutionthat is long enough to be reliably different from the natu-ral blink, but still fast enough for the user to turn the page atthe last second, granting a seamless flow in between readingmusic notes and signaling the page to turn. In the end users

are able to quickly turn the pages without posing a tediouslong task.

Figure 12. screenshot of the digital music sheet with calibrationmode on before use

3.5. Web Camera Support

In addition to the eye tracker version, we also workedwith the webcam only version to increase the accessibil-ity of our page turning system. Instead of recording thevideo via Pupil Labs device, we obtain the real-time eyeimage from the built in webcam. This allows us to caterto a broader audience of potential users and significantlysimplifies the set up process. We used Dlib Library [1] toextract the eye region. First, we find the face region withthe Dlib face detection model based on Hessian of Gaus-sian (HoG) features and SVM. Then we locate the faciallandmarks which including eyes, eyebrows, nose, ears, andmouth critical feature points and extract the eye featurespoints.

Figure 13. Eye lankmark points

From the eye feature points in Figure 13 , we could de-rive the eye aspect ratio (EAR) which is close related toblink state.

EAR =‖p2− p6‖+ ‖p3− p5‖

2 · ‖p1− p4‖(8)

Note that, the eye aspect ratio is approximately constantwhile the eye is open, but will rapidly fall to zero whena blink is taking place. Detecting blink would be signifi-cant to our page turning system due to fast movement of theeyeballs. Recall that we design the page turning system tobe triggered only when it first detected the corner gaze andthen followed by blinking. Since people would blink un-consciously, we ask the users to blink for a certain amountof time, approximately last for 1s for the algorithm to de-tect. In this way, the page turning would not be accidentallybe triggered by sweeping the eyeballs to the corner nor un-consciously blinking.

The Eye landmarks also play an important role in gazedetection. On the one hand, The eye corner coordinatewould be obtained from the eye features points set basedon the geometric structure of the eye. We could then de-rive the geometric center from the corner coordinates. Onthe other hand, we also make use of the eye features pointsto calculate the boundary of eye, and crop out the eye re-gion. The cropped eye image would then be feed into theabove algorithm to detect the pupil center. With pupil centerand geometric center, we could calculate the gaze directionbased on Formula 7.

However, the webcam application is not robust since thethird-party library sometimes could not detect face regionand thus might tamper the application in some cases. Mean-while, webcam captures eye image at a distance, thus wehave less number of eye pixels and add more difficulty inreconizing the corner gaze. It requires further parametertuning to accurately obtain the gaze direction.

4. Results and Evaluation

Figure 14. example frame capture of live demo.

Figure 14 shows a live demo of our final page turningsystem. The music score book is turning to the next pagewhen the user looks at the right corner. In this section, wewill evaluate our working page turning systems with the fol-

lowing aspects: run time and accuracy.We used a subset of the labelled pupils in the wild (LPW)

dataset [6, 10] to evaluate the pupil detection algorithm.LPW dataset contains 66 high-quality, high-speed eye re-gion videos. The ground truth coordinates of the pupil isgiven for each frame in the video. We apply both pupil de-tection pipelines on 1000 frames and compare the resultingpupil center coordinates with the ground truth pupil centercoordinates. The performance of the algorithm is evaluatedby calculating the percentage of frames in which pupil cen-ter are precisely located with less then 5px error. Note thataveraging over all frames reflects little about the algorithmquality, since a false detection would greatly increase theMSE and affect the results sharply. Thus, we only calculatethe MSE of frames when pupil is precisely located. The re-sult is given in Table 2. Since real time implementation ofthe page turning system is critical due to the practice con-sider, run time of the proposed algorithm for both pupil de-tection and gaze detection are given in table 1,

Method Run TimePupil detection 1(FRST) 6.107sPupil detection 2(thresholding) 0.0235sGaze detection 0.0328s

Table 1. Run time evaluation of proposed algorithm pipeline

Method Accuracy trueMSEPupil detection 1(FRST) 75.34% 3.675Pupil detection 2(thresholding) 70.09% 3.799Gaze detection - -

Table 2. Accuracy evaluation of proposed algorithm pipeline

From the above tables, We could see that pupil detectionalgorithm 2 with thresholding method is computationallymore efficient than pupil detection algorithm 1 with fast ra-dial symmetry transform. However, with regard to accuracyand mean square error, the FRST performed slightly bet-ter than thresholding method. This might result in the factthat FRST related on the geometrical symmetrical structure,and thus are more robust under different light conditions.Both algorithm are strongly affected when there are thickeye lashes on the frames, which would result in false pupilcenter coordinates and greatly increase the MSE value asmentioned above.

As for gaze detection, we are unable to find an appro-priate data set labelled with ground-truth gaze to evaluateour gaze detection at this time. Instead, we perform a smallsubjective case study to evaluate the user experience withour page turning system by measuring the percentage of ef-

fective page turning operations. The results are given inTable 4. We could see that better accuracy are obtainedwhen detecting the right corner gaze. This might result inthe relatively different movement of eye ball required forlooking left and look right. The pupil center shift might bemore obvious when looking right than looking left.

look left look rightcorrect detection 125 138

total try 151 150accuracy 82.7% 91.4%

Table 3. Accuracy of the page turning system

5. Discussion and Future Work• During our evaluation trials we noticed that users who

wear glasses are not consistently receiving accurate re-sults. This proved to be an outcome of our algorithmwrongfully detecting glares on the glasses as pupil.This problem can be easily solved by adding a colordetection fragment to our pipeline that matches theoriginal color of the detected region to the possiblepupil color, simply put if the color of the detected pupilis not black it should not be classified as pupil.

• Another user specific issue that affected our algorithmquality is posed by the false detection of the pupil re-gion, which is caused by misinterpreting dark coloredlashes as a pupil. If this happens during runtime, thepage can be accidentally turned when the user did notintend to do so. However, if this happens during the setup process, which happened only once in our trial runs,then the center of the eye will not be detected correctlywhich will produce unexpected results for turning. Theproblem seems to only affect users with black, well de-fined long lashes. As it appears to be this is a knownproblem in other pupil tracking techniques as well, andthe only way to overcome this is by making our pupildetection algorithm more robust.

• As an extension for our project we explored using thebuilt in web camera for video recording instead of thespecialized PupilLabs hardware. For that we utilizedDlib facial recognition library that allowed us to seg-ment out user’s eyes from the rest of the backgroundin every frame. However, using Dlib library eye track-ing considerably slowed down the entire pipeline to thepoint where the real time page turning was not effec-tive. Instead, we can try using OpenCV facial recogni-tion library to specify the region of the user’s face anddo the eye segmentation ourselves by using templatematching or possibly a training dataset which would

recognize the eye region. This will allow us to identifypotential bottlenecks and address them accordingly tospeed up the entire process.

• Also, as we collect more reference data and obtainmore usage time, we will need to address the issueof quality testing. Right now, we are using the out-put video from the user’s session to manually evalu-ate when the page turning is triggered and compare itagainst the turn history of our algorithm. This is nota trivial task and is not scale-able. As a potential so-lution we can automize this process by simultaneouslyrunning the gaze detection provided by PupilLabs andusing that output as the source of ground truth againstour custom output. Since PupilLabs is a library reliedon by a large number of other users and research it issafe to assume their gaze tracking is fairly reliable. Wecan overlay the graphical output we produce with theirvideo output and compare the two pupil outlines frameby frame in an automated manner which will producea percentage of identical frames. Ideally, we shouldstrive for ¿99

6. External LinkInitial result with prerecorded eye movement video data:video link here

Final result with live video stream data: video link here

References[1] Dlib Library, http://dlib.net/.[2] Pupil Labs, https://pupil-labs.com/.[3] R. K. Bellamy, B. E. John, J. T. Richards, C. B. Swart, J. C.

Thomas, S. M. Trewin, et al. Correcting systematic calibra-tion errors in eye tracking data, Oct. 10 2017. US Patent9,782,069.

[4] N. H. Cuong and H. T. Hoang. Eye-gaze detection with asingle webcam based on geometry features extraction. In2010 11th International Conference on Control AutomationRobotics & Vision, pages 2507–2512. IEEE, 2010.

[5] K. G. Derpanis. The harris corner detector. York University,2004.

[6] Y. S. A. B. Marc Tonsen, Xucong Zhang. Labelled pupils inthe wild (LPW) dataset.

[7] I. Martinikorena, R. Cabeza, A. Villanueva, I. Urtasun, andA. Larumbe. Fast and robust ellipse detection algorithm forhead-mounted eye tracking systems. Machine Vision and Ap-plications, 29(5):845–860, 2018.

[8] C. H. Morimoto and M. R. Mimica. Eye gaze tracking tech-niques for interactive applications. Computer vision and im-age understanding, 98(1):4–24, 2005.

[9] P. B. Nguyen, J. Fleureau, C. Chamaret, and P. Guillotel.Method for calibration free gaze tracking using low costcamera, Oct. 23 2018. US Patent App. 14/389,783.

[10] M. Tonsen, X. Zhang, Y. Sugano, and A. Bulling. Labelledpupils in the wild: a dataset for studying pupil detection inunconstrained environments. In Proceedings of the Ninth Bi-ennial ACM Symposium on Eye Tracking Research & Appli-cations, pages 139–142. ACM, 2016.

[11] M. Turk and A. Pentland. Eigenfaces for recognition. Jour-nal of cognitive neuroscience, 3(1):71–86, 1991.