Fundus tracking with the scanning laser ophthalmoscope

D. P. Wornson, George W. Hughes, and Robert H. Webb

A technique is described of using the video output of the scanning laser ophthalmoscope to monitor thepositions of fundus features with respect to an input laser raster. The monitoring performance characteris-tics are discussed as well as tracking methods and possible applications in psychophysics and laser photocoag-ulation.

1. Introduction

A number of different techniques are used for moni-toring eye position, some with very high accuracy andbandwidth. Almost all of them derive their signalsfrom the front of the eye or from apparatus attached tothe front of the eye.1 We describe here an instrumentand technique capable of tracking features in the backof the eye-the fundus.

For some purposes the aim of an eye-tracker is toproduce a constant or highly controlled spatial corre-spondence between a particular locality on the sub-ject's fundus and, for example, (1) a graphic stimulus,(2) a photocoagulation beam, (3) a light detector, or (4)the location of this area of interest within a fundusimage (e.g., feature registration). Using a computer-based tracking system linked to the scanning laserophthalmoscope (SLO) it is possible to (1) select afeature or area of interest on the fundus, (2) establishthe position of a light input or detector, directly ob-serving this position until satisfactory, and (3) keepthe light input or detector at the selected position withrespect to the fundus. Since the SLO signal is in videoformat, a record is available by videotape recording.

Fundus position is monitored 60 times/s by real-time analysis of images as they are digitized and storedin a frame buffer. The analysis is done on a microcom-puter (an SBE 200). The tracking programs can betailored to the task at hand, if necessary compromisinga less important factor to enhance a critical one. Suchfactors include signal delay, slew rate, and precision.The particular program described here represents acompromise among all these factors.

The authors are with Eye Research Institute of Retina Founda-tion, 20 Staniford Street, Boston, Massachusetts 02114.

Received 1 July 1986.0003-6935/87/081500-05$02.00/0.© 1987 Optical Society of America.

U. Scanning Laser Ophthalmoscope

The SLO is both a camera and projector. 2 Just as aTV sweeps a thin electron beam across a TV screen, theSLO sweeps a spot of laser light across the retinaforming a raster. In the same way that TV produces apicture by varying the intensity of the electron beam,the SLO is capable of projecting graphics on the retinaby modulating the intensity of the laser beam. Lightwhich is reflected back through the pupil is collectedand converted to an electronic signal which becomesthe video input to a CRT monitor.

The particular version of the SLO used here was anonconfocal one in which the horizontal scan was donewith an acoustooptic deflector rather than the movingmirror used for other versions. The light input con-sisted of yellow and red lasers combined.

The laser raster can be physically divided into Car-tesian coordinates, the Y coordinate being a line num-ber and the X coordinate, a pixel number. The pixelnumber results from dividing up the horizontal scantime with a crystal clock. Pixel clocks are used in boththe SLO raster graphics controller and the frame-grab-ber which converts its video images to data in comput-er memory. These clocks can be synchronized, mak-ing input (graphics) positions correspond exactly tooutput (grabbed image) positions having the same co-ordinates. The primary output of the fundus trackerdescribed here is a stream of X- Y coordinates in thisnative Cartesian system. Thus a given position read-ing means "along the path traveled by the input laserbeam at line Y after X pixel-clock cycles." The mainpurpose here will be to describe how position readingsare obtained by analyzing the image memory of theframe-grabber.

How to use this position information depends on theparticular application. In the tracker's most basicform, the position information is used to write a fidu-cial cross into frame buffer memory for display (Fig. 1)or for projection onto the fundus via the SLO raster.Because of this projection option, the system is actual-ly a tracker even in this basic form and not simply a

1500 APPLIED OPTICS / Vol. 26, No. 8 / 15 April 1987

Fig. 1. To obtain these fiducial crosses, a 6-bit frame-grabber was set to continual-grab mode with the SLO as its source; the grabbed imageswere analyzed in real time by a computer; and a fiducial cross was written into frame-grabber memory prior to output for display. The frame-grabber output, including cross, was videotaped and studied frame-by-frame. Error in eye-movement monitoring was investigated bymeasuring the amount of variation in the placement of the fiducial, which had been placed approximately over the left edge of the vessel.Since each frame consists of two successive images, it can be seen that in one photo the velocity of eye motion is almost zero, while in the otherphoto the velocity is 180'/s. (The white optic disk is -5° in diameter.) Note: The images seen here have been degraded by sixreproductions into different media before printing-video to 6-bit digital to video to tape to video to CRT to film. Frame-by-frame analysiswas done at the CRT stage, where it was judged that resolution of cross position vs vessel position by human inspection was possible to aboutone-fifth the width of the fiducial cross (13 min of arc). After reproduction to film, this resolution dropped to about one-third of the width of

the vessel (-20 min of arc), probably because the film represents integration of about sixteen frames of paused VCR signal.

monitor. In another tracker configuration which willbe described, the raster itself was steered to compen-sate for eye movement, providing a relatively stableimage of the fundus feature regardless of eye positionwithin the tracking field.

Ill. Fundus-Tracking Soffware

The particular program which will be evaluated herewas written as a study of the possibility of locking ontoa blood vessel for automatically guided laser photoco-agulation. The algorithm has two stages which consistof a coarse approximation and fine adjustment. Thefirst stage calculates the centroid of the cup region ofthe optic disk. The second stage locates the bloodvessel using a special template. The algorithm is per-formed once per video field (60 times/s) with eachiteration requiring 1/120 s.

The centroid method of calculating the position ofthe optic disk was selected from a number of methods,some of which included spatial derivative information.The other methods, which included edge detection andcenter-of-mass approaches, were tried and rejectedmostly because they required much more time toachieve a given level of reliability and accuracy. Thecentroid method was excellent as a first approximationbecause it was extremely fast, moderately accurate,and reliable.

The centroid algorithm begins by positioning a soft-ware window around the last known location of thedisk in frame-grabber memory. Pixel values within

the window are sampled in a grid pattern to determinetwo things: (1) the coordinates of those pixels whosebrightness exceeds the current threshold value and (2)the brightest pixel in the grid, from which is derivedthe current threshold value to be used in the nextiteration of the algorithm. The current threshold isdefined as the brightest pixel value in a grid minus asmall constant. The centroid calculation itself simplyrequires finding the average X and average Y coordi-nates of those pixels brighter than the threshold.

The vessel-finding stage in the algorithm uses a tem-plate-correlation method (Fig. 2). In essence the tem-plate searches for two parallel edges of opposite gradi-ent which are close to each other and at the correctorientation. The nine-point template has to be pre-pared beforehand so that it has the proper size andorientation to fit the particular locality in the particu-lar eye. The template is moved horizontally across thearea where the vessel is expected, and the position withthe highest match value indicates the vessel's location.The match value is computed by adding the values atlocations corresponding to the six off-vessel points ofthe template and subtracting twice the values at loca-tions corresponding to the three on-vessel points.

IV. Characteristics of the Eye-Movement Signal

Theoretical analysis and frame-by-frame inspectionof a videotape were used to help describe the eye move-ment monitoring signal of the tracker. These were theindicated characteristics:

15 April 1987 / Vol. 26, No. 8 / APPLIED OPTICS 1501

........

. l, 0

I

I

Fig. 2. Two stages of the fundus tracking algorithm are illustratedhere. First, pixels are sampled and thresholded in a grid patterncovering the optic disk. The centroid of the brightest pixels in thedisk is calculated. In the second stage, at a fixed offset from thiscentroid, a vessel-recognizing template is swept horizontally insearch of the vessel. A template-correlation value is calculated byadding the values of the six off-vessel positions and subtracting twicethe values at the three on-vessel positions. The template position

having the highest correlation value is the one selected.

A. Resolution

The SLO field of view used was 55°, and the frame-grabber resolution was 256 X 256; thus the width of apixel at the fundus was 13 min of arc. The resolutionof the vessel-finding algorithm itself was limited to this13 min of arc, since it was selected among various pixelpositions.

B. Extent of Tracking

The area being tracked remained clear and unob-structed enough to track motions within a field of 160when the eye was dilated. The program assumed ran-dom movement within this range.

C. Sampling Rate

Timing aspects of any SLO-based system are, ofcourse, highly constrained by the video format of theSLO signal, which yields one image (one sample) every1/60 s.

D. Tracking Rate (Slew Rate)

The disk portion of the tracking algorithm cannot beexpected to perform well if any part of the cup regionbeing tracked moves outside of its software window.The window used was 110 square, and the area beingtracked was 1.5° in diameter, so the maximum veloci-ty for accurate disk-tracking was theoretically 4.50/field or 270°/s. Somewhat larger movements wouldlead to only semiaccurate tracking with a maximumindicated motion of 5.5°. The second stage of thealgorithm-the vessel-finder-should be capable ofcatching up horizontally by another 1.50. Thus themaximum distance which can be followed from one

field to the next should be 5.5° vertically and 70horizontally. This corresponds to average velocities of330 and 4200/s, respectively, over any 16-ms period.

In the videotape analyzed, no limit was found to thistracking rate. The largest distance covered betweenany two examined fields was 7 3/40 during a 140 horizon-tal saccade. This corresponds to a velocity of 4650/s.Accuracy in the placement of the fiducial mark on theimage was unaffected by the higher velocities.

E. Signal Delay

The computer algorithm processed pixel data moreslowly than the frame-grabber digitized pixels; for thisreason it was possible to begin both processes simulta-neously without risk of the computer outrunning thedigitization and as a result reading data from the previ-ous image. Synchronization was obtained by buildinga current line register which consisted of an interfacebetween the frame-grabber address lines and a com-puter I/O port. Since data collection and computerprocessing took place concurrently, the time delay forthe monitoring component of the tracker was simplythe time taken for computer processing-8 ms.

V. Measuring Deviation

Video frames, which consisted of two interlaced im-ages, were reviewed one at a time, and in each onemeasurements were made to quantify the variation inthe computer's placement of fiducial crosses on theimages. An average position was calculated first andthen rms deviation from the average. The first meth-od used was to register manually the fundus imageagainst a transparency consisting of an image templateand rulings. The calculated horizontal deviation was8 min of arc rms during periods of relative stillness and13 min of arc during velocities of more than 1200/s.Variation in the vertical dimension contained twocomponents, jitter and drift. The jitter componentwas almost the same as the horizontal rms deviation,measuring 7 min of arc during relative stillness and 13min of arc during velocities exceeding 1200/s. Thedrift component ranged from zero in the center of thefield to 1 pixel of drift per 8 pixels of motion at theperiphery. Since this slippage was along the vesseland was relatively small for a given locality, it wasdeemed compatible with photocoagulation, and no at-tempt was made to correct for it in the algorithm.

It was felt that the ambiguities inherent in the abovemeasuring technique added to the calculated error.When velocity was large, the individual images weregeometrically distorted; when velocity was low, the twoimages in the frame were difficult to distinguish. Also,since measurement was geared to fundus image regis-tration, it did not evaluate precision according to themain purpose of the program: to find the blood vessel.

A better method of measurement in the horizontaldirection was to gage the distance between the rightedge of the vessel and the endpoint of the fiducialcross. Since the distance was of the order of 1 pixel,and since both fiducial and vessel edge were relativelydistinct and unchanging, this type of measurement

1502 APPLIED OPTICS / Vol. 26, No. 8 / 15 April 1987

C. _iEEftii r

was judged to be more accurate. Estimates of errorwere made using the fiducial itself to derive the unit ofmeasure. By this method the error in the fiducial'splacement approached 0.3 pixel rms, which is the theo-retical limit imposed by the one-pixel resolution of thevessel-locating algorithm. The actual calculated errorwas 5 min of arc rms and is probably the best measureof the tracker's monitoring precision in the directionorthogonal to a well-defined vessel. This figure didnot deteriorate in the presence of velocities up to 270°/S.

VI. Discussion

The algorithm's cup-centroid technique of monitor-ing eye position has advantages and disadvantages.Within a locality it can deliver a position reading whichis two-dimensional and just as precise as the vessel-locating technique. It is robust due to the uniquebrightness of the cup of the disk, which is easily identi-fied in the image. However, over the full range of thetracking field it is susceptible to drift in relation toother features-especially if the input light intensityor the detection are not uniform for all areas of theraster. Such nonuniformity was present in the earlyversion of the SLO used for this study: there wasvignetting in the detection optics, and the acoustoopticdeflector used to create the horizontal scan deliveredmore light to the center of the lines than to the sides.Possibly as a result, the shape and relative position ofthe thresholded area of the disk changed slightly as theeye moved, giving the vertical drift already noted.The drift was always continuous and opposite the di-rection of eye motion. However, the magnitude of thisslippage, which peaked at 1 pixel of slip per 8 pixels ofmotion, suggests that other factors were involved. Itmay be possible to remove most of the centroid drift byuse of a calibration lookup table or a mapping function,but this possibility was not explored in depth.

While the centroid portion of the algorithm wasfound to result in drift, the vessel-finding portionyielded a stable fiducial. Evidently, the templatetechnique is not much affected by unevenness of illu-mination or detection or by parallax between the partsof the vessel at opposite ends of the 1.5° template.Despite the use of various vessels and locations, how-ever, this problem was never encountered. The ves-sel-template technique appears to perform equivalent-ly in all parts of the tracking field.

The tracking rate, measured from field to field on astop-frame VCR, is 60-80 disk-diam/s during 2-3 disk-diam saccades.

The bandwidth of this tracking technique could beincreased by tracking more than one area of the funduswith the areas well-separated vertically or by increas-ing the frame rate of the SLO itself. The first of thesewould restrict the tracking field, while increasing theframe rate would require some redesigning of the SLO.

There are trade-offs in the tracking program config-uration. Increased sampling density gives increasedprecision and decreased noise, but costs increased sig-

nal delay. Increased window size leads to increasedslew rate again with increased signal delay.

There is a great deal of information available in thefundus image which can be used for purposes otherthan direct tracking calculations. Since the computerwas idle for 8 ms after every 8-ms iteration of thealgorithm, there was plenty of computing time left todo other types of image analysis, for example, measure-ment of light levels or feature size. In this program,the extra time was used to detect blinks and otherdisturbances in the signal.

VII. Tracker Applications

The tracker described here is relatively easy to usefor applications in which the SLOs own coordinatesystem is acceptable. Such applications might in-clude raster presentation of psychophysical stimuli inwhich it is important to have a relatively stable place-ment of graphics on the retina. For such presentationit may be necessary to use two lasers of different wave-length to avoid having the raster graphics interferewith the image information necessary for tracking.The two lasers are aligned before input to the scanningoptics, but light returning from the eye is detectedseparately using filters. It may be possible to use IRas the imaging wavelength, leaving the graphics as theonly visible component of the raster. If the graphicsare to be presented at areas of the fundus other thanthe tracked area, all this can be done (as we did ) usingonly one laser. It should be pointed out that there canbe serious delays involved whenever stabilized graph-ics are attempted on a video-based system. For exam-ple, if the current image is an even-field image, theeven field part of the graphics will not be updated untiltwo fields later. This is a 33-ms delay.

A closed-loop tracker was built to obtain a relativelymotionless fundus image on the display. The comput-er was used to generate feedback signals to guide theSLO raster by directing dc offsets to the instrument'shorizontal and vertical mirror drives. The total delayin the tracking was -8 ms, 4.5 for the computer algo-rithm and 3.5 settling time for physically turning themirrors. The loop was closed in the sense that feed-back signals steered the raster in such a way as to keepthe tracked feature (cup of the disk) always at the sameraster-coordinate position. The display was in theend relatively stable but had the characteristic thatregistration of images on the display was always onefield behind the motion. Thus for every motion therewas a slight stutter produced by the not-yet-registeredimage.

For some possible applications of this tracking tech-nique, such as laser photocoagulation, the raster coor-dinates may not be the natural units of measure. Inge-nuity will be required to couple the two systems. Onepossible approach, of course, is calibration and map-ping of the photocoagulator guidance system onto ras-ter space at the fundus. Another approach, which ispossible on the new confocal version of the SLO,3 is tohave the photocoagulator position detected by theSLO itself. At the instant in which the photocoagula-

15 April 1987 / Vol. 26, No. 8 / APPLIED OPTICS 1503

tor spot on the fundus is confocal with the SLO outputscanning optics, the spot is detected and its locationrecorded in frame-grabber memory. This position iscompared to the desired position as calculated by thetracker, and appropriate controls are implemented.

In any tracking system (of anything) allowance mustbe made for leading a moving object. At 50'/s, our 8-ms delay causes a 24-min of arc lag in the monitoredeye position. We have built in no lead to compensatethis.

Vil. Summary and Conclusion

The most important facts about the eye-trackingtechnique described here are that (1) the tracking re-lies mostly on real-time software analysis of video im-ages from the scanning laser ophthalmoscope and (2)the tracking information is derived from the back ofthe eye, the fundus.

Because it is video-based, the bandwidth of thetracking technique is 30 Hz. The delay in the eye-movement monitoring signal was 8 ms, and the slewrate was 2700/s. For locating a well-defined vessel, theprecision was 5 min of arc rms, not including the lagproduced by signal delay.

The tracker specifies the location of tracked fundusfeatures in terms of particular beam positions in theraster of the scanning laser ophthalmoscope. For thisreason it may be useful in applications such as eye-tracked photocoagulation, imaging, or presentation ofvisual stimuli via raster graphics stabilized againstdrift and small motions.

The eye-tracking described here is only a beginning,and much remains to be tested and explored. Atpresent the technique has been tried on only a handfulof subjects, who were all experienced users of the SLO.It remains to be seen how difficult it will be to workwith naive subjects and those with visual problems.On the positive side, we are seeing that images from thenew confocal SLO give dramatically better contrast,detail, and uniformity of illumination and detection.Not only is there more available information in theimages, but the feature information remains more con-stant in the presence of eye motion. Algorithm devel-opment to take advantage of these instrument im-provements has hardly begun. More positivecommercial development of powerful low-cost com-puter-vision tools is proceeding quite rapidly. Thetechnique may eventually find a useful place where it isdesirable to have a convenient noncontact method formedium performance tracking of the fundus.

Requests for reprints should be addressed to R. H.Webb.

References1. L. R. Young and D. Sheena, "Survey of Eye Movement Recording

Methods," Behav. Res. Methods Instrum. 7, 394 (1975).2. R. H. Webb and G. W. Hughes, "Scanning Laser Ophthalmo-

scope," IEEE Trans. Biomed. Eng. BE-28, 488 (1981).3. R. H. Webb and G. W. Hughes, "Confocal Scanning Laser Ophth-

almoscope," Appl. Opt. 26, 1492 (1987).

I Elementary School ScienceInstruction Programs

NSF announces the second of twoprogram solicitations intended toencourage partnerships among publishers,school systems, and scientists/scienceeducators for the purpose of providing anumber of competitive, high quality,alternative science programs for use intypical elementary schools. Proposals aresought for projects that will improve thecontent, increase the role of the child asan active agent in the learning process,and lead to an increase in the time allottedto science instruction in elementaryschools.

Preliminary proposals, which arerequired, must be received byJune 1, 1987. The deadline for formalproposals is Aug. 3, 1987. RequestBrochure NSF 87-12. For furtherinformation, contact Mr. Jerry Theise, Ms.Mary Kohlerman, or Ms. Alice Moses,Instructional Materials DevelopmentProgram (357-7066).

1504 APPLIED OPTICS / Vol. 26, No. 8 / 15 April 1987