tick talk - cogsci journal archivecsjarchive.cogsci.rpi.edu/proceedings/2003/pdfs/73.pdftick talk...

Tick Talk

Douglas J. Davidson ([email protected])Kathryn Bock ([email protected])David E. Irwin ([email protected])

Beckman InstituteUniversity of Illinois

405 N. MathewsUrbana, IL 61801 USA

Abstract

Three experiments on the production of time ex-pressions are reported. In each experiment, speak-ers produced time expressions to analog clock faceswhile their eye movements were recorded with aheadband-mounted eye tracker. Previous work hasshown that speakers can apprehend the visual-conceptual information in a clock display veryquickly, and that this is followed by relatively slow,incremental formulation of the time expressions.Analyses of fixation patterns from the present ex-periments lend additional support to this model.

Introduction

Clock time-telling offers an unusually efficient andeffective means of studying how speakers formulatelinguistic expressions to relatively simple visual dis-plays. American English speakers can produce ei-ther absolute (e.g., nine thirty) or relative (e.g., halfpast nine) time forms for the same clock, and thisprovides a contrast between two opposite word or-ders for the same visual display. Clock faces alsocontain a reasonably large response set. Consider-ing just the times at five minute intervals (e.g., 9:30,9:35, 9:40 . . . ) an analog clock face contains a smallcombinatorial explosion of at least 276 unique timeexpressions, each corresponding to exactly the samehour and minute hand arranged in different orienta-tions. In addition, there is an unambiguous mappingfrom the hands of a clock display to the componentsof time expressions. This mapping allows techniquessuch as eye tracking to be applied to study the rela-tionship between gaze patterns and speech.

Eye tracking has emerged as a powerful tool to in-vestigate language production because, among otherthings, it can be used to observe whether speakersseek visual information corresponding to the indi-vidual terms of a multi-word expression before theybegin to articulate those terms. Under some circum-stances, the properties of the recorded eye move-ments (e.g., gaze position, duration, sequence) canbe related to the properties of the recorded speech(timing, expression choice, fluency) to provide evi-dence for or against hypotheses about the mecha-nisms of speech planning.

Griffin and Bock (2001) proposed a general de-composition of the process of formulating descrip-tions based on data from experiments in which par-ticipants produced spontaneous descriptions of sim-ple visual events. The first phase in this account,apprehension, involves the rapid extraction of vi-sual features and conceptual relationships relevantto the upcoming utterance. The second phase, lin-guistic formulation, involves retrieving words andconstructing the expression form to convey the in-tended description. Importantly, linguistic formula-tion occurs more slowly than apprehension, and it isaccompanied by gaze patterns that match the orderof the spoken words.

Time telling provides a strong test of this hypoth-esized decomposition because unlike the descrip-tion of a novel scene using a novel sentence, bothclock reading and time telling are well-rehearsedprocedures that many speakers perform habituallythroughout the day. In addition, time expressionsform a finite set. In principle, speakers could mapeach conceptual representation of a time to a geo-metrical representation of a corresponding clock inmemory. This, in turn, could permit a fast mappingfrom the perception of a clock display to the formu-lation of the time expression, potentially bypassingslow formulation requirements.

Bock, Irwin, Davidson & Levelt (2003) showedthat this bypass is not taken. In their experiments,speakers started to look to regions of the clock facesthat corresponded to the first term of the ultimateexpression that they produced approximately 400ms after the onset of the clock displays, suggestingvery fast apprehension for the clock displays. Thesignature pattern of successive gazes to regions cor-responding to successive terms of utterances was asrobust for clock displays as in earlier studies of scenedescription. The experiments reported here extendthe results of Bock et al. by investigating severalalternative explanations for the data patterns thatwere originally observed.

The speakers in Bock et al. were asked to producetime expressions as either absolute or relative timeforms. By knowing the format required for the timeexpression in advance, in principle speakers couldhave adopted a task set that biased them to indi-

294

viduate the clock hands in the order of the requiredexpression. For example, if speakers are aware thatthey should produce a time consisting of first thehour and then the minute value, an efficient strat-egy would be to first look for the hour hand. Thetask set, in this case, would be the primary determi-nant of the observed eye movement patterns, ratherthan an intrinsic tendency to link gaze patterns withspeech planning processes. Left to their own devices,without explicit instructions to produce one or theother form, speakers could provide eye movementpatterns that are less closely linked to planning pro-cesses. In Experiment 1 reported below, speakerswere simply asked to produce time expressions with-out specific instructions about the required form. Inaddition, half of the participants heard instructionsthat presented a relative time expression as an exam-ple utterance, while the other half heard an absoluteexample.

The hands of analog clock displays are differen-tially informative, insofar as the hour hand conveysinformation about both the hour and the minute.This is the case because the hour hand moves para-metrically from one hour to the next for each in-crement of the minute (see the top right clock inFigure 1). For example, one can deduce that thetime is 9:30 just by observing that the hour handis halfway between 9 and 10 –the hour hand couldprovide all the information necessary for the clocktime. The hands are differentially informative be-cause while the hour hand conveys this informationabout the value of the minute, the minute hand doesnot convey any information about the hour. Thelocation of the minute hand at the 6 position, forexample, does not provide the exact angular posi-tion of the hour hand. It only conveys that the hourhand is midway between some (unspecified) pair ofhours.

Figure 1: Numberless fixed, numberless parametric,numbered, and tick-marked clocks.

The analog clock displays used in Bock et al. did

not have parametrically-arranged hour hands. Intheir stimuli, the hour hand remained fixed at one lo-cation for varying locations of the minute hand (seetop left in Figure 1). It is possible that Bock et al.observed gaze paths that mimicked word order pat-terns because unlike a regular clock, the hour handregion of the clock face did not contain all of the in-formation usually found in a clock display. Speakersmight have needed to inspect both hands in those ex-periments to understand the time, whereas on a reg-ular clock, the differential relationship of the handscould in principle permit speakers’ gaze to favor thehour more. Experiment 1 investigates this possibil-ity.

Among the requirements of time telling, a speakermust understand both where the hands are located,and to what the hands refer. On a clock face withoutany numbers, speakers must determine the angularrelationship between the hand locations and somereference point, and they must determine which hourand minute regions are relevant.

Numbered clock faces may make this process sim-pler, because the digits present on an ordinary num-bered clock face provide both identity and locationinformation. The value for the hour can be acquiredby simply reading the digit to which the hour handpoints, and the value of the minute can be acquiredby calculating the appropriate minute value afterthe digit has been read, or by retrieving this valuefrom memory once the digit has been identified. Dig-its also encode location information by providing apoint of reference for nearby hands, relative to theorientation of the entire clock face. It could be thateither identity and location information is enough tocircumvent the need for incremental gaze patternsduring time phrase production. Experiments 2 and3 compare numberless clock faces like those in Ex-periment 1 with numbered and tick-marked clockfaces (see Figure 1) to examine the contribution ofthese factors to time telling.

Experiment 1

MethodParticipants Sixteen participants from the Psy-chology Department subject pool at the Universityof Illinois took part for course credit or for monetarypayment. All participants had normal or correctedto normal vision, and spoke American English astheir native language. None had taken part in theother experiments reported here.

Design and Materials The experiment was atwo-factor design with type of clock (fixed hourhand, parametric hour hand) and instruction (abso-lute biased, relative biased) as between-participantfactors. Each participant saw 144 clock faces, con-sisting of all combinations of hour and 5-minutetimes. The fixed hour hand clock faces were con-structed so that the hour hand always pointed to

295

the location of the hour, while the parametric clockfaces were constructed so that the hour hand ad-vanced for each 5-minute increment of the minutehand.

Apparatus Eye movements were recorded usingan headband-mounted Eyelink (SR Research, Ltd.)eye tracker. Clock displays were presented on a 51cm monitor, each clock subtending approximately 23deg of visual angle. Participants pressed a button ona hand-held button box to begin and end each trial.Speech was recorded using a sound card and a pre-amplified microphone.

Procedure Participants were simply asked to pro-duce time expressions to the clock faces. One ex-ample of an absolute expression was provided inthe absolute-biased instructions, and one example ofa relative expression was provided for the relative-biased instructions. Participants were not given ex-plicit instructions about the form of the time expres-sions.

Each trial began with the presentation of a centralfixation point. Participants pressed a button to starta trial, and drift correction was applied while theyheld fixation to the center point. If drift correctionwas successful and participants held fixation, a clockface was presented for a maximum of five seconds,or until participants pressed a button indicating thatthey had finished the trial. The clock face remainedon screen for a minimum of three seconds. The nexttrial began after a one second delay.

Data Analysis Participants’ fluent utteranceswere classified as absolute, relative, or full -hour timeexpressions. Full hour time expressions were ex-cluded from further analysis, because there was nochoice of construction for those cases. Also, timescorresponding to displays in which the number andminute hand overlapped (e.g., 2:10 ) were excluded,as these did not contain different fixation target re-gions for hours and minutes.

For the fixation duration data, an extension of thestandard Cox proportional hazards model was usedto compare cumulative hazard rate functions of fix-ation durations in different experimental conditions.This extension allows for the analysis of repeatedmeasurements and random effects in survival anal-ysis (Therneau & Grambsch, 2000; also known as“frailty” models).

Results and Discussion

Speech Table 1 shows the average proportion ofrelative time expressions observed in all conditions.Speakers preferred to produce absolute expressionsfor both types of clocks and for both types of in-struction by a wide margin. Given this preference,speakers were more likely to produce a relative ex-pression when the instructions contained a relativeexample (p < .001), but there was no difference be-

tween the two clock types.

Table 1: Average Proportion of Relative Expres-sions.

Clock TypeInstruction Bias Orig. Param. Ave.Relative 0.113 0.103 0.108Absolute 0.000 0.032 0.016Ave. 0.056 0.068

Further inspection of the data revealed that mostof the relatives were produced by just two speak-ers. When they produced relative expressions, thesespeakers were more likely to do so for times nearthe top of the hour. An analysis of the proportionof expressions that were relative showed that therewas a linear main effect of minute value (p < .001),as well as a quadratic trend (p < .001), indicatingthat these participants produced a few relatives forminute values 0-15, almost no relatives for times nearthe half hour, and progressively more relatives forminute values greater than 40. See Bock, Irwin, andDavidson (in press) for an account of this pattern.

There were no statistically significant differencesfor speech onset times for the speakers who saw thefixed clocks (mean latency: 1658 ms) versus thespeakers who saw the parametric clocks (1769 ms).

Eye Movements Figure 2 shows a global profileof the gaze patterns for absolute expressions. It plotsthe average proportion of fixations to the hour andminute hands relative to the onset of the clock dis-play (top plot) and to the onset of speech (bottom)for absolute expressions. The proportions were cal-culated by summing the number of fixations in 100ms intervals over subjects.

The top plot shows that the two types of clockfaces led to similar distributions of fixations to thehour and minute hands. Speakers first fixated thehour hand region and then the minute, regardlessof whether the clock faces were fixed or parametric.The bottom plot of Figure 2 shows that, relativeto speech, the peak in the proportion of fixationsto the hour hand occurred earlier than the peak ofthe minute hand, and that the peak of the minutehand occurred near the onset of speech. Both thehour and minute distributions appear to be shiftedsomewhat earlier for the parametric clocks, whichcould be due to differences between the two subjectgroups.

There was a median of four fixations before theonset of speech across the experimental conditions.Table 2 shows the pattern of saccade transitions forthe first four fixations, providing a more fine-grainedanalysis of the data shown in Figure 2. Each cellcontains the proportion of transitions to the hour,minute, and other regions for each transition pair

296

0 1000 2000 3000 4000

0.0

0.4

0.8

Fixations to Parametric and Fixed Clocks

Time (ms) From Display Onset

Ave

. Pro

port

ion

of F

ixat

ions

HourMinute

−3000 −2000 −1000 0 1000 2000 3000

0.0

0.4

0.8

Time (ms) From Voice Onset

Ave

. Pro

port

ion

of F

ixat

ions

Figure 2: Average proportions of fixations to fixed(solid) and parametric (dotted) clock faces relativeto display onset (top) and speech onset (bottom).

(transitions to the center region are not shown). Ta-ble 2 shows that the early fixations 1-4 were morelikely to target the hour hand region. For example,at the transition from the first to the second fixationafter leaving the center, 14.6% of transitions werefrom the minute to the hour, but there were fewer ofthe reverse, hour-to-minute transitions, 1.4%. Thisinitial fixation to the hour hand was then most oftenfollowed by a series of refixations to the hour hand.Later, at the transition from fixation 4 to 5, therewere more transitions to the minute region.

0 500 1000 1500 2000

01

23

45

67

Fixed Hour Hand

Fixation Duration (ms)

Cum

ulat

ive

Haz

ard

Fix 1

Fix 2

Fix 3

Fix 4

0 500 1000 1500 2000

01

23

45

67

Parametric Hour Hand

Fixation Duration (ms)

Cum

ulat

ive

Haz

ard

Fix 1Fix 2

Fix 3Fix 4

Figure 3: Cumulative hazard functions for the firstfour fixations of the fixed and parametric clocks.

Figure 3 plots cumulative hazard functions for fix-ations 1-4 after participants’ gaze left the center re-gion. These fixations correspond to fixations 1-4 inTable 2. The slope of the cumulative hazard functionat a particular time point can be interpreted as the

Table 2: Transition matrices for fixations 1-4 (sourceregion along the left side, destination region alongthe top).

Transition: Center:1Hour Minute Other

Center 0.436 0.215 0.311

Transition: 1:2Hour Minute Other

Hour 0.409 0.014 0.014Minute 0.146 0.013 0.054Other 0.139 0.021 0.150







rate at which the fixations end in a saccade at thattime point. Figure 3 shows that the first fixationafter clock onset did not last as long as the fixationsthat came later, as the estimated cumulative haz-ard function for the first fixation has a steeper slope(exp(coef)=0.36, χ2=1077, p < 0.005) than the laterfixations, which appear to have similar, more shallowslopes. In addition, both the fixed and parametrichour hand clocks show this pattern.

The data from Experiment 1 suggest that speakersproduced time expressions to fixed analog clock facesmuch like they do for parametrically-arranged faces.A similar global profile of fixations was observedfor both types of clocks, and speakers did not ap-pear to have more (or less) trouble with fixed clocks.The initial fixation after clock onset appeared to beshorter than the other fixations for both the fixedand parametric clocks.

297

Experiment 2

MethodExperiment 2 compared the performance of speak-ers to numbered clock faces versus numberless clockfaces. In most respects except the design and stim-uli, the experiment was like Experiment 1 (differ-ences are outlined below).

Participants Eight participants from the Univer-sity of Illinois community took part for course creditor pay. All participants were native speakers ofAmerican English, and none of the participants hadtaken part in the other experiments reported here.

Design and Materials Experiment 2 was a one-factor within-participant design with clock type(numbered, numberless) as the main experimentalfactor. The numberless clocks were the same stimulias the parametric condition of Experiment 1 (seeFigure 1, top right). The numbered clocks wereagain the same clocks, but with digits placed at theperiphery (see Figure 1, bottom left). Each partici-pant saw 144 trials total, consisting of 72 numberedand 72 numberless clocks. The stimulus lists for pre-senting the clocks were counterbalanced so that fourof the participants saw one version of a clock as anumbered clock, while the other four saw the sameclock as numberless (and vice versa). Instructionsto participants were like those in Experiment 1, ex-cept that both relative and absolute time expressionswere provided as examples (participants were toldthat either form would be an acceptable response).

Results and DiscussionLike Experiment 1, two speakers produced the ma-jority of relative expressions in Experiment 2. Over-all, a similar proportion of relatives (0.08 percent ofthe expressions) were produced as in Experiment 1,and they were more common near the top of thehour.

For the absolute expressions, speakers were fasterto start time expressions for the numbered clocks(1638 ms) compared to the numberless clocks (2326ms), t(7) = 8.47, p < 0.001, as expected. There weretoo few relative expressions across subjects to makea meaningful comparison.

Figure 4 shows the aggregate profile of fixations tothe hour and minute hands over time for numberlessand numbered clocks for absolute expressions. Thedata pattern suggests that speakers spent less gazetime at the time-relevant regions for the numberedclocks. The top plot of Figure 4 shows that afterdisplay onset, fixations to the hour hand started todecrease earlier, and fixations to the minute handincrease earlier for the numbered clocked comparedto the numberless clocks.

The plot of fixations relative to speech (bottomplot) shows that the span between the peak of thehour hand region fixations and voice onset was some-

0 1000 2000 3000 4000

0.0

0.4

0.8

Fixations to Numberless and Numbered Clocks


Ave

. Pro

port

ion

of F

ixat

ions

−3000 −2000 −1000 0 1000 2000 3000

0.0

0.4

0.8


Ave

. Pro

port

ion

of F

ixat

ions

HourMinute

Figure 4: Average proportions of fixations to num-berless (solid) and numbered (dotted) clock faces rel-ative to display onset (top) and speech onset (bot-tom).

what shorter for numbered clocks compared to num-berless clocks.

The data from Experiment 2 suggest that the dis-play of number information accelerates the produc-tion of the hour term, but does not change the in-cremental pattern of formulation. The global pro-file of fixations showed that speakers were faster toleave the hour region to start fixations to the minuteregion for numbered clocks. Nonetheless, the samebasic pattern of incremental looking first to the hourhand and then to the minute hand in concert withspeech was observed for both types of clocks.

Experiment 3Experiment 3 examined performance on clock faceswithout tick-marks to clock faces with tick-marks.In most respects except the design and stimuli, theexperiment was like Experiment 2. The differencesare outlined below.

MethodParticipants Eight participants from the Univer-sity of Illinois community took part in the experi-ment for course credit or pay. All participants werenative speakers of American English, and none hadparticipated in the other experiments reported here.

Design and Materials Experiment 3 was a one-factor within-participant design with clock type(tick-free, tick-marked) as the main experimentalfactor. The tick-free clocks were the same stimulias the parametric condition of Experiment 1 (seeFigure 1, top right), while the tick-marked clockswere the same clocks as the tick-free, but with smalldots placed at the periphery (see Figure 1, bottomright). Both types of clocks had parametric hour

298

hands. Each participant saw 144 trials total, con-sisting of 72 tick-free and 72 tick-marked clocks. Thestimulus lists for presenting the clocks were counter-balanced as in Experiment 2, and the instructionswere the same.

Results and Discussion

In Experiment 3, one participant produced the ma-jority of relative expressions (57), while two othersproduced 15 or less (out of the 132 opportunities foreach subject). The majority of the relative expres-sions had minute values at 45, 50, and 55, as in theother experiments.

For the absolute expressions, speakers were fasterto produce times to the tick-marked clocks (1764 ms)compared to the tick-free clocks (2004 ms), t(7) =4.17, p < 0.005. There were too few relatives acrosssubjects for a meaningful comparison.

Figure 5 shows the aggregate proportion of fixa-tions to hour and minute hands for clocks with andwithout tick-marks. Like Experiment 2 with num-bered clocks, it appears that speakers were some-what faster to leave the hour hand and start fixat-ing the minute hand with the tick-marked clocks,although the effect is smaller than with the num-bered clocks. There appears to be little effect of thetick-marks for the span between fixations to the hourregion and the onset of speech.

0 1000 2000 3000 4000

0.0

0.4

0.8

Fixations to Tick−Free and Tick−Marked Clocks


Ave

. Pro

port

ion

of F

ixat

ions

−3000 −2000 −1000 0 1000 2000 3000

0.0

0.4

0.8


Ave

. Pro

port

ion

of F

ixat

ions

HourMinute

Figure 5: Average proportions of fixations to tick-free (solid) and tick-marked (dotted) clock faces rel-ative to display onset (top) and speech onset (bot-tom).

Experiment 3 shows that the tick marks do notgreatly affect the patterns of fixations or perfor-mance of time telling to the clock faces. Thesame incremental pattern of fixations was observed,although shorter latencies were observed for tick-marked clocks.

General DiscussionIt appears that speakers’ gaze patterns are largelyinsensitive to small changes in the details of analogclock faces, while at the same time, speech perfor-mance parameters may be more strongly affected.Across all three manipulations, most speakers choseto produce absolute time expressions, and they pro-vided very similar fixation patterns. When numberswere present on the clock faces, speakers were fasterto start utterances and appeared to be faster to movefrom one hand to the next, but otherwise providedsimilar fixation profiles. If speakers had faster accessto phonological information by reading the numbersin Experiment 2, this may have enabled them tospend less time gazing at the relevant time regions.These patterns would be consistent with results fromMeyer and van der Meulen (2000) suggesting that asubstantial factor influencing fixation times duringobject naming is phonological form retrieval. The ef-fect of tick-marks on the fixation profiles was small,but did affect utterance latencies as well.

In general, the data from these experiments pro-vide additional support for hypothesized propertiesof formulation in language production, and for theusefulness of eyetracking and measures of eye-voicecoordination during time telling in assessing thoseproperties. Across all three experiments, when pro-ducing absolute expressions, speakers appeared tolocate the hour hand region quickly (within the firstor second fixation). This was followed by a seriesof fixations that reflected the order of the wordsthat participants produced. It appears that evenwith highly formulaic expressions and a fixed andfinite vocabulary, the eyes still foreshadow the pathof speech.

AcknowledgmentsThis research was supported by research and train-ing grants from the National Science Foundation(SBR 94-11627 and SBR 98-73450) and the NationalInstitute of Mental Health (RO1-MH66089 and T32-MH18990).

ReferencesBock, J. K., Irwin, D., Davidson, D. J. & Levelt,

W. J. M. (2003). Minding the clock. Journal ofMemory and Language, 48, 653-685.

Bock, J. K., Irwin, D. E., & Davidson, D. J. (inpress). Putting first things first. J. M. Hender-son & F. Ferreira (Eds.), The interface of lan-guage, vision, and action: What we can learn fromfree-viewing eyetracking. New York: PsychologyPress.

Griffin, Z. & Bock, J. K. (2000). What the eyes sayabout speaking. Psychological Science, 11, 274-279.

Meyer, A. & van der Meulen, F. F. (2000). Phono-logical priming effects on speech onset latenciesand viewing times in object naming. PsychonomicBulletin & Review, 7, 314-319.

Therneau, T. M. & Grambsch, P. M. (2000). Model-ing survival data: Extending the Cox model. NewYork: Springer-Verlag.299

tick talk - cogsci journal archivecsjarchive.cogsci.rpi.edu/proceedings/2003/pdfs/73.pdftick talk...

Documents