B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

ava i l ab l e a t www.sc i enced i rec t . com

www.e l sev i e r. com/ l oca te /b ra in res

Research Report

When a high-intensity “distractor” is better thena low-intensity one: Modeling the effect of an auditoryor tactile nontarget stimulus on visual saccadic reaction time

Adele Diedericha, Hans Coloniusb,⁎aSchool of Humanities and Social Sciences, Jacobs University Bremen P.O. Box 750 561, D–28725 Bremen, GermanybDepartment of Psychology, University of Oldenburg, P.O. Box 2503, D–26111 Oldenburg, Germany

A R T I C L E I N F O

⁎ Corresponding author. Fax: +49 441 798 5158E-mail addresses: a.diederich@jacobs-uni

0006-8993/$ – see front matter © 2008 Elsevidoi:10.1016/j.brainres.2008.05.081

A B S T R A C T

Article history:Accepted 29 May 2008Available online 10 June 2008

In a focused attention task saccadic reaction time (SRT) to a visual target stimulus (LED) wasmeasuredwith an auditory (white noise burst) or tactile (vibration applied to palm) nontargetpresented in ipsi- or contralateral position to the target. Crossmodal facilitation of SRT wasobserved under all configurations and stimulus onset asynchrony (SOA) values ranging from−250 ms (nontarget prior to target) to 50 ms. This study specifically addressed the effect ofvarying nontarget intensity. While facilitation effects for auditory nontargets are somewhatmore pronounced than for tactile ones, decreasing intensity slightly reduced facilitation forboth types of nontargets. The time course of crossmodalmean SRT over SOA and the patternof facilitation observed here suggest the existence of two distinct underlying mechanisms:(a) a spatially unspecific crossmodal warning triggered by the nontarget being detected earlyenough before the arrival of the target plus (b) a spatially specific multisensory integrationmechanism triggered by the target processing time terminating within the time window ofintegration. It is shown that the timewindow of integration (TWIN)model introduced by theauthors gives a reasonable quantitative account of the data relating observed SRT to theunobservable probability of integration and crossmodal warning for each SOA value under ahigh and low intensity level of the nontarget.

© 2008 Elsevier B.V. All rights reserved.

Keywords:Multisensory integrationSaccadic reaction timeTime window of integrationEffect of intensity

1. Introduction

Based on neurophysiological and behavioral findings inhumans, monkey, and cat, several authors have suggested theexistence of a critical spatiotemporal “window” for multi-sensory integration to occur (e.g., Bell et al., 2005; Meredith,2002; Corneil et al., 2002). Given that the strength of a multi-sensory effect is often lawfully related to the spatial andtemporal configuration of the stimuli from different modalities(Stein and Meredith, 1993), the notion of a “window” has beenproposed to be a critical determinant for multisensory integra-

.versity.de (A. Diederich), h

er B.V. All rights reserved

tion at both neural and the behavioral levels of observation.With respect to the temporal dimension, the idea simply is thatthevisual and the auditory stimulus, say,must not bepresentedtoo far away in time for bimodal integration to occur. Thisintegrationmaymanifest itself in the formof an increased firingrate of a multisensory neuron (relative to unimodal stimula-tion), anaccelerationof saccadic reaction time (Frens et al., 1995;Diederich et al., 2003), an effective audiovisual speech integra-tion (Van Wassenhove et al., 2007), or in an improved ordegraded judgment of temporal order of bimodal stimulus pairs(cf. Spence and Squire, 2003). Depending on context, estimates

ans.colonius@uni-oldenburg.de (H. Colonius).

.

220 B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

of thewidth of the timewindowdifferwidely, however, rangingfrom 40 to 600 ms. This is not really surprising given the hugedifferences in theabove-mentionedexperimental paradigms. Infact, it seems quite questionable that the broad usage of thenotion of a time window for multisensory integration reflect aconcept that goes beyond a metaphoric level of explanatorypower. In this paper, the role of a timewindow is investigated inthe limited context of an explicit model for saccadic reactiontime to visual targets in the presence of either an auditory ortactile nontarget (“distractor”).

In this focused attention paradigm, crossmodal stimuli arepresented and participants are instructed to respond only to theonset of a stimulus from a specifically defined target modality,such as the visual, and to ignore the remaining nontargetstimulus, tactile or the auditory.When a stimulus of a nontargetmodality, a tone, say, appears before the visual target at somespatial disparity, there is no overt orienting response toward thetone if the participant is following the task instructions. Never-theless, the nontarget stimulus, though being uninformativewith respect to the spatial location of the target, has been shownto modulate the saccadic response to the target: Depending onthe exact spatiotemporal configuration of target and nontarget,the effect can be a speed-up or an inhibition of SRT (see, e.g.,Amlot et al., 2003; Diederich and Colonius, 2007a), and thesaccadic trajectory can be affected as well (Doyle and Walker,2002). Specifically, as longas thenontargetoccurs less thanabout200 ms before the target stimulus (stimulus onset asynchrony,SOA, greater or equal to 200ms), a speed-up of bimodal vs. visualSRT is observed when target and nontarget are presented inspatial proximity, and the speed-up diminishes, or turns intoinhibition, with increasing spatial disparity (Van Opstal andMunoz, 2004; Diederich and Colonius, 2007b; Hughes et al., 1998).The spatial effect disappears when the nontarget is presentedeven earlier, but a spatially unspecific facilitation is sometimespreserved evenwith an SOAof −500ms (Diederich andColonius,2007a, 2008a). This latter effect may be considered a warningeffect, or exogenous shift of attention, rather than a true multi-sensory integration phenomenon (e.g., McDonald et al., 2000).

1.1. Time-window-of-integration (TWIN) model

The distinction between warning and multisensory integra-tion has been taken into account by a stochastic modeldeveloped by the authors (Colonius and Diederich, 2004;Diederich and Colonius, 2004a, 2007a,b, 2008a,b) yielding anestimate of the relative contribution of either mechanism forany specific SOA value. The TWIN model distinguishes twoserial stages of saccadic reaction time: an early, afferent stageof peripheral processing (first stage) followed by a compoundstage of converging subprocesses (second stage). In the firststage, a race among the peripheral neural excitations in thevisual and auditory (or tactile) pathways triggered by acrossmodal stimulus complex takes place. The second stagecomprises neural integration of the input and preparation ofan oculomotor response. Thus, the model retains the classicnotion of a race mechanism as an explanation for crossmodalinteraction (cf. Colonius and Diederich, 2006), but restricts it tothe very first stage of stimulus processing.

The assumption of only two stages is certainly an over-simplification. Note, however, that the second stage is defined

by default: it includes all subsequent, possibly temporallyoverlapping, processes that are not part of the peripheralprocesses in the first stage (for a similar approach, see VanOpstal and Munoz 2004). Moreover, Whitchurch and Takaha-shi (2006) collected (head) saccadic reaction times in the barnowl finding further support for the notion of a race betweenearly visual and auditory processes depending on the relativeintensity levels of the stimuli. In particular, their data suggestthat the faster modality initiates the saccade and the slowermodality remains available to refine saccade trajectory.

The TWIN model makes specific assumptions about thetemporal configuration needed formultisensory integration tooccur (see also Colonius and Diederich 2004; Diederich andColonius 2007a,b, 2008a,b, for further details):

(1) Time-window-of-integration assumptions: In the focusedattention paradigm, crossmodal interaction occurs only if ((a))a nontarget stimulus wins the race in the first stage, openinga “time window” such that ((b)) the termination of the targetperipheral process falls in the window. The duration of the“time window” is a constant.

The idea here is that the winning nontarget will keep thesaccadic system in a heightened state of crossmodal reactivitysuch that the upcoming target stimulus, if it falls into the timewindow, will trigger crossmodal interaction. At the neurallevel, this might correspond to a gradual inhibition of fixationneurons (in superior colliculus) and/or omnipause neurons (inmidline pontine brain stem). In the case of the target being thewinner, no discernible effect on saccadic reaction time ispredicted, analogous to the unimodal situation.

The window of integration acts as a filter determiningwhether the afferent information delivered from differentsensory organs is registered close enough in time for cross-modal interaction to take place. Passing this filter is necessaryfor crossmodal interaction to occur. It is not a sufficient con-dition because interaction also depends on the spatial config-uration of the stimuli. Rather than assuming the existence of ajoint spatiotemporal window of integration permitting inter-action to occur only for both spatially and temporally neigh-boring stimuli, the TWINmodel allows for interaction to occureven for rather distant stimuli of different modalities, as longas they fall exactly within the time window. The two-stagestructure of the TWINmodel suggests an additional, importantassumption concerning the interplay of spatial and temporalfactors:

(2) Assumption of spatiotemporal separability: The amountof interaction ((facilitation or inhibition)) in second-stageprocessing time is a function of the spatial configuration ofthe stimuli, but it does not depend on their ((physical))presentation asynchrony ((SOA)).

These assumptions are part of a more general frameworkmaking a distinction between intra- and crossmodal stimulusproperties. Crossmodal properties are defined when stimuli ofmore than one modality are present, like spatial distance oftarget to nontarget or similarity between stimuli of differentmodalities. Intramodal properties, on the other hand, refer toproperties definable for a single stimulus, no matter whetherthis property is definable in all modalities (like intensity) or inonly one modality (like wavelength or frequency).

221B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

Intramodal properties can affect the outcome of the race inthe first stage and, thereby, the probability of an interaction.Crossmodal properties may affect the amount of crossmodalinteraction (Δ) occurring in the second stage. Note that cross-modal features cannot influence first stage processing timesince the stimuli are still being processed in separate pathways.Initial empirical evidence for these assumptions has been foundin Colonius and Diederich (2004) for visual–tactile stimulationandinArndtandColonius (2003) for visual–auditory stimulation.

The third assumption refers to the role of the nontargetstimulus as a warning cue. It can be formulated in two dif-ferent versions, a decision for one version or the other willhave to be decided upon empirical evidence:

(3) Assumption of warning mechanism: (Version A) If thenontarget wins the processing race in the first stage by amargin wide enough for the time window of integration toclose again before arrival of the target, then subsequent pro-cessing will be facilitated or inhibited without depending onthe spatial configuration of the stimuli. (Version B) If thenontarget wins the processing race in the first stage by a wideenough margin, then subsequent processing will in part befacilitated or inhibited without dependence on the spatialconfiguration of the stimuli.

The time margin by which the nontarget may win againstthe target will be called headstart. Version A then stipulatesthat the headstart is at least as large as the width of the timewindow. This precludes simultaneous occurrence of warningand multisensory interaction within the same trial. Version Bis less restrictive: all that is needed for the nontarget to act as awarning signal is a “large enough” headstart against the targetin the race. The actual value of the headstart criterion is aparameter to be estimated in fitting the model under eitherversion of the third assumption.

It follows that the occurrence of warning depends onintramodal characteristics of the target and the nontarget,such as modality or intensity. Assuming that increasing sti-mulus intensity goes along with decreased reaction time (forauditory stimuli see, e.g., Frens et al., 1995; Arndt and Colonius,2003; for tactile, Diederich and Colonius, 2004b), TWIN makesspecific predictions regarding the effect of nontarget intensityvariation thatwill be investigated here. For instance, an intenseauditory nontarget may have a higher chance to win the racewith a headstart compared to a weak tactile nontarget. Ingeneral, increasing the intensity of the nontarget (1) increasesthe probability of it becoming awarning signal, and (2)makes itmore likely for the nontarget to win the peripheral race againsttheprocessing of the target. Depending on SOA, however, it willbe shown below that the latter either results in an increasedprobability of integration if the target arrives “on time” to fallinto the time window, or it may decrease the probability ofintegration if the nontarget process finishes too early so thatthe window is already closed before the target arrives.

1.2. Quantitative specification of TWIN

For the race in the first stage of the model, we assign inde-pendent nonnegative random variables V and A to the peri-pheral processing times for the visual target and auditorynontarget stimulus, respectively. For ease of exposition, we

only consider the case of auditory nontargets. Whenever thenontarget is tactile instead of auditory, symbol A will have tobe replaced by symbol T in the expressions below. With τ asSOA value and ω as integration window width parameter, thetime window of integration assumption is equivalent to the(stochastic) event I, say,

I ¼ Aþ sbVbAþ sþ xf g:

Thus, the probability of integration to occur, P(I), is a functionof both τ and ω, and it can be determined numerically once thedistribution functions ofA andVhave been specified (see below).

The warning mechanism of the nontarget is triggeredwhenever the nontarget wins the race by a certain margin orheadstart γA. Thus, its occurrence corresponds to the event:

W ¼ Aþ sþ gAbVf g:

The probability of this event, P(W), is a function of both τ andγA, and its value can be determined numerically as soon asthe distribution functions of A and V have been specified (seebelow). Under version A of assumption (3), the headstart γA islargeenough for the integrationwindowtocloseagain, implying

gANxz0 and; therefore; P I \Wð Þ ¼ 0:

The next step is to compute expected total reaction time forthe unimodal and crossmodal conditions. From the two-stageassumption, total reaction time in the crossmodal conditioncan be written as a sum of two random variables:

RTcrossmodal ¼ S1 þ S2; ð1Þ

where S1 and S2 refer to the first and second stage processingtime, respectively. For the expected saccadic reaction time inthe crossmodal condition then follows (under version A of thewarning assumption):

E RTcrossmodal½ � ¼ E S1½ � þ E S2½ � ¼ E S1½ � þ P Ið ÞE S2jI½ � þ P Wð ÞE S2jW½ �þ 1� P Ið Þ � P Wð Þf gE S2jIc \Wc½ �

¼ E S1½ � þ E S2jIc \Wc½ � � P Ið Þ E S2jIc \Wc½ � � E S2jI½ �f g�P Wð Þ E S2jIc \Wc½ � � E S2jW½ �f g;

where E[S2|I], E[S2|W], and E[S2|Ic∩Wc ] denote the expected sec-ond stage processing time conditioned on interaction occurring(I),warning occurring (W ), or neither of themoccurring (Ic∩Wc ),respectively (Ic, Wc stand for the complement of events I,W.).Setting

DuE S2jIc \Wc½ � � E S2jI½ �juE S2jIc \Wc½ � � E S2jW½ �

this becomes

E RTcrossmodal½ � ¼ E S1½ � þ E S2jIc \Wc½ � � P Ið Þ � D� P Wð Þ � j ð2Þ

In the unimodal condition, no integration or warning ispossible. Thus,

E RTunimodal½ � ¼ E S1½ � þ E S2jIc \Wc½ �;

andwe arrive at a simple expression for the combined effect ofmultisensory integration andwarning, crossmodal interaction(CI),

CIuE RTunimodal½ � � E RTcrossmodal½ � ¼ P Ið Þ � Dþ P Wð Þ � j: ð3Þ

Note that Δ and κ can separately take on positive or ne-gative values (or zero) depending on whether multisensory

Table 1 – Restrictions to model parameters in theestimation routine

Parameters Restrictionlimits

1/λV 20−200 Mean peripheral processingtime for visual stimulus

1/λTH20−200 Mean peripheral processing time for

tactile stimulus, higher intensity1/λAH

20−200 Mean peripheral processing time forauditory stimulus, higher intensity

1/λTL20−200 Mean peripheral processing time for

tactile stimulus, lower intensity1/λAL

20−200 Mean peripheral processing time forauditory stimulus, lower intensity

µ N0 Mean central processing timeωT b300 Window of integration for tactile

stimulusωA b300 Window of integration for auditory

stimulusΔTipsi

None Amount of crossmodal interactiondue to tactile stimulus presentedipsilaterally

ΔTcontraNone Amount of crossmodal interaction

due to tactile stimulus presentedcontralaterally

ΔAipsiNone Amount of crossmodal interaction

due to auditory stimulus presentedipsilaterally

ΔAcontraNone Amount of crossmodal interaction

due to auditory stimulus presentedcontralaterally

γTHDependingon model

Headstart for tactile stimulus,higher intensity

γAHDependingon model

Headstart for auditory stimulus,higher intensity

γTLDependingon model

Headstart for tactile stimulus,lower intensity

γALDependingon model

Headstart for auditory stimulus,lower intensity

κ ≥0 Amount of warning cue effect

222 B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

integration and warning have a facilitative or inhibitoryeffect.

Under version B of the warning mechanism assumption, itis assumed for simplicity that the effects on RT of the twoevents I and W, integration and warning, combine additively.Interestingly, it can then be shown that the crossmodalinteraction prediction of this model version is captured bythe same equation as under version A, i.e., Eq. (3). The onlydifference is in the order restriction for the parameters, γANω,under version A.

1.3. Model predictions

Even without making explicit assumptions about the prob-ability distributions of the peripheral processing times, a num-ber of qualitative predictions from the model can be made.Note, first, that for a given spatial configuration and nontargetmodality there are no sign reversals or changes in magnitudeof Δ or κ across all SOA values. In contrast, both the probabilityof integration P(I) and the probability of warning P(W) dochange with SOA. In particular, when the nontarget ispresented very late relative to the target (large positive SOA),its chances of winning the race against the target and thusopening thewindowof integration becomevery small.When itis presented rather early (large negative SOA), it is likely to winthe race and to open the window, but the window may beclosed by the time the target arrives. Again, the probability ofintegration, P(I), is small. Therefore, the largest integrationeffects are expected for some mid-range SOA values. On theother hand, the probability of warning P(W) decreases mono-tonically with SOA: the later the nontarget is presented thesmaller are its chances to win the race against the target withsomeheadstart γ. Obviously, there alsohas to be anupper limitto the headstart for the warningmechanism to become active:if the nontarget occurs several seconds before the target itmayno longer act as a warning cue at all. Therefore, in order to fit aparametric version of the model to empirical data, a certainnumerical range for each possible parameter valuewill have tobe specified (see Table 1).

In the parametric version of the TWINmodel, all peripheralprocessing times are assumed to be exponentially distributed(cf. Colonius and Diederich 2004). Assuming explicit probabil-ity distributions is a requirement for calculating numericalvalues of P(I) and P(W), but the particular choice of the expo-nential is not: it is chosen here for computational simplicityonly. The exponential distribution is characterized by a singlequantity, the intensity parameter λ (for a full description of thecomputation of P(I) and P(W) we refer to Diederich andColonius, 2008a).

To illustrate the mean SRT predictions of TWIN as afunction of SOA, we choose a set of arbitrary (but plausible)parameters. Let the parameter for the visual target processingtime be λV=0.025 [1/ms] which amounts to a mean peripheraltime for visual unimodal stimuli of 40ms (1/λV). The parameterfor second stage central processing time when neitherintegration nor warning occurs, μ≡E[S2|Ic∩Wc], is set to 100[ms]. Note that we need not make any distributional assump-tions about second stage processing time as long as we restrictour predictions to the level of mean SRT. Neither λV nor μ aredirectly observable, but the sum of the peripheral and central

processing time for the visual target stimulus constitutes aprediction for unimodal mean SRT:

E RTunimodal½ � ¼ 1=km þ l:

For a prediction of crossmodal SRTs we have to specify theremaining parameters. Let the parameter of nontarget proces-sing time be λA=0.015 [1/ms], corresponding to a mean peri-pheral time of 66.7ms. The timewindowof integration is set toω=150ms. The parameter for multisensory integration Δ is setto 20 ms. Finally, the warning cue effect also reducing meanSRT to bimodal stimuli, κ, is set to 5 ms. Fig. 1 plots predictedmeanSRTas a functionof SOA.Thehorizontal line refers to theunimodal condition, the red curve to the crossmodal conditionwithout warning, and the six blue dashed lines are the predic-tions for different γ values of the headstart, from γ=50 ms(bottom curve) to γ=300 ms (top curve) in steps of 50. Withoutwarning, mean SRT approaches the unimodal mean for largeenoughnegative SOAvalues. For theseSOAvalues, the effect ofthe headstart is most pronounced, as to be expected. Here, thedifference between the two versions of the warning mechan-ism becomes apparent: for γ values smaller than ω=150 ms

Fig. 2 – TWIN model prediction for mean SRT. Mean SRTas a function of SOA and headstart γ, with other parametersas in Fig. 1.

223B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

(three bottom curves), the warning adds to the multisensoryintegrationeffect aroundSOA=−100msbecause the effects areadditive,whereas for largerheadstart valueswarning effects inthat SOA range are no longer possible.

Amore comprehensive illustration of the effects ofwarningis given in Fig. 2 depicting mean crossmodal SRT as a functionof the headstart parameter, γ, varying from 10 to 300 ms insteps of 10 ms, and of the SOA varying from −300 to 100.

Fig. 3 shows, as a function of SOA, the probability of inte-gration (red line) and the corresponding probability of warningfor the six different values of the headstart parameter γ (bluedashed curves, most right hand γ=50, most left γ=300). Theseprobabilities are not observable but their time course illus-trates the interplay between warning and integration: (i) theprobability of warning decreases towards zero as SOA in-creases, with the headstart parameter controlling the locationof decline: the larger γ the smaller the probability of warningat a given SOA; (ii) the probability of integration is a peakedfunction of SOA:

If target and nontarget are presented in two distinct spatialconditions, ipsilateral and contralateral, say, onewould expectΔ to take on two different values, Δi and Δc, whereas P(W) κ, theexpected non-spatial warning effect, should remain the sameunder both conditions. Subtracting the corresponding cross-modal interaction terms then gives, after canceling thewarning effect terms (Eq. (3)),

CIi � CIc ¼ P Ið Þ � Di � Dcð Þ; ð4Þ

an expression that should yield the same qualitative behavior(especially, peakednesss), as a function of SOA, as P(I).

Finally, increasing the intensity level of the nontargetperipheral process (λA) affects both the probability of warningand of integration as a function of SOA: (i) the higher λA the

Fig. 1 – TWIN model prediction with and without warning.Mean unimodal SRT (dashed line) and mean crossmodalSRT without warning (red line) as a function of SOA.The blue dashed curves are mean crossmodal SRT for 6values of the headstart parameter γ, from 50 ms (bottomcurve) to 300 ms (top curve) in steps of 50; with kV=.025,kA=.015, ω=150.

higher the chances of the nontarget to win against the targetwith the headstart required for the warning to take place(dashed curves in Fig. 4); (ii) the effect on integration prob-ability is more complex: for large negative SOA, i.e., when the(auditory) nontarget arrives very early – so thatwarning occurswith high probability – further increasing the auditory in-tensitymakes itmore likely for the timewindowof integrationto close before the target arrives and therefore results in lowerP(I) values; for SOA values in the range of about −120 to −40(given the selected parameter values in Fig. 4) turns the effectof around: P(W) is already low and increasing auditoryintensity now improves its chances to win against the targetand to open the window, as illustrated by the five crossing redlines in Fig. 4).

Fig. 3 – TWIN model prediction. Probability of integration(red line) and probability of warning (dashed curves)with γ=50,100,150,200 or 250 ms, from right to left. Here,kV=.025, kA=.015, ω=150.

224 B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

2. Results

Saccades were screened for anticipation errors (SRTb80 ms,0.6% of all trials)), misses (SRTN500 ms, none), and accuracy:trials with saccade amplitudes deviating more than threestandard deviations from the mean amplitude were excludedfrom the analysis (0.9% of all trials). Given that apart fromindividual speed levels all six participants exhibited verysimilar results across stimulus conditions, all data presentedbelow are averages over participants.

2.1. Qualitative test

We first consider the test implied by Eq. (4),

CIi � CIc ¼ P Ið Þ � Di � Dcð Þ: ð5Þ

Since according to the model the difference Δi−Δc does notdepend on SOA, this expression should have the samequalitative shape as P(I), i.e., it should go to zero for small andfor large enough SOA values and peak in between, at aroundSOA=100 ms. Fig. 5 displays the results. Apart from variabilitydue to sampling error, the shapes of both curves (circles fortactile, triangles for auditory nontargets, lines for strongerintensities, dashed lines for weaker intensities) are partially inagreement with the predicted shape of P(I). Specifically, thetactile curve (TH) peaks before the auditory curve (AH) and thetwo auditory curves cross as predicted by the model.

2.2. Quantitative tests

Table 1 (left column) lists all parameters of TWIN that have tobe estimated. The middle column of Table 1 lists the para-meter restrictions that were imposed on the estimationroutine. The λ values were restricted to fall within a range

Fig. 5 – Qualitative test of integration and warningmechanism. Panel (A) Probability of integration for auditoryand tactile nontargets with high and low intensity levelunder warning assumption A; Panel (B) Crossmodalintegration difference ipsi- vs contralateral as function of SOAthat should share qualitative features with curves in (A).

Fig. 4 – TWIN model prediction. Probability of integration(red lines) and probability of warning (dashed curves)for different values of auditory intensity kA=.01, .015, .02,.025, or .03. Here, γ=200, other parameter values are thesame as in Fig. 3.

consistent with neurophysiological estimates for peripheralprocessing times (Stein and Meredith 1993; Groh and Sparks,1996). The width of the the time window of integration for atactile and an auditory nontarget, ωT and ωA, respectively, waslimited to an upper bound of 300 ms. These 17 parameterswere estimated from 45 means from 25,366 valid observations(40 bimodal and five unimodal mean SRT).

The parameters were estimated byminimizing the Pearsonχ2 statistic

v2 ¼XNn¼1

X2i¼1

X2j¼1

X2k¼1

SRT i; j; k;nð Þ �bSRT i; j; k;nð ÞrSRT i;j;k;nð Þ

!2

þX5i¼1

SRTuni ið Þ �bSRTuni ið ÞrSRTuni ið Þ

!2

ð6Þ

Table 2 – Estimated parameters [in ms] for both versionsof the warning mechanism

Estimated parameters No restriction γ≥ω

1/λV 58 471/λTH

103 921/λAH

79 671/λTL

131 1201/λAL

110 101µ 103 114ωT 111 156ωA 101 98ΔTipsi

27 28ΔTcontra

11 16ΔAipsi

17 28ΔAcontra

5 16γTH

90 175γAH

34 98γTL

102 190γAL

59 104κ 21 22χ2 122 150

225B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

using the FMINSEARCH routine of MATLAB. Here SRT i; j; k;nð ÞandbSRT i; j; k;nð Þ are, respectively, the observed and the fittedvalues of the mean SRT to bimodal stimuli (visual–auditory,i=1; visual–tactile, i=2) with intensities (low, j= i, high, j=2)presented in spatial positions (ipsilateral, k=1; contralateral,k=2) with SOA (referred to by n to N=5); rSRT i;j;k;nð Þ

and rSRTuni ið Þare the respective standard errors. The second term in Eq. (6)refers to the five unimodal condition. All parameter estimatesare collected in Table 2 (left column: no restriction on theordering of γ andω, as in Version B of thewarning assumption;right column: under the restriction of γ≥ω, as in version A).The minimized χ2 values (df=28) are shown in the last row ofTable 2.

Figs. 6 and 7 plot the observedmean SRT values (±standarderror) and model predictions across SOAs for all nontargetconditions for version A and B, respectively. Upper panelsshow auditory nontargets with high (left panel) and low (rightpanel) intensities. Lower panels show the respective tactilenontarget conditions. Circles (squares) refer to mean SRTs tobimodal stimuli presented ipsilaterally (contralateral), corre-sponding model predictions are presented by a dashed line(ipsilateral) and a line (contralateral). The horizontal dashedline marks the estimated mean unimodal (visual) SRT.

Figs. 8 and 9 plot the probability of integration (lines) andthe probability of warning (dashed lines) computed from themodel parameter estimates for version A and B, respectively.The differences between the dashed and the solid linesexpress the effect of changing the intensity of the nontarget:the warning probability decreases with decreasing intensity,whereas integration probability first increases and then, forlarger SOA, decreases with intensity, as predicted (see Figs. 3and 4).

3. Discussion

Inspecting the five intensity parameters (λ values) in Table 2,the estimates are found not to be in line with the expectation

that the visual peripheral latencies are longer than theauditory or somatosensory ones. However, peripheral laten-cies are also known to depend on the relative intensity levels;here, the stimuli were not matched so as to achieve acrossmodal subjective equivalence in intensity. Reassuringly,the order of high vs. low intensity within the auditory and thesomatosensory modality, respectively, was reflected correctlyin the parameter values. Second, by definition “peripherallatency” in TWIN is the processing time before any crosstalkbetween the senses occurs; it is not entirely clear (to us)whether the estimates for peripheral latencies from theneurophysiology literature are in one-to-one correspondenceto this definition. Finally, some of the discrepancymay simplyresult from problems in estimating these parameter values. Infuture work, the stability and reproducibility of the parameterestimates should be investigated systematically.

Visual inspection of Figs. 6 and 7 indicates that both TWINmodel versions reflect the main qualitative properties of thedata relatively well: (i) for large negative SOA (−250 ms), thereis facilitation but no difference between ipsi- and contralateralpresentation of target and nontarget, (ii) the difference be-tween the ipsi- and contralateral condition increases untila maximum around SOA=−100 to −50 is achieved, and(iii) facilitation decreases toward zero with increasing SOAfurther. This study specifically addressed the effect of varyingnontarget intensity. The plots indicate that decreasing inten-sity somewhat flattens the increase of the SOA curve for boththe auditory and the tactile nontargets such that facilitationbecomes lesser. This is in line with the decrease of integrationprobability in this SOA range predicted by the model (seeFig. 4). Comparing the nontarget modality conditions suggeststhat facilitation effects for auditory nontargets are slightlymore pronounced than for tactile ones. Again, this could bepredicted from the corresponding probability of integration/warning plots (Figs. 8, 9): the probability of warning levels offmuch earlier for the tactile than for the auditory nontargets(under both model versions).

The prediction alluded to in the title of this paper may beinterpreted as being in conflict with the well known principleof “inverse effectiveness” according to which a decrease ofstimulus intensity should go along with an enhancement ofmultisensory interaction, i.e. in this context, a reduction ofSRT (Stein and Meredith 1993). As explicated above (cf. Fig. 4),the effect of decreasing nontarget intensity on integrationprobability critically depends on the SOA range. In particular,inverse effectiveness would only be expected for rather largenegative SOAs, i.e., when the nontarget leads the target by asufficient amount of time. For SOA values of −120 to −40,where the largest facilitation effects occur, however, decreas-ing intensity would have the opposite effect since it woulddiminish the nontarget's chance of winning the race andopening the time window. In this context, it should be notedthat the validity of the principle of inverse effectiveness, aswell as its precise definition, have become a matter debaterecently (Holmes 2007; Ross et al. 2007). These predictions ofTWIN certainly need further empirical testing, with a highernumber of intensity levels, before a definite resolution of thisissue is possible.

The crossmodal spatial interaction effects in saccadesreported here are in line with findings from other labs using

Fig. 6 – Model fit for TWIN with warning version A. Observed and predicted mean SRT for model with warning mechanismversion A as a function of SOA for auditory (top panels) and tactile (bottom panels) nontargets with high (left panels)and low (right panels) intensity. Horizontal dashed line: unimodal visual observed data; blue lines and points:contralateral condition, red lines and points: ipsilateral condition.

226 B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

visual targets and either auditory or tactile nontargets(e.g., Amlôt et al. 2003; Bell et al. 2005; Frens et al. 1995;Harrington and Peck 1998) and also with our own results(Colonius and Diederich 2004; Diederich et al. 2003; Dieder-ich and Colonius 2007a,b, 2008a). The time course of cross-modal mean SRT over SOAs and the pattern of facilitationobserved here suggest the existence of two distinct under-lying mechanisms: (a) a spatially unspecific crossmodalwarning triggered by the nontarget being detected earlyenough before the arrival of the target plus (b) a spatiallyspecific multisensory integration mechanism triggered bythe target processing time terminating within the timewindow of integration.

On the other hand, due to the large number of observations(about 100 trials/condition), a χ2 goodness of fit test has largepower and clearly rejects both versions of the model (pb .001).

There are several, notmutually exclusive explanations for thisfailure: First, the model, although not really simple, may notcapture all relevant factors affecting the timing of the saccadicresponse. Second, there may be idiosyncratic effects ofindividual participants that have not been averaged outgiven their small number. Third, since the number of para-meters is high, the estimation procedure may not yet havedelivered the best estimates possible.

Consequently, it appears sensible to consider the χ2 valuemore as a descriptive measure of model appropriatenessthan as a criterion to reject themodel outright. In this respectversion B, i.e., no restriction of γ relative to ω, accountsslightly better for the data as reflected in the smaller χ2 valuein Table 2. This suggests that the restriction that in any giventrial only one mechanism can be operative – warning orintegration – may be too strong. In other words, the

Fig. 7 – Model fit for TWIN with warning version B. Same as Fig. 6 for model under warning mechanism version B.

227B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

nontarget may be effective both as a warning signal and asan element of the bimodal stimulus configuration inducingmultisensory integration, and its influence in either roledepends on SOA in a manner depicted in Fig. 3. Thus, thisfinding goes beyond our previous study (Diederich andColonius, 2008a) where no advantage of either version wasdiscernable.

The time window of integration model introduced inColonius and Diederich (2004) and extended by a warningmechanism inDiederich andColonius (2008a) has bynowbeentested in a variety of ways: it accounts for the spatial con-figuration of the stimuli (Diederich andColonius 2007b), for theeffect of increasing the number of nontargets presentedtogether with the target (Diederich and Colonius 2007a), forthe warning effect occurringwith large negative SOAs (Dieder-ich and Colonius, 2008a) and, as shown here, for the effects ofincreasing the intensity of the nontarget. The qualitative testpresented here, based on Equation (4) and depicted in Fígure 5,showing that the peakedness of the crossmodal integrationcurves over SOA can been recovered constitutes further strong

support for the central assumption of spatiotemporal separ-ability in TWIN, independent of the specific distributionalassumptions for the peripheral latencies.

Finally, it should be noted that the idea of multisensoryintegration being determined by a time window had alreadybeen suggested in Meredith, Nemitz, and Stein (1987) record-ing from SC neurons. It, together with the concept of a raceamong peripheral processes, now underlies many studies ofcrossmodal temporal interaction (e.g., Van Opstal and Munoz,2004; Lewald and Guski, 2003; Morein-Zamir, Soto-Faraco,Kingston, 2003; Spence and Squire, 2003; see Whitchurch andTakahashi, 2006, for head saccades in the barn owl).

4. Experimental procedures

In a focused attention paradigm, saccadic reaction times toa visual target stimulus were measured in the presence orabsence of an auditory or tactile accessory (nontarget)stimulus.

Fig. 8 – Probability of warning and integration. Probability ofintegration and of warning for high and low intensity ofnontarget (top panel: tactile, bottom panel: auditory)with parameters from Table 2 for model under warningmechanism version A.

Fig. 9 – Probability of warning and integration. Probability ofintegration and of warning for high and low intensity ofnontarget (top panel: tactile, bottom panel: auditory)with parameters from Table 2 for model under warningmechanism version B.

228 B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

4.1. Participants

Six undergraduate students, aged 20 to 23, one female, servedas paid voluntary participants. All had normal or corrected tonormal vision. They were screened for their ability to followthe experimental instructions (proper fixation, few blinksduring trial, saccades towards visual target). They gave theirinformed consent prior to their inclusion in the study. Theexperiment was conducted in accordance with the ethicalstandards described in the 1964 Declaration of Helsinki.

4.2. Apparatus and stimulus presentation

Two red light-emitting diodes (LED, 25mA, 3.3 mcd) presentedagainst a black background served as visual targets (V). Theywere placed on a table 600 mm in front of the participant, 20°to the left or right of a central fixation point (fixation LED, red,

25 mA, 5.95 mcd). The tactile stimulation (50 Hz, 1–2 mmamplitude) was generated by two silenced oscillation exciters(Mini-Shaker, Type 4810, Bruel and Klær) placed on basessituated under the table. Positioned in each shaker was ametal rod extending through a hole in the table approximately20 mm above the surface. On each rod was a wooden ball of14 mm diameter, which rested in the palm of the participantand transmitted the vibration to the hand. They were placed20° to the left and right of the fixation LED and 620mm in frontof the participant. Two intensities were employed: a lowerintensity with 0.25 V and denoted (t) and a higher intensitywith 0.75 V, denoted (T). Auditory stimuli were bursts ofwhite noise with two levels of intensity, a lower intensity with25.5 dBA SPL, (a), and a higher intensity with 45 dBA SPL). Theywere generated by two speakers (Canton Plus XS). Thespeakers were placed at 20° to the left and right of the fixation

229B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

LED at the same hight as the participants' ear level and120 cm in front of the participants. The black table was(180×130×75 cm) with a recess to sit in. One PC controlled thestimulus presentation, and two other interlinked PCs con-trolled the EyeLink program.

4.3. Data collection

Saccadic eye movements were recorded by an infrared videocamera system (EyeLink II, SR Research) with a temporalresolution of 500 Hz and a horizontal and vertical spatialresolution of 0.01°. Criteria for saccade detection on a trial bytrial basis were velocity (35°/s) and acceleration (9500°/s2). Therecorded eye movements from each trial were checked forproper fixation at the beginning of the trial, eye blinks andcorrect detection of start and end point of the saccade.

4.4. Spatial and temporal stimulus configuration

The visual target was presented (unimodally) or in the presenceof an auditory or a tactile nontarget. For bimodal presenta-tions, two spatial and five temporal stimulus configurationswere used. Spatial configuration: (i) target (V) and high- or lowintensity nontarget (AH, AL, TH, TL) were presented in the samehemifield (20° left or right fromfixation) (ipsilateral), or (ii) targetand nontarget were presented in opposite hemispheres(20° left and right, or vice versa) (contralateral). Temporal con-figuration: five different stimulus onset asynchrony levels(SOAs) were used between targets and nontargets. The non-targetwas presented 250ms, 100ms, or 50ms before the targetappeared (SOAs=−250, −100, −50), or target and nontargetwere presented simultaneously (SOA=0), or the nontarget waspresented 50 ms after the target appeared (SOA=50).

4.5. Procedure

The participants were seated in a completely darkened, soundattenuated room with the head positioned on a chin rest andthe elbows and lower arms resting comfortably on a table.They were unable to see their hands during the experiment.Every experimental session began with 10 min dark adapta-tion during which themeasurement systemwas adjusted andcalibrated. During this phase, participants put the hands at theposition used during the entire experimental block.

Each bimodal trial began with the appearance of thefixation point. After a variable fixation time (800–1500 ms),the fixation LED disappeared and, simultaneously, the visualtarget stimulus was turned on (i.e., no gap). Participants wereinstructed to gaze at the visual target as quickly and as accu-rately as possible ignoring any auditory or tactile nontargets(focused attention paradigm). The visual target appeared incombination with either a tactile or an auditory nontarget inipsi- or contralateral position. The visual stimuli were pre-sented for 500 ms; the auditory and tactile nontargets wereturned off simultaneously with the visual stimulus. Thus theirduration varied between 750 and 450 ms, depending on SOA.Stimulus presentation was followed by a break of 2000 ms incomplete darkness, before the next trial began, indicated bythe onset of the fixation LED. 240 bimodal trialswere presentedwithin one block of trials randomized across all experimental

conditions (two nontarget modalities, two intensities, fiveSOA, two lateralities). Two blocks of trials were conductedwithin 1 h. After extensive training (60–120 min), eachparticipant completed a total of 3840 bimodal trials (96 trialsper condition). Unimodal blocks consisted of 250 trials andwere conductedwithin 30min.Oneblockwaspresented beforeand one after the entire bimodal presentation, resulting in atotal of 500 unimodal trials (100 trials per condition).

Acknowledgments

This research was supported by grants from DeutscheForschungsgemeinschaft Di 506/8–1 and /8–3. We are gratefulto Dr. Annette Schomburg, Rike Steenken, and Stefan Rach forhelpful discussions of this work.

R E F E R E N C E S

Amlôt, R., Walker, R., Driver, J., Spence, C., 2003. Multimodalvisual-somatosensory integration in saccade generation.Neuropsychologia 41, 1–15.

Arndt, A., Colonius, H., 2003. Two separate stages in crossmodalsaccadic integration: Evidence from varying intensity ofan auditory accessory stimulus. Exp. Brain Res. 150, 417–426.

Bell, A.H., Meredith, A., Van Opstal, A.J., Munoz, D.P., 2005.Crossmodal integration in the primate superior colliculusunderlying the preparation and initiation of saccadic eyemovements. J. Neurophysiol. 93, 3659–3673.

Colonius, H., Diederich, A., 2004. Multisensory interaction insaccadic reaction time: A time-window-of-integration model.J. Cogn. Neurosci. 16, 1000–1009.

Colonius, H., Diederich, A., 2006. Race model inequality:Interpreting a geometric measure of the amount of violation.Psychol. Review 113 (1), 148–154.

Corneil, B.D., VanWanrooij, M., Munoz, D.P., Van Opstal, A.J., 2002.Auditory–visual interactions subserving goal-directedsaccades in a complex scene. J. Neurophysiol. 88, 438–454.

Diederich, A., Colonius, H., 2004a. Modeling the time course ofmultisensory interaction in manual and saccadic responses.In: Calvert, G., Spence, C., Stein, B.E. (Eds.), Handbook ofMultisensory Processes. MIT Press, Cambridge, MA.

Diederich, A., Colonius, H., 2004b. Bimodal and trimodalmultisensory enhancement: effects of stimulus onset andintensity on reaction time. Percept Psychophys. 66 (8),1388–1404.

Diederich, A., Colonius, H., 2007a. Why two “distractors” are betterthan one: modeling the effect of nontarget auditory and tactilestimuli on visual saccadic reaction time. Exp. Brain Res. 179,43–54.

Diederich, A., Colonius, H., 2007b. Modeling spatial effects invisual–tactile saccadic reaction time. Percept. Psychophys.69 (1), 56–67.

Diederich, A., Colonius, H., 2008a. Crossmodal interaction insaccadic reaction time: separating multisensory from warningeffects in the timewindow of integrationmodel. Exp. Brain Res.186, 1–22.

Diederich, A., Colonius, H., Schomburg, A., 2008b. Assessingage-related multisensory enhancement with thetime-window-of-integration model. Neuropsychologia.

Diederich, A., Colonius, H., Bockhorst, D., Tabeling, S., 2003.Visual–tactile spatial interaction in saccade generation.Exp. Brain Res 148, 328–337.

230 B R A I N R E S E A R C H 1 2 4 2 ( 2 0 0 8 ) 2 1 9 – 2 3 0

Doyle, M.C., Walker, R., 2002. Multisensory interactions in saccadetarget selection: Curved saccade trajectories Exp. Brain Res.142, 116–130.

Frens, M.A., Van Opstal, A.J., Van der Willigen, R.F., 1995. Spatialand temporal factors determine auditory–visual interactions inhuman saccadic eye movements. Percept. Psychophys. 57,802–816.

Groh, J.M., Sparks, D.L., 1996. Saccades to somotosensory targets.III. Eye-position-dependent somatosensory activity inprimates superior colliculus. J. Neurophysiol. 75, 439–453.

Harrington, L.K., Peck, C.K., 1998. Spatial disparity affectsvisual–auditory interactions in human sensorimotorprocessing. Exp. Brain Res. 122, 247–252.

Holmes, P.N., 2007. The law of inverse effectiveness in neuronsand behavior: Multisensory integration versus normalvariability Nicholas P. Holmes. Neuropsychologia 45,3340–3345.

Hughes, H.C., Nelson, M.D., Aronchick, D.M., 1998. Spatialcharacteristics of visual–auditory summation in humansaccades. Vis. Res. 38, 3955–3963.

Lewald, J., Guski, R., 2003. Cross-modal perceptual integration ofspatially and temporally disparate auditory and visual stimuli.Cogn. Brain Res 16, 468–478.

McDonald, J.J., Teder-Sälejärvi, W.A., Hillyard, S.A., 2000.Involuntary orienting to sound improves visual perception.Nature 407, 906–908.

Meredith, M.A., Nemitz, J.W., Stein, B.E., 1987. Determinantsof multisensory integration in superior colliculusneurons. I. Temporal factors. J. Neurosci 10, 3215–3229.

Meredith, M.A., 2002. On the neural basis for multisensoryconvergence: a brief overview. Cogn. Brain Res. 14, 31–40.

Morein-Zamir, S., Soto-Faraco, S., Kingstone, A., 2003. Auditorycapture of vision: examining temporal ventriloquism.Cogn. Brain Res. 17, 154–163.

Ross, L.A., Saint-Amour, D., Leavitt, V., Javitt, D.C., Foxe, J.J., 2007.Do you see what I'm saying? Optimal visual enhancement ofspeech recognition in noisy environments. Cereb. Cortex. 17,1147–1153.

Spence, C., Squire, S., 2003. Multisensory integration: maintainingthe perception of synchrony. Curr. Biol. 13, R519–R521.

Stein, B.E., Meredith, M.A., 1993. The merging of the senses.The MIT Press, Cambridge MA.

Van Opstal, A.J., Munoz, D.P., 2004. Auditory–visual interactionssubserving primate gaze orienting. In: Calvert, G., Spence, C.,Stein, B.E. (Eds.), Handbook of Multisensory Processes.MIT Press, Cambridge, MA, pp. 373–393.

Van Wassenhove, V., Grant, K.W., Poeppel, D., 2007. Temporalwindow of integration in auditory–visual speech perception.Neuropsychologia 45, 598–607.

Whitchurch, E.A., Takahashi, T.T., 2006. Combined auditory andvisual stimuli facilitate head saccades in the barn owl(Tyto alba). J. Neurophysiol. 96, 730–745.