research report modality-specific processing streams in verbal

Ž .Cognitive Brain Research 6 1997 95–113

Research report

Modality-specific processing streams in verbal working memory:evidence from spatio-temporal patterns of brain activity

Daniel S. Ruchkin a,), Rita S. Berndt b, Ray Johnson Jr. c, Walter Ritter d, Jordan Grafman e,Howard L. Canoune a

a Department of Physiology, School of Medicine, UniÕersity of Maryland, Baltimore, MD 21201-1559, USAb Department of Neurology, School of Medicine, UniÕersity of Maryland, Baltimore, MD 21201-1559, USA

c Department of Psychology, Queens College, City UniÕersity of New York, Flushing, NY, USAd Department of Neuroscience, Albert Einstein College of Medicine, Bronx, NY, USA

e CognitiÕe Neuroscience Section, Medical Neurology Branch, National Institute of Neurological Disorders and Stroke, Bethesda, MD, USA

Accepted 20 May 1997

Abstract

The present study was concerned with whether there are separate, modality-specific processing ‘‘streams’’ in verbal working memoryŽ .for information that is heard or read. We used event-related brain potentials ERPs recorded from scalp of normal humans to show

between-modality differences in spatio-temporal patterns of brain activity during retention in working memory of aurally or visuallypresented verbal information. The ERP patterns suggested that a sustained, automatically maintained auditory store was activated byauditory presentation and a transient, visual-verbal store was activated by visual presentation. In addition to these modality-specificdifferences, the ERPs indicated that the phonological loop was activated in both modalities and further suggested that the onset ofphonological loop activation was earlier for auditory presentation. q 1997 Elsevier Science B.V.

Keywords: Working memory; Short-term memory; Verbal; Modality; Auditory; Visual; Event-related potential; Slow wave

1. Introduction

Short-term storage of verbal information utilizes work-w xing memory. Baddeley 3 proposed a model of working

Ž .memory that consists of three independent components: 1a central executive that determines what information ismade available for conscious processing by exerting con-

Ž .trol over voluntary action; 2 a verbal rehearsal system;Ž . Žand 3 a visuo-spatial rehearsal system visuo-spatial

.sketchpad . The verbal rehearsal system was posited toconsist of a phonological component having a limitedduration passive buffer for phonological codes and anarticulatory rehearsal process that refreshes the buffer.There is considerable evidence that verbal informationpresented in the auditory modality, after phonological anal-ysis, directly accesses the phonological buffer, while ver-bal information presented in the visual modality requiresexplicit translation to phonological format before it can

w xaccess the phonological buffer 45 . Specifically, it has

) Ž . Ž .Corresponding author. Fax: q1 410 706-8341; E-mail Bitnet :[email protected]

been suggested that verbal information presented in printundergoes visual analysis, followed by a re-coding fromvisual to phonological format, before the re-coded informa-tion enters the phonological store through articulatory re-

w xhearsal 45 .w xPenney 27 extended Baddeley’s model to include

modality-specific sensory processing ‘‘streams’’ that fur-ther contribute to the retention of verbal information inworking memory. She hypothesized that, along with thephonological loop, auditory and visual stores are automati-cally activated by presentation of verbal information in therelevant modality, with the auditory store being more

Žhighly structured and durable than the visual store seew x.also Cowan 8 . To support her modality-specific ‘‘sep-

w xarate streams’’ model, Penney 27 cited a number of linesŽ .of evidence: 1 memory is improved when different items

are presented in different modalities relative to presenta-Ž .tion of different items in a single modality; 2 presenta-Ž .tion of additional stimuli at the ends of lists suffix effect

has different effects upon recall as a function of modality;Ž .3 recall is enhanced when items are organized by modal-

Ž .ity than by time of presentation; 4 two concurrent verbal

0926-6410r97r$17.00 q 1997 Elsevier Science B.V. All rights reserved.Ž .PII S0926-6410 97 00021-9

( )D.S. Ruchkin et al.rCognitiÕe Brain Research 6 1997 95–11396

tasks can be more effectively performed when differentinput modalities are utilized in comparison with a singleinput modality.

The question of whether verbal working memory in-volves separate, modality-specific processing streams hasbeen addressed in a recent positron emission tomographyŽ . w xPET study by Schumacher et al. 44 . Subjects performeda continuous processing task requiring the maintenance ofthree letters in working memory for 9 s. Letters werepresented either aurally or visually in different blocks oftrials, with a 3 s inter-letter interval. Regional cerebral

Ž .blood flow rCBF was summed over 60 s intervals duringthe memory task and a control detection task. Consistentwith previous PET studies of verbal working memoryw x2,12,26,29,46 , when activation values generated duringthe control task were subtracted from the activation valuesgenerated during the memory task, memory-related rCBFwas revealed in the left hemisphere in dorsal-lateral frontalcortex, Broca’s area, supplementary motor area, and pre-motor cortex, and bilaterally in anterior cingulate, andsuperior and posterior parietal cortices. There was almostcomplete overlap of the rCBF activation patterns in the

w xtwo modalities, leading Schumacher et al. 44 to suggestthat the verbal working memory system was amodal.

Thus, there appears to be a contradiction between Schu-w xmacher et al.’s 44 PET findings and the behavioral data

w xreviewed by Penney 27 . Schumacher et al. argued againstthe possibility that lack of sensitivity in their PET ap-proach may have hidden evidence for separate auditoryand visual processing streams by contrasting auditory andvisual modality scans, separately for the memory andcontrol tasks. These subtractions revealed modality-specificrCBF activation which was attributed to encoding pro-cesses. While encoding is a plausible interpretation for themodality-specific rCBF activation, it may not be a com-plete explanation. For example, automatically activatedsensory-specific stores also may have contributed to themodality-specific activation. Furthermore, if some of thehypothesized processing in the modality-specific process-ing streams is sufficiently rapid, then the rCBF activationassociated with it may not be reliably distinguished fromnoise.

The issue of the existence of modality-specific process-ing streams in verbal working memory can also be ad-dressed using the spatio-temporal properties of event-re-

Ž .lated brain potentials ERPs recorded from the scalp ofnormal humans. In contrast with imaging techniques suchas PET, ERPs have a high time resolution relative to theduration of the cognitive processes being investigated.Thus, encoding, retention and recall processes can beisolated by suitable separation along the time line, and thespatio-temporal characteristics of transient working mem-ory operations can be delineated. The issue of modality-specific processing streams in verbal working memory canbe addressed by between-modality comparisons of ERPscalp topographic distributions at designated times in the

retention interval. Different ERP scalp topographies duringretention of verbal information presented selectively in theauditory or visual modality only will occur if the patternsof underlying brain activation differ for auditory or visualpresentation of verbal information. Hence, finding suchtopographic differences would be evidence for the exis-tence of separate, modality-specific processing streams.

A number of recent human ERP studies of workingmemory have found long-duration slow wave ERPs thatwere specific to the type of information held in workingmemory and sensitive to the attributes and amount of

w x w xmaterial retained 17,38–40 . Ruchkin et al. 38–40 em-ployed delayed match-to-sample tasks and found two ERPslow waves during the delay interval that were specific toverbal working memory tasks: a bilateral posterior positivewave followed by a left frontal negative wave. The ampli-tudes of both waves were related directly to verbal mem-ory load and were negligible in a non-memory controltask. The left frontal negativity appeared to index, in part,articulatory rehearsal operations, since it was sustaineduntil the end of the delay interval and located in thevicinity of brain regions that participate in the articulatory

w xrehearsal process 49 . Moreover, the variation of the leftfrontal negativity with memory load covaried directly with

w xmeasures of speech rate 38 . The functional significanceof the posterior positivity is less clear. Since it started priorto the left anterior negativity and terminated during thedelay interval, it is not likely to be an index of therehearsal process. Rather, it may index either processesinvolved in initiating rehearsal or the operation of anadditional transient store. The posterior positivity wasfound only in verbal tasks, and not in visuo-spatial work-

w xing memory tasks 23,38,40,41 . Instead, during retentionof visuo-spatial material, there was negative slow waveactivity over posterior and central scalp, largest over theright hemisphere, that varied directly with visuo-spatialmemory load.

w xLang et al. 17 reported some ERP evidence of modal-ity-specific effects in verbal working memory. They foundthat stimulus modality had a significant effect upon ERPnegative slow wave activity during presentation and reten-tion of three digits in a short-term memory task. Forauditory stimuli, the negativity was larger over the frontalscalp, while for visual stimuli the negativity was largerover posterior temporal regions. The posterior positivitywas not delineated by Lang et al., possibly because theyutilized a relatively low and constant memory load. Lang

w xet al. 17 also found that modality-specific differences ofthe negative slow waves gradually diminished during the 3s retention interval. These findings are consistent with thepresence of modality-specific separate processing streamsin verbal working memory and suggest that the modality-specific working memory operations are transient.

The current experiment, while extending the findings ofw xthe Lang et al. 17 study, was designed to examine more

definitively and with more rigorous topographic analyses

( )D.S. Ruchkin et al.rCognitiÕe Brain Research 6 1997 95–113 97

whether verbal working memory has modality-specificprocessing streams. Hence, a delayed match-to-sample taskwas employed, with a modality contrast and a memoryload manipulation. Given the results of our previous verbal

w xworking memory studies 38–40 , we conjectured that theposterior positivity might provide additional informationwith respect to modality-specific processing streams. Thus,to insure that the posterior positivity would be clearlydelineated, relatively large memory loads were employed.Non-words, consisting of strings of consonant-vowel sylla-bles, were presented either aurally or visually, and memoryload was manipulated by varying the number of syllablesin the non-words.

The available evidence indicates that the slow negativi-ties index, at least in part, working memory operations.However, an alternative interpretation is that the slownegativities only index general preparation for the re-sponse to S2 and are not specific to memory tasksw x20,32,33,47 . To rule out this latter possibility, a controldetection task was developed in which subjects searchedthe non-words for repeated syllables; the outcome of thesearch was reported at the end of the delay interval. Bothmemory and detect tasks involved preparation for an ef-fortful response to S2, with the amount of effort varyingdirectly with the number of syllables in the S1 non-word.However, in the detect task, memory demand was heldvery low during the delay interval, so that indices ofworking memory retention operations presumably madelittle contribution to the detect task slow wave activity.Thus, the contrast between memory and detect ERPs pro-vided a means of delineating ERP indices of workingmemory retention operations. Due to the length of theexperiment, it was feasible to run the detect task in onlyone modality, although we did conduct a single subjectpilot experiment in which the detect task was run in bothmodalities. In the main experiment, the auditory modalitywas employed in the detect task.

2. Methods

2.1. Subjects

Eighteen volunteers with English as their first languagewere tested for their ability to perform the verbal memorytasks. Fifteen were accepted into the study. The data fromtwo subjects were not used due to the subjects’ failure tocomply fully with instructions 1. The average age of the

1 To minimize activation of long-term memory, non-words were em-ployed, and subjects were instructed to use only inner-speech rehearsal ofthe sounds of the non-words in the memory task. Subjects were furtherinstructed not to make a conscious effort to retain non-words in thecontrol detection task. Despite these instructions, one subject attempted toimprove performance in the memory task by forming associations be-tween the names of acquaintances and the non-words, and a secondsubject used inner-speech rehearsal during the delay interval in approxi-mately 50% of the control detection trials.

Ž . Žremaining 13 subjects 7 females was 29 years range,.22–44 years . Their average amount of education was 17.7

Ž .years range, 16–19 years . All had normal or corrected-to-normal vision and their average laterality quotient on

w x Žthe Edinburgh handedness inventory 25 was 0.77 range,0.0–1.0; q1.0 corresponds to maximally right-handed;

.y1.0 corresponds to maximally left-handed . Informedconsent was obtained in accord with the procedures of theU.S.P.H.S. Subjects were paid US $15.00rh.

2.2. Experimental design and procedure

Subjects were presented with non-words consisting ofsequences of spoken or written consonant-vowel syllables,presented at 400 ms intervals. There were two types oftasks, memorization and detection. Prior to each trial, adisplay indicated the task to be performed. A trial began

Ž .with the presentation of a non-word S1 , followed by a3700 ms delay interval, after which there was a visual

Ž .probe stimulus S2 which tested task performance.

2.2.1. Memory taskŽ .In the memory task either auditory or visual modality ,

subjects attempted to retain the S1 non-word in workingmemory over the delay interval. Information load wasvaried between two levels by varying the number ofsyllables in the non-word. For low-load trials, the non-words consisted of three syllables. The numbers of sylla-bles in the high-load trials were adjusted for each subjectsuch that high-load memory-task error rates were approxi-mately matched for auditory and visual modalities. Foreight subjects, five syllables were employed in both audi-tory and visual high-load trials. For four subjects, fivesyllables were employed in the high-load auditory trials,and four syllables were employed in the high-load visualtrials. For one subject, four syllables were employed inboth auditory and visual high-load trials. Working memorywas tested with a visual S2 probe consisting of one of thesyllables from the non-word, and the subject’s task was torespond verbally with the next syllable in the non-word.

2.2.2. Detect taskThe contribution of anticipation of S2 to the slow

negativities was evaluated by contrasting the results of theauditory detect and memory tasks. The primary differencebetween the detect and memory tasks was in the memoryretention effort during the delay interval, with a substantialmemory load that varied with S1 non-word length in thememory task, and a negligible, invariant memory loadŽ .remember ‘‘yes’’ or ‘‘no’’ in the detect task. However,the detect and memory tasks were similar at the S2 probe,which required substantial processing effort that varieddirectly with S1 non-word length in both tasks. Thus, thecontrast between the memory and detect tasks’ slow nega-tivities during the delay interval reveals indices of memoryretention operations in the delay interval, since memory


load was different but the anticipated processing of S2 wascomparable in the two tasks.

In the auditory detect task, subjects searched the S1non-word for repeated syllables. As in the memory task,there were two load levels, with the non-word consistingof three syllables in the low-load condition. The number ofsyllables for the high-load condition corresponded to thenumber of syllables employed in the memory task high-loadtrials. The S2 probe display consisted of the words ‘‘yes’’and ‘‘no’’, with each word followed by a combination of a

Ž .letter and a number ‘‘1’’ for low load, ‘‘3’’ for high load .Ž .Subjects responded either yes there had been a repetition

Ž .or no there had not been a repetition by speaking theletter whose position in the alphabet followed the letterassociated with ‘‘yes’’ or ‘‘no’’ by the amount of thenumber. Thus, on low-load trials, subjects responded withthe next letter in the alphabet, while on high-load trialsthey responded with the letter that was three places beyond

Žthe displayed letter e.g., on a high-load trial, if the.displayed letter was ‘‘g’’, the correct response was ‘‘j’’ .

Consequently, the control detect task resembled the mem-ory task in that the same S1 was employed in the twotasks. Furthermore, anticipation of S2 was comparable inthe two tasks since the response to S2 in the detect taskhad some of the properties of the response in the memory

Ž .task: 1 the response to the probe could not be determinedŽ .until onset of the probe display; 2 determination of the

Ž .response required effortful processing; 3 the requiredeffort varied directly with the S1 non-word length.

2.2.3. Stimulus materialsOne hundred and twenty 3-syllable, 4-syllable and 5-

syllable non-words were generated by randomly combin-ing syllables from a set of 29 consonant-vowel syllablesselected for their differences from real English words 2.No syllable was repeated in a non-word. The 29 conso-nant-vowel syllables, spoken by a male voice, were digi-tized at a 20 kHz sampling rate, edited and combined into

Žnon-words in accordance with the computer-generated.lists . Fifty non-words in each of the sets were reserved for

the experiment. The remaining non-words were used inpreliminary sessions for practice and for determining num-bers of syllables for the high-load trials.

Each non-word was used once in a memory trial andonce in a detection trial. The memory trial probe syllableswere randomly selected from all but the last syllable of anon-word. Ten percent of the detection non-words wereconverted to repetitions by randomly selecting one syllableand replacing the adjacent syllable with it. The letters usedin the detection trial probes were randomly selected from

2 The 29 syllables used to generate the non-words were as follows: BACO DA FA FU GA JA JI JO KA KI LA NA PO PU RA SA VA VO VUWA ZA ZI ZO DE GE RE VE ZE. Examples of 3-, 4- and 5-syllablenon-words are as follows: REKACO DEGAPU VEVUZIDA ZEVU-KICO PUCOVOLAZI FAVUJOJIPO.

the portion of the alphabet starting with ‘‘d’’ and endingwith ‘‘w’’.

Onsets of the last syllable of the 3-, 4- and 5-syllablenon-words were kept the same by delaying onsets of the 3-and 4-syllable non-words. For the 3-syllable non-words,

Žnon-verbal filler stimuli slanted lines for visual trials;.humming sounds for auditory trials were presented in the

time slots corresponding to the first two syllables of the5-syllable non-word.

2.3. General procedures

Subjects were seated in a dimly lit, electrically shieldedŽ .sound-attenuating chamber. Auditory stimuli 50 dB SL

were presented by a loudspeaker located directly above thesubject’s head. The visual stimuli were white letters on adark grey background displayed on a cathode ray tubemonitor located 140 cm in front of the subject. Intensitywas adjusted so that the image could be clearly distin-

w xguished with minimal after-image effects 6 . Each syllablewas 0.38 high and 0.448 wide. The visual non-words werepresented in upper-case, with a 350 ms duration for eachsyllable. The probe display consisted of lower-case charac-ters.

For terminological convenience, the time correspondingto the onset of the first syllable of the 5-syllable non-wordswill be referred as the start of the stimulus presentationŽ .S1 epoch. A trial began when the subject moved the rightindex finger into the path of an optical sensing switchbeam, at which time the pre-trial display terminated. TheS1 epoch started after a 1.0 s delay. Subjects responded

Ž .vocally to the probe S2 , which started 5700 ms after theŽ .onset of the S1 epoch. Their reaction time RT from the

onset of the S2 display was measured by a voice onsettime switch connected to the microphone. The S2 displayterminated with the subject’s response. Across subjects,the average of the median inter-trial interval was 16.6 s.

Subjects were instructed that performance accuracy wasmore important than response speed. They were told torehearse the sounds of the non-words, regardless of pre-sentation modality, using silent articulation with no overtvocalization movements. They were further instructed notto use visual imagery as a memory aid.

The six combinations of task and load were equiproba-ble and presented in a random sequence. There were 300

Žtrials in the experiment 50 trials for each combination of.task, modality and load . All subjects were presented the

same sequence of six blocks of 50 trials. Prior to the ERPrecording session, each subject participated in at least twopractice sessions, with a minimum of 330 practice trials.No feedback was given in either the practice or experimen-tal sessions.

2.4. Debriefing questionnaire

After completing the experiment, subjects responded toa debriefing questionnaire in which they described their


performance strategies. They also rated the overall diffi-culty of each combination of task, modality and memory

Ž .load on a scale of 1–7 and indicated when in a trial theyinitiated rehearsal operations.

2.5. Recording procedures

AgrAgCl electrodes were placed on the posterior-ante-Ž .rior midline Pz, Cz, Fz , and over left and right hemi-

Ž .spheres at: occipital scalp O1, O2 , inferior-posteriorŽ . Ž . 3cerebellar scalp Cb1, Cb2 , posterior temporal and

Ž .parietal scalp T5, P3, P4, T4 , temporal and central scalpŽ . Ž .T3, C3, C4, T4 , frontal scalp F7, F3, F4, F6 , prefrontal

Ž . Ž .scalp Fp1, Fp2 , 2 cm below the canthi E1, E2 , on theŽ .temporal-central midline 1 cm below the tragus A1, A2 .

The A1 electrode was the reference for the other 24electrodes. Impedances were below 2000 V . The ampli-fiers were set to a gain of 10 000, an upper cutoff fre-

Ž .quency y3 dB of 30 Hz and an AC coupling timeŽ .constant of 5.3 s y3 dB attenuation frequency: 0.03 Hz .

A digital computer controlled stimulus selection andtiming and measured RT. The electroencephalogram wasdigitized for 6000 ms, beginning 300 ms prior to the

Ž .beginning of the S1 stimulus sampling intervals20 msand stored in digital format for subsequent data analyses.The ERPs were digitally converted to linked A1 and A2reference wave forms.

Slow phasic cephalic skin potential artifacts were coun-tered by gently abrading the skin with a sterile lancet prior

w xto applying the electrode 30 . DC drift artifact was esti-mated and approximately removed by the following proce-

Ž .dure. a The DC potential in each channel was measuredbefore and after each block of 50 trials by a DC amplifierŽ .gains50 that operated in parallel with the AC-coupled

Ž .amplifier. b The slope of the drift artifact for eachchannel in a block was estimated by dividing the differ-ence between the pre- and post-block DC potentials by the

Ž .duration of the block. c For each channel, the estimatedslope for the block was removed from the wave forms for

w xall trials in the respective block 14 . The order of magni-tude of the drift slope was about y1.5 mVrs, referred tothe A1 reference electrode.

The data were smoothed by a zero phase shift low-passdigital filter with a y3 dB frequency of 4.4 Hz using

w xprocedures described previously 36 . A calibrated ampli-fier ground was used for the conversion of the averageERPs obtained with the 5.3 s time constant to wave formsapproximating those that would have been obtained by DC

w xrecording 9,11,37 .Subjects were not required to make special efforts to

3 In order to reduce the presence of electromyographic artifacts and thepossibility of electrocardiographic artifacts in the recordings from Cb1and Cb2, the electrodes were placed 1.2 cm above the standard interna-tional 10–20 system Cb1 and Cb2 locations.

suppress blinks and eye movements. Eye movement arti-facts were removed via a spatio-temporal dipole modeling

w x w xprocedure 4,5,18,19 , implemented in Scherg’s 43 brainŽ .source localization program BESA . Empirically deter-

mined eye movement source components were used toestimate and remove the contribution of the electro-oculo-gram to the averaged ERPs, separately for each of the sixexperimental conditions for each subject 4. As a conserva-tive measure, we excluded trials with very large andprolonged eye movement activity from the average ERPs

Žas well as trials with artifactually high amplitudes ;5–.15% of the total trials for a subject .

2.6. Data analysis procedures

For each subject, average ERPs for correct responsetrials were computed at all electrode sites for each combi-nation of modality, task and load. In order that the averageERPs for the two tasks were based upon S1 non-wordswith the same properties, only non-repetition trials wereused for the detect task. The utilization of BESA toremove electro-oculographic artifacts entailed a reductionof the time range of the analysis interval to 270 ms prior tothe start of S1 to 3640 ms after S1 termination, and anincrease in the sampling interval to 30 ms.

2.6.1. Amplitude measuresThe average amplitude over the original 300 ms pre-

stimulus epoch was used as the baseline for all amplitudemeasurements. To quantify the slow wave brain processesduring presentation of the last syllable of the S1 non-wordand in the delay interval, average amplitudes were com-puted over eight 400 ms windows centered at latencies of1800, 2500, 3000, 3500, 4000, 4500, 5000, and 5440 msafter the start of S1. To ensure that the amplitudes fromeach subject contributed equally to the averages and statis-tical tests, the amplitude measurements were scaled suchthat the average absolute amplitude over all latency win-dows, electrodes and experimental conditions for eachsubject equaled the across-subjects grand mean averageabsolute amplitude.

4 Approximate removal of EOG artifact was implemented by using asurrogate source analysis to simultaneously model ERP and EOG activityw x19 . EOG surrogate sources were estimated from the topographies ofcalibration data obtained during the subject’s performance of saccadeŽ .208 and blink maneuvers. Blinks and horizontal and vertical saccadeEOG activity were each modeled by different combinations of surrogatesources. The ERPs were modeled by regional source dipoles which fitted

Žthe ERP activity but were not resolved into source dipoles corresponding.to ‘‘actual’’ neural generators . The EOG-corrected ERP wave forms

were obtained by removing from the model the activity due to the EOGsources. This method has a theoretical advantage over eye movementartifact correction by regression, since it does not entail the assumptionthat recordings from eye movement monitor channels reflect EOG activ-

w xity alone 7 .


2.6.2. Statistical analysesRepeated-measures global ANOVAs tested the effects

Žof modality, load and their interaction with electrode 23.channels in each of the eight measurement intervals for

the memory task. There was no a priori reason to expectthat ERP recordings from all of the electrodes would besensitive to significant changes in brain electric activity asexperimental conditions were manipulated. The sites of theelectrodes were selected to give as broad coverage over thehead as possible, thereby facilitating the use of currentsource density maps, and source modeling in the removalof eye artifacts. Thus, 23-channel tests could be prone todiluting the statistical significance of ‘‘real’’ experimentaleffects. Hence, a significance level of 0.10 was the crite-rion for justifying performance of further post-hoc testsrestricted to electrode subsets that, on the basis of previousstudies, were expected to be sensitive to experimentaleffects. A probability level of 0.05 was the criterion forsignificance of the post-hoc tests. Corrected degrees offreedom were obtained by multiplying the original degreesof freedom by the Greenhouse–Geisser epsilon and trun-cating the result to an integer.

2.6.3. Topographic profile analysisTopographic profile analyses were used to determine

whether amplitude measurements, obtained in differentexperimental conditions, reflected the activity of more thanone combination of neural generators. It is assumed thatERP activity recorded on the scalp is due to a combinationof neural sources located in different brain regions androrwith different orientations. If, in different experimentalconditions, the relative strengths of activation of the vari-ous brain sources are the same, then the correspondingshapes of scalp topographies will be the same. Conversely,if the shapes of the scalp topographies are different indifferent experimental conditions, then the relativestrengths of activation of the various brain sources con-tributing to the scalp ERP activity must also be different.

If the differences in shapes of scalp topographies aresystematic, then the ERP amplitudes should have a signifi-cant interaction between scalp location and experimentalmanipulation. However, in some instances it is possible fora difference in amplitude alone, with no change in topo-graphic shape, to also produce a significant interaction

w xbetween scalp location and experimental manipulation 21 .Thus, to ensure that the topographic comparisons wereconfined to shapes alone, the data were first scaled to

w xremove the effects of amplitude differences 41 .

2.6.4. Current source densityIn order to obtain reference-free images of scalp ERP

activity with increased spatial resolution, approximationsŽ .of scalp current source density CSD were computed from

the across-subjects averaged ERP wave forms. The spheri-w xcal spline method of Perrin et al. 28 was used to estimate

the CSD maps.

3. Results

3.1. BehaÕioral performance, ratings of difficulty and per-formance strategies

Ž .In the memory task, error rates Table 1 were compara-Ž . Žble for the two modalities F s0.54 , while RTs Table1, 12

. Ž2 tended to be longer for the auditory modality F s1, 12.4.69, Ps0.051 . Error rates increased with load in both

Žmemory and detect tasks memory: F s115.48, P-1, 12.0.00001; detect: F s4.76, P-0.05 . RTs also in-1, 12

Žcreased with load in both tasks memory: F s42.65,1, 12.Ps0.00003; detect: F s42.27, Ps0.00003 , and1, 12

Žwere longer in the detect task F s79.48, Ps1, 12.0.00001 .

Ž .Difficulty ratings on a 1–7 scale indicated that sub-jects found both the auditory and visual memory tasks

Ž .more difficult at high loads 4.9 and 4.8, respectively thanŽ .at low loads 1.4 . Taken together, the error rates and

subject ratings indicated that the difficulty of the memorytasks was essentially the same for both modalities. For thedetect task, the low- and high-load difficulty ratings were1.3 and 2.6, respectively, indicating that, for low loads, thedetect task difficulty was comparable to that in the mem-ory task. At high loads, the greater overall difficulty of thememory task probably reflected processing differencesduring the delay interval, since load was substantiallyhigher in the memory task in that interval. The substan-tially longer RTs in the detect task were an indication thatprocessing of the S2 probe required more effort than in thememory task. Thus, the detect data should provide asensitive measure of ERP activity associated with prepara-tion in these conditions.

All subjects used inner-speech to rehearse the non-words. In addition to inner-speech rehearsal, four subjects

Žoccasionally also employed visual imagery less than 4%.of the trials during the delay interval. One subject made

Žextensive use of visual imagery 50% of the auditory trials,.75% of the visual trials . She both visualized the characters

and rehearsed the sound of the non-word.

3.2. Memory task ERP waÕe forms

For the memory task, the effects of auditory and visualpresentation upon brain activation are illustrated by theacross-subject average ERP wave forms presented in Fig. 1and Fig. 2. During the S1 interval, the ERPs displayed aseries of phasic deflections, synchronized with the pre-

Table 1Ž .Error rates percentage

Low load High load

Visual memory 0.9 14.8Auditory memory 0.0 17.9Auditory detect 2.6 5.3


Table 2Ž .Median RTs ms for correct response trials

Low load High load

Visual memory 1279 1787Auditory memory 1303 2019Auditory detect 2113 3099

sentation of fillers and non-word syllables, that indexedinitial processing of the stimuli. For both modalities therewas frontal negative slow wave activity, largest over theleft hemisphere, that started during S1 for auditory stimuliand after S1 termination for visual stimuli and persisteduntil the end of the delay interval. The across subjectsaverage amplitudes in the 4000 ms and 5440 ms latency

Ž .windows Figs. 3 and 4 indicate that the left frontalnegativity in the delay interval generally increased alongwith memory load. The timing of the left frontal negativityŽlargest late in the delay interval, when retention-rehearsal

.operations were most likely to be predominant , scalp

Žtopography maximal in the vicinity of brain regions in-w x.volved in articulatory rehearsal 49 and sensitivity to

memory load suggest that the left frontal negativity in-dexed retention-rehearsal operations.

w xConsistent with Penney’s 27 model, there were modal-ity differences in slow wave activity during the delayinterval at both low- and high-memory loads. There were

w xamplitude differences since, as in the Lang et al. 17study, the frontal negativity was larger for auditory stimuli.There also were timing differences since for auditorystimuli onset of the frontal negativity was in the S1interval, shortly after presentation of the first syllable,while for visual stimuli onset was later, during the delayinterval. The earlier onset for auditory stimuli presumablyreflected the direct access of auditory information to thephonological store and hence earlier onset of retention-re-hearsal operations for aurally presented verbal information.In addition, for visual stimuli, there was marked positiveslow wave activity over the centro-parietal scalp that startedduring S1 and persisted for approximately 2000–2500 ms

Fig. 1. Across-subject average ERPs at all electrode sites for both auditory and visual modalities in the low-load memory tasks. In this and subsequentfigures, the vertical lines indicate S1 onset, the onset of the second filler, onset of the first syllable of the low-load non-word, and S1 offset, respectivelyŽ .S1 interval durations2000 ms . The vertical level of the time axis for each recording site corresponds to the average amplitude in the 300 ms pre-S1

Ž .interval, and wave forms are plotted with negative amplitude up with respect to a digitally linked A1 and A2 reference. The time base extends from 270ms prior to S1 onset to 5640 ms after S1 onset. The wave form layout is in rough correspondence with the layout of the electrodes on the head. The top ofthe figure corresponds to the front of the head and the columns to the right of midline correspond to the right hemisphere. The E1 and E2 sites were located2 cm below the outer canthi of the eyes.


after termination of S1. In contrast, for auditory stimuli,there was mostly negative slow wave activity over thecentro-parietal scalp during the interval corresponding towhen the slow posterior positivity was active in the visual

Ž .modality. The persistence i.e., after termination of S1 andmodality specificity of the posterior positivity suggest thatit indexed transient visual verbal memory operations.

3.3. ERP amplitude measures and CSD maps in the mem-ory task

Modality differences in the scalp distributions of theamplitude measurements provided quantitative support for

w xPenney’s 27 separate processing streams model. TheŽ . Ž .amplitudes in the 2500 ms early , 4000 ms middle and

Ž . Ž .5440 ms late latency windows Figs. 3 and 4 and theŽ .CSD maps Fig. 5 indicate that the pattern of brain

activation during the delay interval was different afterhearing and reading the non-words. The data in Figs. 3–5show that the long duration frontal negativity, the putativeindex of rehearsal operations, was focused over left frontal

Žscalp, emerging earlier for auditory stimuli during the S1. Žinterval than for visual stimuli after the 2500 ms latency.window . While the data in Figs. 3–5 show that the

topography of the frontal negativity tended to become

more alike for the two modalities as time progressed,nevertheless the amplitudes and maps for the 5440 mslatency window indicate that, even at the end of the delayinterval, the auditory frontal negativity was larger andsomewhat more bilateral than the visual frontal negativity.

Figs. 3–5 also show marked differences between audi-tory and visual slow wave topography over the posteriorscalp, starting during the S1 interval and extending forabout 2000–2500 ms after termination of S1. For visualstimuli, there was posterior slow wave activity, a putativeindex of transient visual verbal short-term memory opera-tions that consisted of positivity at centro-parietal sitesŽ .e.g., Cz, Pz and negativity at posterior temporal sitesŽ . ŽT5, T6 Fig. 4 – 2500 and 4000 ms panels, Fig. 5 – 1800

.ms to 4500 ms maps . The very distinct positive andnegative foci of the posterior visual CSD suggest that thevisual posterior positivity was due to neural generators inbilateral temporal-parietal cortex, with negative and posi-tive poles oriented towards lateral and central regions,respectively. In contrast, for auditory stimuli, with theexception of some low-amplitude positivity at occipitaland parietal sites in the beginning of the delay interval, the

Ž .amplitudes Fig. 3 were negative over centro-posteriorŽ .scalp, and the auditory CSD maps Fig. 5 indicate a more

diffuse pattern of CSD over posterior scalp compared with

Fig. 2. Across-subject average ERPs at all electrode sites for both auditory and visual modalities in the high-load memory tasks.


the visual CSDs. The auditory CSD displayed little or noŽ .negativity over left posterior temporal scalp T5 and less

pronounced negativity than the visual CSD over rightŽ .posterior temporal scalp T6 . Taken together, the visual

and auditory topographies suggest that there were transientvisual verbal short-term memory operations in the vicinityof temporal-parietal cortex in both hemispheres.

Ž .The results of global 23-electrode ANOVAs on theŽ .amplitudes in the eight latency windows Table 3 and

Ž .profile comparisons Table 4 provide statistical supportw xfor Penney’s 27 separate processing streams model. The

ANOVAs indicated that the amplitudes in all measurementintervals were systematically influenced by whether sub-jects heard or read the non-words. The interactions ofmemory load with electrode location further indicated that

Žamplitudes in the later measurement intervals i.e., after.2500 ms varied with the number of syllables in the

non-word, and that the load effects differed over electrodesites. As the number of syllables increased, amplitudes

became more negative at frontal sites and more positive atŽ .posterior sites Figs. 3 and 4 . The global profile compar-

Ž .isons Table 4 indicated that the differences between theŽ .scalp distributions of slow wave amplitudes Figs. 3 and 4

for auditory and visual presentation of the non-words weresystematic, and that the differences were more pronouncedin the earlier latency windows. This means that differentpatterns of brain activation were operative during retentionof non-words that were heard or read.

3.4. Changes oÕer time

w xThe ERP data are also consistent with Penney’s 27view that the auditory verbal short-term memory is moredurable than the visual verbal short-term memory. Topo-graphic profile comparisons across the 2500, 4000 and

Ž .5440 ms latency windows Table 5 indicated that changesin slow wave topography over time at posterior sites weresignificant for both modalities, but the changes were more

Fig. 3. Amplitude measures in the delay interval for both low and high loads in the auditory memory task, averaged across subjects. Note that in order tomake clear the effect of load upon the ERPs, in contrast with Fig. 1 and Fig. 2, the amplitudes in this figure and in Fig. 4 are plotted with low- andhigh-load conditions overlaid. In this and subsequent figures the amplitude measurements were obtained by computing the average amplitude over 400 ms

Ž . Ž . Ž .wide latency windows centered at 2500 ms top panel , 4000 ms middle panel , and 5440 ms bottom panel . The leftmost column provides a sagittal viewalong the midline. The five columns to the right provide coronal views at five positions along the posterior-to-anterior axis.


Fig. 4. Amplitude measures in the delay interval for both low and high loads in the visual memory task, averaged across subjects, for the latency windowsŽ . Ž . Ž .centered at 2500 ms top panel , 4000 ms middle panel , and 5440 ms bottom panel .

Table 3Memory task – auditory and visual modality ERPs – global ANOVAs. All electrode sites

Time window 1800 2500 3000 3500 4000 4500 5000 5440

e 0.115 0.129 0.121 0.117 0.117 0.126 0.127 0.129

LoadF 0.12 2.80 2.39 1.87 0.76 0.76 0.28 0.19P

L=elecF 1.04 2.34 3.27 4.12 5.66 5.11 4.83 4.24P 0.051 0.026 0.0082 0.012 0.015 0.023

ModalityF 31.20 36.28 27.95 23.61 25.65 24.36 24.56 17.86P 0.00012 0.00006 0.00019 0.00039 0.00028 0.00034 0.00033 0.0012M=elecF 18.58 24.51 20.68 21.48 20.13 17.35 11.87 8.61P 0.00001 -0.00001 -0.00001 -0.00001 -0.00001 0.00001 0.00013 0.0010

Degrees of freedom: main – 1, 12; interaction – 22, 264.In this and subsequent tables, the latencies at the top of each column indicate the centers of the 400 ms latency windows over which amplitudes weremeasured.


Fig. 5. Estimated across-subjects averaged current source density maps in the memory task for the four combinations of modality and load, derived fromŽ .the average amplitudes over the last syllable of the non-word 1800 ms and the seven 400 ms wide latency windows in the delay interval. The shaded

regions of the maps indicate positive current densities, and the unshaded regions indicate negative current densities. The difference between contourintervals corresponds to a density increment of 1 mVrcm2. Electrode positions are indicated by the dots. The maps are 908 projections, with the electrodes

Ž .on the circumference corresponding in clockwise order, starting at the right of the nose to Fp2, F8, T4, T6, O2, O1, T5, T3, F7, Fp1.

marked for the visual modality. At frontal sites the visualslow wave topography varied significantly over time, butthe auditory slow wave topography was essentially invari-

Ž .ant Table 5 . Thus, the trend towards greater similaritybetween the auditory and visual brain activation patterns astime progressed was due to visual stimuli activation ap-

Table 4Memory task – global profile comparisons of auditory and visual modality ERPs. All electrode sites

Time window 1800 2500 3000 3500 4000 4500 5000 5440

e 0.109 0.126 0.120 0.116 0.112 0.122 0.123 0.126F 13.03 17.16 21.65 21.82 16.41 10.70 5.78 4.98P 0.00010 0.00001 -0.00001 -0.00001 0.00002 0.00028 0.0072 0.013

Degrees of freedom: interaction – 22, 264.

Table 5Memory task – profile comparisons across the 2500 ms, 4000 ms and 5440 ms latency windows, separately for each modality. For all electrodes, posteriorelectrodes, and frontal electrodes

All electrodes T5 P3 Pz P4 T6 F7 F3 Fz F4 F8

Auditory Visual Auditory Visual Auditory Visual

df 44, 528 44, 528 8, 96 8, 96 8, 96 8, 96e elec 0.130 0.120 0.492 0.459 0.519 0.606e time 0.684 0.866 0.912 0.831 0.554 0.729F 5.17 10.28 6.16 24.32 0.28 10.92P 0.0037 -0.00001 0.0014 -0.00001 0.76 0.00002


Table 6Memory task – profile comparisons of auditory and visual modality ERPs. Anterior electrodes: F7 F3 Fz F4 F8

Time window 1800 2500 3000 3500 4000 4500 5000 5440

e 0.525 0.537 0.560 0.586 0.602 0.581 0.580 0.561F 14.65 15.19 11.41 11.23 10.06 5.94 3.82 3.72P 0.00006 0.00005 0.00028 0.00026 0.00051 0.0073 0.035 0.038


proaching the pattern of activation that emerged early in atrial for the auditory stimuli.

Although the auditory and visual frontal negative slowwave topographies tended to become more alike with the

Ž .progression of time Figs. 3–5 , profile comparisons re-stricted to the frontal sites extending laterally from F7 toF8 indicated that the frontal topographies for the twopresentation modalities were still significantly different at

Ž .the end of the delay interval Table 6 . In contrast, profilecomparisons, restricted to the parietal-temporal sites ex-tending laterally from T5 to T6, indicated that although thedifference between the posterior topographies for the twomodalities were statistically significant in the first seven

Ž .measurement intervals i.e., prior to 5200 ms , the differ-ence was not significant in the last measurement intervalŽ .Table 7 . Thus the most durable and reliable differencesin brain activation as a function of modality appear to belocated in frontal regions.

3.5. Distinctions between amodal and specific auditoryprocessing streams

While at the end of the delay interval the slow wavetopographies were similar over frontal sites for the two

Žmodalities maximally negative at F3 and least negative at.F8 , nevertheless the amplitudes were about 3–4 mV higher

Žat all frontal sites in the auditory task Figs. 3 and 4,.bottom panels . Coupled with the differences in scalp

distribution, these amplitude differences suggest that thefrontal negativity may have been a composite of two

Ž .components: 1 a component that was lateralized to theŽ .left hemisphere maximal at F3 that was common to

Ž .auditory and visual modalities; 2 a component with abilateral distribution that was most pronounced for theauditory modality. The left-lateralized component may in-dex amodal rehearsal operations, while the bilateral com-

ponent may index retention operations associated with theauditory processing stream.

3.6. Load effects

3.6.1. Left anterior negatiÕityDuring the latter part of the delay interval, the ampli-

tudes of the negativity over left anterior scalp were largerŽfor high-memory loads Figs. 3 and 4, middle and bottom

.panels . An ANOVA on the amplitudes at left anteriorŽ .sites F7, F3, E1, Fp1 in the 3800–5640 ms interval,

pooled over auditory and visual modalities, indicated thatthe increased negativity as memory load increased was

Ž .significant F s6.34, Ps0.027 . The direct relation-1, 12

ship between amplitude and memory load was significantŽ .for auditory trials F s6.16, Ps0.029 and marginally1, 12

Ž .significant for visual trials F s3.51, Ps0.085 .1, 12

3.6.2. Posterior positiÕityDuring the delay interval, the amplitudes of the slow

waves at centro-posterior sites were more positive forhigh-memory loads. For auditory presentation, the shift

Žtowards positivity as load increased superimposed upon.negative slow wave activity was largest in the 2300–5200

Žms interval at parietal sites P3, Pz, P4; F s4.97,1, 12.Ps0.046 . For visual presentation, the increment in the

posterior positivity as load increased had a shorter durationŽ .largest in the 2300–3700 ms interval and included both

Žright central and parietal sites C4, P3, Pz, P4; F s6.28,1, 12.Ps0.028 .

w xAs in our prior studies 38,40 , more than one brainprocess contributed to the posterior slow waves in thevisual modality, since the topography of the posterior high-minus low-load amplitude increment differed from thetopography of the unsubtracted posterior slow wave ampli-

Table 7Memory task – profile comparisons of auditory and visual modality ERPs. Posterior electrodes: T5 P3 Pz P4 T6

Time window 1800 2500 3000 3500 4000 4500 5000 5440

e 0.537 0.535 0.536 0.572 0.555 0.530 0.465 0.470F 4.81 4.06 5.25 4.19 5.02 8.69 7.47 3.41P 0.017 0.030 0.013 0.026 0.014 0.0014 0.012 0.078



tudes. The unsubtracted amplitudes of the posterior slowwave were positive over centro-parietal scalp and negativeover posterior temporal scalp, while the high- minus low-

Žload difference amplitudes were positive at all sites Fig. 4,.top and middle panels . A profile comparison between the

low-memory load and difference amplitudes in the 2300–3700 ms interval over T5, P3, Pz, P4, T6, C4 indicated

Žthat this topographic difference was significant F s5, 60.4.97, es0.483, Ps0.038 .

3.7. Detect task – indices of anticipatory actiÕity

The detect task was used to evaluate the relative contri-butions of memory operations and anticipation of S2 to theslow negativities in the delay interval. Subjects retained asubstantial amount of information in the memory task, asopposed to remembering only the yesrno decision in thedetect task. If the slow negativities indexed, at least in part,memory operations, then they should be larger in thememory task. In contrast, if the slow negativities onlyindexed anticipation of S2, then they should be about thesame magnitude in both tasks. Across-subject average ERPwave forms for the detect task are presented in Fig. 6, anda contrast between ERPs for high-load detect and auditory

memory trials is presented in Fig. 7. The detect taskamplitudes in the 2500 ms, 4000 ms and 5440 ms latencywindows are presented in Fig. 8. At posterior sites, theslow negativities in the delay interval were similar in thetwo tasks. The key differences between detect and memoryslow negativities were at central and frontal sites, indicat-ing that memory operations were reflected in the frontalnegativities.

During the S1 interval, subjects encoded, retained andsearched the non-words for syllable repetitions in thedetect task, as opposed to only encoding and retaining thenon-words in memory trials. In the S1 interval, the centro-frontal negativity was larger for the detect task. The detecttask centro-frontal negativity was maximal near the termi-nation of the S1 interval and decreased over the delayinterval. In contrast, for memory trials, the negativity at allsites increased as time progressed, with its amplitude beinglargest at the end of the delay interval, such that during thelast 3000 ms of the delay interval the amplitude of the

Ž .negativity over left anterior scalp F7, F3, E1, Fp1 wasŽsignificantly larger in the memory task F s7.96, Ps1, 12

.0.015, see Fig. 7 . In further contrast with the memorytask, the global ANOVAs on the detect task amplitudesindicated that manipulation of information load had no

Fig. 6. Across-subject average ERPs at all electrode sites for both low and high loads in the detect task.


Fig. 7. Across-subject average ERPs at all electrode sites for auditory memory and detect tasks in the high-load condition.

systematic effects upon ERP amplitudes in any of the eightŽlatency windows P)0.47 for all main effects and load=

.electrode interactions .The frontal negativity in the memory task was lateral-

ized to the left hemisphere, with the lateralization beingŽ .more pronounced at high loads Fig. 3 , while the frontal

negativity in the detect task tended to be bilaterally sym-Ž .metric Fig. 8 . A profile comparison of the auditory

memory and detect amplitudes at F7, F3, F4, and F8 in the2200–5640 ms interval indicated that the difference in

Žlateralization at high loads was significant high and lowloads combined: F s3.52, es0.677, Ps0.046; high3, 36

load: F s6.41, es0.66, Ps0.019; low load: Ps3, 36.0.39 .

3.8. Pilot experiment – automatic Õerbal auditory memory

While it is clear that the left-lateralized component ofthe frontal negativity indexes memory operations, the func-tion indexed by the bilateral component of the frontalnegativity is not clear. The occurrence of bilateral frontalnegativity in the detect task possibly means that the bilat-eral component indexed anticipatory activity. However, in

w xview of the results from a prior study 39 and the pilotexperiment, an anticipatory interpretation is unlikely. The

contrast between the frontal negativity in the auditorydetect task of the current experiment and the negligiblefrontal negativity in the visual detect task of the earlier

w xRuchkin et al. 39 verbal working memory experimentsuggest that the bilateral frontal negativity may have in-dexed S1 processing. However, because the earlier experi-ment utilized a less demanding detect task S2 responsethan in the current experiment, it is not clear whether thedifference between frontal negativities in the two detecttasks was due to S2 response requirements or S1 modality.If the amplitude difference was due to S1 modality, then,since the bilateral frontal negativity was active long aftertermination of S1, in both the memory and detect tasks itmay have indexed the automatically maintained verbal

w xauditory memory postulated in the Penney 27 model.

Table 8Pilot experiment ERP amplitudes, in mV, at F3. Averaged over the 5440ms latency window

Modality Load Memory task Detect task

Auditory Low y6.59 y6.38High y9.63 y8.26

Visual Low y4.73 y1.09High y5.74 y0.13


Fig. 8. Amplitude measures in the delay interval for both low and high loads in the detect task, averaged across subjects, for the latency windows centeredŽ . Ž . Ž .at 2500 ms top panel , 4000 ms middle panel , and 5440 ms bottom panel .

The results of the pilot experiment, which employedboth auditory and visual detect tasks, suggested that S1modality in the detect task influenced the frontal negativitylong after termination of S1. ERP amplitudes obtained

Ž .from the left frontal site F3 at the end of the delayŽ .interval Table 8 replicated the modality, load and task

effects found in the main experiment. In addition, theamplitude of the frontal negativity in the visual detect taskwas markedly less than in the auditory detect task, eventhough the S2 task was the same for both modalities.

4. Discussion

The between-modality differences in ERP activationw xduring the delay interval support Penney’s 27 modality-

specific sensory processing ‘‘streams’’ in verbal workingmemory, while the similarities indicate common verbalworking memory processes in both modalities. There wasa temporal dimension to this pattern of results since thedegree of difference between auditory and visual ERPactivity in the delay interval diminished as time pro-

gressed, suggesting that modality-dependent processing ul-timately converges to an amodal process.

4.1. Amodal and auditory processing streams – frontalnegatiÕity

It appears that the frontal negativity indexed, at least inpart, retentionrrehearsal operations: its amplitude varieddirectly with memory load, and it persisted until the end of

Ž .the delay interval 3640 ms after termination of S1 . Thefrontal negativity displayed modality differences along thedimensions of timing, topography and amplitude. As in the

w xLang et al. study 17 , the amplitude of the frontal negativ-ity was larger for the auditory modality, and its onsetoccurred about the time of presentation of the first twosyllables of the non-word. In contrast, in the visual modal-ity, onset of the frontal negativity occurred about 500–1000ms after the end of the non-word presentation. The earlieronset of the frontal negativity in the auditory modality maybe a reflection of the direct access to the phonologicalstore that has been hypothesized for auditory verbal pre-sentation. In this framework, the delayed onset of the


visual frontal negativity may be due to visually presentedinformation undergoing re-coding to phonological formbefore entering the phonological store through articulatory

w xrehearsal 45 .The finding of a left frontal focus of slow wave activa-

tion late in the delay interval of both the visual andauditory memory tasks is consistent with the results of anumber of PET studies of verbal working memory thatfound increased rCBF in left frontal brain regionsw x1,2,10,13,22,26,44,46 . The persistence to the end of thedelay interval of the left frontal focus of negativity, whensubjects were engaged in rehearsal operations, supports

w x w xAwh et al.’s 2 and Paulesu et al.’s 26 supposition thatthe rCBF found in the vicinity of Broca’s area was associ-ated with rehearsal in preparation for output.

Although the differences between the auditory and vi-Žsual frontal topographies early in the delay interval Figs. 3

.and 4, top panels may be attributed to earlier onset ofretention-rehearsal operations for auditory presentation, asignificant modality difference between auditory and vi-sual frontal topographies remained even at the end of the

Ždelay interval 3240–3640 ms after S1, Figs. 3 and 4,.bottom panels . In this context, it is important to note that

subjects estimated that they began rehearsal operationswell before the middle of the delay interval for bothmodalities. Thus, although subjects were in the retention-rehearsal stage during the latter part of the delay intervalfor both modalities, their patterns of brain activation weredifferent for the two modalities.

Evidence for auditory-specific verbal working memoryoperations in the latter part of the delay interval can beseen in the frontal slow waves. While the frontal negativitywas concentrated over the left hemisphere for both modali-ties, it was larger and less lateralized for auditory stimuli.One interpretation is that the frontal negativity in theauditory modality consists of a focus of activation over the

Ž .left hemisphere present for both modalities superimposedŽupon bilateral activation only present for the auditory

.modality . Supporting this idea of two separate frontalprocesses during retention of aurally presented verbal in-formation, the amplitude of the bilateral activation ap-peared to be comparable at central and frontal sites, givingit a more posterior distribution than the left-lateralizedfrontal negativity. A possible role for the process indexedby this bilateral activity is discussed below.

4.2. Contribution of anticipation of S2 to the slow negatiÕi-ties

The differences between the frontal negativities in theauditory and visual memory tasks argue against an ‘‘onlyanticipation of S2’’ explanation. The S2-probe task wasthe same for the two modalities, and the error rates andRTs for the two modalities were similar. However, thefrontal negativities in the delay interval were clearly differ-

Žent for the two modalities i.e., no bilateral frontal negativ-

.ity for visual stimuli , indicating that the bilateral frontalnegativity indexed, at least in part, processing of S1 infor-mation.

The ERPs elicited in the detect task further indicatedthat the left-lateralized frontal negativity in the delay inter-val of the memory task was not likely to have arisen solely

Ž .from anticipatory processes: 1 as time progressed, thefrontal negativity increased in the memory task but de-

Ž .creased in the detect task; 2 the amplitude of the frontalnegativity was lateralized to the left in the memory task

Ž .and bilaterally symmetric in the detect task; 3 in thelatter part of the delay interval the amplitude of the frontalnegativity was significantly larger over the left hemisphere

Ž .in the memory task; 4 the amplitude of the frontalnegativity over the left hemisphere varied directly with S1information load in the memory task and was invariant inthe detect task.

Those aspects of the slow negativities that were similarŽ . Ž .1 in both tasks and 2 in both modalities of the memory

Ž .task i.e., did not vary with S1 were the most likely tohave been indices of anticipation of S2. The slow negativi-ties during the final 1500–2000 ms of the delay interval atparietal, posterior temporal, and occipital sites came clos-est to meeting this criterion. Since this activity did notinclude the frontal electrodes, it is reasonable to concludethat the frontal negativities indexed, at least in part, mem-ory operations. In view of its appearance in both auditorymemory and detect tasks, the bilateral frontal negativitymay have indexed an automatic memory process.

4.3. Automatic memory processes

w xPenney 27 argued that auditory verbal information isŽautomatically encoded into both a phonological code di-

.rect access to the phonological buffer and an auditoryŽ .code the basis for the auditory-specific processing stream .

In her model, if there is no subsequent input to over-writeit, the auditory code can be maintained without attentionŽ .i.e., automatically for long intervals. In this framework,the bilateral centro-frontal activation that is superimposedupon the lateralized frontal negativity for auditory stimulimay index processing that supports automatic maintenanceof an auditory code. If this is the case, then the bilateralfrontal negativity should appear only for auditory and notfor visual stimuli when there is no explicit memory re-quirement. Support for this explanation came from thepilot experiment. For both the detect and memory tasks inthe pilot experiment, the frontal slow negativity in thedelay interval was markedly less for visual than for audi-tory S1s. This was the case despite the fact that the S2probe stimulus and response did not vary with modalityand, in the detect task, the error rates and RTs wereessentially identical for both modalities. This result sug-gests that part of the negativity in the delay interval of theauditory detect task indexed processing of some form of‘‘memory’’ for S1, since the S1 modality effect was foundmore than 3000 ms after termination of S1.


w xStudies by Rohrbaugh et al. 34,35 have providedfurther indications of modality-dependent long durationnegative waves that reflect processing of a preceding S1rather than anticipation of an impending S2. Unpaired

Ž .stimuli e.g., acoustic tone burst, light flash elicited longŽ .duration 1500–2000 ms negative after-waves that were

generally larger for the auditory stimuli. While more dataare needed, the current data are consistent with the ideathat the bilateral centro-frontal negativity in both the audi-

w xtory memory and detect tasks relates to Penney’s 27postulated automatically maintained auditory code.

4.4. Visual processing stream – Õisual posterior positiÕity

The early onset and persistence of the visual posteriorŽpositivity from early in the S1 interval to about 2000–2500

.ms after the end of S1 suggest that it may index opera-tions that support the re-coding from visual to phonologi-cal format andror initiation of retention processes. The

w xresults of the current study and previous studies 23,38–41indicate that the posterior positivity elicited by visualstimuli is specific to verbal short-term memory for writtenmaterial. The negative posterior slow wave activity in theauditory memory task makes clear that the posterior posi-tivity indexed processing that was specific to the visualmodality. The specificity of the posterior positivity toworking memory has been demonstrated in a study show-ing that it appears in a working memory task but not in adetection task that used the same stimuli but with no

w xmemory requirement 39 . The nature of the processingindexed by the visual centro-posterior positivity has beenfurther delimited to a verbal visual store, since it was not

Ž .elicited: 1 during retention of two-dimensional patternsw x Ž .of randomly placed alphabetic characters 38,40 ; 2 dur-

ing retention of faces or movement of an object in two-di-w x Ž .mensional space 41 ; or 3 of shapes or locations ofw xgeometric figures 23 .

It has been hypothesized that the working memorysystem for visually presented verbal information involves ashort-term store that operates on information that hasundergone initial visual analysis but has not been phono-

Ž w xlogically recoded e.g., see Shallice and Vallar 45 , Fig.w x .1.1, or Vallar and Papagno 48 , Fig. 6.1 . In view of the

timing of the posterior positivity and the fact that itappears to be elicited exclusively by visually presentedverbal information, it is reasonable to propose that theposterior positivity indexes the operation of this hypothe-sized visual, non-phonological verbal store.

4.5. Comparison of PET and ERP results

The apparent conflict between the finding of modalitydifferences in the current ERP study and the absence of

w xmodality differences in the Schumacher et al. 44 PETstudy may be due to methodological differences and theway Schumacher et al. interpreted their results. The lack of

a rCBF concomitant of the ERP visual posterior positivityw xin the Schumacher et al. 44 study may have been due to

differences in memory load. Schumacher et al.’s memoryrequirement of three alphabetic letters, presented at a rate

Žof one letter per 3 s 7.5 times slower than in the current.study , may not have been enough load to strongly engage

the process indexed by the ERP visual posterior positivity.Thus, given the transient character of the ERP posteriorpositivity, after summation over the PET scan the putativerCBF concomitant of the ERP posterior positivity may nothave been reliably distinguishable from noise.

With respect to the postulated automatic auditory verbalstore, rCBF activation associated with such a store would

w xbe canceled in Schumacher et al.’s 44 memory minusw xcontrol subtraction. Schumacher et al.’s 44 interpretation

that the rCBF activity revealed in their auditory minusvisual scan subtractions was associated only with encodingis open to question, since the time course of the brainprocesses reflected by rCBF is not known. The rCBFrevealed by the auditory minus visual scan subtractionsmay have reflected both sustained automatic sensory mem-ory operations and phasic encoding operations. In thiscontext, ERP studies that probed automatic auditory sen-

Ž .sory memory i.e., the ERP mismatch negativity indicatedthat the memory operations occurred in primary auditory

w xcortex 15,16,24,31,42 , suggesting that encoding and con-comitant automatic sensory memory processes happen inthe same region. Analogously, the processing regions re-vealed by the PET auditory minus visual subtractionscould be involved in both encoding and automatic memoryoperations.

4.6. Conclusion

The results of the current experiment are consistent withthe concept of separate sensory-specific working memorystreams that converge after the auditory or visual informa-tion is converted to a phonological code. The ERP signs ofthe visual and auditory processing streams were, respec-tively, a phasic visual posterior positivity and a sustainedauditory bilateral centro-frontal negativity. The memorysystem common to both modalities was manifested by asustained left anterior negativity that developed early forauditory presentation and later for visual presentation.Given the left anterior negativity’s topography and tempo-ral coincidence with rehearsal, we assume that the com-mon memory system is involved in retention of a phono-logical code. The trend towards convergence was mani-fested by the decay of the visual centro-posterior positivityand the ultimate build-up of the left anterior negativity inthe visual condition. By the end of the recording epoch,the ERP difference between the two modalities reducedprimarily to the presence of the auditory bilateral centro-frontal negativity. We can only speculate about when andhow this residual difference might resolve. We hypothesizethat the auditory trace associated with the bilateral centro-


frontal negativity finally either decays or is over-written.Then only the phonologically based trace, common to bothmodalities, would remain.

Acknowledgements

We thank John Mertus of the Department of Cognitiveand Linguistic Science of Brown University for providingus with his Barus Laboratory Interactive Speech SystemŽ .BLISS software and assisting us in adapting it to anevent-related potential experiment framework. This workwas supported in part by NINDS grants to D.S. RuchkinŽ . Ž .NS11199 and W. Ritter NS30029 .

References

w x1 N.C. Andreasen, D.S. O’Leary, S. Arndt, T. Cizadlo, R. Hurtig, K.Rezai, G.L. Watkins, L.L.B. Ponto, R.D. Hichwa, Short-term andlong-term verbal memory: a positron emission tomography study,

Ž .Proc. Natl. Acad. Sci. USA 92 1995 5111–5115.w x2 E. Awh, J. Jonides, E.E. Smith, E.H. Schumacher, R.A. Koeppe, S.

Katz, Dissociation of storage and rehearsal in verbal working mem-ory: evidence from positron emission tomography, Psychol. Sci. 7Ž .1996 25–31.

w x3 A.D. Baddeley, Working Memory. Oxford Psychology Series No.11, Clarendon Press, Oxford, 1986, pp. 1–289.

w x4 P. Berg, M. Scherg, Dipole modelling of eye activity and itsapplications to the removal of eye artifacts from the EEG and MEG,

Ž .Clin. Phys. Physiol. Meas. 12A 1991 49–54.w x5 P. Berg, M. Scherg, Dipole models of eye movements and blinks,

Ž .Electroenceph. Clin. Neurophysiol. 79 1991 36–44.w x Ž .6 J.L. Brown, Afterimages, in: C.H. Graham Ed. , Vision and Visual

Perception, Wiley, New York, 1965, pp. 479–503.w x7 C.H.M. Brunia, J. Mocks, M.M.C.V. Berg-Lenssen, M. Coelho,

M.G.H. Coles, T. Elbert, T. Gasser, G. Gratton, E.C. Ifeachor, B.W.Jervis, W. Lutzenberger, L. Sroka, A.W.V. Blokland-Vogelesang,G.V. Driel, J.C. Woestenburg, P. Berg, W.C. McCallum, P.D. Tuan,P.V. Pocock, W.T. Roth, Correcting ocular artifacts in the EEG: a

Ž .comparison of several methods, J. Psychophysiol. 3 1989 1–50.w x8 N. Cowan, Attention and Memory, an Integrated Framework, Ox-

ford University Press, Oxford, 1995, pp. 1–321.w x9 T. Elbert, B. Rockstroh, Some remarks on the development of a

Ž .standardized time constant, Psychophysiology 17 1980 504–505.w x10 J.A. Fiez, E.A. Raife, D.A. Balota, J.P. Schwarz, M.E. Raichle, S.E.

Petersen, A positron emission tomography study of the short-termŽ .maintenance of verbal information, J. Neurosci. 16 1996 808–822.

w x11 A.W.K. Gaillard, R. Naatanen, Some baseline effects on the CNV,¨¨ ¨Ž .Biol. Psychol. 10 1980 31–39.

w x12 P.M. Grasby, C.D. Frith, K.J. Friston, C. Bench, R.S.J. Frackowiak,R.J. Dolan, Functional mapping of brain areas implicated in audi-

Ž .tory-verbal memory function, Brain 116 1993 1–20.w x13 P.M. Grasby, C.D. Frith, K.J. Friston, J. Simpson, P.C. Fletcher,

R.S.J. Frackowiak, R.J. Dolan, A graded task approach to thefunctional mapping of brain areas implicated in auditory-verbal

Ž .memory, Brain 117 1994 1271–1282.w x14 E. Hennighausen, M. Heil, F. Rosler, A correction method for DC¨

Ž .drift artifacts, Electroenceph. Clin. Neurophysiol. 86 1993 199–204.

w x15 D.C. Javitt, C.E. Schroeder, M. Steinschneider, J.C. Arezzo, H.G.Vaughan Jr., Demonstration of mismatch negativity in the monkey,

Ž .Electroenceph. Clin. Neurophysiol. 83 1992 87–90.

w x16 D.C. Javitt, M. Steinschneider, C.E. Schroeder, H.G. Vaughan Jr.,J.C. Arezzo, Detection of stimulus deviance within primate primaryauditory cortex: intracortical mechanisms of mismatch negativityŽ . Ž .MMN generation, Brain Res. 667 1994 192–200.

w x17 W. Lang, A. Starr, V. Lang, G. Lindinger, L. Deecke, Cortical DCpotential shifts accompanying auditory and visual short-term mem-

Ž .ory, Electroenceph. Clin. Neurophysiol. 82 1992 285–295.w x18 O.G. Lins, T.W. Picton, P. Berg, M. Scherg, Ocular artifacts in

recording EEGs and event-related potentials. I. Scalp topography,Ž .Brain Topogr. 6 1993 51–63.

w x19 O.G. Lins, T.W. Picton, P. Berg, M. Scherg, Ocular artifacts inrecording EEGs and event-related potentials. II. Source dipoles and

Ž .source components, Brain Topogr. 6 1993 65–78.w x20 W.C. McCallum, Potentials related to expectancy, preparation and

Ž .motor activity, in: T.W. Picton Ed. , Handbook of Electroen-cephalography and Clinical Neurophysiology, Revised Series, vol. 3:Human Event-Related Potentials, Elsevier, Amsterdam, 1988, pp.427–534.

w x21 G. McCarthy, C.C. Wood, Scalp distributions of event-related poten-tials: an ambiguity associated with analysis of variance models,

Ž .Electroenceph. Clin. Neurophysiol. 62 1985 203–208.w x22 P.K. McGuire, D.A. Silbersweig, R.M. Murray, A.S. David, R.S.J.

Frackowiak, C.D. Frith, Functional anatomy of inner speech andŽ .auditory verbal imagery, Psychol. Med. 26 1996 29–38.

w x23 A. Mecklinger, E. Pfeifer, Event-related potentials reveal topograph-ical and temporal distinct neuronal activation patterns for spatial and

Ž .object working memory, Cogn. Brain Res. 4 1996 211–224.w x24 R. Naatanen, Attention and Brain Function, Lawrence Erlbaum,¨¨ ¨

Hillsdale, NJ, 1992, pp. 1–494.w x25 R.C. Oldfield, The assessment and analysis of handedness: the

Ž .Edinburgh inventory, Neuropsychologia 9 1971 97–113.w x26 E. Paulesu, C.D. Frith, R.S.J. Frackowiak, The neural correlates of

Ž .the verbal component of working memory, Nature 362 1993342–345.

w x27 C.G. Penney, Modality effects and the structure of short-term verbalŽ .memory, Mem. Cogn. 17 1989 398–422.

w x28 F. Perrin, J. Pernier, O. Bertrand, J.F. Echallier, Spherical splines forscalp potential and current density mapping, Electroenceph. Clin.

Ž .Neurophysiol. 72 1989 184–187.w x29 M. Petrides, B. Alivisatos, E. Meyer, A.C. Evans, Functional activa-

tion of the human frontal cortex during the performance of verbalŽ .working memory tasks, Proc. Natl. Acad. Sci. USA 90 1993

878–882.w x30 T.W. Picton, S.A. Hillyard, Cephalic skin potentials in electroen-

Ž .cephalography, Electroenceph. Clin. Neurophysiol. 33 1972 419–424.

w x31 W. Ritter, D. Deacon, H. Gomes, D.C. Javitt, H.G. Vaughan Jr., Themismatch negativity of event-related potentials as a probe of tran-

Ž .sient auditory memory: a review, Ear Hear. 16 1995 52–67.w x32 B. Rockstroh, T. Elbert, A. Canavan, W. Lutzenberger, N. Bir-

baumer, Slow Cortical Potentials and Behavior, Urban andSchwarzenberg, Baltimore, MD, 1989, pp. 1–267.

w x33 B. Rockstroh, M. Muller, M. Wagner, R. Cohen, T. Elbert, ‘‘Prob-¨ing’’ the nature of the CNV, Electroenceph. Clin. Neurophysiol. 87Ž .1993 235–241.

w x34 J.W. Rohrbaugh, K. Syndulko, D.B. Lindsley, Cortical slow nega-tive waves following non-paired stimuli: effects of task factors,

Ž .Electroenceph. Clin. Neurophysiol. 45 1978 551–567.w x35 J.W. Rohrbaugh, K. Syndulko, D.B. Lindsley, Cortical slow nega-

tive waves following non-paired stimuli: effects of modality, inten-sity and rate of stimulation, Electroenceph. Clin. Neurophysiol. 46Ž .1979 416–427.

w x36 D.S. Ruchkin, Measurement of event-related potentials: signal ex-Ž .traction, in: T.W. Picton Ed. , Handbook of Electroencephalogra-

phy and Clinical Neurophysiology, Revised Series, vol. 3: HumanEvent-Related Potentials, Elsevier, Amsterdam, 1988, pp. 7–43.


w x37 D.S. Ruchkin, AC-to-DC inverse filtering of event-related potentials,Ž .in: W.C. McCallum, S.H. Curry Eds. , Slow Potential Changes in

the Human Brain, Plenum, New York, 1993, pp. 91–97.w x38 D.S. Ruchkin, J. Grafman, G.L. Krauss, R. Johnson Jr., H. Canoune,

W. Ritter, Event-related brain potential evidence for a verbal work-Ž .ing memory deficit in multiple sclerosis, Brain 117 1994 289–305.

w x39 D.S. Ruchkin, R. Johnson Jr., H. Canoune, W. Ritter, Short-termmemory storage and retention: an event-related brain potential study,

Ž .Electroenceph. Clin. Neurophysiol. 76 1990 419–439.w x40 D.S. Ruchkin, R. Johnson Jr., J. Grafman, H. Canoune, W. Ritter,

Distinctions and similarities among working memory processes: anŽ .event-related potential study, Cogn. Brain Res. 1 1992 53–66.

w x41 D.S. Ruchkin, R. Johnson Jr., J. Grafman, H. Canoune, W. Ritter,Multiple visuo-spatial working memory buffers: evidence from spa-

Ž .tio-temporal patterns of brain activity, Neuropsychologia 35 1997195–209.

w x42 M. Sams, R. Hari, Magnetoencephalography in the study of humanŽ .auditory information processing, Ann. NY Acad. Sci. 620 1991

102–117.w x43 M. Scherg, Fundamentals of dipole source potential analysis, in: F.

Ž .Grandori, G.L. Romani, M. Hoke Eds. , Auditory Evoked Electricand Magnetic Fields. Topographic Mapping and Functional Local-

ization. Advances in Audiology, vol. 6, Karger, Basel, 1990, pp.40–69.

w x44 E.H. Schumacher, E. Lauber, E. Awh, J. Jonides, E.E. Smith, R.A.Koeppe, PET evidence for an amodal verbal working memory

Ž .system, Neuroimage 3 1996 79–88.w x45 T. Shallice, G. Vallar, The impairment of auditory-verbal short-term

Ž .storage, in: G. Vallar, T. Shallice Eds. , Neuropsychological Im-pairments of Short-Term Memory, Cambridge University Press,Cambridge, 1990, pp. 11–53.

w x46 E.E. Smith, J. Jonides, R.A. Koeppe, Dissociating verbal and spatialŽ .working memory using PET, Cereb. Cortex 6 1996 11–20.

w x Ž .47 J.J. Tecce, L. Cattanach, Contingent negative variation CNV , in: E.Ž .Niedermeyer, F. Lopes da Silva Eds. , Electroencephalograpy: Ba-

sic Principles, Clinical Applications and Related Fields, Urban andSchwarzenberg, Baltimore, MD, 1987, pp. 657–679.

w x48 G. Vallar, C. Papagno, Neuropsychological impairments of short-Ž .term memory, in: A.D. Baddeley, B.A. Wilson, F.N. Watts Eds. ,

Handbook of Memory Disorders, Wiley and Sons, Chichester, 1995,pp. 135–165.

w x49 G.S. Waters, E. Rochon, D. Caplan, The role of high-level speechplanning in rehearsal: evidence from patients with apraxia of speech,

Ž .J. Mem. Lang. 31 1992 54–73.

research report modality-specific processing streams in verbal

Documents