processing abstract auditory features in the human auditory cortex

Processing abstract auditory features in the human auditory cortex

Oleg A. Korzyukov,a,b,c,* Istvan Winkler,a,d Valentina I. Gumenyuk,a and Kimmo Alhoe

a Cognitive Brain Research Unit, Department of Psychology, University of Helsinki, Finlandb Helsinki Brain Research Center, Helsinki, Finland

c Department of Psychiatry, Yale University and VA-Connecticut Healthcare System, West Haven, CT, USAd Institute for Psychology, Hungarian Academy of Sciences, H-1394 Budapest, P.O. Box, 398, Hungary

e General Psychology Division, Department of Psychology, P.O. Box 09, FIN-00014 University of Helsinki, Finland

Received 7 May 2003; revised 8 August 2003; accepted 14 August 2003

Abstract

Using electric and magnetic brain responses we tested whether violations of an abstract auditory regularity are processed in auditorycortex and whether abstract auditory regularities are retained for at least 10 s. The mismatch negativity (MMN) event-related potential andits magnetic counterpart (MMNm) were recorded to infrequent tone pairs of descending pitch (the second tone having a lower frequencythan the first one) embedded in a sequence of tone pairs of ascending pitch, whereas the absolute frequency of both ascending anddescending tone pairs varied on seven levels. Results showed that the dominant generators of the electromagnetic activity elicited byviolations of the pitch-ascension rule lie within auditory cortex. We also found that the memory representation of pitch-ascension is retainedfor at least 10 s. When short trains of ascending-pitched tone pairs were followed by a silent period of 8–12 s, descending-pitched probetone pairs elicited the MMN component when a single reminding pair with ascending pitch was presented before the probe. The reactivatingeffect of the reminder was similar to what has been previously shown for concrete auditory regularities, such as the constancy of tone pitch.The present results support the view that auditory cortical functions can process sensory and categorical information in a similar manner.© 2003 Elsevier Inc. All rights reserved.

Introduction

In music as well as in speech, patterns carrying informa-tion are formed by combining a limited number of discretesound elements (phonemes or notes) in accordance withcommon rules (see Zatorre et al., 2002). Therefore, under-standing speech or enjoying music requires fast detectionand encoding of the higher order regularities (i.e., ones thatare not tied to concrete levels of acoustic features) that arepresent in sound sequences. Recent evidence suggested thatthe analysis of higher order regularities is performed for allsounds, irrespective of their relevance to the current goal-oriented behavior (Na¨atanen et al., 2001). Higher orderregularities include “abstract” rules, that is, rules that aredefined by relationships between stimulus features indepen-

dently of the absolute feature levels. For example, there isevidence that the brain can represent the regular aspects ofan unattended sound sequence composed of tone pairs ofvarying frequencies, when the common feature of the pairsis that the frequency of the second tone of each pair ishigher than that of the first tone (“pitch ascension”; Saarinenet al., 1992; Paavilainen et al., 1995, 1999, for a review, seeNaatanen et al., 2001). This type of abstract regularity isespecially interesting for research because, for example, itcan shed light on the processing of “intonation” in spokenlanguage (Patel et al., 1998) and of “melodic contour” inmusic (Trainor et al., 2002).

There is a particularly intriguing question regarding thememory of auditory features: How can one recognize (byauditory information alone) melodies, musical instruments,or a person’s voice when memory of fine acoustic details,such as the exact pitch of a sound, is lost within a fewseconds (Cowan, 1984)? In previous studies, we found amemory process termedreactivation that may allow humanlisteners to recognize sounds even when the immediate

* Corresponding author. Department of Psychiatry, Yale UniversitySchool of Medicine, VA Connecticut Healthcare System, 950 CampbellAvenue, West Haven, CT 06516, Fax:�1-203-937-4791.

E-mail address: [email protected] (O.A. Korzyukov).

NeuroImage 20 (2003) 2245–2258 www.elsevier.com/locate/ynimg

1053-8119/$ – see front matter © 2003 Elsevier Inc. All rights reserved.doi:10.1016/j.neuroimage.2003.08.014

memory traces for these sounds have been lost (Cowan etal., 1993; Ritter et al., 1998; Winkler et al., 1996, 2002; fora similar findings obtained by behavioral methods, seeRovee-Collier and Hayne, 1987). In natural situations, how-ever, we usually recognize objects by their “abstract” (re-lational) properties rather than by concrete acoustic featurelevels. For example, we recognize a melody even if it isplayed on a different absolute pitch level than the last timewhen we listened to it. That is, we recognize the intersoundpitch relationships defining the melody contour rather thanthe actual sounds that we heard. This suggests that thereactivation process observed in our previous studies shouldalso operate on abstract regularities.

Reactivation of concrete (nonabstract) acoustic memorytraces has been demonstrated with the help of an event-related brain potential (ERP) component termed the mis-match negativity (MMN; for recent reviews, see Naatanenand Winkler, 1999; Picton et al., 2000). MMN is elicited bysounds violating some regular aspect of the preceding soundsequence, such as occasionally presenting a deviant soundin a repetitive sound sequence (the auditory oddball para-

digm). MMN is generated irrespective of whether thesounds are task-relevant (Alho et al., 1994; Naatanen, 1990;Sams et al., 1985; Sussman et al., 2003). It has been estab-lished that the MMN-generating process involves memoryrecords of acoustic regularities (the “standard” ; Jacobsenand Schroger, 2001; Korzyukov et al., 1999; for reviews,see Naatanen, 1990; Naatanen and Winkler, 1999; Ritter etal., 1995; Winkler, 1996).

MMN is elicited not only by sounds deviating from anexactly repeating stimulus or stimulus feature but also bysounds breaking abstract (relational) rules, such as pitchascension within a tone pair (the second tone of the pairhaving a higher frequency than the first tone; see Fig. 1,middle panel; Paavilainen et al., 1995, 1999; Saarinen et al.,1992) or the relationship between two features of a soundwithin the sequence (e.g., the higher the pitch of a tone, thesofter its intensity; Paavilainen et al., 2001).

The current study was designed to answer two questions:(1) to test the assumption that the processing of abstractauditory rules involves auditory cortex (see Naatanen et al.,2001) and (2) to test whether the memory record of an

Fig. 1. Schematic illustration of a fragment of the auditory stimulus sequences presented in the experiments. The frequency of the pure sinusoidal tonespresented in these sequences is shown on the vertical axes. Time is schematically represented by the horizontal axis. Descending-frequency tone paris aremarked by thick outline, and ascending ones by thin outline. Gray shading of the boxes indicates deviant and Probe tone pairs.

2246 O.A. Korzyukov et al. / NeuroImage 20 (2003) 2245–2258

abstract auditory regularity can be reactivated similarly tothe reactivation of memory records describing concrete au-ditory regularities (such as the repetition of a single tone).These two issues are important both to psychological and

neuroscientific theories of the storage and processing ofsensory and categorical forms of information. Traditionalviews of information processing have assumed that abstractrules are stored in categorical memory representations and

Fig. 2. Grand-averaged (N � 7) frontal (Fz) and right mastoid (RM) ERPs measured in the three electrical experiments. Gray area shading (middle and bottompanel) indicates significant difference between the overplotted standard vs deviant and Probe vs Probe control ERP waveforms (the MMN responses).

Fig. 3. (Top panel) MEG sensor locations over the right and left hemisphere, the sensors used for modeling the responses are shown with black lines. (Middlepanel) Across-subject mean ECD locations for N1m (yellow circle), MMNm elicited by frequency change (blue square), and for the MMNm elicited byabstract-regularity violations (red square). (Bottom panel) Frontal (Fz, top) and central (Cz, bottom) ERP responses measured for the standard (blackcontinuous line) and deviant (dashed line) tones in the Oddball (left) and Frequency-change (right) experiments. The deviant-minus-standard differencewaveforms are marked with continuous red (Oddball. experiment) and blue (Frequency-change experiment) lines. Gray area shading indicates the MMNresponses.

2247O.A. Korzyukov et al. / NeuroImage 20 (2003) 2245–2258

Fig. 4. (Top panel) Magnetic responses in one representative subject (four selected channels directly over the approximate location of the right auditory cortex)elicited by the frequent (black solid lines) and infrequent (dashed lines) stimuli in the Frequency-change (left) and the Oddball experiment (right), together with theirrespective difference waveforms (blue line for the Frequency-change and red line for the Oddball experiment). (Bottom panel) ECDs projected on a tilted horizontal(left side) and right sagittal (right side) MRI slice obtained of the same subject. The level and orientation of the MRI slices crossing the superior temporal cortexare shown (by white line) on the corresponding MRI slices. The ECDs shown (blue square for the MMNm obtained in the Frequency-change, red square for theMMNm in the Oddball experiment, and yellow circle for N1m) were calculated at the peak of the corresponding magnetic activity.

are processed separately from sensory information (which isstored in sensory memory codes) in non-sensory-specificareas of the brain. The current study was designed to testthese assumptions: Whether sensory and abstract informa-tion can be processed together and whether abstract (cate-gorical) information can be accessed by processes whoseneural substrate is located in sensory-specific areas of thebrain (in the present case, auditory cortex).

In the auditory reactivation paradigm, short trains ofsounds are separated by a relatively long silent interval(6–30 s in previous studies). Standard trains consist ofregular sounds, e.g., a repeating tone (see, e.g., Winkler etal., 2002). Occasionally or after each standard train, a reac-tivation train is presented. Reactivation trains start with aregular stimulus (the “ reminder” ) immediately followed bya deviant stimulus. If the deviant stimulus elicits MMN,reactivation has occurred. This is because it has been shownthat no MMN is elicited by a deviant stimulus delivered inthe first position of a train or a stimulus change occurring atthe beginning (the first two positions) of a train that has notbeen preceded by a corresponding standard train (Cowan etal., 1993). The elicitation of MMN requires the presence ofa memory record representing some auditory regularity,which is violated by the deviant stimulus (see above). Fur-thermore, the memory representation of a regularity be-comes inactive after a few seconds (Grau et al., 1998;Winkler et al., 2001). The reminder reestablishes the corre-sponding memory record as a “standard” against whichfurther incoming stimuli are compared, but only if the re-minder closely matches the standard. Winkler et al. (2002)showed that when the standard was a repeated tone, a sounddiffering from the frequency of the repeated tone by only3% did not reactivate the memory record of the standard andno MMN was elicited by the following deviant tone. Reac-tivation has been demonstrated even after a 30-s-long silentinterval (Winkler et al., 2002) and also when random soundsintervened between the standard and the reactivation train(Winkler et al., 1996). The long retention interval, at whichpoint subjects were not able anymore to discriminate thestandard tone from another similar tone (Winkler et al.,2002) as well as reactivation after substantial amount ofinterference suggest that the memory trace of the standardtone has been eliminated from the immediate memorybuffer and it could only be made accessible again to mentaloperations by the reminder (i.e., through the reactivationprocess). Winkler et al. (2002) suggested that, in addition tosensory buffering, auditory sensory information is stored inmore durable memory traces. This latter form of storageencodes auditory sensory information in the context of thepattern or regularity within which the sounds were encoun-tered. The reminder reactivates the memory record of thewhole regularity, thus making sensory details also availablewhen they have already faded from the sensory buffer. Inthe current study, if MMN was elicited by the deviantstimulus immediately following the reminder, this wouldsuggest that a memory record of an abstract regularity can

be reactivated, similarly to the reactivation of concrete stim-ulus and stimulus-feature repetition rules (Cowan et al.,1993; Ritter et al., 1998).

Materials and Methods

Subjects

Electrical brain responses were studied from sevenhealthy right-handed subjects (22–35 years of age, threefemale). Subjects were instructed to watch a subtitled moviewithout sound and to ignore the test sounds throughout theexperiments.

Stimuli and procedures

Stimuli were eight different pure sinusoidal tones (30 msin duration with a 5-ms rise and a 5-ms fall times included,50 dB in intensity above the subjective hearing threshold,binaurally presented through headphones). Tone frequen-cies were taken from the 500 to 1104 Hz range in equal 12%steps (Fig. 1). The tones were presented in pairs. Tone pairseither descended or ascended in frequency by one frequencystep. Pairs with different absolute frequencies were deliv-ered in a random order and with a uniform timing. Thewithin-pair onset-to-onset interval was 120 ms and the in-terpair onset-to-onset interval was 1 s.

Three experiments, Equiprobable, Oddball, and Reacti-vation, were administered in two sessions (seven blocks ofsounds, in each; 168, 440, and 440 stimulus pairs per blockin the Equiprobable, Oddball, and Reactivation experi-ments, respectively), which were carried out on separatedays. In the Equiprobable experiment (Fig. 1, top panel)ascending and descending tone pairs were presented inrandom order with equal probability. In the Oddball exper-iment (Figure 1, middle panel), the probability of the as-cending-pitch tone pairs was 90% (standards), whereas thatof the descending-pitch tone pairs was 10% (deviants). TheReactivation experiment consisted of two conditions: Testand Control. In both conditions, two short trains of tonepairs alternated (Fig. 1, bottom panel). The absolute fre-quencies of the tone pairs varied, as in the other two exper-iments. In the Test condition, “Standard Trains” consistedof six tone pairs of ascending pitch and were followed by asilent interval of 8, 9, 10, 11, or 12 s, presented in randomorder with equal probability. “Reactivation Trains” con-sisted of four tone pairs, starting with an ascending-pitch“Reminder,” followed by a descending-pitched “Probe,”and ending with two ascending-pitch tone pairs. Reactiva-tion Trains were separated from the following StandardTrain by a silent interval of 3, 4, or 5 s, again, presented inrandom order with equal probability. The timing of thetrains was identical in the Control condition. However,Standard trains were exchanged for “No-Standard Trains,”which had an equal number of ascending and descending


tone pairs presented in random order. In the Control condi-tion, Reactivation Trains were exchanged for ComparisonTrains. The first two tone pairs of these trains were just likethe corresponding pairs in the Reactivation Trains (ascend-ing-pitch “Reminder Control” and descending-pitch “ProbeControl” ), but were followed by one ascending- and onedescending-pitched pair in random order. The Control con-dition allowed us to determine the elicitation of MMNwithout confounding factors. In particular, because the“Probe Control” was physically identical to the “Probe,” itfollowed the same stimuli as the “Probe” (the “ReminderControl” ), as well as it was presented with the same tem-poral schedule as the “Probe,” it can be expected that theobligatory ERP components elicited by the “Probe Control”(such as the P1, N1, and P2, which may partly overlap theMMN response and are elicited with a high amplitude afterlong silent intervals) will have the same parameters (ampli-tude and latency) as those elicited by the “Probe” itself.Therefore, the difference between the ERPs elicited by the“Probe” and the “Probe Control” should provide a pureestimate of the MMN(m) response.

Data collection and analysis

The electroencephalogram (EEG) was recorded with Ag/AgCl electrodes from 15 scalp locations [F7, F3, Fz, F4, F8,C3, Cz, C4, T5, P3, Pz, P4, T6 (10–20 system) and the leftand right mastoids] with the nose as the common reference.Horizontal and vertical eye movements were recorded, re-spectively, between electrodes placed at the outer canthi ofthe two eyes and between electrodes placed above andbelow the left eye (horizontal and vertical electrooculo-gram, EOG). The EEG was digitized (SynAmps amplifiersand Scan software, NeuroScan, Inc.) with 250 Hz samplingrate and 0–40 Hz filter limits. The signals were off-linefiltered between 1 and 20 Hz EEG epochs (beginning100 ms before and ending at 700 ms after the onset of thetone pairs) were averaged separately for each condition andstimulus type (ascending and descending pairs in theEquiprobable experiment; standard and deviant pairs in theOddball experiment; Probe and Probe Control, Reminder

and Reminder Control pairs in the Reactivation experi-ment). In the Oddball Experiment, epochs elicited by stan-dard pairs immediately following a deviant pair were omit-ted from averaging. Responses contaminated byextracerebral artifacts causing EEG or EOG changes ex-ceeding 80 �V in any recording channel were also excludedfrom averaging. At least 100 acceptable responses wereaveraged for each stimulus type after the artifact rejection.

For testing the elicitation of MMN, we compared themean voltages in the MMN interval (see below) between theERPs elicited by deviant and standard pairs (Oddball ex-periment) and Probe and Probe Control pairs (Reactivationexperiment). (In the Oddball experiment, no separate con-trol condition is needed, because the relatively fast stimuluspresentation rate largely reduces the amplitude of the oblig-atory ERP components, thus the deviant-minus-standarddifference provides a good estimate of the MMN response;see, e.g., Naatanen, 1992.) Amplitude measurements werereferred to the average voltage of the 100-ms prestimulusperiod. Measurement intervals were established on the basisof the group averaged ERP responses. The MMN intervalwas defined as the 40-ms interval centered on the negativedeviant-minus-standard (Probe-minus-Probe-Control) dif-ference peak within 100 to 200 ms from the onset of devi-ation (220–320 ms from pair onset, because the secondtone, which determines whether the pair is ascending ordescending in pitch, commenced 120 ms after the onset ofthe first tone).

Statistical comparisons were performed by repeatedmeasures ANOVA. The first ANOVA test had StimulusType (ascending vs descending in the Equiprobable exper-iment; standard vs deviant in the Oddball experiment; Probevs Probe Control in the Reactivation experiment) and twoelectrode location factors (Anterior-Posterior: F-line vs C-line vs P-line; and Laterality: Left vs Middle vs Right, basedon the 3 � 3 matrix of F3, Fz, F4, C3, Cz, C4, P3, Pz, P4).A subsequent ANOVA was calculated only for the MMNresponses established by the first ANOVA test. ThisANOVA had Stimulus Type and Electrode Location (Fz vsright mastoid vs left mastoid) as factors, checking the ex-pected polarity reversal of the MMN (difference between

Table 1Group-averaged equivalent current dipoles for the N1m and the two MMNm responses (Mean � Standard Deviation; latencies measuredfrom the tone/pair onset)

Type of ECD Latency(ms)

Location of ECD Volume(mm3) ofconfidence

Goodnessof fit (%)

Moment(nAm)

x (mm) y (mm) z (mm)

Left hemisphereN1m (standard tone) 95 � 8 �47 � 10 5 � 9 70 � 17 9.6 � 7 82 � 6 15 � 10MMNm (frequency) 146 � 12 �57 � 9 12 � 9 67 � 12 80 � 78 76 � 5 9 � 2MMNm (abstract regularity) 261 � 19 �53 � 7 13 � 7 74 � 10 329 � 319 74 � 12 7 � 4Right hemisphereN1m (standard tone) 97 � 12 47 � 9 15 � 8 66 � 6 1089 � 2117 85 � 9 16 � 6MMNm (frequency) 162 � 18 56 � 3 22 � 8 69 � 5 72 � 54 74 � 7 10 � 3MMNm (abstract regularity) 264 � 25 55 � 8 28 � 7 70 � 7 250 � 252 68 � 12 7 � 4


Fig. 5. Magnetic responses from one representative subject (left side) visualized with minimum current estimates at the peak of the abstract-rule-violationMMNm (top), frequency-change MMNm (middle) and N1m (bottom). The region of interest that has been chosen for the minimum current estimates isdepicted with red color on the schematic brain images shown at the right side.

the deviant and the standard or between the Probe and theProbe control) response that would emerge as a significantinteraction between the two factors. Huynh–Feldt correctionwas applied when appropriate.

An additional magnetoencephalographic study was car-ried out to locate the brain areas involved in generating theMMN response in the Oddball experiment.

Magnetoencephalographic study

The purpose of this experiment was to locate the sourceof the MMNm elicited by violations of an abstract regularityand to compare it with the source location of the widelystudied frequency-change MMNm and the N1m generator(Alho, 1995; Korzyukov et al., 1999).

Reactivation experiment

Fig. 6. Scalp distribution of the deviant-minus-standard and probe-minus-probe-control difference potentials at the peak of the MMN response in the electricalOddball (top panel) and Reactivation (bottom panel) experiments (latencies are marked separately for the two conditions). The color scale for the amplitudes(in microvolts) is shown in the middle of the figure.


The Oddball experiment was replicated, measuring mag-netic event-related fields with a 306-channel whole-headSQUID magnetometer (4-D Neuroimaging Oy, Finland).An additional group consisting of nine healthy right-handedsubjects (20–35 years, 5 female) was studied. MEG data ofone male and one female subject were excluded from theanalysis due to low signal to noise ratio. In addition to theOddball experiment, a sound sequence consisting of a singlerepetitive tone (606 Hz in frequency, all other parameterswere identical to those described above), replaced randomlyin 10% of the trials by a tone differing only in frequency(533 Hz) was administered (Frequency-change experiment;the onset-to-onset interval was 1000 ms).

The EEG (frontal and central midline electrodes only)and magnetoencephalogram (MEG) were recorded with abandpass of 0.1–200 Hz and a 600-Hz sampling rate. MEGepochs starting 100 ms before and ending 1000 ms aftereach tone/tone-pair onset were averaged separately for eachstimulus type and condition. Epochs with artifacts (EOGvariation exceeding 100 �V or MEG variation exceeding3000 fT/cm), and epochs for standard tones/tone pairs im-mediately following a deviant were omitted from averaging.At least 100 acceptable trials were collected for each stim-ulus type after artifact rejection. Responses were digitallyfiltered with a bandpass of 1–20 Hz (for details, see alsoAlho et al., 1998).

Equivalent current dipoles (ECDs) that optimally, in theleast-squares sense, reproduced the N1m and MMNm mag-netic fields (Hamalainen et al., 1993), were determined [setsof 38 sensor units (114 recording channels), centered at theapproximate location of the left and right auditory cortices,Fig. 3, top panel] using a spherical head model. The ECDlocations and orientations were calculated in a coordinatesystem in which the x axis has a medial–lateral, the y axis aposterior–anterior, and the z axis an inferior–superior direc-tion. N1m and MMNm amplitudes were measured withrespect to the mean signal amplitude in the 100-ms pre-stimulus period. Only ECDs with an orientation producinga negative electric response over the frontocentral midlinescalp and only those explaining more than 50% of themeasured signal were accepted. One-way ANOVAs withrepeated measures were performed to compare the ECDlocations in 3-dimensional space. For illustrative purposes,single-dipole MMNm and N1m ECD models were mappedonto the MRI image in one subject.

The N1m and MMNm responses of seven subjects werevisualized using minimum current estimates (MCE; for thedetails of the MCE method, see Uutela et al., 1999), whichcan resolve several local or distributed MEG current sourceswithout explicit a priori information about the number ofsources. A spherical model of the brain was used in theforward calculation; the boundary element model deter-mined the point set used in the calculation and the imagesused in the visualization.

Results

Equiprobable and Oddball experiments

No significant amplitude differences were found betweenthe ERPs elicited by tone pairs with ascending and descend-ing pitch when they were presented with equal probability(Equiprobable experiment; Fig. 2, top panel). In the electri-cal Oddball experiment, the MMN peak latency was 270 �25 ms (Mean � SD) from the pair onset (150 ms from theonset of the second tone; Fig. 2, middle panel). TheANOVA test showed a significant interaction between stim-ulus type and the anterior–posterior factor [frontal vs centralvs parietal electrode locations; F(2,12) � 7.70, P � 0.024,� � 0.59]. Subsequent Newman–Keuls tests (within frontaland parietal ERP amplitudes � stimulus type) showed thatthe significant interaction was caused by the larger negativedifference between the deviant and standard responses at thefrontal than the posterior leads (P � 0.01). The subsequentANOVA showed a significant interaction between Stim-ulus Type and Electrode Location factors [F(2,12) �42,36 P � 0.001], which was caused by the polarityreversal of the deviant-minus-standard difference at themastoid leads.

The magnetic responses showed that, in both hemi-spheres, the across-subject mean ECD locations for both theMMNm elicited by the violation of the abstract auditoryregularity (Oddball experiment) and for the MMNm elicitedby frequency change (Frequency-change experiment) werelocated in the vicinity of auditory cortex, anterior to theN1m generator location (Fig. 3). In the right hemisphere, thesource of the MMNms elicited by frequency-deviant tonesand by abstract-rule violations were significantly anterior[F(2,12) � 19.74, P � 0.01, � � 0.73] to the ECD locationof the N1m elicited by standard tones (Table 1), whereas nosignificant differences were found between the ECD loca-tions along the x or z axes. Furthermore, a subsequentNewman–Keuls test showed that the ECD of the MMNmelicited by the abstract-rule violation was significantly moreanterior not only compared with the N1m ECDs but alsocompared with the MMNm elicited by the frequency-devi-ant tones (P � 0.05; see Fig. 3). Figure 4 shows the locationof the ECDs mapped onto the MRI scan of one representa-tive subject. MCE of the right-hemispheric electromagneticbrain activation at the peak of N1m and the two MMNmpeaks are shown on Fig. 5 for one subject. In 4 of 7 subjectsMCEs show that the MMN generator responding to changein the abstract regularity was anterior to the MMN generatoractivated by a simple frequency change; no subject showedan opposite spatial relationship between the two MMNgenerators. In the left hemisphere, the ECDs modeling thetwo MMNm responses were significantly more anterior[F(2,12) � 4.68, P � 0.05, � � 0.82] and medial [F (2,12)� 9.06, P � 0.05, � � 0.6] to the N1m ECD location.


Reactivation experiment

A statistically significant MMN response was elicited bythe Probe tone pairs in the Reactivation experiment (Fig. 2,bottom panel). The first ANOVA test showed significantinteraction between stimulus type (Probe vs. Probe Control)and the anterior–posterior factor [frontal vs central vs pari-etal electrode locations; F(2,12) � 4.41, P � 0.05, � �0.89]. Subsequent Newman–Keuls tests revealed that thesignificant interaction was caused by the larger negativeamplitude difference between the Probe and the Probe Con-trol responses at the frontal than at the posterior leads (P �0.01). The second ANOVA showed a nonsignificant polar-ity reversal at the mastoid leads. However, the close simi-larity between scalp distributions of the MMNs elicited inthe electrical Oddball and Reactivation experiments (Fig. 6)suggests that the MMN elicited after reactivation of theabstract rules was generated by at least partly the sameneural populations that were responsible for the abstract-rule-violation MMN elicited in the Oddball experiment.

Discussion

Generator location of the MMN(m) elicited by violatingan abstract regularity

Corroborating evidence from present EEG and MEGstudies have indicated that the MMN(m) elicited by viola-tions of an abstract regularity is, at least partly, generated inauditory cortex. The scalp distribution of the MMN elicitedin the electrical Oddball experiment was compatible withthe notion of auditory cortical generators underlying theMMN (see Fig. 6, top). For the first time, a clear andstatistically significant polarity reversal across the temporalcortex was demonstrated for the abstract-regularity basedMMN (Fig. 2, Oddball experiment). Recording electricalpotentials of opposite polarities from the two sides of theSylvian fissure is an important indicator that the generatorsof the MMN component lies within auditory cortex (Alho etal., 1986; Giard et al., 1995). The replication of the Oddballexperiment with magnetoencephalographic measurementprovided compatible results. The ECD source location forthe MMNm elicited by the abstract regularity violation waslocated in the vicinity of auditory cortex, close to the sourceof the frequency-change MMN and anterior to the source ofthe N1m component (see Figs. 3–5), as was found in manyprevious studies (e.g., Korzyukov et al., 1999; for a review,see Alho, 1995).

A number of previous studies have shown that the MMNresponse elicited by violations of concrete regularities re-ceives contribution from at least two separate cortical gen-erators (e.g., Giard et al., 1990, Rinne et al., 2000, Yago etal., 2001, Opitz et al., 2002). The primary generator ofMMN (and its magnetic counterpart, the MMNm) has beenlocated within auditory cortex (Celsis et al., 1999; Halgren

et al., 1995; Kropotov et al., 1995; Opitz 1999 for a review,see Alho, 1995). According to some results, this activationof the temporal MMN generator precedes that of the second,frontal MMN generator (Rinne et al., 2000). Although theprecise neuronal mechanisms of the frontal MMN generatorare poorly understood, it has been suggested that this com-ponent is associated with triggering an involuntary attentionswitch toward the deviant auditory event (Giard et al., 1990;Naatanen, 1992; Escera et al., 2000).

However, the frontal MMN generator found in theabove-mentioned studies cannot account for the presentfinding of a more frontal ECD location (Figs. 3 and 4) andMCE activation (Fig. 5) for the MMNm elicited by theabstract-regularity violations (compared with that for thefrequency-deviation MMNm) because the previously ob-served frontal MMN generator is less detectable by MEG(Levanen et al., 1996; Rinne et al., 2000). One possibleexplanation of this finding can be given if we assume theexistence of an additional, presumably prefrontal MMNgenerator that is active only when violations of abstractsequential regularities are processed. This suggestion iscompatible with previous studies that found evidence of theprocessing of abstract regularities in prefrontal cortex(Huettel et al., 2002; Wallis et al., 2001).

Alain et al. (1998) studied the effects of unilateral lesionsof the superior temporal and dorsolateral prefrontal corticeson the MMN. Both types of cortical lesions attenuated theMMN elicited by a frequency change in a repetitive tonesequence as well as the MMN elicited by violations of amore complex rule, i.e., by occasional frequency repetitionsembedded in a sequence of two regularly alternating fre-quencies. Results of Alain et al. (1998) suggested that awidely distributed neocortical circuit, which includes bothtemporal and dorsolateral prefrontal cortices, are involvedin storing auditory sensory memory traces. These circuitsare also involved in generating the MMN response elicitedby violations of complex regularities. These authors dis-cussed the possibility that more complex stimuli such asspeech, music, or environmental sounds likely place greaterdemands on memory access than the detection of changes insimple auditory stimuli. Thus our results might reflectgreater demands on memory access inherent to the process-ing of abstract regularities that more strongly activate pre-frontal brain areas mediating monitoring processes that per-tain to the accuracy of information retrieved from memoryfor error checking (Fletcher et al., 1996; Henson et al.,1999).

Alternatively, it is possible that violations of an abstractrule cause more widespread activation of auditory cortexthan simple frequency change, which has “pushed” the ECDlocation (representing the center of the activation) in theanterior direction (cf. Alho et al., 1996). This alternativeinterpretation receives support from findings showing thatthe anterior regions of superior temporal sulcus respondpreferentially to acoustic spectral variation (Zatorre andBelin, 2001), whereas pure tone responses are generally best


observed in primary auditory regions (Rauschecker et al.,1995; Wessinger et al., 2001). This observation apparentlyreflects the integration of different frequencies over timeand may therefore be tied to interactions between unitstuned to different frequency bands (Brosch et al., 1999;Zatorre and Belin, 2001). Moreover lesion study in the rightauditory cortex suggests spatially separate (within subre-gions of auditory cortex) processing the direction of pitchchange between two notes and the same pitches alone(Johnsrude et al., 2000).

Finally, simultaneous contribution to the generation ofabstract-rule violation MMNm response from neuronal net-works that are localized in auditory fields of the temporallobe and in prefrontal brain regions (a combination of theabove two alternative explanations) might also be consid-ered as an appropriate interpretation of our results. Thispossibility is compatible with electrophysiological record-ing and anatomical tracing results that demonstrated theexistence of pathways originating in auditory fields of thesuperior temporal region and terminating in distinct regionsof the frontal lobe (Romanski et al., 1999).

Memory records involved in the MMN-generating process

Probe pairs elicited MMN in the Reactivation experi-ment as was shown by the comparison between the re-sponses elicited by the Probe and Probe Control tone pairs.The elicitation of the MMN component by the Probe pairshows that at the time it was presented, ascending-pitchserved as the “standard” (the detected regularity) for theMMN-generating process. Because pervious studiesshowed that the first stimulus in a train following silentintervals of ca. 10 s does not elicit MMN (e.g., Cowan et al.,1993), therefore, the reminder must have reactivated thememory record of the standard formed during the Standardtrains. This reactivated regularity representation was thenmismatched by the Probe pair.

The similar timing and close correspondence betweenscalp distributions of the MMNs elicited in the Oddball andReactivation experiments (Fig. 6) suggest that the MMNelicited after reactivation of an abstract rule (ReactivationExperiment) was generated by largely the same neural pop-ulations that were responsible for the abstract-rule violationMMN observed in the Oddball experiment. This assumptionis compatible with the notion that reactivated information isprocessed by the same brain regions that were engagedduring its encoding (Nyberg et al., 2000; Buckner andWheeler, 2001; Gandhi, 2001) and that the reactivation of aconsolidated memory presumably returns it to a labile state(Nader, 2003).

Our Reactivation experiment result provides importantevidence for filling the apparent gap between the limitedtemporal capacity of the sensory buffers and our everydayexperience of being able to recognize particular objects (notjust object categories) by detailed sensory information.Real-life objects show characteristic regularities, which are

of sensory nature but cannot be defined by a few absolutefeature levels. For example, a person’s voice is character-ized by complex spectral and dynamic (temporal) features,which, together, are sufficiently unique to identify thespeaker. However, there is no simple sound feature, such aspitch or intensity, the concrete level of which would alwaysbe present in the voice of a given person. The characteristicfeatures of a person’s voice are of relational nature, such asthe relative amounts of energy emitted in different spectralbands. Identifying a person by his/her voice requires somememory record of these characteristics, one that is suffi-ciently rich in acoustic details to distinguish it from othermembers of some category (e.g., male or female voices).This information must be stored in long-term memory,which would not be possible, if all sensory details were lostwithin a few seconds (as has been shown for example forthe exact pitch of isolated tones) or if the informationretained for long term would be of purely categorical nature.

The current results demonstrate the existence of an in-termediate phase of the memory system, in which the in-formation, which has already been eliminated from thesensory buffer can be reactivated when similar stimuli areencountered. However, the formation of true long-termmemory records requires more time than the intervals usedin the current experiment and involves additional brainstructures, such as the hippocampus (Hasselmo and McClel-land, 1999) and parietal and prefrontal cortex (Buckner andWheeler, 2001; Otten et al., 2002; for reviews see Nader,2003). Conversion of the abstract regularity records dem-onstrated in the current experiment to long-term memoryrecords may occur during sleep, as has been suggested bythe results of Atienza et al. (2001 and 2002).

Summary

The view emerging from our results is that there areprocesses that operate similarly on sensory and categoricalinformation (for compatible evidence, see Winkler et al.,1999). The deviance detection process reflected by theMMN(m) electric brain potential and magnetic field exem-plifies these type of mental operations. Importantly, a majorpart of the MMN-generating processes have been located inauditory cortex, both, when the deviation was of sensory aswell as when it was of abstract nature. Thus it appears thatauditory cortex is involved not only in processing sensorybut also categorical forms of information, supporting higherlevel cognitive operations than it was previously assumed(cf. Naatanen et al., 2001).

Acknowledgments

This research was supported the Academy of FinlandGrants 77322, 55606, and 102316; the Academy of Finlandfunding the Graduate School of Psychology; and the Hun-garian National Research Fund (OTKA T034112)


References

Alain, C., Woods, D.L., Knight, R.T., 1998. A distributed cortical networkfor auditory sensory memory in humans. Brain Res. 812, 23–37.

Alho, K., 1995. Cerebral generators of mismatch negativity (MMN) and itsmagnetic counterpart (MMNm) elicited by sound changes. Ear Hear.16, 38–51.

Alho, K., Paavilainen, P., Reinikainen, K., Sams, M., Naatanen, R., 1986.Separability of different negative components of the event-related brainpotential associated with auditory stimulus processing. Psychophysiol-ogy 23, 613–624.

Alho, K., Tervaniemi, M., Huotilainen, M., Lavikainen, J., Tiitinen, H.,Ilmoniemi, R.J., Knuutila, J., Naatanen, R., 1996. Processing of com-plex sounds in the human auditory cortex as revealed by magnetic brainresponses. Psychophysiology 33, 369–375.

Alho, K., Winkler, I., Escera, C., Huotilainen, M., Virtanen, J., Jaaskel-ainen, I.P., Pekkonen, E., Ilmoniemi, R.J., 1998. Processing of novelsounds and frequency changes in the human auditory cortex: magne-toencephalographic recordings. Psychophysiology 35, 211–224.

Alho, K., Woods, D.L., Algazi, A., 1994. Processing of auditory stimuliduring auditory and visual attention as revealed by event-related po-tentials. Psychophysiology 31, 469–479.

Atienza, M., Cantero, J.L., 2001. Complex sound processing during humanREM sleep by recovering information from long-term memory asrevealed by the mismatch negativity (MMN). Brain Res. 901, 151–160.

Atienza, M., Cantero, J.L., Dominguez-Marin, E., 2002. Mismatch nega-tivity (MMN): an objective measure of sensory memory and long-lasting memories during sleep. Int. J. Psychophysiol. 46, 215–225.

Brosch, M., Schulz, A., Scheich, H., 1999. Processing of sound sequencesin macaque auditory cortex: response enhancement. J. Neurophysiol.82, 1542–1559.

Buckner, R.L., Wheeler, M.E., 2001. The cognitive neuroscience of re-membering. Nat. Rev. Neurosci. 2, 624–634.

Celsis, P., Boulanouar, K., Doyon, B., Ranjeva, J.P., Berry, I., Nespoulous,J.L., Chollet, F., 1999. Differential fMRI responses in the left posteriorsuperior temporal gyrus and left supramarginal gyrus to habituation andchange detection in syllables and tones. Neuroimage 9, 135–144.

Cowan, N., 1984. On short and long auditory stores. Psychol. Bull. 96,341–370.

Cowan, N., Winkler, I., Teder, W., Naatanen, R., 1993. Short- and long-term prerequisites of the mismatch negativity in the auditory event-related potential (ERP). J. Exp. Psychol. Learn. Mem. Cogn. 19, 909–921.

Deouell, L.Y., Bentin, S., Giard, M.H., 1998. Mismatch negativity indichotic listening: evidence for interhemispheric differences and mul-tiple generators. Psychophysiology 35, 355–365.

Fletcher, P.C., Shallice, T., Frith, C.D., Frackowiak, R.S., Dolan, R.J.,1996. Brain activity during memory retrieval: the influence of imageryand semantic cueing. Brain 119, 1587–1596.

Escera, C., Alho, K., Schroger, E., Winkler, I., 2000. Involuntary attentionand distractibility as evaluated with event-related brain potentials. Au-diol. Neuro-Otol. 5, 151–166.

Gandhi, S.P., 2001. Memory retrieval: reactivating sensory cortex. Curr.Biol. 11, 32–34.

Giard, M.H., Lavikainen, J., Reinikainen, K., Perrin, F., Bertrand, O.,Pernier, J., Naatanen, R., 1995. Separate representation of stimulusfrequency, intensity and duration in auditory sensory memory: Anevent-related potential and dipole-model analysis. J. Cogn. Neurosci. 7,133–143.

Giard, M.H., Perrin, F., Pernier, J., Bouchet, P., 1990. Brain generatorsimplicated in the processing of auditory stimulus deviance: a topo-graphic event-related potential study. Psychophysiology 27, 627–640.

Grau, C., Escera, C., Yago, E., Polo, M.D., 1998. Mismatch negativity andauditory sensory memory evaluation: a new faster paradigm. Neuro-Report 9, 2451–2456.

Halgren, E., Baudena, P., Clarke, J.M., Heit, G., Liegeois, C., Chauvel, P.,Musolino, A., 1995. Intracerebral potentials to rare target and distractorauditory and visual stimuli. I. Superior temporal plane and parietallobe. Electroencephalogr. Clin. Neurophysiol. 94, 191–220.

Hamalainen, M., Hari, R., Ilmoniemi, R.J., Knuutila, J., Lounasmaa, O.V.,1993. Magnetoencephalography-theory, instrumentation and applica-tions to non-invasive studies of the working human brain. Rev. Mod.Phys. 65, 413–497.

Hasselmo, M.E., McClelland, J.L., 1999. Neural models of memory. Curr.Opin. Neurobiol. 9, 184–188.

Henson, R.N., Shallice, T., Dolan, R.J., 1999. Right prefrontal cortex andepisodic memory retrieval: a functional MRI test of the monitoringhypothesis. Brain 122, 1367–1381.

Huettel, S.A., Mack, P.B., McCarthy, G., 2002. Perceiving patterns inrandom series: dynamic processing of sequence in prefrontal cortex.Nat. Neurosci. 5, 485–490.

Jacobsen, T., Schroger, E., 2001. Is there pre-attentive memory-basedcomparison of pitch? Psychophysiology 38, 723–727.

Johnsrude, I.S., Penhune, V.B., Zatorre, R.J., 2000. Functional specificityin the right human auditory cortex for perceiving pitch direction. Brain123, 155–163.

Korzyukov, O., Alho, K., Kujala, A., Gumenyuk, V., Ilmoniemi, R.J.,Virtanen, J., Kropotov, J., Naatanen, R., 1999. Electromagnetic re-sponses of the human auditory cortex generated by sensory-memorybased processing of tone-frequency changes. Neurosci. Lett. 276, 169–172.

Kropotov, J.D., Naatanen, R., Sevostianov, A.V., Alho, K., Reinikainen,K., Kropotova, O.V., 1995. Mismatch negativity to auditory stimuluschange recorded directly from the human temporal cortex. Psychophys-iology 32, 418–422.

Levanen, S., Ahonen, A., Hari, R., McEvoy, L., Sams, M., 1996. Deviantauditory stimuli activate human left and right auditory cortex differ-ently. Cerebr. Cortex 6, 288–96.

Naatanen, R., 1990. The role of attention in auditory information process-ing as revealed by event-related potentials and other brain measures ofcognitive function. Behav. Brain Sci. 13, 201–288.

Naatanen, R., 1992. Attention and Brain Function. Erlbaum, Hillsdale, NJ.Naatanen, R., Tervaniemi, M., Sussman, E., Paavilainen, P., Winkler, I.,

2001. ’Primitive intelligence’ in the auditory cortex. Trends Neurosci.24, 283–288.

Naatanen, R., Winkler, I., 1999. The concept of auditory stimulus repre-sentation in cognitive neuroscience. Psychol. Bull. 125, 826–859.

Nader, K., 2003. Memory traces unbound. Trends Neurosci. 28, 65–72.Nyberg, L., Habib, R., McIntosh, A.R., Tulving, E., 2000. Reactivation of

encoding-related brain activity during memory retrieval. Proc. Nat.Acad. Sci. USA 97, 11120–11124.

Opitz, B., Mecklinger, A., von Cramon, D.Y., Kruggel, F., 1999. Combin-ing electrophysiological and hemodynamic measures of the auditoryoddball. Psychophysiology 36, 142–147.

Opitz, B., Rinne, T., Mecklinger, A., von Cramon, D.Y., Schroger, E.,2002. Differential contribution of frontal and temporal cortices toauditory change detection: fMRI and ERP results. NeuroImage 15,167–174.

Otten, L.J., Henson, R.N.A., Rugg, M.D., 2002. State-related and item-related neural correlates of successful memory encoding. Nat. Neuro-sci. 5, 1339–1344.

Paavilainen, P., Jaramillo, M., Naatanen, R., Winkler, I., 1999. Neuronalpopulations in the human brain extracting invariant relationships fromacoustic variance. Neurosci. Lett. 265, 179–182.

Paavilainen, P., Saarinen, J., Tervaniemi, M., Naatanen, R., 1995. Mis-match negativity to changes in abstract sound features during dichoticlistening. J. Psychophysiol. 9, 243–249.

Paavilainen, P., Simola, J., Jaramillo, M., Naatanen, R., Winkler, I., 2001.Preattentive extraction of abstract feature conjunctions from auditorystimulation as reflected by the mismatch negativity (MMN). Psycho-physiology 38, 359–365.


Patel, A.D., Peretz, I., Tramo, M., Labrecque, R., 1998. Processing pro-sodic and musical patterns: A neuropsychological investigation. Brainand Language 61, 123–144.

Picton, T.W., Alain, C., Otten, L., Ritter, W., Achim, A., 2000. Mismatchnegativity: different water in the same river. Audiol. Neuro-Otol. 5,111–139.

Rauschecker, J.P., Tian, B., Hauser, M., 1995. Processing of complexsounds in the macaque nonprimary auditory cortex. Science 268, 111–114.

Rinne, T., Alho, K., Ilmoniemi, R.J., Virtanen, J., Naatanen, R., 2000.Separate time behaviors of the temporal and frontal MMN sources.NeuroImage 12, 14–19.

Ritter, W., Deacon, D., Gomes, H., Javitt, D.C., Vaughan Jr., H.G., 1995.The mismatch negativity of event-related potentials as a probe oftransient auditory memory: a review. Ear Hear. 16, 52–67.

Ritter, W., Gomes, H., Cowan, N., Sussman, E., Vaughan Jr., H.G., 1998.Reactivation of a dormant representation of an auditory stimulus fea-ture. J. Cogn. Neurosci. 10, 605–614.

Romanski, L.M., Tian, B., Fritz, J., Mishkin, M., Goldman-Rakic, P.S.,Rauschecker, J.P., 1999. Dual streams of auditory afferents targetmultiple domains in the primate prefrontal cortex. Nat. Neurosci. 2,1131–1136.

Rovee-Collier, C., Hayne, H., 1987. Reactivation of infant memory: im-plications for cognitive development. Adv. Child Dev. Behav. 20,185–238.

Saarinen, J., Paavilainen, P., Schroger, E., Tervaniemi, M., Naatanen, R.,1992. Representation of abstract attributes of auditory stimuli in thehuman brain. NeuroReport 3, 1149–1151.

Sams, M., Paavilainen, P., Alho, K., Naatanen, R., 1985. Auditory fre-quency discrimination and event-related potentials. Electroencepha-logr. Clin. Neurophysiol. 62, 437–448.

Sussman, E., Winkler, I., Wang, W.J., 2003. MMN and attention: Com-petition for deviance detection. Psychophysiology 40, 430–435.

Trainor, L.J., McDonald, K.L., Alain, C., 2002. Automatic and controlledprocessing of melodic contour and interval information measured byelectrical brain activity. J. Cogni. Neurosci. 14, 430–442.

Uutela, K., Hamalainen, M., Somersalo, E., 1999. Visualization of mag-netoencephalographic data using minimum current estimates. Neuro-Image 10, 173–180.

Wallis, J.D., Anderson, K.C., Miller, E.K., 2001. Single neurons in pre-frontal cortex encode abstract rules. Nature 411, 953–956.

Wessinger, C.M., VanMeter, J., Tian, B., van Lare, J., Pekar, J., Raus-checker, J.P., 2001. Hierarchical organization of the human auditorycortex revealed by functional magnetic resonance imaging. J. Cogn.Neurosci. 13, 1–7.

Winkler, I., 1996. Necessary and sufficient conditions for the elicitation ofthe mismatch negativity, in: Ogura, C., Koga, Y., Shimokochi, M.(Eds.), Recent Advances in Event-Related Brain Potential Research,Elsevier, Amsterdam, pp. 36–44.

Winkler, I., Cowan, N., Csepe, V., Czigler, I., Naatanen, R., 1996. Inter-actions between transient and long-term auditory memory as reflectedby the mismatch negativity. J. Cogn. Neurosci. 8, 403–415.

Winkler, I., Korzyukov, O., Gumenyuk, V., Cowan, N., Linkenkaer-Han-sen, K., Alho, K., Ilmoniemi, R.J., Naatanen, R., 2002. Temporary andlonger retention of acoustic information. Psychophysiology 39, 530–534.

Winkler, I., Lehtokoski, A., Alku, P., Vainio, M., Czigler, I., Csepe, V.,Aaltonen, O., Raimo, I., Alho, K., Lang, A.H., Iivonen, A., Naatanen,R., 1999. Pre-attentive detection of vowel contrasts utilizes both pho-netic and auditory memory representations. Cogn. Brain Res. 7, 357–369.

Winkler, I., Schroger, E., Cowan, N., 2001. The role of large-scale memoryorganization in the mismatch negativity event-related brain potential. J.Cogn. Neurosci. 13, 59–71.

Yago, E., Escera, C., Alho, K., Giard, M-H., 2001. Cerebral mechanismsunderlying orienting of attention towards auditory frequency changes.NeuroReport 12, 2583–2587.

Zatorre, R.J., Belin, P., 2001. Spectral and temporal processing in humanauditory cortex. Cerebr. Cortex 11, 946–953.

Zatorre, R.J., Belin, P., Penhune, V.B., 2002. Structure and function ofauditory cortex: music and speech. Trends Cogn. Sci. 6, 37–46.


processing abstract auditory features in the human auditory cortex

Documents