NeuroImage 59 (2012) 3701–3712

Contents lists available at SciVerse ScienceDirect

NeuroImage

j ourna l homepage: www.e lsev ie r .com/ locate /yn img

Chronometry of word and picture identification: Common andmodality-specific effects

Leen Van Doren a, Maarten Schrooten a,b, Katarzyna Adamczuk a, Patrick Dupont a, Rik Vandenberghe a,b,⁎a Laboratory for Cognitive Neurology, Katholieke Universiteit Leuven, Belgiumb Neurology Department, University Hospitals Leuven, Belgium

⁎ Corresponding author at: Neurology Department,Herestraat 49, 3000 Leuven, Belgium. Fax: +32 16 3444

E-mail address: rik.vandenberghe@uz.kuleuven.ac.b

1053-8119/$ – see front matter © 2011 Elsevier Inc. Alldoi:10.1016/j.neuroimage.2011.11.068

a b s t r a c t

a r t i c l e i n f o

Article history:Received 10 June 2011Revised 8 November 2011Accepted 23 November 2011Available online 4 December 2011

Keywords:ERPP3Visual awareness negativityDm

Based on a previous fMRI connectivity analysis, we previously proposed that long-distance connections be-tween left inferior frontal sulcus and left occipitotemporal sulcus mediate access to visual short-term mem-ory both for written words and pictures enhancing conscious perception and successful encoding in anamodal manner. Using a 64-channel event-related potential electrode system in 19 young cognitively intactvolunteers, we determined the chronometry of common and input-modality specific effects of word and pic-ture identification and subsequent memory retrieval. Stimulus durations were calibrated per subject, modal-ity and run so as to reach a 50% positive identification report. The earliest main effect of a positiveidentification report occurred between 180 and 200 ms, was common for both input-modalities, had a pos-itive polarity and was located at around CPz. This effect was followed between 270 and 450 ms by additionalcommon positive-polarity effects at centrofrontal electrode sites and by common negative effects at P7/P8,TP7/TP8 and T8. Each of the later effects was closely associated not only with identification but also with sub-sequent memory retrieval. The earliest input-modality specific effect of conscious identification that wedetected occurred from 280 till 440 ms at P8. Our findings are in line with a model where the initial stagesof perceptual identification and visual short-term memory access rely on long-distance connections thatare shared between written words and pictures.

© 2011 Elsevier Inc. All rights reserved.

Introduction

The substrate for shared versus input-modality specific processingof meaning of written words versus pictures has received a great dealof attention, both in patient lesion studies (Chertkow et al., 1997; Rappet al., 1993; Rogers and McClelland, 2004) and in functional imagingof the intact brain (Buckner et al., 2000; Thierry and Price, 2006; VanDoren et al., 2010; Vandenberghe et al., 1996; Vandenbulcke et al.,2006; Vandenbulcke et al., 2007; Wagner et al., 1997). Many tasksthat have been used to study commonality between cognitive proces-sing of written words and pictures have required at least some degreeof explicit task-related retrieval of the associations and meaning ofthe referent. Part of the overlap may therefore be attributable to ex-plicit strategic processes rather than word and picture identificationper se.

In a recent fMRI study (Van Doren et al., 2010) we addressed thisquestion using a method adopted from the consciousness researchfield (Bar et al., 2001; Carmel et al., 2006; Kanwisher, 2001; Maroiset al., 2004; Wilenius-Emet et al., 2004). At brief stimulus durations,conscious stimulus identification is a probabilistic process. For each

University Hospitals Leuven,285.

e (R. Vandenberghe).

rights reserved.

individual and each input modality, we selected a duration of wordor picture presentation so that subjects reported conscious identifi-cation of the stimulus only with a probability of 50% across items(Carmel et al., 2006; Marois et al., 2004; Wilenius-Emet et al., 2004)and also within items across subjects. In this way sensory input wasmatched between events that were positively associated with percep-tual identification and events in which processing remained belowthe conscious identification threshold. Subjects had to indicate by keypress whether they had been able to identify the stimulus. Any com-monality therefore between cognitive processing of words and picturescould not be attributed to a common verbal output pathway or explicitcategorical or other semantic judgments. We verified the reliability ofthe identification report by means of surprise subsequent memory re-trieval. For stimuli which subjects had reportedly identified, subsequentmemory retrieval success was significantly higher than for stimuliwhich they denied to have identified (Van Doren et al., 2010). Forboth words and pictures, two regions were associated with a positiveidentification report as well as with subsequent memory retrievalsuccess: the middle third of the left inferior frontal sulcus (IFS) andthe border between posterior and middle third of the left occipitotem-poral sulcus (OTS) (Van Doren et al., 2010).

In a next step (Van Doren et al., 2010) we compared 3 models thatspecified how these 2 amodal regions, left IFS and left OTS, may inter-act with word- or picture-specific systems defined by the contrast

3702 L. Van Doren et al. / NeuroImage 59 (2012) 3701–3712

between written words and pictures. The word-specific system con-sisted of the left posterior superior temporal sulcus (STS) and inferiorfrontal gyrus (IFG), the picture-specific system of the middle occipitalgyrus bilaterally and the left collateral sulcus. In each of the models,left IFS and left OTS were reciprocally connected with each other. Inone model, both amodal nodes, the left IFS and left OTS, were recipro-cally connected with the language- and picture-specific systems. Inthe second model the connectivity between IFS and the input-modality specific systems was entirely mediated through left OTS.According to this model only OTS was directly connected with thelanguage- or picture-specific systems. A third model was analogousto the second but the roles of OTS and IFS were reversed (Van Dorenet al., 2010). A Bayesian comparison using dynamic causal modeling(DCM) was in favor of the second model (Van Doren et al., 2010).From a functional perspective, this winning DCM model led us to thehypothesis that long-distance loops between left IFS and OTS en-hance perceptual processing and feed into the systems specializedfor word and picture processing.

In the framework of the Neural Theory of Visual Attention (NTVA)(Bundesen, 1990; Bundesen and Habekost, 2008; Bundesen et al.,2005), we attributed the OTS and IFS activations to visual short-term memory (VSTM) access. If perceptual units gain access toVSTM, the consciously perceived unit is made available (‘broad-casted’) to other cognitive brain circuits, such as declarative mem-ory (Baars, 1988; Baars, 2002) or phonological retrieval (Jackendoff,2007) systems. Once visual categorizations have been encoded inVSTM (i.e. retained in feedback loops of the VSTM system (Bundesenand Habekost, 2008)), the contents of VSTM can be reported verballyor otherwise expressed through any voluntary output mechanisms:they can be named, conceptualized, propositionally thought about andencoded into long-term memory. We postulated that OTS and IFSreflected access to short-termmemory for bothwords and pictures, en-hancing detailed perceptual analysis, conscious identification and theefficiency of episodic memory encoding.

If long-distance loops between left OTS and left IFS mediate VSTMaccess and increase the probability of conscious perception of bothwords and pictures, we would expect to see amodal effects associatedwith conscious stimulus identification at a relatively early processingstage, before or simultaneously with input-modality specific effectsof conscious perception. Alternatively, if conscious perception arisesfrom activity within input-modality specific systems which subse-quently converge onto amodal regions such as IFS, early modality-specific effects of conscious perception would be expected to precedeamodal effects. We tested this empirically using ERP. This provided uswith timing information on the amodal and modality-specific effectsin the exact same paradigm as we used before in fMRI (Van Dorenet al., 2010).

Materials and methods

Subjects

Twenty-four healthy native Dutch speakers (13 women, 11 men,between 18 and 25 years of age) participated. All participants werestrictly right-handed (Oldfield, 1971), free of psychotropic and vaso-active medication and without neurological or psychiatric history.They all gave written informed consent in accordance with the Decla-ration of Helsinki. The experiments were approved by the EthicsCommittee, University Hospitals Leuven.

Stimuli and tasks

Subjectswere sitting 75 cm in front of a 19 inch screen (1280×1024pixels, refresh rate of 75 Hz). The mean luminance of the stimuli was162 cd/m2. Subjects first received 3 runs of 128 trials each: Subjectshad to respond by key press whether they had been able to identify

the briefly presented stimuli (see below). These trials will be calledthe incidental encoding trials. Following the encoding runs, 2 surpriseforced-choice recognition runs were administered. These will be calledthe retrieval runs.

During incidental encoding, each trial consisted of a forward mask(duration 200 ms), a written word (duration xwordms) or a picture(duration xpicturems) and a backward mask (200 ms), followed by afixation point which remained on the screen for a variable period ran-domized between 2500 and 6000 ms till the next trial started (Fig. 1).Each run contained 40 picture trials and 40 word trials. Half of thesubjects held the response box in their right hand, half in their lefthand. During the encoding runs subjects were instructed to selectone of two buttons depending on whether or not they had beenable to identify the word stimulus as a real-world stimulus. Theywere instructed to respond within the interval before a next trialcould come up at the earliest, i.e. within 2500 ms. The variablesxword and xpicture were calibrated for each individual, each modality(word versus picture) and each run so as to reach a proportion of ap-proximately 50% affirmative responses per individual and modality.For each individual, the starting values for the word and picture dura-tions were determined on the basis of the percentage positive re-sponses during a prior test run using a training set of stimuli thatwas not used for ERP. After each of the runs we again checked thepercentage positive reports per modality. If it was below 37.5% orabove 62.5%, the presentation duration for that modality was length-ened or shortened, respectively, with 13.3 ms in the next run. Afterthe backward mask there was a variable interval between 2500 msand 6000 ms (on average 4250 ms) before the start of the next trial.

Each run contained 8 catch trials (4 foil word and 4 foil picturetrials). These catch trials were introduced in order to measure moreobjectively the level of detail at which subjects processed the stimulito reach their decision. During these trials, we presented chimaericstimuli that did not refer to a real-world entity. They were constructedby dividing a word or a picture in halves and by combining the wordor picture halves into chimaeric word or chimaeric picture stimuli, pre-serving word length and continuity of picture outline, respectively.Since these foil stimuli did not belong to the real world, subjects hadto respond negatively to the foils (similarly to the object decisiontask (Riddoch and Humphreys, 1993)).

Each run additionally contained 40 mask-only events per run.Subjects were instructed to respond negatively to the mask-only tri-als. In half of the mask-only trials the interval between the forwardand the backward mask matched that of the picture trials, in theother half it matched that of the word trials. Together with theword (n=40), the picture (n=40) and the catch trials (n=8), thisresulted in a total of 128 trials per run. A negative key press was re-quired when subjects had been unable to identify the word or picture,when they had identified the stimulus as an entity that did not belongto the real world (catch trials), and in mask-only trials.

The 3 encoding runs were followed by the retrieval runs. Half ofthe entities were from the preceding incidental encoding runs(‘old’) and half were novel. Words and pictures were presented fove-ally (stimulus duration 500 ms) in separate lists and this was coun-terbalanced between subjects. Subjects had to indicate by key presswhether the stimulus had been presented in the preceding encodingruns. The response windowwas identical to that of the encoding runs.

Our stimulus pool consisted of 240 concrete entities (108 bio-logical, 132 non-biological). Entities were pseudorandomly allocatedfor each subject to be used either during the 3 encoding runs or asnovel stimuli during the forced-choice recognition phase only. Foreach entity this was counterbalanced across subjects. During incidentalencoding, a given entity was used during one run only per individual,once as a word and once as a picture. Half of the entities were shownfirst as a word and in one of the later trials as a picture and half viceversa. For each entity, themodality of first presentation was counterba-lanced across subjects.

Fig. 1. Stimuli and tasks. For each subject, run and modality, the variable x is titrated so as to reach a proportion of approximately 50% positive responses.

3703L. Van Doren et al. / NeuroImage 59 (2012) 3701–3712

Word length was two to six letters (mean=4.5, s.d.=1.0) and 1to 3 syllables (mean=1.4, s.d.=0.5), word size 0.34 to 2.29 visualdegrees, the base-10 logarithms of the word frequencies (per million)ranged from 0.0 to 3.0 (mean=1.2, s.d.=0.7) (Baayen et al., 1993).Picture size was 0.38 to 2.98 visual degrees. Pictures were taken fromthe Snodgrass–Vanderwart set (Snodgrass and Vanderwart, 1980) butonly if their names did not exceed an upper limit of 6 letters inlength. The pictures were supplemented by outlines of concrete en-tities that were closely similar in appearance to the Snodgrass–Vanderwart set. The forward and backward masks were chosenpseudorandomly out of a pool of 40 different masks constructed byscrambling the pictures.

Behavioral findings

Behaviorally, the reaction times of the identification responses topictures or words during the encoding runs were analyzed usingrepeated-measures analysis of variance (ANOVA) with input modality(words or pictures) and response (affirmative or negative) as within-subject factors.

In the forced-choice recognition phase of the subsequent memoryretrieval experiment, overall recognition accuracy was quantified as d′(MacMillan and Creelman, 1990). We also compared d′ between olditems that subjects had identified during the preceding identificationruns and old items that they had failed to identify (Student's t test).Furthermore, we compared the true-positive response rate to olditems that subjects had failed to identify during the preceding identifi-cation runs with the false-positive rate to novel items (Student's t test).

ERP recording

EEG was acquired during the encoding runs using a WaveGuard(Advanced Neuro Technology (ANT), Enschede, The Netherlands)electrode systemwith 64 channels configured using the 10/20 systemof electrode placement (Ag/AgCl electrodes). For each channel, im-pedance was lower than 5 kΩ. Following recording, data were writtento file after re-referencing to a pair of linked electrodes placed on themastoid processes. The data were sampled at a rate of 512 Hz andstored for offline analysis. All channels were amplified using an Ad-vanced Source Analysis (ASA)-lab amplifier (ANT, Enschede, TheNetherlands) and bandpass filtered from 0.01 to 40 Hz using ASAsoftware.

ERP analysis

ERP data were analyzed using EEGLAB (Swartz center for compu-tational neuroscience, http://sccn.ucsd.edu/eeglab/). An epoch wasdefined as the period starting 200 ms prior to onset of the forwardmask and ending 1400 ms later. The period from 200 to 50 ms prior

to onset of the forward mask was used for baseline correction. Epochscontaining signal deviations with amplitudes below−75 μV or above75 μV were excluded prior to averaging. Blink artefacts were removedusing second-order blind identification (SOBI) (20 sources and usinga blink signature). Subjects were included in the analysis only if atleast a total of 15 trials for each of the 4 main conditions (positive ver-sus negative identification report for words or pictures) were avail-able for further analysis after preprocessing and 10 trials/run foreach of the two mask-only conditions. Nineteen subjects fulfilledthis criterion.

Primary analysisAverage ERPs were calculated for each of the word and picture

conditions for each run after subtraction of the responses evoked bythe mask-only trials for the respective modality. Per subject asample-weighted average was calculated over runs (John et al.,2001). The weight was based on the relative number of acceptedepochs of a given type in that run. For each subject the following con-trasts were calculated:

• Main effect of identification report: (words to which subjectsresponded positively−words to which subjects responded nega-tively)+(pictures to which subjects responded positively−picturesto which subjects responded negatively), and inversely

• Main effect of input modality: words minus pictures, and inversely• Interaction effect between identification report and input modality.

We conducted for each time point and for each electrode positiona 2×2 factorial ANOVA of the ERP sample voltages with identificationreport (2 levels: positive vs negative) and input modality (2 levels:word vs picture) as within-subject factors. Inference was based on asignificance threshold of Pb0.05 corrected for nonsphericity (Geis-ser–Greenhouse correction) and for the number of electrodes (Bon-ferroni correction).

Our design also allowed us to sort trials based on subsequentmemory retrieval rather than a positive identification report, as amore objective criterion for sorting events. The analysis was other-wise identical to that described above.

If main or interaction effects were significant, we conducted posthoc analyses of the simple effects using a statistical threshold ofPb0.05 uncorrected.

Secondary analysesIn a secondary analysis we recalculated the contrasts of our main

analysis excluding picture or word items which were identified byless than 15% or more than 85% of the subjects.

A positive identification report and subsequent memory retrievalsuccess were closely related. In order to disentangle the effect of apositive identification report from that of subsequent memory re-trieval, we conducted a secondary analysis. We restricted the analysis

3704 L. Van Doren et al. / NeuroImage 59 (2012) 3701–3712

to those trials to which subjects gave a positive identification reportand contrasted items which were subsequently recognized withitems which subjects failed to subsequently recognize. In a separateanalysis, we restricted the analysis to those trials to which subjectsgave a negative identification report and contrasted items that weresubsequently recognized with items that were missed.

The secondary analyses were restricted to electrodes that showedsignificant effects for the primary analysis. For the secondary analysis,the significance threshold was set at an uncorrected Pb0.05.

Relationship with fMRI foci

To determine the correspondence between amodal and modality-specific effects of conscious perception in the ERP data and similareffects obtained in the previous fMRI experiment (Van Doren et al.,2010), we conducted a current source density analysis. Based onthe differential ERP distribution, the standardized low resolutionbrain electromagnetic tomography (sLORETA) software was used(http://www.uzh.ch/keyinst/loreta.htm) to compute the cortical three-dimensional distribution of current density (Pascual-Marqui, 2002).In sLORETA, computations are made in a realistic head model (Fuchset al., 2002) in Montreal Neurological Institute (MNI) space with thethree-dimensional solution restricted to cortical gray matter. The in-tracerebral volume is partitioned in 6239 voxels at 5 mm spatial reso-lution. sLORETA images represent the standardized electric activity ateach voxel as the magnitude of the estimated current density.

Results

Behavioral data

The calibration procedure yielded an average word presentationduration of 26.9 ms (s.d. 8.4) and an average picture presentation du-ration of 50.1 ms (s.d. 11.8) across runs and subjects (F(1,18)=248.8,Pb10−10). Subjects responded positively to the words in 56% (s.d. 13)and to the pictures in 63% (s.d. 12) of trials (t(18)=2.12, Pb0.05).Omissions occurred in 0.74% of the words and 0.81% of the pictures(t(18)=0.30, P=0.77). Subjects gave a positive response to 25.9%(s.d. 17.0) of picture foils, and to 26.9% (s.d. 17.5) of word foils (t(18)=0.18, P=0.86). They provided a false-positive response tomask-only trials in 0.75% (range 0–3.74%) of instances.

Reaction times were significantly faster when subjects respondedaffirmatively compared to negatively (F(1,18)=21.5, Pb0.0002)(Table 1). There were no significant differences between modalities(F(1,18)=2.96, P=0.10) and no significant interaction effects (F(1,18)=0.004, P=0.95).

During the forced-choice recognition phase, the true-positiverecognition rate, i.e. the proportion of old stimuli that subjects recog-nized correctly, was 45.2% (s.d. 11.9) for words and 47.1% (s.d. 12.5)for pictures (t(18)=0.41, P=0.85). The false-positive recognitionrate, i.e. the proportion of novel stimuli to which subjects responded

Table 1Incidental encoding runs: columns 2–3: reaction times (in ms) (mean and standard de-viation). Columns 4–5: number of trials included in the primary analysis averaged overthe 19 subjects who participated (absolute numbers) (mean and standard deviation).

Reaction times Number of trialsretained persubject

Mean S.d. Mean S.d.

Positive report for words 884 136 57 12Negative report for words 1023 200 47 18Positive report for pictures 910 137 64 14Negative report for pictures 1046 160 38 15Mask events (word duration) 777 135 52 7Mask events (picture duration) 771 139 51 7

as if they had been presented in the previous runs, was 20.0% (s.d.12.6) for words and 20.4% (s.d. 10.6) for pictures (t(18)=0.15,P=0.88). The true recognition rate differed significantly between oldstimuli to which subjects had responded positively in the precedingruns (60.4%, s.d. 12.9) and old stimuli to which subjects had respondednegatively (25.3%, s.d. 10.2) (Student's t(18)=15.6, Pb10−11). Thiswas confirmed by analysis of the sensitivity measure d′ (MacMillanand Creelman, 1990) (items to which subjects had previouslyresponded positively: d′=1.35, s.d.=0.93; items to which subjectshad previously responded negatively: d′=0.36, s.d.=0.98, Student'st(18)=14.1, Pb10−10). We also determined whether old stimuli towhich subjects had given a negative identification report were rec-ognized differently from novel items. The true-positive responserate for old stimuli to which subjects had given a negative identifica-tion report (25.3%, s.d. 10.2, see above) was higher than the falsepositive response rate for novel stimuli (19.8%, s.d. 11.1) (t(18)=1.87, P=0.08).

Primary ERP analysis

The average number of epochs per subject included in the analysisis listed in Table 1.

Main effect of identification reportThe spline voltage map for the main effect of a positive identifica-

tion report and the main effect of subsequent memory retrieval areshown in Figs. 2A and D, respectively.

At 180–200 ms an early effect with positive polarity occurredat CPz/C1/C2 for a positive versus a negative identification report(Figs. 3A, C, D; Table 2). The association with subsequent memoryretrieval did not reach significance (Table 2). The simple effect reachedsignificance only for words (Table 3). Inspection of the ERP indicatedthat a similar trend was present for pictures (Figs. 3A, C, D).

At a later interval, between 330 and 410 ms, slightly more an-terior centrofrontal sites (Figs. 3B, E–H; Figs. 4A–B) showed a pos-itive effect associated both with a positive identification report andsubsequent memory retrieval success (Table 2). The simple effectswere significant both for words and for pictures (Fig. 3; Table 3).

In parallel, an effect of a positive identification report with nega-tive polarity from 270 till 450 ms which was also significantly asso-ciated with subsequent memory retrieval success occurred at P7/P8, TP7/TP8 and T8 (Figs. 4C–D, 5E–J; Table 2). At TP8/T8 the simpleeffects were significant for both words and pictures (Table 3).

At parietooccipital, and parietal electrode sites an effect of positiveidentification report with a negative polarity was seen in the same la-tency range (Figs. 4E, 5B–F; Table 2), which was more pronounced forpictures than for words (Table 3; interaction effect see below).

Finally, between 800 and 900 ms, an effect with a positive polaritywhich was significantly associated with subsequent memory re-trieval, was found at CP4/C4 (Table 2).

Main effect of input modalityERP differences between modalities were widespread and started

as early as 130 ms. Positivity to pictures compared to words occurredfrom 180 ms onwards at occipital, parietooccipital, temporoparietaland parietal electrodes. Negativity for pictures compared to words oc-curred from 130 ms onwards bilaterally at centroparietal and centralelectrodes and spreading to central frontal and frontal electrodes overthe following 50 ms (Fig. 2B).

Interaction effect between identification report and input modalityTo the right, P8 showed a significant interaction between identifica-

tion report and input modality, reaching significance from 320 ms to450 ms (peak at 332 ms, Pb0.0001) (Figs. 2C, 4D, 5F). For picturesthe simple effect of identification report was significant from 280 to440 ms (Pb0.0000005) but not for words (Figs. 2C, 5F; Tables 3 and 4).

Fig. 2. Spline voltage maps. A. Main effect of identification report. Contrast between trials to which subjects gave a positive identification report and trials to which subjects gave anegative identification report. B. Main effect of input modality. Contrast between picture trials minus word trials. C. Interaction between identification report (positive versus neg-ative) and input modality (pictures versus words). D. Contrast between trials associated with subsequent memory retrieval success and trials associated with subsequent memoryretrieval failure. E. Interaction between subsequent memory retrieval (success versus failure) and input modality (pictures versus words). Legend: Color scale indicates the ampli-tude of the voltage maps.

3705L. Van Doren et al. / NeuroImage 59 (2012) 3701–3712

At a later timepoint, between 520 and 530 ms, a significant inter-action occurred at Cz/FC2 (Figs. 3B, H): in this time segment, a posi-tive identification report yielded an effect with an opposite polarityfor words compared to pictures (Table 4).

Overall the interaction effect between subsequent memory re-trieval and input modality was very similar to the interaction effectbetween identification report and input modality (Figs. 2C vs E).

Secondary ERP analyses

We re-analyzed the data excluding the picture and the word itemswhich were identified by more than 85% or less than 15% of subjects.The main effect of identification report was confirmed at a correctedPb0.05 at C2, Cz, FCz/FC1/FC2/Fz/F2, CP6, TP8, T8, P6, P7/8, PO7/8 and O2 (Fig. 6).

In order to verify whether any of the picture-specific effects couldbe explained by the skewed distribution of picture identifiability(Fig. 7) we divided the pictures into sextiles based on the percentageof positive responses to each item. We contrasted the upper sextile ofidentifiability with the lower sextile and looked for any differences atP8. The ERP plot strictly overlapped making it unlikely that thepicture-specific effects of a positive identification report at thesesites were due to between-item differences (Fig. 7).

When we restricted the analysis to those trials to which subjectshad given a positive identification report and compared between tri-als associated with retrieval success versus retrieval failure, the effectat FCz/FC1 (Pb0.02, peak at 348 and 432 ms, respectively) and Cz(Pb0.008, peak at 352 ms) remained significant, as well as the effectsat TP7 (Pb0.05, peak 312 ms), TP8 (Pb0.005, peak 379 ms), T8(Pb0.005, peak 379 ms) and the late effect at CP4 (Pb0.005, peak832 ms).

When we restricted the analysis to those trials to which subjectshad given a negative identification report and compared between tri-als associated with retrieval success versus retrieval failure, the effectat FCz and C1 (Pb0.05, peaks at 324 and410 ms, respectively) and atP7 and TP7 remained significant (Pb0.05, peaks at 283 ms and at363 ms, respectively).

Current source density mapping

To investigate whether the FCz/Cz effect was compatible with aninferior frontal source, we calculated the current source densitymaps from the average ERP signal from 340 to 404 ms (the intervalwhen FCz showed a significant main effect of conscious identifica-tion) (Fig. 8A). As a control, we performed the same analysis in the in-terval −200 to −136 ms. The current source density in the leftinferior frontal region (MNI coordinates −42, 36, 15), described inour previous fMRI analysis (Van Doren et al., 2010), was 1.02 μA/mm2 in the interval 340 to 404 ms, compared to 0.005 μA/mm2 inthe control interval −200 to −136 (Fig. 8B). We then calculatedthe current source density of the average ERP signal in each 100 msinterval between −200 and 1000 ms in this left frontal volume-of-interest to obtain the time course (Fig. 8B).

Discussion

Our study provides novel insights in the time course of amodaland input-modality specific effects of conscious stimulus identifica-tion. The temporal predictions derived from our previous fMRI con-nectivity analysis (Van Doren et al., 2010) were confirmed by thecurrent ERP study: amodal effects of conscious stimulus identifica-tion, i.e. effects that are common between written words and

Fig. 3. Main effect of identification report. Grand average waveforms in each of the 4 conditions at electrode positions CPz (A), Cz (B), C1/2 (C, D), FCz (E), Fz (F) and FC1/2 (G, H).The dotted vertical line corresponds to the onset of the forward mask. The zero timepoint (full vertical line) corresponds to the onset of the word or the picture. Legend: Red: pic-tures, blue: words, full line: positive identification report, dotted line: negative identification report. (For interpretation of the references to color in this figure legend, the reader isreferred to the web version of this article.)

3706 L. Van Doren et al. / NeuroImage 59 (2012) 3701–3712

pictures, occurred relatively early, preceding input-modality specificeffects of conscious stimulus identification (Figs. 3, 5).

Based on the differential effects of conscious identification report,we could distinguish 4 successive time phases: a main effect withpositive polarity and central distribution peaked at 180 ms (Figs. 3A,C, D) and was followed between 330 and 410 ms by a second wavefor both words and pictures, which extended more frontally(Figs. 2A, 3B–H). During this time interval, the effects were associatednot only with a positive identification report but also with subsequentmemory retrieval success (Table 2). A third phase, at around 500 ms,was characterized by an interaction at FC2/Cz between identificationreport and input-modality (Table 4; Figs. 2C, 3B, E). In parallel withthese positive polarity effects, effects with negative polarity occurredat parietal and temporoparietal electrode sites reaching significancefrom 280 ms onwards (Table 2; Figs. 2A, 5E–J), with a significant inter-action between identification report and input-modality at P8 between320 and 450 ms (Table 4; Figs. 2C, 5F). Finally, an amodal differentialeffect predictive of subsequent memory retrieval (Dm) (Friedmanand Johnson, 2000; Neville et al., 1986; Paller and Wagner, 2002;

Paller et al., 1987; Paller et al., 1988; Sanquist et al., 1980; Voss andPaller, 2009) occurred between 800 and 900 ms at C4/CP4 (Fig. 2D).

From a sensory perspective, written words and pictures differsubstantially, causing extensive ERP differences even before any ofthe effects of consciousness report (130 ms) (Fig. 2B). Our mainfindings however are based on the main effect of consciousness re-port and the interaction effect between a positive consciousness re-port and input-modality because these effects are free of anysensory confound. Our objective was to define the temporal se-quence of amodal and modality-specific effects of a positive con-sciousness report, not to define the earliest differences in ERPresponses to words versus pictures. We cannot exclude that apartfrom sensory effects, the main effect of input-modality may alsocontain effects related to higher-order levels of processing ofwords or pictures that are unaffected by consciousness report or re-trieval success.

Differences in duration between pictures and words cannot ex-plain any of the effects reported since these differences were sub-tracted out in each of the contrasts of interest. To control for any

Table 2First column: electrode position. Columns 2–5: main effect of a positive versus a neg-ative identification report. Columns 6–9: main effect of subsequent memory retrieval.Bold: Pb0.05 corrected for the number of electrodes. If an electrode showed a signifi-cant effect for one of the contrasts but not for the other, we also evaluated whetherthe other contrast showed any effect at a lower threshold of uncorrected Pb0.05(plain). If that is the case, we report the peak of the effect and the significance level.Abbreviations: vs: versus; N.S.: not significant.

Positive vsnegativeidentificationreport

Subsequentmemoryretrieval vsfailure

+/− Interval Peak Pb +/− Interval Peak Pb

(ms) (ms) (ms) (ms)

CPz + 178–186 182 .0006 + 203 0.01C1 + 180–186 182 .0006 + 205 0.005C2 + 178–197 186 .0005 + 208 0.05FCz + 340–404 391 .0001 + 324–332 430 .0005

+ 420–436FC1 + 383–406 391 .0001 + 430–438 434 .0005FC2 + 340–396 348 .0005 + 430 0.001Fz + 334–365 350 .0005 + 430 0.005F2 + 346–359 348 .0007 + 428 0.01Cz + 344–402 389 .0005 + 420–477 432 .0001C1 + 324–334 436 .0005

+ 424–480CP5 − 330–369 342 .0005 − 344 0.005CP6 − 354–439 393 .0001 − 357 0.005TP7 − 281–398 320 .00001 − 242–371 318 .00001TP8 − 271–449 389 .0000005 − 324–412 369 .000005T8 − 365–418 383 .00001 − 365–416 381 .000005P4 − 385–408 398 .0005 N.S.P5 − 313–369 346 .0005 − 283 0.005P6 − 289–441 389 .000005 N.S.P7 − 283–389 346 .00005 − 238–346 244 .0005P8 − 283–438 389 .0000005 − 320–350 336 .0005PO6 − 387–404 396 .0005 N.S.PO7 − 311–363 320 .00005 − 318 0.001PO8 − 348–404 393 .0001 N.S.O2 − 348–406 398 .0005 N.S.CP4 + 832 0.001 + 820–838 832 .0005C4 + 826 0.005 + 801–900 871 .0005FP1 − 760 0.005 − 824–838 828 .0005FPz − 824–836 828 .0005

Table 3Simple effects for contrast between positive and negative identification report for thoseelectrode sites where identification report had a significant main effect. The interval isonly mentioned for electrode positions that reach a corrected significance threshold.Bold: Pb0.05 corrected. Standard font: Pb0.05 uncorrected.

Pos. vs neg. forwords

Pos. vs neg. forpictures

+/− Interval Peak Pb +/− Interval Peak Pb

(ms) (ms) (ms) (ms)

CPz + 178 0.005 N.S.C1 + 180 0.005 N.S.C2 + 178 .001 N.S.FCz + 346–359 352 .0005 + 396 0.02FC1 + 385 0.005 + 398 0.05FC2 + 348 0.001 + 396 0.05Fz + 352 0.005 + 334 0.01F2 + 352 0.005 + 355 0.05Cz + 348 0.005 N.S.CP5 − 336 0.05 − 342 0.01CP6 − 293 0.005 − 395 0.001TP7 − 340 0.005 − 281–395 318 .0001TP8 − 285–297 291 .0005 − 271–449 404 .00000005

− 385–395T8 − 369–385 379 .00001 − 391–418 408 .00005P4 N.S. − 393 0.01P5 − N.S. − 314 0.005P6 − N.S. − 324–404 361 .0001P7 − 285 0.005 − 303–389 348 .0001P8 − 293 0.01 − 283–438 359 .0000005PO6 N.S. − 387–404 398 .0005PO7 N.S. − 313–352 326 .0005PO8 N.S. − 348–404 363 .000005O2 − 180 0.01 − 348–406 359 .00005

3707L. Van Doren et al. / NeuroImage 59 (2012) 3701–3712

effects related to the mask onset or offset, we also systematically sub-tracted the response to the mask-only trials before analyzing anyother effects.

Picture trials were more frequently associated with a positiveidentification report than word trials. The main effect of identificationreport is not affected by this difference: the terms used to calculatethe main effect are the 4 average ERPs, one for each of the 4 cells ofthe 2×2 factorial design, thus removing any effect from differencesin absolute number of trials between any of the 4 conditions. Weapplied a minimum criterion for the number of trials available tocalculate these averages, excluding subjects when this criterion wasnot met.

We strived for a 50% identification rate per subject across allstimuli for each modality (words or pictures). Ideally, this shouldbe true also per stimulus. For words (Fig. 7A) the distribution wasspread relatively evenly around 50% but for pictures the distribu-tion was skewed (Fig. 7B). Since the interaction effect at P8 wasmainly driven by the picture stimuli (Fig. 5F), we had to excludethat the picture-specific effects of identification report at theseelectrode sites were a consequence of stimulus differences betweenpictures to which subjects responded positively versus negatively.We contrasted the set of pictures with highest identifiability againstthe set with lowest identifiability and did not find even a trend in P8(Fig. 7C), the site of the strongest interaction effect (Fig. 5F). We also

re-analyzed the ERP data exclusive of all stimuli which were identifiedby more than 85% or less than 15% of subjects and were essentiallyable to replicate our main findings, including the interaction effect(Fig. 6D). The interaction effect between input modality and identi-fication report therefore does not appear to be due to between-itemdifferences in identifiability. The latency of the interaction effect alsogoes against a purely exogenous stimulus-driven effect.

The earliest effect of a positive identification report occurred atcentral electrode sites around 180 ms (Figs. 3A, C, D). Central effectswith a similar latency between 180 and 200 ms have been describedbefore when stimuli are presented for 20 ms and subjects had to re-port an animate–inanimate judgment by key press (Thorpe et al.,1996) and also during the attentional blink paradigm when subjectsdetect the second target compared to no detection (Sergent et al.,2005). According to our data, the ERP effect at 180 ms was moreclosely associated with identification than with subsequent memoryretrieval (Table 2). For this reason, we speculate that the early centralERP effects around 180 ms (Figs. 3A, C, D) reflect processes of atten-tional selection and/or stimulus identification (Xu, 2009) precedingaccess to VSTM.

A positive-polarity wave between 330 and 410 ms was stronglyassociated not only with a positive consciousness report for bothwords and pictures but also with subsequent memory retrieval(Table 2; Fig. 3). An ERP effect very similar in latency and distribution(Fabiani and Donchin, 1995; Fabiani et al., 1986; Fabiani et al., 1990;Karis et al., 1984) has been linked to subsequent memory retrievalbefore. It occurs after stimulus evaluation is completed (Kutas et al.,1977) and has been related to perceptual identification or ‘initialencoding’ (Karis et al., 1984), as opposed to elaborative encoding(Fabiani and Donchin, 1995; Fabiani et al., 1990). Its amplitude isproportional to the amount of change that is required in workingmemory by the environmental events (Donchin and Coles, 1988;Karis et al., 1984). The updating of working memory refers to theloading of new stimulus representations into working memory

Fig. 4. Temporal comparison between the centrofrontal and occipitotemporal, amodaland modality-specific effects. Differential waveforms obtained from the contrast of apositive minus a negative identification report for words and pictures (A–C) and forpictures (D–E). A. FCz. B. Cz. C. P7. D. P8. E. PO8. The onset of the divergence of thewaveforms is approximately simultaneous for the differential waveforms illustrated.Legend: Green filling: significance threshold uncorrected Pb0.05. Blue line: significancethreshold Pb0.05 corrected for the total number of electrodes tested. (For interpreta-tion of the references to color in this figure legend, the reader is referred to the webversion of this article.)

3708 L. Van Doren et al. / NeuroImage 59 (2012) 3701–3712

(Donchin and Coles, 1988) and is closely related to what is called‘access to VSTM’ in NTVA (Bundesen and Habekost, 2008). Sepa-rate studies have associated a similar P3 effect with visual aware-ness (Koivisto and Revonsuo, 2010; Lamy et al., 2009; Melloni et al.,Jan, 2011). The current experiment brings these two processes,conscious identification and subsequent memory retrieval success,together within the same study. The rationale for studying themwithin the same experiment is the central role of VSTM access forboth functions (Bundesen and Habekost, 2008). The ERP effect of

subsequent memory retrieval (Fabiani and Donchin, 1995; Fabianiet al., 1986; Fabiani et al., 1990; Friedman and Johnson, 2000; Karis etal., 1984; Neville et al., 1986; Paller, 1990; Paller et al., 1987; Paller etal., 1988; Sanquist et al., 1980; Voss and Paller, 2009) and the effectof conscious perception (Koivisto and Revonsuo, 2010; Lamy et al.,2009; Melloni et al., Jan, 2011; Wilenius and Revonsuo, 2007;Wilenius-Emet et al., 2004) have been studied extensively butlargely in separation from each other. We compared these two pro-cesses within one experiment allowing for direct comparisons. Cog-nitively, according to NTVA (Bundesen and Habekost, 2008), thecommon component is access to VSTM, and, electrophysiologically,a centrofrontal positive-polarity effect at 330–410 ms (Fig. 3).

One of the main purposes of the current ERP study was to obtaintiming information using the same paradigm that we had alreadystudied in great detail using fMRI (Van Doren et al., 2010). The fMRIstudy revealed the middle third of the left IFS as one of the amodalregions involved in conscious stimulus identification as well as sub-sequent memory retrieval (Van Doren et al., 2010). The left IFSeffect therefore has close functional similarities with the ERP ef-fect at 330–410 ms reported here (Table 2; Fig. 3). The left IFS focus(−42, 36, 15) identified by the fMRI study (Van Doren et al., 2010)overlapped with one of the sources contributing to the Cz/FCz/FC1 ef-fect according to the current source density analysis (Fig. 8). Our ERPexperiment therefore provides novel information on when IFS con-tributes to conscious stimulus identification during visual cognitiveprocessing. A same or nearby IFS focus has been implicated in visualawareness using fMRI (−46, 48, 14) (Lau and Passingham, 2006) aswell as in subsequent memory retrieval (−42, 36, 15) according toa dual pulse transcranial magnetic stimulation study (Machizawa etal., 2010). Activation of this IFS region by the oddball effect which isknown to elicit a P3 effect is also in agreement with our combinedERP and fMRI findings (Kirino et al., 2000). Taken together, thesefindings provide converging evidence for a critical role of the mid-dle third of IFS in conscious stimulus identification and subsequentmemory retrieval at a time interval between 330 and 410 ms post-stimulus onset.

In parallel with the Cz/FCz/FC1 effect, left and right occipitotem-poral electrode positions showed a negativity linked with consciousperception and subsequent memory retrieval both for words and pic-tures (Table 2; Fig 5). Previous studies have revealed a negative po-larity effect with a right-sided dominance similar to the currenteffect (Melloni et al., Jan, 2011; Wilenius and Revonsuo, 2007) labeledthe ‘visual awareness negativity’ (VAN) (Koivisto and Revonsuo, 2010).In a study using stimuli very similar to ours, its latency was around300–460 ms (Wilenius and Revonsuo, 2007). An important aim of thecurrent ERP study was to temporally compare amodal effects ofconscious identification with input-modality specific effects of con-scious identification. Electrode site P8 showed the earliest effect ofinteraction between input modality and identification report thatwe could detect (Fig. 5F). Critically, this effect happened between320 and 450 ms, clearly after the earliest effect of identification re-port at CPz (180 ms). Interestingly, close inspection of the wave-forms at P8 and PO8 suggest that the interaction effect is precededby a negative effect of consciousness report at around 180 ms thatis shared between words and pictures and parallels the positiveamodal effect at CPz with the same latency (Figs. 5D, F). This couldsuggest a first wave related to attentional selection regardless ofinput-modality followed by a wave related to the VSTM phase thatis picture-specific at these electrode sites. In our original fMRI study,we obtained a near-significant interaction between input-modalityand subsequent memory retrieval in the right middle occipital gyrus(Van Doren et al., 2010). Lesion studies have also suggested that rightoccipototemporal cortex is involved in picture-specific cognitive pro-cesses, for instance related to mnemonic retrieval of features of realobjects (Vandenbulcke et al., 2006) and conceptual processing of non-verbal stimuli (Thierry and Price, 2006).

Fig. 5. Main effect of identification report. Grand average waveforms in each of the 4 conditions at left-sided (first column) and right-sided occipitotemporal electrode positions(second column). A, B. O1/O2. C, D. PO7/PO8. tect E, F. P7/P8. G, H. TP7/TP8. I, J. T7/T8. Legend: see Fig. 3.

Table 4Interaction effects between input-modality and identification report. Bold: Pb0.05corrected. Abbreviations: ID: identification report.

Interaction between input-modality and identification

Interval Peak Pb

(ms) (ms)

P8 320–338 332 .0001443–451

FC2 523–533 529 .0004Cz 523–533 527 .0003

3709L. Van Doren et al. / NeuroImage 59 (2012) 3701–3712

We also found an interaction between input-modality and identi-fication report at 523–533 ms at Cz/FCz (Fig. 3B), following the strongamodal effect at this location. In our previous fMRI experiment, wefound subthreshold interaction effects between input-modality andconscious perception in the left frontal operculum and left superiortemporal sulcus (STS) (Van Doren et al., 2010). Speculatively, the lateinteraction effect could possibly reflect word-specific cognitive proces-sing following VSTM access, e.g. phonological retrieval (Jackendoff,2007).

The amodal effects around 300 ms were also predictive of subse-quent memory retrieval when the analysis was restricted to trials to

Fig. 6. Event-related responses after exclusion of items to which more than 85% or less than 15% of subjects gave a positive identification report. Selection of electrodes based onprimary analysis. A, B. PO7/PO8. C, D. P7/P8. E, F. TP7/TP8. H. Cz. H. FCz. Legend: see Fig. 3.

3710 L. Van Doren et al. / NeuroImage 59 (2012) 3701–3712

which subjects had given a negative consciousness report. This mayreflect processes of stimulus identification occurring below thethreshold of conscious perception but sufficient to be encoded intoepisodic memory. Dissociations between subjective awareness andcognitive processing have received a lot of attention in the literature.For instance, subjects may fail to detect a familiar word but beable to guess its identity or lexical status (Merikle and Reingold,1990). Masked priming experiments (Dehaene et al., 2001; Diazand McCarthy, 2007; Forster, 1998; Grainger and Holcomb, 2009;Holcomb and Grainger, 2006; Qiao et al., 2009) also provide clearevidence that there is extraction of meaning subliminally. In pre-vious ERP studies of visual awareness, subjects sometimes guessedcorrectly about the stimulus identity but gave a negative conscious-ness report thereafter (Cul et al., 2007; Lau and Passingham, 2006).As we propose that the amodal effects between 300 and 400 ms re-late to VSTM access, the dissociation between identification andconsciousness report may indicate that perceptual units may winthe race for access to VSTM, be encoded successfully but still failto reach the threshold for a positive consciousness report.

The model that we set out to test with this ERP study was based onlong-distance connections between left IFS and OTSmediating amodal

VSTM access at an early stage of stimulus identification. This systemfor conscious word and picture perception resembles that proposedby Bar et al. which consists of orbitofrontal and ventral occipitotem-poral nodes involved in speeded stimulus identification (Bar, 2003;Bar et al., 2001; Bar et al., 2006). There are however two notable differ-ences: the orbitofrontal node in that model lies anterior to our IFSfocus and, secondly, the long-distance loop in the model proposedby Bar et al. (Bar, 2003; Bar et al., 2001; Bar et al., 2006) is supposedto rely on magnocellular pathways which would imply a faster time-course than we observed. To the best of our knowledge, no empiricaldata are available on the exact timing of the orbitofrontal and occipitaleffects in Bar's model (Bar et al., 2001; Bar, 2003; Bar et al., 2006).

Our findings are in accordance with models where consciousvisual perception relies on feedback loops within large-scale dis-tributed circuits (Dehaene et al., 2006) which are shared betweenwords and pictures. The alternative account, that conscious stimu-lus identification arises from input-modality specific areas, was re-futed. Our combined fMRI and ERP data strongly suggest that oneof these long-distance loops connects inferior frontal and occipito-temporal nodes enhancing access to VSTM in an amodal mannerand facilitating conscious word or picture processing. The DCM

Fig. 7. A, B. The word and picture stimulus set are binned according to the % of subjects that identified the stimulus in this experiment. C. Contrast between the upper and lowersextile of the histogram in P8 the electrode showing a significant interaction effect in the primary analysis.

3711L. Van Doren et al. / NeuroImage 59 (2012) 3701–3712

analysis of the fMRI data did not provide any evidence for directprojections of input-modality specific systems onto left IFS (VanDoren et al., 2010). Instead, according to the DSM, IFS was indi-rectly connected with the input-modality specific systems vialeft OTS. The occipitotemporal ERP effects of conscious stimulusidentification may reflect feedback from frontal sources to sensorycortex.

Fig. 8. Current source density map A. Time window 340–404 ms. This corresponds to the tidensities at −42, 36, 15, the MNI coordinate of the left IFS focus implicated before access t

To conclude, this ERP experiment provides us with novel insightinto the temporal relationship between amodal and input-modalityspecific effects of conscious perception and reveals the close electro-physiological link between conscious stimulus identification andsubsequent memory retrieval success. Together with an earlier fMRIstudy using the same paradigm, our findings corroborate a modelwhere VSTM access is mediated by long-distance connections between

me window when the FCz effect achieves significance B. Time course of current sourceo visual short-term memory.

3712 L. Van Doren et al. / NeuroImage 59 (2012) 3701–3712

inferior frontal and ventral occipitotemporal cortex at a relatively earlystage of cognitive processing both for words and pictures.

Acknowledgments

This work was supported by Research Foundation Flanders (FWO)[G0076.02, G0668.07 to R.V.], K.U. Leuven [OT/04/41, OT/08/056, EF/05/014 to R.V.], and Federaal Wetenschapsbeleid belspo [IUAP P6/29]. We thank Dr. Maarten De Vos for his help with the blink artefactremoval.

References

Baars, B., 1988. A Cognitive Theory of Consciousness. Cambridge University Press,Cambridge, UK.

Baars, B., 2002. The conscious access hypothesis: origins and recent evidence. TrendsCogn. Sci. 6, 47–52.

Baayen, H., Piepenbrock, R., Rijn, H.V., 1993. The CELEX Lexical Database (CD-ROM).Linguistic Data Consortium, Philadelphia, PA.

Bar, M., 2003. A cortical mechanism for triggering top-down facilitation in visual objectrecognition. J. Cogn. Neurosci. 15, 600–609.

Bar, M., Kassam, K., Ghuman, A., Boshyan, J., Schmid, A., Dale, A., Hamalainen, M.,Marinkovic, K., Schacter, D., Rosen, B., Halgren, E., 2006. Top-down facilitationof visual recognition. Proc. Natl. Acad. Sci. 103, 449–454.

Bar, M., Tootell, R., Shacter, D., Greve, D., Fischl, B., Mendola, J., Rosen, B., Dale, A., 2001.Cortical mechanisms specific to explicit visual object recognition. Neuron 29,529–535.

Buckner, R., Koutstaal, W., Shacter, D., Rosen, B., 2000. Functional MRI evidence for arole of frontal and inferior temporal cortex in amodal components of priming.Brain 123, 620–640.

Bundesen, C., 1990. A theory of visual attention. Psychol. Rev. 97, 523–547.Bundesen, C., Habekost, T., 2008. Principles of visual attention. Linking Mind and Brain.

Oxford, UK, Oxford University Press.Bundesen, C., Habekost, T., Kyllingsbaek, S., 2005. A neural theory of visual attention:

bridging cognition and neurophysiology. Psychol. Rev. 112, 291–328.Carmel, D., Lavie, N., Rees, G., 2006. Conscious awareness of flicker in humans involves

frontal and parietal cortex. Curr. Biol. 16, 907–911.Chertkow, H., Bub, D., Deaudon, C., Whitehead, V., 1997. On the status of object con-

cepts in aphasia. Brain Lang. 58, 203–232.Cul, A.D., Baillet, S., Dehaene, S., 2007. Brain dynamics underlying the nonlinear thresh-

old for access to consciousness. PLoS Biol. 5, e260.Dehaene, S., Changeux, J., Naccache, L., Sackur, J., Sergent, C., 2006. Conscious, preconscious

and subliminal processing: a testable taxonomy. Trends Cogn. Sci. 10, 204–211.Dehaene, S., Naccache, L., Cohen, L., LeBihan, D., Mangin, J., Poline, J., Rivere, D., 2001.

Cerebral mechanisms of word masking and unconscious repetition priming. Nat.Neurosci. 4, 752–758.

Diaz, M., McCarthy, G., 2007. Unconscious word processing engages a distributed net-work of brain regions. J. Cogn. Neurosci. 19, 1768–1775.

Donchin, E., Coles, M., 1988. Is the p300 component a manifestation of context updating?Behav. Brain Sci. 11, 357–427.

Fabiani, M., Donchin, E., 1995. Encoding processes and memory organization: a modelof the von restorff effect. J. Exp. Psychol. Learn. Mem. Cogn. 21, 224–240.

Fabiani, M., Karis, D., Donchin, E., 1986. P300 and recall in an incidental memory para-digm. Psychophysiology 23, 298–308.

Fabiani, M., Karis, D., Donchin, E., 1990. Effects of mnemonic strategy manipulation in avon restorff paradigm. Electroencephalogr. Clin. Neurophysiol. 75, 22–35.

Forster, K., 1998. The pros and cons ofmasked priming. J. Psycholinguist. Res. 27, 203–232.Friedman, D., Johnson, R., 2000. Event-related potential (ERP) studies of memory

encoding and retrieval: a selective review. Microsc. Res. Tech. 51, 6–28.Fuchs, M., Kastner, J., Wagner, M., Hawes, S., Ebersole, J.S., 2002. A standardized bound-

ary element method volume conductor model. Clin. Neurophysiol. 113, 702–712.Grainger, J., Holcomb, P., 2009. Watching the word go by: on the time-course of com-

ponent processes of visual word recognition. Lang. Linguist. Compass 1, 128–156.Holcomb, P., Grainger, J., 2006. On the time course of visual word recognition: an

event-related potential investigation using masked repetition priming. J. Cogn.Neurosci. 18, 1631–1643.

Jackendoff, R., 2007. Language, consciousness, culture. Essays on Mental Structure. MITPress.

John, M.S., Dimitrijevic, A., Picton, T.W., 2001. Weighted averaging of steady-stateresponses. Clin. Neurophysiol. 112, 555–562.

Kanwisher, N., 2001. Neural events and perceptual awareness. In: Dehaene, S. (Ed.),The Cognitive Neuroscience of Consciousness. Elsevier Science publishers,Amsterdam, pp. 89–113.

Karis, D., Fabiani, M., Donchin, E., 1984. P300 and memory: individual differences in thevon restorff effect. Cogn. Psychol. 16, 177–216.

Kirino, E., Belger, A., Goldman-Rakic, P., McCarthy, G., 2000. Prefrontal activationevoked by infrequent target and novel stimuli in a visual target detection task:

an event-related functional magnetic resonance imaging study. J. Neurosci. 20,6612–6618.

Koivisto, M., Revonsuo, A., 2010. Event-related brain potential correlates of visualawareness. Neurosci. Biobehav. Rev. 34, 922–934.

Kutas, M., McCarthy, G., Donchin, E., 1977. Augmenting mental chronometry: the p300as a measure of stimulus evaluation time. Science 197, 792–795.

Lamy, D., Salti, M., Bar-Haim, Y., 2009. Neural correlates of subjective awareness andunconscious processing: an ERP study. J. Cogn. Neurosci. 21, 1435–1446.

Lau, H.C., Passingham, R.E., 2006. Relative blindsight in normal observers and the neuralcorrelate of visual consciousness. Proc. Natl. Acad. Sci. U. S. A. 103, 18763–18768.

Machizawa, M.G., Kalla, R., Walsh, V., Otten, L.J., 2010. The time course of ventro-lateral prefrontal cortex involvement in memory formation. J. Neurophysiol.103, 1569–1579.

MacMillan, N., Creelman, C., 1990. Response bias: characteristics of detection theory,threshold theory and non-parametric indexes. Psychol. Bull. 107, 403–413.

Marois, R., Yi, D., Chun, M., 2004. The neural fate of consciously perceived and missedevents in the attentional blink. Neuron 41, 465–472.

Melloni, L., Schwiedrzik, C.M., Mïller, N., Rodriguez, E., Singer, W., Jan 2011. Expectationschange the signatures and timing of electrophysiological correlates of perceptualawareness. J. Neurosci. 31, 1386–1396.

Merikle, P.M., Reingold, E.M., 1990. Recognition and lexical decision without detection:unconscious perception? J. Exp. Psychol. Hum. Percept. Perform. 16, 574–583.

Neville, H., Kutas, M., Chesney, G., Schimidt, A., 1986. Event-related brain potentialsduring initial encoding and recognition memory of congruous and incongruouswords. J. Mem. Lang. 25, 75–92.

Oldfield, R., 1971. The assessment and analysis of handedness: the Edinburgh inventory.Neuropsychologia 9, 97–113.

Paller, K., McCarthy, G., Wood, C., 1988. ERPs predictive of subsequent recall and recog-nition performance. Biol. Psychol. 26, 269–276.

Paller, K.A., 1990. Recall and stem-completion priming have different electrophysiolog-ical correlates and are modified differentially by directed forgetting. J. Exp. Psychol.Learn. Mem. Cogn. 16, 1021–1032.

Paller, K.A., Kutas, M., Mayes, A.R., 1987. Neural correlates of encoding in an incidentallearning paradigm. Electroencephalogr. Clin. Neurophysiol. 67, 360–371.

Paller, K.A., Wagner, A.D., 2002. Observing the transformation of experience intomemory.Trends Cogn. Sci. 6, 93–102.

Pascual-Marqui, R.D., 2002. Standardized low-resolution brain electromagnetic tomog-raphy (sLORETA): technical details. Methods Find Exp. Clin. Pharmacol. 24 (Suppl.D), 5–12.

Qiao, E., Vinckier, F., Szwed, M., Naccache, L., Valabrégue, R., Dehaene, S., Cohen, L.,2009. Unconsciously deciphering handwriting: subliminal invariance for hand-written words in the visual word form area. NeuroImage 49, 1786–1799.

Rapp, B., Hillis, A., Caramazza, A., 1993. The role of representations in cognitive theory:more on multiple semantics and the agnosias. Cogn. Neuropsychol. 10, 235–249.

Riddoch, M., Humphreys, G., 1993. Birmingham Object Recognition Battery. LawrenceErlbaum Associates Ltd., Hove, UK.

Rogers, T., McClelland, J., 2004. Semantic cognition. A Parallel Distributed Approach.Massachusetts Institute of Technology Press, Cambridge, MA.

Sanquist, T.F., Rohrbaugh, J.W., Syndulko, K., Lindsley, D.B., 1980. Electrocortical signsof levels of processing: perceptual analysis and recognition memory. Psychophys-iology 17, 568–576.

Sergent, C., Baillet, S., Dehaene, S., 2005. Timing of the brain events underlying access toconsciousness during the attentional blink. Nat. Neurosci. 8 (10), 1391–1400.

Snodgrass, J., Vanderwart, M., 1980. A standardized set of 260 pictures: norms for nameagreement, image agreement, familiarity and visual complexity. J. Exp. Psychol.Hum. Learn. Mem. 6, 174–215.

Thierry, G., Price, C., 2006. Dissociating verbal and nonverbal conceptual processing inthe human brain. J. Cogn. Neurosci. 18, 1018–1028.

Thorpe, S., Fize, D., Marlot, C., 1996. Speed of processing in the human visual system.Nature 381, 520–522.

Vandenberghe, R., Price, C., Wise, R., Josephs, O., Frackowiak, R., 1996. Functional anatomyof a common semantic system for words and pictures. Nature 383, 254–256.

Vandenbulcke, M., Peeters, R., Dupont, P., Van Hecke, P., Vandenberghe, R., 2007. Wordreading and posterior temporal dysfunction in amnestic mild cognitive impair-ment. Cereb. Cortex 17, 542–551.

Vandenbulcke, M., Peeters, R., Fannes, K., Vandenberghe, R., 2006. Knowledge of visualattributes in the right hemisphere. Nat. Neurosci. 9, 964–970.

Van Doren, L., Dupont, P., De Grauwe, S., Peeters, R., Vandenberghe, R., 2010. Theamodal system for conscious word and picture identification in the absence ofa semantic task. NeuroImage 49, 3295–3307.

Voss, J.L., Paller, K.A., 2009. Recognition without awareness in humans and its implica-tions for animal models of episodic memory. Commun. Integr. Biol. 2, 203–204.

Wagner, A., Desmond, J., Demb, J., Glover, G., Gabrieli, J., 1997. Semantic repetitionpriming for verbal and pictorial knowledge: a functional MRI study of left inferiorprefrontal cortex. J. Cogn. Neurosci. 9, 714–726.

Wilenius, M.E., Revonsuo, A.T., 2007. Timing of the earliest ERP correlate of visualawareness. Psychophysiology 44, 703–710.

Wilenius-Emet, M., Revonsuo, A., Ojanen, V., 2004. An electrophysiological correlate ofhuman visual awareness. Neurosci. Lett. 354, 38–41.

Xu, Y., 2009. Distinctive neural mechanisms supporting visual object individuation andidentification. J. Cogn. Neurosci. 21, 511–518.