spatial frequency requirements for audiovisual speech perception

Copyright 2004 Psychonomic Society Inc 574

Perception amp Psychophysics2004 66 (4) 574-583

When auditory speech perception is impaired by noiseor by a degraded acoustic signal being able to see a talk-errsquos face saying the words significantly increases intelli-gibility In one of the first experimental demonstrationsof this effect Sumby and Pollack (1954) showed that per-ception of spoken words was enhanced across a range ofacoustic signal-to-noise ratios when subjects looked at atalker seated a few feet away This visual contribution tospeech perception has been demonstrated under a varietyof conditions including point-light displays (eg Rosen-blum Johnson amp Saldantildea 1996) animated faces (egMassaro 1998) and degraded visual images (Vitkovichamp Barber 1996) In this study we extended this work bymanipulating the spatial frequency content of the facial

videos in order to investigate the nature of visual imageprocessing during audiovisual speech perception

Understanding how visual information is integratedwith auditory speech information requires a detailedspecification of the nature of the visual speech informa-tion as well as the nature of visual information processingIn recent years some progress has been made in both areasStudies of facial kinematics during speech productionhave indicated that dynamic visual information must below in temporal frequency (see Munhall amp Vatikiotis-Bateson 1998 for a review) For example Ohala (1975)showed that the modal temporal frequency of jaw motionin continuous oral reading is below 5 Hz Although speechmay contain some high-frequency movement compo-nents the time course of opening and closing of the vocaltract for syllables is relatively slow

The primary locus of visual speech information is aroundthe mouth and jaw owing to their principal role in speechsound generation However the motion of articulationspreads across the entire face (Vatikiotis-Bateson Mun-hall Hirayama Kasahara amp Yehia 1996) These motionshave been shown to correlate with the changing acousticspectrum and RMS amplitude of the speech (Yehia Rubinamp Vatikiotis-Bateson 1998) From a perceptual stand-point this distributed facial information has been shownto contribute significantly to intelligibility the more of theface that is visible the greater the intelligibility (egBenoicirct Guiard-Marigny Le Goff amp Adjoudani 1996)

This research was supported by grants from the Natural Sciences andEngineering Research Council of Canada the NIH the National Insti-tute of Deafness and Other Communications Disorders (Grant DC-05774) and the Communication Dynamics Project ATR Human In-formation Science Laboratories Kyoto Japan The authors thank ClareMacDuffee Amanda Rothwell and Helen Chang for assistance in test-ing subjects Martin Pareacute and Doug Shiller made helpful suggestionsabout an earlier draft Correspondence concerning this article should beaddressed to K G Munhall Department of Psychology Queenrsquos Uni-versity Kingston ON K7L 3NC Canada (e-mail munhallkpsycqueensuca)

NotemdashThis article was accepted by the previous editorial teamheaded by Neil Macmillan

Spatial frequency requirements for audiovisual speech perception

K G MUNHALLQueenrsquos University Kingston Ontario Canada

C KROOS and G JOZANATR Human Information Science Laboratories Kyoto Japan

and

E VATIKIOTIS-BATESONATR Human Information Science Laboratories Kyoto Japan

and University of British Columbia Vancouver British Columbia Canada

Spatial frequency band-pass and low-pass filtered images of a talker were used in an audiovisualspeech-in-noise task Three experiments tested subjectsrsquo use of information contained in the differentfilter bands with center frequencies ranging from 27 to 441 cyclesface (cface) Experiment 1 demon-strated that information from a broad range of spatial frequencies enhanced auditory intelligibility Thefrequency bands differed in the degree of enhancement with a peak being observed in a mid-rangeband (11-cface center frequency) Experiment 2 showed that this pattern was not influenced by view-ing distance and thus that the results are best interpreted in object spatial frequency rather than inretinal coordinates Experiment 3 showed that low-pass filtered images could produce a performanceequivalent to that produced by unfiltered images These experiments are consistent with the hypothe-sis that high spatial resolution information is not necessary for audiovisual speech perception and thata limited range of spatial frequency spectrum is sufficient

SPATIAL FREQUENCY AND SPEECH PERCEPTION 575

People are adept at using this rich visual informationand when image quality is degraded visual speech hasbeen shown to be quite robust In one of the first studies ofthis kind Brooke and Templeton (1990) manipulated spa-tial resolution by varying the quantization level of a dis-play showing the mouth region articulating English vow-els Varying the quantization level between 8 8 and128 128 pixels Brooke and Templeton found that per-formance on a silent lipreading task decreased when thedisplay was reduced below 32 32 spatial resolutionmdashfor example 16 16 pixels Similar methods have beenused by C Campbell and Massaro (1997) and MacDonaldAndersen and Bachmann (2000) to study the McGurkeffect and by Vitkovich and Barber (1996) to studyspeechreading of digits Both Campbell and Massaroand MacDonald et al found that the McGurk effect per-sisted to some degree even at low spatial resolutionVitkovich and Barber found no effect of their pixel den-sity manipulation but did find changes in performancewhen the grayscale resolution (ie the number of graylevels) of the images was drastically reduced Most re-cently Thomas and Jordan (2002) used Gaussian blur-ring to study the importance of fine facial detail in visualspeech and face processing In a series of studies theyfound that visual speech perception was not impaireduntil the blurring cutoff was 8 cycles per face or less

All of these studies suggest that a visual contributionto speech perception does not require the extraction ofhigh spatial resolution information from the facial imageThe performance of subjects with point-light displays ofvisual speech and the influence of gaze during audio-visual speech are consistent with this interpretationWhen the facial surface kinematics are reduced to themotions of a collection of light points subjects still showperceptual benefit in an acoustically noisy environment(Rosenblum et al 1996) and can perceive the McGurkeffect (Rosenblum amp Saldantildea 1996) Recently PareacuteRichler ten Hove and Munhall (2003) examined the dis-tribution of gaze during the McGurk effect Their resultsshowed that the eyemouth regions dominated the gazepatterns (see also Lansing amp McConkie 1999 Vatikiotis-Bateson Eigsti Yano amp Munhall 1998) however thestrength of the McGurk effect was not influenced by wherein the face region the subjects fixated More importantPareacute et al demonstrated that the McGurk effect did notdisappear when subjects directed their gaze well beyondthe facial region Thus high-acuity foveal vision doesnot seem to be a requirement for audiovisual integration

Although these studies suggest that the visual informa-tion used for speech perception can be relatively crudethere have been no direct tests of the range of spatial fre-quencies that are critical for audiovisual speech perceptionIt is widely accepted that the perception of visual imagesis carried out in the nervous system by a set of spatial-frequencyndashtuned channels For the purpose of studyingthis processing images can be decomposed into sets ofnonoverlapping spatial frequency bands and studies ofthe recognition of static objects in which such stimuli

have been used indicated that these different spatial fre-quency bands do not contribute equally to the objectidentification process (eg Parish amp Sperling 1991)Perception of such objects as letters and faces appears toinvolve distinct processing of portions of the spatial fre-quency spectrum This may be due to task- or domain-specific processing (Wenger amp Townsend 2000) For ex-ample Vuilleumier Armony Driver and Dolan (2003)reported that the amygdala carries out selective processingof the low spatial frequency portions of faces showingemotional expression whereas the fusiform cortex whichis thought to be more involved in identification is influ-enced more by the high-frequency portion of the facialimage Alternatively selective processing could be de-termined by the properties of a stimulus Solomon andPelli (1994) and Majaj Pelli Kurshan and Palomares(2002) have argued that the identification of text is me-diated by a small portion of broadband spectrummdasha sin-gle channel one to two octaves wide determined by in-formation in the fonts

In this research we manipulated the spatial resolutionof facial images in order to understand the separate con-tributions of different spatial frequency components tovisual speech processing Previous work on static faceprocessing (Gold Bennett amp Sekuler 1999 Naumlsaumlnen1999) has suggested that optimal recognition of faces in-volves a relatively narrow band of spatial frequenciesGold et al for example tested the discrimination of fa-cial identity across a range of bandwidths and spatial fre-quency ranges They observed peak efficiency of faceprocessing with a two-octave band-pass filtered stimuluswith a center frequency of 62 cyclesface (cface seeNaumlsaumlnen 1999 for similar results)

Our extension of this approach to the perception ofspoken language is motivated by two factors First thereis behavioral evidence that static and dynamic facial im-ages convey independent information (eg LanderChristie amp Bruce 1999) and neurological evidence thatdistinct neural substrates process static and dynamic vi-sual speech (Calvert amp R Campbell 2003 R CampbellZihl Massaro Munhall amp Cohen 1997 Munhall Ser-vos Santi amp Goodale 2002) Second the spatial fre-quency range that is important for a given stimulus typemay be tied to the particular task being carried out withthe stimuli (Schyns amp Oliva 1999 Wenger amp Townsend2000) In natural face-to-face conversations a wide rangeof visual tasks are carried out in parallel Listeners per-ceive information about the talkerrsquos identity emotionaland physical state focus of attention and degree of so-cial and conversational engagement as well as linguisticinformation about the utterance Each of these aspects ofcommunication may involve different stimulus proper-ties with information conveyed by different spatial fre-quency ranges (eg Vuilleumier et al 2003)

The experiments reported here extended the study ofthe role of visual stimulus properties to dynamic facialimages that conveyed linguistic information Our exper-imental task speech perception in noise was quite se-

576 MUNHALL KROOS JOZAN AND VATIKIOTIS-BATESON

lective and subjects had no explicit demands to monitorany information beyond the spoken words Thus the aimof the series of experiments was to characterize the vi-sual information properties involved in a single subtaskin communication the visual enhancement of speech in-telligibility In Experiment 1 we used band-pass filteredfacial images in a speech-in-noise task and comparedperformance of different bands against the full-videoand auditory-only conditions In Experiment 2 the dif-ferent band-pass filtered images were tested at differentviewing distances In Experiment 3 low-pass filtered ver-sions of the same audiovisual sentences as those used inExperiments 1 and 2 were tested In combination thesethree experiments allowed us to examine what spatial res-olution is sufficient for audiovisual speech The use ofthe band-pass filtered images permitted tests of the in-formation contained within each spatial frequency bandand provided evidence about the retinal specificity of thevisual information used in visual speech The low-pass fil-tered images provided a direct test of the sufficiency oflow-frequency information in visual speech perception

GENERAL METHOD

StimuliThe Central Institute for the Deaf (CID) ldquoeveryday sentencesrdquo

(Davis amp Silverman 1970) were spoken by a native American En-glish speaker and were recorded on high-quality videotape (Beta-cam SP) For Experiments 1 and 2 the sentences were digitized asimage sequences converted to grayscale and band-pass filteredusing a two-dimensional discrete wavelet transformation (for a de-scription of the application of wavelets to image processing seePrasad amp Iyengar 1997 for details of the specific adaptation of theprocess used here see Kroos Kuratate amp Vatikiotis-Bateson 2002)Five one-octave band-pass filtered sets of sentence stimuli werecreated (see Table 1) with filter center frequencies ranging from 27to 441 cface1 Figure 1 shows single images from the different fil-ter conditions Note that the set of filters tested did not cover thecomplete range of spatial frequency components in the original im-ages The low-frequency cutoff was 18 cface and high-frequencycutoff was 59 cface (Table 1)

The original audio track and a commercial multispeaker babbletrack (Auditec St Louis MO) were recorded to Betacam SP video-tape in synchrony with the original grayscale image sequence andwith each of the five band-pass image sets From the tape a CAVvideodisc was pressed for use in the perception experiments

EquipmentThe videodisc was played on a Pioneer (Model LD-V8000)

videodisc player Custom software was used to control the videodisctrials

ScoringLoose key word scoring (Bench amp Bamford 1979) was carried

out on the subjectsrsquo responses using the standard CID key words(Davis amp Silverman 1970) This scoring method ignores errors ininflection because these errors compound for subsequent words(eg plural nouns and verb agreement) Percentage of key wordscorrectly identified was used as the dependent variable in all thestatistical analyses

EXPERIMENT 1

Static face images can be recognized with only a smallrange of spatial frequencies Although this fact has beendemonstrated numerous times a number of different fre-quencies have been proposed as the critical bands Goldet al (1999) reviewed a broad collection of studies re-porting critical identification bands ranging from 1 to25 cface This wide discrepancy is presumably due totask differences and discriminability of the faces in thetest set Recent studies that have taken into account suchdesign limitations have reported that recognition of fa-cial images and static facial expressions is best for nar-row band (around two octaves) mid-to-low spatial fre-quency images (5ndash20 cface eg Naumlsaumlnen 1999) In thepresent experiment we examined whether the perceptionof dynamic facial images during audiovisual speech showsthe same sensitivity to band-pass filtered images

MethodSubjects Twenty-one native speakers of English were tested All

had normal or corrected-to-normal vision and none reported hear-ing problems or a history of speech or language difficulties

Stimuli Seventy sentences from the full set of 100 CID sentenceswere selected for use in the perceptual testing The CID sentence setis organized in groups of 10 sentences that are balanced for thenumber of key words phonetic content and average duration of theutterance Seven of these sentence groups were used to test theseven audiovisual conditions (full image auditory only and fiveband-pass filter)

Equipment The subjects watched the displays on a 20-in videomonitor (Sony PVM 1910) The acoustic signals were amplifiedand mixed (Tucker-Davis System II) and were played throughspeakers (MG Electronics Cabaret) placed directly below the mon-itor The testing took place in a double-walled sound isolation booth(IAC Model 1204)

Design Each subject participated in a single session consistingof 70 sentences presented in noise There were seven conditions(unfiltered video no-video and five band-pass filter conditions)The 70 sentences were divided into groups of 10 sentences with anequal number of key words Each group of 10 sentences was as-signed to one of the seven stimulus conditions for a subject As-signment of groups of sentences to a condition was counterbal-anced across subjects The presentation of stimuli was randomizedacross condition within subjects

Procedure A small table and chair were positioned 171 cm fromthe monitor and the subjects viewed the stimuli with the head po-sition fixed using a chin- and headrest After the presentation of atrial the subjects verbally repeated as much of the sentence as theycould When the experimenter recorded the response the next trialwas initiated The auditory signal-to-noise level was held constantfor all the subjects Pilot testing was used to find a level that pro-duced auditory-only response accuracy below 50 This was nec-essary to permit the significant increase in intelligibility that oc-curs when visual stimuli are present without ceiling effects

Table 1Spatial Frequency Bands Used in Experiment 1

Bandwidth Center Frequency Center Frequency(cface) (cface) (cyclesdeg of visual angle)Width Width Width

18ndash37 27 03637ndash73 55 07373ndash15 110 14615ndash29 220 29229ndash59 441 585


Results and DiscussionThe percentage of key words correctly identified was

used as the dependent variable As can be seen in Fig-ure 2 the average performance differed across the audio-visual conditions The greatest number of key wordswere identified with the unfiltered visual stimuli whereasthe poorest performance was observed for auditory onlyThe spatial frequency filtered bands showed an invertedU-shaped performance curve with a peak in accuracy in

the images with 11-cface center frequency A one-wayanalysis of variance (ANOVA) showed a main effect foraudiovisual condition [F(6120) 1957 p 001]Tukeyrsquos HSD post hoc tests revealed that the full-facevideo condition had a significantly ( p 05) higher num-ber of key words identified than did all the conditions ex-cept the 11-cface band ( p 08) The latter conditionwas also found to be different from the unfiltered facewith additional statistical power2 Furthermore all of theaudiovisual conditions with the exception of two spatialfrequency filters (27 and 22 cface) showed significantlyhigher intelligibility than did the auditory-only condition( p 05)

When the performances in the filter conditions wereanalyzed separately a significant quadratic trend wasobserved ( p 01) The observed peak (11-cface cen-ter frequency) corresponds well with Naumlsaumlnenrsquos (1999)8- to 13-cface maximum sensitivity and Costen Parkerand Crawrsquos (1996) 8- to 16-cface critical spatial fre-quency range for static faces

These results show that visual enhancement of speechperception in noise occurs across a range of spatial fre-quencies All but two of the spatial frequency bands thatwe tested exceeded performance in the auditory-onlycondition On the other hand the full-face condition ex-ceeds the accuracy level of all of the band-pass conditionssuggesting that the linguistic information contained withinany single band is incomplete Either the summing ofsome of the filter bands or the broadening of the band-width may raise performance to the level of the full-facecondition The filtering in the present experiment wascarried out with a one-octave bandwidth3 Previous stud-

A B C

D E F

Figure 1 Images of the talker showing the different visual conditions used in Experiments 1 and 2(A) full video (B) 27-cface center frequency (C) 55-cface center frequency (D) 11-cface center fre-quency (E) 22-cface center frequency and (F) 441-cface center frequency

Figure 2 Percentage of key words correctly identified in Ex-periment 1 as a function of the spatial frequency bands (27- 55-11- 22- and 441-cface center frequencies) The unfiltered full-face and auditory-only (AO) conditions are also shown The errorbars show the standard errors of the means


ies have shown that optimal performance for static facerecognition can be found with a bandwidth close to twooctaves (Gold et al 1999 Naumlsaumlnen 1999) If audio-visual speech perception has the same optimal band-width its use might have reduced differences betweenthe full-face image and the band-pass conditions Wewill return to this issue in Experiment 3

One curious difference between Gold et alrsquos (1999)studies of static face identity and the present dynamicspeech perception experiment is that the one-octave stim-uli used here produced performance that approached thatof the unfiltered stimuli whereas Gold et alrsquos subjectsfound their one-octave stimuli impossible to discrimi-nate Although there are differences in task and stimulusdimensions (eg identity vs speech information) be-tween the two studies the discrepancy in performancefor the one-octave stimuli is more likely due to the pres-ence of motion in the present experiment Motion influ-ences object perception in a number of ways Motion canprovide additional cues to shape either through motion-defined form or simply by providing multiple viewpointsfor the face Independently the facial kinematics canspecify the spatiotemporal signature for an individualthat can be used to perceive identity or gender as well asbe perceived itself as a form of biological motion (egHill amp Johnston 2001 Stone 1998) The independenceof facial motion information from form cues has beendemonstrated with point-light audiovisual speech (Rosen-blum et al 1996) and preserved visual speech perceptionin a case of visual form agnosia (Munhall et al 2002)Knappmeyer Thornton and Buumllthoff (2003) on the otherhand have demonstrated that facial motion and facialform information are integrated during identity judg-ments The study of static face processing is thus ahighly impoverished condition removed from the naturalcontext of dynamic object perception

Despite this difference in the effectiveness of the one-octave bandwidth the spatial frequency effects for staticstudies of identity and emotion discrimination show a re-markable similarity to the present dynamic study of fa-cial behavior For both dynamic and static conditionsvery low frequencies seem less useful than high fre-quencies (see also Gold et al 1999) and peak perfor-mance is observed in the lowndashmid spatial frequencyrange This similarity of performance across differentstimuli and tasks suggests either that visual stimulusproperties of faces determine performance for all ofthese tasks in a bottom-up fashion or that the pattern ofperformance reflects a property of the visual system thatis implicated in all facial processing (cf Majaj et al2002)

EXPERIMENT 2

One approach to examining the relative contributionsof the visual system and the object properties to percep-tual performance is to vary the size of or the viewing dis-tance to the object In this experiment we contrastedspatial frequency defined with reference to the object

(rate of change of contrast per face) with spatial fre-quency defined in terms of retinal coordinates (rate ofchange of contrast per degree of visual angle) We did soby manipulating viewing distance to the display over a31 range Because image size on the retina is inverselyproportional to viewing distance retinal spatial frequencywill vary proportionally with distance to the display (seeTable 2) If the spatial frequency sensitivity in Experi-ment 1 is determined by retinal frequency this viewingdistance manipulation should shift the peak of the intel-ligibility function

For a range of different stimuli it has been reportedthat visual perception is scale invariant When viewedfrom different distances performance on the detectionof sine wave gratings (eg Banks Geisler amp Bennett1987) static face recognition (Naumlsaumlnen 1999) primingof face recognition (Brooks Rosielle amp Cooper 2002)letter identification (Parish amp Sperling 1991) and ob-ject recognition (Tjan Braje Legge amp Kersten 1995)is quite stable This remarkable scale stability for staticobject recognition is also found in dynamic audiovisualspeech perception Jordan and Sergeant (2000) manipu-lated distance from 1 to 30 m in an audiovisual task andfound that vision improved performance with congruentauditory speech at all distances In this experiment weexamined whether this scale invariance would extend toband-pass filtered images and thus whether subjectsconsistently use object-centered information to perceivespoken words

MethodSubjects Ninety individuals served as subjects All were native

speakers of English had normal or corrected-to-normal vision andreported no hearing problems and no history of speech or languagedifficulties

Stimuli Ninety CID sentences were used as stimuli In order totest five filter conditions at three viewing distances these sentenceswere divided into 15 groups of six sentences The number of keywords was approximately equal for each group

Equipment The subjects watched the displays on a 20-in videomonitor (Quasar QC-20H20R) The acoustic signals were ampli-fied and mixed (Mackie Micro Series 1202-VLZ mixer) and wereplayed through headphones (Sennheiser HD265) Because of theseparation required between the monitor and the subject the test-ing took place in a quiet laboratory room rather than in a soundbooth

Design Each subject participated in a single session consistingof 90 sentences presented in noise There were five band-pass fil-tered conditions and three viewing distances The 90 sentences


Center 114-cm 228-cm 342-cmFrequency Viewing Distance Viewing Distance Viewing Distance

(cface) (cyclesdeg (cyclesdeg (cyclesdegWidth visual angle) visual angle) visual angle)

27 032 065 09855 066 133 198

110 132 265 398220 265 531 796441 532 1064 1596


were divided into 15 groups of 6 sentences with an approximatelyequal number of key words Each group of 6 sentences was as-signed to one of the five band-pass filter conditions at a viewingdistance for a subject The subjects were tested at each viewing dis-tance separately Order of viewing distance was counterbalancedacross subjects Assignment of a group of sentences to a conditionwas counterbalanced across subjects so that every sentence oc-curred in each condition combination (band-pass filter viewingdistance order of viewing distance) The presentation of stimuliwas randomized across band-pass filter condition within subjectswithin a viewing distance

Procedure The subjects watched the monitor from three view-ing distances (114 228 and 342 cm) with their head in a headrestTable 2 shows the center frequencies in object and retinal coordi-nates In order to equate listening conditions the auditory stimuliwere presented through headphones The auditory signal-to-noiselevel was held constant for all the subjects Pilot testing was used tofind a level that produced auditory-only response accuracy below50 As in Experiment 1 the presentation of stimuli was subjectpaced The subjects verbally repeated as much of each sentence asthey could When the experimenter recorded the response the nexttrial was initiated

Results and DiscussionAs in Experiment 1 an inverted U-shaped function is

present in the data with a peak in intelligibility againbeing observed for the 11-cface filter condition In gen-eral the three viewing distances did not influence thepattern of results with all viewing distances showing thesimilar percentages of key words perceived for the dif-ferent spatial frequency bands (see Figure 3)

A 3 5 ANOVA was carried out to test the effect ofviewing distance (114 228 and 342 cm) and band-pass fil-ter (27- 54- 11- 22- and 441-cface center frequency)

As can be seen in Figure 3 no effect of distance was ob-served [F(2178) 066 p 5] however the band-passfilters significantly altered the intelligibility of the speech[F(4356) 727 p 001] A significant quadratic trendwas observed ( p 001) in these data A small filter distance interaction was present [F(12712) 201 p 05] The performance for the high-frequency filter (441cface) was significantly worse at 342 cm than at the av-erage of the other two distances

The present findings are consistent with studies of sta-tic faces that have indicated that object spatial frequencyrather than retinal spatial frequency best accounts for theperceptual results (faces Hayes Morrone amp Burr 1986and Naumlsaumlnen 1999 letters Parish amp Sperling 1991 cfMajaj et al 2002) This presumably accounts for theability of subjects to perceive audiovisual speech (Jordanamp Sergeant 2000) and letters (Legge Pelli Rubin ampSchleske 1985) over a large range of viewing distancesExtracting speech information by using a face-based co-ordinate system has ecological advantages since thespatial range of conversation varies considerably

As in Experiment 1 the intelligibility function wasnot completely symmetrical around the 11-cface bandFor two of the viewing distances (114 and 228 cm) thehigh-frequency band showed better performance thandid the corresponding low-frequency condition For thefurthest viewing distance high-frequency performancedropped off significantly This drop-off was presumablydue to optical attenuation of the high spatial frequencieswith increased distance (see Naumlsaumlnen 1999 for similarchanges in sensitivity to high-frequency bands with dis-tance for static facial images)

EXPERIMENT 3

The final experiment was motivated by two related ob-servations (1) In the first experiment the band-pass fil-tered stimuli did not produce performance equal to theunfiltered facial images and (2) the observed intelligi-bility function for the band-pass filtered images in Ex-periments 1 and 2 was not symmetrical around the peakat 11 cface One explanation for these observations isthat visual speech information is distributed across theindividual spatial frequency bands This could occur be-cause the optimal bandwidth for visual speech is actu-ally wider than the one-octave bands that we tested (cfGold et al 1999 Naumlsaumlnen 1999) or because nonredun-dant phonetic information is distributed across the spa-tial frequency spectrum In the latter case performancefor the higher frequency bands may have been better thanthat for the lower frequency bands because the filterbandwidth is proportional to frequency High frequencybands would contain more frequencies than would lowerbands and thus perhaps more information On the otherhand it may be that the high spatial frequency band cap-tures important phonetic information that is not reflectedin bands below its lower cutoff (29 cface) C Campbelland Massarorsquos (1997) and MacDonald et alrsquos (2000) data

Figure 3 Percentage of key words correctly identified in Ex-periment 2 as a function of the five spatial frequency bands (27-55- 11- 22- and 441-cface center frequencies) and the threeviewing distances (114 228 and 342 cm)


showed a monotonic increase in performance with eachincrease in level of quantization Although the slope ofthis function decreased markedly for higher levels ofquantization these results support the idea that the higherfrequency bands carry some unique information thataids visual speech

In the present experiment we addressed these issuesby testing the same recordings and task as those used inthe first two experiments with low-pass rather than band-pass spatial frequency filtered images By doing this wedirectly tested whether lower frequency information issufficient for visual speech processing (eg MacDonaldet al 2000) and indirectly tested whether the optimalbandwidth for visual speech is greater than the one-octavebandwidth used in the first two experiments Previouswork has supported the idea that subjects can reach max-imum performance with low-pass filtered stimuli how-ever it is unclear to what extent C Campbell and Mas-sarorsquos (1997) and MacDonald et alrsquos findings showinghigh-frequency contributions are due to the use of quanti-zation rather than other forms of image filtering Quan-tization introduces spurious high-frequency informationand the accuracy function shown in those articles mayhave been due in part to release from the effects of thishigh-frequency artifact with increasing quantizationrather than to the information below the cutoff frequencyThis interpretation is supported by Thomas and Jordanrsquos(2002) findings

MethodSubjects Two groups of 24 subjects (48 in total) served as sub-

jects All were native speakers of English had normal or corrected-to-normal vision and reported no hearing problems and no historyof speech or language difficulties Each of the two groups of sub-jects was tested at a different auditory signal-to-noise level

Stimuli Eighty CID sentences were used as stimuli The samerecordings as those used in Experiments 1 and 2 were processed ina different manner for this study The video was digitized as a se-quence of images converted to grayscale and low-pass filteredusing rotationally symmetric Gaussian filters (for details of the pro-cess used here see Jozan 2001) Six low-pass filtered sets of sen-tence stimuli were created (see Table 1) with cutoff frequenciesranging from 18 to 59 cface

The original audio track and a commercial multispeaker babbletrack (Auditec St Louis MO) were recorded to Betacam SP video-tape in synchrony with the original grayscale image sequence andwith each of the six low-pass image sets From the tape a CAVvideodisc was pressed for use in the perception experiments

Equipment The subjects watched the displays on a 20-in videomonitor (Sharp Aquos) The acoustic signals were amplified andmixed (Tucker-Davis System II) and were played through speakers(MG Electronics Cabaret) placed directly below the monitor Thetesting took place in a double-walled sound isolation booth (IACModel 1204)

Design Each subject participated in a single session consisting of80 sentences presented in noise These 80 stimuli were divided intogroups of 10 sentences with each group of 10 sentences assignedto one of the eight stimulus conditions for a subject (unfilteredvideo auditory only and the six low-pass conditions) Assignmentof groups of sentences to a condition was counterbalanced acrosssubjects The presentation of stimuli was randomized across con-ditions within subjects A second set of subjects carried out thesame experiment at a different signal-to-noise ratio

Procedure A small table and chair were positioned 171 cm fromthe monitor and the subjects viewed the stimuli with the head po-sition fixed using a chin- and headrest After the presentation of atrial the subjects verbally repeated as much of the sentence as theycould When the experimenter recorded the response the next trialwas initiated

Two different auditory signal-to-noise levels were chosen for thetwo groups of subjects For each group the signal-to-noise levelwas held constant for all the subjects As in the previous experi-ments pilot testing was used to find noise levels that produced twodifferent auditory-only response accuracies below 50

Results and DiscussionThe different signal-to-noise levels resulted in distinct

levels of performance The mean percentages of keywords correct averaged across conditions for the twogroups of subjects were 447 and 591 Howeverboth groups showed the same pattern across the differentaudiovisual conditions Figure 4 shows mean percent-ages of key words correct for the eight audiovisual con-ditions averaged across the two signal-to-noise levels Ascan be seen all but the lowest spatial frequency filter con-dition showed visual enhancement over the auditory-onlycondition Significantly the performance in the low-passconditions asymptotes with the condition composed ofimages filtered with a 73-cface cutoff frequency Forthis condition and conditions with higher cutoffs perfor-mance equaled that found for the unfiltered images

A 2 8 ANOVA showed these patterns to be statisti-cally reliable Main effects were found for signal-to-noise level [F(146) 1125 p 01] and audiovisualcondition [F(7322) 357 p 01] with no signifi-cant interaction Tukeyrsquos HSD post hoc tests showed thatall the audiovisual conditions with the exception of thelowest spatial frequency condition (18 cface) were re-liably better than the auditory-only condition ( p 05)and that there was no significant difference between theunfiltered facial stimuli and the stimuli with cutoffs of73 15 29 and 59 cface ( p 1)

Figure 4 Percentage of key words correctly identified in Ex-periment 3 as a function of the low-pass filter cutoffs (18 37 7315 29 and 59 cface) The unfiltered full-face and auditory-only(AO) conditions are also shown The error bars show the stan-dard errors of the means


These data help to clarify the interpretation of the firsttwo experiments The asymptotic performance at a rela-tively low spatial frequency suggests that the high spatialfrequency speech information either was redundant withthat contained in other bands or at the very least is notuseful in this context Lower spatial frequency informationis sufficient to equate performance with the unfiltered con-dition This raises the strong possibility that a broaderbandwidth (eg Gold et al 1999) would have raised theaccuracy level in Experiments 1 and 2 for the optimalband to equal performance in the unfiltered images

GENERAL DISCUSSION

In all the experiments the intelligibility of speech innoise varied as a function of the spatial frequency con-tent of the accompanying video images Experiment 1showed that all but the lowest spatial frequency band thatwe tested enhanced auditory speech perception howevernone of the individual spatial frequency bands reachedthe accuracy level of the unfiltered images The band-pass conditions showed a quadratic intelligibility pat-tern with peak intelligibility occurring in the mid-rangefilter band with a center frequency of 11 cface Experi-ment 2 showed that this pattern did not vary as a functionof viewing distance and thus that object-based spatialfrequency best characterized the data Experiment 3 in-dicated that a summation of the lower frequency bandsprovided all of the information required for audiovisualspeech

The visual enhancement of intelligibility even fromthe 55-cface center frequency band is consistent withstudies using quantized facial images for audiovisualspeech perception (C Campbell amp Massaro 1997 Mac-Donald et al 2000) Low-frequency images in thosestudies and in the present experiments influenced audi-tory perception This suggests that listeners do not needto foveate on the mouth to acquire visual phonetic infor-mation Studies of gaze during audiovisual speech haveindicated that listeners scan the eyes and mouth in astereotyped manner (Pareacute et al 2003 Vatikiotis-Batesonet al 1998) Our findings predict that irrespective ofthe location of gaze audiovisual speech perception willnot vary and indeed Pareacute et al have demonstrated thatthis is the case Whether subjects fixated on the eyes orthe mouth they showed the same level of McGurk effect

The band-pass stimuli show a maximum enhancementof intelligibility for the band with a center frequency of11 cface This corresponds with values found for staticface identification and facial expression discrimination(eg Naumlsaumlnen 1999) suggesting a commonality be-tween these disparate perceptual processes This con-trasts with previous research supporting a distinction be-tween the perceptual processes and neural architecturesresponsible for face identification and visual speech pro-cessing (eg Bruce amp Young 1986) The present find-ings are more consistent with data indicating that facialspeech processing and face identification are not inde-

pendent (Walker Bruce amp OrsquoMalley 1995) In face-to-face communication there may be advantages to usingthe same set of spatial frequencies for multiple face-related tasks It allows resources to be focused on a lim-ited scope of information and by rigidly restricting theprocessing bandwidth an efficient automated skill canevolve Pelli and colleagues have demonstrated suchldquorigidrdquo processing in the selective spatial frequency pro-cessing of letters (Majaj et al 2002 Solomon amp Pelli1994 cf Schyns amp Oliva 1999) and the use of letter ver-sus word processing of written text (Pelli Farell amp Moore2003)

Whether the spatial frequency patterns are the resultof the information distribution within the face itself (ob-ject statistics) or a property of the visual system cannotbe directly tested here4 One technique that has beenused for face (Gold et al 1999) and letter (Parish ampSperling 1991) perception is the calculation of the effi-ciency of the subjectsrsquo perceptual processing This in-volves comparing the performance of subjects with thatof an ideal observer (eg Geisler 1989) to see whetherthe human subjects are using all of the available infor-mation Given the dynamic nature of the speech stimuliand the open-set identification task potentially drawingon the full lexicon this is not a tractable approach foraudiovisual speech However ideal observer analysis forstatic face identification indicates that human observersdo not optimally use all of the information available inthose facial images and thus the peak in sensitivity as-sociated with a mid-range spatial frequency band is notreplicated in the ideal observerrsquos behavior (eg Goldet al 1999) Since the observed spatial frequency sensi-tivity curves appear to be similar between static facesand dynamic speech information we think it likely thatour results also do not reflect a data limitation Indeedstatistical analysis of the facial kinematics during speechindicates a remarkably rich information structure (Yehiaet al 1998) that is strongly correlated with the time-varying acoustic spectrum of speech

What do these findings suggest about the process ofaudiovisual speech perception In practical terms thedata indicate that viewing distance and gaze are not tightlyconstrained by the requirements of visual speech per-ception During natural conversations gaze has many so-cial and information-gathering functions Direction ofgaze signals turn taking and social attention Objects andgestures attract the gaze away from the face and a lateralreflective gaze accompanies some cognitive processingOur data and the recent findings of Pareacute et al (2003)suggest that visual speech processing can tolerate con-siderable deviations in fixations beyond the mouth re-gion and thus that speech visual processing might notbe significantly impaired by the parallel uses of gazeduring conversation

Lansing and McConkie (1999) have recently proposedthat subjects foveate the region of the face that carriesthe best information for a task This gaze direction as-sumption predicts that subjects will fixate on the mouth


for tasks requiring segmental perception and will fixateon other regions of the face during prosodic and emotionjudgments Although their data are consistent with thisassumption it is likely that their particular task silentspeechreading within a laboratory setting influencedtheir findings Different distributions of gaze have beenreported by Vatikiotis-Bateson et al (1998) and Pareacuteet al (2003) for different tasks The determinants ofthese patterns of gaze during speech perception are likelyto be a combination of oculomotor factors and cognitiveprocessing strategies However it is an entirely separateissue whether foveation on specific facial areas is nec-essary for audiovisual speech perception or even helpfulwhen it does occur (see the discussion above)

The conditions that promote integration of auditoryand visual information in speech are not completely un-derstood The integration of simpler nonverbal signalsappears to involve synchronous stimulation across themodalities (eg Calvert Hansen Iversen amp Brammer2001) but behavioral evidence suggests that this is not thecase for speech (eg Grant amp Greenberg 2001 MunhallGribble Sacco amp Ward 1996) One possible solutionthat has been proposed is that the dynamics of articula-tion provide a temporal signature in both visual and au-ditory speech that is important for cross-modal integra-tion For example slow changes in the acoustics thatreflect the syllabic alternation of opening and closing thevocal tract may directly correspond to the visual kine-matics (Greenberg amp Arai 2001 Munhall et al 1996Remez Fellowes Pisoni Goh amp Rubin 1998 Yehiaet al 1998) By this view the tracking of visual motionis the basis of audiovisual integration

Facial motion is a combination of rigid motion of thehead and nonrigid deformation of the face Whereas headmotion is strongly associated with prosody (NicholsonBaum Cuddy amp Munhall 2001 Yehia Kuratate ampVatikiotis-Bateson 2002) the soft tissue deformationsof the mouth and face provide segmental phonetic infor-mation Most research on motion perception has focusedon rigid motion (see Lu amp Sperling 2001) and work onbasic processes in the perception of shape deformation isonly beginning (eg Loffler amp Wilson 2001) Sincespatial-frequencyndashtuned channels within the visual systemplay an important role in motion perception determiningthe contribution of these two classes of motion to speechperception will require studies that systematically con-trol spatial and temporal frequency of the facial images

REFERENCES

Banks M Geisler W amp Bennett P (1987) The physical limits ofgrating visibility Vision Research 27 1915-1924

Bench J amp Bamford J M (Eds) (1979) Speech-hearing tests andthe spoken language of hearing-impaired children London AcademicPress

Benoicirct C Guiard-Marigny T Le Goff B amp Adjoudani A(1996) Which components of the face do humans and machines bestspeechread In D G Stork amp M E Hennecke (Eds) Speechreadingby humans and machines Models systems and applications (pp 315-328) Berlin Springer-Verlag

Brooke N M amp Templeton P D (1990) Visual speech intelligibilityof digitally processed facial images Proceedings of the Institute ofAcoustics 12 483-490

Brooks B E Rosielle L J amp Cooper E E (2002) The priming offace recognition after metric transformations Perception 31 297-313

Bruce V amp Young A (1986) Understanding face recognition BritishJournal of Psychology 77 305-327

Calvert G A amp Campbell R (2003) Reading speech from still andmoving faces The neural substrates of visible speech Journal ofCognitive Neuroscience 15 57-70

Calvert G A Hansen P C Iversen S D amp Brammer M J(2001) Detection of audio-visual integration sites in humans by ap-plication of electrophysiological criteria to the BOLD effect Neuro-Image 14 427-438

Campbell C amp Massaro D (1997) Perception of visible speech In-fluence of spatial quantization Perception 26 129-146

Campbell R Zihl J Massaro D Munhall K amp Cohen M M(1997) Speechreading in the akinetopsic patient LM Brain 1201793-1803

Costen N P Parker D M amp Craw I (1996) Effects of high-passand low-pass spatial filtering on face identification Perception ampPsychophysics 58 602-612

Davis H amp Silverman S R (Eds) (1970) Hearing and deafness(3rd ed) New York Holt Rinehart amp Winston

Geisler W S (1989) Sequential ideal-observer analysis of visual dis-criminations Psychological Review 96 267-314

Gold J Bennett P amp Sekuler A (1999) Identification of band-pass filtered letters and faces by human and ideal observers VisionResearch 39 3537-3560

Grant K amp Greenberg S (2001) Speech intelligibility derivedfrom asynchronous processing of auditory-visual information InProceedings of AVSP-01 (pp 132-137) Scheelsminde Denmark

Greenberg S amp Arai T (2001) The relation between speech intel-ligibility and the complex modulation spectrum In Proceedings ofthe 7th European Conference on Speech Communication and Tech-nology (Eurospeech-2001 pp 473-476) Aalborg Denmark

Hayes T Morrone M C amp Burr D C (1986) Recognition of pos-itive and negative bandpass-filtered images Perception 15 595-602

Hill H amp Johnston A (2001) Categorizing sex and identity fromthe biological motion of faces Current Biology 11 880-885

Jordan T amp Sergeant P (2000) Effects of distance on visual andaudiovisual speech recognition Language amp Speech 43 107-124

Jozan G (2001) Analysis of talking faces Tools for filtering and fea-ture tracking (ATR Tech Rep No TR-HIS-0004 pp 1-18) KyotoATR Laboratories

Knappmeyer B Thornton I amp Buumllthoff H (2003) The use offacial motion and facial form during the processing of identity VisionResearch 43 1921-1936

Kroos C Kuratate T amp Vatikiotis-Bateson E (2002) Video-based face motion measurement Journal of Phonetics 30 569-590

Lander K Christie F amp Bruce V (1999) The role of movement inthe recognition of famous faces Memory amp Cognition 27 974-985

Lansing C R amp McConkie G W (1999) Attention to facial regionsin segmental and prosodic visual speech perception tasks Journal ofSpeech Language amp Hearing Research 42 526-538

Legge G E Pelli D G Rubin G S amp Schleske M M (1985)Psychophysics of reading I Normalization vision Vision Research25 239-252

Loffler G amp Wilson H R (2001) Detecting shape deformation ofmoving patterns Vision Research 41 991-1006

Lu Z-L amp Sperling G (2001) Three-systems theory of human vi-sual motion perception Review and update Journal of the OpticalSociety of America A 18 2331-2370

MacDonald J Andersen S amp Bachmann T (2000) Hearing byeye How much spatial degradation can be tolerated Perception 291155-1168

Majaj N J Pelli D G Kurshan P amp Palomares M (2002) Therole of spatial frequency channels in letter identification Vision Re-search 42 1165-1184

Massaro D W (1998) Perceiving talking faces From speech percep-tion to a behavioral principle Cambridge MA MIT Press


Munhall K G Gribble P Sacco L amp Ward M (1996) Temporalconstraints on the McGurk effect Perception amp Psychophysics 58351-362

Munhall K G Servos P Santi A amp Goodale M (2002) Dy-namic visual speech perception in a patient with visual form agnosiaNeuroReport 13 1793-1796

Munhall K G amp Vatikiotis-Bateson E (1998) The moving faceduring speech communication In R Campbell B Dodd amp D Burn-ham (Eds) Hearing by eye Pt 2 The psychology of speechreadingand audiovisual speech (pp 123-139) London Taylor amp FrancisPsychology Press

Naumlsaumlnen R (1999) Spatial frequency bandwidth used in the recogni-tion of facial images Vision Research 39 3824-3833

Nicholson K G Baum S Cuddy L L amp Munhall K G (2001)A case of multimodal aprosodia Impaired auditory and visual speechprosody perception in a patient with right hemisphere damage InProceedings of AVSP-01 (pp 62-65) Scheelsminde Denmark

Ohala J J (1975) The temporal regulation of speech In G Fant ampM A A Tatham (Eds) Auditory analysis and perception of speech(pp 431-453) London Academic Press

Pareacute M Richler R C ten Hove M amp Munhall K G (2003)Gaze behavior in audiovisual speech perception The influence of oc-ular fixations on the McGurk effect Perception amp Psychophysics 65553-567

Parish D H amp Sperling G (1991) Object spatial frequencies reti-nal spatial frequencies noise and the efficiency of letter discrimina-tion Vision Research 31 1399-1415

Pelli D G Farell B amp Moore D C (2003) The remarkable in-efficiency of word recognition Nature 423 752-756

Prasad L amp Iyengar S S (1997) Wavelet analysis with applica-tions to image processing Boca Raton FL CRC Press

Remez R E Fellowes J M Pisoni D B Goh W D amp RubinP E (1998) Multimodal perceptual organization of speech Evi-dence from tone analogs of spoken utterances Speech Communica-tion 26 65-73

Rosenblum L D Johnson J A amp Saldantildea H M (1996) Point-light facial displays enhance comprehension of speech in noise Jour-nal of Speech amp Hearing Research 39 1159-1170

Rosenblum L D amp Saldantildea H M (1996) An audiovisual test ofkinematic primitives for visual speech perception Journal of Experi-mental Psychology Human Perception amp Performance 22 318-331

Schyns P amp Oliva A (1999) Dr Angry and Mr Smile When cate-gorization flexibly modifies the perception of faces in rapid visualpresentations Cognition 69 243-265

Simoncelli E P amp Olshausen B A (2001) Natural image statis-tics and neural representation Annual Review of Neuroscience 241193-1216

Solomon J A amp Pelli D G (1994) The visual filter mediating let-ter identification Nature 369 395-397

Stone J V (1998) Object recognition using spatiotemporal signaturesVision Research 38 957-951

Sumby W H amp Pollack I (1954) Visual contribution to speech in-telligibility in noise Journal of the Acoustical Society of America26 212-215

Thomas S M amp Jordan T R (2002) Determining the influence ofGaussian blurring on inversion effects with talking faces Perceptionamp Psychophysics 64 932-944

Tjan B S Braje W L Legge G E amp Kersten D (1995) Humanefficiency for recognizing 3-D objects in luminance noise Vision Re-search 35 3053-3069

Vatikiotis-Bateson E Eigsti I-M Yano S amp Munhall K G(1998) Eye movement of perceivers during audiovisual speech per-ception Perception amp Psychophysics 60 926-940

Vatikiotis-Bateson E Munhall K G Hirayama M Kasahara Yamp Yehia H (1996) Physiology-based synthesis of audiovisual speechIn Proceedings of the 4th Speech Production Seminar Models andData (pp 241-244) Autrans France

Vitkovich M amp Barber P (1996) Visible speech as a function ofimage quality Effects of display parameters on lipreading ability Ap-plied Cognitive Psychology 10 121-140

Vuilleumier P Armony J L Driver J amp Dolan R J (2003)Distinct spatial frequency sensitivities for processing faces and emo-tional expressions Nature Neuroscience 6 624-631

Walker S Bruce V amp OrsquoMalley C (1995) Facial identity and fa-cial speech processing Familiar faces and voices in the McGurk ef-fect Perception amp Psychophysics 57 1124-1133

Wenger M J amp Townsend J T (2000) Spatial frequencies in short-term memory for faces A test of three frequency-dependent hy-potheses Memory amp Cognition 28 125-142

Yehia H C Kuratate T amp Vatikiotis-Bateson E (2002) Link-ing facial animation head motion and speech acoustics Journal ofPhonetics 30 555-568

Yehia H C Rubin P E amp Vatikiotis-Bateson E (1998) Quanti-tative association of vocal-tract and facial behavior Speech Commu-nication 26 23-44

NOTES

1 The spatial frequency values used throughout the article are basedon image width measures

2 The lack of statistical difference ( p 08) between the unfilteredface and the 11-cface band could be due to a lack of power or simplyto the fact that there is no real difference between these conditionsGiven the great intersubject variability found in speech-in-noise exper-iments we strongly suspected the former explanation We tested an ad-ditional group of 21 subjects and found exactly the same pattern of re-sults as that found in Experiment 1 The combined data for the twogroups of subjects showed that the 11-cface band was reliably lower inintelligibility than the unfiltered face by Tukeyrsquos HSD ( p 01) For theremainder of this article we will treat this unfiltered face as showingbetter performance than do all of the filtered conditions

3 The stimuli used in this experiment are a by-product of a video-based face-tracking project (Kroos et al 2002) The one-octave band-width was determined by the requirements of that work rather than beingthe optimal bandwidth for the perception of facial stimuli Since thetracking system becomes less accurate at the highest spatial frequen-cies an important validation of the adequacy of that system was to showthat linguistically relevant speech motion is recoverable at medium andlower frequencies

4 Of course the object statistics and properties of the visual systemmight be equivalent (see Simoncelli amp Olshausen 2001)

(Manuscript received October 21 2001revision accepted for publication September 25 2003)


People are adept at using this rich visual informationand when image quality is degraded visual speech hasbeen shown to be quite robust In one of the first studies ofthis kind Brooke and Templeton (1990) manipulated spa-tial resolution by varying the quantization level of a dis-play showing the mouth region articulating English vow-els Varying the quantization level between 8 8 and128 128 pixels Brooke and Templeton found that per-formance on a silent lipreading task decreased when thedisplay was reduced below 32 32 spatial resolutionmdashfor example 16 16 pixels Similar methods have beenused by C Campbell and Massaro (1997) and MacDonaldAndersen and Bachmann (2000) to study the McGurkeffect and by Vitkovich and Barber (1996) to studyspeechreading of digits Both Campbell and Massaroand MacDonald et al found that the McGurk effect per-sisted to some degree even at low spatial resolutionVitkovich and Barber found no effect of their pixel den-sity manipulation but did find changes in performancewhen the grayscale resolution (ie the number of graylevels) of the images was drastically reduced Most re-cently Thomas and Jordan (2002) used Gaussian blur-ring to study the importance of fine facial detail in visualspeech and face processing In a series of studies theyfound that visual speech perception was not impaireduntil the blurring cutoff was 8 cycles per face or less

All of these studies suggest that a visual contributionto speech perception does not require the extraction ofhigh spatial resolution information from the facial imageThe performance of subjects with point-light displays ofvisual speech and the influence of gaze during audio-visual speech are consistent with this interpretationWhen the facial surface kinematics are reduced to themotions of a collection of light points subjects still showperceptual benefit in an acoustically noisy environment(Rosenblum et al 1996) and can perceive the McGurkeffect (Rosenblum amp Saldantildea 1996) Recently PareacuteRichler ten Hove and Munhall (2003) examined the dis-tribution of gaze during the McGurk effect Their resultsshowed that the eyemouth regions dominated the gazepatterns (see also Lansing amp McConkie 1999 Vatikiotis-Bateson Eigsti Yano amp Munhall 1998) however thestrength of the McGurk effect was not influenced by wherein the face region the subjects fixated More importantPareacute et al demonstrated that the McGurk effect did notdisappear when subjects directed their gaze well beyondthe facial region Thus high-acuity foveal vision doesnot seem to be a requirement for audiovisual integration

Although these studies suggest that the visual informa-tion used for speech perception can be relatively crudethere have been no direct tests of the range of spatial fre-quencies that are critical for audiovisual speech perceptionIt is widely accepted that the perception of visual imagesis carried out in the nervous system by a set of spatial-frequencyndashtuned channels For the purpose of studyingthis processing images can be decomposed into sets ofnonoverlapping spatial frequency bands and studies ofthe recognition of static objects in which such stimuli

have been used indicated that these different spatial fre-quency bands do not contribute equally to the objectidentification process (eg Parish amp Sperling 1991)Perception of such objects as letters and faces appears toinvolve distinct processing of portions of the spatial fre-quency spectrum This may be due to task- or domain-specific processing (Wenger amp Townsend 2000) For ex-ample Vuilleumier Armony Driver and Dolan (2003)reported that the amygdala carries out selective processingof the low spatial frequency portions of faces showingemotional expression whereas the fusiform cortex whichis thought to be more involved in identification is influ-enced more by the high-frequency portion of the facialimage Alternatively selective processing could be de-termined by the properties of a stimulus Solomon andPelli (1994) and Majaj Pelli Kurshan and Palomares(2002) have argued that the identification of text is me-diated by a small portion of broadband spectrummdasha sin-gle channel one to two octaves wide determined by in-formation in the fonts

In this research we manipulated the spatial resolutionof facial images in order to understand the separate con-tributions of different spatial frequency components tovisual speech processing Previous work on static faceprocessing (Gold Bennett amp Sekuler 1999 Naumlsaumlnen1999) has suggested that optimal recognition of faces in-volves a relatively narrow band of spatial frequenciesGold et al for example tested the discrimination of fa-cial identity across a range of bandwidths and spatial fre-quency ranges They observed peak efficiency of faceprocessing with a two-octave band-pass filtered stimuluswith a center frequency of 62 cyclesface (cface seeNaumlsaumlnen 1999 for similar results)

Our extension of this approach to the perception ofspoken language is motivated by two factors First thereis behavioral evidence that static and dynamic facial im-ages convey independent information (eg LanderChristie amp Bruce 1999) and neurological evidence thatdistinct neural substrates process static and dynamic vi-sual speech (Calvert amp R Campbell 2003 R CampbellZihl Massaro Munhall amp Cohen 1997 Munhall Ser-vos Santi amp Goodale 2002) Second the spatial fre-quency range that is important for a given stimulus typemay be tied to the particular task being carried out withthe stimuli (Schyns amp Oliva 1999 Wenger amp Townsend2000) In natural face-to-face conversations a wide rangeof visual tasks are carried out in parallel Listeners per-ceive information about the talkerrsquos identity emotionaland physical state focus of attention and degree of so-cial and conversational engagement as well as linguisticinformation about the utterance Each of these aspects ofcommunication may involve different stimulus proper-ties with information conveyed by different spatial fre-quency ranges (eg Vuilleumier et al 2003)

The experiments reported here extended the study ofthe role of visual stimulus properties to dynamic facialimages that conveyed linguistic information Our exper-imental task speech perception in noise was quite se-



GENERAL METHOD








EXPERIMENT 1

















A B C

D E F







EXPERIMENT 2












27 032 065 09855 066 133 198

110 132 265 398220 265 531 796441 532 1064 1596










EXPERIMENT 3




















GENERAL DISCUSSION












REFERENCES




























































NOTES








GENERAL METHOD








EXPERIMENT 1

















A B C

D E F







EXPERIMENT 2












27 032 065 09855 066 133 198

110 132 265 398220 265 531 796441 532 1064 1596










EXPERIMENT 3




















GENERAL DISCUSSION












REFERENCES




























































NOTES












A B C

D E F







EXPERIMENT 2












27 032 065 09855 066 133 198

110 132 265 398220 265 531 796441 532 1064 1596










EXPERIMENT 3




















GENERAL DISCUSSION












REFERENCES




























































NOTES










EXPERIMENT 2












27 032 065 09855 066 133 198

110 132 265 398220 265 531 796441 532 1064 1596










EXPERIMENT 3




















GENERAL DISCUSSION












REFERENCES




























































NOTES















EXPERIMENT 3




















GENERAL DISCUSSION












REFERENCES




























































NOTES























GENERAL DISCUSSION












REFERENCES




























































NOTES










REFERENCES




























































NOTES



































NOTES






spatial frequency requirements for audiovisual speech perception

Documents