Doğu ErdenerDoğu Erdener

Auditory-visual speech perception in Auditory-visual speech perception in young children: young children: language and age-language and age-

specific factorsspecific factors

@@

6 December 2007

Development of Speech Perception Development of Speech Perception

First 6 months phonetically-based, language-general

Beyond 1st year phonologically-based, language-specific Consonants 7-11 months (Best, 1995) Vowels 4-6 months (Kuhl et al., 1992) Tones 6-9 months (Burnham & Mattock, 2006)

A working heuristic (Burnham et al., 2002) Acoustic / Phonetic Stage (0-6 months) Phonological Stage (6-12 months) Semantic Stage (12-24 months) Orthographic Stage (6-8 years)

Amodal (e.g.,Schwartz, et al., 1997) Phonetic (e.g.,Burnham & Dodd, 2004) Phonological / Postcategorical (e.g.,Massaro, 1998) none/some/all of the above?

Pre-categorical

Auditory-Visual Speech Perception (AVSP): Integration Auditory-Visual Speech Perception (AVSP): Integration

in early infancy (till ~6 months) is phonetically based beyond first year of life phonological

Recall that Recall that auditory-only auditory-only speech perception …. speech perception ….

AVSP integration research: AVSP integration research: We need differential and ontogentic data!We need differential and ontogentic data!

AVSP: Cross-Language (Differential) StudiesAVSP: Cross-Language (Differential) Studies

Cross-language phonemic differences e.g., Werker et al. (1992) Visual influence from /th/ in the McGurk

stimulus [A]-/ba/ + [V]-/tha/ :• anglophones > francophones• more experienced francophones > less experienced francophones

Amount of visual speech influence across langauges: American > Japanese (Sekiyama & Tohkura, 1993) Japanese > Mandarin (Sekiyama, 1997)

Sekiyama studies Possible reasons: Cross-cultural studies: Less eye/face contact in some cultures +

Japanese Ss tend to look at the eye area whereas American Ss on the mouth area (Yuki et al., 2007).

Linguistic 1: Tonal (e.g. Cantonese) and pitch-accent (e.g., Japanese) information are not visually salient.

Linguistic 2: Fewer visually distinct consonant clusters and vowels in Japanese than in English.

Development of Auditory-Visual Speech PerceptionDevelopment of Auditory-Visual Speech Perception

Infancy Infants match auditory and visual speech information

(Burnham & Dodd, 2004, Rosenblum et al., 1997)

Childhood McGurk & MacDonald (1976) 3 to 8 years + adults Massaro et al. (1986) 4-6 years + adults

Adulthood AV integration adults > children (Sekiyama &

Burnham, in press; Massaro et al.,1986) Late adulthood older Ss > younger Ss on vısual

speech information when auditory sensitivity is diminished (Cienkowski & Carney, 2002).

Focus: Developmental Data from Infants and ChildrenFocus: Developmental Data from Infants and Children

Why is there an age-related increase?

Infancy and childhood are marked with several novel language-relevant and language-specific challenges, such as phonetic phonological semantic vocabulary reading

Japanese and Australian English SpeakersThree child (6-, 8-, and 11-year-olds) groups

and a control group of adults.Method

McGurk stimuli DV: Visual Speech Index (VSI) Score based

on the proportion of auditory-based responses Identification Task: /ba/, /da/, or /ga/?

Sekiyama & Burnham (in press): Sekiyama & Burnham (in press): an ontogenetic + differential study an ontogenetic + differential study

AV+ minus A

Augmentation

A minus AV-

Interference

Visual Speech

Index (VSI)

[AV+] minus [AV-]

Japanese

0

0.25

0.5

0.75

1

-4 4 12 Clear

0

0.25

0.5

0.75

1

1 2 3 4

0

0.25

0.5

0.75

1

-4 4 12 Clear

0

0.25

0.5

0.75

1

-4 4 12 Clear

SN Ratio

Australian English

0

0.25

0.5

0.75

1

-4 4 12 Clear

Prop. Correct Aud Rs

0

0.25

0.5

0.75

1

-4 4 12 Clear

Prop. Correct Aud Rs

0

0.25

0.5

0.75

1

-4 4 12 Clear

Prop. Correct Aud Rs

0

0.25

0.5

0.75

1

-4 4 12 Clear

SN Ratio

Prop. Correct Aud Rs

6-yos

8-yos

11-yos

Adults

Sekiyama and Burnham (in press) found: Developmental increase in visual speech influence with age…

• In English speakers but• Not in Japanese speakers

So…..

What factors modulate the development of auditory-visual What factors modulate the development of auditory-visual speech perception speech perception (measured by VSI), (measured by VSI),

especially especially between 6 and 8between 6 and 8 years of age? years of age?

Experiment 1: Factors InvestigatedExperiment 1: Factors Investigated

Language Specific Speech Perception (LSSP) Effect of phonological experience in native language: Native

Speech Perception minus Non-native Speech Perception (N-NN) (Burnham, 2003).

Reading Reading ability is positively related to LSSP in children

(Burnham, 2003). Evidence for reading-lipreading link:

(de Gelder & Vroomen, 1998; Cavé et al., 2007)

Articulation ability is positively related to visual speech influence in children (Desjardins et al., 1997), and in adults with cerebral palsy (Siva et al., 1995)

Experiment 1Experiment 1

AVSPLanguage-specific Speech Perception

(LSSP) Reading

Articulation

Regression

Correlation

Experiment 1: MethodExperiment 1: Method

Language Specific Speech Perception (LSSP) Test: (Native minus Non-native speech perception N-NN)18 N [voiced vs. voiceless] + 18 NN [voiced vs. prevoiced]

Reading Test WRAT-3 reading subtest: 15 letters + 42 words = 57 items

Articulation Test Queensland Articulation Test (QAT) Naming task: 64 pictures Aus English consonants: initial, medial, & final positions

Auditory-Visual Speech Perception (AVSP) TestAuditory-Visual Speech Perception (AVSP) Test McGurk paradigm: A & V combinations of /ba/, /da/, and /ga/ 48 McGurk stimuli: (24 AV, 12 AO, and 12 VO),

3 consonants x 4 talkers (2 Eng & 2 Jap) x 2 congruence types x 2 background noise (Clear & noisy [+4 dB]) versions

Visual Speech Index (VSI) AV (+) minus AV (-)

Experiment 1: MethodExperiment 1: Method

Participantsn=96; Australian English 5-, 6-, 7-, and 8-year-oldsn=48; Australian English adults

N=144

Procedure Individual testing (~60 minutes) Task orders counterbalanced Clear & Noise (+4dB SN Ratio) Each stimulus presented twice

Exp. 1 Results: AVSP TestExp. 1 Results: AVSP Test

0

0.25

0.5

0.75

1

5 years 6 years 7 years 8 years Adults

Age

Visual Speech Index Score (/1)

Native Speaker Clear VSIForeign Speaker Clear VSINative Speaker Noise VSIForeign Speaker Noise VSI

VSI scores: Linear increase with age (p <.01) + Quadratic between 5 and 6 years (p <.01).

VO scores: Linear increase with age (p <.01).

AO scores: Linear increase with age (p <.01).

0

0.25

0.5

0.75

1

5 years 6 years 7 years 8 years Adults

Age

Correct VO Score (/1) Native VO Scores

Non-native VO Scores

0

0.25

0.5

0.75

1

5-year-olds 6-year-olds 7-year-olds 8-year-olds Adults

Age

Proportion of Correct AO Response (/1)

Native Speaker ClearForeign Speaker ClearNative Speaker NoiseForeign Speaker Noise

Exp. 1 Results: LSSP, reading and articulation Exp. 1 Results: LSSP, reading and articulation

LSSP scores: No age-based differences (p >.01) but native > non-native (p< .01)

Reading scores: Increase with age (p< .01)

Articulation scores: Increase with age (p <.01) + adults > children (p <.01).

0

0.25

0.5

0.75

1

5 years 6 years 7 years 8 years Adults Mean Discrimination Index Score (/1)

Native DI Non-Native DI

Age

0

0.25

0.5

0.75

1

5 years 6 years 7 years 8 years Adults

Age

Reading Test Scores (/1) Reading Test Score (WRAT-3)

0

0.25

0.5

0.75

1

5 years 6 years 7 years 8 years Adults

Age

Articulation Test Scores (/1) Articulation Test Score

Children

Age + AO + VO + LSSP + Articulation + Reading VSI

Adults

Age + AO + VO + LSSP +Articulation + Reading VSI

Exp. 1 Results: Regression Analyses Exp. 1 Results: Regression Analyses (DV = VSI Scores)(DV = VSI Scores)

Exp. 1 Results: Correlation Coefficients Exp. 1 Results: Correlation Coefficients

VSI Age AO VO LSSP Artic. Read.

VSI -

Age .14 -

AO .07 .43** -

VO .33** .32** .26** -

LSSP .21* -.15 -.01 .05 -

Articulation .07 .69** .30** .25** -.15 -

Reading .13 .86** .46** .35** -.06 .68** -

* Sig. at á=.05, **Sig. at á= .01

Experiment 1: DiscussionExperiment 1: Discussion

AVSP and lipreading (VO) increase with age This increase is sharp at around 6 years of age, around the

onset of reading instruction. No age-based LSSP differences

native and non-native difference is largest at 6 years Reading instruction a new challenge new sources of

speech information used? Regression Analyses

Children: lipreading + LSSP VSI Adults : AO VSI

Correlation Analyses Articulation is correlated with AO, lipreading (VO), and reading.

Exp. 1 Discussion Exp. 1 Discussion Experiment 2 Experiment 2

Experiment 1: LSSP & Lipreading AVSP

Schooling and/or new language-specific challenges seem to increase visual speech influence.

Towards Experiment 2 (3- & 4-year-olds): Status of LSSP AVSP link in pre-orthographic

stages?? Are AVSP and basic Cognitive abilities related? Are AVSP and Vocabulary Knowledge related?

Experiment 2: Method Experiment 2: Method

Participants: N=48; 3-year-olds & 4-year-olds AVSP – material, stimuli and procedure

AVSP: AX discrimination task Three conditions AV(36), AO(12), VO(12) AX-VSI (/1): based on different trials

LSSP – material, stimuli and procedure 18 Native [pa-pha] & 18 Nonnative [ba-pa] trials. AX-based category change paradigm.

Vocabulary Test – material, stimuli, and procedure PPVT (Dunn & Dunn,1997) age-based word sets of 12 items on average 5-6 sets.

Cognitive Flexibility – material, stimuli and procedure FIST (Jacques & Zelazo, 2001) 15 items (shapes, sizes & colours)

Exp. 2 Results: AVSP TestExp. 2 Results: AVSP Test

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

3-year-olds 4-year-olds

VSI-AX Score (/1)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

3-year-olds 4-year-olds

Correct Response Rate (/1)

Auditory-Only

Visual-Only

VSI-AX scores: 4-year-olds > 3-year-olds (p <.01)

VO scores: 4-year-olds ≈ 3-year-olds (p =.07)

AO scores: 4-year-olds > 3-year-olds (p <.05)

Exp. 2 Results: LSSP, vocabulary and cognitive flexibilityExp. 2 Results: LSSP, vocabulary and cognitive flexibility

LSSP scores:

4-year-olds > 3-year-olds (p <.01)

No stimulus language effect or contrast/age interaction (p >.05)

Vocabulary scores: 4-year-olds > 3-year-olds (p <.01)

FIST scores: 4-year-olds > 3-year-olds (p <.01)

0

0.05

0.1

0.15

0.2

0.25

3-year-olds 4-year-olds

Discrimination Index Score (/1)

Native DINon-Native DI

0

10

20

30

40

50

60

70

80

3-year-olds 4-year-olds

Receptive Vocabulary Score

0

2

4

6

8

10

12

14

16

3-year-olds 4-year-oldsMean Executive Function Test Score (/15)

Rule Abstraction

Cognitive Flexibility

Exp. 2 Results: Regression Analyses Exp. 2 Results: Regression Analyses

VSI scores as dependent variable

Age + AO + VO + LSSP + Vocabulary + FIST VSI-AX

Exp. 2 Results: Correlation Coefficients Exp. 2 Results: Correlation Coefficients

VSI-AX Age AO VO LSSP Vocab. FIST

VSI-AX -

Age .45** -

AO .54** .42** -

VO .21 .21 .37** -

LSSP .30* .10 .21 .10 -

Vocab. .45** .66** .56** .27* .41** -

FIST .51** .36** .35** .36** .42** .54** -

* Sig. at á=.05, **Sig. at á= .01

Experiment 2: DiscussionExperiment 2: Discussion

AVSP, LSSP, vocabulary and FIST improve with age AVSP improves between 3 and 4 years. Lipreading p=.07 – same age-based trend, but small age

difference? Regression Analyses

AO & FIST VSI-AX similar to adults in Exp. 1 (AO VSI)

Why does AO predict VSI-AX? AVSP product of AO speech perception Three and 4 years: object-word pairing is a challenge, but not as

much as reading – so no need for extra info. AVSP is determined by inherent abilities?

e.g., AO speech perception and cognitive function.

Regression Analyses: Regression Analyses: LSSP, LSSP, aage, language tests regressed onto AVSPge, language tests regressed onto AVSP

5,6,7, 8 years LSSP + VO + AO + Articulation + Reading AVSP

AVSP is related to LSSP and lipreading

AdultsLSSP + VO + AO +Articulation + Reading AVSP

AVSP is related only to auditory speech perception

3, 4 years

LSSP + VO + AO+ Cognitive + Vocab. AVSP

AVSP is related only to auditory speech perception

General DiscussionGeneral Discussion

Visual speech influence develops with age AV: e.g., Massaro et al. (1986) VO: e.g., Desjardins et al. (1997)

(though p=.07 in Exp.2)

AVSP follows a phonetic phonological trend AVSP is amodal and phonetic

(Burnham & Dodd, 2004; Rosenblum et al., 1997) AVSP is phonological (Massaro, 1998) But more importantly, AVSP seems to occur at all

levels of language processing.

AVSP- Articulation Link Vocabulary and articulation challenges in early childhood Tests of AVSP, LSSP, and articulation for children (12-24

months), with and without speech disorder (Dodd et al.,in press). AVSP-Reading Link

Burnham (2003): reading LSSP LSSP AVSP So, integrative ability phonemic/phonological processing This may be the case for English but not for other languages?

Around school age, language-relevant challenges emerge Reading phoneme-grapheme mapping LSSP intensifies (orthographic stage: Burnham et al., 2002) Children use all speech information available

• Hyperarticulated speech styles (Lees & Burnham, 2005).e.g., teacherese (Håkansson, 1987).

• Cognitive skills activated + maturation

General DiscussionGeneral Discussion

Past Past Current Current Future Future

AVSP and reading link (Cave et al., 2007; de Gelder & Vroomen,

1998)

Burnham (2003)

Articulation-lipreading link (Desjardins et al.,1997)

AVSP (integrative module)

Auditory-only speech perception

Lipreading

Speech Production(articulation)

Reading

Some questions awaiting answerSome questions awaiting answer

Speech Production-Perception Link: the perception-production link: a developmental

perspective; e.g., testing children with speech disorders (Dodd et al., in press).

Visual Speech and Reading Link:Research on the link between study strong correlations between visual aspects of speech and reading This study correlations between reading and AO,

lipreading and articulation. Cavé et al. (2007); de Gelder & Vroomen (1998). Cross-language reading x AVSP studies.

Acknowledging…….Acknowledging…….

Prof. Denis Burnham

Prof. Barbara Dodd, Asc.Prof. Chris Davis, Prof.Kaoru Sekiyama

MARCS Auditory Laboratories & MARCS Baby Lab.

College of Arts, University of Western Sydney

Australian Postgraduate Award

Aprica Foundation, Osaka, Japan

Participating schools, teachers, parents and kids in Sydney