Language Perception

Eva M. FernándezQueens College & Graduate Center

CUNY

ABRALIN

22-FEB-05

2

Language Is…

SIGNAL MEANING

grammar&

lexicon

PERCEPTION

PRODUCTION

logic knowledge about

the real world

3

Production

SIGNAL MEANING

LexicalRetrieval

StructuralAssignment

PhonologicalEncoding

• I’ll give you my undevoted attention!

• You’ll earn her eternal grapefruit.

• This restaurant hasn’t been awake very long.

• Put the oven on at a very low speed.

• We have a lot of churches in our minister.• They roasted a cook.• If you give the nipple an infant…• You ordered up ending.

• phonological fool• a glear plue sky• spattergrain

4

Perception

SIGNAL MEANING

StructureBuilding

LexicalAccess

PhonologicalDecoding

LexicalRetrieval

StructuralAssignment

PhonologicalEncoding

5

Not Present in (Speech) Signal:

phonemes

word boundaries

clause boundaries

location of empty categories

intended attachments for locally or globally ambiguous strings

hidden intents of the speaker!

SO HOW COME WE’RE SO GOOD AT DECODING?

Visual Illusions

when the experiences people report don’t correspond to physical properties of the stimulus

very cool… but also very informative about the way the

visual / perception system works

(which is: modularly)

7

The Hermann Grid Illusion

                                                 

           How many grey dots do you see at the “cross-roads”?

Source: http://dragon.uml.edu/psych/illusion.htmlA great page to visit for many more visual illusions.

8

A Face Can’t Be Hollow!

A face is always perceived as convex… not concave.http://www.kyb.tuebingen.mpg.de/de/bu/demo/Max Planck Institut für Biologische Kybernetik

Perceptual Illusions

also very cool… and also very informative about the way the

language perception system works

its modularity ensures its speed and accuracy,

which are both in turn compromised when the signal is

AMBIGUOUS

10

McGurk Effect

by Arnt Maasø, of the University of Oslo: http://www.media.uio.no/personer/arntm/McGurk_english.html

11

Perceptual Displacement andPhonemic Restoration

“The state governors met with their respective legislatures convening in the capital city.”

12

“legislatures”, with “cough!” (~ 145 msec) spliced in

13

“legislatures”, intact --- [s] ~ 145 msec

14

Bottom ~ Top study with Broca’s patients (Pollack & Picket, 1964)

The apple the boy is eating is red.

The girl the boy is chasing is tall.

bait, date, gate study (Garnes & Bond, 1976)

Here’s the fishing gear and the ___.

Check the time and the ___.

Paint the fence and the ___.

15

Structure Building: The Parser

its input is a string of lexical items

its job is to build syntactic structure

its output is sent to a mechanism that decodes meaning

it probably has limited access to information that’s not in the grammar or in the lexicon

it probably operates following a very small set of strategies, grounded on limitations imposed by working memory

RSVP Paradigm

Center-screen, word-by-word display

Timing: N ms per word (here: N = 500 ms)

Sentence-recall task

17

Thebeautifulblackcatchasedthecolorfulball.

The beautiful black cat chased the colorful ball.

18

Blackcolorfultheballchasedcatbeautifulthe.

Black colorful the ball chased cat beautiful the.

19

The Garden Path Sentence

The soldiers marched into the desert surprised the Persian forces.Since Joel always jogs a mile seems like a short distance to him.Carmela put the candy on the table in her mouth.Konstantin understood the problem had no solution.Everybody at the party knew Ann’s date was a total fool.

Local ambiguityDisambiguation downstream,which goes against parser’s preferencesReanalysis… or meltdown!

20

The Garden Path Theory

Lyn Frazier & Janet Fodor, late 1970s

Minimal Attachment: build the simplest tree

Late Closure: attach locally

Minimal Chain Principle / Active Filler Strategy: posit shortest possible chain / posit gaps for fillers ASAP

(the parser is lazy)

21

Minimal Attachment

The soldiers marched into the desert surprised the Persian forces.Since Joel always jogs a mile seems like a short distance to him.Carmela put the candy on the table in her mouth.Konstantin understood the problem had no solution.Everybody at the party knew Ann’s date was a total fool.

building complex structure = processing cost

22

Late Closure

John said Mary will arrive last night.

Physicists are thrilled to explain what they are doing to people.

Under the glistening tree there was a gift for a boy in a box.

Professor Humperdinck artfully avoided looking at the exams of the students that were sitting in his office ungraded.

Two sisters reunited after 18 years in check-out counter!

attaching non-locally = processing cost

23

Mary saw a gift for a boy…

NP

PP

P

for

NP

a boy

NP

a gift

Mary saw a gift for a boy in a box.

PP

in a box

attaching non-locally = processing cost

Late Closure

24

Late Closure

John said Mary will arrive last night.

Physicists are thrilled to explain what they are doing to people.

Under the glistening tree there was a gift for a boy in a box.

Professor Humperdinck artfully avoided looking at the exams of the students that were sitting in his office ungraded.

Two sisters reunited after 18 years in check-out counter!

attaching non-locally = processing cost

25

The RC Attachment Ambiguity

N1

The plot concerns the guardian of the princewho was exiled from the country for decades

RC

La trama es sobre el guardián del príncipeque fue exiliado del país por décadas

N2

26

Cross-Linguistic Differences

N1 attachment rates (%), in studies using questionnaire instruments where:

• RC was long• N1/N2 were equal in animacy• Complex NP was in canonical object position for the language

SP 63   67 62 63 55

EN 42 46 43(US)

40(UK)

47 48

Cuetos &

Mitchell, 1988

Bradley et al., 2003

Carreiras, 1992

Ehrlich et al., 1999

Fernández, 2000/2003

Hem

forth et al., submittd

27

Cross-Linguistic Differences

As in previous table, for languages other than English & Spanish,listed (for lack of a better strategy!) in alphabetical order:

SP-like 55   64 66 62

94!! (Fra)

60 (Can) 62 65 66   61 57   82  

EN-like   44         48(FS)

    32 42   28

 

AF

RIK

AA

NS

AR

AB

IC

BU

LGA

RIA

N

CR

OA

TIA

N

DU

TC

H

FR

EN

CH

GE

RM

AN

HE

BR

EW

ITA

LIAN

JAP

AN

ES

E

NO

RW

EG

IAN

PO

RT

. (BR

AZ

IL)

PO

RT

. (EU

RO

PE

)

RO

MA

NIA

N

RU

SS

IAN

SW

ED

ISH

28

Cross-Linguistic Differences

… could be driven by …

genetic relationship?

syntactic properties?

existence of unambiguous alternatives?

distribution of unambiguous strings in input?

prosody?

PARSERPROSODY

(phonology)PRAGMATICS

29

Pragmatics?

Grice’s Cooperative Principle

“Make your conversational contribution such as is required, at the stage at which it occurs, by the accepted purpose or direction of the talk exchange in which you are engaged”

The plot concerns

the guardian of the prince who was exiled

the prince’s guardian who was exiled

* the prince’s guardians who was exiled

30

Long RCs are Informationally Heavy

The plot concerns the guardian of the prince

… who was exiled.

… who was exiled from the country for decades.

Long RC has more lexical content, so it’s more informative.Does informativeness influence attachment?

RC length effect, confirmed in:English, Spanish; Arabic, Bulgarian, Croatian, French, German, Hebrew, Hindi, Portuguese…

N1 INTERP MORE LIKELY

31

Long RCs are Prosodically Heavy

The plot concerns the guardian of the prince who was exiled.

The novel’s plot concerns the guardian of the prince who was exiled.

The plot concerns the guardian of the prince who was exiled from the country for decades.

The novel’s plot concerns the guardian of the prince who was exiled from the country for decades.

32

Elicited Production

N = 8 native US English speakers — 5F, 3M

N = 6 4 sentences,RC Length Matrix-Subject Weight

RC = 1 versus 3 prosodic words…who was exiled ( from the country for decades )

MX = 1 versus 2 prosodic wordsThe ( unusual ) plot…

Fernández, Bradley & Taylor, in prep

33

The unusual plot concerns the guardian of the prince.The prince was exiled from the country for decades.

1

34

Acoustic Analysis: Regions

The( unusual )

plot concerns theguardian

of

theprince

who wasexiled

( from the countryfor decades )

Wt S V N1 N2 RC1 RC3

Duration: Uniform acoustic signature of phrasal break

Fernández, Bradley & Taylor, in prep

35

Acoustic Analysis: Regions

S ] [ V V ] [ N1 N1 ] [ N2 N2 ] [ RC

Fernández, Bradley & Taylor, in prep

36

0

200

400

600

800

1000

1200

Wt S V N1 N2 RC1 RC3

Mea

n D

urat

ion

(mse

c)MX1, RC1 MX1, RC3MX2, RC1 MX2, RC3

N2] [RC

Fernández, Bradley & Taylor, in prep

37

Region = N2

450 500 550 600 650 700 750

RC1

RC3

Mean Duration (msec)

MX2MX1

Fernández, Bradley & Taylor, in prep

38

Elicited Production: SummaryMX: F1(1,7) = 2.80, p=.138 RC: F1(1,7) = 11.46, p<.02

F2(1,5) = 2.07, p=.209 F2(1,5) = 9.96, p<.05

Interaction MX x RC: F1 < 1, F2 (1,5) = 1.62, p > .25

… N2 ] [ RC — and nowhere else

Likelihood of break grades with RC length and matrix weight, additively, i.e., with sentence length

Fernández, Bradley & Taylor, in prep

39

Questionnaire Procedure

“Reading comprehension test” 36 targets, 108 fillers (1:3 ratio) Comprehension question after each sentence

Example of target The plot concerns the guardian of the prince who was exiled from the country for decades.

Who was exiled? the guardian the prince

Example of filler The sneaky burglars took all the stereo equipment but overlookedthe computer system.

What was stolen? the stereo the computer

Fernández, Bradley & Taylor, in prep

40

Questionnaire ParticipantsN = 44, Queens College students

US English speakers

Language-history questionnaire, non-nativespeakers excluded/replaced

Rejected/replaced for errors > 15% in fillers

Fernández, Bradley & Taylor, in prep

41

Questionnaire Results

30

40

50

60

70

1 PWd 3 PWds

Relative Clause Length

% N

1 A

tta

chm

en

t

MX, 2 PWds

MX, 1 PWd

Relative Clause Length F1(1,40) = 24.95, p<.001 F2(1,32) = 30.12, p<.001

Matrix Subject Weight F1(1,40) = 5.51, p<.05 F2(1,32) = 9.43, p<.01

Interaction F1 < 1 F2 < 1

Fernández, Bradley & Taylor, in prep

42

The Implicit Prosody Hypothesis (IPH)

“In silent reading, a default prosodic contour is projected onto the stimulus, and it may influence syntactic ambiguity resolution” (Fodor 1998, 2002)

the brother of the bridegroom who snores

the brother of the bridegroom ][ who snores

43

Prosody and Syntax Align

the brother of the bridegroom ][ who often unknowingly snores

the brother of the bridegroom who snores

NP

N1 PP

NPP

RCN2

NP

N1 PP

NPP RC

N2

el hermano del novio ][ que a menudo inconscientemente roncaba

el hermano del novio ][ que roncaba

prosodic discontinuity

syntactic discontinuity

Selkirk, 1986

44

Empirical Support for the IPH

Behavioral evidence on how RCs are interpreted during silent reading

existing dataset: Hemforth et al. (submitted)

Evidence on how the N-of-N-RC construction is produced in discourse-neutral speech

elicited production experiment

Do the patterns in the two datasets match up?

45

Materials in English and Spanish:

with short and long RCs

N1-N2-RC placed post- and pre-verbally

Behavioral Evidence

The guest impressed X. X impressed the guest.

El invitado impresionó a X. X impresionó al invitado.

X = the brother of the bridegroomwho (often unknowingly) snores

el hermano del novioque (a menudo inconscientemente) roncaba

Hemforth et al. (submitted)

46

Behavioral Evidence

Post-VerbalObjects

Pre-VerbalSubjects

Who snores?The brother (N1)

20

30

40

50

60

Short RC Long RC Short RC Long RC

% H

igh

Att

ach

me

nt

English

Spanish

Post-Verbal Objects:

Cross-linguistic difference

RC length effect

Pre-Verbal Subjects:

RC length effect reduced

Cross-linguistic difference reduced

Hemforth et al. (submitted)

47

N2][RCN2][RC

RC.]RC.]

N2][RCN2][RC

RC][VRC][V

ENGLISH SPANISH

The guest impressed the brother of the bridegroom who often unknowingly snores.

El invitado impresionó al hermano del novioque a menudo inconscientemente roncaba.

The brother of the bridegroom who often unknowingly snores impressed the guest.

El hermano del novio que a menudo inconscientemente roncaba impresionó al invitado.

48

Experiment: Elicited ProductionParticipants, N = 8 per language

English New York

Spanish Madrid

Materials, N = 8 4 per language(selected from Hemforth et al.’s 32 4)

Post- and pre-verbal of identical length

RC’s right boundary with same lexical content, whether short or long

The guest impressed X. X impressed the guest.

X = the brother of the bridegroomwho (often unknowingly) snores

Fernández, Bradley, Igoa & Teira, 2003; Fernández & Bradley, 2004

49

Analyses: N2 & RC’s Verb

Duration: Presence of Boundary

Pitch movement: Type of Boundary

The guest impressed

the brother of the bridegroom ][ who … snores.]

N2][RC RC.]

The brother of the bridegroom ][ who … snores ][ impressed

N2][RC RC .][V the

guest.

Fernández, Bradley, Igoa & Teira, 2003; Fernández & Bradley, 2004

50

ENGLISH SPANISH

Mean duration (ms)

Pre-VerbalSubjects

Long RC

Short RC

100 msMonolinguals: N2 Durations

Placement × Length InteractionF1(1,14) = 5.77, p < .05, F2(1,14) = 12.37, p < .005

123 ms Post-Verbal

68 ms Pre-Verbal

Post-VerbalObjects

RC-Length =

Mean duration (ms)

51

Monolinguals: RC Vb DurationsPlacement × Length InteractionF1(1,14) = 6.38, p < .025; F2(1,14) = 5.90, p < .05

–10 ms Post-Verbal

35 ms Pre-Verbal

Long RC

Short RC

100 ms

Pre-VerbalSubjects

Post-VerbalObjects

RC-Length =

ENGLISH SPANISH

Mean duration (ms) Mean duration (ms)

52

Mea

n ris

e (H

z) p

er 2

00 m

s

Monolinguals: N2 PitchPlacement × Language InteractionF1(1,14) = 16.56, p < .002, F2(1,14) = 14.43, p < .002

0.4 Hz/200 ms English

23.6 Hz/200 ms SpanishPlacement =

ENGLISH SPANISH

Long RC

Short RC

Post Pre

53

Monolinguals: RCVb Pitch

Me

an

ris

e (

Hz)

pe

r 2

00

ms

ENGLISH SPANISH

Interaction: Placement × LanguageF1(1,14) = 6.05, < .05, F2(1,14) = 14.72, < .002

8.7 Hz/200 ms English

38.6 Hz/200 ms SpanishPlacement =

Long RC

Short RC

Post Pre

54

N2][RCN2][RC

RC.]RC.]

N2][RCN2][RC

RC][VRC][V

ENGLISH SPANISH

Pre-VerbalSubjects

Post-Verbal Objects

Duration & Pitch: Monolinguals

Post-Verbal, Short

Pre-Verbal, Short

Pre-Verbal, Long

Post-Verbal, Long

Post-Verbal, Short

Pre-Verbal, Short

Pre-Verbal, Long

Post-Verbal, Long

55

Summary of Data Outcomes

Pitch Movements: Type of Boundaryand Cross-Linguistic Differences

Spanish: N2 falls pre-verbally, rises post-verbally

English: N2 uniformly falls, pre- and post-verbally

Duration: Presence of Boundary and Cross-Linguistic Similarities

In both languages: Likelihood of breaks before RC is modulated by position

Fernández, Bradley, Igoa & Teira, 2003; Fernández & Bradley, 2004

56

Conclusions and Speculations

Behavioral similarities and differences are indexed in the prosodic patterns of Spanish and English

But what is the source for the contrasting sentence-medial tunes in Spanish?

Are such patterns projected entirely within the syntax-prosody interface?

Or are such patterns the result of an interplay of syntax, prosody, and information structure?