‘Joining in’spontaneous conversation andimprovisational music-making

Sarah Hawkins1, Richard Ogden2, Ian Cross1

{sh110, ic108}@cam.ac.uk rao1@york.ac.uk

Language, Music and Interaction, Philological Society,QMUL, Nov. 2012

1 Centre for Music & ScienceUniversity of Cambridge

2 Dept. of Language and Linguistic Science,University of York

What processes underpin interaction in music-making and speech?

Spontaneous interaction:conversation and musical improvisation

• A controlled environment in which to elicit (relatively!) natural joint (inter)action in unrehearsed• talking• music-making• non-musical play

• Initial observations and hypotheses

2

with unfamiliar instruments / objects

Speaker–listener neural coupling underlies successful communication

Stephens, Silbert, and Hasson (2010)

fMRI as speaker tells a story; another P listens.• a speaker’s brain activity is spatially and temporally

coupled with the listener’s activity• but only when the listener understands the speaker• most correlated patterns in listener are delayed relative

to speaker’s; but some are anticipatory• greater anticipatory speaker–listener coupling

→ greater understanding (independent comp. measure)

3

Stephens, Silbert, and Hasson (2010) PNAS 107(32) 14425-14430Neural: Charles Schroeder group; Edward Large group e.g.

Schroeder et al. (2008) TICS 12(3), 106-113 Fujioka, Trainor, Large & Ross (2012) J. Neurosci. 32(5), 1791-1802

Sociophonetics: Garrod & Pickering (2004) TICS 8(1), 8-11

Hypothesis: if coupling of neural oscillations underpins successful

communication, then we should find:

similar processes in music and speech differing only by• demands of the medium• function of the particular interaction

our aim: find a set of comparable tasks & measuresin music-making and conversing

CA framework: alignment and disalignment

4

Alignment and disalignment in talk

Superordinate, multidimensional terms

We don’t expect binary classification every time

5

hasn’t he got nice eyes?

Alignment DisalignmentTiming YES! ( 200 ms ) YES!

Phonetic form YES! Yes

Syntactic form Yes, he has Yes

Word choiceYes, they’re really gorgeous!

They’re ok

In music, we can expect similar patterns (Turino)

• How did you get here?• What do you think of the room?

6

demographic questions; initial consent

10 non-musical play

10 musical play

10 conversation

• card houses• tallest tower: blocks• market stall: playdough

• xylophone, kalimba• drums, claves

detailed musical questions; final consent; £8

• 9/11; Princess Diana’s death…• important event you shared

≥ 5 conversationMinutes

no shakers!

at least 2

Who?

“Pilot”:5 dyads,• 3 musician pairs• 2 non-musician• various tasks

“Experiment 1”6 dyads,• 3 musician pairs• 3 non-musician• 2-3 prescribed

tasks

7

Dyads:• friends• same-sex• native

speakers of English

• 18-30• university

educated• both musicians,

or bothnon-musicians

Starting set-up: familiarisation

• card house

• tallest tower of blocks

• playdough market stall

8

• Sri Lankan drums (2 types, one not shown)

• claves• circular

xylophone• kalimba

(mbira)

• ≥ 10 years’ formal training

• currently actively engaged in music at least once a month

• ≤ 7 years’ formal training

• no active music-making in past 4 years

9

non-musiciansmusicians

Recording• 4 video cameras• overhead omni mike• stereo pair (music)• 2 close-talking mikes

10

Recording• 4 video cameras• overhead omni mike• stereo pair (music)• 2 close-talking mikes

11

Looking for co-ocurrences

• Focus: alignment and disalignment• rhythmically: entrainment and failure to entrain

• Body movement is well established as marking important events (beats) in both speech and music

• What happens when such beats carry across the two modalities?

• A single framework for labelling events• tracking beats in speech and music: currently, Cummins

12

Example(s) of what we found placeholder

• Alignment “magic”: E1_MF1 29:43-29:51• beat continuation across modalities, and between

participants• perfect coordination• unscripted (music not ‘counted in’)

• Disalignment: E1_MF1 26:31-26:47

13

Alignment: beat continuation across modalities, and between participants

• The criterion is (for alignment): there is speech before or after where

• the music changes, and the music 'works'.

• The questions are: to what extent is the speech and music beat coordinated?

• and how does this compare when there is breakdown or less successful interaction?

• what happens with body movements and eye gaze?

14

Tentative Hypotheses (to be completed)

15

• look at the effector:• hands when playing• faces when speaking• presumably faces when singing together

• look at times of uncertainty….

Next steps

• Quantify: proportion of positive instances of categories• Theory: which? is there only one?

• Why? (Causes) do people entrain willy nilly or element of prediction from one or other

• if we can’t tell bottom up from top down, and there’s not a clear listener vs clear talker, what are we dealing with – the holy spirit?

• Theory: top down and bottom expectancies mesh: me, Narmour, Pearce/Wiggins….we need to work actively to get this working for a general theory – and using sp and music as our test bed seems an exciting way forward. Form a working group???

16

17

Daniel Halford

Satinder Gill

Rein Ove Sikveland

David Greatrex

Thanks to!

Hannah LeachNewton Trust, U. Cambridge; BA/Leverhulme Foundation Small Grant

Some random slides from past presentations that may help orient you re

our way of thinking

18

So why do music and speech seem fairly different, but we feel they are the same?

1. music and speech typically differ in the relative balance accorded to conveying phatic vs referential meaning—but this is a very loose difference

• functions of both modalities dictate what is important, and where we should look for guiding principles

• languages will differ – as will musics (structure)

2. these different balances in large part dictate the greater predictability of rhythm in music than in speech

19

some things that music and speech share

• wide range of rates, affected by many factors• phrase-final lengthening• predictable tonal endings: cadence, nuclear tone• internal/local rate change: rubato, asides, emotion,

floor holding• deviation from rhythmicity indicates

• emotion • phrasing• in speech, the demands of the actual words used

• regular rhythms are constructed, in both speech and music• consistent with most other perceptual approaches

20

Summary of a video clip (not included)• the background speaker, L, maintains a beat of about 460 ms in

speaking; and appears to lead the transition into music:• though R talks more, and is talking very casually,

she seems to entrain to L’s speaking beat

• they start playing about on ‘the current beat’: c. 800-900 ms

• gradually increase tempo to c. 700 ms

• look at the effector:• hands when playing• faces when speaking• presumably faces when singing together

21