new creating a new communication system : gesture has the upper … a new... · 2015. 11. 24. · :...

Powerpoint Templates

Creating a New Communication System

: Gesture has the Upper Hand

인지과학 협동과정

방효석

Table of Contents

Backgrounds

Overview of Present Research

Experiment 1

Experiment 2

Discussion

Backgrounds

Previous Research

: Creating a communication system from scratch: gesture beats vocalization

hands down (Fay et al., 2014)

Does the modality of communication drives the creation of shared

inventory of sign-meaning mapping?

Backgrounds

Previous Research

: Creating a communication system from scratch: gesture beats vocalization

hands down (Fay et al., 2014)

1. Communication success will be higher for gesture than for non-

linguistic vocalization.

2. Communication success will be higher in the combined modality

compared to gesture-alone.

3. There will be greater alignment in the gestural modality than in

the vocalization modality.

1. Language Origin

• Proto-speech account

: Rudimentary vocalization came to have communicative meaning

• Proto-sign account

: Language evolved first from manual gestures, before shifting to the

vocal modality

• Multimodal account

: The earliest forms of language were not restricted to a single modality

Basic Concepts

Backgrounds

Theories of Origin of

Languages

1. Proto-sign account

2. Proto-speech account

3. Multimodal account

Experimental Design

1. Gesture-only condition

2. Non-linguistic vocalization

condition

3. Multimodal condition

2. Types of Signs(Pierece’s triadic model of sign)

1) Iconic: Signs where the signifier resembles the signified(e.g., a

portrait, a cartoon, sound effects, or a statue)

2) Indexical: Signs where the signifier is caused by the signified(e.g.,

smoke signifies fire)

3) Symbolic: The signifier is assigned arbitrarily or is accepted as

societal convention. The relationship must be learned (i.e. alphabet,

mathemetical signs, computer codes)

Basic Concepts

Backgrounds

3. Referential Communication Task

Referential communication?

: Communication occured by making reference

Ex) You can refer one of the cars on the right,

by using

• names (the Corvette)

• noun modifiers, like adjectives

(the light green car)

• prepositional phrases

(the car in the upper right-hand corner)

• relative clauses

(the car that you just put on the shelf)

Basic Concepts

Backgrounds

4. Measures

1) Effectiveness: measures how successful the signs were at identifying

their referent (communication success)

2) Alignment: : measures the similarity between the sign a participant

produced, and the sign their partner produced on the previous game

when communicating the same concept (basis of shared inventory)

Basic Concepts

Backgrounds

5. Motivated signs & Arbitrary signs

1) Motivated signs: Signs that are linked to meaning by structural

resemblance or by natural association (i.e. Iconic/indexical signs)

2) Arbitrary signs: Signs where the relation between signifier and signified

is purely conventional and culturally specific (i.e. Symbolic signs; name,

language...)

Basic Concepts

Backgrounds

Research Overview

Purpose of the present research

1. Limitation of previous referential task studies

: lack of the number of concepts used

2. Provide the comparison between the vocal and gestural

modalities to figure out the differences of affordances of sign

motivations of gesture and non-linguistic vocalization

Experiment 1

Hypothesis

1. Sign motivation would be higher for gestured signs than for

signs produced using non-linguistic vocalization

2. Communication success would be higher for gesture than for

non-linguistic vocalization

3. Alignment would be higher in the gesture-only condition than

in the vocal-only condition

Experiment 1

Method

1)Participants:

106 undergraduate students(63 females)

Placed into unacquainted pairs

All were free from auditory, visual, speech and motor impairment

2)Materials:

18 target concepts were sampled without replacement

(Adjective, Noun, Verb)

Presented with six distractor concepts

A different set of concepts were used by each pair

Experiment 1

Method

3) Task & Procedure:

• 2 task conditions (gesture/vocal) with 6 games in each condition

• Pairs 2 participants (matcher/director)

matcher: Select the same concept more than once within the

same game

director: Produce as many gestures and vocalizations as they

wished for each concepts

• Counterbalancing of modality across participants

gesture-only: face one another

vocal-only: face away from each other(eliminate gesture effects)

Experiment 1

Result & Discussion

Three-way ANOVA (modality, game, concept)

1) Motivation

• 7-point likert scale: 0 (entirely symbolic) to 6 (highly motivated, either

iconic or indexical)

• Gesture outperformed vocalization [F(1, 52) = 700.33, p < .001]

• Increase in sign motivation across games 1-6 in both conditions [F(5,

260) = 39.89, p < .001]

Experiment 1

Result & Discussion

2) Communication Success

Experiment 1

Result & Discussion

3) Alignment

• 7-point likert scale, 0 (did not copy partner’s sign) to 6 (nearly identical

copy of partner’s sign)

• Gesture showed higher alignment than vocalization(p<.001)

• Alignment increased in both modalities across games(p<.001)

Experiment 1

Result & Discussion

4) Limitation

• Concurrent feedback effect in gesture-only condition

: Participants in gesture-only condition may have and advantage

seeing their partner’s facial expressions of confusion or

comprehension

→ Experiment 2 addresses this issue

Experiment 2

Method

1)Participants:

60 undergraduate students(42 females)

2)Materials:

Same corpus of concepts used in experiment 1 was used

Fewer concepts were sampled(540)

3) Task & Procedure:

Participants communicated signs to a video camera

Only 2 games were played

Experiment 2

Result & Discussion

1) Motivation

Experiment 2

Result & Discussion

2) Communication Success

• Gestured signs were communicated more successfully than vocal

signs(p<.001)

3) Alignment

• Alignment was higher in gesture-only condition[F(1, 29)=283.41,

p<.001]

Experiment 2

Result & Discussion

• Experiment 2 replicates the pattern of results observed in experiment 1

: Indicates that the benefit of gesture over non-linguistic vocalization is

due to the modality itself, rather than concurrent feedback

Experiment 2

Result & Discussion

Based on interaction effect of 3 variables:

Discussion

If gesture is better than non-linguisitc vocalization in terms of sign

motivation, why do we use conventional spoken language instead of

conventional sign language?

There are several studies showing evidence that people also

spontaneously produce motivated vocal language in their speech. If

both gestural and vocal language have same effect of motivation,

which of these do you think the origin of conventionalized language?

new creating a new communication system : gesture has the upper … a new... · 2015. 11. 24. · :...

Documents

creating a communication system from scratch: gesture...

a new approach for dynamic gesture recognition using

creating new possibilities

welcome to gesture - mat uc santa barbara · welcome to...

creating new brands

evolving lifestyles, creating new health...

creating new cultures pdf

creating web parts new

creating newsletters new

gesture final report new

american new urbanism as a gesture spread out through...

creating new norms

multi-touch: a new tactile 2-d gesture interface for human

gesture input and gesture recognition algorithms

creating new tools

creating new growth platform

creating new horizons

towards natural gesture synthesis: evaluating gesture...

creating new knowledge

request #: hutrr93 title: gesture sensors spec release: 1...