1 visually-guided motor actions in humans ken nakayama and joo-hyun song harvard university vision...

1

Visually-guided Motor Actions in Humans

Ken Nakayama and Joo-Hyun SongHarvard University

VISION SCIENCESLABORATORY

2

Actions: Interaction with The External World

3

.

• Discrete responses • Continuous interaction

Actions: Read Out Cognitive Processing

• Visually-guided actions: Instantaneously read out internal processes

Yet, the importance of motor actions is overlooked

4

Cognitive Theories in Human Behaviors

• Perception Cognitive decision Action

• Issue of motor control

- Limited to post-cognitive decision-making - Mere reflection of completed cognitive decisions

5

Target without distractors Target among distractors

Visually-Guided Manual-Pointing Task

+

Single targetEasy Task

+

Odd-color target Difficult Task

6

.

Hand movement recording: 120Hz with Polhemus Fastrak

Visually-Guided Manual-Pointing Task

QuickTime™ and aSorenson Video 3 decompressorare needed to see this picture.

7

Outline

• Part 1: Role of focal attention on latencies and trajectories of manual pointing

• Part 2: Concurrent processing of manual pointing to competing stimuli

• Part 3: Automatic adjustment of visuo-motor readiness

8

Outline




9

Visual Attention: Selection for Perception & Action

10

• Broadly distributed attention for entire displays is sufficient to detect and localize the target

Distributed vs. Focused Attention

• Focused attention needs to be allocated to the target for feature discrimination

( Atkinson & Braddick, 1989; Folk & Egeth, 1989; Johnson & Pashler, 1990; Sagi & Julesz, 1985 a, b; Green, 1992)

11

Diagnostic Visual Search Paradigm

Mixed Condition

Target: Red Green Red

+ + +

Blocked Condition

Red Red RedTarget:

+ + +

(Bravo & Nakayama, 1992) Number of distractors

React

ion

Tim

es

Detection(distributed)

MixedBlocked

Discrimination (focused)

MixedBlocked

Saccadic tasks: No subtle discrimination required

(McPeek et al., 1999)

12

Role of Focused Attention on Latencies & Trajectories of Manual Pointing

Number of distractorsR

eact

ion

Tim

es

Detection (distributed)

MixedBlocked

Discrimination (focused)

MixedBlocked

+ + +

B. Blocked Condition

+ + +

Green GreenGreen

A. Mixed Condition

Green GreenRed

• Full Movement trajectories over the time course: Interaction between cognitive-decision making for the correct target & motor control

13

Reaction Time Indexes

(Initiation)Latency

MovementDuration

Total Time

+

Stimulus onset

Finger lift-off

Target touch

Time

14

Total Time (ms)

7 Subjects

Exp 1: Effect of Perceptual Grouping of Distractors

680

700

720

740

760

2 5 8 11

Number of distractors

Mixed

Blocked

280

300

320

340

2 5 8 11Number of distractors

MixedBlocked

Initial latency (ms)

380

400

420

440

460

2 5 8 11Number of distractors

MixedBlocked

Movement duration (ms)

15


-5 0 50

2

4

6

8

10

12

-5 0 50

2

4

6

8

10

12

-5 0 50

2

4

6

8

10

12

-5 0 50

2

4

6

8

10

12 # Distractor=0

Vert

ical M

ovem

en

t

Horizontal Movement (Inches)

Mixed condition

Baseline condition

-5 0 50

2

4

6

8

10

12# Distractors=5

0

2

4

6

8

10

12

-5 0 5-5 0 50

2

4

6

8

10

12# Distractors=2 # Distractors=11

C.

Blocked condition

16


-5 0 50

6

12

Horizontal movement (inches)

Ver

tical

mov

emen

t (in

ches

)

• Initially toward distractors & corrected to the target

• Size & frequency of curved trajectories are correlated with the strength of target-distractor competition (McPeek, Han & Keller, 2003).

17

Max Curvature=Max(D)/L

-50

5

05

1002468

1012

Forward

Left Right

D

L

.

0.07

0.08

0.09

0.1

0.11

0.12

0 2 4 6 8 10 12


Baseline

Mixed

Blocked


18

680

700

720

740

760

2 5 8 11


Mixed

Blocked

Position within Same Color Sequence

. . .1 8. . .

RT

5

Target:

(Maljkovic & Nakayama, 1994; Bichot & Schall, 1999; McPeek et al., 1999)

Exp 2: Effect of Perceptual Priming

Total Time Attended target feature repetition

19


650

675

700

725

750

775

0 1 2 3 4 5 6 7

Order in sequence

Blocked Baseline

7 Subjects

Total time (ms)

Order in sequence400

425

450

475

500

0 1 2 3 4 5 6 7

Blocked Baseline

Movement duration

Order in sequence

240

245

250

255

260

265

270

0 1 2 3 4 5 6 7

Blocked Baseline

Initial latency

20

Maximum curvature


0.05

0.075

0.1

0.125

0.15

0.175

0 1 2 3 4 5 6 7

Order in sequence

Max

imum

cur

vatu

re

Blocked Baseline

21

Discussion

• Focused attention is closely associated with manual-pointing process though subtle discrimination is not required. • Trajectory curvature is reduced as decreasing competition between target and distractors by perceptual grouping and cumulative priming

• Interactive processes between target selection and motor control may challenge assumption of serial processing in traditional cognitive theories

22

Outline




23

Assumption of Serial Processing & Optimal Planning

Perception

Cognitive decision

Action• A single movement is planned & executed serially at a time

• An optimal trajectory is planned• Straight trajectories• Bell-shaped velocity profile

( Flash & Hogan, 1985; Uno, Kawato, & Suzuki, 1989; Engelbrecht, 2001)

24

• Reaching trajectories swerves away from distractors ( Tipper, Lortie, & Baylis, 1992; Tipper, Howard, & Houghton, 1998, 2000)

• Dorsal premotor areas encode multiple action plans concurrently, even before cognitive decision is made (Cisek & Kalaska, 2002, 2005)

Multiple Reaching Plans For potential Targets

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

26

Are they concurrently planned?

• Do we execute every hand movement after target selection is finalized? • Or, do we sometimes initiate hand movements before completing target selection? • Can movements to competing stimuli be concurrently processed?

When selecting a target among multiple distractors…

+

Odd-color target

27

Concurrent Processing of Saccades

Target

InitialsaccadeCorrective

saccade

(McPeek, Skavenski, & Nakayama, 2000)

28

Concurrent Processing of Visually-Guided Manual Pointing

vs.

Tiny mass/inertia

High mass/inertia

Hand/arm

29

+

Trial 1: Odd-color target

Main Experiment

+

Trial 2: Single target (Baseline)

Trial 3

Trial 4+

+

. .

30

Down

Left Right

Up

Single target trials

Movement Trajectories:Single Target

31

Down

Left Right

Up



32

Down

Left Right

Up



33

Odd-color target trials

Down

Left Right

Up

Movement Trajectories:Odd-Color Target


34

Down

Left Right

Up


Odd-color target trialsSingle target trials

35

Down

Left Right

Up



36

Down

Left Right

Up



37

Are Initial & Corrective Movements Planned Concurrently ?

• Curved trajectories- Influences of competing distractors on planning & execution of pointing movements

• Are initial & corrective movements concurrently planned, overlapping in time?

• Difficulty of decomposing curved trajectories into discrete two movements

38

Estimate of Corrective Movement Onset

Average Baseline Trajectory

Time (ms)

Pos

ition

(inch

es)

0 40 80 120 160 200 240 280 320 360-5

-4

-3

-2

-1

0

1

2

3

4

5

Left

Cen

ter

Rig

ht

-1.5SD

+1.5SD

InitialMovement

CorrectiveMovement

StimulusOnset

+

Initial latency

Corrective latency

39

Corrective Latency vs. Initial Latency

8 subjects

243 Trials

0

10

20

30

40

50

60

0 50 100 150 200 250 300 350 400 450 500

Latency(ms)

Number of trials

Corrective latencyInitial latency

40

Difference Between Corrective & Initial Latencies

243 Trials

0

5

10

15

20

25

30

35

-400 -300 -200 -100 0 100 200 300 400

Differece betweencorrective and initial latencies (ms)

Number of trials

8 subjects

41

Time Cost Of Executing two Concurrently Planned Movements

Vel

ocity

alo

ng t

he c

urve

500 10000

10

20

30

40

Time(ms)

Curved Straight

Straight

Curved

Odd-Color

+

Non-optimal properties

42

Time Cost Of Executing two Concurrently Planned Movements

8 subjects

0

100

200

300

400

500

600

700

800

Initial latency Movementduration

Total time

Time (ms)

Curved trajectory

Straight trajectory

43

Displaced target

Initial target

~150-250 ms

+

++

Fixation

Time

35% Trials: Displaced

65% Trials: Baseline

Time

+

+

Fixation

Control Experiment:Double-Step Task

44

Control Experiment:Double-Step Task

931 Trials

0

50

100

150

200

250

300

0 50 100 150 200 250 300 350 400 450 500

Latency(ms)

Number of trials

Corrective latencyInitial latency

45

Discussion

• The cost of planning corrective movements may be negligible because these can be planned in parallel with initial movements.

• Like saccades, pointing movements to competing stimuli can be concurrently processed.

• Hand movements are sometimes initiated before cognitive decision-making for the correct target is completed.

• The visuo-motor system is less concerned with planning a single optimal trajectory as long as the ultimate talk goal is achieved

46

Outline




47

Down

Left Right

Up

Single Target

• Fast initiation & straight trajectory

Influence of Trial Difficulty on Manual Pointing Strategies

+


48

Single Target

Down

Left Right

Up



+


49

Single Target

Down

Left Right

Up



+


50

Odd-Color Target

• Slow initiation & straight trajectory

• Fast initiation but curved trajectory


+


51

Odd-Color Target




+


52

Odd-Color Target




+


53

• How does intermixing trials of varying difficulty influence latency & accuracy criteria of visually-guided manual pointing?

• What mechanism adjusts latency & accuracy criteria in visually-guided manual-pointing tasks?

• How does the time course of latency & accuracy criteria adjust?

Questions

54

• Predictability Knowing upcoming trial type may lead to initiation

latency & accuracy adjustment

• Trial Type RepetitionContinuous repetition of the same type trials may lead

to initiation latency & accuracy adjustment

Initiation latency & accuracy criteria for trials of

varying difficulty may be influenced by either …

Predictability vs. Repetition

55

Alternating(Odd-color)

Mixed

++

+

+

. .

.

+

. .

.

+

+

+

(Single)

+

+

+

++

+

+

+

. .

. .

. .

Exp1. Dissociation of Predictability & Past Experience

To compare influences of predictability and past trial type repetition on initiation latency & accuracy

adjustment

Blocked

56

Exp1. Dissociation of Predictability & Past Experience

Prediction for Adjustment

Blocked Mixed Alternating

Predictability Yes No Yes

Trial Repetition

Yes No No

Cognitive knowledge of the next trial but frequent trial type switch

57



Trial Repetition

Yes No No

200

250

300

350

400

Blocked Mixed Alternated

Single targetOdd color target

N=9

Initiation Latency

√

Adjustment of Initiation Latency Criteria

Alternating

Initi

atio

n la

tenc

y(m

s)Not simply due to task

switchingNo difference in error

rates ( over 98%)

58



Trial Repetition

Yes No No

Movement duration

√


300

350

400

450

500


Movement duration(ms)

59

.

200

250

300

350

400



N=9


Alternating

• Homogenization: Initiation latency difference vanished in the mixed & alternating conditions

• No initiation latency differentiation by cognitive knowledge in the alternating condition

• Adjustment of initiation latency criteria by trial type repetitions

+ +++ +Trial sequence

0 0 1 0 1

+

2

Initiation Latency

60

-50

-25

0

25

50

75

100

1 over 2

Number of the same type trial repetition

0

.

+ +++ +Trial sequence

0 0 1 0 1

+

2

Trial-by-Trial Initiation Latency Adjustment

Moment-to-Moment cumulative learning

61

.

200

250

300

350

400



N=9

Dissociation of Predictability & Past Experience in Initiation Latency

Alternating

-50

-25

0

25

50

75

100

1 over 2


0

.

No initiation latency differentiation by cognitive knowledge

Cumulative learning from recent experience

Dissociation of the two sources Motor initiation latency is

adjusted by recent experience not by cognitive knowledge

62

Adjustment of Accuracy Criteria

Cost of sub-optimal adjustment of latency criteria?

.

200

250

300

350

400



N=9

Initiation Latency

Alternating

63

Movement Trajectories

Left Right (Inch)

Up

(Inc

h)D

own

Single target

Odd-color target


-5 0 50

2

4

6

8

10

12

-5 0 50

2

4

6

8

10

12

-5 0 50

2

4

6

8

10

12

-5 0 50

2

4

6

8

10

12

-5 0 50

2

4

6

8

10

12

-5 0 50

2

4

6

8

10

12

64

Maximum curvatures.

0.00

0.05

0.10

0.15

0.20



N=9

Alternating

More curved trajectory trials in mixed & alternated conditions

65

.

200

250

300

350

400



N=9

Exp2: Cumulative Learning in Predictable Sequence

Alternating

-50

-25

0

25

50

75

100

1 over 2


0

.

Cumulative learning from recent experience

Questions• Time course with longer sequence?• Predictable sequence?

66

Exp2: Cumulative Learning in Predictable Sequence

+ +

++ +

0

+

+

+

++

1 2 3 4 0 1 2 3 4

Odd-color

200

250

300

350

400

0 1 2 3 4

N=9


Singe target

Odd-color target

switching

• 5 single/odd-color target alternation• Alternation informed in advance & Visual cue

67

• Intermixing trials of varying difficulty eliminates differences between initiation latencies of easy and difficult trials

Discussion

.

200

250

300

350

400



N=9

Alternating

• It is Not task switching : both easy & difficult trials slowed by intermixing

(Allport et al., 1994)

Initiation latency in Exp1

68

• Repetition of trials with the same difficulty gradually differentiates initiation latencies by cumulative learning

Discussion

200

250

300

350

400

0 1 2 3 4

N=9


Singe target

Odd-color target

• It is not perceptual priming : both easy and difficult trials facilitated by repetition

(Maljkovic & Nakayama, 1994)

Initiation latency in Exp 2

69

• It is not specific to input or output systems: Also, no difference between easy and difficult trials in reading aloud tasks (Lupker et al., 1997)

Discussion

70

Fast initiation

• Central mechanism?: Determine sensorimotor readiness by very recent experience: Adjustable gain or threshold setting

Discussion

Time

Threshold

Deci

sion(A

ctiv

ati

on)

Slow initiation

71




72

• Simple visually-guided action requires the allocation of focused attention to the target • Multiple planning & execution of manual pointing toward competing stimuli can occur

• Visuo-motor action & target selection process are interacting continuously even after motor execution

Conclusion

73

• Visuo-motor system is less concerned with planning a single optimal trajectory as long as the ultimate task goal is successfully achieved

• Application: Instantaneous read-out Perceptual/cognitive processing

Conclusion

74

Acknowledgements

• Ken Nakayama• Patrick Cavanagh• Yuhong Jiang• Anne Grossetete• Harvard Vision Lab

75

Thank you!

1 visually-guided motor actions in humans ken nakayama and joo-hyun song harvard university vision...

Documents