computational motor control: state space models for motor adaptation (jaist summer course)

Computational Motor Control Summer School03: State space models for motor adaptation.

Hirokazu Tanaka

School of Information Science

Japan Institute of Science and Technology

State-space modeling of motor adaptation.

In this lecture, we will learn:

• Motor adaptation paradigms

• Continuous-time state-space models

• Discrete-time state-space models

• Controllability

• Observability

• State-space description for motor adaptation

• Multi-rate models

• Motor memory of errors

• Mirror reversal (non-error based learning)

Motor adaptation paradigms to dynamical perturbations: Force-field adaptation.

Shadmehr & Mussa-Ivaldi (1994) J Neurosci

Baseline (no field) Initial exposures

adaptation catch trials

Motor adaptation paradigms to kinematical perturbations: Visuomotor rotation.

Krakauer et al. (2000) J Neurosc; Krakauer (2009) Progress in Motor Control

Adaptation to prism displacements.

Martin et al. (1996) Brain ;Kitazawa et al. (1995) J Neurosci

Adaptation to prism displacements.

Kitazawa et al. (1995) J Neurosci

1 1n n ne e ke 1

1

1

n

in

i

e e k e

Continuous-time state-space models.

F ma mx x v

Fv am

x

v

x

Newton’s equation of dynamics

0 1 0

0 0 1/

x xF

v v m

x Ax Bu

x Ax Bu

State-space representation

A x B u

State-space vector

Discrete-time state-space models.

Discrete-time representation

k k t x x

1 ( 1)k

k k

k k k

k k

k t

t

t

t t

x x

x x

x Ax Bu

I A x B u

1ˆ ˆ

k k k x Ax Bu

2

2ˆ

ˆ t

t

t t

te t

e

t

A

A

A I A

B BB

t

Δt 2Δt 3Δt (k-1)Δt kΔt (k+1)Δt0

k-1 k k+10 1 2 3

time (continuous)

time steps (discrete)

Deterministic and stochastic state-space models.

1k k k

k k

x Ax Bu

z Cx

1

k

kk k

kk

k

w

v

x Ax Bu

z Cx

Deterministic Stochastic

XkXk-1 Xk+1

zk-1 zk zk+1

uk-1 uk uk+1

Linear time-variant and time-invariant state-space models.

1k k k

k k

k k

k

x x u

xCz

A B

Time-variant model

1k k k

k k

x x u

xCz

A B

Time-invariant model

Throughout these lectures, we will use linear time-invariant (LTI) models for mathematical simplicity.

State-space models in an explicit component form.

1k k k x Ax Bu

k kz Cx

1, 1

2, 1

, 1

k

k

N k

x

x

x

1,

2,

,

k

k

N k

x

x

x

11 12 1

21 21

11 1

N

N

a a a

a a

a a

11

21

1

1

L

N NL

b b

b

b b

1

L

u

u

= +

1,

2,

,

k

k

N k

x

x

x

1

M

z

z

11 12 1

1 112

N

MM

c c c

c c c

=

Process equation

Measurement equation

N vector N×N matrix N vector N×L matrix L vector

M vector M×N matrix N vector

Controllability: the ability of driving a system into desired final state.

1k k k x Ax Bu

, , ,N L N N

k k

N L x u A B

Controllability is the ability of external inputs {uk} to drive a state from any initial condition to any final condition in a finite time. A state-space model is controllable if the N×NL controllability matrix has full row rank:

2 1n B AB A B A B

Sketch of proof:

0

1 1

2

2 2 1

1

1

0

2

N N N

N N

NN

N

N

N

x Ax Bu

A x ABu Bu

u

uA x B AB A B

uKalman (1963) SIAM J Contr

Observability: determining hidden state from measurements.

k kz Cx

, ,N M N

k

M

k

x z C

Observability is the ability to determine a (latent) state from a sequence of measurements {zk}. A state space model is called observable if the MN×N observability has full rank N:

Kalman (1963) SIAM J Contr

1N

C

CA

C A

State-space models for dynamic (force-field) motor adaptation.

Thoroughman & Shadmehr (2000) Nature; Donchin et al. (2003) J Neurosci

1n n n

n n n

x Ax Bu

z Cx Du

State-space models for kinematic (visual rotation) motor adaptation.

Tanaka et al. (2006) J Neurophysiol

T

1

k k k k

k k k

z

z

x Ax BH

H x

Trial-by-trial generalization width reflects directional tuning width.

1

1i i N

i

N

g

g

g

r r r r Rg

T

k k k rR g

1 1 1

T

k k k k k k k k R R g R g gr g

Suppose that, for target direction θ, the motor output is a weighted sum of population activity {gi(θ)} multiplied with preferred directions {ri}:

A gradient descent learning rule specifies the change of preferred directions according to the movement error Δrk and the population activity {g(θk)} :

This change affects the motor output at the next trial as:

Two-rate model of motor adaptation: fast and slow learners.

Smith et al. (2006) PLoS Biol

1n n nu x Ax B

f ff f

1

s ss s

1

0

0

n n

n

n n

x xa bu

x xa b

n nz Cx

f

f s1

1 1s

1

1 1n

n n n

n

xz x x

x

f s s f,a a b b

There are two learners in the brain; the fast learner (x(f)) learns quickly but forgets quickly, while slow learner (x(s)) learns slowly but maintains its memory longer.

Motor output is a sum of the fast and slow learners.

State vector consists of fast (x(f)) and slow learners (x(s)).

The model explains savings, spontaneous recovery.


Savings Spontaneous recovery

The prediction of spontaneous recovery is confirmed in humans.


The slow process contributes to motor memory consolidation.

Joiner & Smith (2008) J Neurophysiol

The slow process, but not the fast process, contributes to motor memory consolidation.

Explicit (strategic) and implicit (error-based) learning.

Mazzoni & Krakauer (2006) J Neurosci

Strategy (aiming the adjacent target) cancels the “error” without any adaptation!

Explicit (strategic) and implicit (error-based) learning.

Mazzoni & Krakauer (2006) J Neurosci

What is “motor error?”: Aiming error and target error.

Taylor & Ivry (2011) PLoS Comp Biol; Taylor & Ivry (2014) Prog Brain Res

State-space model for strategic and error-based learning.


yn: target directionrn: rotation anglexn: adaptation variablesn: strategy variable

yn

sn

sn-rn+xn

aiming

n n n nn n ne s s r x r x

target

n n n n n n nn ne y s r x y sx r

aiming

netarget

ne

State-space model for strategic and error-based learning.


yn

sn

sn-rn+xn

aiming

n n n nn n ne s s r x r x

target

n n n n n n nn ne y s r x y sx r

aiming

netarget

ne

aiming

targ

1

e

1

t

n n n

nn n

x ax be

s cs de

a=0.99, b=0.015,

c=0.999, d=0.022

Steepest descent learning rule for optimization.

Lecture 6, in Neural Networks for Machine Learning, Geoff Hinton

E( 1) ( )n n E

w w

w

optimum

E

w

E

w

Descent learning rule:

RPROP: Adjustment of learning rate.

E

wgradient

learning rate

1 1

Motor memory of experienced errors.

Herzfeld et al. (2014) Science

( ) ( ) ( )

( 1) ( ) ( ) ( )

ˆ

ˆ ˆ

n n n

n n n n

e y y

x ax e

( )

( )

( )

( )

( )

: perturbation

ˆ : estimated perturbation ("belief")

: sensory consequence

ˆ : predicted sensory consequence

: control signal

n

n

n

n

n

x

x

y

y

u

State-space model: memory of environments

Population-coding model: memory of errors

( ) ( )n n

i i

i

w g e

2

2exp

2

i

i

e eg e

( 1)

( 1) ( 1) ( 1) ( )

T ( 1) ( 1)sgn

n

n n n n

n n

ee e

e e

gw w

g g error

acti

vity

w1 w2 w3 wn

η

Motor memory of experienced errors.

Herzfeld et al. (2014) Science

Displacement and left-right reversal: Why so different?

Martin et al. (1996) Brain; Sekiyama et al. (2000) Nature

Displacement prism

… takes only few dozen trials. … takes a few weeks.

Left-right reversed prism

Day 3

Day 34

Mirror reversal: a distinct form of motor adaptation?

Taglen et al. (2014) J Neurosci; Lilicrap et al. (2013) Exp Brain Res

Movement number Movement number

Ab

solu

te e

rro

r

Ab

solu

te e

rro

r

Visual rotation Mirror reversal

Summary

• A state-space model consists of a process equation (temporal transition) and an observation equation (measurement).

• Humans are flexible to a novel environment, known as motor adaptation, such as perturbations of force fields and visual transformation.

• State-space modeling has been very successful in describing trial-by-trial adaptation processes in humans.

References

• Thoroughman, K. A., & Shadmehr, R. (2000). Learning of action through adaptive combination of motor primitives. Nature, 407(6805), 742-747.

• Donchin, O., Francis, J. T., & Shadmehr, R. (2003). Quantifying generalization from trial-by-trial behavior of adaptive systems that learn with basis functions: theory and experiments in human motor control. The Journal of Neuroscience, 23(27), 9032-9045.

• Tanaka, H., Sejnowski, T. J., & Krakauer, J. W. (2009). Adaptation to visuomotor rotation through interaction between posterior parietal and motor cortical areas. Journal of Neurophysiology, 102(5), 2921-2932.

• Smith, M. A., Ghazizadeh, A., & Shadmehr, R. (2006). Interacting adaptive processes with different timescales underlie short-term motor learning. PLoS Biol, 4(6), e179.

• Joiner, W. M., & Smith, M. A. (2008). Long-term retention explained by a model of short-term learning in the adaptive control of reaching. Journal of Neurophysiology, 100(5), 2948-2955.

• Inoue, M., Uchimura, M., Karibe, A., O'Shea, J., Rossetti, Y., & Kitazawa, S. (2015). Three timescales in prism adaptation. Journal of Neurophysiology, 113(1), 328-338.

• Mazzoni, P., & Krakauer, J. W. (2006). An implicit plan overrides an explicit strategy during visuomotor adaptation. The Journal of Neuroscience, 26(14), 3642-3645.

• Taylor, J. A., & Ivry, R. B. (2011). Flexible cognitive strategies during motor learning. PLoS Comput Biol, 7(3), e1001096-e1001096.

• Taylor, J. A., & Ivry, R. B. (2014). Cerebellar and prefrontal cortex contributions to adaptation, strategies, and reinforcementlearning. Progress in Brain Research, 210, 217.

• Taylor, J. A., Krakauer, J. W., & Ivry, R. B. (2014). Explicit and implicit contributions to learning in a sensorimotor adaptation task. The Journal of Neuroscience, 34(8), 3023-3032.

• Telgen, S., Parvin, D., & Diedrichsen, J. (2014). Telgen, S., Parvin, D., & Diedrichsen, J. (2014). Mirror reversal and visual rotation are learned and consolidated via separate mechanisms: Recalibrating or learning de novo?. The Journal of Neuroscience, 34(41), 13768-13779.?. The Journal of Neuroscience, 34(41), 13768-13779.

• Lillicrap, T. P., Moreno-Briseño, P., Diaz, R., Tweed, D. B., Troje, N. F., & Fernandez-Ruiz, J. (2013). Adapting to inversion of the visual field: a new twist on an old problem. Experimental Brain Research, 228(3), 327-339.

• Ostry, D. J., Darainy, M., Mattar, A. A., Wong, J., & Gribble, P. L. (2010). Somatosensory plasticity and motor learning. The Journal of Neuroscience, 30(15), 5384-5393.

Exercise

• Simulate the state-space model proposed by Taylor and Ivry.

• Mirror reversal is different from most adaptation paradigms in that learning from error worsens the performance. Can we consider a state-space model for mirror reversal?