neural prosthetics iv: pushing preferred directions around beata jarosiewicz laboratory of andrew...

Neural Prosthetics IV:

Pushing preferred directions around

Beata JarosiewiczLaboratory of Andrew Schwartz

The explanatory gap• Absolute gap:

Why is neural activity accompanied by any conscious experience at all?

• Comparative gap:Why does certain neural activity give rise to, say, visual rather than auditory experience? Or to the experience of red instead of green?

‘Red’

670 nm wavelength light

Neural activity resulting from 670 nm

light

The explanatory gap• Absolute gap:

Why is neural activity accompanied by any conscious experience at all?

• Comparative gap:Why does certain neural activity give rise to, say, visual rather than auditory experience? Or to the experience of red instead of green?

‘Red’

670 nm wavelength light

Neural activity resulting from 670 nm

light

absolute gap

The explanatory gap• Absolute gap:

Why is neural activity accompanied by any conscious experience at all?

• Comparative gap:Why does certain neural activity give rise to, say, visual rather than auditory experience? Or to the experience of red instead of green?

‘Red’ ‘Green’

670 nm wavelength light

Neural activity resulting from 670 nm

light

530 nm wavelength light

Neural activity resulting from 530 nm

light

comparative gap

absolute gap

Cortical “deference” vs. “dominance”

from “Neural Plasticity and Consciousness” (Hurley & Noë, Biology & Philosophy, 2003)

Cross-modal remapping: “vOICe”

Left and Right Image is scanned left to right, one snapshot per second. So left and right get encoded by time (and stereo).

Dark and Light Brightness is encoded by loudness: the brighter the visual pattern, the louder the sound.

Up and Down Elevation is encoded by pitch: the higher the position of the visual pattern, the higher the pitch.

Movie by Alex Storer, BU

Cross-modal remapping: “vOICe”2000:

“Well the other day I was again washing dishes. I had let the water out of the sink and turn to get a towel to dry my hands. Then when I turn back to rinse the sink I was stunned to see the sink in a “depth” like image. I stepped away from the sink and walked slowly up to it again to see if my mind was playing tricks on me. No, the feeling of seeing depth in the sink bowl was still there.”

2002:“Just sound?.... No, It is by far more, it is sight! There IS true light perception generated by the vOICe. When I am not wearing the vOICe the light I perceive from a small slit in my left eye is a grey fog. When wearing the vOICe the image is light with all the little greys and blacks. Yet a definite light image.”

- Pat Fletcher, long-term vOICe user

Cross-modal remapping: “vOICe”2000:

“Well the other day I was again washing dishes. I had let the water out of the sink and turn to get a towel to dry my hands. Then when I turn back to rinse the sink I was stunned to see the sink in a “depth” like image. I stepped away from the sink and walked slowly up to it again to see if my mind was playing tricks on me. No, the feeling of seeing depth in the sink bowl was still there.”

2002:“Just sound?.... No, It is by far more, it is sight! There IS true light perception generated by the vOICe. When I am not wearing the vOICe the light I perceive from a small slit in my left eye is a grey fog. When wearing the vOICe the image is light with all the little greys and blacks. Yet a definite light image.”

- Pat Fletcher, long-term vOICe user

What caused the switch from cortical dominance to cortical deference?

Maybe what’s important is how the information is interpreted downstream.

Is there a way we can experimentally change how the activity of particular neurons is interpreted downstream, and test whether they can change to acquire their newly-assigned function?

What caused the switch from cortical dominance to cortical deference?

Maybe what’s important is how the information is interpreted downstream.

Is there a way we can experimentally change how the activity of particular neurons is interpreted downstream, and test whether they can change to acquire their newly-assigned function?

What caused the switch from cortical dominance to cortical deference?

Maybe what’s important is how the information is interpreted downstream.

Is there a way to experimentally manipulate the way neural activity is interpreted downstream, and test whether the neurons “oblige”?

There is a way, with brain control!

• With the brain control paradigm, we know the precise relationship between the activity of each of the neurons under study and the “behavioral” output.

• This allows for a new kind of perturbation that can target selected subsets of neurons by altering the way those neurons contribute to the object’s movement, and to test whether neurons can change their activity specifically and selectively when the monkey learns to compensate for the perturbation.

3D center-out task under “brain control”

zzyyxx vbvbvbb 0

Cosine tuning with (intended) movement

direction

cosvbbf 0

iii b(t)f(t)v ''

Population Vector Algorithm

1) Adaptive: • Each cell’s tuning function (PD and modulation depth) was estimated by

iteratively regressing firing rate against target direction as the monkey performed the brain control task.

• Once the monkey’s performance stabilized, the tuning functions obtained from this iterative regression were frozen and used for decoding (“dPDs”).

2) Control 1 (C1): • Monkey performed brain control using the decoding

parameters obtained from the adaptive session.

3) Perturbation (P): • A subset (~25%) of the recorded units (the

“perturbed units”) were given reassigned dPDs by rotating their original dPDs 90 degrees about the x, y, or z axis (chosen randomly each day).

4) Control 2 (C2): • The perturbation was removed (i.e. the original

dPDs were reinstated).

Experimental setup: 4 sessions

Direction of perturbation

Deflection in early trajectory

Monkey learns to control cursor by end of Perturbation session

C1

Early P

Late P

Possible neural bases1) The perturbed units could have suppressed their contribution to

the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths.

2) The perturbed units could have shifted their actual PDs toward their reassigned dPDs.

3) The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

Possible neural bases1) The perturbed units could have suppressed their contribution to

the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths.

2) The perturbed units could have shifted their actual PDs toward their reassigned dPDs.

3) The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

Possible neural bases1) The perturbed units could have suppressed their contribution to

the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths.

2) The perturbed units could have shifted their actual PDs toward their reassigned dPDs.

3) The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

Target

Deflection caused by

perturbation

Re-aiming direction

Testing the possible neural bases

• To obtain each unit’s “measured” tuning function in each session, the same linear regression was done on all firing rates vs. target directions in that session (but post-hoc):

• measured PD = direction for which its firing rate was the highest (direction at the peak of the cosine fit)

• measured modulation depth = the magnitude of the fitted cosine (the difference between the peak firing rate and the baseline firing rate)

zzyyxx dbdbdbbf 0

Modulation depths did not change

Possible neural bases1) The perturbed units could have suppressed their contribution to

the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths.

2) The perturbed units could have shifted their actual PDs toward their reassigned dPDs.

3) The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

Target

Deflection caused by

perturbation

Re-aiming direction

Testing for shift in measured PDs

O

azimuth

MC1

0

0

MR

R

elev

atio

n

M

M

C1 -> P

P -> C2

“Dose-response”

Possible neural bases1) The perturbed units could have suppressed their contribution to

the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths.

2) The perturbed units could have shifted their actual PDs toward their reassigned dPDs.

3) The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

Target

Deflection caused by

perturbation

Re-aiming direction

Conclusions• This study demonstrates selective functional plasticity, even when

only a global feedback signal is available.

• How can the brain do that? Perhaps noise allows it to explore synaptic weight spaces that optimize cursor control?

• We still don’t know where or how the changes took place that underlie the PD shifts in motor cortex.

• Further studies using the brain control paradigm promise a deeper understanding of plasticity as a neural substrate of learning.

• In combination with the dominance/deference studies, this result might have interesting implications for “neural coding”: perhaps it isn’t as important what “activates” a neuron (i.e. its “tuning function”) – as how that neuron’s activity is used downstream.

Thank you!

• Rob Kass• Meel Velliste• George Fraser• Chance Spalding • Ingrid Albrecht• Nathaniel Daw• Valerie Ventura

• NIH-NINDS-NO1-NS-2-2346

Introduction

• Many studies have shown that neurons can change their “coding” properties in association with learning.

• In motor cortex, neurons can change their tuning functions when monkeys adapt to perturbations that interfere with the execution (e.g. Bizzi and collaborators) or visual feedback (e.g. Wise, Paz, Vaadia) of their movements.

• With the brain control paradigm, we know the precise relationship between the activity of each of the neurons under study and the “behavioral” output.

• The brain control paradigm also allows for a new kind of perturbation that can target selected subsets of neurons by altering the way those neurons contributes to the object’s movement; this makes it possible to test whether neurons can change their activity specifically and selectively when the monkey learns to compensate for the perturbation.

Cursor deflection resulting from perturbation

Expected Actual

Rotation axes: red = X

green = Y blue = Z

Expected cursor perturbations

rotation axis:

X = redY = greenZ = blue

cos)(tvbbf(t) 0

ii

0m

0i bbb

b(t)f(t)v

ii

i '

The “Population Vector” algorithm:

= firing rate at time t

= baseline firing rate

= cell’s preferred direction

= (intended) movement velocity at time t

= angle between the movement direction and the cell’s preferred direction

b

)(tv

0b

f(t)

ib '

imb

= normalized preferred direction of cell i

= maximum firing rate of cell i

Estimating preferred directions:

zzyyxx vbvbvbb 0

Re-aiming occurs later in trajectory