Modelin

gdecisio

nmak

ingdefi

citsin

frontostriatal

diso

rders

Mich

aelFran

kLab

orato

ryforNeu

ralComputatio

nan

dCognitio

nBrownUniversity

Computatio

nal

Psych

iatryan

d...

Neu

rogen

ocomputomics

•Man

ydiso

rders

broad

lych

aracterizedbych

anges

inmotiv

ation

•Sev

eralfro

nto-striatal

diso

rders

hav

esu

bstan

tialgen

eticheritab

ility

•Individual

differen

cesin

reinforcem

entlearn

ing?

Computatio

nal

Psych

iatryan

d...

Neu

rogen

ocomputomics

•Man

ydiso

rders

broad

lych

aracterizedbych

anges

inmotiv

ation

•Sev

eralfro

nto-striatal

diso

rders

hav

esu

bstan

tialgen

eticheritab

ility

•Individual

differen

cesin

reinforcem

entlearn

ing?

•But...

Can

didate

gen

eeffects

aregen

erallysm

all

•Which

gen

es?Which

task?Which

measu

re?

Computatio

nal

Psych

iatryan

d...

Neu

rogen

ocomputomics

•Man

ydiso

rders

broad

lych

aracterizedbych

anges

inmotiv

ation

•Sev

eralfro

nto-striatal

diso

rders

hav

esu

bstan

tialgen

eticheritab

ility

•Individual

differen

cesin

reinforcem

entlearn

ing?

•But...

Can

didate

gen

eeffects

aregen

erallysm

all

•Which

gen

es?Which

task?Which

measu

re?

•N

eedth

eoreticalm

od

el!(an

dco

nvergin

gp

harm

acolo

gy/im

agin

g)

Fran

k&

Fossella,2011;

Maia

&Fran

k,2011;

Huyset

al,2011

Rein

forcem

entlearn

ingan

ddopam

ine:

pred

ictionerro

rs

Positiv

ePE:

Neg

ativePE:

dopam

ine:

Montag

ue,D

ayan

&Sejn

owksi96;

Doya,2002;

O’Reilly,

Fran

k,H

azy&

Watz

06...

δ(t)

=(

r(t)

+γV̂(t+

1))

−V̂(t)

Rein

forcem

entlearn

ingan

ddopam

ine:

pred

ictionerro

rs

Positiv

ePE:

Neg

ativePE:

dopam

ine:

Montag

ue,D

ayan

&Sejn

owksi96;

Doya,2002;

O’Reilly,

Fran

k,H

azy&

Watz

06...

δ(t)

=(

r(t)

+γV̂(t+

1))

−V̂(t)

D1effects

onstriatal

learning:Positiv

ePE

D1effects

onstriatal

learning:Positiv

ePE

Three

factorlearn

ing:presy

nap

tic,postsy

nap

tican

dDA

D2effects

onstriatal

learning:Neg

ativePE

Fran

k2005

Neu

ralmodel

ofbasal

gan

glia

anddopam

ine

Integ

ratesawideran

geofdata

into

asin

gle

coheren

tfram

ework

Sep

arateGoan

dNoGopopulatio

nsinteg

ratestatistics

ofrein

forcem

ent

Input

Stria

tum

Thalamus

GPi/S

Nr

preSMASTN

SNc

Go

NoGo

GPe

ge ≈i w

ij >+

i wij

≈V

m−Θ

[]+

Vm

−Θ[

]++

1jnet

<x

x

βN=

yγ

γge ge

Vm

=

c

gg

g iigll

+ ++...

Ee

m V[

]

EV

[i

]mE

V[

l]m

∆w

ij≈

(xpi y

pj )−(xti y

tj )

Fran

k,2005,

2006JCogNeu

rosci,N

eural

Netw

orks

Max

imizin

gRew

ardvia

RTAdap

tation:

Tem

poral

Utility

Integ

rationTask

01000

20003000

40005000

Tim

e (ms)

0.00.10.20.30.40.50.60.70.80.91.0

ProbabilityC

EV

DE

VIE

VC

EV

R

Rew

ard F

requ

ency

01000

20003000

40005000

Tim

e (ms)

0 50

100

150

200

250

300

350

# Points Gained

CE

VD

EV

IEV

CE

VR

Rew

ard M

agn

itud

e

01000

20003000

40005000

Tim

e (ms)

0 5 10 15 20 25 30 35 40 45 50 55 60

Expected Value (freq*mag)

CE

VD

EV

IEV

CE

VR

Exp

ected V

alue

RLmodel:

Fitto

data

across

allsu

bjects

RLmodel

:ad

just

RTsas

afunctio

nofrew

ardpred

ictionerro

rs

Fran

k,D

oll,O

as-Terp

stra&

Moren

o(2009,

Natu

reNeu

roscien

ce)

Neu

rogen

etican

dpharm

acological

modulatio

nof

reinforcem

entlearn

ingparam

eters

Fran

k&

Fossella,2011

Single

subject

Data...

010

2030

4050

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000S

ingle Subject C

EV

RT (ms)

Trial

010

2030

4050

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000S

ingle Subject D

EV

RT (ms)

Trial

010

2030

4050

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000S

ingle Subject IE

V

RT (ms)

Trial

010

2030

4050

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000S

ingle Subject C

EV

R

RT (ms)

Trial

Exploratio

nvsExploitatio

n

•Byexploitin

glearn

edstrateg

ies,weknow

wecan

get

acertain

amount

ofrew

ard

•Butdon’tknow

how

gooditcan

get.

⇒Need

toE

xplo

re

•Theo

ry:Explore

based

onrelativ

euncertain

tyab

outwheth

erother

actionsmightyield

better

outco

mes

than

statusquo

(Day

an&

Sejn

owksi96)

Exploratio

nvsExploitatio

n

•Byexploitin

glearn

edstrateg

ies,weknow

wecan

get

acertain

amount

ofrew

ard

•Butdon’tknow

how

gooditcan

get.

⇒Need

toE

xplo

re

•Theo

ry:Explore

based

onrelativ

euncertain

tyab

outwheth

erother

actionsmightyield

better

outco

mes

than

statusquo

(Day

an&

Sejn

owksi96)

Uncertain

ty-Based

Exploratio

n

510

1520

2530

3540

4550

−4000

−3000

−2000

−1000 0

1000

2000

3000

4000E

xploration

Trial

RT Diff (ms)

Single S

ubject, CE

V Model E

xp termR

T diff

PFCGen

e-Dose

Effect

onUncertain

ty-Based

Exploratio

n

0.000.050.100.150.200.250.300.350.400.450.50

(x 1e4)val/valval/m

etm

et/met

CO

MT

gene-dose effectsU

ncertainty-exploration parameter

ε

Fran

k,Doll,O

as-Terp

stra&

Moren

o(2009,

Natu

reNeu

roscien

ce)

Does

thebrain

trackrelativ

euncertain

tyforexploratio

n?

Does

thebrain

trackrelativ

euncertain

tyforexploratio

n?

ǫ>

0(’ex

plorers’)

explorers

>non-ex

plorers

Bad

re,Doll,L

ong&

Fran

k,under

review

EEG

reveals

temporal

dynam

ics

EEG

reveals

temporal

dynam

ics

Relativ

euncertain

tyrep

resented

prio

rto

choice,

andmore

soin

explorato

rytrials

Cav

anag

h,Cohen

,Figuero

a&

Fran

k,under

review

Neg

ativesymptomsin

schizo

phren

ia:Uncertain

ty-Based

Exploratio

n

(uncert)0.00

0.050.100.15

0.200.250.30

0.350.40

(x 1e4)

SZ

CN

Un

certainty-d

riven exp

loratio

n

ε

**

ε0

12

34

Glo

bal A

nh

edo

nia

-1.2-1.0-0.8-0.6-0.4-0.2 00.20.40.60.8

(x1e4)

An

hed

on

ia & E

xplo

ration

r = -.44, p =

.002

ε•

Anhed

onia

=beh

avioral

componen

tofrew

ardseek

ing(e.g

.,initiatin

gsocial/

recreational

activities)

notcap

acityto

experien

cepleasu

re

•Anhed

onia

relatedto

exploratio

nan

dnotlearn

ingfro

mrew

ardpred

ictionerro

rs

Strau

sset

al,2011,

Biological

Psych

iatry

Obsessiv

eCompulsiv

eDiso

rder:

Aversio

nto

Uncertain

ty

gainslosses

-0.4

-0.2

0.0

0.2

0.4

0.6

(x 1e4)

CN

OC

D

Un

certainty-d

riven exp

loratio

n

ε

prelim

inary

data,N

=17

per

group

with

Masch

avan

’tWout,B

enGreen

berg

,Stev

eRasm

ussen

Summary

•Dopam

inemodulates

reinforcem

entlearn

ingan

dch

oice

based

on

positiv

ean

dneg

ativeoutco

mes:

patien

ts,pharm

acology,g

enetics,

imag

ing

•Prefro

ntal

cortex

tracksoutco

meuncertain

tyso

asto

reduce

it

•Disru

ptio

nofthese

mech

anism

sisasso

ciatedwith

fronto-striatal

diso

rders,

Park

inson’s,sch

izophren

ia,OCD

•Models

integ

ratebetw

eenmultip

lelev

elsofan

alysis:

neu

ralmech

anism

toab

stractcomputatio

n(see

Thomas

Wieck

i

dem

onstratio

ntomorro

w!).

Than

ksTo...

Brad

leyDoll

Christin

aFiguero

aJim

Cav

anag

hDav

idBad

reJeff

Cock

burn

AnneCollin

sThomas

Wieck

iJim

Gold

Ken

tHutch

ison

Masch

avan

’tWout

Nico

leLong

MikeCohen

Ahmed

Moustafa

ScottSherm

an

Thepatien

ts

Lab

forNeu

ralComputatio

nan

dCognitio

n