Lecture 4

XH

XII

XII

f--O

X =L .

t- I f: 2 f=3 4=4 .

X =/ , 2g ... , N

'

t - o , I , 2, - -

, Ty# of paths - NT

Define

• It) : state at time t (SCH - I , 2)

• vans := Max { RCH : SHI - x } .

-

score , reward .

• VCQD is the best we can do

• Goal i find Noll ) and optimal strategy .

Cost , rctix) , need N computations .

# Vctix) = NT

Tt of operations ⇒ N'T

Vail) -+4 K2, = -12 VC 3117=+3 .

UH ,D=t6

↳→

This.

Ub2) -46 VG,2)=t1V =-3 .

C-=3

Dynamioprogrammingprina.pl#

Botta problem

> info THI - JI! Let,xHhoaDdtt # GLADIt . x.Glatt, XHI , Oltl) i Xlto ) '- Xo -

Dehne the value function V : Cfo ,tixlRd→lR .÷÷.s÷::::::÷: ::*::5=0 , Eaxo F V Xo)

TheoremCDynamicprog.P-nhapeDPPJForevery.ESC- Cto its] ,SET

,2- ElRd , we have .

Yes . Z) = ingfffjhctixltlioetlldt t VCTIXK) ) }×

x. Ctl = AH, cacti) e tf I] , X a z

a

¥.

t¥** ,2.

EkeI

+to S T t ,

Prot

Define J': ' 'hefffjhct, xai.atDdt + VEXED} .

① "

J? E VCs, -23.11

Fix E>0 , pick Q : [Sit . T → ⑦ sit .

JCQJ S VCs, E)T E . C by dean of inf MV )Under this control , we have .

VCE, XK) ) 's JE'

hctixlth Htt ICXCti ) )

⇒ JEE SE Ht ,xHhatDdt t VE, xcel )s + JE' ) Lctixethoetildt + INCH ) - JEETS VCs, Z) t e .

⇒ JE S VCS, 2-7 .

Step 2 :"

J'Z VCs . 2- I

"

fix ETO r then I Qi : Csis → OH ft .

fjhctixltioccti ) Itt Vance)) £ JT + E . T

F- Q2 : cc, tis -9 sit . g) tJet' Ktxa cacti ) at + Elton ) S V K,Xk) ) T E .

Definea a f Oct) te CSE]

Out) tf TG ti

Jst' LHIXHI . Oltl ) dtt Efx Hit ) E J" t 2E .

-UGH S JEET

⇒ us,z , s J2t2# II.

Hamilton-facobi-BeflmanEquationCHJB-pxn.fr . - X ez .

Dpp : VBA) = ihofffghcfixuy, acts ) dtt VCTIXKD }µ infinitesimal F- Stas .

VCs, a infffgtttshc - - - sat + Vestas , xcstas)) }

S

X(Stas) = Hs) t Jst"

fat , XHI , O dt

= Eggs -1 As FCS, O ) t o CAS)2-

VC Stasi XCSTASDI VCS,Z ) -1 IsVCs, Z) - AS

+ VCSZJITFCSZ , OCS) ) AS t o (As)

fgst.NL XHI , Idf = LIS, Z , OG) ) Is + OCHS )

#5=inf{¥t Asf 2srcs.at#Vcsa5TfCsizocsD-

Thesz, T.to#J}⇒ 2sVCsz)toin¥{zVGz55fSR, + Usa, 071=0{ It , ,z ) = ICE) . ( terminal cost )

Hamilton- Jacobi - Bellman Equation .

Vlfyz ) - Eez ) { flow . 8th#Ffv)

IsV t 'out fffzv ) O t £02 ). ⇒C-IR.

⇒.

Implicational ✓ original omtrol problem .

Veto , Xo) =ihofJCQT@ssum.e)Fix s. 2- I E's.z CS , til -1 ⑦ is optimal .

①PP ⇒Vcs, z Ia chef Let, x , O Idt t VEIXK) ) ).

= Jst LH, xs.IM , Ofa Al )It t VK , KD{O*:= OI

. ix. h. J Taylor expansion .

- as""""

:÷g÷÷:¥÷÷s¥i÷";

- LCS,x7s) DTD ) - Fans, CSDTFCS , Msd , OTH )

} - Hs, Toko) - Davis,x* Tfa, O )

dehne p* = - Thus , ytf O C- Q

p fcs , xD , EG) ) - Lcsixcsl , GD Z- Ctx)

HP" Tfcssx-4010) - LCS, , O )

Q Oo

⇒ Still a necessary condition .

SufficientconditlhttssumeEr: Storti → ⑦ Satisfies

I be its controlled trajectory .

⇒ A-Vet , IAD t ×VHiXHDTfCt,xHh8HD-

IfVA, Ray+ KHANEH, so

⇒ integrate from to tot ,

vet, , I Cti ) ) - Uto, Ects ) x JIT kn - - Idt so .

- -

Elkan) iIoT¥ running cost

(x-

Jeong

⇒ JEET = M£807

Generate optimal control

① Solve HSB . → V '

② OTH - afgmfx×VCtnMtD¥x"Hid1- LH, tho) y

closed loop .