Upload File

introduction to reinforcement learning and fintech case study

30
Introduction to RL and Liquid Labs Case Study Jorge Davila-Chacon CSMLS Meetup - October 15, 2015 - Hamburg, Germany Data Scientist Research Associate Liquid Labs University of Hamburg

Upload: jorge-davila-chacon

Post on 06-Apr-2017

378 views

Category:

Data & Analytics


3 download

Report

TRANSCRIPT

Page 1: Introduction to Reinforcement Learning and FinTech Case Study

Introduction to RL andLiquid Labs Case Study

Jorge Davila-Chacon

CSMLS Meetup - October 15, 2015 - Hamburg, Germany

Data Scientist Research Associate Liquid Labs University of Hamburg

Page 2: Introduction to Reinforcement Learning and FinTech Case Study

20 mins

Introduction to RL

10 mins

Overview of Architecture

15 mins

Implementation with iPython Notebook

10 mins

Analysis of Results

5 mins

Wrap Up

2

Page 3: Introduction to Reinforcement Learning and FinTech Case Study

Who?

● Curious people!

● Math background?

● Academia, industry?

● Semantics > Syntactics

3

Page 4: Introduction to Reinforcement Learning and FinTech Case Study

RL

Introduction

4

Page 5: Introduction to Reinforcement Learning and FinTech Case Study

What?● Learn by experience

● Learn with rewards

● Learn continuously

RL

5

Page 6: Introduction to Reinforcement Learning and FinTech Case Study

RL Cycle

Figure 1. From “Introduction to Reinforcement Learning”, Sutton and Barto (1998).

https://webdocs.cs.ualberta.ca/~sutton/book/ebook/the-book.html6

Page 7: Introduction to Reinforcement Learning and FinTech Case Study

How?● Markov Decision Process

● Bellman’s equation

● Policy iteration

7

Page 8: Introduction to Reinforcement Learning and FinTech Case Study

Markov Decision Process

8

Page 9: Introduction to Reinforcement Learning and FinTech Case Study

Bellman Equation

9

Page 10: Introduction to Reinforcement Learning and FinTech Case Study

Bellman Equation

10

Page 11: Introduction to Reinforcement Learning and FinTech Case Study

Policy Iteration

11

Page 12: Introduction to Reinforcement Learning and FinTech Case Study

Policy Iteration

● SARSA

● Q-Learning

12

Page 13: Introduction to Reinforcement Learning and FinTech Case Study

Policy Iteration

● SARSA

● Q-Learning

13

● Exploration

● Exploitation

Page 14: Introduction to Reinforcement Learning and FinTech Case Study

Why?

With love…

● From: Monte Carlo

● To: TD Learning

● Eligibility Traces

14

Page 15: Introduction to Reinforcement Learning and FinTech Case Study

From MC to TD

15

Page 16: Introduction to Reinforcement Learning and FinTech Case Study

Eligibility Traces

● SARSA

● Q-Learning

16

Page 17: Introduction to Reinforcement Learning and FinTech Case Study

Eligibility Traces

17

Page 18: Introduction to Reinforcement Learning and FinTech Case Study

18

RL

Case Study:Simulation before deployment

Page 19: Introduction to Reinforcement Learning and FinTech Case Study

iPython Notebook

● Colorado example

● Architecture

● Implementation

19

Page 20: Introduction to Reinforcement Learning and FinTech Case Study

Graphs!

● Long run results

● Short run results

● Future Work

20

Page 21: Introduction to Reinforcement Learning and FinTech Case Study

Long Run - Without Lambda

21

Page 22: Introduction to Reinforcement Learning and FinTech Case Study

Long Run - With Lambda

22

Page 23: Introduction to Reinforcement Learning and FinTech Case Study

Long Run - With Lambda

23

Page 24: Introduction to Reinforcement Learning and FinTech Case Study

Long Run - With Lambda

24

Page 25: Introduction to Reinforcement Learning and FinTech Case Study

Short Run - With Lambda

25

Page 26: Introduction to Reinforcement Learning and FinTech Case Study

Short Run - With Lambda

26

Page 27: Introduction to Reinforcement Learning and FinTech Case Study

Short Run - With Lambda

27

Page 28: Introduction to Reinforcement Learning and FinTech Case Study

Future Work - Non-Monotonic Epsilon

28

Page 29: Introduction to Reinforcement Learning and FinTech Case Study

Wrap Up

● Learn from scratch

● Adaptive learning

● On-line learning

● Research possibilities!

29

Page 30: Introduction to Reinforcement Learning and FinTech Case Study

Thank you for coming!

Jorge Davila-Chacon

[email protected]

or LinkedIn

30

mailto:[email protected]
mailto:[email protected]

Reinforcement Learning - Multi-Agent Reinforcement

Deep Learning for Reinforcement Learning in Pacman · Deep Learning for Reinforcement Learning in Pacman Deep Learning für Reinforcement Learning in Pacman Vorgelegte Bachelor-Thesis

Bayesian Reinforcement Learning - mlg.eng.cam.ac.ukmlg.eng.cam.ac.uk/rowan/files/BayesianReinforcementLearning.pdf · Introduction Bayesian Reinforcement Learning Bayesian Reinforcement

Cooperative Inverse Reinforcement Learning...Cooperative Inverse Reinforcement Learning Dylan Hadfield-Menell CS237: Reinforcement Learning May 31, 2017

Reinforcement Learning Chapter 13 What is Reinforcement Learning? Q-Learning Examples 1

Generalization in Reinforcement Learning: Successful ...papers.nips.cc/paper/1109-generalization-in-reinforcement-learning... · Generalization in Reinforcement Learning: Successful

Reinforcement Learning

Reinforcement Learning Das Reinforcement Learning-Problem Alexander Schmid

Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning

Reinforcement Learning - uni-freiburg.degki.informatik.uni-freiburg.de/.../recordings/reinforcement.pdf · Reinforcement Learning 3 What is Reinforcement Learning? Learning from interaction

Multi-Objective Reinforcement Learning using Sets of Pareto … · 2020. 10. 19. · learning and multi-objective reinforcement learning. 2.1 Reinforcement Learning A reinforcement

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Reinforcement Learning Lecture Inverse Reinforcement Learningipvs.informatik.uni-stuttgart.de/mlr/wp-content/uploads/2017/07/09... · Reinforcement Learning Inverse Reinforcement

Eick: Reinforcement Learning. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning

Reinforcement Learning or Active Inference?karl/Reinforcement Learning or Active... · Reinforcement Learning or Active Inference? ... From the point of view of reinforcement learning

Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

Deep Learning for Reinforcement Learning in · PDF fileDeep Learning for Reinforcement Learning in ... Deep Learning for Reinforcement Learning in Pacman Deep Learning für ... Während

Reinforcement Learning and Deep Reinforcement Learningcse.ucdenver.edu/.../Class-22-Reinforcement-learning-DL.pdf · 2018. 11. 28. · Outlines 1 Principles of Reinforcement Learning

Reinforcement Learning - 4. Model-free reinforcement Learning

Reinforcement Learning & Apprenticeship Learning

Inverse Reinforcement Learning CS885 Reinforcement

Inverse Reinforcement Learning - Peoplecbfinn/_files/bootcamp_inverserl.pdf · Apprenticeship Learning via Inverse Reinforcement Learning. Good introduction to inverse reinforcement

Reinforcement Learning: Learning algorithms