deep reinforcement learning for robotics using...

PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen, Steven Bohez, Elias De Coninck, Sam Leroux, Pieter Van Molle Bert VanKeirsbilck, Pieter Simoens, Bart Dhoedt [email protected]

Upload: others

Post on 29-Mar-2020

27 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

PUBLIC

Deep Reinforcement Learning for Robotics Using DIANNETim Verbelen, Steven Bohez, Elias De Coninck, Sam Leroux, Pieter Van Molle

Bert VanKeirsbilck, Pieter Simoens, Bart [email protected]

Page 2: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

How can we build robots that are able to execute complex tasks without programming them explicitly ?

Page 3: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Kuka Youbot

5 axis armLength: 66 cm

Gripper

Omnidirectional wheelsMax speed: 0.8 m/s

Battery operated

Embedded PC

Page 4: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Hokuyo Laser rangefinder

Kuka soft gripper

Reinforcement learning

EnvironmentAgent

Reinforcement learning

Environment

Observation

Agent

Reinforcement learning

Environment

Action

Agent

Reinforcement learning

EnvironmentReward

Agent

Deep Reinforcement learning

● The actor needs to process high dimensional observations to determine the next action.● Our favorite processing block: deep neural networks

Observation Action

Page 10: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

How can we train without destroying our robot ?

Page 11: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

https://docs.google.com/file/d/0B6aNwc1Zdoa9Z3p2QzVabHhpcVk/preview

Page 12: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

V-REP simulator

Page 13: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Multiple simulator instances gathering experience on CPU

Page 14: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Multiple simulator instances gathering experience on CPU

GPU system training the model

Page 15: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

How can we evaluate our models on the robot ?

Page 16: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Brain transplantation !

Page 17: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

How can we connect the different components ?

Dianne

• Modular software framework for designing, training and evaluating neural networks.

• Distributed training and evaluation

• Java based

• Easy integration (service based architecture)

• GUI

• Open source (AGPL 3)

Page 20: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Deployed agent

Page 21: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Experience Pool

Deployed agent

Page 22: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Experience Pool

Repository

TrainingDeployed

agentDeployed

agent

Page 23: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Experience Pool

Repository

TrainingDeployed

agentDeployed

agent

Page 24: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Deep Reinforcement learning algorithms

Page 25: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

DQN

“Playing Atari with Deep Reinforcement Learning” (Mnih et al, 2013)

Expected future return for each possible action

raw laser scanner measurements

(512 values)

Q Values

Page 26: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

https://docs.google.com/file/d/0B6aNwc1Zdoa9anl4NHRLNDd3UTg/preview

Page 27: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

DDPG

Continuous control with Deep Reinforcement Learning (Lillicrap, et al. 2015)

Actor network

Critic network

raw laser scanner measurements

(512 values)

Continuous action

Expected future return

Page 28: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

https://docs.google.com/file/d/0B6aNwc1Zdoa9UkFTRXIwYTZjM1E/preview

Page 29: Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Visit dianne.intec.ugent.be for more information

PUBLIC

Abstraction layer with ROS

Base

Sensor

Arm

Introduction to Deep Reinforcement Learning Model-free Methods · •People started to apply deep learning into robotics Deep RL for Robotics Levine, Sergey, and Chelsea Finn. "Deep

CSC2621 Imitation Learning for Roboticsflorian/courses/imitation_learning/lectures/... · CSC2621 Imitation Learning for Robotics Florian Shkurti Week 10: Inverse Reinforcement Learning

Reinforcement Learning in Robotics

Reinforcement Learning Methods for Military Applications Malcolm Strens Centre for Robotics and Machine Vision Future Systems Technology Division Defence

Model-Based Reinforcement Learning in Robotics...Model-Free Example Rainbow Algorithm [2] 1𝑓= 1 60 𝑠 200.000.000𝑓=…=925,925ℎ=38,58 Not feasible in robotics! MODEL-BASED

Deep Reinforcement Learning for Robotics: Frontiers and Beyondwnzhang.net/teaching/past-courses/cs420-2018/slides/guest-shixiang-gu.pdf · 01 Deep Reinforcement Learning for Robotics:

Introduction to Robotics Reinforcement Learning in Roboticsipvs.informatik.uni-stuttgart.de/mlr/marc/teaching/14-Robotics/10... · Five approaches to RL ... Teaching and learning

WWV2015: Branko schuurman_geert verbelen - dhl

DIANNE - A distributed deep learning framework on OSGi - Tim Verbelen

Extending the OpenAI Gym for robotics: a toolkit for ...erlerobotics.com/whitepaper/robot_gym.pdfExtending the OpenAI Gym for robotics: a toolkit for reinforcement learning using ROS

CSE 591 (Fall 2016): Human-aware Robotics T Th 9:00AM – 10 ...yzhan442/teaching/CSE... · Potentialtopics:human>awareplanning,!Markov!decision!process!and!reinforcement! learning,and!inverse!reinforcement!learning.!

Deep Reinforcement Learning for Robotics · Deep reinforcement learning . Robotics . Percepts . Hand-engineered state-estimation . Many-layer neural network with many parameters to

Cite this article as: Verbelen T, Godinas L, Maleux G

Deep Reinforcement Learning for Robotics

Reinforcement learning in robotics - Columbia Universityallen/S19/rl_robotics_survey.pdfrobotics, Figure 2 provides a schematic illustration of two axes of problem variability: the

Carbon Fibre and Aluminium Foam Composites - Robotics UWArobotics.ee.uwa.edu.au/theses/2011-REV-SAE-CarbonFibre-Craven.pdf · The short fibre reinforcement technique was validated

Reinforcement Learning in Robotics: A Surveycse.unl.edu/~lksoh/Classes/CSCE475H_Spring16/seminars/...Reinforcement Learning Provides feedback in terms of a scalar objective function

Profil: AI och Maskininlärning - Linköping University€¦ · Learning •Deep learning •Bayesian learning •Reinforcement learning Robotics / Cyber-Physical Systems Decision

Getting to the Next Level with Eclipse Concierge - Jan Rellermeyer + Tim Verbelen + Jochen Hiller

Getting a kick out of humanoid robotics - UvA · Getting a kick out of humanoid robotics Using reinforcement learning to shape a soccer kick Christiaan W. Meijer 0251968 Master thesis

Fast and robust learning by reinforcement signals ...sro.sussex.ac.uk/7691/1/HuertaNowotnyNECO.2009.2E03-08-733.pdf · Centre for Computational Neuroscience and Robotics, Department

Improved Deep Reinforcement Learning for Robotics Through ... · Improved Deep Reinforcement Learning for Robotics Through Distribution-based Experience Retention Tim de Bruin 1Jens

OSGi for IoT: the good, the bad and the ugly - Tim Verbelen

Reinforcement Learning in Robotics: Applications and Real-World Challenges

Exploration in Reinforcement Learning Jeremy Wyatt Intelligent Robotics Lab School of Computer Science University of Birmingham, UK [email protected]

lecture 5 - reinforcement learningholly/teaching/4500/spring2018/...Reinforcement Learning COMP 4500 Mobile Robotics I Spring 2018 Prof. Yanco Many of the slides in this presentation

CSE-571 Probabilistic Robotics - University of WashingtonCSE-571 Probabilistic Robotics Reinforcement Learning for Active Sensing Manipulator Control Reinforcement Learning • Same

Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

IEEE ROBOTICS AND AUTOMATION ... - yuxiangsun.github.io · IEEE ROBOTICS AND AUTOMATION LETTERS, VOL. 5, NO. 2, APRIL 2020 1247 High-Speed Autonomous Drifting With Deep Reinforcement

Hierarchical Solutions in Reinforcement Learning using Graph Algorithms Project Presentation Control and Robotics Laboratory Electrical Engineering Faculty

PPO Reinforcement Learning for Dexterous In-Hand Manipulation · Main challenges for using Reinforcement Learning in Robotics –Differences between reality and simulation need to

Robert Jan Verbelen and the United States Government (Report, 1988)

Reinforcement Learning for Robotics - Columbia …RL Class of Methods Value Iteration methods (Q-Learning, SARSA) DQN: Playing Atari with Deep Reinforcement Learning (Mnih etal 2015)

Overview on Reinforcement Learning for Robotics · Reinforcement Learning(RL) offers robotics tasks a framework and set of tools for the design of sophisticated and hard-to-engineer

Reinforcement Learning in Robotics: ASurvey