hierarchical rl (dai).ppt
TRANSCRIPT
![Page 1: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/1.jpg)
Hierarchical Hierarchical Reinforcement Reinforcement
LearningLearning
Hierarchical Hierarchical Reinforcement Reinforcement
LearningLearningAmir massoud FarahmandAmir massoud [email protected]@SoloGen.net
![Page 2: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/2.jpg)
Markov Decision Problems
• Markov Process: Formulating a wide range of dynamical systems
• Finding an optimal solution of an objective function
• [Stochastic] Dynamics Programming• Planning: Known environment• Learning: Unknown environment
![Page 3: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/3.jpg)
MDP
![Page 4: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/4.jpg)
Reinforcement Learning (1)
• Very important Machine Learning method
• An approximate online solution of MDP– Monte Carlo method– Stochastic Approximation– [Function Approximation]
![Page 5: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/5.jpg)
Reinforcement Learning (2)
• Q-Learning and SARSA are among the most important solution of RL
![Page 6: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/6.jpg)
Curses of DP• Curse of Modeling
– RL solves this problem
• Curse of Dimensionality– Approximating Value function– Hierarchical methods
![Page 7: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/7.jpg)
Hierarchical RL (1)• Use some kind of hierarchy in order to …
– Learn faster– Need less values to be updated (smaller
storage dimension)– Incorporate a priori knowledge by designer– Increase reusability– Have a more meaningful structure than a
mere Q-table
![Page 8: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/8.jpg)
Hierarchical RL (2)• Is there any unified meaning of
hierarchy? NO!• Different methods:
– Temporal abstraction– State abstraction– Behavioral decomposition– …
![Page 9: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/9.jpg)
Hierarchical RL (3)• Feudal Q-Learning [Dayan, Hinton]• Options [Sutton, Precup, Singh]• MaxQ [Dietterich] • HAM [Russell, Parr, Andre]• HexQ [Hengst]• Weakly-Coupled MDP [Bernstein, Dean & Lin,
…]• Structure Learning in SSA [Farahmand, Nili]• …
![Page 10: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/10.jpg)
Feudal Q-Learning• Divide each task to a few smaller sub-tasks• State abstraction method• Different layers of managers• Each manager gets orders from its super-
manager and orders to its sub-managers
Super-Manager
Manager 1 Manager 2
Sub-Manager 1 Sub-Manager 2
![Page 11: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/11.jpg)
Feudal Q-Learning• Principles of Feudal Q-Learning
– Reward Hiding: Managers must reward sub-managers for doing their bidding whether or not this satisfies the commands of the super-managers. Sub-managers should just learn to obey their managers and leave it up to them to determine what it is best to do at the next level up.
– Information Hiding: Managers only need to know the state of the system at the granularity of their own choices of tasks. Indeed, allowing some decision making to take place at a coarser grain is one of the main goals of the hierarchical decomposition. Information is hidden both downwards - sub-managers do not know the task the super-manager has set the manager - and upwards -a super-manager does not know what choices its manager has made to satisfy its command.
![Page 12: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/12.jpg)
Feudal Q-Learning
![Page 13: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/13.jpg)
Feudal Q-Learning
![Page 14: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/14.jpg)
Options: Introduction
• People do decision making at different time scales– Traveling example
• It is desirable to have a method to support this temporally-extended actions over different time scales
![Page 15: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/15.jpg)
Options: Concept• Macro-actions• Temporal abstraction method of Hierarchical RL• Options are temporally extended actions which
each of them is consisted of a set of primitive actions
• Example:– Primitive actions: walking NSWE– Options: go to {door, cornet, table, straight}
• Options can be Open-loop or Closed-loop
• Semi-Markov Decision Process Theory [Puterman]
![Page 16: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/16.jpg)
Options: Formal Definitions
![Page 17: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/17.jpg)
Options: Rise of SMDP!• Theorem: MDP + Options = SMDP
![Page 18: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/18.jpg)
Options: Value function
![Page 19: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/19.jpg)
Options: Bellman-like optimality
condition
![Page 20: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/20.jpg)
Options: A simple example
![Page 21: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/21.jpg)
Options: A simple example
![Page 22: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/22.jpg)
Options: A simple example
![Page 23: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/23.jpg)
Interrupting Options• Option’s policy is followed until it
terminates.• It is somehow unnecessary
condition– You may change your decision in the middle
of execution of your previous decision.
• Interruption Theorem: Yes! It is better!
![Page 24: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/24.jpg)
Interrupting Options:An example
![Page 25: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/25.jpg)
Options: Other issues
• Intra-option {model, value} learning
• Learning each options– Defining sub-goal reward function
![Page 26: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/26.jpg)
MaxQ• MaxQ Value Function
Decomposition• Somehow related to Feudal Q-
Learning• Decomposing Value function in a
hierarchical structure
![Page 27: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/27.jpg)
MaxQ
![Page 28: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/28.jpg)
MaxQ: Value decomposition
![Page 29: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/29.jpg)
MaxQ: Existence theorem
• Recursive optimal policy.• There may be many recursive optimal policies with
different value function.• Recursive optimal policies are not an optimal policy.• If H is stationary macro hierarchy for MDP M, then all
recursively optimal policies w.r.t. have the same value.
![Page 30: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/30.jpg)
MaxQ: Learning
• Theorem: If M is MDP, H is stationary macro, GLIE (Greedy in the Limit with Infinite Exploration) policy, common convergence conditions (bounded V and C, sum of alpha is …), then with Prob. 1, algorithm MaxQ-0 will converge!
![Page 31: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/31.jpg)
MaxQ• Faster learning: all states updating
– Similar to “all-goal-updating” of Kaelbling
![Page 32: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/32.jpg)
MaxQ
![Page 33: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/33.jpg)
MaxQ: State abstraction
• Advantageous– Memory reduction– Needed exploration will be reduced– Increase reusability as it is not
dependent on its higher parents
• Is it possible?!
![Page 34: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/34.jpg)
MaxQ: State abstraction• Exact preservation of value function• Approximate preservation
![Page 35: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/35.jpg)
MaxQ: State abstraction• Does it converge?
– It has not proved formally yet.
• What can we do if we want to use an abstraction that violates theorem 3?– Reward function decomposition
• Design a reward function that reinforces those responsible parts of the architecture.
![Page 36: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/36.jpg)
MaxQ: Other issues
• Undesired Terminal states• Non-hierarchical execution (polling
execution)– Better performance– Computational intensive
![Page 37: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/37.jpg)
Learning in Subsumption Architecture
• Structure learning– How should behaviors arranged in
the architecture?
• Behavior learning– How should a single behavior act?
• Structure/Behavior learning
![Page 38: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/38.jpg)
SSA: Purely Parallel Case
build maps
explore
avoid obstacles
locomote
manipulatethe world
sensors
![Page 39: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/39.jpg)
SSA: Structure learning issues
• How should we represent structure?– Sufficient (problem space can be
covered)– Tractable (small hypothesis space)– Well-defined credit assignment
• How should we assign credits to architecture?
![Page 40: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/40.jpg)
SSA: Structure learning issues
• Purely parallel structure– Is it the most plausible choice
(regarding SSA-BBS assumptions)?
• Some different representations– Beh. learning– Beh/Layer learning– Order learning
![Page 41: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/41.jpg)
SSA: Behavior learning issues
• Reinforcement signal decomposition: each Beh. has its own reward function
• Reinforcement signal design: How should we transform our desires into reward function?– Reward Shaping– Emotional Learning– …?
• Hierarchical Credit Assignment
![Page 42: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/42.jpg)
SSA: Structure Learning example
• Suppose we have correct behaviors and want to arrange them in an architecture in order to maximize a specific behavior
• Subjective evaluation: We want to lift an object to a specific height while its slope does not become too high.
• Objective evaluation: How should we design it?!
![Page 43: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/43.jpg)
SSA: Structure Learning example
![Page 44: Hierarchical RL (DAI).ppt](https://reader035.vdocuments.mx/reader035/viewer/2022062220/55617da2d8b42a72118b54f3/html5/thumbnails/44.jpg)
SSA: Structure Learning example