reinforcement learning
TRANSCRIPT
WELCOME TO ALL
REINFORCEMENT LEARNING
NAME : S. THARABAI
COURSE: M.TECH(CSE) II YEAR PT
REG. No : 121322201011
REVIEW No: 2
WHAT IS REINFORCEMENT LEARNING?
REINFORCEMENT IS DEFINED AS CHARACTERISING THE LEARNING PROBLEM
REINFORCEMENT LEARNING WORKS ON PRIOR AND
ACCURATE MEASUREMENTSIT WORKS ON SOME
CALCULATIONS AND PLACES WHERE HUMAN CAN'T REACH. IT
IS INTANGIBLE
IT IS NOT A REAL WORLD PROBLEM BUT TRYING TO
REACH THINGS BEYOND THE WORLD!
Reinforcement learning is
learning what to do--how to map situations to actions--so as to maximize a numerical reward signal.
These two characteristics--trial-and-error search and delayed reward--are the two most important distinguishing features of
reinforcement learning.
EXAMPLES
➢ CHESS BOARD MOVEMENTS➢ ROBOTICS MOVEMENTS➢ DEPTH OF THE PETROLEUM REFINERY
AND ADAPTOR
Comparison Between Machine Learning And Reinforcement Learning.
Machine learning works classified and unclassified data sets. Reinforcement learning requires prior – accurate measurements. Example, Robotics Arms should be measured and given, how many steps to move and in what direction based on temperature adaptors.Machine learning is a real-world problem. Reinforcement learning works on some calculations, and helps in places where man can not reach. Example, Petroleum Refineries.Machine learning is tangible. Reinforcement learning is intangible. Example, Chandrayan-1 project.Machine learning works on sequence of step, reinforcement learning works on episodesepisodes
ELEMENTS OF REINFOREMENT LEARNING
➢ A POLICY➢ A REWARD FUNCTION➢ A VALUE FUNCTION➢ A MODEL ENVIRONMENT
POLICY, REWARD AND VALUE
➢ POLICY DEFINES LEARNING AGENTS'S WAY OF BEHAVING AT A GIVEN TIME.
➢ REWARD DEFINES A GOAL IN A REINFORECEMENT PROBLEM
➢ VALUE FUNCTION SPECIFIES WHAT IS GOOD IN THE LONG RUN
ENVIRONMENT
The fourth and final element of some reinforcement learning systems is a model of the environment. This is something that mimics the behavior of the environment. For example, given a state and action, the model might predict the resultant next state and next reward. Models are used for planning
An -Armed Bandit Problem
An analogy is that of a doctor choosing between experimental treatments for a series of seriously ill patients. Each play is a treatment selection, and each reward is the survival or well-being of the patient.
In our -armed bandit problem,
Each action has an expected or mean reward given that the action is selected; let us call this the value of that action. If you knew the value of each action, then it would be trivial to solve the -armed bandit problem: you would always select the action with highest value. We assume that you do not know the action values with certainty, although you may have estimates.
Monte Carlo Methods
Monte Carlo methods are ways of solving the reinforcement learning problem based on averaging sample returns. To ensure that well-defined returns are available, we define Monte Carlo methods only for episodic tasks. That is, we assume experience is divided into episodes, and that all episodes eventually terminate no matter what actions are selected. It is only upon the completion of an episode that value estimates and policies are changed. Monte Carlo methods are thus incremental in an episode-by-episode sense, but not in a step-by-step sense. The term "Monte Carlo" is often used more broadly for any estimation method whose operation involves a significant random component. Here we use it specifically for methods based on averaging complete returns.
The backup diagram for Monte Carlo estimation of
Approximate state-value functions for the blackjack policy that sticks only on 20 or 21, computed by Monte Carlo policy evaluation.
The spectrum ranging from the one-step backups of simple TD methods(step-by-step value) to the
up-until-termination backups of Monte Carlo methods(next episode reward)
Performance of -step TD methods. The performance measure shown is the root mean-squared (RMS) error between the true values of states and the values found by the learning methods, averaged over the 19 states, the first 10 trials, and 100 different
sequences of walks.
A simple maze (inset) and the average learning curves for Dyna-Q agents varying in their number of planning steps per real step. The task is to travel from S to S as
quickly as possible.
Action-Value Methods
we denote the true (actual) value of action a as, Q*(a) and the estimated value at the r th play as Qt(a) . Recall that the true value of an action is the mean reward received when that action is selected. One natural way to estimate this is by averaging the rewards actually received when the action was selected. And r be the number of plays in which Ka is the times prior to r th play.
yielding rewards, r1+r2+....rk
then its value is estimated to be,
DYNAMIC PROGRAMMING
The reinforcement learning is more useful in tracking the Non-stationary problems.
In such cases it makes sense to weight recent rewards more heavily than long-past ones. Ex. DYNAMIC PROGRAMMING.
The Agent-Environment Interface
The reinforcement learning problem is meant to be a straightforward framing of the problem of learning from interaction to achieve a goal. The learner and decision-maker is called the agent. The thing it interacts with, comprising everything outside the agent, is called the environment.
The Markov Property
In the reinforcement learning framework, the agent makes its decisions as a function of a signal from the environment called the environment's state. In particular, we formally define a property of environments and their state signals that is of particular interest, called the Markov property.
Example 1: Bioreactor Suppose reinforcement learning is being applied to determine moment-by-moment temperatures and stirring rates for a bioreactor (a large vat of nutrients and bacteria used to produce useful chemicals)
Example 2: Pick-and-Place Robot Consider using reinforcement learning to control the motion of a robot arm in a repetitive pick-and-place task. If we want to learn movements that are fast and smooth, the learning agent will have to control the motors directly and have low-latency information about the current positions and velocities of the mechanical linkages.
Sample Remote RCXLisp Code Illustrating Sensor-Based Motor Control
LISP CODING
(DEFUN full-speed-ahead (r s dir)
“This will make the rcx in stream R go at speed S
in direction DIR until touch sensor on its ‘2’ port returns 1.”
(LET ((result 0))
(set-effector-state ‘(:A :B :C) :power :off r)
;in case things are in an inconsistent state,
;turn everything off first
(set-effector-state ‘(:A :C) :speed s r)
(set-effector-state ‘(:A :C) :direction dir r)
; dir is eq to :forward, :backward, or :toggle
; no motion will occur until the
; next call to set-effector-state
(set-sensor-state 2 :type :touch :mode :boolean r)
(set-effector-state ‘(:A :C) :power :on r)
(LOOP ;this loop will repeat forever until sensor 2 returns a 1
(SETF result (sensor 2 r))
(WHEN (AND (NUMBERP result)
;needed to keep = from causing error if
;sensor function returns nil due to timeout.
(= result 1))
(RETURN)))
(set-effector-state ‘(:A :C) :power :float r))))
(DEFUN testing ()
(with-open-com-port (port :LEGO-USB-TOWER)
(with-open-rcx-stream (rcx10 port :timeout-interval 80 :rcx-unit 10)
; increase/decrease serial timeout value of 80 ms depending on
;environmental factors like ambient light.
(WHEN (alivep rcx10)
(full-speed-ahead rcx10 5 :forward)))))
Integrating Planning, Acting, and Learning
When planning is done on-line, while interacting with the environment, a number of interesting issues arise. New information gained from the interaction may change the model and thereby interact with planning. It may be desirable to customize the planning process in some way to the states or decisions currently under consideration, or expected in the near future.
Relationships among learning, planning, and acting.
Dyna-Q, a simple architecture integrating the major functions needed in an on-line planning agent. Each function appears in
Dyna-Q in a simple, almost trivial, form
The complete algorithm for Dyna-Q.
A simple maze (inset) and the average learning curves for Dyna-Q agents varying in their number of
planning steps per real step. The task is to travel from S to S as quickly as possible.
Policies found by planning and nonplanning Dyna-Q agents halfway through the second episode. The arrows
indicate the greedy action in each state; no arrow is shown for a state if all of its action values are equal. The
black square indicates the location of the agent.
Dimensions of Reinforcement Learning
The dimensions of the reinforcement learning have to do with the kind of backup used to improve the value function. The vertical dimension is whether they are sample backups (based on a sample trajectory) or full backups (based on a distribution of possible trajectories). Full backups of course require a model, whereas sample backups can be done either with or without a model (another dimension of variation).
A slice of the space of reinforcement learning methods.
The horizontal dimension corresponds to the depth of backups, that is, to the degree of bootstrapping. At three of the four corners of the space are the three primary methods for estimating values: DP, TD, and Monte Carlo.
A third important dimension is that of function approximation. Function approximation can be viewed as an orthogonal spectrum of possibilities ranging from tabular methods at one extreme through state aggregation, a variety of linear methods, and then a diverse set of nonlinear methods
In addition to the four dimensions just discussed, we have identified a number of others throughout the book:
Definition of return
Is the task episodic or continuing, discounted or undiscounted?
Action values vs. state values vs. afterstate values
What kind of values should be estimated? If only state values are estimated, then either a model or a separate policy (as in actor-critic methods) is required for action selection.
Action selection/exploration
How are actions selected to ensure a suitable trade-off between exploration and exploitation? We have considered only the simplest ways to do this: -greedy and softmax action selection, and optimistic initialization of values.
Synchronous vs. asynchronous
Are the backups for all states performed simultaneously or one by one in some order?
Replacing vs. accumulating traces
If eligibility traces are used, which kind is most appropriate?
Real vs. simulated
Should one backup real experience or simulated experience? If both, how much of each?
Location of backups
What states or state-action pairs should be backed up? Modelfree methods can choose only among the states and state-action pairs actually encountered, but model-based methods can choose arbitrarily. There are many potent possibilities here.
Timing of backups
Should backups be done as part of selecting actions, or only afterward?
Memory for backups
How long should backed-up values be retained? Should they be retained permanently, or only while computing an action selection, as in heuristic search?
Elevator Dispatching
Waiting for an elevator is a situation :
We press a button and then wait for an elevator to arrive traveling in the right direction.
We may have to wait a long time if there are too many passengers or not enough elevators.
Just how long we wait depends on the dispatching strategy the elevators use to decide where to go.
For example, if passengers on several floors have requested pickups, which should be served first? If there are no pickup requests, how should the elevators distribute themselves to await the next request?
Elevator dispatching is a good example of a stochastic optimal control problem of economic importance that is too large to solve by classical techniques such as dynamic programming.
Four elevators in a ten-story building
Job-Shop Scheduling
Many jobs in industry and elsewhere require completing a collection of tasks while satisfying temporal and resource constraints. Temporal constraints say that some tasks have to be finished before others can be started; resource constraints say that two tasks requiring the same resource cannot be done simultaneously (e.g., the same machine cannot do two tasks at once). The objective is to create a schedule specifying when each task is to begin and what resources it will use that satisfies all the constraints while taking as little overall time as possible. This is the job-shop scheduling problem.
JOB SHOP SCHDULING APPLICATION IN NASA
The NASA space shuttle payload processing problem (SSPP), which requires scheduling the tasks required for installation and testing of shuttle cargo bay payloads. An SSPP typically requires scheduling for two to six shuttle missions, each requiring between 34 and 164 tasks. An example of a task is MISSION-SEQUENCE-TEST.
CONCLUSION
All of the reinforcement learning methods we have explored in this ppt have three key ideas in common.
First, the objective of all of them is the estimation of value functions.
Second, all operate by backing up values along actual or possible state trajectories.
Third, all follow the general strategy of generalized policy iteration (GPI), meaning that they maintain an approximate value function and an approximate policy, and they continually try to improve each on the basis of the other.
We suggest that value functions, backups, and GPI are powerful organizing principles potentially relevant to any model of intelligence.
THANK YOU!