e xploring m arkov d ecision p rocess v iolations in r einforcement l earning jordan fryer –...
Post on 18-Dec-2015
216 Views
Preview:
TRANSCRIPT
1
EXPLORING MARKOV DECISION PROCESS VIOLATIONS IN REINFORCEMENT LEARNINGJordan Fryer – University of Portland
Working with Peter Heeman
2
OUTLINE
Background: Reinforcement Learning (RL) RL and symbolic reasoning to learn system
dialogue policy Background: Markov Decision Processes
(MDP) The Problem
Attempting to find absolute convergence Simplification process Evaluation tools
Discussion
3
BACKGROUND: REINFORCEMENT LEARNING
Inputs States
How the agent represents the environment at a certain time Actions
How the agent interacts with the environment Cost Function
A probabilistic mapping of a state-action pair to a value Most of the costs may be assigned at terminal state
Simulated User So can system can try out different dialogue behaviors
Outputs: Optimal Policy A mapping of a state to an action
How it learns Iteratively: evaluate current policy and explore
alternatives, and then update policy
4
BACKGROUND: REINFORCEMENT LEARNING
Keep track of Q score for each state-action pair Cost to get to the end from state following that
action For each dialogue simulation, take final cost
and propagate it back over the state-action pairs in the run
S1
S2
S3
S4
S5
Utt: 1 Utt: 1 Utt: 1 Utt: 1 SQ: 10Total: 14
a1
a2
a3
a4
Q: 11 Q: 12 Q: 13 Q: 14
5
BACKGROUND: MARKOV DECISION PROCESSES
RL guaranteed to converge for Markov Decision Processes
Only use current state to decide what action to do next
System + User + Environment must satisfy: Pr {st+1 = s’ | st,at, st-1,at-1, …, s0,a0} = Pr {st+1 = s’ | st,at}
Pr {rt+1 = r | st,at, st-1,at-1, …, s0,a0} = Pr {rt+1 = r| st,at}
How detailed should states be? Too detailed, becomes brute force and explodes state
space Too vague, violates the MDP assumptions
RL learns solution very fast due to “merging” of states
6
WHY RL FOR DIALOGUE?
There is a delayed cost to dialogue. The correctness of a dialogue is not really known
until the end of the dialogue and the task has been performed
Modeling after humans isn’t always correct Many things a computer can do that a human
can’t and many things a human can do that a computer can not
Hard to handcraft a policy
7
THE PROBLEM: FINDING ABSOLUTE CONVERGENCE RL is guaranteed to converge for MDP in the
limit
How do we know if we have an MDP violation?
How long do we have to wait for convergence?
How do we measure convergence?
We will use QLearning with ε-greedy (20%)
8
DOMAIN
Ran on toy Car Domain (from CS550 course) Database of 2000 cars (differ in color, year,
model …) User has one of the 2000 cars in mind System asks questions and reports list of cars
that match the user’s car State:
11 Questions: Boolean (asked or not) carBucket: Number (bucketized number of cars) Done: Boolean (reported cars or not)
Cost Function: 1 cost per utterance, 5 cost per extra reported car
9
FULL VERSION OF PROBLEM
Every 1000 epochs (100 dialogue runs) test the current policy
Could not get it to converge
10
SIMPLIFIED VERSION OF PROBLEM
Let’s simplify problem (common CS approach) Removed some attributes, reduced buckets from 4
to 2, force output and exit when only one car left Reduced number of state-action pairs
Were able to use exact user distribution for testing
Tool to examine the degree of convergence Compare results from multiple policies (existing
tool) Keep track of the minimum testing score seen
while training a policy Calculated the percentage of test sessions that
are at the minimum testing score
11
E # AVE MIN C % SA 19000 20 7.95082 7.95082 0.5 0.989 216 20000 20 7.95082 7.95082 0.5 0.990 216 21000 20 7.95082 7.95082 0.5 0.990 216 22000 20 7.95082 7.95082 0.5 0.990 216 23000 20 7.95082 7.95082 0.5 0.991 216 24000 20 7.95082 7.95082 0.5 0.991 216 25000 20 7.95082 7.95082 0.5 0.991 216 26000 20 7.95082 7.95082 0.5 0.991 216 27000 20 7.95082 7.95082 0.5 0.992 216 28000 20 7.95082 7.95082 0.5 0.992 216 29000 20 7.95082 7.95082 0.5 0.992 216 30000 20 7.95082 7.95082 0.5 0.992 216 31000 20 7.95082 7.95082 0.5 0.992 216 32000 20 7.95082 7.95082 0.5 0.992 216 33000 20 7.95082 7.95082 0.5 0.993 216 34000 20 7.95082 7.95082 0.5 0.993 216 35000 20 7.95082 7.95082 0.5 0.993 216 36000 20 7.95082 7.95082 0.5 0.993 216 37000 20 7.95082 7.95082 0.3 0.993 216 38000 20 7.95082 7.95082 0.1 0.994 216 39000 20 7.95082 7.95082 0.1 0.997 216 40000 20 7.95082 7.95082 0.0 0.999 216 41000 20 7.95082 7.95082 0.0 0.999 216 42000 20 7.95082 7.95082 0.0 0.999 216 43000 20 7.95082 7.95082 0.0 1.000 216 44000 20 7.95082 7.95082 0.0 1.000 216 45000 20 7.95082 7.95082 0.0 1.000 216 46000 20 7.95082 7.95082 0.0 1.000 216 47000 20 7.95082 7.95082 0.0 1.000 216 48000 20 7.95082 7.95082 0.0 1.000 216 49000 20 7.95082 7.95082 0.0 1.000 216 50000 20 7.95082 7.95082 0.0 1.000 216
E # AVE MIN C % SA 1 20 7.96066 7.95082 0.1 1.000 182 2 20 7.95738 7.95082 0.3 0.975 187 5 20 7.95656 7.95082 0.4 0.933 193 10 20 7.95328 7.95082 0.5 0.925 200 25 20 7.95082 7.95082 0.5 0.940 208 50 20 7.95082 7.95082 0.5 0.950 212 100 20 7.95164 7.95082 0.5 0.950 213 200 20 7.95082 7.95082 0.5 0.956 214 300 20 7.95082 7.95082 0.5 0.961 214 400 20 7.95082 7.95082 0.5 0.965 214 500 20 7.95082 7.95082 0.5 0.968 216 700 20 7.95082 7.95082 0.5 0.971 216 1000 20 7.95082 7.95082 0.5 0.973 216 1500 20 7.95082 7.95082 0.5 0.975 216 2000 20 7.95082 7.95082 0.5 0.977 216 2500 20 7.95082 7.95082 0.5 0.978 216 3000 20 7.95082 7.95082 0.5 0.979 216 4000 20 7.95082 7.95082 0.5 0.980 216 5000 20 7.95082 7.95082 0.5 0.981 216 6000 20 7.95082 7.95082 0.5 0.982 216 7000 20 7.95082 7.95082 0.5 0.983 216 8000 20 7.95082 7.95082 0.5 0.984 216 9000 20 7.95082 7.95082 0.5 0.985 21610000 20 7.95082 7.95082 0.5 0.985 21611000 20 7.95082 7.95082 0.5 0.986 21612000 20 7.95082 7.95082 0.5 0.987 21613000 20 7.95082 7.95082 0.5 0.987 21614000 20 7.95082 7.95082 0.5 0.987 21615000 20 7.95082 7.95082 0.5 0.988 21616000 20 7.95082 7.95082 0.5 0.988 21617000 20 7.95082 7.95082 0.5 0.989 21618000 20 7.95082 7.95082 0.5 0.989 216
12
E # AVE MIN C % SA
1 20 7.96066 7.95082 0.1 1.000 182
2 20 7.95738 7.95082 0.3 0.975 187
500 20 7.95082 7.95082 0.5 0.968 216
43000 20 7.95082 7.95082 0.0 1.000 216
13
SIMPLIFIED VERSION OF PROBLEM
Bugs found: Would converge and then go out of convergence
Alpha rounding errors Would find a minimum score within first 10
epochs that it could never find again States not seen yet in training, but seen in testing,
must choose same action during the test session
Got convergence Convergence achieved when all SA pairs
explored
14
A BIT MORE COMPLEXITY
Make domain more complex: Added back all attributes, 2 buckets, no exit
constraint Absolute convergence before all SA pairs
seen E # AVE MIN C % SA 125000 7 6.82600 6.82600 0.1 0.997 19380 126000 7 6.82600 6.82600 0.1 0.997 19390 127000 7 6.82600 6.82600 0.1 0.997 19398 128000 7 6.82600 6.82600 0.1 0.997 19401 129000 7 6.82600 6.82600 0.1 0.997 19404 130000 7 6.82600 6.82600 0.1 0.997 19415 131000 7 6.82600 6.82600 0.1 0.997 19428 132000 7 6.82600 6.82600 0.1 0.997 19434 133000 7 6.82600 6.82600 0.1 1.000 19443 134000 7 6.82600 6.82600 0.1 1.000 19450 135000 7 6.82600 6.82600 0.1 1.000 19457 136000 7 6.82600 6.82600 0.1 1.000 19463 137000 7 6.82600 6.82600 0.0 1.000 19468 138000 7 6.82600 6.82600 0.0 1.000 19475 139000 7 6.82600 6.82600 0.0 1.000 19481 140000 7 6.82600 6.82600 0.0 1.000 19492 141000 7 6.82600 6.82600 0.0 1.000 19499
E # AVE MIN C % SA…
1000000 7 6.82600 6.82600 0.0 1.000 214451001000 7 6.82600 6.82600 0.0 1.000 214451002000 7 6.82600 6.82600 0.0 1.000 214461003000 7 6.82600 6.82600 0.0 1.000 214481004000 7 6.82600 6.82600 0.0 1.000 214491005000 7 6.82600 6.82600 0.0 1.000 214491006000 7 6.82600 6.82600 0.0 1.000 214501007000 7 6.82600 6.82600 0.0 1.000 214501008000 7 6.82600 6.82600 0.0 1.000 214511009000 7 6.82600 6.82600 0.0 1.000 214511010000 7 6.82600 6.82600 0.0 1.000 214541011000 7 6.82600 6.82600 0.0 1.000 214551012000 7 6.82600 6.82600 0.0 1.000 214551013000 7 6.82600 6.82600 0.0 1.000 214571014000 7 6.82600 6.82600 0.0 1.000 214571015000 7 6.82600 6.82600 0.0 1.000 21459
…
15
QTRAIN AND QTEST
Qtrain: The Q values of an SA pair, used by RL in training by following the policy and exploring Should converge to Q* (values for the optimal
policy), but never know what Q* is Our group has also been using Qtest
The Q values of an SA pair, determined through testing the optimal policy
Qtest and Qtrain should converge to Q* and so should converge to same value
16
QTRAIN AND QTEST
Helps to show absolute convergence
17
NOW THE FULL VERSION
Moved up to 4 buckets, no absolute convergence
Noticed difference in Qtest and Qtrain Can further analyze Qtest,
Different states “merge” If MDP, path should not matter
18
VISUALIZING MDP VIOLATION
CYM5
CYM15
CDYM1
CDYM5
CDYM15
Action: AskDoors
4.255
2.0
2.0
6.333
5.0
2.0
6.333
4.4425
19
MDP VIOLATION
Why is there an MDP violation? CarBuckets: caused states to be treated as equal
when clearly they are not. How to remove MDP violation?
Keep a more accurate history Not always possible: State space explodes
Just keeping track of the order in which questions are asked leads to ~40 million states
Barto & Sutton admit that most problems are not perfect MDPs but that RL can deal with it
20
DISCUSSION
Does having a MDP violation hurt you? Despite non-convergence the 4 buckets did
better than the 2 buckets. 6.8091 vs 6.8260
RL can deal with some MDP violation Car gas mileage analogy
21
DISCUSSION
Does this mean we don’t care about MDP violation? Rueckert (REU last year) removed an MDP
violation did not increase the state space dramatically improved the policy learned
One should be aware of any MDP violations Our tool can find them (or some of them) Major MDP violations need to be fixed
22
ACKNOWLEDGEMENTS AND QUESTIONS
Thanks to: My fellow interns Pat Dickerson & Kim Basney Peter Heeman Rebecca Lunsford Andrew Rueckert Ethan Selfridge Everyone at OGI
Questions?
top related