chapter 13 deterministic dynamic programming
DESCRIPTION
Chapter 13 DETERMINISTIC DYNAMIC PROGRAMMING. Math 305 2008 We will cover 9.1-9.4 plus some material not in the text. Dynamic Programming. Technique for making a sequence of interrelated decisions Problem solving strategies, e.g. find a route from here to LA - PowerPoint PPT PresentationTRANSCRIPT
Chapter 13 DETERMINISTIC DYNAMIC PROGRAMMING
Math 305 2008
We will cover 9.1-9.4 plus some material not in the text
Dynamic Programming
Technique for making a sequence of interrelated decisionsProblem solving strategies, e.g. find a route from here to LA
– forward: enumerate all possibilities– backward: figure which ways one can get to the desired end
General type of problem: consecutive stages– at each stage you are in one of a number of possible states– each state has one or more possible policies from which to choose – the policy you choose determines your state at the next stage
Method
– start with a solution for a small part of the problem – expand
A useful approach when you don’t want to check all possiblities
Prototype problem (stage coach)
A traveller is going from east to west coast on a series of stage coaches that travel from one state to another and wants the safest routeS/he selects a life insurance policy for each stage, reasoning that
cheapest = safest
Approach
4 stagesStates at each stage:
1: 1 2: 2,3,43: 5, 6, 74: 8,9
First stage: travel from 1 to 2, 3, or 4Second stage: travel from state at that stage to next stageDecision variables: xn (n=1,..,4) = destination on nth stage
Approach
fn(s,xn) = total cost of best policy for remaining stages given you are in state s at stage n and select policy xn (n = stage, s = state, xn = decision)fn(s) = cost of best policy given you are in state s at stage n
= min fn(s, xn) over possible choices for xn
fn(s, xn) = c s xn + fn+1(xn) (c s xn
= cost of policy from s to xn)
goal: find f1(1)f1(1) = min of f1 (1,2) = 2 + f2(2) f1 (1,2) = 5 + f2(3) f1 (1,4) = 1 + f2(4)
Start at Stage 4
f2*(2) = ? ...Better: go backwardFind the best policy from stage 4 on, stage 3 on,...
Stage 4: states 8 and 9 → we want f4*(8) and f4*(9) f4(8,10) = 1 = f4*(8) f4(9,10) = 4 = f4*(9)
s\x4 f4*(s) x4*8 1 109 4 10
Stage 3
States 5, 6, 7 → we want f3*(5), f3*(6). f3*(7) f3(5,8) = 7 + f4*(8) = 8f3(5,9) = 5 + f4*(9) = 9 f3(6,8) = 3 + f4*(8) = 4f3(6,9) = 4 + f4*(9) = 8 f3(7,8) = 7 + f4*(9) = 8 f3(7,9) = 1 + f4*(9) = 5
So far this isn't any better thanexhustive search
s\x3 8 9 f3*(s) x3*
5 8 9 8 8
6 4 8 4 8
7 8 5 5 9
Stage 2
States 2, 3, 4f2(2,5) = 10 + f3*(5) = 18f2(2,6) = 12 + f3*(6) = 16 f2(3,5) = 5 + f3*(5) = 13f2(3,6) = 10 + f3*(6) = 14f3(3,7) = 7 + f3*(7) = 12 f2(4,6) = 15+f3*(6) = 19
s\x2 5 6 7 f2*(s) x2*
2 18 16 - 16 6
3 13 14 12 12 7
4 - 19 18 18 7
Stage 1
f1(1,2) = 2 + f2*(2) = 18f1(1,3) = 5 + f2*(3) = 17f1(1,4) = 1 + f2*(4) = 19
s\x1 2 3 4 f1(s) x1*
1 18 17 19 17 3
Shortest route:
1 -> 3
3->7
7 -> 9
9->10
s\x1 2 3 4 f1(s) x1*
1 18 17 19 17 3
s\x2 5 6 7 f2*(s) x2*
2 18 16 - 16 6
3 13 14 12 12 7
4 - 19 18 18 7
s\x3 8 9 f3*(s) x3*
5 8 9 8 8
6 4 8 4 8
7 8 5 5 9
s\x4 f4*(s) x4*
8 1 10
9 4 10
Computational Efficiency
This method? 16 additions, 9 comparisons
How else could we solve this?– list all paths (14) and total # additions (3 on each) => 42– shortest route ?
Compare n+1 stages with n choices at each stage except lastDynamic programming: n nodes with n additions each =n2 Exhaustive search: nn paths with n additions = nn+1
E.g. n=10: 100 versus 1011
13.3 Inventory Theory
What is inventory?– something that is produced– has a demand– needs to be stored until used– cookies at Dories, beer at the Fargo, flash drives at the bookstore
What is an inventory policy?– when to order/produce more– how much at a time
What costs are associated with inventory?– cost per unit (variable)– setup or ordering– holding– shortage
Inventory Theory
What are we trying to optimize?Assumptions
– time is broken into periods– production occurs at the beginning of the period– each period has an associated demand which is met from items held
over from the last period and/or produced in the current period
13.3 Example
Demand for a product Costs
c(x)=cost of producing x units=3 + 1x
Other restrictions– at most 5 units can be produced each month– at most 4 units can be carried over to the next month– 0 units on hand at the beginning of month 1
What are the stages, states, and decisions?– stage: beginning of a month– state: entering inventory– decision: fn(s) = amt produced at beginning of n, given s units on hand
Month Demand1 12 33 24 4
Cost AmountSetup 3Variable 1Holding .50
Stage 4
f4(i) = cost of entering period 4 with i units = cost of producing 4 - i units = c(4-s) f4(0) = set up + cost of 4 units = 3 + 4
f4(1) = set up + cost of 3 units = 3 + 3
Demand is 1 3 2 4
f4(i) x4*
0 7 4
1 6 3
2 5 2
3 4 1
4 0 0
f3 (i) = min {c(x) + 0.5(x + i - 2) + f4(i+x -2) } x=0..4 ordering holding stage 4 on
Demand is 1 3 2 4
i 0 1 2 3 4 5 f3 (i) x3(i)0 12 12.5 13 13.5 12 21 11 11.5 12 12.5 10 10 52 7 10.5 11 11.5 9 7 03 6.5 10 10.5 8 6.5 04 6 9.5 7 6 05 5.5 6 5.5 0
Stage 3
i x cost f3 (i) x3(i)3 0 6.5 6.5 03 1 103 2 10.53 3 84 0 6 6 04 1 9.54 2 75 0 5.5 5.5 05 1 6
i x cost f3 (i) x3(i)0 2 12 12 20 3 12.50 4 130 5 13.51 1 11 10 51 2 11.51 3 121 4 12.51 5 102 0 7 7 02 1 10.52 2 112 3 11.52 4 9
Alternate Notation for Stage 3
Stage 2
f2 (i) = min {c(x) + 0.5(x + i - 3) + f2(i+x -3) } for x = 0, 1, ...,5 x=0,1,2,3,4
Demand is 1 3 2 4
i 0 1 2 3 4 5 f3 (i) x3(i)0 18 18.5 16 16 51 17 17.5 15 16 15 42 16 15.5 14 15 16 14 33 12 14.5 13 14 15 12 04 10.5 12 13 14 10.5 05 ...
Alternate Notation for Stage 2i x cost f2 (i) x2(i)0 3 18 16 50 4 18.50 5 161 2 17 15 11 3 17.51 4 151 5 162 1 16 3 12 2 15.52 3 142 4 152 5 16
i x cost f2 (i) x2(i)3 0 12 12 03 1 14.53 2 133 3 143 4 154 0 10.5 10.5 04 1 124 2 134 3 14
Stage 1
f1 (i) = min {c(x) + 0.5(x + i - 1) + f1(i+x - 1) } for x = 0, 1, ...,5 x=0,1,2,3,4
Demand is 1 3 2 4(text also finds f1 (i) for i = 1,2,3,4)
i 1 2 3 4 5 f1(i) x1(i)
0 20 20.5 21 20.5 20.5 20 1
There must be an easier way(not in text)
Only produce when entering inventory=0– don't carry inventory to meet part of the demand if you will have to produce in the period
Demand is 1 3 2 4
f4(i) x4*
0 7 4
4 0 0
i x cost f3 (i) x3(i)0 2 12 12 22 0 7 7 0
i x cost f2 (i) x2(i)0 3 18 16 50 4 18.53 0 12 12 0
i x cost f1(i) x1(i)0 1 20 20 10 4 21.5
Network Representation
2,0
1, 0
2,1
2,2
2,3
2,4
5,0
(i,j): i = period, j = beginning inventory
13.4 Resource Allocation
I have 5 blocks of time to study and want to maximize the sum of my grades
gi(xi) = grade in subject i given I study xi blocks.3 decisions: xn = # blocks to spend on subject nstage = subjectstate = time available to allocate to remaining stages
hours Eng Econ Phys
0 40 65 40
1 65 70 55
2 75 89 63
3 88 91 78
4 93 95 81
5 99 98 85
Resource Allocation
gi(xi) = grade in subject i given xi blocks
Objective: max gi(xi) subject to xi= 5
fn(s) = max effectiveness of s hours in stages n through 3
fn(s, xn) = gn(xn) + fn+1(s - xn) = grade from xn hours to subject n plus best effect of remaining hours in later stages
fn(s) = max { gn(xn) + fn+1(s - xn)} xn=o..s
Resource Allocation
f3(s) f2(s)
f1(s)
Solution: English 2, Econ 2, Physics 1
s f3(s) x3
0 40 01 55 12 63 23 78 34 81 45 85 5
s\x2 0 1 2 3 4 5 f2(s) x2
0 105 105 0
1 120 119 120 02 128 125 129 129 23 143 133 131 144 2
4 146 148 152 146 135 152 25 150 151 154 167 150 138 167 2
s\x1 0 1 2 3 4 5 f1(s) x1
5 207 217 219 217 213 204 219 2
Continuous Functions
Suppose we can allocate fractional units of time and grade is a continuous function of time spent
English: g1(x) = 65 - x2 + 11x max at (5.5, 95)Econ: g2(x) = 80 - 2x2 + 13 x max at (3.25, 101.125)Physics: g3(x) = 55 + 7x
f3(s) = 55 + 7s x3 = sf2(s, x2) = g2(x2) + f3(s - x2) = 80 -2 x2
2 + 13x2 + 55 + 7(s - x2) = 135 - 2 x2
2 + 6x2 + 7s df2/dx2 = -4x2 + 6 = 0 when x2 = 3/2 d2f2/dx2
2 = -4 -> max at x2 = 3/2 (if s ≥ 3/2)otherwise max at x2 = s
Continuous Functions
Case 1, s < 3/2 x2 = s, f2(s) = 80 - 2s2 + 13s + 55 + 7(s - s) = 135 + 13s - 2s2
Case 2, 3/2 s f2(s) = 80 + 2(3/2)2 + 13(3/2) + 55 + 7(s - 3/2) = 150 + 7s
Implications – if available time < 3/2, put it all into econ– if ≥ 3/2, put 3/2 into econ and surplus into physics (compare slopes of the two graphs before and after 3/2)
x2 f2(s) s < 3/2 s 135 + 13s - 2s2
3/2 <= s 3/2 150 + 7s
Continuous Functions
f1(5, x1) = g1(x1) + f2(5- x1)case1, 5 - x1 < 3/2 (x1 > 3.5) f1(5, x1) = 65 - x1
2 + 11x + 135 + 13(5 - x1) - 2(5 - x1)2 df1/d x1 = -2 x1 + 11 - 13 - 4(5- x1) = 2 x1 - 22 < 0 for 3.5 < x1 5
-> max at x1 = 3.5 if 3.5 < x1 5. f1(5. 3.5) = 241.25case 2, 5 - x1 3/2 (0 x1 3.5) f1(x1, 5) = 65 - x1
2 + 11 x1 + 150 + 7(5- x1) = 250 + 4 x1 - x1 2
df1/d x1 = 4 - 2 x1 = 0 at x = 2 d2f1/dx12 = -2 -> max at x = 2 f1(5) = f1(5,2) = 254
x2 f2(s) s < 3/2 s 135 + 13s - 2s2
3/2 <= s 3/2 150 + 7s
Continuous Functions
Decision: x1 = 2Thus 5 - x1 = 3 left for remaining stages 3 > 3/2 -> x2 = 3/2 -> x3 = 3 - 3/2 = 3/2 Solution: x1 = 2 x2 = 3/2 x3 = 3/2, sum of grades = 254
x3 f3(s)
0 ≤ s ≤ 5 s 55 + 7s
x2 f2(s)
s < 3/2 s 135 + 13s - 2s2
3/2 ≤ s 3/2 150 + 7s
x1 f1(s)
0 ≤ x1 < 3.5 2 254
3.5 ≤ x1 ≤ 5 3.5 241.25
Probabilistic Models
The state at the next stage is not completely determined by decision at current stage, rather determines a probability distribution for the next stage.
Objective: maximize the expected value.Example: job interview
– a job candidate has up to three interviews– at each, she will be offered a job which is terrific, good or fair– she must decide then whether to accept the job or interview again– a terrific job is worth 3 points, good: 2, fair: 1
Stages: interviewsState: job status at stage n (T, G, or F)Decision: interview or accept
Job Prob. Value
T .2 3
G .5 2
F .3 1
Probabilistic Models
fn(s) = max expected value if in state s at stage nfn(s, xn) = max expected value if in state s at stage n and make decision xn (xn = i or a)
Job Prob. Value
T .2 3
G .5 2
F .3 10
F
G
T T T
GG
F F
Stage 3
f3(T, i) = 3(.2) + 2(.5) + 1(.3) = 1.9f3(T,a) = 3 -> f3(T) = 3, x3 = af3(G, i) = 3(.2) + 2(.5) + 1(.3) = 1.9f3(G,a) = 2 -> f3(G) = 2, x3 = a f3(F, i) = 3(.2) + 2(.5) + 1(.3) = 1.9f3(F,a) = 1 -> f3(F) = 1.9, x3 = i
s\ x3 i a f3(s) x3
T 1.9 3 3 a
G 1.9 2 2 a
F 1.9 1 1.9 i
Stage 2
f2(T, i) = p(T)f3(T) + p(G)f3(G) + p(F)f3(F) = .2(3) + .5(2) + .3(1.9) = 2.17
f2(T,a) = 3 -> f2(T) = 3, x2 = a
f1(i) = .2f2(T) + .5f2(G) + .3f2(F) = .2(3) + .5(2.17) + .3(2.17) = 2.336
Strategy at stage I: interview; II: interview in G or F.; III: interview in F
s\ x2 i a f2*(s) x2
T 2.17 3 3 a
G 2.17 2 2.17 i
F 2.17 1 2.17 i
Stage 1
Knapsack Problem (back to 13.4)
A thief breaks into a house.Around the thief are various objects: a diamond ring, a silver candelabra, a Bose Wave Radio, a large portrait of Elvis Presley painted on a blackvelvet background (a "velvet-elvis"), and a large tiffany crystal vase. The thief has a knapsack that can only hold a certain capacity (8). Each of the items has a value and a size, and cannot hold all of the items inthe knapsack.
Which items should the thief take? There are three thieves: greedy, foolish and slow, and wise (ref for this example)
Item Size Value
Ring 1 15
Candelabra 5 10
Radio 3 9
Elvis 4 5
Knapsack Problem
The greedy thief breaks into the window, and sees the items. He makes a mental list of the items, and grabs the most expensive item first. The ring goes in first, leaving a capacity of 7, and a value of 15. Next, he grabs the candelabra, leaving a knapsack of size 2 and a value of 25. No other items will fit in his knapsack, so he leaves.
The foolish and slow thief climbs in the window, and sees the items. This thief was a programmer, downsized as part of the "dot-bomb" blowout. Possessing a solid background in boolean logic, he figures that he can simply compute all combinations of the objects and choose the best. So, he starts going through the binary combinations of objects - all 25 of them. While he is still drawing the truth table, the police show up, and arrest him. Although his solution would certainly have given him the best answer, it just took long to compute.
Item Size Value
Ring 1 15
Candelabra 5 10
Radio 3 9
Elvis 4 5
Knapsack Problem, Wise Thief
The wise thief appears, and observes the items. He notes that an empty knapsack has a value of 0. He notes that a knapsack can either contain each item, or not. Further, his decision to include an item will be based on a quick calculation - either the knapsack with some combination of the previous items will be worth more, or else the knapsack of a size that will fit the current item was worth more. So, he does this quick computation, and figures out that the best knapsack he can take is made up of items 1,3, and 4, for a total value of 29
Item Size Value
Ring 1 15
Candelabra 5 10
Radio 3 9
Elvis 4 5
Generalized Resource Problem (p. 767)
w units of a resource availableT activities in which the resources can be allocatedxt : the level at which activity t is implementedgt(xt ): # of units of the resource used by activity trt(xt ): the resulting benefitStates: each activityStages: how much of the resource is available for remaining stagesDecision: how much to use at this stageFormulation
maximize Σ rt(xt ) s.t. Σ gt(xt ) ≤ w t = 1,...T t = 1,...T
ft(d) = max benefit if d units are allocated to activities t through Tft(d) = max {rt(xt ) + ft+1(d - xt ) fT+1(d) = 0 xt
Wise Thief, Stages 4 and 3
Item Size Value
Ring 1 15
Candelabra 5 10
Radio 3 9
Elvis 4 5
s\x4 0 4 f4*(x4) x4
0 0 0 04 0 5 5 4
s\x3 0 3 f3*(x3) x3
0 0 0 0
3 0 9 9 3
4 4 9 9 3
7 4 14 14 3
f3(d) = max {r3(x3 ) + f4(d - x3 )f3(3) = max {r3(0) + f4(3 ) = 0 + 0 r3(3) + f4(0 ) = 9 + 0}
f3(4) = max {r3(0) + f4(4 ) = 4 r3(3) + f4(1 ) = 9}
f3(7) = max {r3(0) + f4(7) = 4 r3(3) + f4(4 ) = 9 + 5 =14
Stage 2
Item Size Value
Ring 1 15
Candelabra 5 10
Radio 3 9
Elvis 4 5
s\x3 0 3 f3*(x3) x3
0 0 0 0
3 9 9 4
4 4 9 9 3
7 4 14 14 14
f2(d) = max {r2(x2) + f43(d - x2)f2(3) = max {r2(0) + f3(3 ) = 4}
f3(4) = max {r2(0) + f3(4) = 9}
f3(5) = max {r2(0) + f3(5) = 3 r2(5) + f3(0 ) = 10}f3(7) = max {r2(0) + f3(7) = 14 r2(5) + f3(2 ) = 10}f3(8) = max {r2(0) + f3(8) = 14 r2(5) + f3(3) = 10 + 9 = 19}
s\x2 0 5 f2*(x2) x2
0 0 0 0
3 9 9 0
4 9 9 0
5 9 10 10 5
7 14 10 14 0
8 14 19 19 5
Stage 1
Item Size Value
Ring 1 15
Candelabra 5 10
Radio 3 9
Elvis 4 5
f1(8) = max {r1(0) + f2(8 ) = 0 + 19} r1(1) + f2(7) = 15 + 14}s\x1 0 1 f1*(x1) x1
8 19 29 29 1
s\x2 0 5 f2*(x2) x2
0 0 0 0
3 9 4 0
4 9 0
5 9 10 10 5
7 14 10 14 0
8 14 19 19 5
How is this an LP?
xi = # item imax 15x1 + 10x2 + 9x3 + 5x4 s.t. x1 + 5x2 + 3x3 + 4x4 ≤ 8
Item Size Value
Ring 1 15
Candelabra 5 10
Radio 3 9
Elvis 4 5
Turnpike Theorem
Let cj = benefit from item j, wj = weight of item jOrder items by benefit per unit weight
c1/w1 ≥ c2 /w2 ≥ c3 /w3 ...
If there is a unique "best" item #1, e.g. c1/w1 > c2 /w2 when the max weight w ≥ w* = (c1w1)/(c1 - w1 ( c2 /w2 )
the optimal solution contains at least one of item 1Thief problem:
15/1 > 9/3 > 10/5 > 5/4w* = 15*1/(15 - 1*9/3) = 15/12 = 1.25 < 8
Why is this any use? – start with one ring and reduce computation
Why turnpike– for a long trip you might go a little out of your way to maximize time
on a turnpike.
Item Size Value
Ring 1 15
Candelabra 5 10
Radio 3 9
Elvis 4 5
Proof of Turnpike TheoremWithout using any type 1 items we cannot do better than include w/w2 type 2This would earn c2w/w2 Suppose we fill the knapsack with as many type 1 items as possibleWe can fit in at least (w/w1 1) type 1 items ‑These items would earn a benefit of c1(w/w1 1)‑Thus if c1(w/w1 1)‑ c2w/w2 (1) there must be an optimal solution using a type 1 item(1) holds if w(c1/w1 c‑ 2/w2)c1
c1w1
or w = w* ‑‑‑‑‑‑‑‑‑‑‑‑‑‑‑c1 c‑ 2w1/w2
Thus if knapsack can hold at least w* pounds, there will be an optimal solution using at least one type 1 item