uct for tactical assault battles in real-time strategy games
DESCRIPTION
UCT for Tactical Assault Battles in Real-Time Strategy Games. Radha-Krishna Balla 19 February, 2009. Overview. Introduction Related Work Method Experiments & Results Conclusion. Introduction Related Work Method Experiments & Results Conclusion. Domain. RTS games Resource Production - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/1.jpg)
RADHA-KRISHNA BALLA19 FEBRUARY, 2009
UCT for Tactical Assault Battles in Real-Time Strategy Games
![Page 2: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/2.jpg)
Overview
I. IntroductionII. Related WorkIII. MethodIV. Experiments & ResultsV. Conclusion
![Page 3: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/3.jpg)
I. IntroductionII. Related WorkIII. MethodIV. Experiments & ResultsV. Conclusion
![Page 4: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/4.jpg)
Domain
RTS games Resource Production Tactical Planning
Tactical Assault battles
![Page 5: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/5.jpg)
RTS game - Wargus
Screenshot of a typical battle scenario in Wargus
![Page 6: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/6.jpg)
Planning problem
Large state spaceTemporal actionsSpatial reasoningConcurrencyStochastic actionsChanging goals
![Page 7: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/7.jpg)
I. IntroductionII. Related WorkIII. MethodIV. Experiments & ResultsV. Conclusion
![Page 8: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/8.jpg)
Related Work
Board games – bridge, poker, Go etc., Monte Carlo simulations
RTS games Resource Production
Means-ends analysis – Chan et al., Tactical Planning
Monte Carlo simulations – Chung et al., Nash strategies – Sailer et al., Reinforcement learning – Wilson et al.,
Bandit-based problems, Go UCT – Kocsis et al., Gelly et al.,
![Page 9: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/9.jpg)
Our Approach
Monte Carlo simulationsUCT algorithm
Advantage Complex plans from simple abstract actions Exploration/Exploitation tradeoff Changing goals
![Page 10: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/10.jpg)
I. IntroductionII. Related WorkIII. MethodIV. Experiments & ResultsV. Conclusion
![Page 11: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/11.jpg)
Method
Planning architectureUCT AlgorithmSearch space formulationMonte Carlo simulationsChallenges
![Page 12: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/12.jpg)
Planning Architecture
Online Planner
State space abstraction Grouping of units
Abstract actions Join(G) Attack(f,e)
![Page 13: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/13.jpg)
UCT Algorithm
Exploration/Exploitation tradeoffMonte Carlo simulation – get subsequent
statesSearch tree
Root node – current state Edges – available actions Intermediate nodes – subsequent states Leaf nodes – terminal states
Rollout-based constructionValue estimates
![Page 14: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/14.jpg)
UCT Algorithm – Pseudo Code 1
At each interesting time point in the game:build_UCT_tree(current state);choose argmax action(s) based on the UCT policy; execute the aggregated actions in the actual game;wait until one of the actions get executed;
build_UCT_tree(state):for each UCT pass do
run UCT_rollout(state);
(.. continued)
![Page 15: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/15.jpg)
UCT Algorithm – Pseudo Code 2
UCT_rollout(state): recursive algorithm
if leaf node reached thenestimate final reward; propagate reward up the tree and update value functions; return;populate possible actions;
if all actions explored at least once thenchoose the action with best value function; else if there exists unexplored actionchoose an action based on random sampling;
run Monte-Carlo simulation to get next state based on current state and action;
call UCT_rollout(next state);
![Page 16: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/16.jpg)
UCT Algorithm - Formulae
𝑄+ሺ𝑠,𝑎ሻ= 𝑄ሺ𝑠,𝑎ሻ + 𝑐× ඨ𝑙𝑜𝑔𝑛ሺ𝑠ሻ𝑛ሺ𝑠,𝑎ሻ
𝜋ሺ𝑠ሻ= 𝑎𝑟𝑔𝑚𝑎𝑥𝑎 𝑄+ሺ𝑠,𝑎ሻ
𝑛ሺ𝑠,𝑎ሻ ← 𝑛ሺ𝑠,𝑎ሻ + 1 𝑛ሺ𝑠ሻ ← 𝑛ሺ𝑠ሻ + 1
𝑄ሺ𝑠,𝑎ሻ ←𝑄ሺ𝑠,𝑎ሻ + 1𝑛ሺ𝑠,𝑎ሻሾ𝑅− 𝑄ሺ𝑠,𝑎ሻሿ
Action Selection:
Value Updation:
![Page 17: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/17.jpg)
Search Space Formulation
![Page 18: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/18.jpg)
Monte Carlo Simulations
![Page 19: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/19.jpg)
Domain-specific Challenges
State space abstraction - Grouping of units (proximity-based)
Concurrency of actions - Aggregation of actions
- Join actions – simple - Attack actions – complex (partial
simulations)
![Page 20: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/20.jpg)
Planning problem - revisited
Large state space – abstractionTemporal actions – Monte Carlo simulationsSpatial reasoning – Monte Carlo simulationsConcurrency – aggregation of actionsStochastic actions – UCT (online planning)Changing goals – UCT (different objective
functions)
![Page 21: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/21.jpg)
I. IntroductionII. Related WorkIII. MethodIV. Experiments & ResultsV. Conclusion
![Page 22: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/22.jpg)
Experiments
# Scenario Name
# of friendly groups
Friendly groups
composition# of enemy
groupsEnemy groups composition
# of possible ‘Join’ actions
# of possible ‘Attack’ actions
Total # of possible actions
1 2vs2 2 {6,6} 2 {5,5} 1 4 5
2 3vs2 3 {6,2,4} 2 {5,5} 3 6 9
3 4vs2_1 4 {2,4,2,4} 2 {5,5} 6 8 14
4 4vs2_2 4 {2,4,2,4} 2 {5,5} 6 8 14
5 4vs2_3 4 {2,4,2,4} 2 {5,5} 6 8 14
6 4vs2_4 4 {2,4,2,4} 2 {5,5} 6 8 14
7 4vs2_5 4 {2,4,2,4} 2 {5,5} 6 8 14
8 4vs2_6 4 {2,4,2,4} 2 {5,5} 6 8 14
9 4vs2_7 4 {3,3,6,4} 2 {5,9} 6 8 14
10 4vs2_8 4 {3,3,3,6} 2 {5,8} 6 8 14
11 2vs4_1 2 {9,9} 4 {4,5,5,4} 1 8 9
12 2vs4_2 2 {9,9} 4 {5,5,5,5} 1 8 9
13 2vs4_3 2 {9,9} 4 {5,5,5,5} 1 8 9
14 2vs5_1 2 {9,9} 5 {5,5,5,5,5} 1 10 11
15 2vs5_2 2 {10,10} 5 {5,5,5,5,5} 1 10 11
16 3vs4 3 {12,4,4} 4 {5,5,5,5} 3 12 15
Table 1: Details of the different game scenarios
![Page 23: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/23.jpg)
Planners
UCT Planners UCT(t) UCT(hp)
Number of rollouts – 5000Averaged over – 5 runs
![Page 24: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/24.jpg)
Planners
Baseline Planners Random Attack-Closest Attack-Weakest Stratagus-AI Human
![Page 25: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/25.jpg)
Video – Planning in action
Simple scenario<add video>
Complex scenario<add video>
![Page 26: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/26.jpg)
Results
Figure 1: Time results for UCT(t) and baselines.
![Page 27: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/27.jpg)
Results
Figure 2: Hit point results for UCT(t) and baselines.
![Page 28: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/28.jpg)
Results
Figure 3: Time results for UCT(hp) and baselines.
![Page 29: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/29.jpg)
Results
Figure 4: Hit point results for UCT(hp) and baselines.
![Page 30: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/30.jpg)
Results - Comparison
Figures 1, 2, 3 & 4: Comparison between UCT(t) and UCT(hp) metrics
Time results Hit point results
UCT(t)
UCT
(hp)
![Page 31: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/31.jpg)
Results
Figure 5: Time results for UCT(t) with varying rollouts.
![Page 32: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/32.jpg)
I. IntroductionII. Related WorkIII. MethodIV. Experiments & ResultsV. Conclusion
![Page 33: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/33.jpg)
Conclusion
Conclusion
Future Work Engineering aspects Machine Learning techniques Beyond Tactical Assault
![Page 34: UCT for Tactical Assault Battles in Real-Time Strategy Games](https://reader031.vdocuments.mx/reader031/viewer/2022012922/568161e2550346895dd1f929/html5/thumbnails/34.jpg)
Thank you