games computers (cannot) play graham kendall automated scheduling, optimisation and planning...
TRANSCRIPT
Games Computers (cannot) Play
Graham KendallAutomated Scheduling, Optimisation and
Planning Research Group (ASAP)
Games Computers (cannot) Play
Checkers: Why was it considered “beaten”?
Two approaches to Checkers
Games in ASAP
Poker (if time)
Contents
Games Computers (cannot) Play
1959. Arthur Samuel started to look at Checkers2
The determination of weights through self-play ( adapted, remained fixed)
39 Features
Included look-ahead via mini-max
Computers & Game Playing : A Potted History
2 Samuel A. Some studies in machine learning using the game of checkers. IBM J. Res. Develop. 3 (1959), 210-229
Games Computers (cannot) Play
Samuels’s program defeated Robert Nealy, although the victory is surrounded in controversy
Was he state champion?
Did he lose the game or did Samuel win?
Computers & Game Playing : A Potted History
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : Just about to make move 16
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program)
Forced Jump
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program)
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program)
Strong(Try
to keep)Trapped
Only advance
to Square 28
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program)
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 23 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program)
Games Computers (cannot) Play
This was a very poor move.
It allowed Samual to retain his “Triangle of Oreo”
AND.. By moving his checker from 19 to 24 it guaranteed Samuel a King
This questioned how strong a player Nealy really was
Computers & Game Playing : A Potted History
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
16-12 then 5-1, Chinook said would be a draw
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
This checker is lost
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
KThis checker
could run (to 10) but..
K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
Forced Jump
K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
K
Games Computers (cannot) Play
Computers & Game Playing : A Potted History
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
White (Nealey)
Red (Samuel’s Program) : After Move 25
23
K
Victory
Games Computers (cannot) Play
Two Mistakes by Nealy
Allowing Samuel to get a King
Playing a move that led to defeat when there was a draw available
Computers & Game Playing : A Potted History
Games Computers (cannot) Play
The next year a six match rematch was won by Nealy 5-1.
Three years later (1966) the two world championship challengers (Walter Hellman and Derek Oldbury) played four games each against Samuel’s program. They won every game.
Computers & Game Playing : A Potted History
Games Computers (cannot) Play
Checkers
Chinook
Blondie 24 (aka Anaconda)
Computers & Game Playing : A Potted History
Games Computers (cannot) Play
Perfect
Each Player has complete knowledge of the game state
Usually only two players, who take alternate turns
Examples include Chess, Checkers, Awari, Connect-Four, Go, Othello
Types of Games
Games Computers (cannot) Play
Imperfect
Some of the game state is hidden
Examples include Poker, Cribbage, Bridge
Types of Games
Games Computers (cannot) Play
Games with an element of chance
The game moves have some stochastic element
For example, Backgammon
Types of Games
Games Computers (cannot) Play
Types of Games
Solved or Cracked
Over Champion
World Champion
Grand-Master
Amateur
Connect-Four Checkers (8x8)
Chess Go (9x9) Go (19x19)
Qubic Othello Backgammon
Nine Men’s Morris
Go_moku
Awari
6 Jaap van den Herik H., Uiterwijk and van Rijswijck J. Games Solved: Now and in the future. Artificial Intelligence 134 (2002) 277-311
Games Computers (cannot) Play
Case Study 1: Checkers
Samuel’s work, perhaps, restricted the research into Checkers until 1989 when Jonathan Schaeffer began working on Chinook
He had two aims
To develop the worlds best checkers player
To “solve” the game of checkers
Games Computers (cannot) Play
Case Study 1: Checkers
Chinook, at its heart, had an evaluation function
Piece count (+30% for a King)
Runaway checker
“Dog Hole”
The weights were hand-tuned
Games Computers (cannot) Play
Case Study 1: Checkers
Opening game database from published work (with corrections they found)
Initially 4000 openings, leading to an eventual 40,000
“Cooks” – innovative lines of play that could surprise an opponent
The aim was to take opponents into unknown territory
Games Computers (cannot) Play
Case Study 1: Checkers
Endgame database: Started writing in May 1989
The 8-piece endgame database finished on February 20th 1994
Games Computers (cannot) Play
Case Study 1: Checkers
1 120
2 6,972
3 261,224
4 7,092,774
5 148,688,232
6 2,503,611,964
7 34,779,531,480
8 406,309,208,481
Games Computers (cannot) Play
Case Study 1: Checkers
9 4,048,627,642,976
10 34,778,882,769,216
11 259,669,578,902,016
12 1,695,618,078,654,976
13 9,726,900,031,328,256
14 49,134,911,067,979,776
15 218,511,510,918,189,056
16 852,888,183,557,922,816
Games Computers (cannot) Play
Case Study 1: Checkers
17 2,905,162,728,973,680,640
18 8,568,043,414,939,516,928
19 21,661,954,506,100,113,408
20 46,352,957,062,510,379,008
21 82,459,728,874,435,248,128
22 118,435,747,136,817,856,512
23 129,406,908,049,181,900,800
24 90,072,726,844,888,186,880
TOTAL 500,995,484,682,338,672,639
Games Computers (cannot) Play
Case Study 1: Checkers
With a 4-piece database Chinook won the 1989 Computer Olympiad
In the 1990 US National Checkers Championship Chinook was using a 6-piece database.
It came second, to Marion Tinsley, defeating Don Lafferty on the way who was regarded at the worlds second best player.
Games Computers (cannot) Play
Case Study 1: Checkers
Marion Tinsley
Held the world championship from 1951 to 1994
Before playing Chinook, Tinsley only lost 4 competitive games (no matches)
Games Computers (cannot) Play
Case Study 1: Checkers
The winner of the US Championship has the right to play for the world championship. Finishing second (with Tinsley first) entitled Chinook to play for the world championship
The American Checkers Federation (ACF) and the European Draughts Association (ADF) refused to let a machine compete for the title.
Games Computers (cannot) Play
Case Study 1: Checkers
In protest, Tinsley resigned
The ACF and EDF, created a new world championship, “man versus machine” and named Tinsley as the world champion.
At this time Tinsley was rated at 2812, Chinook was rated at 2706
Games Computers (cannot) Play
Case Study 1: Checkers
The match took place 17-29 Aug 1992.
The $300,000 computer used in the tournament ran at about half the speed of a 1GHz PC
The match finished 4-2 in favour of Tinsley (with 34 draws)
Games Computers (cannot) Play
Case Study 1: Checkers
A 32 game rematch was held in 1994
8-piece end game
Processors four times as fast (resulted in a factor of 2 speed up due to more complex evaluation function and the overhead of parallel processing)
Opening book of 40,000 moves
In preparation Chinook no losses in 94 games against Grandmasters
Games Computers (cannot) Play
Case Study 1: Checkers
Six games in (1-1, with 4 draws) Tinsley resigned for health reasons. His symptoms were later diagnosed as pancreatic cancer.
Tinsley died on 3rd April 1995 (aged 68). Undoubtedly the best player ever to have lived
Chinook was crowned the man versus machine champion. The first automated game player to have achieved this.
A 20-match with Don Lafferty resulted in a draw (1-1, with 18 draws)
Games Computers (cannot) Play
Case Study 1: Checkers
Opening Game Database(40,000) moves
End Game Database(8-pieces)
Hand CraftedEvaluationFunction
(/ search)
Won the World (ManVersus Machine)Championship
in 1994…
…defeating the world whohad held the title
for 40 years
Marion Tinsley lost his 5th,6th and 7th games to
Chinook
Schaeffer J. One Jump Ahead:Challenging Human Supremacy
in checkers, Springer, 1997
Games Computers (cannot) Play
Case Study 2: Anaconda
Project started in the summer of 1998, following a conversation between David Fogel and Kumar Chellapilla
It was greatly influenced by the recent defeat of Kasparov by Deep Blue
Chess was seen as too complex so “draughts” was chosen instead
The aim is to evolve a player – rather than build in knowledge
Games Computers (cannot) Play
Case Study 2: Anaconda
Reject inputting into a neural network what humans think might be important
Reject inputting any direct knowledge into the program
Reject trying to optimise the weights for an evaluation function
Games Computers (cannot) Play
Case Study 2: Anaconda
The Gedanken Experiment
I offer to sit down and play a game with you. We sit across an 8x8 board and I tell you the legal moves
We play five games, only then do I say “You got 7 points.”I don’t tell you if you win or lost
We play another five games and I say “You got 5 points”
You only know “higher is better”
Games Computers (cannot) Play
Case Study 2: Anaconda
The Gedanken Experiment
How long would it take you to become an expert at this game?
We cannot conduct this experiment but we can get a computer to do it
Games Computers (cannot) Play
Case Study 2: Anaconda
Samuel’s Challenge: “Can we design a program that would invent its own features in a game of checkers and learn how to play, even up to the level of an expert?”
Games Computers (cannot) Play
Case Study 2: Anaconda
Newell’s Challenge: “Could the program learn just by playing games against itself and receiving feedback, not after each game, but only after a series of games, even to the point where the program wouldn’t even know which programs had been won or lost?”
Newell (and Minsky)7 believed that this was not possible, arguing that the way forward was to solve the credit assignment problem.
7 Minsky M. Steps Towards Artificial Intelligence. Proceedings of the IRE, 1961, 8-30
Games Computers (cannot) Play
Case Study 2: Anaconda
I1
I32
HL11
HL140
HL21
HL210
O
.
.
....
.
.
.
# weights=1741
Evaluation used for MiniMax
Later changed toan explicit piece
differential
Games Computers (cannot) Play
Case Study 2: Anaconda
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
23
K
Games Computers (cannot) Play
Case Study 2: Anaconda
1
7
8
14
25
27
+1
+1
+1
-K
-1
-1
All other neurons have an value of zero
Games Computers (cannot) Play
Case Study 2: Anaconda
Algorithm
Initialise 30 Networks
Each network played 5 games as red against random opponents
Games were played to completion or until 100 moves had been made (a draw)
+2 for a win, 0 for a draw, -1 for a loss
15 best performing networks were saved for the next generation and copies were mutated
Games Computers (cannot) Play
Case Study 2: Anaconda
Observations
The points for a win, lose draw were set such that wins were encouraged. No experimentation with different values were tried
Players could play a different number of games. This was, purposefully, not taken into account
Mutation was carried out using an evolutionary strategy
Games Computers (cannot) Play
Case Study 2: Anaconda
After 10 Generations
After 10 generations the best neural network was able to beat both its creators and a simple (undergraduate project) program which, by the authors admission was “weak”
Note: 400MHz PC
Games Computers (cannot) Play
Case Study 2: Anaconda
ACF Ratings
Grand (Senior) Master 2400+ Class E 1000-1199
Master 2200-2399 Class F 800-999
Expert 2000-2199 Class G 600-799
Class A 1800-1999 Class H 400-599
Class B 1600-1799 Class I 200-399
Class C 1400-1599 Class J <200
Class D 1200-1399
Games Computers (cannot) Play
Case Study 2: Anaconda
After 100 Generations
Playing on zone.com
Initial rating = 1600
Beat a player ranked at 1800 but lost against a player in the mid 1900’s
After 10 games their ranking had improved to 1680. After 100 games it had improved to 1750
Typically a 6-ply search but often 8-ply
Games Computers (cannot) Play
Case Study 2: Anaconda
Observations
The highest rating it achieved was 1825
The evolved King value was 1.4, which agrees with perceived wisdom that a king is worth about 1.5 of a checker
In 100 generations a neural network had been created that was competitive with humans
It surpassed Samuel’s program
The challenge set by Newell had been met
Games Computers (cannot) Play
Case Study 2: Anaconda
The Next Development
Alpha-Beta Pruning introduced and evolved over 250 generations
Over a series of games, Obi_WanThe Jedi defeated a player rated at 2134 (48 out of 40,000 registered) and also beat a player rated 2207 (ranked 18)
Final rating was 1902 (taking into account the different orderings of the games)
Games Computers (cannot) Play
Case Study 2: Anaconda
The Next Development
Spatial nature of the board was introduced as at the moment it just “saw” the board as a vector of length 32
Games Computers (cannot) Play
Case Study 2: Anaconda
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
23
36 3x3 Overlapping
squares
Games Computers (cannot) Play
Case Study 2: Anaconda
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
23
25 4x4 Overlapping
squares
Games Computers (cannot) Play
Case Study 2: Anaconda
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
23
16 5x5 Overlapping
squares
Games Computers (cannot) Play
Case Study 2: Anaconda
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
23
9 6x6 Overlapping
squares
Games Computers (cannot) Play
Case Study 2: Anaconda
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
23
4 7x7 Overlapping
squares
Games Computers (cannot) Play
Case Study 2: Anaconda
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20
21 22 24
25 26 2728
29 30 31 32
23
1 8x8 Overlapping
squares
Games Computers (cannot) Play
Case Study 2: Anaconda
The Next Development
36+25+16+9+4+1 = 91 inputs
5,046 weights
Games Computers (cannot) Play
Case Study 2: Anaconda
I1
I32
O
.
.
.
# weights=5046
36 3x3
25 4x4
16 5x5
9 6x6
4 7x7
1 8x8
HL1
(91 nodes)
HL2
(40 nodes)
HL3
(10 nodes)
Sum of 32 Board Inputs
Games Computers (cannot) Play
Case Study 2: Anaconda
2 months and 230 generations later!!
After 100 games the rating was 1929
A 27 point increase over the previous network. Nice but not decisive
Maybe it was due to there being three times more weights but the training period was the same?
Games Computers (cannot) Play
Case Study 2: Anaconda
6 months and 840 generations later!!
After 165 games it was rated at 2045.85 (sd 33.94)
Rated in the top 500 at zone.com (of the 120,000 players now registered)
That is better than 99.61% of the players
Games Computers (cannot) Play
Case Study 2: Anaconda
Playing Chinook8
In a ten match series against Chinnok novice level it had two wins, two losses and 4 draws
8 Fogel D. B. and Chellapilla K. Verifying Anaconda’s expert rating by competing against Chinook: experiments in co-evolving a neural checkers player, Neurocomputing 42 (2002) 69-86
Games Computers (cannot) Play
Case Study 2: Anaconda
Blondie
The neural checkers player went through a number of names
David0111
Anaconda
Blondie24
Games Computers (cannot) Play
Case Study 2: Anaconda
Games Computers (cannot) Play
Case Study 2: Anaconda
References
Fogel D.B. Blondie24: Playing at the Edge of AI, Morgan Kaufmann, 2002
Fogel D. B. and Chellapilla K. Verifying Anaconda’s expert rating by competing against Chinook: experiments in co-evolving a neural checkers player, Neurocomputing 42 (2002) 69-86
Chellapilla K and Fogel D. B. Evolving neural networks to play checkers without expert knowledge. IEEE Trans. Neural Networks 10(6):1382-1391, 1999
Chellapilla K and Fogel D. B.Evolution, neural networks, games, and intelligence, Proc. IEEE 87(9):1471-1496. 1999
Chellapilla K and Fogel D. B. Evolving an expert checkers playing program without relying on human expertise. IEEE Trans. Evolutionary Computation, 2001
Chellapilla K and Fogel D. B. Anaconda Defeats Hoyle 6-0: A Case Study Competing an Evolved Checkers Program Against Commercially Available Software. Proc. Of CEC 2000:857-863