Learning Othello

The quest for general strategy building.

Objectives and Goals

• Develop a program that is capable of learning to play Othello.

• More interested in how the program learns.

• Learn which techniques are more affective than others

• NOT building Deep Blue!

The Big Picture

• Just a small piece in a much bigger puzzle.

• Using games to explore building a general strategy ontology.

• Is it possible to find patterns in different games to form similar, working strategies?

What is Othello?

• Two player grid-based game.• Black and white discs used.• Must move if you can, skip turn if you

can’t.• Object is to “outflank” opponent’s discs and

convert them to yours.• Game over when no more moves possible.• Winner is player with most discs.

The Board

System Design

Knowledge Base

LearningAgent(GA)

Strategy Strategy

GameControllerPlayer

AgentPlayerAgent

GUI Display Text Display

System Developed Iteratively

• Iteration 1– Knowledge base will contain moves and

genetic algorithm information.

• Iteration 2– Game logic contained in controller moved to

knowledge base.

• Iteration 3– All player logic stored in knowledge base.

Game Controller

• Initialize all components of the system.

• Setup the default board.

• Prepare the display.

• Use learning agent to prepare player strategies.

• Control game flow.

• Control training sessions.

System Design

Knowledge Base

LearningAgent(GA)

Strategy Strategy

GameControllerPlayer

AgentPlayerAgent

GUI Display Text Display

Game Controller Cont.

• Iteration 1– Present player agent with list of possible

moves.– Take move from player agent and control

board.

• In iteration 2, a lot of this logic will be contained in the KB.

Player Agent

• Consult strategy to return a move to the Game Controller.

• Iteration 1– Game Controller will give agent all possible

moves to chose from.

• Iteration 2– Game controller will give agent copy of the

board, and the agent will find and chose move.

System Design

Knowledge Base

LearningAgent(GA)

Strategy Strategy

GameControllerPlayer

AgentPlayerAgent

GUI Display Text Display

Learning Agent

• Majority of the focus for the project and iteration 1.

• Uses genetic algorithm approach for building strategy.

• Moderate risk because not directly applicable to the problem domain.

Steps for GA

• Encode problem in terms of chromosomes.

• Define fitness function for chromosomes.

• Generate initial population.

• Measure fitness and select fittest to “mate.”– Cross over– Mutation

• Repeat

The Genetics of Othello

• Moves will be the genes.– Horizontal_skip, Horizontal_jump– Vertical_skip, Vertical_jump– Diagonal_skip, Diagonal_jump– Hybrid (?)

• In iteration 1, a chromosome will contain all possible moves.

• Iteration 2, a chromosome may not contain all moves.

Fitness

• Simplest measure is your fit if you win.

• More complicated, and possibly more accurate is a measure of the number of discs taken.– Scoring system from online game.

The Birds and Bees of Othello

• Crossover– Swapping the order of the moves in a

chromosome.– Need to avoid duplicates.

• Mutation– Change the jump/skip value.– True mutation because of the coupling?– Defined differently in hybrid moves?

Knowledge Base

• Iteration 1– Contain the definition of the genetic operators (crossover

and mutation)– Contain the moves and their relation to the genetic

operators

• Iteration 2– More game logic stored so fitness functions can be

defined and tied closer to the genetic algorithm information

• Iteration 3– All game logic contained in KB.

Overall Application Flow

• Game controller started and components initialized.

• Learning agent populates initial chromosome base.

• Game controller directs games.

• Learning agent “mates” chromosomes based on game outcomes.

Issues

• Defining a strategy simply by moves may be too simplistic

• How exactly to mate the chromosomes– One chromosome that is operated on

• Might work if it contains all moves

– Multiple chromosomes to mix and match• How play multiple simultaneous games

• Encoding the game knowledge to make it “computational”

Summary

• Good first step towards a very complex problem.

• Allows the ability to explore different areas of AI with the same infrastructure.– Swappable learning agents