cs 680: game aisanti/teaching/2012/cs680/... · 2012. 2. 20. · reminders • progress self-check...

CS 680: GAME AI WEEK 6: STORY EVALUATION AND

PLAYER MODELING

2/20/2012 Santiago Ontañón [email protected] https://www.cs.drexel.edu/~santi/teaching/2012/CS680/intro.html

Reminders • Progress self-check indicator:

•  Your progress is good is you have: •  Project 1 completed. •  You know what are you going to do for project 2 and have looked into

the source code of GraphicIFEngine and understand the basics of it.

Outline •  Student Presentation:

“The Corridor Map Method: a General Framework for Real-Time High-Quality Path Planning”

•  Student Presentation: “Structuring Content in the Façade Interactive Drama Architecture”

•  (Very brief Introduction to Machine Learning) •  Story Evaluation •  Player Modeling •  SE and PM for Drama Management

•  Project Discussion

Machine Learning • Branch of AI that studies how to:

•  Infer general knowledge from data: supervised learning •  Infer behavior from data: learning from demonstration •  Find hidden structure in data: unsupervised learning •  Infer behavior from trial an error (data): reinforcement learning

• Underlying principle is inductive inference: •  E.g. after seeing 100 times that objects fall down, we can infer by

induction the general law of gravity. •  We cannot deduce gravity from observation, we can only induce it.

Machine Learning • Branch of AI that studies how to:

•  Infer general knowledge from data: supervised learning •  Infer behavior from data: learning from demonstration •  Find hidden structure in data: unsupervised learning •  Infer behavior from trial an error (data): reinforcement learning

• Underlying principle is inductive inference: •  E.g. after seeing 100 times that objects fall down, we can infer by

induction the general law of gravity. •  We cannot deduce gravity from observation, we can only induce it.

For today’s class, we only care about

supervised learning

Supervised Learning • Given labeled (supervised) training data:

(x1,y1) (x2,y2) (x3,y3) … (xn,yn)

•  Find the general function that maps x to y: •  f(x) = y

Examples

Supervised Learning Example •  Learning when to cross the street:

•  X is defined as a list of features: (“cars coming?”, “pedestrian traffic light color”, “time”)

•  Y defined as: cross, stay •  Examples:

•  x1 = (no, green, 3pm) y1 = cross •  x2 = (no, red, 8am) y2 = don’t cross •  x3 = (yes, red, 5pm) y3 = don’t cross •  x4 = (yes, green, 9pm) y4 = don’t cross •  x5 = (no, green, 10am) y5 = cross

•  Outcome of machine learning? •  f(x) = “cross if no cars coming and light color green”

Supervised Learning • Examples can be:

•  Fixed-length vector of attributes: •  Numerical •  Categorical

•  Structured representations (graphs, trees, logical clauses)

• Machine learning works better with: •  Few representative attributes •  Lots of data

Simplest Method: Nearest Neighbor • Examples are fixed vectors of real numbers •  To predict f(x):

•  f(x) = yi where (xi,yi) is the example with xi most similar to x (nearest neighbor)



x



x Nearest neighbor f(x) = gray

Machine Learning Methods: • K-Nearest Neighbor • Decision Trees (ID3) •  Linear Regression • Bayesian Models (Naïve Bayes) • Boosting • Support Vector Machines • Neural Networks •  etc.

Machine Learning Methods: • K-Nearest Neighbor • Decision Trees (ID3) •  Linear Regression • Bayesian Models (Naïve Bayes) • Boosting • Support Vector Machines • Neural Networks •  etc.

Which one is the best?

Supervised Learning Methods: • K-Nearest Neighbor • Decision Trees (ID3) •  Linear Regression • Bayesian Models (Naïve Bayes) • Boosting • Support Vector Machines • Neural Networks •  etc.

Which one is the best?

NONE! Each one has its own biases

Drama Management

Game Engine

Drama Manager

Player

Game Adaptation

Evaluation Function Narrative

Story Evaluation in Drama Management •  The interaction between the player and a computer game

can be seen as a form of storytelling

• Each new game the player plays is a new story

•  Idea: •  Using game tree search, the drama manager can foresee the

stories that will unfold depending on the actions executed by the player

•  If the DM can automatically evaluate the quality of such stories, it can modify the game to guide the player towards the best ones

Can the Quality of a Story be Automatically Evaluated? • Given a story:

•  “Alice arrived at the mansion and headed towards the basement. There, she found some secret documents containing information about the Verlag family, but nothing about the death of William. She decided to go out and search in the library …”

•  Is that a good story? Is it boring? interesting?

• Can we automatically assign a number between 0 and 1 to assess whether it is interesting or not?




• Can we automatically assign a number between 0 and 1 to assess whether it is interesting or not?

In general: NO

“interesting” is not well defined, since “interesting” is a subjective

perception, different for each individual player




• Can we automatically assign a number between 0 and 1 to assess whether it is interesting TO A PARTICULAR USER or not?




• Can we automatically assign a number between 0 and 1 to assess whether it is interesting TO A PARTICULAR USER or not?

The answer is still NO

“interesting” to a specific individual is still subject so contextual factors

(e.g. mood, or what happened in the immediate past).

But it can be approximated!

Story Evaluation Approaches • Synthetic: manually capturing aesthetic knowledge into

mathematical functions. •  E.g.: measuring how much the excitement curve of a story

approaches an Aristotelian curve

• Machine Learning: Learning story quality from examples. •  Use machine learning




•  (Very brief Introduction to Machine Learning) •  Story Evaluation

•  Synthetic Approaches •  Machine Learning

•  Player Modeling •  SE and PM for Drama Management


Synthetic Approaches to Story Quality • Even if it’s very hard to capture “quality” as a

mathematical function, there are certain measurable aspects of stories we can focus on.

• Main idea: •  1) Identify a series of measurable aspects •  2) Define a function for each aspect that, given a story, assigns a

numerical value to it •  3) Combine all of those functions in a single number (e.g.

averaging) to provide a final measure of quality

Measurable Aspects •  Tea-For-Three (Weyhrauch, 1997):

•  Thought Flow •  Activity Flow •  Options (Agency) •  Motivation •  Momentum •  Intensity •  Manipulation

•  Façade (Mateas & Stern, 2005): •  Aristotelian Arc

•  Fabulist (Riedl, 2004): •  Motivation (for character actions)

Thought Flow • Avoids fragmented stories:

•  Intuition: stories where one event is related to the previous one are better than stories where one event is disconnected in topic from the previous one.

•  Specifically: “Measures the percentage of events in the player’s experience related to the previous events”

• Associate each event in the game (plot point or beat) to a topic

• Given a sequence of events, thought flow is easy to measure

Thought Flow Example (Anchorhead) Plot Point Topic Thought Flow Get-safe-combo safe - Examine-album family - Read-basement-clippings family yes Read-bedroom-pages William - Open-safe safe - Get-crypt-key crypt - Find-williams-coffin crypt, William yes Get-skull crypt, William yes Get-library-book family -

Thought Flow(S) = 3/9

GraphicIFEngine (Project 2) •  Plot-points in the story given to you have already been

annotated for Thought Flow. For example: <plotpoint name="find-williams-coffin" plot="william" thought="william crypt">

<preconditions> <condition type="and">

<condition type="not"><condition type="plotpoint" name="find-williams-coffin"/></condition>

<condition type="plotpoint" name="get-crypt-key"/> </condition>

</preconditions> <trigger>

<condition type="action"> <action type="open" actor="player” object="crypt"/>

</condition> </trigger>

</plotpoint>

Activity Flow • Avoids the player moving from place to place:

•  Intuition: stories where each event happens in a different location (player has to constantly change location) are worst than if things have a spatial continuity.

•  Specifically: “Measures the percentage of events in the player’s experience that happen in the same location as the previous event”

• Given a sequence of events, activity flow is also easy to measure

• Avoids the player moving from place to place: •  Intuition: stories where each event happens in a different location

(player has to constantly change location) are worst than if things have a spatial continuity.

•  Specifically: “Measures the percentage of events in the player’s experience that happen in the same location as the previous event”

• Given a sequence of events, activity flow is also easy to measure

As mentioned before: Quality “in general” cannot be measured.

Many of you would disagree with this

measure. Activity flow was designed to capture the particular story aesthetics

that Weyhrauch appreciates.

Activity Flow

Activity Flow Example (Anchorhead) Plot Point Location Activity Flow Get-safe-combo bedroom - Examine-album basement - Read-basement-clippings basement yes Read-bedroom-pages bedroom - Open-safe bedroom yes Get-crypt-key bedroom yes Find-williams-coffin backyard - Get-skull backyard yes Get-library-book library

Activity Flow(S) = 4/9

Activity Flow Example (Anchorhead) Plot Point Location Activity

Flow Get-safe-combo bedroom -

Examine-album basement -

Read-basement-clippings

basement yes

Read-bedroom-pages

bedroom -

Open-safe bedroom yes

Get-crypt-key bedroom yes

Find-williams-coffin

backyard -

Get-skull backyard yes

Get-library-book library


Plot Point Location Activity Flow



basement yes

Get-safe-combo bedroom -

Read-bedroom-pages

bedroom yes




backyard -




Activity Flow Example (Anchorhead) Plot Point Location Activity

Flow Get-safe-combo bedroom -



basement yes

Read-bedroom-pages

bedroom -




backyard -




Plot Point Location Activity Flow



basement yes

Get-safe-combo bedroom -

Read-bedroom-pages

bedroom yes




backyard -




Even if “Activity Flow” is a weird measure in principle. In practice, it’s quite consistent with

common sense.

Sequence in the right makes more sense than the one on the left.

Since activity flow can capture that. The DM can guide the player to wards the sequence

in the right, and avoid the one on the left.

Options (Agency) • Ensures player feels in control of the outcome of the

game: •  Intuition: stories where there is more than “one right path” provide

the user more satisfaction than linear stories. Moreover, this is more important at the beginning of the game than towards the end.

•  Specifically: Measures the number of possible options the player has at any given point in time.

• Given a sequence of events, we need to know the set of

possible actions that affect the story that could have been executed at each given point in time.

Options Example • Could be measured in “actions” or in “plot-points” • Example for actions:

current state!

state! state! state!

state!

state! state!

action!

action! action!

action! action!action!

3 possible actions

1 possible action

2 possible action

Weight Actions 1 3

0.66 1 0.44 2

Options(S) = 4.54

The weight is useful to consider the number of actions at the beginning as more important

than towards the end.

Here I used 0.66^t for the weight (where t is time)

In GraphicIFEngine • Only relevant actions for the story should be considered •  There is a function that gives you this • You can see the percentage of those actions with satisfied

preconditions (except for location) at each instant of time

Motivation • Player should perform actions only when a motivation

exists: •  Intuition: some events in the game provide motivations for others.

The motivating events should happen earlier •  Specifically: Measures the number of times a motivating event

occurs before the motivated event.

• Given the list of plot points or beats, the authors needs to

provide the system with which ones motivate with others.

Motivation Example •  List of plot points with motivation relations

Plot Point Motivation Get-safe-combo - Examine-album - Read-basement-clippings - Read-bedroom-pages - Open-safe - Get-crypt-key no Find-williams-coffin - Get-skull - Get-library-book yes

In GraphicIFEngine • Motivations are not specified in the story provided to you •  If any one wants to work on this aspect, you will have to

introduce them by hand

Measurable Aspects •  Tea-For-Three (Weyhrauch, 1997):

•  Thought Flow: continuity in topics •  Activity Flow: continuity in space •  Options (Agency): player in control of the story •  Motivation: measure whether user actions have proper motivation •  Momentum: author specifies pairs of actions that should happen in

sequence •  Intensity: Aristotelian arc (complex!) •  Manipulation: assign a cost per each DM action, try to minimize it

•  Façade (Mateas & Stern, 2005): •  Aristotelian Arc (complex!)

•  Fabulist (Riedl, 2004): •  Motivation (for character actions): characters can only do motivated

actions




•  (Very brief Introduction to Machine Learning) •  Story Evaluation

•  Synthetic Approaches •  Machine Learning

•  Player Modeling •  SE and PM for Drama Management


Machine Learning Approaches to Story Quality • As we have seen, it is hard to capture quality as a

computer function

• Alternative option: •  Collect a large enough set of stories •  Ask humans to read them and rate them for quality •  Define a set of features of a story •  Learn an approximation of the function that maps stories to quality

using the examples provided by humans

Machine Learning Approaches to Story Quality • As we have seen, it is hard to capture quality as a

computer function

• Alternative option: •  Collect a large enough set of stories •  Ask humans to read them and rate them for quality •  Define a set of features of a story •  Learn an approximation of the function that maps stories to quality

using the examples provided by humans

This is the key element. Which features?

Story features •  Features that characterize a story:

•  Number of topics •  Number of actions •  Number of actions of each time (histogram) •  Percentage of unique actions •  Number of plot points •  Number of actions of each topic (topic as in “throught flow topic”) •  Any of the measures above (thought flow, activity flow, etc.)

Story features •  Features that characterize a story:

•  Number of topics •  Number of actions •  Number of actions of each time (histogram) •  Percentage of unique actions •  Number of plot points •  Number of actions of each topic (topic as in “throught flow topic”) •  Any of the measures above (thought flow, activity flow, etc.)

As you can see, not very good features

Machine Learning for Story Quality • Has been studied in the past (e.g. McIntyre 2011)

• Still an open problem

• Other approaches, such as Markov Models could be used, but, to the best of my knowledge, it’s an unexplored area.

Machine Learning for Story Quality • Has been studied in the past (e.g. McIntyre 2011)

• Still an open problem

• Other approaches, such as Markov Models could be used, but, to the best of my knowledge, it’s an unexplored area.

• Why did I explain this to you? points out the difficulty of dealing with “subjective” aspects like “story interestingness” or “quality” using formal approaches, like AI.

Project 2 •  If you decide to use “story evaluation” measures, I

recommend: •  Thought flow •  Activity flow •  Options

• Motivation is a better feature, but you’ll have to modify the stories.xml file

Player Models • Complementary to “Story Evaluation” in the context of

Drama Management

•  In this context, a player model is a computational function that predicts some aspect of the player. For example: •  A function that predicts the next action the player will execute •  A function that predicts how much will the player like a given beat •  A function that predicts if the player is getting bored •  etc.

Types of Player Models

Scope Purpose Domain Source

Individual Generative Actions

(Machine) Induced

Class (Human) Interpreted

Universal Descriptive Human Reactions

Analytic

Hypothetical Synthetic

(Smith, Lewis, Hullett, Smith, Sullivan, 2011)




(Machine) Induced



Analytic


Try to model a specific individual (e.g. the current player)

Try to model a class of individuals (e.g. “players that select being a

wizard in Skyrim”)

Try to model general aspects for every single player

Do not model any individual in particular, but are useful. For

example, in minimax, the opponent model assumes optimal play.




(Machine) Induced



Analytic


Generate data that could have been asked to the player. E.g.: did

you like event X? which action would have you executed in

situation Y?

Describe general patterns about the player: e.g. movement

patterns. Useful to understand the behavior of the player AFTER the

game




(Machine) Induced



Analytic


Predict player actions: which is the most likely action that will be executed in situation X? or a

probability distribution over the actions

Predicts aspects that are outside of the game world: boredom, interest,

fun, frustration, etc.




(Machine) Induced



Analytic


Models learned from data using machine learning techniques

Models generated by hand by humans, but based on data

Models obtained by optimizing some function. For example, to obtain a model

of optimal game play.

Models created by hand based on established theories or on personal

impressions.




(Machine) Induced



Analytic





(Machine) Induced



Analytic


Example of analytic player model: bastet (Bastard Tetris)

Generative Human Reactions Synthetic Models



(Machine) Induced



Analytic


• Story Evaluation as Player Modeling: •  Measures such as “Through Flow”, “Options”, etc. can be seen as

Generative, Human Reactions, Synthetic player models

Generative Action Synthetic Models



(Machine) Induced



Analytic


Generative Action Synthetic Models • Probabilistic rule-based models

• Probabilistic rule: P(a|c) •  Probability distribution of the actions given a condition c

•  For example, probabilities of actions given last one was “move”: •  P(examine|move) = 0.6 •  P(move|move) = 0.2 •  P(take|move) = 0.1 •  P(use|move) = 0.05 •  P(talk|move) = 0.05

Generative Action Synthetic Models • Player action model used in the game Haunt 2 (Magerko

2006):

Figure 17, the model used for Haunt 2 is composed of both an operator hierarchy very

similar to the hierarchy used to define the HauntBot agents and the hypothesized

knowledge base of the player. This hypothesized knowledge base is the main source

of information that the director has about the individual currently playing the game;

the player model, as currently designed, is a general model of player behavior. The

operators used in the Haunt 2 player are a subset of those used by the HauntBots for

autonomous behavior in the game. Operators do not decompose into atomic actions

(e.g. move forward one step), but instead only decompose into executable abstract

actions, such as move-to-area. The player is modeled to instantaneously move to the

target area with some associated cost in time when move-to-area is selected and

applied. This is a rudimentary player model used to show the benefits of a predictive

director. Future work entails research on defining more rigorous and adaptive

models.

Figure 17. Probabilistic model of player behavior in Haunt 2.

Model operators are run on the simulated environment just as if the director

agent were proposing, selecting and applying operators in the real world (as the

player). When choosing which model operator should be selected at a given time, the

director chooses one of the operators that are applicable in the current situation based

on probabilistic control knowledge per the Soar decision cycle described in Chapter

80

Actions vs Human Reactions Models in Drama Management

•  In the context of search-based DM (Drama management based on game-tree search):

•  Actions model: can be used to predict which branches have higher probability of occurring

•  Human Reactions model: can be used as the evaluation function

Actions vs Human Reactions Models in Drama Management • Expectimax algorithm:

•  Base algorithm assumes all player actions equally likely •  Better player model can help predicting things like the effect of

hints current state!

state! state! state!

state!

state! state!

actiona!

action2!

action3!

DM!action!

DM!action!

DM action!

state!

state! state!

actionb!

action2!

action3!

state!

state! state!

actionc!

action2!

action3!

Hint at action 2

0.5 0.5 0.5 0.5 0.25 0.75

Class Generative Action Synthetic Models



(Machine) Induced



Analytic


Class Generative Action Synthetic Models • Game Engine will have a collection of player models

•  One action model per class of player: •  Wizard •  Warrior •  Thief •  etc.

• At run-time, the game will use one or the other, according to the choices of the player

Machine Learning for Player Modeling



(Machine) Induced



Analytic


Machine Learning for Player Modeling •  Learn to predict actions/human reactions of the player at

hand from data

•  Two main issues: 1.  How to collect data 2.  How to learn a model from the data

Collecting Data for Player Modeling • Human subjects play the game (with no drama manager)

•  For each subject playing the game collect: 1.  Background information: age, gender, gaming experience 2.  Action trace 3.  Numerical questionnaire:

1.  For each event in the game (plot point/beat/room/object) how much they enjoyed it

2.  At each point in time during their game play: how bored/excited/frustrated they were

Collecting Data for Player Modeling • Human subjects play the game (with no drama manager)

•  For each subject playing the game collect: 1.  Background information: age, gender, gaming experience 2.  Action trace 3.  Numerical questionnaire:

1.  For each event in the game (plot point/beat/room/object) how much they enjoyed it

2.  At each point in time during their game play: how bored/excited/frustrated they were

Important for learning action model

Important for learning human reactions model

Learning a Player Model •  (Recall intro to machine learning) Supervised machine

learning can learn functions: f(x) = y •  What is X and Y here?

• Y is what we want to predict (actions / human reactions) • X is whatever we can observe from the player: list of

actions executed till now (there is nothing else we can observe!)

Learning an Action Model • Available data (collected):

• Current player:

Actions: A1 A2 An event1: hate event2: like

… eventm: - Boredom: low low

background low

Actions: B1 B1 B3 What is B4 going to be?

Learning an Action Model • Available data (collected):




background low

Actions: B1 B1 B3 What is B4 going to be?

Only relevant information

• Examples:

• X are traces of actions. For applying standard supervised learning, we need to represent them using a fixed set of numerical features: •  Histogram of action types (percentage of moves, examines, etc.) •  Percentage of unique actions •  Last action type •  etc.

Y:

Learning an Action Model

X: A1

X: A1 A2 A3

Y: A2

Learning an Human Reaction Model • Available data (collected):




background low

Actions: B1 B1 B3 How much will the player like event2?

• Examples:

• X are traces of actions (as before, we need to turn them into a fixed set of features).

• Y is how much the player likes or dislikes the target event. • We can learn a model for each event, or have a generic

model that learn all at once (next slide)

Y: hate

Learning an Human Reaction Model

X: A1

X: A1 A2

Y: like A2 An

An

X,Y"X,Y"

X,Y" X,Y"

like dislike

indifferent

pp1 pp2

ppn

Player"Trace"

X : action"trace"

Nearest"Neighbor"

Player Model""

0.8 0.2

0.5

pp1 pp2

ppn

Learning an Human Reaction Model • Nearest neighbor

•  Instead of the closest neighbor, we can take the average of the k closest

• Player model will predict the average interest of the k players with traces most similar to the current one

Passage (Thue)

Project 2 •  If anyone is interested in player modeling, I suggest:

•  Synthetic models easier for this class because: •  1) They don’t need to be very accurate •  2) For Machine learning you need to collect data

•  Use a synthetic action model to guide the expectimax algorithm •  At most, have several synthetic action models, and ask the player

some questions at the beginning (e.g. gamer/non gamer) to choose which model to use

Drama Management

Game Engine

Drama Manager

Player

Game Adaptation

Evaluation Function Narrative

Drama Management

Game Engine

Drama Manager

Player

Game Adaptation

Player Model Narrative

Drama Management

Game Engine

Drama Manager

Player

Game Adaptation

Player Model Narrative

In summary, Drama management is: •  Narrative representation •  Modeling the current player •  Adaptation of the game using a player

model

Project Discussion • Project 2 ideas:

•  If anyone is interested in story evaluation and player modeling, and there is no time to build a full drama manager a solution is:

•  Focus on building a player model, but do not incorporate it into the drama manager

•  For example: after playing a game, provide an analysis of the resulting story (thought flow, motivation, etc.)

•  If you do this, the story evaluation/player component should be quite strong (to compensate the fact that you will not create a drama manager)

Project Discussion • Project 1:

•  Questions? Technical Problems? Assistance?

• Project 2: •  Questions? Technical Problems? Assistance?

cs 680: game aisanti/teaching/2012/cs680/... · 2012. 2. 20. · reminders • progress self-check...

Documents