E. Altman INRIA, France

Advances in Evolutionary Games

BioneticsBioneticsDcember 2010

Overview of the talk

1. Background on Evolutionary Games and Population Games and Examples

2. Adding time varying states

3. Adding controlled state transitions

4. Examples

5. Mathematical model

6. Computing Equilibria

EGs and PGs in Biology and Engineering

BIOLOGY CONTEXT: Central tool defined by Meynard Smith (1972) for explaining and predicting dynamics of large competing populations with many limited local interactions. EG.

TELECOM CONTEXT: Competition between protocols, technologies. Can be used to design and regulate evolution

ROAD TRAFFIC CONTEXT: Competition between cars over routes. Introduced by Wardrop (1952). PG.

Framework

Large population Several strategies (behavior of individuals). Call

all those who use a strategy a subpopulation EG: Competition between the strategies through

a very large number of interactions each involving a small number of individuals typically pairwize interactions

PG: interactions with an infinite number of players

Evolutionary Games: Definitions

Evolutionary Stable Strategy (ESS): At ESS, the populations are immuned from being invaded by other small populations (mutations).

ESS more robust than standard Nash equilibrium.

“State” vector X: fractions of users that belong to different populations. Or fractions of strategies in a population

Fitness: J(p,q):= utility when playing pure strategy p and all others play q. J(x,y) fitness when using mixed strategies x and y, resp.

Evolutionary Stable StrategyEvolutionary Stable Strategy (ESS):

p is ESS if for all q there is d(q) s.t. for all 0<e<d(q)

J(p,p) >= (1-e) J(q,p) + e J(q,q)

Equivalent condition:

J(p,p) > J(q,p) or

J(p,p) = J(q,p) and J(q,q) < J(p,q)

Ex 1: Hawk and Dove Game

Large population of animals. Occasionally two animal find themselves in competition on the same piece of food. An animal can adopt an aggressive behavior (Hawk) or a peaceful one (Dove).

D-D: peaceful, equal-sharing of the food. fitness of 0.5 to each player.

H-D or D-H: 0 fitness to D and 1 for H that gets all the food no fight

HD Game

H-H: fight in which with equal chances to obtain the food but also to be wounded. Then the fitness of each player is 0.5-d, -d is the expected loss of fitness due to being injured.

Modeling competition: Generalized HD Game

•Generalized game: A11<A22<A12 and A21<A22.

•Simple conditions for H to be unique ESS and for mixed ESS

Ex 2: Competition between protocols

There are various flow control protocols to regulate traffic in the Internet.

Huge number of file transfers every second Interactions occur between limited number of

connections that use the same bottleneck link The average speed of transfer, the delay etc

depend on the versions of the protocol involved in the interaction

Competition between protocols

Ex 3: Population Games (PG) in Wireless communications

• Cellular network contains many mobiles. One base station (BS) per cell

•CDMA: At each time an individual sends a packet it interacts with all mobiles in the same cell

•A mobile can transmit with different power levels q1 < … < qK. Higher power is more costly

•Objective of : max_k J(k,w) :=

where wk is the fraction of mobiles that use qk

Replicator Dynamics

Delayed case: present growth rate depends on past fitness

K and tau : design parameters. Determine speed of convergence and stability

Architecting evolution: impact of K

stability iff K tau< θ. Oscillations mean no convergence to ESS.

Individual States in EG and PG

Different behaviors may be a result of different inherent characteristics – individual states

Example: weather conditions, age, The individual state can be random Description through a Markov chain EG: Local interactions with players chosen at

random; their state is unknown PG: Global interactions, the state can be known

Indiv. states in HD Game

The decisions H or D determine whether a fight will occur

There is also a true identity -- Strong or Weak We call this the individual STATE

If there is a fight then the states determine the outcome.

Note: the decision H/D are taken without knowing the state of the other.

Indiv. States in Networks

Flow control protocol: large end to end delay slows the protocol and decreases its throughput

Wireless:

- the power received may depend on the radio channel conditions

- the transmitted power may depend on the energy level of the battery

MDEG: Markov Decision EG ASG: Anonymous Sequential G

Each player has a controlled Markov chain (MDP)

A player has finite or infinite life time. It has several interactions each time with another randomly selected player (MDEG) with a large population (ASG)

Each interaction results in an immediate fitness that depends on the actions and states of the players involved

The states and actions of a player determine also the probability distribution of the next state

Assumptions, References

A player maximizes the total expected or average fitness

EG average fitness: EA & YH IEEE trans Autom Contr, June 2010 (theory) EG total expected fitness: Infocom 2008 (power control) Evolutionary Ecology Research, 2009 (the theory) (EA, YH, R El-Azouzi, H. Tembine) SAG: Jovanovic & Rosenthal, J Math. Econ, 1988 (disc cost)

Assume: The transition probabilities of the MDP of a player depend only on its own actions and states

Ex 1 (MDEG): Hawk and Dove game

A bird that looses becomes weaker (less energy)

A very weak bird dies

State: Energy level

Would a weaker bird be more or less aggressive?

If the result of the fight are determined by the energy level then the transitions are determined by states and actions of both birds.

Ex 2: (MDEG or ASG): Battery dependent power control

Transmitting at higher power empties faster the battery

A battery with little energy left is not able to support transmissions at high power

The state: remaining energy in the battery

The transitions do not depend on other mobiles

Ex 3: channel dependent power control

The decision to transmit at power qk may depend on the channel state

Seems “degenerate”: the mobile does not control the transitions

Restriction: discrete power set; if a power level is chosen then the next power cannot differ by more than one unit.

This creates non-trivial transitions. The state = (Channel state, current power level)

MDEG: Local interactions

Each local interaction is described by a stochashtic game with partial monitoring

The stochastic game has an equilibrium. The game is equivalent to a matrix game where the pure actions of a player are its pure stationary policies

Allows us to transform the problem into a standard EG with a huge action space (action=pure policy)

We show: equivalence to a polytope game in the space of marginal stationary occupation measures Cardinality: no. of states times number of actions

Model of Individual player

Each player is associated with a MDP with

POWER CONTROL ASG

The model for an individual

•State of an individual corresponds to the battery level.

•Set of actions available at state s:

• Qs decreases with the energy: smaller powers are available when the battery has less energy

•Transitions: the probability to stay at a state s if q is used is

•Recharging: P0N is the probability to move from 0 to N

The model: Interactions

Global state: fraction of mobile in each individual state:

Proportion of mobiles using qk at time t is The Reward:

Stationary policy the ptob to choose qk in s

Interactions and System model

In stationary regime:

Interactions and System model

The expected reward:

A stationary policy u is an equilibrium if

Results (1)

Define the interference of u:

Denote the probability distribution with mass 1 at q

Results (2)

Threorem. An equilibrium exists within

Results (3)

Future work

Branching MDPs: a state-action pair of an individual determines the immediate fitness, the transition probabilities and the number of off-springs