Download - Co-evolution, Games, and Social Behaviors
�� � � �� � �� �� �� � �� � �� � � �� �� � �� ����
��� � �� �! " #%$& '() " *+ $, " $ ( " -. /
012 � �3 �4 5 �6 798: ;< 01 � ;� 1 ; => 2 ; ?� �@ ;< A�: B �4 C� <6 � � D2 �6
E F DHG � A: � � = C� <6 � � D2 �6 CI JK > > = ?� �: ; FL � � D F �6
?M NO 2 : : 7O P P!Q Q QR 1 AR G 2 �6 R �1R 8S PUT V� �
W XYZ [\] ^Y_ _ `a YZ `b\ Y ] c\ bd e `f_ g `Z ch ] i
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
� � � � � � �� � � � � �� ��
IR �2 �: D �6 ; A
K R �2 B D �6 ; A
�R �2 B 1 ��� ;@ �3 8: � � �
�R 8: �6 �: � 1 6 � F 8 3 �< � A �: � � � � 4 3 ; �<� � � D A B A: ;6 A
JR > 2 ;! "#$ D �6 ;
%R & 8 3 : � 73 ;3 ;@ ;3 A �4 1 � � 7 ;< �: � � � �� F < ; 798: �: � � �
'R 5 � � 1 3 8 F� � D M ;6 �<S A
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
� � � � � ��
��
��
��
��
5 � � 7 ;< �: ;
$ ;4 ;1 :
5 � � 7 ;< �: ; $ ;4 ;1 :
�� �� � � �� � ��� ���� � �� � �� ��� � � ���� � � � � �� �� � !�" ��$# !% � � � � � � �� � �& �� �' � � � !
( ) * ),+ ) - � !� !�� � ! � � - . + . * . ( � " % + . / ( 0 - 12 � &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
�� � �� �� �� � � � � � � � �� � � �� � � � � � � �
$ �@ � A� �� 1 �6 6 ;� : ; F: 2 �:
> 2 ;� � 73 � B ;< 1 � A ; �! #$ /2 � A D < ; �: ;< D ;� ;< �3 �: B �� F � 7 73 � 1 �G �3 �: B: � < ; �3 � 3 �4 ;
A�: 8 �: � � � AR "� � F F�: � � � : � : 2 ; 7< � G 3 ;6 A �4 ;� ;< D B 1 � � A ;<@ �: � � � = ;1 �3 � D B = �� F
� @ ;< 7 � 798 3 �: � � � =6 �� B �: 2 ;< < ; �3 � 3 �4 ; 7< � G 3 ;6 A 1 �� G ; < ; 7< ; A ;� : ; F G B: 2 ;
! #$ 7 �< � F� D6 R
5 �3 6 �� = �� F� 3 �� 1 ; �� F 8 G ;<6 �� 2 �@ ; �3 A� � � F� 1 �: ; F: 2 �: : 2 ;! "# $ � A� � 8 �3 �: �: �@ ;3 B F�� ;< ;� : �4 < �6 : 2 ;K "#$ �� F: 2 �: �R R R 1 ;<: �� � A: < �: ; D� ; A: 2 �: Q � <S Q ;3 3
4 � < �� F�@ � F 8 �3 A � � : 2 ; # < � A� � ;<�� A$ �3 ;6 6 �4 ��3 � � 3 �< D ; D < � 8 7 AR �
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
� � �� � � � � � �
> 2 ;! "#$ D �6 ; 1 �� G ; F ;� � ; F G B: 2 ;4 �3 3 � Q � � D: 2 < ; ; 7< � 7 ;<: � ; AO
IR ; �1 2 73 � B ;< 4 �1 ; A: Q � 1 2 � � 1 ; A G ;: Q ; ;� 1 � � 7 ;< �: � � � � 5 / �� F F ;4 ;1: � � � �$ /�
K R : 2 ;$ � 7: � � � � A F �6 � � �� : 4 � < ; �1 2 73 � B ;< =�R ;R = ; �1 2 � A G ;: : ;< � � 1 2 � � A�� D$ : 2 ��
5 � � 6 �: : ;< 2 � Q 6 �� B � 4 : 2 ; �: 2 ;< 73 � B ;< A 1 2 � � A ; 5 �
�R : 2 ; F �6 � � �� : $ A: < �: ; D� ; A � � : ;< A ;1: � � � F ;� 1 � ;� : ; � 8 �3 � G < � 8 6 R "� 7 �<: � 1 8 3 �< =: 2 ;
� 8: 1 �6 ; �4 �3 3 73 � B ;< A 1 2 � � A ;: 2 ;� < � � � � F �6 � � �� : 5 A: < �: ; D� ; A � A 7< ;4 ;< �G 3 ;4 < �6
;@ ;< B 73 � B ;<�� A 7 � � � : �4 @ � ;Q : � : 2 ; � � ; �� Q 2 � 1 2 ;@ ;< B� � ; 1 2 � � A ; A$ = G 8: � � � � ;
� A6 �: �@ �: ; F: � F ;@ � �: ; 8 � �3 �: ;< �3 3 B4 < �6 $ R
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
� � �� � � � � � � � � ��� � � � ��
73 � B ;< $
5
� � ���
! 8 6 G ;< �4 1 � � 7 ;< �: � < A �6 � � D: 2 ; < ;6 �� � � � D � � 73 � B ;< A
�
���
���
� � �� � �
�� �� � �
�� �� � �� �� � ��� �� � � �� � �� � �� � � �� � � � � �� �� � !�" ��# !� � � � � � � �� � � ��� � �� �� � ��� � � � � �" ��� �" % �� � �" ! � !�� � !�� � !� �%� / 1 � . �� ��� � �� � �� � /� 1 � � � . �� " % �� � � . �� ��� � � � � � � � / � 1 �� . / � 0 ��� � 12 � ��� � � � � � � & �� �
��� ��� � �� � �� � ! ! � � � �� � � � ��� �� � � � � � � �� &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
� � �� � � � � � � � � � � � � � �
73 � B ;< $
5
� � ���
! 8 6 G ;< �4 1 � � 7 ;< �: � < A �6 � � D: 2 ; < ;6 �� � � � D � � 73 � B ;< A
�
��
��
� � �� � �
� � � � � � �
� � � � �
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
� � � � � � � � � � �
�> 2 ;! #$ 1 � < < ; A 7 � � F A: � �: < 8 3 B < ;6 �<S �G 3 ; < �� D ; �4 < ; �3 � Q � <3 F A� 1 � �3 7< � G 3 ;6 A = G 8:
�4 ;Q A�6 73 ; ; V �6 73 ; A Q �3 3 A8 � 1 ;R � � R 5 �3 6 �� =� �� � � ��� � � �� �� ��� � ��
� � � �� � # ;< D �6 � � # < ; A A =I �� K R � 7 7R I J '�� I J � /
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
� � � � � � � �� � � � �� �
�2 �: Q � 8 3 F B� 8 F � � A � < �: � � � �3 �� F�@ � F 8 �3 �
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
�� � � � � � �� � � � � � � � � �� � � � � � �
ER DR =: 2 ; ;� ;< D B 1 < � A� A =: 2 ; F < � 8 D2 : � � C < �: �� � � � : 2 ; A8 6 6 ;< �4 I � ' % = ;: 1R
� ": � A ;@ � F ;� : 3 B �� ; �1 2 � � F�@ � F 8 �3 � A < �: � � � �3 A ;3 4 � � � : ;< ; A: : � � D� � < ;: 2 ; 1 �3 3 4 � <
< ; A: < �� � : � < < ; A 7 ;1 : �@ ; �4 : 2 ; 1 2 � � 1 ; A �4 : 2 ; �: 2 ;< AR C 8: � �� F: 2 � A � A: 2 ; < 8 G
� �4 ;@ ;< B� � ; 78 < A8 ; A � � F�@ � F 8 �3 < �: � � � �3 �: B � � : 2 � A Q � B =: 2 ; B �< ; Q � < A ; � �
: 2 �� �4 ;@ ;< B� � ; � A6 �: �@ �: ; F G B 1 �3 3 ;1 : �@ ; < �: � � � �3 �: B �� F ; V ;< 1 � A ; A < ; A: < �� � : R
"4 ;@ ;< B� � ;: < � ; A: � G ; � �4 < ; ; < � F ;< �: 2 ;� � ��� � � ; D ;: A � < � F ; �: �3 3 R �
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
� � �� � � � � � � � � � � �� � � � �
�� < A: F� A1 8 A A ; F G B �< F� � � � I � %�R
� > 2 ;< ; �< ; A� V4 �<6 ;< A Q �: 2 � � ; 1 � Q ; �1 2 Q ;� D2 : ; F �: I � � �3 GR
� > 2 ; B A2 �< ; � 1 �6 6 � � 7 � A: 8 < ; Q 2 � 1 2 1 �� � � 3 B A8 A: �� � A� V 1 � Q AR
� E �12 � F F�: � � � �3 1 � Q Q �3 3 < ; F 8 1 ;: 2 ; Q ;� D2 : �4 ;@ ;< B 1 � Q G B I � �3 GR
� E �12 4 �<6 ;< � A �3 Q � B A G ;: : ;< � � : � 2 �@ ; �� � F F�: � � � �3 1 � Q < ; D �< F3 ; A A � 4 : 2 ;
12 � � 1 ; A � 4 �: 2 ;< 4 �<6 ;< AR
� "4 : 2 ; B �3 3 2 �@ ; � � ; � F F�: � � � �3 1 � Q =: 2 ; B ;� F 8 7 Q �: 2 : Q � � � �� 3 GR 1 � Q AR
0 8 1 2 : < � D ; F� ; A �< ;� �: 8 � 1 �6 6 � � � � � 8 < 2 � D2 3 B 1 �6 7 ;: �: �@ ; Q � <3 FR ": � A A8 D D ; A: ; F
: 2 �: �: 2 ; �6 7 � @ ;< � A2 6 ;� : � 4 A6 �3 3 4 �<6 ;< A � � E� D3 �� F F 8 < � � D: 2 ; 7 ;< � � F �4 : 2 ;
;� 1 3 � A8 < ; A � � : 2 ; ;� D2 : ; ;� : 2 1 ;� : 8 < B6 � B2 �@ ; G ; ;� ; V �1 ;< G �: ; F G B: 2 � A
72 ;� �6 ;� � � R �
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � �� � � �� � �� ��
IR 5 �� ;@ �3 8: � � � � 1 1 8 < A4 < ; � 8 ;� : 3 B � � ! �: 8 < ;R
K R 5 �� ;@ �3 8: � � � F � ; A� �: < ; � 8 � < ; �� B � V ; F �: � ; A A4 8 � 1 : � � � R
�R 5 �� ;@ �3 8: � � � 7< � @ � F ; A �� � 1 ; � 7 7< � �1 2 : � 3 ; �<� Q �: 2 � 8: ; V 7 ;<: S � � Q 3 ; F D ; �� F �
: ; �12 ;<R
A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 2
'
&
$
%
Introduction: Evolutionary Learning
Very straightforward conceptually!
1. Initialise population, X(t = 1).
2. Evaluate fitness of each population member.
3. Select parents from X(t) based on fitness.
4. Generate offspring from parents to obtain X(t + 1)
5. Repeat steps (2-4) until some termination criteria are met.
A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 3
'
&
$
%
Two Approaches to Evolutionary Learning
Things might get a little trickier.
Michigan Approach: Holland-style learning classifier systems (LCS), whereeach individual is a rule. The whole population is a complete (learning)system.
Pitt Approach: Each individual is a complete system.
This talk deals only with the Pitt-style evolutionary learning since it is morewidely used.
A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 4
'
&
$
%
Current Practice in Evolutionary Learning
fitness evaluation and selection
"genetic" operators
mutation
. . . . . .
a population of individuals(learning systems, e.g., ANNs orrule−based systems)
best individual
Pitt Style Evolutionary Learning
crossover
Figure 1: A general framework for Pitt style evolutionary learning.
A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 5
'
&
$
%
Fitness Evaluation
1. Based on the training error.
2. Based on the training error and complexity (regularisation), e.g.,
1fitness
∝ error + α ∗ complexity
A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 6
'
&
$
%
What If No Error Function Is Available
1. Or, we don’t know how to obtain the fitness function required to evaluatethe fitness of a population member, e.g.,, if we want to evolve game-playingstrategies.
2. In other words, the exact teacher/target information is unavailable.
A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 7
'
&
$
%
Well ... We have Co-evolutionary Learning
1. Initialise population, X(t = 1).
2. Evaluate fitness through interactions between population members.
3. Select parents from X(t) based on fitness.
4. Generate offspring from parents to obtain X(t + 1)
5. Repeat steps (2-4) until some termination criteria are met.
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
# � ' �#
� �" �� �� � �" ! /� � � � !� � � ! 1
������������ ����
��������
������
��� ���
��� �!"�$#
� � �� � �� � � � � � �� ��� &�
�� & %
� & &
� & �
� &��
& %
& &
& �
�� � � � �� ' � ! �� ! �� � � � �� " ! � �� � � � ( �� � �� � � � !� �� � � �" �� � � &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� �� � � � � �� � � � � � � � � � � �
IR 5 � � 7 ;< �: �@ ; G ;2 �@ � � 8 < A 1 �� G ; ;@ �3 @ ; F4 < �6 � 7 � 798 3 �: � � � �4 < �� F �6 A: < �: ; D� ; A
Q �: 2 � 8: A 7 ;1 �4 B� � D �� ; V 73 � 1 �: �� F � V ; F �: � ; A A4 8 � 1 : � � � R
K R 1 � � 7 ;< �: �@ ; 7 � 798 3 �: � � � 1 �� G ;@ 8 3 � ;< �G 3 ;: � F ;4 ;1 : �@ ; � � : < 8 F ;< AR
�R 5 �� ;@ �3 8: � � � �< B3 ; �<� � � D6 � B� �: G ; �G 3 ;: � 7< � F 8 1 ; A: < �: ; D� ; A Q 2 � 1 2 D ;� ;< �3 � A ;
Q ;3 3 � D �� � A: 8 � A ; ;� � 7 7 � � ;� : AR
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � � � �� �� � � � �� � � � � �� � � � � � � �
IR E@ �3 @ � � D � A: < �: ; D B: 2 �: � A 1 � 7 �G 3 ; �4 F ; �3 � � D Q �: 2 �3 3 S � � F A �4 � 7 7 � � ;� : A 1 �� G ;
2 �< FR
K R & � BG ; Q ; 1 �� 8 A ; �: ; �6 �4 � � F�@ � F 8 �3 A � A � A: < �: ; D B A� : 2 �: ; �1 2 � � F�@ � F 8 �3 � A
� � 3 B F ; �3 �� D � � � ; � < : Q � : B 7 ; A �4 � 7 7 � � ;� : AR 0 8 1 2 � � F�@ � F 8 �3 A A2 � 8 3 F G ; � 8 �: ;
; � A B: � ;@ �3 @ ;R
�R C 8: 2 � Q : � ;@ �3 @ ; �: ; �6 �4 � � F�@ � F 8 �3 A Q 2 � �< ; 1 �6 73 ;6 ;� : �4 ; �1 2 �: 2 ;< �
�R <: � � 1 � �3 A 7 ;1 � �: � � � � 7 7 ; �< A: � G ; �� ;� ;1 : �@ ;: ;1 2 � � � 8 ;: � �12 � ;@ ;: 2 �: R
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� �� � � � � � � � � � � � �� � �� � � � � � � � � � � � �� �
IR D ;� ;: � 1 �3 D � < �: 2 6 Q � A 8 A ; F: � ;@ �3 @ ; A: < �: ; D� ; A4 � < 73 � B� � D: 2 ;K "#$ R
K R "6 73 � 1 �: �: � ; A A A2 �< � � D Q � A 8 A ; F: � 4 � <6 F�� ;< ;� : A 7 ;1 � ; A � A 7 ;1 � �3 � A: A /R
�R D �: � � D �3 D � < �: 2 6 Q � A 8 A ; F: � F ;1 � F ; Q 2 � 1 2 A 7 ;1 � ; A A2 � 8 3 F < ; A 7 � � F: � ��
8 � S � � Q � � 7 7 � � ;� : R
�R E V 7 ;< �6 ;� : �3 < ; A8 3 : A A2 � Q ; F: 2 �: � 8 < 6 ;: 2 � F Q � <S ; F@ ;< B Q ;3 3 R
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� �� � � � � �� � �� � � � � � � � � � � � � � �� � �� � �
quality strategythe opponent uses, and uses the best
Finds which high-
counter-strategy.
GateStrategy 3
Strategy 3
Module 1
Module 2
Module 3
counter-strategies
Find best
Strategy 1
Strategy 2
speciated GAspecies fromExpert-level
Opponent uses
strategy. an expert-level
�� �� � �� � ��� �" � � � ��� �� � � ��� � � �� �" �" � % �� �� �" � �� � �� � � ! � �" � � � � � �� � �"� � � �� � �� &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � �� � � � � � � � �� �
� � �
0: < �: ; D B � � � A > � ; A @ ;< � D ; 01 � < ;
� � � Q � �: 2 ;<�� A
G ; A: R A< �R � % � �R � J � IR �K K IR JI �
D �: ;R A< �R % � � �R � J � IR JK � IR K � �
� �� � � � ' � ! �� ! � �� �" !� � � �� � !� � � !� � �� � �� � ! �� � � � � � ���� � � � � �" � �� �� � ' � ! ��� � � ���� � ( �� � �� � � � � � �� �� �% � � !� �� � � � � & �� �� � ! �� ! � �� � � ' �� � � �% � ' �� � �� " !&
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � � �� � � � � � � � � �� � � � � � �� � � � � � � � � � �
IR & �S �� D � � F�@ � F 8 �3 A F�� ;< ;� : O
� � �
��
� ��� � /�� � / � 1� / � 1 1 � 0 � � / � 1 �
Q 2 ;< ;
� � / � 1 � / � � / � 1� � / � 1 1 �� � /�� / � 1� � / � 1 1
� / � 1 � A: 2 ; ;� A ;6 G 3 ; � 8: 798: R
K R 3 3 � � F�@ � F 8 �3 A �< ;3 ; �<� ; F� �� � �� �� �� R
�R � < � F� ;� : F ; A1 ;� : �3 D � < �: 2 6 A = A8 1 2 � A C # =1 �� G ; 8 A ; FR
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � � �� �� � � � � � � � � � � � � � � � � �� �
IR �2 �: � A: 2 ; �6 7 �1 : � 4 2 � A: � < B3 ;� D: 2 �6 ;6 � < B / � � : 2 ; ;@ �3 8: � � � �
K R �2 �: � A: 2 ; �6 7 �1 : � 4 D < � 8 7 A�� ; � � : 2 ; ;@ �3 8: � � � �
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
� � � � � �� � � �� � � � � � � �� � �
1
1.5
2
2.5
3
3.5
4
0 100 200 300 400 500 600 700 800 900 1000
Ave
rage
pay
off
Generation
3 prisoners, history 2
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � �� � � �� � � � � � � �� � �
1.5
2
2.5
3
3.5
4
0 100 200 300 400 500 600 700 800 900 1000
Ave
rage
pay
off
Generation
3 prisoners, history 3
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � �� � � �� � � � � � � �� � �
2.2
2.4
2.6
2.8
3
3.2
3.4
3.6
3.8
4
0 100 200 300 400 500 600 700 800 900 1000
Ave
rage
pay
off
Generation
3 prisoners, history 4
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � �� � � �� � � � � �
&�
& �
& &
& %�
� � � � � & � % � � � � � � � & � % � � � �
� ' �� � � � ��� ����
� �" �� �� � �" !
� � � � � �� ! �� � !� �� � �# � � !&� � � �#
�� �� � �� � �� � � � � ( �� � �� � � � !� �� � � �� � � � �� �� � �" � � �� � � ! � � !� � �� � �� � � � &
�" � � � � � � � �� " ! �� � � � � � �� � � � � � � � � � �� �� � �" �� " % �" � � � " �� � !� � � � � � !�� � � % � � �� � � �" & �� � � �� � ��" � � � � �" � ! �� � � �� � � !� � � � � � � � � � � � � � � " % % � � � �' �� !� �� � � � �� �� � �" &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � �� � � �� � � � � �
�� & �
�� & �
�� & �
�� & �
&� � � � � & � % � � � � � � � & � % � � � �
� ' �� � � � ��� ����
� �" �� �� � �"
� � � � � �� ! �� � !� �� � � / �� � �� ! 1
� � !& � � � �
�� �� � &� � �� � � � � ( �� � �� � � � !� �� � � �� � � � �� �� � �" � � �� � ! � � !� � � � �� � �� � � � &�� � � �� � ��" � � � � �" � ! �� � � �� � � !� � � � � � � � � � � � � �� " % % � � � �' �� ! � �� � � � �� �� � �" &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � �� � � �� � � � � �
��
&�
%�
� �
� �� � � � � & � % � � � � � � � & � % � � � �
� ' �� � � � ��� ����
� �" �� �� � �" !
% � � � � �� ! �� � !� �� � � /� &� �� ! 1� � !& % � � �
% � �
� � �
� � �
�� �� � �� � �� � � � % ( �� � �� � � � !� �� � � �� � � � �� �� � �" " �' �� � � �� � � !& �� � � �� � ��" � � �� �" � ! �� � � �� � � !� � � � � � � � � � � � � �� " % % � � � �' �� ! � �� � � � �� �� � �" & � � % � � �" !� � �� �� � �� � � � ! �� " !� � ' � � �" ' �� � �% � �� �� � � !� � � !� � �� � � ! � �� � � � � � � �� � �" !&
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � � �� � � �� � � � � �
� �
& %
� �� �
� �
� &� %
� �� � � � � & � % � � � � � � � & � % � � � �
� ' �� � � � ��� ����
� �" �� �� � �"
& � � � � �� ! �� � !� �� � �# � � !& & � � �#
� � �
� � �
% � �
�� �� � %� � �� � � � & ( �� � �� � � � !� �� � � �� � � � �� �� � �" " �' �� � � �� � � !& �� �� �� � ��" � � �� �" � ! �� � � �� � � !� � � � � � � � � � � � � ��� " % % � � � �' �� ! � �� � � � �� �� � �" &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � �� � � � � �� � � � �� � � � � �
� & �
� & � �� & &
� & & �� & �
� & � �� & %
� & % �� � � � � & � % � � � � � � � & � % � � � �
� � � !
� �" �� �� � �"
� � � ! �" � � � � �� � �" ! � � � ��" � % ( �� � �� � � � !� �� � �
� � !& % � � �
�� �� � �� (" �� " ! � �� � � % ( �� � �� � � � !� �� � � �� � �� �� � � � �� �� � �" " �' �� � � �� � � ! �� � �� � � ! % � � �" !� � �� � !� � �� � � � � � � � �� � �" ! � � ' � � �" ' �� � �% & � � � ! � !� � � � ' �� � � �
�� � � �� � � �" � �� � � � � !� �� � � �" �" � ' � � � �" �� � � � � !�� � �" & �� � � ! � � � & � � � �� " !� � �� � �" � ' �� � � � � �� � �� �� � � !�� � �" � � !� �" ' �� � �% � � ��� � �� � � �
� ��� �� � � �
� � &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� �� � � � � �� � � � � � � � � � � � �
IR & ;6 � < B2 ;3 7 A3 ; �<� � � DR N � � D2 � A: � < B3 ;� D: 2 A ;� 1 � 8 < � D ; 1 � � 7 ;< �: � � � R
K R N �< D ; D < � 8 7 A6 �S ;: 2 ; ;@ �3 8: � � � �4 1 � � 7 ;< �: �@ ; A: < �: ; D� ; A6 � < ; F� � 1 8 3 : R
�� �� �� � =
IR � ; < �< ;3 B 1 �6 6 �: � 8 < A ;3 @ ; A: � � � ; Q � B � < : 2 ; �: 2 ;<R � ; 7< ;4 ;< : � G ; A�6 ;Q 2 ;< ;
�� : 2 ;6 � F F3 ;R
K R � ; �< ;6 � < ;3 �S ;3 B: � 2 �@ ; A2 � <: D �6 ; A Q �: 2 A2 � <: 6 ;6 � < BR
�R � ;6 � B� �: G ; �G 3 ;: � � � : ;< �1 : Q �: 2 ;@ ;< B� � ; �3 : 2 � 8 D2 Q ;S � � Q : 2 ;6 G B� �6 ; A
�< ; 798: �: � � � /R
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � �� � � � � � � � � �� � � �� � � � � � ��
� � �� 0 �� 0 � � �� � �� �� �� �� � � �� �� �
0 �� �� �� � ���
0 � �� � �� �
�� �� � �� �� � ��� ��� � �� � �� ��� � � � ��� � � � � �� �� � !�" ��$# ! % � � � � � � �� � � �� �� � �� �� � � � � � ! � �� � � � �� �� � �" &
> 2 ; 7 � B� � : � 73 � B ;< � A D�@ ;� G BO
�� � � � �� � � �� � 0 � � � ) /� � � � )� � � 1 / 1
Q 2 ;< ; � � �� F � � �< ;: 2 ; 1 � � 7 ;< �: � � � 3 ;@ ;3 A � 4 : 2 ;: Q � 73 � B ;< A = Q 2 � 1 2 �< ; F� A1 < ;: � � ; F
� � : � 4 � 8 < 1 2 � � 1 ; A �4 1 � � 7 ;< �: � � � R
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
� � � �� � ��
� "� 4 � <6 �: � � � �G � 8: � D �6 ; ��R ;R =� � : ;< �1 : � � � � A6 � F ; �@ ��3 �G 3 ;: � �3 3 �: 2 ;< AR � � 8
1 �� A ; ; A�6 ;� � ;� A < ; 78: �: � � � Q �: 2 � 8: � � : ;< �1 : �� D Q �: 2 2 �6 P2 ;< F� < ;1 : 3 BR
� �� < A: 1 � � A� F ;< ; F G B ! � Q �S �� F 0� D6 8 � FR
� C 8: � � 3 B � � K � 73 � B ;< "# $ D �6 ; A Q �: 2 : Q � 3 ;@ ;3 A �4 1 � � 7 ;< �: � � � R
� ! � ;@ �3 8: � � � �< B �3 D � < �: 2 6 Q � A 8 A ; F � � : 2 ;� < Q � <S R #3 � B ;< A F� F� �: ;@ �3 @ ; �� : 2 ;� <
1 � A ;R
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � � � � � � � ��
�2 �: � A: 2 ; �6 7 �1 : � 4 6 8 3 : � 73 ;3 ;@ ;3 A �4 1 � � 7 ;< �: � � � � � < ; 798: �: � � � � � : 2 ; ;@ �3 8: � � �
�4 1 � � 7 ;< �: � � � �
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � �� � � � � �� � �
� E �12 A: < �: ; D B � A < ; 7< ; A ;� : ; F G B �: Q ��� 3 � B ;< 4 ; ; F� 4 � < Q �< F� ;8 < �3 � ;: Q � <S Q �: 2 K �
2 � F F ;� � � F ; AR
� > 2 ;< ; � A � � ; � 8: 798: � � F� 1 �: � � D: 2 ; 1 8 < < ;� : �1 : � � � : �S ;� G B: 2 ; 73 � B ;<R
� > 2 ;< ; �< ; � @ ; � � 78: AO �I /: 2 ; 73 � B ;<�� A � Q � 7< ;@ � � 8 A6 � @ ; � �K /: 2 ; � 7 7 � � ;� : � A
7< ;@ � � 8 A6 � @ ; � � � / Q 2 ;: 2 ;< : 2 ; � 7 7 � � ;� : ; V 73 � �: ; F: 2 ; 73 � B ;< � � � / Q 2 ;: 2 ;< : 2 ;
73 � B ;< ; V 73 � �: ; F: 2 ; � 7 7 � � ;� : � �� F � J / < ; 798: �: � � � �4 : 2 ; � 7 7 � � ;� : R
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � �� � � � � � � � �� �� � � � � �� � � � � � �� �� � �� � � � �� �
1
1.5
2
2.5
3
3.5
4
0 50 100 150 200 250 300
Ave
rage
pay
off
Generation
No reputation, pop 100, 10 iterations per game, 2 levels of coop
’p100it10c2norep.a/pop.av’ u 3:9’p100it10c2norep.b/pop.av’ u 3:9’p100it10c2norep.c/pop.av’ u 3:9’p100it10c2norep.d/pop.av’ u 3:9’p100it10c2norep.e/pop.av’ u 3:9’p100it10c2norep.f/pop.av’ u 3:9’p100it10c2norep.g/pop.av’ u 3:9’p100it10c2norep.h/pop.av’ u 3:9’p100it10c2norep.i/pop.av’ u 3:9’p100it10c2norep.j/pop.av’ u 3:9
�� �� � � � � � � � �" �� � � ! �" � � � �� �� �� � �" ! � !� � � �� � � ! � � ! !�" � �" � � ' �� � � � � � �� �� �
� � � " ��� �� � � ! � � � � �� �� �� � �" !&
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � �� � � � � �� � �� �� � � � � �� � � � � � �� �� �� � � � �� �
1
1.5
2
2.5
3
3.5
4
0 50 100 150 200 250 300
Ave
rage
pay
off
Generation
No reputation, pop 100, 5 iterations per game, 2 levels of coop
’norep100it5c2.a/pop.av’ u 3:9’norep100it5c2.b/pop.av’ u 3:9’norep100it5c2.c/pop.av’ u 3:9’norep100it5c2.d/pop.av’ u 3:9’norep100it5c2.e/pop.av’ u 3:9’norep100it5c2.f/pop.av’ u 3:9’norep100it5c2.g/pop.av’ u 3:9’norep100it5c2.h/pop.av’ u 3:9’norep100it5c2.i/pop.av’ u 3:9’norep100it5c2.j/pop.av’ u 3:9
�� �� � �� � � � � � �" �� � � ! �" � � � �� �� �� � �" ! � !� � � �� � � ! � � ! ! �" � �" � � ' �� � � � � � �� �� �&
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � �� � �� � � � � � �� � � � � � �� �� � �� � � � �� �
1
1.5
2
2.5
3
3.5
4
0 50 100 150 200 250 300
Ave
rage
pay
off
Generation
Reputation, pop 100, 10 iterations per game, 2 levels of coop
’p100it10c2rep.a/pop.av’ u 3:9’p100it10c2rep.b/pop.av’ u 3:9’p100it10c2rep.c/pop.av’ u 3:9’p100it10c2rep.d/pop.av’ u 3:9’p100it10c2rep.e/pop.av’ u 3:9’p100it10c2rep.f/pop.av’ u 3:9
’p100it10c2rep.g/pop.av’ u 3:9’p100it10c2rep.h/pop.av’ u 3:9’p100it10c2rep.i/pop.av’ u 3:9’p100it10c2rep.j/pop.av’ u 3:9
�� �� � �� � � � � � �" �� � � ! �" � � � �� �� �� � �" ! � !� � � �� � � ! � � ! !�" � �" � � ' �� � � � � � �� �� � �� � � � � � �� � �" � � � � !� � � � �� �� � �" � �� � � � � �� � &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � �� � �� � � � � � �� � � � � � �� �� �� � � � ��
1
1.5
2
2.5
3
3.5
0 50 100 150 200 250 300
Ave
rage
pay
off
Generation
Reputation, pop 100, 1 iteration per game, 2 levels of coop
/home/darwen/src/nipd/runs/p100it1c2rep
�� �� � �� � � � � � �" �� � � ! �" � � �� �� �� � �" � !� � � �� � � ! � � ! ! �" � �" � � ' �� � � � � � �� �� �&
� � � � � � �� � �" � � � � ! �� � !� � �� � �� �� � � � � � � ' � � �� � �" &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � � � � � � � � � �� � � � � � �� � �� �� � � � � � �� � � � � � ��
1
1.2
1.4
1.6
1.8
2
2.2
2.4
2.6
2.8
0 50 100 150 200 250 300
Ave
rage
pay
off
Generation
No reputation, pop 150, 20 iterations per game, 8 levels of coop
’p150it20c8norep.b/pop.av’ u 3:9’p150it20c8norep.c/pop.av’ u 3:9’p150it20c8norep.d/pop.av’ u 3:9’p150it20c8norep.e/pop.av’ u 3:9’p150it20c8norep.f/pop.av’ u 3:9’p150it20c8norep.g/pop.av’ u 3:9’p150it20c8norep.h/pop.av’ u 3:9’p150it20c8norep.i/pop.av’ u 3:9’p150it20c8norep.j/pop.av’ u 3:9
�� �� � �� � � � � � �" �� � � !� � �� �� �� � �" ! � � �� � % � � � � � � ! � �� � � � �� �� � �" �� " % % � � ���� � �" % � � �" �� � !&
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � �� � �� � � � � � �� � � � � � �� � � �
1
1.5
2
2.5
3
3.5
4
0 50 100 150 200 250 300
Ave
rage
pay
off
Generation
Reputation, pop 150, 20 iterations per game, 8 levels of coop
’p150it20c8rep.b/pop.av’ u 3:9’p150it20c8rep.c/pop.av’ u 3:9’p150it20c8rep.d/pop.av’ u 3:9’p150it20c8rep.e/pop.av’ u 3:9’p150it20c8rep.f/pop.av’ u 3:9’p150it20c8rep.g/pop.av’ u 3:9’p150it20c8rep.h/pop.av’ u 3:9’p150it20c8rep.i/pop.av’ u 3:9’p150it20c8rep.j/pop.av’ u 3:9
�� �� � &� � � � � � �" �� � � ! � � �� �� �� � �" ! � � �� � % � � � � � � ! � � � � � � �� �� � �" & � �� �� � � � �� � �" �� � � � �� �� � �" � ! � � � � �� � � � � �� � &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � �� � �� �� �� �� �� �
E@ ;� Q 2 ;� : 2 ;� 8 6 G ;< �4 1 � � 7 ;< �: � � � 3 ;@ ;3 A � A2 � D2 R
1
1.5
2
2.5
3
3.5
4
0 50 100 150 200 250 300
Ave
rage
pay
off
Generation
Reputation, pop 150, 10 iterations, 64 levels of coop
’p150it10c64rep.a/pop.av’ u 3:9’p150it10c64rep.b/pop.av’ u 3:9’p150it10c64rep.c/pop.av’ u 3:9’p150it10c64rep.d/pop.av’ u 3:9’p150it10c64rep.e/pop.av’ u 3:9’p150it10c64rep.f/pop.av’ u 3:9’p150it10c64rep.g/pop.av’ u 3:9’p150it10c64rep.h/pop.av’ u 3:9’p150it10c64rep.i/pop.av’ u 3:9’p150it10c64rep.j/pop.av’ u 3:9
�� �� � �� � � � � � �" �� � � ! � �� �� �� � �" ! � � �� � �� �� �� �� � � �� � � � �� �� � �" & � �� �� � � � �� � �" �� � � � �� �� � �" � ! !� � � � � � � �� � &
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � �
�� � � � � �� �
IR "#$ D �6 ; A �< ; 8 A ;4 8 3 6 � F ;3 A � 4 < ; �3 � 3 �4 ; A�: 8 �: � � � AR
K R ! "#$ D �6 ; A �< ;6 � < ; < ; �3 � A: � 1 �� F D ;� ;< �3 : 2 �� : 2 ;K "#$ � � ;R
�R & 8 3 : � 73 ;3 ;@ ;3 A �4 1 � � 7 ;< �: � � � F� A1 � 8 < � D ; 1 � � 7 ;< �: � � � R
�R M ; 798: �: � � � 1 �� 6 �: � D �: ;: 2 � A �� F ;� 1 � 8 < � D ; 1 � � 7 ;< �: � � � R
JR M ; 798: �: � � � 1 �� ;� 1 � 8 < � D ; 1 � � 7 ;< �: � � � ;@ ;� Q 2 ;� : 2 ; D �6 ; �� F2 � A: � < B3 ;� D: 2 A
�< ; ; V: < ;6 ;3 B A2 � <: R
%R "#$ D �6 ; A Q �: 2 6 8 3 : � 73 ;3 ;@ ;3 A � 4 1 � � 7 ;< �: � � � � A Q ;3 3 � A < ; 798: �: � � � G ;2 �@ ;
� 8 �3 �: �: �@ ;3 B F�� ;< ;� : 3 B4 < �6 1 3 � A A� 1 �3 "#$ D �6 ; AR
'R 5 �� ;@ �3 8: � � � �< B3 ; �<� � � D � A �� ;� ;1: �@ ;3 ; �<� � � D: ;1 2 � � � 8 ;R
�� � �� � � �� �� � � � �� � � � � � �� � � �� � � �� �� � � �
� � � � � � � � � �� � �
IR # �3 �: � 1 �3 A B A: ;6 A P A�: 8 �: � � � A = ;R DR =1 �3 F Q �< =6 � A A�3 ; 1 < � A� A = ;: 1R
K R E1 � � �6 � 1 A B A: ;6 A = ;R DR = 8 A: < �3 � �� 7 ;: < �3 6 �<S ;: =� � : ;<� �: � � � �3 1 � � ; ;6 �<S ;: =
;: 1R�R C� �3 � D� 1 �3 �� F ;1 �3 � D� 1 �3 A B A: ;6 AR
�R 0 � 1 � �3 A B A: ;6 A = ;R DR = � 8 < F� � � ;< : � � � D2 : �
A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 9
'
&
$
%
A Simple Research Question
• If I invent a wonderful co-evolutionary learning algorithm and use it toco-evolve a really intelligent game-playing strategy (e.g., for chess, carracing, iterated prisoner’s dilemma, or others), how do I know it wouldperform well against a new opponent that it has never seen before?
• Can we say anything at all about the ability (performance) of our co-evolvedsolutions in a new and unseen environment?
• Sounds like generalisation to me.
A Rigorous Theoretical Framework for Measuring Generalisation of Co-evolutionary Learning 15
'
&
$
%
True Generalization Performance
1. Given a co-evolved strategy i, let test strategies j be obtained from strategyspace S. The true generalization performance of strategy i, Gi, is:
Gi = EP1(j)[Gi(j)] =∫
S
Gi(j)P1(j)dj, (1)
where Gi is the expectation of strategy i’s performance against j, Gi(j),w.r.t. distribution P1(j) over strategy space S.
2. A simplified form:
Gi =1M
M∑
j
Gi(j), (2)
which is simply its average performance against all strategies j.
More Information
• S. Y. Chong, P. Tino and X. Yao, “Measuring Generalization Performance in Co-evolutionary Learning,” IEEE Transactions on Evolutionary Computation, 12(4):479-505, August 2008. – IEEE Transactions on Evolutionary Computation
Outstanding 2008 Paper Award (bestowed in 2010).