40pta short introduction to coco · • bbob-noisytest suite with 30 functions (45 algorithms) •...
TRANSCRIPT
![Page 1: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/1.jpg)
A Short Introduction to COCO
Tea Tušar
Computational Intelligence GroupDepartment of Intelligent SystemsJožef Stefan InstituteLjubljana, Slovenia
July 15, 2018
Workshop on Game-Benchmark for Evolutionary AlgorithmsGenetic and Evolutionary Computation Conference, GECCO 2018Kyoto, Japan
1
![Page 2: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/2.jpg)
Why benchmark optimization algorithms?
No free lunch theorem ⇒ No algorithm works best for alloptimization problems
Purpose of benchmarking: To be able to select the best algorithmfor the given real-world optimization problem
Preconditions
• The real-world problem with some known properties• Test problems with similar properties to those of the real-worldproblem
• Results of several optimization algorithms on these testproblems for any number of evaluations
2
![Page 3: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/3.jpg)
How to benchmark optimization algorithms?
The COCO platform
• COCO (Comparing Continuous Optimizers)• https://github.com/numbbo/coco• Automatized benchmarking of optimization algorithms
• Test problems with known properties• Data of previously run algorithms available for comparison• Provides interfaces to C/C++, Python, Java, Matlab/Octave
• Being developed at Inria Saclay, France, since 2007
3
![Page 4: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/4.jpg)
Benchmarking with COCO
COCOexperiments
C
Results of youralgorithm
Log �les
COCOpostprocessing
Python
Results of otheralgorithms
Log �les
Data pro�les
0 2 4 6log10(# f-evals / dimension)
0.0
0.2
0.4
0.6
0.8
1.0
Frac
tion
of fu
nctio
n,ta
rget
pai
rs
RANDOMSEARF1-CMAESRL-SHADE-GP1-CMAESCGA-ring1BFGS-P-raMCS huyer1plus2mirsimplex pEDA-PSO eL-BFGS-B-CMAES posoPOEMS kuCMAES-APOALPS hornIP-500 liVNS garciNIPOPaCMADE posik best 2009bbob f1-f24, 5-D
51 targets: 100..1e-0815 instances
v2.2.1.417
Tables
C/C++interface
C
Pythoninterface
Python
Javainterface
Java
Matlab/Octaveinterface
Matlab
Your favoritealgorithm
Matlab
Requirements: C compiler and Python (other languages are optional)
4
![Page 5: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/5.jpg)
The fixed-target approach
Interested in the runtime (number of function evaluations) neededto achieve a target value
Runtime (number of function evaluations)
Qual
ity in
dica
tor (
to b
e m
inim
ized)
Fixe
d bu
dget
Fixed target
Convergence graph
5
![Page 6: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/6.jpg)
Data profile
The data profile is the empirical cumulative distribution function(ECDF) of the recorded runtimes
0 20 40 60 80 100Runtime (number of function evaluations)
Qual
ity in
dica
tor (
to b
e m
inim
ized)
Convergence graph
6
![Page 7: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/7.jpg)
Data profile
The data profile is the empirical cumulative distribution function(ECDF) of the recorded runtimes
0 20 40 60 80 100Runtime (number of function evaluations)
Qual
ity in
dica
tor (
to b
e m
inim
ized)
Convergence graph
6
![Page 8: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/8.jpg)
Data profile
The data profile is the empirical cumulative distribution function(ECDF) of the recorded runtimes
0 20 40 60 80 100Runtime (number of function evaluations)
Qual
ity in
dica
tor (
to b
e m
inim
ized)
Convergence graph
0 20 40 60 80 100Runtime (number of function evaluations)
0.0
0.2
0.4
0.6
0.8
1.0
Frac
tion
of ta
rget
s
Data profile
6
![Page 9: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/9.jpg)
Data profile
Data profiles can aggregate performance over multiple runs
0 20 40 60 80 100Runtime (number of function evaluations)
Qual
ity in
dica
tor (
to b
e m
inim
ized)
Convergence graph
7
![Page 10: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/10.jpg)
Data profile
Data profiles can aggregate performance over multiple runs
0 20 40 60 80 100Runtime (number of function evaluations)
Qual
ity in
dica
tor (
to b
e m
inim
ized)
Convergence graph
7
![Page 11: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/11.jpg)
Data profile
Data profiles can aggregate performance over multiple runs
0 20 40 60 80 100Runtime (number of function evaluations)
Qual
ity in
dica
tor (
to b
e m
inim
ized)
Convergence graph
0 20 40 60 80 100Runtime (number of function evaluations)
0.0
0.2
0.4
0.6
0.8
1.0
Frac
tion
of ta
rget
s
Data profile
7
![Page 12: 40ptA Short Introduction to COCO · • bbob-noisytest suite with 30 functions (45 algorithms) • bbob-biobjtest suite with 55 functions (16 algorithms) Algorithm results collected](https://reader036.vdocuments.mx/reader036/viewer/2022071215/6044e73bd25d8f668a78bb6a/html5/thumbnails/12.jpg)
COCO test suites
Test suites and algorithm results
• bbob test suite with 24 functions (173 algorithms)• bbob-noisy test suite with 30 functions (45 algorithms)• bbob-biobj test suite with 55 functions (16 algorithms)
Algorithm results collected at 9 BBOB Workshops (since 2009, mostlyat GECCO conferences)
Under development
• Suite with constrained problems• Suite with large-scale problems• Suites with real-world problems
8