cocopf: an algorithm portfolio framework · callback mechanism to suspend and resume execution of...

arX

iv:1

405.

3487

v1 [

cs.A

I] 1

4 M

ay 2

014

POSTER 2014, PRAGUE MAY 15 1

COCOpf: An Algorithm Portfolio Framework

Petr Baudis

Dept. of Cybernetics, Czech Technical University, Technicka 2, 166 27 Praha, Czech Republic

[email protected]

Abstract. Algorithm portfolios represent a strategy of com-posing multiple heuristic algorithms, each suited to a dif-ferent class of problems, within a single general solver thatwill choose the best suited algorithm for each input. Thisapproach recently gained popularity especially for solvingcombinatoric problems, but optimization applications arestill emerging. The COCO platform [6] [5] of the BBOBworkshop series is the current standard way to measure per-formance of continuous black-box optimization algorithms.

As an extension to the COCO platform, we present thePython-based COCOpf framework that allows composingportfolios of optimization algorithms and running experi-ments with different selection strategies. In our framework,we focus on black-box algorithm portfolio and online adap-tive selection. As a demonstration, we measure the perfor-mance of stock SciPy [8] optimization algorithms and thepopular CMA algorithm [4] alone and in a portfolio withtwo simple selection strategies. We confirm that even a naiveselection strategy can provide improved performance acrossproblem classes.

KeywordsAlgorithm portfolio, continuous black-box optimiza-tion, hyperheuristics, software engineering.

1. IntroductionThe problem of Algorithm Selection is not new [16],

but has only recently gained traction in particular in the fieldof combinatoric problem solvers. [9] There are multipleways to approach the issue, in our work we adopt the ab-straction of algorithm portfolios. [3] The basic idea is thatwe have a set ofheuristic algorithmsat our disposal (eachsuitable for a different class of problem instances), and foreach problem instance on input, we apply aselection strat-egyto pick an algorithm from this portfolio and apply it onthe problem. We can perform the selection once or alonga fixed schedule(offline selection)based on features of theproblem instance, or in multiple rounds first exploring thesuitability of portfolio members, then allocating time to thembased on their previous performance. In successive rounds,

algorithms can be either resumed from their previous state orrestarted; we take the approach of resuming them, reservingthe restart schedule to be an internal matter of each algo-rithm.1

Applying algorithm portfolios on continuous black-boxoptimization is still a fresh area of research.2 The main re-sults so far lie either in population methods, combining a va-riety of genetic algorithms together, e.g. the AMALGAM-SO algorithm [17], or in offline methods based chiefly onexploratory landscape analysis [11].

In our work, we aim to allow the more traditional opti-mization methods to enter the mix as we regard the algo-rithms asblack-box, i.e. we completely avoid modifyingtheir inner working and we simply call them to retrieve a sin-gle result. This allows state-of-art methods to be combinedin a single portfolio easily and offers easy implementationofwhatever selection strategy desired as the existing algorithmmodules can be simply reused without modification. Fur-thermore, we focus on online adaptive selection that selectsalgorithms based on their performance so far and does notrequire possibly expensive feature extraction and training.

The currently accepted de-facto standard for bench-marking optimization methods is theCOmparing Continu-ous OptimisersCOCO platform [6] [5] that was originallydeveloped for the BBOB workshop series. It provides theinfrastructure, glue code for both running experiments andpreparing high quality figures, a set of common referenceresults and the code for a set of benchmark functions. There-fore, we chose to add algorithm portfolio support to this plat-form. The glue code is available in multiple languages, weextended the Python implementation. We are in part mo-tivated by the convenient availability of optimization algo-rithms distributed along the popular SciPy library [8].

We have two options when testing algorithm selectionmethods in practice: either using only previously publisheddata on preformance of individual algorithms, or re-runningoptimization algorithms from scratch under the guide of aparticular selection method. The former approach is cer-tainly an easier option while benchmarking, nevertheless we

1That is, sometimes resumption will simply continue the algorithm toa next iteration, on reaching a local optimum the algorithm may attempt toescape it and failing that, a complete restart may occur.

2Especially if we do not consider applying related methods toindividualoperator selection within genetic algorithms used for optimization.

http://arxiv.org/abs/1405.3487v1

2 P. BAUDIS, COCOpf Algorithm Portfolios

opted for the latter model so that we can also readily apply allcode on practical function optimization aside of mere bench-marking. Apart of that, the available performance data doesnot include information about algorithm iterations, listin-gonly individual function evaluations. That makes it un-clear when is a good time to switch. Lastly, the re-runningmodel allows migration of intermediate solutions betweenalgorithms as a simple extension.

In Section 2, we introduce the COCOpf software, ex-plaining its architecture, general API and a few interestingimplementation points. In Section 3, we establish a refer-ence algorithm portfolio, benchmarking its individual mem-bers and two algorithm selection methods implemented inthe framework. We present the benchmark results in Section4 and conclude with a results summary and outline of a fewdirections of future work in Section 5.

2. COCOpf FrameworkOur framework has been implemented as a Python

module that can be imported by user applications. Theframework provides the means to run the experiment ona given set of benchmark scenarios, maintains a popula-tion of algorithm instances and provides a complex wrapperto individual algorithms that uses an existing per-iterationcallback mechanism to suspend and resume execution ofthe algorithm after a single iteration. Algorithm wrap-pers for the SciPy optimization methods and the refer-ence implementation of CMA are available out-of-the boxas well as two example algorithm selection strategies thatare benchmarked below. The framework is publicly avail-able as free software under the permissible MIT licence athttps://github.com/pasky/cocopf.

In design of such a framework, a choice presents it-self — to either build a generic main program which allowsan experiment to choose a certain configuration and plug incustom callbacks, or to prepare a set of ”lego bricks” thatreplace common parts of the main program but letting eachexperiment provide a separate main program. We opted forthe latter; it reduces encapsulation and requires slightlymorerepetitive code, but it allows a much greater degree of flexi-bility encouraging diverse experiments.

2.1. Experiment Wrapper

TheExperiment class wraps up the common code thatsets up the COCO benchmark, initializing the logs, sets ofdimensions and function instances etc. It also provides aniterator that in turn returns individual function instances tobe minimized, and a pretty-printing progress report func-tion. After an experiment is finished, the standard BBOBpostprocessing tools may be used to examine and plot theperformance data. An example of usage is shown in Fig. 1.

e = Experiment(maxfev=10000, shortname="EG50",

comments="Eps-greedy, eps=0.5")

for i in e.finstances():

minimization(i)

e.f.finalizerun()

e.freport(i)

Fig. 1.A minimalistic main body of a Python script using theExperiment class. Functionminimization is to be pro-vided by the user.

2.2. Minimization Wrappers

All algorithms are executed via theMinimizeMethodclass. Its primary purpose is to produce a callablePython object from a user-provided string name of al-gorithm. By default, it calls thescipy.optimize.minimizeAPI with unchanged method name and wraps it in thescipy.optimize.basinhoppingAPI. However, it is expectedthe user will create a subclass adding custom optimizers;a demo subclass adding the CMA optimization is included.3

We were presented with a software engineering prob-lem of how to suspend, resume and cancel the algorithmsbetween iterations without requiring any modification to thealgorithm code, while all we have is a callback executed aftereach iteration. Our answer is theMinimizeStepping classwhich creates a separate thread for each algorithm instanceand uses the standard PythonQueuemechanism to pause ex-ecution of each thread within the callback code based on in-structions from the main thread. At any time, only a singlethread is running. To cancel execution (when time is up ora solution is found) we pass a special message through thequeue that raises an exception within the callback, eventu-ally cancelling the whole algorithm and thread orderly. An“mlog” facility is provided to log function evaluation statusat the end of each algorithm iteration.

2.3. Portfolio Population

The Population class maintains a population of algo-rithms drawn from the algorithm portfolio; each populationmember corresponds to a tuple of method, solution coordi-nates, current value and number of iterations invested. Usu-ally, the size of population is static, but dynamically addingnew members is also supported.

In a typical case, the population size is the same as theportfolio size, meaning that only a single solution is trackedfor each algorithm. However, in principle this may not be thecase. In fact, we can also make a population from solutionsall using a single algorithm — this special case corresponds

3The custom optimizers may also legitimately want to use the BasinHopping algorithm (see Sec. 3.1), but SciPy allows using custom minimiza-tion routines with the basinhopping API only starting with version 0.14.

https://github.com/pasky/cocopf


to the problem of finding an efficient restart or resume sched-ule for a particular algorithm.

3. Benchmarking Simple PortfoliosTo showcase COCOpf, we composed a reference port-

folio from readily available open source Python optimizersand implemented two selection strategies.

3.1. Reference Algorithm Portfolio

As a testbed algorithm portfolio, we have chosen thesix stock minimizers provided by SciPy v0.13 [8] that areavailable for direct use:4

• Nelder-Meaduses the Simplex algorithm. [12] [19]

• Powell implements a tweaked version of the Powell’sconjugate direction method. [14]

• CG is a nonlinear conjugate gradient method using theFletcher-Reeves method. [13, p. 120–122]

• BFGS uses the quasi-Newton method of Broyden,Fletcher, Goldfarb and Shanno [13, p.136] with numer-ically approximated derivations.

• L-BFGS-B is a limited-memory variant of BFGS withbox constraints. [1] [20]

• SLSQP implements Sequential Least SQuares Pro-gramming [10] with inequalities as the box constraints.

All of these SciPy minimizers perform local mini-mization; to achieve global optimization, we wrap them inthe “Basin Hopping” restart strategy [18] (also provided bySciPy), which is conceptually similar to Simulated Anneal-ing with a fixed temperature. The strategy performs thehopping step 100 times (the default) before restarting fromscratch. On restart, the starting position is determined uni-formly randomly in the[−5,5]k space.

To improve the portfolio performance on more difficultfunctions, we also included the popular CMA algorithm [4](in the “production“ version of its official reference Pythonimplementation), i.e. a genetic algorithm that follows theCovariance Matrix Adaptation evolution strategy.

In accordance with our black-box approach, we keptall the algorithms at their default settings and did not tuneany of their parameters. Where applicable, we set the boxconstraints to[−6,6]k bounds.5

4Other minimizers are also available in SciPy, but they are either unsuit-able for black-box minimization or do not provide a callbackmechanism toallow pausing execution between individual iterations.

5The optimum is guaranteed to be in[−5,5]k , but some minimizersmay behave erraticaly when the optimum is near to or at the bounds.

Each algorithm provides a callback method that is in-voked after a single iteration of the algorithm; we suspendthe algorithm after each callback invocation.

The reference portfolio is not perfectly balanced andcertainly would benefit from algorithms with more diverseperformance. Our motivation for going with these is thatthe implementations are free software, trivial to obtain andtherefore representing a very low barrier of entry. Also,a SciPy-based portfolio gives valuable feedback to SciPyusers regarding the best algorithm to use on a certain func-tion, and provides a gateway to introduce an algorithm port-folio selection strategy to SciPy itself in the future.

3.2. Algorithm Selection Strategies

While the core of our followup work involves research-ing various algorithm selection strategies, in the contextof presentation of the COCOpf framework, we decided todemonstrate just the performance of two simple simplestmethods whose source code is also a part of the framework:

• TheUNIF strategy performs a number of rounds wherein every round, a uniformly randomly selected algo-rithm is run for a single iteration.

• TheEG50strategy performs a number of rounds wherein every round, anepsilon-greedy policywith ε = 0.5

selects the algorithm to run for one iteration. That is,the algorithm with the currently best solution is runwith p = 0.5 and a randomly chosen algorithm is runotherwise.

4. ResultsResults from experiments according to [5] on the

benchmark functions given in [2, 7] are presented in Fig-ures 2, 3 and 4.6 The expected running time (ERT) usedin the figures depends on a given target function value,ft = fopt+∆f , and is computed over all relevant trials as thenumber of function evaluations executed during each trialwhile the best function value did not reachft, summed overall trials and divided by the number of trials that actuallyreachedft [5, 15].

If we review the results in context of choosing a best(black-box function) SciPy optimizer, we can see that thereis no single right choice of a SciPy optimizer that works onall function instances and computational budgets. We canobserve thatPowellgenerally works best for separable func-tions whileBFGS andSLSQP are good choices for manyfunctions, especially in high dimensions. For non-separablefunctions,CMA sooner or later overtakes the SciPy opti-mizers but often with a slow start (which is relevant in case a

6The full results dataset is available athttp://pasky.or.cz/sci/cocopf-scipy.

http://pasky.or.cz/sci/cocopf-scipy


2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

1 Sphere

2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

2 Ellipsoid separable

2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

3 Rastrigin separable

2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

4 Skew Rastrigin-Bueche separ

2 3 5 10 20 400

1

2

3

4

ftarget=1e-08

5 Linear slope

2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

6 Attractive sector

2 3 5 10 20 400

1

2

3

4

5

6

7

ftarget=1e-08

7 Step-ellipsoid

2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

8 Rosenbrock original

2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

9 Rosenbrock rotated

2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

10 Ellipsoid

2 3 5 10 20 400

1

2

3

4

5

6

7

ftarget=1e-08

11 Discus

2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

12 Bent cigar

2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

13 Sharp ridge

2 3 5 10 20 400

1

2

3

4

5

6

7

ftarget=1e-08

14 Sum of different powers

2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

15 Rastrigin

2 3 5 10 20 400

1

2

3

4

5

6

7

ftarget=1e-08

16 Weierstrass

2 3 5 10 20 400

1

2

3

4

5

6

7

ftarget=1e-08

17 Schaffer F7, condition 10

2 3 5 10 20 400

1

2

3

4

5

6

7

ftarget=1e-08

18 Schaffer F7, condition 1000

2 3 5 10 20 400

1

2

3

4

5

6

7

ftarget=1e-08

19 Griewank-Rosenbrock F8F2

2 3 5 10 20 400

1

2

3

4

5

6

ftarget=1e-08

20 Schwefel x*sin(x)

2 3 5 10 20 400

1

2

3

4

5

6

7

ftarget=1e-08

21 Gallagher 101 peaks

2 3 5 10 20 400

1

2

3

4

5

6

7

ftarget=1e-08

22 Gallagher 21 peaks

2 3 5 10 20 400

1

2

3

4

5

6

7

ftarget=1e-08

23 Katsuuras

2 3 5 10 20 400

1

2

3

4

5

6

7

ftarget=1e-08

24 Lunacek bi-Rastrigin

Fig.2. Expected running time divided by dimension versus dimension. Expected running time (ERT in number off -evaluations)divided by dimension for target function value10−8 as log10 values versus dimension. Light symbols give the maximumnumber of function evaluations from the longest trial divided by dimension. Horizontal lines give linear scaling, slanted dottedlines give quadratic scaling. Black stars indicate statistically better result compared to all other algorithms withp < 0.01

and Bonferroni correction number of dimensions (six). Legend: ○:Nelder-Mead,▽:Powell,⋆:CG,◻:BFGS,△:L-BFGS-B,♢:SLSQP,9:CMA, D:UNIF, 7:EG50


separable fcts moderate fcts

0 1 2 3 4 5 6 7 8log10 of (# f-evals / dimension)

0.0

0.2

0.4

0.6

0.8

1.0

Proportion of function+target pairs

CG

BFGS

CMA

Nelder-Mead

L-BFGS-B

SLSQP

UNIF

Powell

EG50

best 2009f1-5,5-D best 2009

EG50

Powell

UNIF

SLSQP

L-BFGSB

Nelder

CMA

BFGS

CG0 1 2 3 4 5 6 7 8

log10 of (# f-evals / dimension)

0.0

0.2

0.4

0.6

0.8

1.0

Prop

ortio

n of fu

nctio

n+target pairs

L-BFGS-B

CG

SLSQP

BFGS

Powell

Nelder-Mead

UNIF

EG50

CMA

best 2009f6-9,5-D best 2009

CMA

EG50

UNIF

Nelder

Powell

BFGS

SLSQP

CG

L-BFGSB

ill-conditioned fcts multi-modal fcts


0.0

0.2

0.4

0.6

0.8

1.0

Prop

ortio

n of fu

nctio

n+target pairs

Powell

CG

L-BFGS-B

SLSQP

BFGS

Nelder-Mead

UNIF

EG50

CMA

best 2009f10-14,5-D best 2009

CMA

EG50

UNIF

Nelder

BFGS

SLSQP

L-BFGSB

CG

Powell0 1 2 3 4 5 6 7 8


0.0

0.2

0.4

0.6

0.8

1.0Pr

oportio

n of fu

nctio

n+target pairs

CG

Nelder-Mead

BFGS

Powell

SLSQP

L-BFGS-B

UNIF

CMA

EG50

best 2009f15-19,5-D best 2009

EG50

CMA

UNIF

L-BFGSB

SLSQP

Powell

BFGS

Nelder

CG

weakly structured multi-modal fcts all functions


0.0

0.2

0.4

0.6

0.8

1.0


BFGS

CG

Powell

UNIF

Nelder-Mead

CMA

L-BFGS-B

SLSQP

EG50

best 2009f20-24,5-D best 2009

EG50

SLSQP

L-BFGSB

CMA

Nelder

UNIF

Powell

CG

BFGS0 1 2 3 4 5 6 7 8


0.0

0.2

0.4

0.6

0.8

1.0


CG

BFGS

Powell

Nelder-Mead

SLSQP

L-BFGS-B

CMA

UNIF

EG50

best 2009f1-24,5-D best 2009

EG50

UNIF

CMA

L-BFGSB

SLSQP

Nelder

Powell

BFGS

CG

Fig.3. Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension(FE-vals/D) for 50 targets in10[−8..2] for all functions and subgroups in 5-D. The “best 2009” line corresponds to the bestERTobserved during BBOB 2009 for each single target.


separable fcts moderate fcts


0.0

0.2

0.4

0.6

0.8

1.0

Prop

ortio

n of fu

nctio

n+target pairs

Nelder-Mead

L-BFGS-B

CG

SLSQP

BFGS

CMA

EG50

UNIF

Powell

best 2009f1-5,20-D best 2009

Powell

UNIF

EG50

CMA

BFGS

SLSQP

CG

L-BFGSB

Nelder0 1 2 3 4 5 6 7 8


0.0

0.2

0.4

0.6

0.8

1.0

Prop

ortio

n of fu

nctio

n+target pairs

Nelder-Mead

Powell

SLSQP

L-BFGS-B

BFGS

CG

EG50

UNIF

CMA

best 2009f6-9,20-D best 2009

CMA

UNIF

EG50

CG

BFGS

L-BFGSB

SLSQP

Powell

Nelder

ill-conditioned fcts multi-modal fcts


0.0

0.2

0.4

0.6

0.8

1.0

Prop

ortio

n of fu

nctio

n+target pairs

Nelder-Mead

Powell

CG

BFGS

L-BFGS-B

SLSQP

UNIF

EG50

CMA

best 2009f10-14,20-D best 2009

CMA

EG50

UNIF

SLSQP

L-BFGSB

BFGS

CG

Powell

Nelder0 1 2 3 4 5 6 7 8


0.0

0.2

0.4

0.6

0.8

1.0

Prop

ortio

n of fu

nctio

n+target pairs

CG

Nelder-Mead

Powell

L-BFGS-B

BFGS

SLSQP

UNIF

EG50

CMA

best 2009f15-19,20-D best 2009

CMA

EG50

UNIF

SLSQP

BFGS

L-BFGSB

Powell

Nelder

CG

weakly structured multi-modal fcts all functions


0.0

0.2

0.4

0.6

0.8

1.0

Prop

ortio

n of fu

nctio

n+target pairs

Nelder-Mead

Powell

EG50

SLSQP

CG

BFGS

L-BFGS-B

UNIF

CMA

best 2009f20-24,20-D best 2009

CMA

UNIF

L-BFGSB

BFGS

CG

SLSQP

EG50

Powell

Nelder0 1 2 3 4 5 6 7 8


0.0

0.2

0.4

0.6

0.8

1.0

Prop

ortio

n of fu

nctio

n+target pairs

Nelder-Mead

Powell

CG

L-BFGS-B

BFGS

SLSQP

UNIF

EG50

CMA

best 2009f1-24,20-D best 2009

CMA

EG50

UNIF

SLSQP

BFGS

L-BFGSB

CG

Powell

Nelder

Fig.4. Bootstrapped empirical cumulative distribution of the number of objective function evaluations divided by dimension(FE-vals/D) for 50 targets in10[−8..2] for all functions and subgroups in 20-D. The “best 2009” linecorresponds to the bestERTobserved during BBOB 2009 for each single target.


function evaluation is very expensive) and is not competitivewith the SciPy optimizers on separable functions.

5. Discussion and ConclusionWe described an easy-to-use framework that allows

users to plug in and experiment with custom algorithm selec-tion strategies and then directly move to use these strategieson practical problems instead of benchmarks. The frame-work currently focuses on online black-box algorithm port-folios, being careful not to require any modification to stockand third-party optimization routines besides a simple call-back mechanism that is commonly provided. The frameworkcould be easily modified to deal with off-line scenarios too.

Our insights may provide guidance for users of theSciPy optimizers. Regarding algorithm portfolios, the re-sults show that even a very naive algorithm selection strat-egy can be beneficial; while it cannot beat the best algorithmCMA in high-dimensional cases on average over all func-tions, it provides a more consistent performance than CMA,translating to best-of-all performance in low dimensions.

5.1. Future Work

The currently chosen reference portfolio is somewhatad hoc, chosen in part to contribute value to the SciPyproject, but not optimal for further general research into al-gorithm portfolios. In the long run, we hope to transition toa better balanced portfolio.

A common scheme of online algorithm selectionschemes is that acredit is assignedto an algorithm in eachround based on its performance (e.g. relative improvement,absolute value) andaccruedover the rounds (e.g. averagingit or using an adaptation rule). Credit-based selection is socommon that it should be directly supported by the frame-work; diverse pluggable strategies would then enable explo-ration of various interesting strategy combinations. A work-in-progress preview implementation is already available.

Actually runningthe individual algorithms is appropri-ate for optimizing non-benchmark functions and useful inthe future when solutions may migrate within the portfoliopopulation. However, a useful feature would be to allow test-ing selection strategies based on just the saved “mlog“ data(see Sec. 2.2) from individual algorithm runs.

AcknowledgementsResearch described in the paper was supervised by Dr.

Petr Posık and supported by the CTU grant SGS14/194/OHK3/3T/13 “Automatic adaptation of search algorithms”.

References[1] BYRD, R. H., LU, P., NOCEDAL, J.,AND ZHU, C. A limited mem-

ory algorithm for bound constrained optimization.SIAM Journal onScientific Computing 16, 5 (1995), 1190–1208.

[2] FINCK , S., HANSEN, N., ROS, R., AND AUGER, A. Real-parameter black-box optimization benchmarking 2009: Presentationof the noiseless functions. Tech. Rep. 2009/20, Research Center PPE,2009. Updated February 2010.

[3] GOMES, C. P.,AND SELMAN , B. Algorithm portfolios. ArtificialIntelligence 126, 1 (2001), 43–62.

[4] HANSEN, N. The CMA evolution strategy: a comparing review. InTowards a new evolutionary computation. Advances on estimation ofdistribution algorithms, J. Lozano, P. Larranaga, I. Inza, and E. Ben-goetxea, Eds. Springer, 2006, pp. 75–102.

[5] HANSEN, N., AUGER, A., FINCK , S.,AND ROS, R. Real-parameterblack-box optimization benchmarking 2012: Experimental setup.Tech. rep., INRIA, 2012.

[6] HANSEN, N., ET AL . Comparing continuous optimisers: Coco.http://coco.gforge.inria.fr/.

[7] HANSEN, N., FINCK , S., ROS, R., AND AUGER, A. Real-parameterblack-box optimization benchmarking 2009: Noiseless functions def-initions. Tech. Rep. RR-6829, INRIA, 2009. Updated February 2010.

[8] JONES, E., OLIPHANT, T., PETERSON, P., ET AL . SciPy: Opensource scientific tools for Python, 2001–.

[9] K OTTHOFF, L. Algorithm selection for combinatorial search prob-lems: A survey.AI Magazine(2014).

[10] KRAFT, D. A software package for sequential quadratic program-ming. DFVLR Obersfaffeuhofen, Germany, 1988.

[11] MUNOZ, M., K IRLEY, M., AND HALGAMUGE , S. The algorithmselection problem on the continuous optimization domain. In Com-putational Intelligence in Intelligent Data Analysis, C. Moewes andA. Nurnberger, Eds., vol. 445 ofStudies in Computational Intelli-gence. Springer Berlin Heidelberg, 2013, pp. 75–89.

[12] NELDER, J. A., AND MEAD, R. A simplex method for functionminimization. The Computer Journal 7, 4 (1965), 308–313.

[13] NOCEDAL, J.,AND WRIGHT, S. Numerical optimization. Springer-Verlag, 2006.

[14] POWELL, M. J. An efficient method for finding the minimum ofa function of several variables without calculating derivatives. Thecomputer journal 7, 2 (1964), 155–162.

[15] PRICE, K. Differential evolution vs. the functions of the secondICEO. In Proceedings of the IEEE International Congress on Evo-lutionary Computation(1997), pp. 153–157.

[16] RICE, J. R. The algorithm selection problem.Advances in Computers15 (1976), 65–118.

[17] VRUGT, J. A., ROBINSON, B. A., AND HYMAN , J. M. Self-adaptivemultimethod search for global optimization in real-parameter spaces.IEEE Trans. on Evolutionary Computation 13, 2 (2009), 243–259.

[18] WALES, D. J., AND DOYE, J. P. K. Global optimization by basin-hopping and the lowest energy structures of lennard-jones clusterscontaining up to 110 atoms.The Journal of Physical Chemistry A101, 28 (1997), 5111–5116.

[19] WRIGHT, M. H. Direct search methods: Once scorned, now re-spectable.Pitman Research Notes in Math. Series(1996), 191–208.

[20] ZHU, C., BYRD, R. H., LU, P.,AND NOCEDAL, J. Algorithm 778:L-bfgs-b: Fortran subroutines for large-scale bound-constrained opti-mization. ACM Transactions on Mathematical Software (TOMS) 23,4 (1997), 550–560.

About Author. . .

Petr Baudis has received his Bachelors degree and Mastersdegree in Theoretical Computer Science at the Charles Uni-versity in Prague. His Master thesis “MCTS with Informa-tion Sharing” presented a state-of-art Computer Go program.

http://coco.gforge.inria.fr/


Currently, he is a PhD student at the Czech Technical Univer-sity with principal research interest in Algorithm Portfoliosand applications of Monte Carlo Tree Search.

cocopf: an algorithm portfolio framework · callback mechanism to suspend and resume execution of...

Documents