COCO: a platform for comparing continuous optimizers in a black-box setting

@article{Hansen2021COCOAP,
  title={COCO: a platform for comparing continuous optimizers in a black-box setting},
  author={Nikolaus Hansen and Anne Auger and Olaf Mersmann and Tea Tusar and Dimo Brockhoff},
  journal={Optimization Methods and Software},
  year={2021},
  volume={36},
  pages={114 - 144}
}
We introduce COCO, an open-source platform for Comparing Continuous Optimizers in a black-box setting. COCO aims at automatizing the tedious and repetitive task of benchmarking numerical optimization algorithms to the greatest possible extent. The platform and the underlying methodology allow to benchmark in the same framework deterministic and stochastic solvers for both single and multiobjective optimization. We present the rationals behind the (decade-long) development of the platform as a… 
View on Taylor & Francis
arxiv.org
156 Citations
Highly Influential Citations
20
Background Citations
30
Methods Citations
69
Results Citations
1

156 Citations

Citation Type

Citation Type

Mixed-integer benchmark problems for single- and bi-objective optimization
TLDR
Two suites of mixed-integer benchmark problems to be used for analyzing and comparing black-box optimization algorithms and some unexpected findings about their properties are provided.
DMS and MultiGLODS: black-box optimization benchmarking of two direct search methods on the bbob-biobj test suite
TLDR
A small defect in the default initialization of DMS and for both algorithms a decrease in relative performance to other algorithms of the original studies, and an under-performance to previously untested stochastic solvers from the evolutionary computation field, especially when the dimension is higher.
Benchmarking large-scale continuous optimizers: The bbob-largescale testbed, a COCO software guide and beyond
TLDR
A new testbed is presented, called bbob-largescale, that contains functions ranging from dimension 20 to 640, compatible with and extending the well-known single-objective noiseless bbOB test suite to larger dimensions, and uses permuted block diagonal orthogonal matrices to reduce the computational demand of the Orthogonal search space transformations.
Benchmarking for Metaheuristic Black-Box Optimization: Perspectives and Open Challenges
  • R. Sala, Ralf Müller
  • Computer Science, Mathematics
    2020 IEEE Congress on Evolutionary Computation (CEC)
  • 2020
TLDR
This communication aims to give a constructive perspective on several open challenges and prospective research directions related to systematic and generalizable benchmarking for black-box optimization.
Using Well-Understood Single-Objective Functions in Multiobjective Black-Box Optimization Test Suites.
TLDR
The rationale behind the approach in terms of groups of functions with similar properties, objective space normalization, and problem instances allows us to easily compare the performance of deterministic and stochastic solvers, which is an often overlooked issue in benchmarking.
Benchmarking discrete optimization heuristics with IOHprofiler
TLDR
This work compiles and assesses a selection of discrete optimization problems that subscribe to different types of fitness landscapes, and compares performances of eleven different heuristics for each selected problem.
The SOS Platform: Designing, Tuning and Statistically Benchmarking Optimisation Algorithms
We present Stochastic Optimisation Software (SOS), a Java platform facilitating the algorithmic design process and the evaluation of metaheuristic optimisation algorithms. SOS reduces the burden of…
The SOS Platform: Designing, Tuning and Statistically Benchmarking Optimisation Algorithms
TLDR
It is argued that the vast majority of these algorithms are simply a reformulation of the same methods and that metaheuristics for optimisation should be simply treated as stochastic processes with less emphasis on the inspiring metaphor behind them.
On Using Real-World Problems for Benchmarking Multiobjective Optimization Algorithms
Although the motivation to study multiobjective optimization algorithms comes from practice, there are only a few challenging real-world problems freely available to the research community. Because…
COCO: The Large Scale Black-Box Optimization Benchmarking (bbob-largescale) Test Suite
TLDR
This documentation presents an approach that replaces a full rotational transformation with a combination of a block-diagonal matrix and two permutation matrices in order to construct test functions whose computational and memory costs scale linearly in the dimension of the problem.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 88 REFERENCES
Mixed-integer benchmark problems for single- and bi-objective optimization
TLDR
Two suites of mixed-integer benchmark problems to be used for analyzing and comparing black-box optimization algorithms and some unexpected findings about their properties are provided.
Using Well-Understood Single-Objective Functions in Multiobjective Black-Box Optimization Test Suites.
TLDR
The rationale behind the approach in terms of groups of functions with similar properties, objective space normalization, and problem instances allows us to easily compare the performance of deterministic and stochastic solvers, which is an often overlooked issue in benchmarking.
Real-Parameter Black-Box Optimization Benchmarking 2009: Experimental Setup
Quantifying and comparing performance of optimization algorithms is one important aspect of research in search and optimization. However, this task turns out to be tedious and difficult to realize…
Comparing results of 31 algorithms from the black-box optimization benchmarking BBOB-2009
TLDR
Results of the BBOB-2009 benchmarking of 31 search algorithms on 24 noiseless functions in a black-box optimization scenario in continuous domain are presented and the choice of the best algorithm depends remarkably on the available budget of function evaluations.
Benchmarking Numerical Multiobjective Optimizers Revisited
TLDR
This work proposes to transfer and adapt standard benchmarking techniques from the single-objective optimization and classical derivative-free optimization community to the field of EMO, and applies this approach to compare three common algorithms on a new test function suite derived from the well-known single- objective BBOB functions.
Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions
TLDR
The testbed of noise-free functions is defined and motivated, and the participants' favorite black-box real-parameter optimizer in a few dimensions a few hundreds of times and execute the provided post-processing script afterwards.
jMetal: A Java framework for multi-objective optimization
TLDR
This paper describes jMetal, an object-oriented Java-based framework aimed at the development, experimentation, and study of metaheuristics for solving multi-objective optimization problems, and includes two case studies to illustrate the use of jMetal in both solving a problem with a metaheuristic and designing and performing an experimental study.
The Impact of Search Volume on the Performance of RANDOMSEARCH on the Bi-objective BBOB-2016 Test Suite
TLDR
It turns out that the suggested region of interest of [-100,100]n (with $n$ being the problem dimension) has a too vast volume for the algorithm to approximate the Pareto set effectively.
Benchmarking RM-MEDA on the Bi-objective BBOB-2016 Test Suite
TLDR
It turns out that, starting from about 200 times dimension many function evaluations, family RM-MEDA shows a linear increase in the solved hypervolume-based target values with time until a stagnation of the performance occurs rather quickly on all problems.
PAVER 2.0: an open source environment for automated performance analysis of benchmarking data
TLDR
PAVER 2.0, an environment for the automated performance analysis of benchmarking data, improves on its predecessor by addressing some of the shortcomings of the original PAVER and making the environment more accessible for the use of and modification by the entire community of potential users.
...
1
2
3
4
5
...