Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking

Existing studies in black-box optimization for machine learning suffer from\nlow generalizability, caused by a typically selective choice of problem\ninstances used for training and testing different optimization algorithms.\nAmong other issues, this practice promotes overfitting and poor-performing user\nguidelines. To address this shortcoming, we propose in this work a benchmark\nsuite, OptimSuite, which covers a broad range of black-box optimization\nproblems, ranging from academic benchmarks to real-world applications, from\ndiscrete over numerical to mixed-integer problems, from small to very\nlarge-scale problems, from noisy over dynamic to static problems, etc. We\ndemonstrate the advantages of such a broad collection by deriving from it\nAutomated Black Box Optimizer (ABBO), a general-purpose algorithm selection\nwizard. Using three different types of algorithm selection techniques, ABBO\nachieves competitive performance on all benchmark suites. It significantly\noutperforms previous state of the art on some of them, including YABBOB and\nLSGO. ABBO relies on many high-quality base components. Its excellent\nperformance is obtained without any task-specific parametrization.\n The OptimSuite benchmark collection, the ABBO wizard and its base solvers\nhave all been merged into the open-source Nevergrad platform, where they are\navailable for reproducible research.\n