Learning How to Optimize Black-Box Functions With Extreme Limits on the Number of Function Evaluations

We consider black-box optimization in which only an extremely limited number\nof function evaluations, on the order of around 100, are affordable and the\nfunction evaluations must be performed in even fewer batches of a limited\nnumber of parallel trials. This is a typical scenario when optimizing variable\nsettings that are very costly to evaluate, for example in the context of\nsimulation-based optimization or machine learning hyperparameterization. We\npropose an original method that uses established approaches to propose a set of\npoints for each batch and then down-selects from these candidate points to the\nnumber of trials that can be run in parallel. The key novelty of our approach\nlies in the introduction of a hyperparameterized method for down-selecting the\nnumber of candidates to the allowed batch-size, which is optimized offline\nusing automated algorithm configuration. We tune this method for black box\noptimization and then evaluate on classical black box optimization benchmarks.\nOur results show that it is possible to learn how to combine evaluation points\nsuggested by highly diverse black box optimization methods conditioned on the\nprogress of the optimization. Compared with the state of the art in black box\nminimization and various other methods specifically geared towards few-shot\nminimization, we achieve an average reduction of 50\\% of normalized cost, which\nis a highly significant improvement in performance.\n