LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso

While Weighted Lasso sparse regression has appealing statistical guarantees\nthat would entail a major real-world impact in finance, genomics, and brain\nimaging applications, it is typically scarcely adopted due to its complex\nhigh-dimensional space composed by thousands of hyperparameters. On the other\nhand, the latest progress with high-dimensional hyperparameter optimization\n(HD-HPO) methods for black-box functions demonstrates that high-dimensional\napplications can indeed be efficiently optimized. Despite this initial success,\nHD-HPO approaches are mostly applied to synthetic problems with a moderate\nnumber of dimensions, which limits its impact in scientific and engineering\napplications. We propose LassoBench, the first benchmark suite tailored for\nWeighted Lasso regression. LassoBench consists of benchmarks for both\nwell-controlled synthetic setups (number of samples, noise level, ambient and\neffective dimensionalities, and multiple fidelities) and real-world datasets,\nwhich enables the use of many flavors of HPO algorithms to be studied and\nextended to the high-dimensional Lasso setting. We evaluate 6 state-of-the-art\nHPO methods and 3 Lasso baselines, and demonstrate that Bayesian optimization\nand evolutionary strategies can improve over the methods commonly used for\nsparse regression while highlighting limitations of these frameworks in very\nhigh-dimensional and noisy settings.\n