PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning

Summary

This paper presents a method to enhance the random sampling component of hyperband. The authors propose replacing it with a combination of random sampling, prior-based sampling, and incumbent-based sampling. They also suggest adjusting the proportion of these samplers based on the current state in the hyperparameter optimization process. The authors perform experiments on a series of benchmarks and compare the proposed method with multiple classical HBO baselines.

Strengths

This work might be useful in some particular situations.

Weaknesses

- The motivation for this work may appear artificial, aiming to reduce the cost of hyperparameter optimization in the age of deep learning. However, the paper lacks a detailed explanation of how the proposed adjustments to the sampling component can make hyperparameter optimization more practical and cost-effective in the era of deep learning. Can the authors provide a more comprehensive reasoning process to support this claim? - Technically, this method is essentially a combination of prior work, including multi-fidelity optimization [1], expert priors [2], and local search [3]. The authors' contributions primarily build upon and benefit from these existing approaches, rather than introducing original ideas. The authors declare that their approach fulfills all the desired requirements for application to deep learning, but this claim is essentially derived from the benefits provided by multi-fidelity optimization. [1] Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1), 6765-6816. [2]Hvarfner, C., Stoll, D., Souza, A., Lindauer, M., Hutter, F., & Nardi, L. (2022). $\pi $ BO: Augmenting acquisition functions with user beliefs for bayesian optimization. arXiv preprint arXiv:2204.11051. [3]Wu, Q., Wang, C., & Huang, S. (2021, May). Frugal optimization for cost-related hyperparameters. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 12, pp. 10347-10354).

Questions

Why modifying the random sampling component of HyperBand can help decrease the expenses associated with tuning deep learning models?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

3 good

Limitations

As a hyperparameter optimization algorithm, the proposed method encompasses several hyperparameters that may significantly influence its final outcomes. These include the hyperparameters associated with the perturbation operation and those control the proportion of the three sampling methods. However, there is currently a dearth of corresponding ablation studies that specifically investigate the individual contributions of these hyperparameters.

Summary

This paper proposes PriorBand, an extension of HyperBand that adds expert priors and a novel sampling technique to replace random sampling in HB, called the Ensemble Sampling Policy (ESP). The ESP allows the algorithm to lean on the expert prior, but also use the current incumbent in case the prior is non-optimal. Under good and bad priors, the authors demonstrate the efficacy of this approach, showing additionally that the ESP also improves the performance of other Successive Halving (SH)-based methods. **Rebuttal:** The authors have clearly addressed my comments in their rebuttal. I have updated my score from 5 to 7.

Strengths

- My favorite part of this paper is Section 7.2, which shows that the ESP proposed in this paper extends to other SH-like HPO methods as well. This contribution should perhaps be highlighted more. - The paper's verbiage is very clear, and examples presented in the introductory sections are useful aids to readers who may not be familiar with HyperBand's workings.

Weaknesses

- My major concern is in the use of average relative rank to demonstrate efficacy. While a lower rank indicates better performance, it does not indicate whether that performance delta is statistically significant. As opposed to confidence intervals over *ranks*, I would much rater see the actual improvement in per-dataset metrics. For example, the original HyperBand paper [1] showed the test error over wall time for multiple HPO algorithms. - minor: In L130, you're missing a period. - L182: typo: modelling --> modeling [1] Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., & Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. The journal of machine learning research, 18(1), 6765-6816.

Questions

- In Alg. 2, L5, can you explain why $p_\pi$ is used instead of $p_{\hat{\lambda}}$? - In Fig. 14, it seems the lines have confidence intervals. How are these computed, and how many repeats of the experiment were used? - In Appendix D.3, the authors describe the way a "good" prior is generated, using the best of 25 configurations. In practice, this would add computational overhead to the overall process of training a model with good hyper-parameters. In that sense, perhaps it should be emphasized that the Bad prior results are more useful to a practitioner, who, following their intuition/expertise/random choice, might possibly use a non-optimal prior (i.e., without running anything). Can you comment on this?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors have addressed the limitations of their work appropriately in the main text and the supplementary materials.

Summary

In the paper "PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning" the authors propose an extension to the well-known HPO methods Hyperband by adapting the way how candidate hyperparameter settings are sampled. To this end, the authors propose to use a weighting mechanism to balance between three sampling distributions: random, locally random close to the incumbent, and a prespecified prior. The latter allows experts to inject beliefs about the optimum. In the empirical study PriorBand is found to perform best on average among the considered methods.

Strengths

+ PriorBand compares favorably to its competitors and allows for hyperparameter optimization with comparably low budgets. + The paper is very well written and the presentation in general is excellent. + The authors put very much effort into making their work reproducible, openly accessible and easy to understand. + For the self-imposed desiderata, PriorBand offers the most desired properties.

Weaknesses

- The work is pretty incremental. The only original contribution of this work is to weight the different probability distributions from which candidates are sampled. - It is not clear what effect the incumbent-sampling has on the overall performance. At least I could not find any ablation studies in this regard. Is it really necessary to include incumbent-sampling? What would happen if only random sampling is balanced against prior-sampling? Is it then harder to retain decent performance in the light of bad priors? - Nothing is stated about the impact on theoretical guarantees that are known to hold for Hyperband. Does the theoretical framework of Hyperband still apply for the changed distributions? - When comparing different hyperparameter optimizers only relative ranks are

Questions

- What is the individual effect of incumbent-based and prior-based sampling? - What is the impact on the theoretical guarantees for Hyperband? Are all the assumptions still fulfilled? - What is the definition of "relative rank"? Is it only a rank or does it also include performance differences? - Where do these desiderata stem from? Although they all seem intuitive it is not clear whether they are exhaustive and what the coverage is like.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

2 fair

Limitations

Limitations of PriorBand are well discussed. However, it is not entirely clear to what extent the desiderata for HPO in the age of deep learning is limited or maybe even exhaustive.

Summary

This paper presents PriorBand, a hyperparameter optimization (HPO) algorithm designed specifically for deep learning models. PriorBand fulfills six key requirements and tackles the shortcomings of existing HPO methods that are unsuitable for DL. It leverages cheap proxy tasks while considering expert input. The algorithm eliminates the need for a naive solution for integrating expert domain knowledge into HPO. Experiments across a wide array of DL tasks are conducted to demonstrate the efficiency and robustness of PriorBand.

Strengths

+ The proposed algorithm is simple yet effective. The augmented prior based on the existing HyperBand algorithm is technically sound and exhibits clear improvement over baselines. + The paper is well-written and well-structured. Motivation is clearly stated before the introduction of detailed algorithms (e.g., Sect. 4).

Weaknesses

- Though the experiments conducted in the paper are quite comprehensive, I found most of them are moved to the appendix where the figures in the main paper are of limited information (most are about robustness to bad priors). I would suggest authors rearrange the paper by moving figures from Appendix F to the main text. - In Table 1, given the fact that all the baseline methods satisfy "Mixed search spaces" + "Speedup under parallelism", I would suggest the authors remove these two rows as they are also not the major technical contributions of the proposed method. Maybe instead, replace with one sentence in the caption highlighting both the baselines and the proposed method can satisfy these two criteria.

Questions

See Weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

See Weakness.

Re: Rebuttal

Thanks for the authors' reply. The rebuttal has resolved my concerns. I am not mainly working on hyperparameter optimization so I cannot faithfully judge the novelty + the technical contribution of the paper compared with the existing SOTA ones, as concerned by Reviewer QGiN and meUK. However, the comprehensive comparison in the paper has convinced me of the method's effectiveness, so I will maintain my positive score.

Re: Rebuttal by Authors

Thank you very much for the thorough rebuttal and the clarifications. Most of my points are reasonably addressed in your comments and revisions. However, the aspect of being incremental remains and I do not find the argument that there exist other incremental works very convincing. Furthermore, I do not consider myself a deep learning practitioner, therefore I unfortunately need to decline the invitation of providing more desiderata. Yet, I would wish for a survey among practitioners, on what they would actually desire from HPO methods to incorporate them into their daily routine when working with such expensive training processes. I would agree with the authors that reducing the cost for HPO might indeed be a key point but maybe not the only one. In general, I would not expect such studies but in this paper on a presentation level, this was very prominently exposed to the reader. Appreciating the work of the authors in their rebuttal and the practical use of PriorBand, I will increase my score to weak accept but only to that level due to the remaining points.

Re: Official Comment by Reviewer meuK

We thank you for your revision; it is much appreciated. We are eager to address any remaining concerns you may have, in order to enhance your confidence in our work. We also welcome feedback to improve the final draft.