Scaling Laws for Hyperparameter Optimization

Summary

The authors present a hyperparameter tuning scheme based on optimizing surrogate powerlaws. They claim substantial improvements over baselines on hyperparameter tuning benchmarks.

Strengths

The writing is generally clear, and the method is well described. The authors report substantial improvements over baselines on what appear to be standard benchmarks. Real improvement here would be significant for the field. I should note here that I am unfamiliar with these HPO baselines and benchmarks. While the reported results seem potentially interesting, I place low confidence in my own assessment here. Input from other reviewers with more familiarity with the methods of assessment and the difficulties one encounters when using HPO in practice will be important here.

Weaknesses

It is very hard to assess Hypothesis 1 from Figure 1. This figure seems like it ought to be a final summary figure after some more illustrative figures showing, for example, parameter vs. performance, with five proposed fits overlaid. More generally, the lack of figures explicitly showing powerlaws in this paper about powerlaws is quite odd and makes it hard to gauge the claims made. For example, I'm quite skeptical that *all* metrics one might want to track follow nice powerlaws as the proposed fitting function assumes -- which seems like it'd largely invalidate the proposed HP tuner -- but I could be convinced by lots and lots of plots showing that metric after metric is indeed forecasted nicely by a powerlaw. This is what I'd expect to see here. For a paper using powerlaw scaling, there's a surprising lack of discussion of [Kaplan et al. (2020)](https://arxiv.org/abs/2001.08361) or the Chinchilla scaling laws, which are foundational results dealing with powerlaws + large models used in practice.

Questions

On "Unfortunately, HPO is not yet feasible for Deep Learning (DL) methods" -- what does this mean? Hyperparameters are optimized all the time.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

NA

Summary

AFTER REBUTTAL: I acknowledge reading the rebuttal. After the reviewer discussion, I will keep my score the same. ---- Gray-box hyperparameter optimization involves optimizing neural network hyperparameters by evaluating performance at low budgets and terminating configurations if they seem unpromising. This paper proposes a gray-box scheme using Bayesian optimization over neural scaling laws to estimate future model performance. The key insight is to fit an ensemble of neural networks that can estimate the power law parameters and then apply BO to estimate new hyperparameter settings. Numerical results demonstrate the power law’s ability to forecast as well as state-of-the-art relative regret in HPO.

Strengths

Overall the paper is well-written, clearly motivated, suggests an intuitive strategy, and rigorously experimented.

Weaknesses

- The function $f(\lambda)$ gets overloaded with $f(\lambda, b)$ or as $f(b)$ in various parts. - The definition of the cost function is not present. It would be nice to have a slower explanation of the experiment setup including the cost function and budget, since the plots tend to just look at normalized budget metrics. - The ideas of learning a meta model of power laws, of using Bayesian optimization over power laws, and using an ensemble of power laws have all been explored in recent works. Although these works do not focus on hyperparameter optimization, it would be nice to differentiate the current work from the previous in terms of methods. - Jain, Achin, et al. "A meta-learning approach to predicting performance and data requirements." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. - Tejero, Javier Gamazo, et al. "Full or Weak annotations? An adaptive strategy for budget-constrained annotation campaigns." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. - Mahmood, Rafid, et al. "Optimizing data collection for machine learning." Advances in Neural Information Processing Systems 35 (2022): 29915-29928.

Questions

Please see the weaknesses

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations are not meaningfully discussed.

Summary

The paper proposes a novel surrogate for multi-fidelity HPO that uses an ensemble of power-law models to estimate the future validation loss of hyper-parameter configurations at intermediate stages of training. The principal novelty of the work lies in exploiting the observation (made in other work) that learning curves tend to follow power laws. Thorough experiments are presented across diverse datasets and tasks, comparing against publicly available strong HPO baselines; SOTA performance is demonstrated. Analysis demonstrates an improvement in learning-curve forecasting for the surrogate over non-power law techniques, as well as an increase in efficiency for DPL over HPO baselines.

Strengths

The paper presents a novel contribution to the HPO methodology literature. The claims of the paper are well supported by experiment, and generally well analyzed. The specification of the separate hypotheses of the work in the analysis gives a pleasantly clear structure to the paper. The results demonstrably advance the SOTA in HPO; this work seems very likely to be used and built upon in future HPO work.

Weaknesses

The final analysis section, evaluating the use of DPL for LLMs, is less convincing than the previous sections. Significant work is done by line 319 “We follow the common practice of conducting HPO with small transformers and then deploying the discovered optimal configuration on the full-scale transformers”. Citations of prior work which discuss this convention or apply the ‘common practice’ should be included (perhaps the paper on Tensor Programs V (Yang et al.2022)). Following the convention of that paper, it would be valuable to include the total computational cost of tuning the small models as a proportion of the computational cost of training the large model once (with the small-model-identified parameters). This would give a much better sense of the practicality of applying such a technique in a LLM setting, where the computational burden of training is paramount. Nits: In Section 5.1, the architectures of LCBench and PD1 are mentioned, but for TaskSet no mention of architecture is made. In the LLM section, total parameter count of the various models should be reported.

Questions

The results on PD1 seem weaker than those for LCBench and TaskSet in Figures 2,3,4, and PD1 is left out of Figure 5 and pushed to the appendix. Do you have any analysis or commentary on the relative performance of the method as a function of the architectures or tasks involved in each benchmark? From the description in Sec 5.1, it seems PD1 is a quite diverse benchmark. While the analysis presented quite reasonably focuses on the aggregated performance of each benchmark, was any analysis performed as to the relative performance on specific architectures, tasks, or hyper-parameter spaces?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The practical limitations of the applicability of this work towards LLMs may be under-explored. While the extension of the method towards giant models is not necessary for the impact of this generally strong contribution, it remains unclear if the efficiency gains over existing HPO methods are sufficient to overcome the general concerns of computational inefficiency which have typically precluded Bayesian HPO methods from general use in larger models.

Summary

This paper proposes to combine power law patterns in learning curves, which have been recently popularized by scaling laws, to improve efficiency and performance of Bayesian optimization (BO) based hyperparameter optimization (HPO). Authors provide detailed discussions on how to model the surrogate function and accuracy of their power law based surrogate function in predicting final performance. Tested on three HPO benchmarks, authors empirically demonstrate that DPL achieves better performance given a limited budget compared to other BO-based HPO baseline algorithms.

Strengths

1. I believe the strategy of incorporating power law patterns from scaling laws into the surrogate function of Bayesian optimization is neat and smart in that it enables an improved prediction of learning curves. 2. In addition, authors have empirically justified the soundness of their method through diverse analyses and experiments. 3. Given its simplicity, I expect this method would generalize across different tasks not explored in this paper. 4. Authors described their experiment settings in detail, which was very helpful in evaluating the significance of their method.

Weaknesses

1. I believe the clarify of the paper could be potentially improved. For example, abstract of this paper is too simple, and I wasn't able to grasp the general direction or the concept of the paper by reading them. Also, I believe moving the algorithm box from Appendix to the main text would help readers to better understand their method. I currently see quite a lot of empty spaces in the paper, so expect, with minor formatting efforts, it would be easy to move the algorithm box to the main text.

Questions

1. I am curious about the applicability of DPL to different ML techniques such as random forest or XGBoost, which is arguably the most popular option for tabular data. I am not an expert in this domain, but I doubt the concept of learning curves also plays an important role in those types of models. If not, I believe stating it clear and focusing writing on the area where their method shines most would actually improve the clarify of the paper. 2. I want to know author's opinion on the applicability of their method on training recent large models. The training cost of recent large models can be astronomical, so multiple training runs assumed by Bayesian optimization may not be realistic despite DPL's improved efficiency compared to the baseline. Even though authors included some transformer experiments, I still think the size of their transformers are quite smaller than recent models. This question is not specifically about DPL, but more about BO-based HPO. Any insights would be much appreciated.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

Authors haven't clearly discussed the limitations of their work. Having one or two sentences on the limitations would be helpful for people who are interested in this direction.

Summary

The paper proposes deep power laws ensembles for hyperparameter optimization. More precisely, it is used as a surrogate for Bayesian Optimization (BO) to estimate the performance of a hyperparameter configuration leveraging ensembles of deep power law functions. Furthermore, it is combined with multi-fidelity optimization to estimate the performance for an upcoming budget, enabling incremental training that can be paused for individual configurations based on the performance estimate of the surrogate.

Strengths

### Originality and Quality The presented approach is interesting, as it combines promising directions of HPO: Multi-Fidelity and Bayesian Optimization based on power laws. It is original in the way that it is the first work (to the best of my knowledge) that leverages Deep Ensembles for power laws as surrogates for BO.Moreover, a simple but original multi-fidelity strategy is presented to dynamically adjust the budget a configuration is evaluated on, allowing incremental training of configurations by pausing and continuing the process dependent on the performance estimate of the surrogate. Overall, none of the parts of the presented approach is innovative on its own, but the combination thereof is. ### Clarity The paper is well structured and written in an understandable manner. The different components and their interaction could be clearly separated and emphasized more. ### Significance The presented method is evaluated on 3 state-of-the-art benchmarks, as well as compared to 7 state-of-the-art HPO tools. The evaluation of 4 well selected hypotheses lead to the conclusion that assumptions are accurate, the approach is working correctly, and the presented method outperforms competitors in most cases.

Weaknesses

### Originality The originality of this paper is partly hard to define, as related work or citations are missing at key aspects of this paper. Especially regarding the usage of power laws in the context of HPO, existing work is not investigated / cited [1, 2]. Furthermore, motivation, explanation, and limitation of multi-fidelity and power laws are mixed up (see e.g. l. 57-59). Moreover, the presented multi-fidelity strategy is not put into context of existing strategies. Furthermore, the authors mix up gray-box HPO and multi-fidelity HPO (e.g. l. 95: “Gray-box (multi-fidelity)”). Multi-fidelity can be classified as gray-box, but not the other way around. In addition, the assumption that every learning curve can be described by a power law function should be verified by a reference or two. Overall, the work could be better embedded into existing work on learning curves, see [2]. ### Clarity and Quality The clarity and quality of the paper can be improved. On one hand, the paper should be self contained, which mainly refers to a missing formal introduction and short background of power law functions. Related work and approach description are partially mixed or not suitably placed, e.g. exploiting a power law assumption with the presented method would have been expected to be explained within the description of the method and not as part of related work of multi-fidelity HPO. On the other hand, the mathematical formulation can be improved. Details below: 1. l. 52-53: “[...] the budget is multiplied by the fraction of discarded hyperparameter configurations and the process continues until the maximum budget is reached.” This would reduce the budget with every step, but it should be increased to reach the maximum budget. Suggestion: “[...] budget is divided by the fraction [...]”. 2. l. 57-59: “However, the only assumption these methods make about the learning curve is that it will improve over time. In contrast, we fit surrogates that exploit a power law assumption on the curves.” There is already work leveraging power-laws for performance prediction of learning curves [1, 2]. 3. l. 79f / Eq. 1: Too much space between $\theta^*$ and the next bracket 4. l. 82: The basic definition of $H$ should contain $N$, as it is defined later in Eq. 2 (l. 87f), otherwise multiple usages of $H$ are ambiguous. 5. l. 83: $\mathcal{A}$ is defined, and directly afterwards the normal $A$ is used. Furthermore, the set definition of all possible $\mathcal{A}$ is missing which should be used for $\arg\min$ (in l. 87f) 6. l. 87f / Eq. 2: “$\lambda_i = \mathcal{A}$” is an unusual notation and more common in programming than math, “$\lambda_i =$” could be removed here to avoid confusion. 7. l. 87f / Eq. 2: $\Omega$ should be $\geq$ (greater equals) the sum of the cost, as the budget should not be exceeded, but can be matched. 8. l. 91: $p$ is not defined, as well as the set definition of all possible $\phi$. 9. l. 138: What is “the loss”? 10 l. 201: SMAC is not necessarily an extension of HB, but only the mf part of SMAC uses HB. This should be properly described. Lastly, there is a grammar issue in l. 55: “elaborated” → “elaborated on”, and Figure 5 is misplaced as it belongs to the earlier section. [1] Buratti, B., & Upfal, E. (2019). Ordalia: Deep Learning Hyperparameter Search via Generalization Error Bounds Extrapolation. In 2019 IEEE International Conference on Big Data (Big Data) (pp. 180-187). IEEE. [2] Mohr, F., & van Rijn, J. (2022). Learning Curves for Decision Making in Supervised Machine Learning--A Survey. arXiv preprint arXiv:2201.12150.

Questions

For easier reference during the rebuttal, the questions are enumerated below: 1. Do you agree or disagree with any of the remarks from section Weaknesses? 2. Do you agree or disagree with the suggested corrections of the mathematical formulation listed in the Weaknesses section? 3. l. 171-172: “HPO budget is defined as the maximum number of steps needed to fully evaluate 20 hyperparameter configurations.” How about the overall CPU hours needed to execute the experiments with different approaches? Do they differ? Are there any additional ecological insights, e.g. from a Green AutoML perspective [3], like one might save wall clock time compared to others? Does this relate to performance improvement? 4. l. 205-206: The description of the used hardware is incomplete as memory is missing. 5. I am wondering why there is no bold conclusion for Hypothesis 4? Final remark regarding the overall rating: I am strongly convinced that the weaknesses can be easily addressed during the rebuttal phase. If the authors do so, I am more than happy to increase my score to an accept as I think that the paper is really good besides the weaknesses above. [1] Buratti, B., & Upfal, E. (2019). Ordalia: Deep Learning Hyperparameter Search via Generalization Error Bounds Extrapolation. In 2019 IEEE International Conference on Big Data (Big Data) (pp. 180-187). IEEE. [2] Mohr, F., & van Rijn, J. (2022). Learning Curves for Decision Making in Supervised Machine Learning--A Survey. arXiv preprint arXiv:2201.12150. [3] Tornede, T., Tornede, A., Hanselle, J., Mohr, F., Wever, M., & Hüllermeier, E. (2023). Towards Green Automated Machine Learning: Status Quo and Future Directions. Journal of Artificial Intelligence Research, 77, 427-457.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Limitations are not explicitly given in the paper if I have not overseen something.

Thanks for the rebuttal

Thanks for the rebuttal with clarifications. I maintain my score, voting for the acceptance. Good luck!

Score Increased

Thank you very much for the detailed response! Since the authors answered all of my questions and will adjust for the remaining points, I am more than happy to increase my score.