We introduce a technique for tuning the learning rate scale factor of any base optimization algorithm and schedule automatically, which we call \textsc{mechanic}. Our method provides a practical realization of recent theoretical reductions for accomplishing a similar goal in online convex optimization. We rigorously evaluate \textsc{mechanic} on a range of large scale deep learning tasks with varying batch sizes, schedules, and base optimization algorithms. These experiments demonstrate that depending on the problem, \textsc{mechanic} either comes very close to, matches or even improves upon manual tuning of learning rates.
Paper
Similar papers
Peer review
Summary
This paper focuses on a method (MECHANIC) for tuning the learning rate of any given base optimizer. This is compatible with any given base optimizer along with a learning rate schedule*. Let u_1, u_2,..u_t be the steps of the base optimizer up until now, then MECHANIC chooses a s_t such that the neural network weights are set to w_{init} + s_t (\sum_i u_i). They do this by leveraging a similar scheme developed for online convex optimization. They implement this method with some minor changes to simplify the implementations and improve the convergence speed (by using momentum and a variant of weight decay). The authors show that in experiments on masked language modeling and image classification, using MECHANIC leads to faster optimization (although it does sometimes lead to worse generalization). *One thing to note is that this is not a replacement for learning rate schedulers. In the CIFAR10 plots (Fig 2), there is still a need for learning rate decay. Rather, it seems to help over the tuned baselines for any given learning rate schedule. Typos: Line 32: The end quote is flipped. Line 17 in algorithm: s_{t+1,n} should be s_{t+1,i}. Line 113: Text is missing. Table 3, last line: MAdamW should be M-LION.
Strengths
Improving over tuned baselines is an important contribution. The authors do so for large scale deep learning datasets such as BERT and JFT-300m. It is a bonus that the authors are able to do so by using a theoretically motivated algorithm.
Weaknesses
The results would have been more impressive if the authors had used more recent baselines such as Mosaic ML’s BERT codebase.
Questions
1. Could the authors provide some intuition about the fact that s_t is multiplied to the sum of the steps taken by the network till now rather than the next step (as is done in learning rate schedulers). 2. It is surprising that the learned value of s (Figure 1 and 3) is usually << .1? For comparison against tuned learning rates I would have expected s to be not significantly smaller/larger than 1. Does this suggest that the learning rates being compared to could have been reduced by a factor of 10 without significant effects? Could the authors provide some intuition about the learnt values of s? 3. The main body says that lamba is always set to .01 while Table 2 says that lambda=.1 in one of the cases. Was lambda tuned for MECHANIC? If so, was weight decay tuned for the base optimizers? 4. I may have missed the justification referred to in “This seems like a significant issue to disregard, but we will provide mathematical justification presently.”. Could the authors point where this is? Also, I think the first line of the proof of 3.1 is assuming this i.e. that g_t is independent of s_t. 5. Line 100: What is “s”? The previous lines only had s^{o}.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
Yes.
Summary
This paper proposes a parameter-free technique to tune the learning-rate scale factor automatically, which can be applied to any given base optimization algorithm to match its performance given carefully tuned hyper parameters. This approach is mainly empirical in nature, though grounded in reduction from recent advancements in theoretical understanding of online convex optimization. The implementation is also "scale-free", i.e. invariant to scaling the stochastic gradients $g_t$ by a constant. The authors provide experimental evidence to support their claims, including both training from scratch as well as transfer learning, and across domains and model architectures.
Strengths
With the rapid rise in both demand and costs to train large models, the compounding effects of hyper-parameter tuning transitioned from being a technical inconvenience to a true bottleneck. While there have been some advancements in parameter-free optimization, any practical contribution to this field is highly valuable. A general-use, easy to implement extension to existing optimizers that act as a force multiplier and improves upon their results falls comfortably into this category. In addition to providing some theoretical justification to their method, the authors also provide a detailed, easy-to-follow background to the field and its main results thus far. The paper also include some interesting intuitions by the authors, instead of post-hoc explanations of empirical findings. The experiments performed by the authors include various domains and model architecture, and notably include both pre-training from scratch as well as fine-tuning which are known to behave differently.
Weaknesses
As stated by the authors themselves: their "primary contribution is not new theoretical development, but instead the translation between theory and practice". Since this paper is focused mainly on empirical evidence rather than new theoretical results, the burden of proof grows larger. While the authors do perform various experiments, there is key information that is missing to fully convince in the efficacy of the approach. 1. The experimental setting is lacking some details. Namely, how were the hyper-parameters for the base algorithm chosen? Following existing procedures while altering some hyper-parameters (e.g. dropping the NSP loss) may affect the results. Additionally, the given hyper-parameters are not guaranteed to be optimal in the first place. To make the claim that MECHANIC helps in achieving competitive results to carefully tuned optimizers, the base optimizers should indeed be carefully tuned. 2. Moreover, statistical significance of the results is not assured, as no measure for variance in the results is given. Very often, the advantage of the MECHANIC based training is small enough to potentially be attributed to the inherent noise in the evaluations. 3. It is not clear that all models converged in the presented results as there are missing details. For example, Table 1 provides fine-tuning results of BERT like models on GLUE, which underperform the results reported by Devlin et al. 2018, and Liu et al. 2019. This may be due to the difference in pre-training corpus, or due to undercutting of the models. In the latter case, additional training may have changed the results. 4. There are missing experiments to show how well does MECHANIC operate when the hyper-parameters of the base algorithm are not well tuned. If it fails in such cases, then one still has to tune the parameters of the base algorithm, and gain nothing by using this approach. 5. there are missing ablation tests - what is the effect of $\lambda$ and the effect of $\beta$? Also, what is the effect of learning rate schedulers on the performance of MECHANIC? 6. While the paper focuses on empirical findings, the authors suggest the method should be robust, but with no guarantees. For example, line 194-196 the authors claim that weight decay should not affect the operation of MECHANIC, but provide no proof (lie 195-196) and no empirical evidence. 7. While the method is presented as parameter-free, there is evidence that tuning its hyper-parameters may indeed be required. For example, lines 175-176 discuss the importance of setting $\beta$. While the authors try to obviate the need to tune it, the choice of defaults may not generalize well to other settings. Also, lines 239-240 mention that tuning $\lambda$ may be beneficial. Some additional comments regarding the presentation of the paper: * Algorithm 2 is referred to quite a lot and is being analyzed in the paper. As such, I think it will be beneficial for the reader if it is included in the body of the paper. * In line 49, you state the "Empirical studies often take the route of "hyper-gradient descent". However, this approach is rarely used in practice, and even in the work you cite, most empirical approaches do not follow it, nor compare to it. * The "scale" buy which the paper measure gains is not consistent. When MECHANIC is outperformed by base algorithms it is said to be "quite competitive", while when it has the advantage by smaller absolute gains, it is said to outperform. For example, see lines 247-248 which discuss the results of Figure 2. There are a few typos and styling issues that should be fixed prior to final submission: 1. line 38, there is a missing "to" between "allowed" and "make" 2. line 93, the words "let and" were swapped 3. line 113, **missing key information** - the sentence of how you set the scaling factor is missing "and so by setting <MISSING> To show..." 4. In Table 2, there are at least 5 cases where the base algorithm achieved **identical** results to those of the MECHANIC tuning, while the table gives only the results of the MECHANIC tuning in boldface which is confusing and misleading. 5. line 218, "works" $\rightarrow$ "work" 6. line 219, "not that" is swapped 7. The bibliography style is inconsistent in its presentation. For example, [7,11,14,33] are all cited from CoLT, each with different styling. [20,21,36] are both from NeurIPS but with very different styles as well. Also 35 cites from arXiv differently from e.g. 38. 8. All references in the paper are not "clickable". Namely, figures, table and citation references are not clickable in the PDF
Questions
1. How were the hyper-parameters of the base algorithms set? How are the results change if the hyper-parameter of the base algorithm are under-optimal? Was there a thorough tuning phase to verify the chosen hyper-parameters (including the total number of update steps) of the baseline algorithm were indeed correctly tuned? 2. What is the standard deviation in the given results? 3. What is the effect of $\lambda$ and the effect of $\beta$ ? Also, what is the effect of learning rate schedulers on the performance of MECHANIC?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors have addressed the limitation of their work adequately. Namely, the authors mention the lack of theoretical guarantees or intuition into some of the results, as well as potential future work. There is no ethical considerations relevant in this case.
Summary
The paper proposes a scheme to automatically tune the learning rate scale factor for any gradient-based optimization algorithm. The method can be viewed as a practical realization of recent theorertical results in online convex optimization (OCO), which reduces the problem to minimizing the regret of a one-dimensional OCO algorithm. The authors test their proposal on a range of large-scale deep learning benchmarks, showing competitive performance compared to strong manually tuned baselines. It is also shown that the method can outperform a recently proposed learning rate tuner, D-adaptation, without requiring the modifications accross base optimizers that D-adaptation does.
Strengths
* Strong empirical evaluation, which shows solid results against strong baselines in large-scale contexts where learning rate tuning is known to play a criticial role. * Great potential for practicaly applicability, as it provides an off-the-shelf wrapper that can be used on top of any gradient-based optimization algorithm. * (At least part of) the approach is theoretically motivated -- and subject to theoretical analysis. Given the above, I believe the project can be turned into a very strong submission -- provided the weaknesses below are addressed.
Weaknesses
Despite the paper's great potential, a major reservation I have, which justifies my rating, is about the clarity of the presentation -- at least for those without strong expertise of the relevant literature. * The background section, which motivates the approach, is based on prior work -- specifically, recent progress in parameter-free online convex optimization (such as ref [28]). I think the authors can do a much better job at reproducing and explaining these known results (the fact that presentation is plagued with misprints does not help). For example, while Theorem 1, which bounds the regret of Mechanic in terms of the regrets of Base and Tuner, is relatively clear, the exposition of how it is exploited (end of Section 2.1 and Section 2.2 ) is very unclear to me. To be specific, the right-hand-side in the last equality on line 97, s dot shows up in both regrets, which suggests some sort of tradeoff between both regrets -- I do not follow the reasoning that the problem is completely relegated to Tuner (for example, the optimal s dot on line 101 from the BASE regret term looks distinct from the optimal s dot on line 119 from the tuner regret). Furthermore, the paragraph between l112-l122 completely lost me. I think this would be highly beneficial to rewamp entirely this exposition for the sake of readability, especially for non OCO experts like myself.(I formulate more specific comments / questions in Questions section. * In Section 3 the specific form of Tuner (even in its simplified form after line 148) seems to come out of the blue. I am not sure whether or not it was supposed to follow from the background analysis; if it does, I think being explicit about the transition would be helpful (of course the empiricial justifications of some specific ingredients, as in 3.2 for weight decay, is completely fine).
Questions
I am confused by the notation (line 41 or line 95) suggesting that the MECHANIC iterates are mere interpolations of the initial point and the BASE iterates. It makes it look like one needs to run the base algorithm once completely to be able to define the MECHANIC iterates, which is clearly not the case from Algorithm 1. I see how MECHANIC depends on the BASE *updates* -- but these are built from gradients that depend on previous MECHANIC iterates, not on the BASE iterates. (The authors seem to acknlowedge this abuse of notation in Footnote 1, but I cannot see the mathematical justification that the footnote is referring to). I am not sure about the editorial choice that consists of motivating the algorithm with the OCO analysis. I feel a better choice could be to first introduce the wrapper as an additional optimization step over the scaling factor $s$ with loss $\ell_t(x_1 - s \sum_t \mbox{base updates}_t)$, which I find intuitive and elegant in and of itself -- as it optimizes in the direction of what would have been a good scale in hindsight ; then to introduce OCO subsequently for the sake of the analysis. Miscalleneous: * l38 allowed make <-- allowed to make * Footnote 1: presently <-- below or shortly * Inconsistent index for s_t between l41 and l42. * From l45, I suggest to just create a new "related work" section. An explicit list of contributions in the introduction would also be helpful. * l93 let and <-- and let * l103 approximates of the gradient <-- approximates the gradient * l110: isn't an example of such an advanced algorithm SGD initialized a s=0? (from l83). * l110: A TUNER subscript is missing in the regret formula. * l113 missing words: "so by setting To show" ... * On Algorithm 1 : l6 gk <-- gt. I suggest adding What is zt? x_ref not updated? * Table 1: it would be nice to see standard deviations reported as well * The last mentioned limitation in section A looks like an advantage to me (no validation set required).
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
1 poor
Contribution
4 excellent
Limitations
Limitations adequately acknowledged in the appendix.
Summary
This paper develops a method to automatically determine the learning rate of an optimization algorithm. Their approach, referred to as Mechanic and motivated by theoretical advances in online convex optimization, determines a new learning rate for iteration t using an update that resembles an adagrad update plus an additional decay factor. Their method is tested in deep learning settings ranging from language to large scale vision, where it approaches and sometimes outperforms a "tuned baseline".
Strengths
This paper has many strengths: - The problem that the authors explore is very important to the community. Even if their method is not a universal solution, any sound progress towards this target is potentially valuable. If the community succeeds in removing the need for tuning learning rates, significant resources use will be eliminated. - Mechanic can be applied to any base algorithm, not only in theory but also in practice: the authors show results for Adam, Lion, and SGD. - Mechanic is tested in various different domains from masked language modeling to large scale JFT pre-training. - Mechanic has only two tunable hyper-paramters, s_init and the vector of beta.
Weaknesses
- The central weakness in this paper is the lack of clarity surrounding the "tuned baseline". How was that baseline tuned? What scheduler is used? Is it the best learning rate and if so out of how many searched? It would be great to see a line plot of LR vs. perf for the baseline and then a horizontal line for mechanic. This weakness is why I've listed the soundness as only fair. - There are no ablation studies for the relevant hyperparameters s_init and the betas. - Minor: There are some claims in the paper which I do not believe to be true. For instance, in the conclusion the authors claim that it is infeasible to manually tune a scale factor for each layer. However, there is work, e.g., mu-transfer which aims to specify a "corrected" learning rate for each layer.
Questions
- Different base algorithms and problem settings result in different performance for Mechanic vs. the baseline. Do the authors have any ideas why this may be? - How is performance affected by s_init and the betas. - I was slightly lost in the explanation for how multiple betas were used in the section where the adagrad like update transforms into more of an adam like update. Could the authors extend this trick so that multiple betas could be used simultaneously in other contexts, e.g., for the adam estimators for the params? - Can you explain the tuning that went into the baseline. Ideally a plot of LR vs. perf for the baseline could be contrasted with mechanic. - Why do the authors compare only to D-adapt and not, e.g., DoG https://arxiv.org/abs/2302.12022.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
Yes, the limitations are discussed.
Thanks for your response.
I look forward to seeing the data for the plot that you mention, and think it will be a good contribution to the paper.
Thank you and rebuttal round #2
Absolutely! Here are some additional data points that you requested. We hope that this resolves your concerns. Please let us know if you need any other information. **Plots of learning rate vs performance of BERT models** ``` Adam Optimizer | LR | | 5e-4 | 1e-3 | 2e-3 | 5e-3 | 1e-2 | Mechanic --------------------------------------------------------------- | Acc | BERT-B | 71.1 | 71.5 | 71.5 | 71.5 | 71.3 | 71.7 | Acc | BERT-L | 75.0 | 75.4 | 75.4 | 74.6 | 74.4 | 75.3 ``` ``` Lion Optimizer | LR | | 5e-5 | 1e-4 | 2e-4 | 5e-4 | 1e-3 | Mechanic --------------------------------------------------------------- | Acc | BERT-B | 70.8 | 70.8 | 71.1 | 71.8 | 71.4 | 72.0 | Acc | BERT-L | 75.1 | 75.6 | 75.7 | 74.7 | Diverged | 75.5 ``` **Ablations of s_init, \lambda and num_betas** Since you were interested, we also re-ran the ablations of s_init, \lambda and num_betas using ViT-B/16 on JFT-300M with *M*-Adam . Definitely let us know if you require any other information. ``` | s_init | 1e-8 | 1e-7 | 1e-6 | 1e-5 | 1e-4 --------------------------------------------------------------- | Acc | 49.8 | 49.8 | 49.9 | 49.7 | 49.6 ``` As we expected from theory, changing s_init by even orders of magnitude does not result in much difference in performance. ``` | wd (or \lambda) | 0 | 1e-3 | 1e-2 | 1e-1 | 1e0 --------------------------------------------------------------- | Acc | 49.7 | 49.8 | 49.9 | 49.7 | Diverged ``` We have observed that while \lambda is helpful in stabilizing Mechanic on some problems, as long as it is set to a reasonable value it does not affect performance by a lot. ``` | num_betas | 2 | 4 | 6 | 8 ----------------------------------------- | Acc | 48.9 | 49.5 | 49.9 | 49.6 ``` Interestingly, we find that having training runs with num_betas < 6 not only perform worse but also leads to instabilities throughout the training run (we will include plots in the final paper). However, increasing num_betas beyond 6 does leads to slightly worse performance.
Thanks for the results. I've raised my score. I am wondering, how do you know it's not a matter of the number of betas but rather what the betas are?
Number of betas vs betas
Thank you so much for increasing your score and supporting our work! ```Are you asking about num_betas ablation results?``` You are right that there may be an optimal single beta value which can recover the best results when using multiple betas but we've found that single optimal beta is problem dependent and needs to be tuned. Thus the main advantage is actually just the fact that we don't have to tune it. Since there is not much cost to adding more beta values we specify the end of the range and don't tune the start of the range. For instance, num_betas = 2 means we use betas = [0.9, 0.99] or num_betas = 4 means betas=[0.9, 0.99, 0.999, 0.9999]. This allows mechanic to do as well as any beta in the list. Of course, the “truly optimal” value may be something more like 0.65 and so not be in the list, but we expect choosing the correct order of magnitude to be reasonably good, similar to how a grid search over learning rates will miss the exact optimal rate but still perform well. Please let us know if you had more questions and we hope that you would consider strongly supporting our work and increase the score to "Accept".
Reply to Authors
>3.1 I meant theorem 1. > "The justification is provided in Theorem 1, for which the regret of the base algorithm is measured with gradients evaluated at rescaled iterates rather than the original base iterates. " > "Second, we prove that for any sequence of vectors" Does this is point to a gap between the theorem and the application? or do you mean the theorem is more general then needed? > "or, if you prefer, a learning rate of 1.0" I see, so it starts with a higher learning rate than the base algorithm and then learns to decrease the learning rate in a near optimal way. Then why is the schedule needed at all i.e. why does MECHANIC not decrease the learning rate automatically?
"Does this is point to a gap between the theorem and the application? or do you mean the theorem is more general then needed?" The theorem is more general than needed. It holds for arbitrary sequences of vectors, when in reality we only require it to hold for sequence of vectors that are stochastic gradients of a loss. This is actually pretty common in the analysis of optimization algorithms, although it seems very unintuitive at first blush - you'd think that the supposedly harder setting makes you lose something, but it turns out that the achieved convergence results cannot be improved even if you do assume stochastic structure. Basically, it turns out that the "worst case" sequence of vectors usually actually tends to be a simple stochastic sequence and so you don't gain much by making the stochastic assumption. Does this also clear up the confusion about independence in the first line of the proof? "I see, so it starts with a higher learning rate than the base algorithm and then learns to decrease the learning rate in a near optimal way" Not quite: it starts with a very small learning rate scale of $s_{init}$ (say 1e-8), and then very very quickly increases the value to reach the "correct" scale - so quickly that it appear near immediate in the plots. In practice, it tends to actually overshoot a bit and then come back down. You should compare $s$ with the scale factor employed after tuning (i.e. the maximum learning rate when a schedule is used), which is usually smaller than 0.1 as you observed. One might hope that mechanic would also learn to produce a schedule as well, but in practice including the schedule seems better (and our theorem only shows that it can find the single best scale for any given schedule, not that it will find an entire schedule).
Thank you for your clarifications. Some follow-up questions.
Thank you for your clarifications. Regarding the BERT/RoBERTa results: The results for BERT provided in Table 1 of Devlin et al. 2018 and Table 5 in Liu et al. 2019 are on the test set, and thus are expected to be lower to begin with. Furthermore, since the primary difference between RoBERTa and BERT is the addition of more training data (book corpus), the removal of the NSP loss (which was shown to improve results) and most importantly longer training, MECHANIC results should be best compared to RoBERTa’s validation results as presented in Table 8 of Liu et al. In every instance, RoBERTa outperforms your baseline, leading to my concern that the models may not have fully converged. Regarding the missing experiments, you mentioned that for the baselines, you used well-tuned optimizers (either from the original authors or by performing a grid search). My concern here is the hyperparameters used for the model trained with MECHANIC. If they aren't calibrated correctly from the outset and the result is subpar, then one would need to perform hyperparameter tuning for the base optimizer in any case. Given that many of the experimental results don't show performance enhancement but rather produce results on par with well-tuned models, I wonder about the benefits of using MECHANIC if one still needs to perform this tuning.
answer to follow-up
Thank you for taking a look at our response promptly! We would be more than happy to provide additional clarification. ```Regarding the BERT/RoBERTa results```: We are again confused. We are looking at the first row of Table 5 from Liu et al. They mention that those results are **Single-task single models on dev** and not the test set. This is the exact setting that applies to our results too. We are comparing our results with **BERT-Large** (first row) from that table. ```Regarding why Roberta is not our baseline```: Note that we removed the NSP loss but the similarity of our setting to Roberta ends there. Most notably, we do **not** use 1) dynamic masking 2) Full doc sentences (bert uses short sentence pairs and we do too) 3) training for longer (like you mentioned) and much larger batch size (with differences in adam optimizer). These changes may explain the difference in performance. You are right that comparing to Roberta’s setting may also be a fruitful evaluation of Mechanic but we hope that this does not discredit our results on vanilla BERT (which is also well understood and a lot of optimization papers use as a baseline e.g. LAMB [1]) ```Regarding model not being fully converged```: If your concern is whether we have linear decayed learning rate for the baseline to convergence, then yes we did! If you are asking why we didn’t train for as long as Roberta (or even longer), we would like to emphasize that pre-training runs are **very** expensive and we do train for a fair comparison with a well-known baseline of vanilla BERT for meaningful results. Additionally, if you are interested, we compare Mechanic on variety of other benchmarks too, most notably in figure 2, we compare on IWSLT14 (LSTM) and BookWiki (GPT Transformer) in the language domain. ```Regarding calibration of mechanic hyperparameters```: We don’t follow the concern here. Mechanic is designed to remove the need to tune the learning rate scale factor and ONLY that need. Of course other hyperparameters (such as a schedule or batch size) might still need to be tuned when using mechanic. However, even just removing the need to tune the learning rate scale factor results in a significant saving as this is usually one of the most critical hyperparameters to get right. In fact, since our experiments used only the tuning from the BASE algorithm for these other hyperparameters, the performance of mechanic would only increase if we were to actually tune for mechanic. We mostly opted not to do this so as to illustrate that mechanic’s good performance is due to the automatic scale factor tuning and not some interaction with other hyperparameters. [1]: https://arxiv.org/abs/1904.00962
I thank the authors for their rebuttal and theclarifications, which will indeed be very welcome in the revision. Since clarity and presentation was my only major concern, for what I think is otherwise a strong submission, I've raised my score.
Decision
Accept (poster)