Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation

Summary

This paper introduces multiple synergistic tricks that allow for structurally growing networks to be competitive with and even sometimes outperform static networks. These methods include a batched schedule of layer widening, initializing opposing neurons, adapting the scale of the existing weights, handling optimizer and using subnetwork-specific dynamic learning rates.

Strengths

Originality: The paper introduces a novel synergy of methods to support growing a neural network, some of which are novel themselves. Quality: The paper provides experimental results demonstrating that the proposed method achieves comparable or better accuracy than large fixed-size models while saving computation time during training across image classification and machine translation benchmarks and backbones architectures. Clarity: Overall, the authors effectively communicate their ideas, methodologies, and results, making it easier for readers to understand the research. Some inconsistencies are noted below. Significance: This is also the most thorough and practical work on growing models to my knowledge, from shallow and deep CNNs to transformers.

Weaknesses

Please check and fix the formatting of your paper. Table 5 violates the right margin. Table 6 is hard to read without sufficient spacing between the columns. Many figures contain text that is too small and completely illegible when printed: for accessibility, please ensure all fonts within figures are legible at only 100% zoom.

Questions

Figure 1(b), notably the input weights of $\delta+\epsilon_1$ and $\delta+\epsilon_2$, and the caption of Figure 2 suggest you are doing “symmetry breaking”, but the rest of your methods do not mention this and rather emphasize that the new weights are two identical copies. Can you clarify this point? Does symmetry breaking help empirically? When including results of methods from related works, did you reimplement/run them yourself or copy their reported results? In the latter case, you must report any differences in experimental protocols beyond methods, including hyperparameters, optimizers, and software/hardware. The learning rate adaptation method based on weight norm is not very clearly motivated. How was this selected: just on the previous works, or is there further justification? How is this affected by hyperparameter choices such as initialization scheme and weight decay? Other strategies exist that may be more intuitive, such as scaling by gradient norm [1]. Simply applying a subnet-wise cosine annealing may also suffice. Lee, Yoonho, Annie S. Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn. "Surgical Fine-Tuning Improves Adaptation to Distribution Shifts." In The International Conference on Learning Representations. 2023. Check the bibtex entry for Chen et al. (2016). Whenever you refer to the appendix (Lines 116, 156, etc.), please refer to a specific appendix section. Line 204 has a typo: CIAFR -> CIFAR.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No limitations nor potential negative societal impacts are explicitly addressed. Including these discussions would be appreciated.

Summary

This paper studies the growth of neural networks in order to reduce training time. Authors study various optimization challenges when training dynamically grown neural networks and propose some novel techniques to address these challenges. The paper is well written in general, though it requires few clarifications.

Strengths

- Important research topic with a lot of technical complexity. The community would benefit a lot from results and techniques given assuming authors plan to open-source their code and make their results reproducible. - This paper studies the optimization for growing networks, which is an understudied, yet very important topic. - This paper proposes a new initialization technique for growing NNs and combines it with a version of LARS optimizer. - Experimental evaluation include transformers trained on natural language. - Authors achieve real-time speed-ups through additional optimization (i.e. increasing batch size).

Weaknesses

## Major - Comparison with other methods in ImageNet is not ideal since baseline numbers might be different in different codebases. Also ideally one would only change the initialization and keep others parts the same to demonstrate that the proposed initialization is better than the alternatives. Looking at the results, it is not clear whether the alternatives would perform worse with the adaptive learning rate. For example one can run Gradmax with the proposed adaptive learning rate and growth schedule and compare the results. In general it would be nice to give more details about the Imagenet experiments and try applying the proposed improvements to other methods and see whether they also improve performance for them. - “Together, they accelerate training without impairing model accuracy – a result that uniquely separates our approach from competitors.” This is not supported as baselines are not as good as the original baseline for the MobileNet-Cifar100 and Imagenet experiments. Can we train longer to get better results? It would be nice to run these experiments and check whether growing methods can match baseline performance with additional training. - Contributions can be more accurate. (1) Function preserving growth is proposed by many (Openai, gradmax, net2net). (2) Stable training dynamics also exist for these methods. If what authors meant is to be more robust to different learning rates, I believe this is a common feature for adaptive optimizers. (3) “Existing methods [26, 38] fail to derive a systemic way for distributing training resources across a growth trajectory.” Not clear this is true. Authors suggest a different heuristics than the ones used in previous work. I don't think there is enough motivation to claim this particular way being more systematic than the others. At the end they are all heuristics and the one proposed by this paper seem to perform better. I would focus on this part. ## Minor - This paper seems to suggest a new initialization technique for neuron adding. You can also cite Gradmax showing that adding neurons to be more effective than splitting neurons. - I would move VGG models to the appendix. For any practical purposes VGGs are a bit irrelevant at this point. - Table 4: Firefly->FireflyOpt - Figure 3 is very difficult to read in printed form, can you make these plots bigger?

Questions

- Will the code be made public? - How many epochs do you train for ImageNet? - In Equation (5) what is f? Does it return momentum statistics? If so, it might be better to use a different letter instead of a function. - Algorithm 1, what is S_n? Do you mean W_n? - In Tables 2-5, how do you calculate the training cost? It would be nice to add this calculation to the appendix? - In Tables 2-5, how do you generate Net2Net and Firefly results. Do you use the same schedule? If different schedules are used, do you do hyper parameter search for these baselines? - Can you also report training loss in the appendix? Often small datasets have overfitting problem as shown in Figure 3, which is often not the case for larger datasets and it would be nice to know where the gains are. - "We re-initialize the momentum buffer at each growing step when using SGD while preserving it for adaptive optimizers" Why optimizer states have different treatments in Adam and SGD? Can you do an ablation on this? Also do you do the same for the baselines, i.e. Net2Net and Firefly?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

There is no limitation section at the moment. I think inability to match baseline numbers for larger experiments can be mentioned here.

Summary

Starting from an optimization and training dynamics perspective, this paper proposes a new method to efficiently growing neural networks, based on a parameterization that dynamically stabilizes weight, activation, and gradient scaling as the architecture grows over time and different subnetworks take the lion's share of the gradient contribution. To achieve this while preserving functional continuity, the approach presented here grows the network by adding new random untrained neurons with initial net 0 functional contribution, and involves a learning rate scheduling mechanism that rebalances the gradient contribution to different subcomponents of the network added over time. The functional preservation reduces training disruptions and instabilities otherwise introduced by this step-function change in the network.This achieves wall-clock improvement at even or better accuracy than training a large fixed-size model.

Strengths

The biggest strength of the paper is that it takes a reasonable, justified, principled approach to improving upon currently available methodologies in network growth, clearly hypothesizes that it might result in certain expected improvements, and then sets out to test this hypothesis through empirical experimentation. This thought process and methodology is clearly outlined in this well-written paper. The paper compares to various competing baselines, not only to the large full-size model, thus providing an empirical comparison among similar solutions in the literature. In particular, the authors break down the performance of their method into incremental performance gains provided by the various components in the system they designed, as show in Table 6. These ablations help answer questions about the relative important of each design choice. I also found the empirical investigation of the growing schedule quite compelling, as I imagine it would be a point of exploration for any practitioner wanting to implement this methodology.

Weaknesses

The study is limited to simple, low scale image classification and machine translation tasks, which aren't representative of the application domains often facing researchers and practitioners these days. Therefore, it remains unproved whether this method would help in the context of large scale deep learning, which is the context in which the problems the authors are trying to solve actually occur. The disconnect between the problem statement and the regime in which the solution is tested weakens the point of the paper, as many promising approaches at low scales have proven completely ineffective at large scales. One general concern with progressively grown architectures, especially with respect to whether the data distribution is / isn't changed with each growing iteration, is about the generalization properties of the final configuration, whether it loose the generalization properties of the equivalent full-scale architecture when the training data distribution stays constant, or whether it affects the task learning abilities in a continual learning setup.

Questions

Just a suggestion: since Figure 3 is too small to be legible, I'd recommend moving it to the appendix.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

There are no major concerns about societal impact of this work. Authors do not over-claim impact in this paper, and they are quite matter-of-fact about the applicability of their work. The scale limitation of their experiments, however, prevents this work from being more impactful.

Summary

The authors study the network growth problem and propose a novel problem to stabilize the optimization difficulty. The authors conduct experiments on various model architectures (including VGG, ResNet, MobileNet, and Transformer) and observe consistent performance improvements.

Strengths

The studied problem is important. The proposed approach is novel. Experiments with various model architectures are conducted and demonstrate consistent performance improvements.

Weaknesses

The most important application for the proposed algorithm is large-scale training. However, the experiments discussed in this study are based on relatively small models, which left doubts on the applicability of the proposed method on large-scale training.

Questions

Is the Transformer used in Section 4.3 constructed in the Post-LN or Pre-LN manner?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

I suggest conducting experiments with pre-training tasks like BERT, which would provide great value to this study. Also, as to the machine translation task, the IWSLT14 dataset is not as reliable as the WMT14 EN-De/Fr dataset. Some additional related work on this topic are: 1. Shallow-to-Deep Training for Neural Machine Translation; 2. On the transformer growth for progressive bert training.

Summary

This paper developed an approach to efficiently grow neural network architecture by parameterization scheme with regard to weight, activation, and gradient scaling. It employed a learning rate adaptation mechanism rebalancing the gradient contribution of the subnetworks. The experimental results show efficiency in wall-clock training time.

Strengths

- The main technical section is described in detail, although there are some missing points (please refer to the Questions section.) - The paper is easy to follow.

Weaknesses

- Wondering if the authors considered the original training cost of the original backbone network. For example, if the original network costs XXX for training from scratch, how much will the proposed approach will cost when all steps’ costs are aggregated, apples to apples? - There is no description of how to calculate train costs. - It is not clear if the subnetworks are trained at a global learning rate or a different learning rate: at line 58, the authors claimed that subnetworks with different lengths have distinct learning rate schedules, while at line 142, it is noted that subnetworks from different growth stages will share a global learning rate. - It would have been better if the authors provided justification for why they picked the LR adaptation rule and comparisons with other LR adaptation rules. That is because a learning rate schedule (although, in many cases, it is overlooked) can have a high impact on the final performance and result. So it would have been nicer if the authors explained more or showed more clues on that. - In (6), C_0 (base case) is not defined. - In (7), the base case is not defined either. Also, are the parentheses (with floor opening and ceiling closing) for rounding to the nearest odd number? - It needs to be clarified why Figure 7 (b) is measured over time/epoch. In other words, why the time spent per epoch matters. It would have been nicer if the authors had added an explanation for that. - It is not clear the overall benefit of growing a network gradually while preserving the function computed by the model at each growth step. Is there any potential possibility/intention to use such an intermediate model? Otherwise, it is not clear to preserve the functions at each step.

Questions

- In Figure 7 (b), why is it measured over time/epoch? - In (6) and (7), what are the base cases? - Why was the certain learning rate adaptation rule selected?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The authors did not mention anything about limitations. It would have been nicer if they were upfront about any limitations to share.

Summary

This paper presents a method for incrementally growing network width across stages. The reader is shown the details of the growing process using a 3-layer feedforward network as an example, with the appendix providing details for other types of networks. Additional algorithmic details, including learning rate and batch size scheduling are provided. The paper then presents experimental results over 3 repeats, including an ablation study. **Rebuttal:** I have read the rebuttal and it addresses my questions. Based on the novelty and contribution of this work, I maintain my original rating of 6.

Strengths

Section 3 of this paper is perhaps the strongest part of this paper. There is no ambiguity in the presentation, and any questions the reader may have (generalization to Adam and other network architectures) are presented either in the appendix or later in the paper. The paper also does not limit itself to a single architecture, and shows details for two other optimizers, as well as other architecture types. It also provides an ablation study showing how each component of the overall approach is useful.

Weaknesses

A few parts of the paper are a bit terse. In particular, Section 4 of the Appendix could benefit from additional exploration of, e.g., more alternatives or motivation/derivation for using the proposed alternative.

Questions

- In Table 1, I'm not entirely sure I see a clear pattern: if we had one more layer, for example, would the last layer (whose dimensions I'll still call $C^h$) at initialization have the old weights rescaled as $\left(\frac{C^h_t}{C^h_{t+1}}\right)^{(3/2)}$, and the new inits be $\frac{1}{(C^h_{t+1})^3}$? - In L143, you say using a global learning rate causes divergent behavior: have you seen this in practice? Can you show loss plots under this scheme? - In Eq. 5, what is the motivation for setting $f$ to the norm function? Is the Appendix Section 4 what the paper refers to when discussing alternative heuristics (L156)? - In Table 2, what do the underscores represent? - In Tables 2-5, what statistical tests did you run to compare your methods? I'm also not sure 3 repeats is sufficient to capture variance in performance effectively. - In Tables 4-5, please change the training costs of Net2Net to bold (since they were in bold in Tables 2-3). - In Table 4, did Net2Net and Firefly achieve the same performance across all 3 runs, or did you only run each once? - In Tables 2-5, how is training cost computed? Is it FLOPS/wall time/etc?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper's limitations are addressed by providing details for how to overcome them. For example, an ablation study shows the necessity of the learning rate adaptation scheme; the appendix provides details on expansion to residual blocks and convolutional blocks.

Post Rebuttal

I read the author's response, and I maintain my original rating.