Summary
The manuscript focuses on training transformers faster by progressively growing the architecture during training. Problems with previous methods in this area include limited growth "dimensions" (e.g. inability to progressively grow the hidden dimension), absence of function preservation in networks with LayerNorm, and a forced choice between function preservation and initialization flexibility. The proposed method (MSG) addresses these problems, and experiments support its effectiveness (i.e., it accelerates training). Ablation and hyperparameter studies further justify the chosen methodology and offer insights that are helpful for understanding the progressive growth problem more broadly.
Strengths
The proposed progressive growth method speeds up training more than competing approaches, and it maintains or improves accuracy/perplexity performances on test datasets (relative to the model without progressive growth).
The study of the impact of different growth dimensions is broadly helpful and clear (Figure 2). This analysis provides intuition for the MSG approach, making it seem less tailored to perform well on just one dataset/architecture (i.e., more generally applicable).
Care is taken to ensure that comparisons across progressive growth methods are fair and that the baseline progressive growth methods are relevant. In Section 4.2, salient features of progressive growth algorithms are isolated appropriately to facilitate analyses of the proposed method, MSG.
The manuscript offers an original and well-designed approach to disentangling the function preservation and initialization decisions of progressive learning. This facilitates the creation of a more flexible growth algorithm and better analysis of the importance of these algorithm components to progressive learning. For example, the final paragraph of Section 4.4 leverages this disentangling to show that both function preservation and flexible initializations matter to performance, supporting the form of the MSG method.
Weaknesses
While the manuscript makes careful comparisons to competing progressive growth methods, some of the baseline performances (without progressive growth) are a little lower than expected. Please see my comments about this in the "Questions" section below.
Questions
Score-affecting:
1. Please ensure all baseline approaches are faithful so that readers know MSG is effective in high-performing training runs. Examples follow.
- In Tables 4 and 5, the SQuAD scores are 2-3% lower than what is reported in the BERT paper.
- In Table 5, GPT-2 perplexity on Wikitext2 is high (41.3). There are strong public GPT-2 implementations; e.g., https://github.com/karpathy/nanoGPT.
- It would be nice to see an MSG version of the LiGO (Wang et al., 2022) paper's Figure 3C: i.e., show perplexity/loss dynamics of GPT trained with and without MSG.
Helpful:
1. Please summarize the MSG operator conclusions/discussion from the referenced technical report (Li et al., 2023).
2. Consider moving tables and figures to the page they are discussed on, or to the next page. This helps readability.
3. Clean up figures and tables. Some example suggestions follow:
- Table 2 could have uniform line lengths.
- Figure 2 could have larger axis text, lines, legend text, etc.
Minor:
1. The manuscript would benefit from a proofreading. Some examples follow:
- "Proof" is not a verb that means "to mathematically demonstrate". Say "we prove" instead of "we proof". A "proof" is what proves something.
- On page 6, rephrase the second sentence of the "Stage Duration" paragraph -- the training steps are what gets split, not the stage steps.
2. I think that not all the inequalities in the second lines of equations 11 and 14 are strict (i.e., change a "<" to a "<=").
3. All of the methods listed in the Related Work section (progressive growth, weight sharing, MoEs, etc.) are discussed in the survey "Compute Efficient Deep Learning" (Bartoldson et al., 2023).
Rating
6: marginally above the acceptance threshold
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.