Summary
Review Summary:
The paper explores the concept of clustering data into skills to establish a curriculum, aiming to increase the efficiency of language model (LM) training. The concept of a 'skill' is introduced as a unit of behavior with a corresponding data distribution. A skill, in this context, signifies that a model's performance improves on other data that assess a similar skill when it is trained on a dataset associated with that skill.
Attempts to use embedding-based clustering for the unsupervised recovery of the curriculum did not yield desired results. However, loss-based clustering proved to be more successful, establishing that validation loss points from the same skill tend to follow similar trajectories. These trajectories were used to cluster by loss per timesteps per run.
Equipped with this understanding, the authors suggest it is possible to generate an ordered skill set and extract a curriculum from it. The existence of such an ordered skill set in real data is demonstrated in the paper.
The paper also presents 'Skill-It', an approach to dynamically create training datasets, adjusting mixture weights based on expected improvements in generalization on the target task. However, the efficacy of this method is highly contingent on the accuracy of the transfer matrix $A$ approximation.
The method employed in the paper consists of two steps:
1. Learning $A$ via brute force or linear approximation, and by approximating $A$ with a smaller model to reduce computational cost.
2. Implementing 'Skill-It' to dynamically sample training data based on the evolution of losses.
Using synthetic and real data, the authors show that such ordered skill sets exist and can enable more data-efficient training. However, the paper also acknowledges that these ordered skill sets do not align with intuitive data groupings based on metadata and embedding clustering. Lastly, a novel online data sampling algorithm, SKILL-IT, is introduced for more efficient learning of skills. The authors demonstrate the effectiveness of their approach with improved accuracy in various training scenarios.
Strengths
1. Theoretical Soundness: The paper presents a robust framework based on the innovative concept of 'skills'. The methodology, which encompasses learning an approximation of the transfer matrix $A$, skill-stratified sampling, and the implementation of 'Skill-It', appears to be well-constructed and theoretically sound.
2. Clarity of Presentation: The writing in the paper is clear, making the complex concepts and methodology accessible and comprehensible. The authors effectively communicate their ideas.
3. Potential Impact: If the proposed method is effectively evaluated and shown to significantly surpass existing baselines, this work could potentially make a substantial contribution to the field of language model training. The concepts of ordered skill sets and their application for more efficient, skill-based training could lead to improved performance of language models and open up new avenues for research in this area.
Weaknesses
Methodological Issues:
1. Overlooking Computational Requirements: As the primary goal of this paper revolves around reducing computational expense, it's essential to consider the additional compute requirements of the proposed methods (e.g., graph learning part) in the evaluation of the results.
2. Emphasis on a Singular Experiment: The additional computational requirements could be offset by significant improvements in larger scale models, making the 125M to 1B experiment particularly important. However, this leaves only the results from Figure 7 / Table 8 and Table 9 as practically relevant for real-data experiments.
3. Computational Cost and Hyperparameters: The methodology is not without its complexities, including a fixed computational cost to learn $A$ and the introduction of a new hyperparameter, $\nabla$. These increase the complexity of the method, making it necessary to demonstrate a strong outperformance over the random baseline to justify their inclusion.
Experimental Shortcomings:
1. Unspecified Deviations: The authors should clarify whether the deviations reported in the tables are standard errors or standard deviations. Without this clarification, the statistical significance of the results remains uncertain.
2. Missing Data in Section 5.1: The lack of standard deviations or standard errors in the main text necessitates referring to the appendix for assessing the significance of results. For the Addition task, the paper does not sufficiently address the observed equivalence between Skill-It, Curriculum, and Anti-Curriculum. In the case of Natural Instructions (real-data), it's uncertain whether Skill-It significantly outperforms random.
3. Incomplete Reporting in Sections 5.2 and 5.3: The omission of standard errors or deviations in Section 5.2, along with the absence of a random baseline, makes it challenging to evaluate the results. Similarly, for Section 5.3 (Figure 6 / Table 7 and Figure 7 / Table 8), it's unclear whether the results are statistically significant due to the absence of reported standard errors or deviations.
4. Need for Statistical Tests in E.4: As the E.4 experiment is likely the one where the initial cost of computing $A$ can be minimized, it's critical to determine if Skill-It is significantly better than skill-stratified sampling. This requires performing appropriate statistical tests with a sufficient number of trials.
5. Absence of Hyperparameter Ablation for Skill-It: While there's an ablation study for $\nabla$ in E.5, one for Skill-It is notably absent. The authors need to demonstrate that Skill-It isn't hypersensitive to its hyperparameters, or otherwise recognize the associated costs of hyperparameter tuning.
6. Limited Comparison to Stronger Baselines: Even without being experts in curriculum learning, one may wonder why the proposed method isn't benchmarked against stronger baselines from this field. This comparison would be valuable in further establishing the strengths and weaknesses of the proposed method.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The limitations aren't discussed.
I suggest some in the Weaknesses section.