SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models

Summary

This paper proposes a data selection approach to reduce the sample size required to conduct supervised fine-tuning (SFT) of LLMs for specific domains. The method achieves it by approximating the training gradients on the full data using only a subset of data. The experiments were done for SFT tasks, including (1) math problem-solving and (2) clinical text summarization. It was found that with 11% of the original dataset, the yielded model can be comparable to the one trained on the full dataset.

Strengths

The paper provides a clear rationale for selecting data for Sparse Fine-Tuning (SFT) by estimating the full gradients with a subset of samples. It conducts a thorough analysis of various variants of the proposed method and includes experiments across diverse scenarios of its application.

Weaknesses

1. Data selection for machine learning has been explored in the literature. Some related works which this paper omits but may need to compare and discuss include: - influence function based approaches [1,2] - reinforcement learning based approaches [3] - data Shapley [4] 2. The generalizability of the proposed theory remains elusive. A small model is used to approximate the gradients. There are two gaps - The small model can have a distinct loss landscape from the large model. Approximating full gradients of a small model not necessarily extrapolate to a large and different models. - There is no theoretical guidance on how this small model should be like, which architecture, which parameter size, to be competent of approximating the gradients of the large model. Considering the black-box nature of deep learning, especially for LLMs, it seems very hard to build such theoretic foundation for the proposed method. [1] Wang, Z., Zhu, H., Dong, Z., He, X., & Huang, S. L. (2020, April). Less is better: Unweighted data subsampling via influence function. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 6340-6347). [2] Kong, S., Shen, Y., & Huang, L. (2021, October). Resolving training biases via influence-based data relabeling. In International Conference on Learning Representations. [3] Yoon, J., Arik, S., & Pfister, T. (2020, November). Data valuation using reinforcement learning. In International Conference on Machine Learning (pp. 10842-10851). PMLR. [4] Ghorbani, A., & Zou, J. (2019, May). Data shapley: Equitable valuation of data for machine learning. In International conference on machine learning (pp. 2242-2251). PMLR.

Questions

From Figure 4, it is surprising that the proposed method and some baselines can outperform the full data performance with around 38% of the samples. How much benefit can the method bring if we continue to scale the data?

Rating

4

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

do not apply

Summary

This paper introduces a data selection method called Small to Large (S2L), which uses the training trajectory of a small model to build clusters and select data based on these clusters. The method is shown to be effective in two domains, math and medicine, with superior performance than other data selection methods and, oftentimes, better than training on the full dataset. A theoretical guarantee is also provided for the convergence of training based on this data selection method. Extensive analysis demonstrates S2L's effectiveness under various compute budgets and across different model series.

Strengths

- The contribution of this work is indeed unique, as most data selection methods in the LLM era are focused on initial or continual pretraining and IFT, with relatively sparse attention given to SFT data in specific domains. This is important for many real-life LLM applications. - The idea is intuitive, as the training trajectory is informative of data characteristics, and effective. Additional theoretical support is provided for the similar effects during training by in-cluster data and the convergence analysis of data selected by S2L. Extensive experiments and analysis demonstrate the overall effectiveness of this method across various compute budgets (Figure 4) and for various model families like Pythia, Phi, and Llama. - Extensive analysis is also being conducted to better understand the S2L method, specifically the synergy between the cluster and fine-tuned model's embedding in terms of data similarity, as well as the robustness across the length and time frame of the training trajectory.

Weaknesses

- As shown in Figure 4, it is clear that for many methods, the SFT has not yet converged. It would be interesting to see how each method affects the convergence of the SFT process and what the relative accuracy looks like, i.e., whether better performance can be further achieved. - The work studies the Pythia model as the small reference model and multiple models from different model families for task evaluation. It is intriguing to know whether other small models can serve as good reference models, and how the alignment between the pretraining data of the small and large models will affect the effectiveness of this data selection strategy. - In the LLM era, the size of SFT data scales up quickly. It will be interesting to see how this method scales with the SFT data size, for instance, at M/B token scale. - It is still tricky to determine whether the effect of such clusters will be influenced by different learning rate schedulers and optimizers.

Questions

Please refer to weaknesses.

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The work addresses the limitations of the limited domain tested and the limited size (7B) of the model used for evaluation.

Summary

This study presents an innovative technique aimed at improving data efficiency in the supervised fine-tuning of large language models for niche domains. The method leverages the training trajectories of smaller models to inform the data selection for larger models, thereby maximizing the utility of the available data. Experimental outcomes on tasks such as mathematical problem-solving and clinical text summarization demonstrate that this technique can significantly reduce the training dataset size while delivering better performance than other standard approaches.

Strengths

- The paper addresses a critical objective: enhancing the efficiency of large models through data selection, utilizing a smaller model to streamline the selection process. - Extensive experiments are conducted to demonstrate the effectiveness of this approach. - The paper is well-structured, with the method and results clearly presented and supported by theoretical analysis.

Weaknesses

- Overclaim: The paper showcases experiments on only two specialized domain datasets, yet it is presented as if the method is universally applicable. While the introduction offers some context, the title remains overly broad, and the methodology lacks domain-specific adaptations. The authors should refine the framing of their contribution to more accurately reflect its actual scope. - Marginal Novelty: Data selection using proxy models is not a new concept, as highlighted by reference [12]. Additionally, the use of training trajectories has been explored in prior studies, such as [1]. - Limited Evaluation: It would be beneficial to conduct experiments with larger language models beyond the 7B scale to validate the proposed method's effectiveness. The results only suggest that the method may be useful for smaller language models. Also, it would be better to calculate the win-rate for the summarization dataset. [1] LESS: Selecting Influential Data for Targeted Instruction Tuning

Questions

1. I am still not sure why using a small fraction of data is better than the performance of using full data on NUMGLUE and MATHEMATICS. Can authors provide more insights on this? 2. How to determine the number of K for clustering?

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

2

Limitations

Yes

Summary

This paper addresses the challenge of data selection in supervised fine-tuning (SFT) of pretrained language models by introducing a novel method called SmallToLarge (S2L). The S2L method involves collecting loss trajectories from a smaller model, clustering these trajectories, and resampling the SFT data to ensure balanced representation across clusters within the given data budget. The authors tested S2L on tasks such as mathematical reasoning and clinical data summarization, demonstrating superior results compared to several existing offline and online data selection methods.

Strengths

1. The motivation of this study is clear, and the proposed S2L method enhances data efficiency in supervised fine-tuning (SFT) for specialized domains. 2. The writing is overall clear and the paper is easy to read. 3. Extensive experiments are conducted to validate the efficacy of the proposed approach. 4. The authors provide detailed analysis about why this method works.

Weaknesses

1. The proposed method may require training the model for multiple rounds, which may not scale up to datasets with millions of examples. 2. The proposed method introduce several additional hyperparameters such as the K in k-means clustering, the training epochs, but the effect of these components are not studied. 3. The connection between the theory and the proposed method is not clear. More explanation on this matter would be appreciated.

Questions

See above

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Yes