Data Selection for Language Models via Importance Resampling

Summary

This paper aims to improve the samples to pre-train language models. It propose a novel method called "Data Selection with Importance Resampling (DSIR)" to select better pre-training samples and found superior fine-tuning performance compared to various other approaches, including random-sampling. The authors show the effectiveness of this approach by reporting 2-2.5% better average performance on the GLUE dataset.

Strengths

This paper addresses a crucial issue "how can we make pre-training more efficient?" It is well written, easy to follow, covers relevant related work, and considers various other methods. Their approach effectively improves performance when fine-tuning models on downstream tasks.

Weaknesses

- While show performance improvement, this paper considers GLUE a rather old, general language benchmark. Therefore, this paper evaluates only a part of the possible tasks. Especially in these times when we see the big success of larger language models on such general tasks using in-context learning, it would be interesting to see how models perform on other more challenging tasks - like reasoning intense ones. - This paper only reports the GLUE performance on the dev set. While we see an improvement, it remains to be seen how this transfer to the test set. This is especially relevant since we do not know whether the authors use the dev set to find the best-performing epoch. From the paper, we know that RoBERTa default hyperparameters were used - which includes early stopping on the dev set - but in this paper nothing is stated regarding this issue. - The authors report that domain-specific pre-training heavily harms downstream tasks' performance in the worst case. This also reflects a crucial limitation of this approach; when pre-training a language model, we aim for an optimal model in general without having specific downstream tasks/domains in mind. But with this work, we introduce new dependence to the target domain, which only sometimes matches the full richness of existing and upcoming language properties/domains.

Questions

- How do you see your approach connected to curriculum learning, where we gradually increase the difficulties during pre-training? - Do you think more recent learning paradigms like in-context learning or prompting can gain in the same way as fine-tuning does?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

Limitations are addressed in the paper.

Summary

The work presents a novel framework for effectively selecting a representative document subset during the pre-training of Large Language Models. Pre-training such models incurs significant costs, prompting efforts to minimize the subset size and associated expenses. The authors commence by highlighting the importance of the subject matter and its practical applications. Subsequently, they identify the limitations and characteristics of existing approaches. Building upon the recognized limitations and compelling evidence of Importance Resampling's applicability in the context of Large Models, the authors propose an innovative framework based on KL reduction. In general, the paper has a good idea and a good novelty factor. The authors claim they improved the state-of-the-art on the text classification task, and, indeed, there is strong evidence by the results presented in the paper. In sum, my unique improvement suggestion is to include statistical treatment of the presented results. In summary, this research exhibits a commendable goal and innovative ideas, demonstrating substantial potential. With minor changes addressed, I recommend accepting this paper. I would like to extend my congratulations to the authors for their extensive experimental work and the promising results they have achieved.

Strengths

S1: The paper is well written. The authors clearly define the evaluated objectives, motivation, and contributions. S2: The implementation details and method-specific hyperparameters were defined. Thus, the paper is (possibly) reproducible. Besides, the authors shared their code as supplementary material with the submission in the OpenReview. S3: The considered datasets are well-known and widely used in the literature. ACL, Sci-ERC, ChemProt, RCT, AGNews, HyperPartisan, Helpfulness, and IMDB. S4: The proposed method was fairly compared to strong baselines. Besides, in terms of text classification methods, the proposed method was applied to RoBERTa, a strong SOTA method in the LLM field. Moreover, the baselines were very well explained in a simple and straightforward way. S5: The authors adopted proper metrics to handle and properly measure the effectiveness on both balanced and unbalanced datasets domains (Accuracy and Macro-F1, respectively). S6: Repetition: It is worth noting that the authors adopted a 5-Fold validation procedure, which demonstrates their commitment to rigorously evaluating the proposed framework.

Weaknesses

W1: There is no statistical treatment of the results (e.g., statistical significance tests), which does not allow to rule out the null hypothesis of equality of results. There are strong evidences of the superiority. However, without tests, any claim of superiority can be considered unsubstantiated. To strengthen the research claims and ensure robust conclusions, it is desirable to include appropriate statistical analyses to validate the significance of the reported results.

Questions

Q1: Overall, the paper demonstrates a strong conceptual framework with a commendable novelty factor. However, there is room for improvement, particularly in addressing the previously mentioned weakness. Regarding this weakness, it is worth noting that the authors have already conducted a 5-Fold random validation procedure. As a result, incorporating a statistical method to strengthen the research claims would require minimal additional effort. In light of this, I highly recommend applying a t-test with Bonferroni correction to account for multiple tests. [1,2] [1] Dacrema, M. F., Cremonesi, P., & Jannach, D. (2019). Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM conference on recommender systems (pp. 101–109). [2] Cunha, Washington, et al. "On the cost-effectiveness of neural and non-neural approaches and representations for text classification: A comprehensive comparative study." Information Processing & Management 58.3 (2021): 102481

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

N/A

Summary

The authors present a novel framework for selecting examples from a large and diverse dataset which are most relevant to a specific target domain. They apply this towards data selection for 'continued pretraining' in which a pretrained LM is trained further on a domain-specific dataset, in order to improve its ultimate downstream performance on domain-relevant tasks. They also define a novel data-evaluation metric which is cheap to calculate and correlates well with downstream task performance. They evaluate the proposed method on continued training for RoBERTa against competing methods for 8 tasks, and demonstrate superior average performance. They present additional experiments ablating the target domain dataset, and demonstrate the utility of the method towards data selection for a 'general domain' LM (via heuristic of formality).

Strengths

This paper was a pleasure to read. The overall problem addressed in the paper is important and especially relevant given the preponderance of general LLMs in contemporary research. The relative tractability of this the proposed method at scale makes it more pragmatic than some competing approaches. The paper is clearly written and well structured, each section flows nicely from the last. The results outperform extant methods, and are thoroughly and engagingly analyzed. The proposed method provides a tractable and approachable means of solving a quite general problem, as well as presents many compelling directions for future work in a similar direction.

Weaknesses

DSIR is general to the kind of feature extractor in use, yet in this work only hashed n-gram and unigram feature are explored. It would be instructive as to demonstrate DSIR with different feature spaces. Naively I would assume this is a more interesting ablation than discriminative vs generative approach to learning the importance weight estimator, as this also might demonstrate a tradeoff of performance and efficiency. However, n-grams are simple and a very reasonable place to start. This does not significantly detract from the overall work, and this omission is rightly mentioned in the limitations as being left for future work. Nits: Figure 3 and Table 3 cover the same data, ideally would be presented on the same page. Table 2 and its analysis (lines 199-212) are also on different pages. Figure 3 references the overall avg performance of the values reported in Table 3, please add that as a final column to the Table itself. Additionally, please add a separate row showing the performance of the baseline model with no continued training. Table 3 might be made more readable (depending on execution) by color-coding the relative performance each dataset. As presented, it is very difficult to parse that ChemProt is stronger on avg than Sci-ERC, that IMDB is weak on average, or that IMDB on HyperPartisan is such a negative outlier. Additionally, it might be useful to standardize the column widths and center the text so the main diagonal is more obviously a diagonal.

Questions

In Table 3, the strong performance of the CS-dataset-target models on non-target domains, especially compared to the relatively poor performance of the Reviews datasets, shows asymmetry in the transfer between domains (line 225). This is an interesting finding and could bear slightly more explanation. In the discussion, it is mentioned (line 319) that amplifying biases in the target data is a potential harm. On the other hand, the possibility of mitigating bias might confer a tremendous benefit; what do the authors think of the potential to use DSIR as a means of removing bias from a giant dataset?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The limitations of the work and potential negative societal impact are sufficiently addressed.

Summary

This paper proposes a simple method for selecting pretraining data for language modeling based on the downstream fine-tuning tasks. It uses ngrams as features for a corpus, and weighs the pretraining data based on importance sampling, which estimates how similar the pretraining data is to the task specific fine-tuning data.

Strengths

1. The method is simple and easy to understand. The presentation of the paper is generally easy to follow 2. The paper tested the method on both domain adaptive continued pretraining and training from scratch

Weaknesses

1. The paper uses a simple n-gram feature for importance weighting, but it’s not clear whether other dense features would bring better performance. Even if using dense features is more expensive or unrealistic, it would be helpful to have a comparison on the small scale, or analysis of the differences in computation required for using different features. 2. The paper did not compare to other automatic data selection methods such as 3. The method does seem like a simple/scalable approach, but there is not a good analysis on the resources(memory/training time) needed.

Questions

1. Have you considered comparing to learned data selection method such as DoReMi(https://arxiv.org/pdf/2305.10429.pdf)?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

The authors begin by highlighting an important issue related to data quality in (encoder only) LMs, motivating the need for an improved way to filter large datasets (e.g. The Pile) for samples which are in distribution to a held out target sample. The authors propose a metric, KL-reduction, which quantifies how well matching a target distribution signals downstream performance after fine-tuning on the data. To sample in-distribution points, they compute a importance weight for each sample, by learning a generative distribution for the raw data and target data. Finally they re-sample data using the weights without replacement. Beyond relying on the KL-reduction, the authors finetune RoBERTa on their filtered data (compared to other baseline filtering/sampling methods) and measure performance improvements on the downstream task, showing several percentage point improvements.

Strengths

* Paper is well written and easy to understand; motivation and set up are clear. * The described KL-reduction is a simple metric which appears to strongly correlate with downstream performance after fine-tuning, saving time and compute costs. * Paper shows strong performance improvements on existing datasets/baselines using the proposed strategy

Weaknesses

* The authors only run experiments using Encoder only models, this is in line with a 2021 paper which their benchmarks (data + a model) are based on. However given the recent attention to decoder only models, and the shown impact data quality has on pretraining (see llama / red pajama), an obvious question is if decoder only models also benefit from this low cost strategy. Including decoder/generative experiments may increase the impact of the method. * The task shares similarities with active learning, that should include be included somewhere in the related work section. * The authors only consider n-gram based features, it’s mentioned in the limitations but it likely wouldn’t take much to run the experiments with other features. On Line 173, the authors mention the potential for BERT embeddings, but there are no related experiments. Clearly, the use of just n-grams is effective, but why are n-grams the best way to determine if something is ‘in distribution’ with a target text?

Questions

* TF-IDF (or bm25), i.e. a simple retrieval system, could be used to retrieve documents relevant to the target set sufficient. Line 263 references experiments but the results should be included. Related to this, if a bm25 approach will yield many duplicate documents, why can’t they just be deduplicated? And why does the proposed approach not yield duplicates? * How are the distributions (q(x) and p(x)) on lines 91/92 computed? Elsewhere it is described as a generative distribution, but is it simply parameterized by counting n-grams in the respective datasets?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The discussion of limitations is adequate.

Further discussion

Dear reviewer, may we ask if you could respond to our comments? In our response, we explore the use of neural features, clarify the comparison to retrieval methods, and addressed other detailed concerns. Please let us know if you have other questions or concerns. Thank you!

Further discussion

Dear reviewer, may we ask if you could respond to our comments? In our response, we explore the use of neural features and clarify the memory/time resources for the data selection method. Please let us know if you have other questions or concerns. Thank you!

Further discussion

Dear reviewer, may we ask if you could respond to our comments? In our response, we clarify that we do not use the GLUE validation set to tune hyperparameters, address the generality of having a target distribution, and address other detailed concerns. Please let us know if you have other questions or concerns. Thank you!