Summary
This work re-examines a well-known technique in the NLP literature, called continued pre-training, that can be utilized to enhance the performance of language models on downstream tasks (in this case, for classification and regression tasks).
The authors have revealed that the conventional approach to continued pre-training is ineffective when applied to sentence-pair tasks and prompt-based fine-tuning settings.
Based on their findings, the authors propose a simple method that involves training on the masking language modeling objective while augmenting the text input with prompts.
This proposed approach has demonstrated its effectiveness across a range of tasks and configurations, suggesting that it can be an effective choice for natural language understanding tasks when using Transformer encoder-based models.
Strengths
- The proposed method is technically simple and effective.
- The work begins with a compelling finding that challenges the effectiveness of the conventional approach, which was previously regarded as effective.
- The authors made efforts to provide analysis from multiple perspectives, aiming to convince readers of the effectiveness of the proposed approach.
Weaknesses
- Although the paper offers a detailed explanation and analysis, there are still ambiguous points that need to be clarified for a better understanding by the readers. Let me ask about this point in the following Questions section.
- There is room for improving the grammar and fluency of the paper's writing.
- I would like to see a thorough analysis of the factors that potentially contribute to the success of the proposed method. For instance, it would be valuable to explore the significance of including labels in continued fine-tuning (although partially considered in Section 4.4) and investigate why sentence-pair tasks do not benefit from the conventional TAPT approach.
Questions
To the best of my understanding, it appears that your method incorporates prompts and their corresponding verbalized labels into the process of continued pre-training. If this is the case, there is a possibility that the tokens for verbalized labels are chosen as the target for masked language modeling. This implies that the method might be performing a similar task during both continued pre-training and fine-tuning. Consequently, there is a potential risk that the effectiveness of the proposed method is primarily attributed to unintentionally prolonged fine-tuning. If this turns out to be true, it could diminish the significance of this work.
To alleviate any ambiguity regarding this point, it would be helpful if you could provide concrete examples demonstrating how your method is applied to real data instances. This would offer a clearer understanding of the methodology and help evaluate its effectiveness more accurately.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The evaluation primarily focuses on the utilization of a single model, specifically RoBERTa-large, for relatively straightforward downstream tasks such as classification. These tasks are considered comparatively easier to solve.