Meet in the Middle: A New Pre-training Paradigm

Summary

The paper proposes a method named "Meet in the Middle" (MIM), an novel inference approach for language models. The uniqueness of MIM lies in its bidirectional training—left-to-right and right-to-left—that not only augments data efficiency but also aligns the token distribution for each position. The method improves the performance on infilling task by employing a dual-direction infilling procedure. Empirical evidence gathered from extensive testing on programming and natural languages substantiates MIM's superior performance over traditional pre-training methodologies in both, left-to-right generation and infilling.

Strengths

1. The novelty of the "Meet in the Middle" (MIM) stands out, particularly with respect to the inference phase in text infilling. 2. MIM successfully addresses the important task of general text infilling, encompassing both natural and programming languages. Its universality as an ensemble approach heightens the performance of language models, agnostic of their underlying neural network architecture. 3. The empirical validation of this research is robust. The efficacy of the proposed method is rigorously assessed against multiple recognized benchmarks and contrasted with strong, widely accepted baselines.

Weaknesses

1. One potential drawback of the proposed MIM model is its increased computational demand and memory usage, attributable to the incorporation of two models, as compared to conventional left-to-right LMs. It would enhance the comprehensiveness of the comparison if the MIM model is benchmarked against an ensemble of two left-to-right LMs pretrained with different random seeds. The experiment is useful even on a smaller scale. 2. The visual presentation of the paper could benefit from improvements. The majority of the tables and figures (except figure 1), are inadequately sized for legibility, particularly when printed. This could impede the reader's understanding and appreciation of the presented research.

Questions

N/A

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

N/A

Summary

The paper proposes a new pre-training paradigm for language models that jointly improves the training data efficiency and the capabilities of the LMs in the infilling task. The first technique is a training objective that aligns the predictions of a left-to-right LM with those of a right-to-left LM, trained on the same data but in reverse order. The second technique is a bidirectional inference procedure that enables both LMs to meet in the middle. The paper reports extensive experiments on both programming and natural language models, outperforming strong baselines.

Strengths

1. A nice idea to pretrain from both directions to improve data efficiency through consistency regularizer. The result models are naturally fit for infilling tasks. 2. Strong results compared to FIM baselines with 3+% gain on most tasks evaluated. 3. Clear ablations to understand where the gain comes from by turning off consistency regularizer.

Weaknesses

1. Computational cost: training requires to train two models (forward and backward), which comes with additional computational cost. 2. Evaluation is restricted to coding. While there is perplexity eval on natural languages, it is unclear whether lower perplexity translates to gains in model capabilities such as few-shot learning.

Questions

Quantify how much additional computational cost there is compared to regular left-to-right pretraining. Ablation of using such additional computation to train the left-to-right longer, and compare results to understand whether the gain reported in this paper comes from computational advantages.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper proposes an extension of a paper "Fill in the middle" (FIM) (Bavarian et al 2022). The idea is to define different objective function by considering two models for both directions and to "Meet in the middle" (MIM). One goal is to be more data efficient and the targeted application is in-filling for code generation.

Strengths

This paper proposes an extension of a paper "Fill in the middle" (FIM) (Bavarian et al 2022). The idea is to define different objective function by considering two models for both directions and to "Meet in the middle" (MIM). One goal is to be more data efficient and the targeted application is in-filling for code generation.

Weaknesses

The method is simple, but the motivations remain unclear. For instance, the paragraph on FIM should be improved. If a reader does not already know about FIM, it is difficult to understand. You could provide some equations. Moreover a discussion, on why MIM is simpler (and could be better) is necessary. Moreover my main concern is about the results: at the end why MIM is really more data-efficient that FIM or than a BERT-like trained from scratch of the same task ?

Questions

The first sentence of the abstract sounds really strange: the masked LM objective is used for instance with BERT, and the "B" means bidirectionnal. Beyond this lazy remark, a real discussion in the paper on this bidirectional point could be insightful for the reader. In the related work section the authors say that there is large body of work on bidirectional training. What are the real motivation of this new criterion ? What are the differences with BERT MLM, or ELMO ? What is expected in terms of performance ? - The experimental part focuses on the comparison between FIM and MIM (in the text). The differences with other models are not really commented ? - The models used could be better described: "a decoder only transformer LM". Could you precise the number of layers, heads, ... (maybe in an appendix) ?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Not really.

Summary

The paper proposes a new pretraining objective called meet in the middle. This is a bidirectional objective consisting of forward plus backward logprob and an agreement term measured by total variation distance between the two. The forward and backward models share parameters and optionally attention. Inference is done by generating in two directions until they meet in the middle with agreement, potentially introducting a 2x factor in parallelism and reduced latency. The model are tested on both code generation accuracy and perplexity on several training set and the pile. They compared to both published baselines training on different data as well as FIM trained on the same data, showing strong results.

Strengths

bidirectional agreement objective that is pretty simple and intuitive in some ways, giving strong results compared to FIM. should be easy enough for others to adopt results and comparisons are strong some helpful ablations are included

Weaknesses

"New paradigm" is perhaps exaggerated for introducing bidirectional model via sharing and agreement still more complexity compared to FIM, and some source of improvements are not very clear. For instance, there are several claims of improved data efficiency due to bidirectionally, but the experiments do not seem to test this directly. If this is the main claim, then some direct left to right comparisons would make more sense. the optional improvements seems pretty important and yielded half the improvement over FIM, so perhaps should not be considered optional. For instance, HE infilling results for MIM, MIM-lambda=0, and FIM are respectively 26.3, 24.7 and 22.8. The caption of table 1 says the 26.3 was without the enhancement from 3.2.2 whereas the ablations in table 5 suggest it was. no comparisons with more recent models like santacoder, starcoder

Questions

given that FIM actually conditions on both the prefix and suffix whereas MIM only conditions on one at a time with weak interactions, any discussion on how MIM outperforms despite that? not very clear what is the backup when the model fails to meet in the middle and how false positives are dealt with, especially for long context generations.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

This paper presents a new pre-training approach called MIM for language models, which promotes agreement between left-to-right and right-to-left models regarding their token distribution for each position. The authors have developed a straightforward and effective infilling inference method that utilizes context from both directions and the tendency of forward and backward models to align. The paper pre-trains language models of different sizes on public code and language data using MIM, and the results of various experiments demonstrate the superiority of MIM over existing baselines.

Strengths

- This paper investigates the bidirectional agreement approach for language models, and the results of various experiments demonstrate the superiority of MIM over existing baselines. - The methodology employed in the paper is robust, and the results are statistically significant. The ablation studies conducted in the paper confirm the effectiveness of the primary proposals during both training and inference.

Weaknesses

- The proposed method is not very novel, as similar ideas have been proposed in the NMT field (treated as conditional language models). However, it is still valuable to investigate the impact of these methods on pure language models. - The bidirectional agreement approach leads to more memory cost and training time.

Questions

1. Have you experimented with MIM using larger model scales, such as 7b and 13b? Have you identified any scaling laws for this approach? 2. The predicted probability distribution from different directions may not the identical due to the different contexts. For instance, predicting the first token from a left-to-right perspective may be challenging, but it may be more accurate when using a right-to-left approach. What is your perspective on this? Could you provide more information on the probability differences observed in various positions? 3. Some of the methods previously used in the NMT field may be directly used for language models. Have you evaluated the performance of these methods in your experiments?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

n/a