BERTs are Generative In-Context Learners

Summary

This paper proposes a finding, indicating that masked language models like BERT can be used as in-context learners. This work suggests that there is potential for a hybrid training, where masked/causal language models can be involved, to take advantage of both objectives. The authors show that masked and generative LMs can outperform each other on different categories of tasks. After reading the authors' response, I think they made some good points and I think I will give accept to this paper.

Strengths

Rethinking and attempting to discover the advantage of masked LM is very impressive, especially in the era of generative large language models, where most researchers put their efforts into improving the generative LLM performance. The figures are pretty clear.

Weaknesses

1. In Figure 1, the authors demonstrate that DeBERTa outperforms GPT-3 in language understanding. But this is more about the natural advantage of the masked language model. The motivations for providing such experimental data are quite vague to me. Additionally, I don't see the advantages of masked models in the other three tasks (Figure 1). In logic, if one method only occasionally shows its advantage, then this method can lack robustness and generalizability. Furthermore, since the masked language models lack large versions (For example 175B), it's quite necessary to justify why use the smaller masked LM over the large and powerful generative LM. 2. This paper mainly uses GPT-3 and DeBERTa to perform experiments. Several issues can weaken this paper. In the era of LLM, GPT-3 can be outdated. There are amount of generative LMs, which are redesigned and well-trained to make them much more lightweight and powerful than GPT-3. Even DeBERTa can demonstrate certain advantages over GPT-3, such experimental results cannot fully support the statement of this paper. In the same way, masked language models have many implementations. The authors need to prove that either DeBERTa is the best masked LM or provide more elements using other masked LMs. 3. As a technical paper, either provide a wide and systemically empirical study or provide a solid mathematical proof.

Questions

As I explained the weaknesses, authors are encouraged to answer the questions I made in the weaknesses section. For details: 1. Which direction do you plan to conduct your research: "either provide a wide and systemically empirical study or provide a solid mathematical proof"? Please specify that and provide the relevant content. 2. At least add Llama-2, Llama-3, Phi-2, Phi-3, Gemma-2B/7B as the exemplar generative LMs. For masked LM, if you cannot prove DeBERTa is the best masked LM (or find literature that supports your model choice), you should at least add BERT, RoBRETa, DistillBERT, T5.

Rating

7

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

1. Lack of experiments to support the statement/conclusion of this paper. 2. Unclear the research conduct path.

Summary

This paper argues that masked language models (MLMs) are just as capable as autoregressive language models at in-context learning. To demonstrate this, the authors propose a generative inference technique that allows DeBERTa to generate text, as well as a hybrid autoregressive/MLM pseudo-log-likelihood estimation technique that allows DeBERTa to rank text sequences by likelihood. DeBERTa and GPT-3 models are compared on a series of NLP benchmarking tasks in an in-context learning setup. While GPT-3 excels at translation and QA, DeBERTa excels at text completion and SuperGLUE tasks.

Strengths

1. This paper argues against the recent exclusive focus on scaling autoregressive language models, showing instead that masked language models also have significant potential as generative in-context learners. Relatedly, it is demonstrated that these two architectures are better for very different types of tasks. This could inform future efforts in training large-scale systems. 2. The limitations are acknowledged explicitly for each of the proposed techniques, and are all written in a constructive way that suggest clear future directions. 3. The writing and visuals are clear and enjoyable to read.

Weaknesses

1. The text ranking procedure seems unfair to autoregressive models: the MLMs are given access to the right context, except for the two tokens that directly follow the one being predicted. This would give them unfair advantages on certain tasks that require more long-term dependency resolution, or which would benefit from lookahead (e.g., syntactic evaluation tasks); thus, the comparison seems a bit unprincipled in these tasks. Would it be possible to devise a technique that does not give the model any of the right context (e.g., shortening the end of the sequence to just [MASK] [MASK] [MASK] [SEP] at each prediction step, instead of giving the rest of the tokens before [SEP])? This would remove any semantic hints. 2. The proposed inference techniques are model-specific. Thus, future efforts using MLMs will need to take into account the quirks of the MLM objective that a particular model uses when writing pseudo-log-likelihood estimation functions. The authors explicitly acknowledge this limitation. 3. Using OPT as a comparison to DeBERTa in the length generalization experiments seems unfair: this model is informally known to perform poorly on many tasks compared to more recent architecturally similar models like Pythia or Llama. Thus, presenting these results side-by-side could mislead readers into believing that MLMs are unilaterally capable of length generalization while autoregressive models are not. I know the stated point of this paper is to compare models released around the same time, but actually, the paper seems to be about comparing different types of LM objectives at performing generative in-context learning. Thus, I think using some of the models from the cited RULER paper would be an acceptable and more fair comparison---or even just a set of Llama or Alpaca models. 4. It is repeatedly stated that hybrid architectures making use of both autoregressive and masked LM elements is a promising future direction. However, no real evidence is given for this. It is true that MLMs and autoregressive models excel at different things, but why would this mean that combining them would give us the best of both worlds, as opposed to the worst (or simply some mean interpolation between their performances, rather than an overall gain over both)? A few papers are cited as examples, but these seem less like hybrid models and more like entirely distinct ideas (e.g., T5 is cited, but encoder-decoder models like these have different pros and cons even relative to encoder-only or decoder-only models).

Questions

Questions: * When computing PLL for sequence ranking, would it be possible to simply remove all of the right context and give three MASK tokens followed by SEP at each probability estimation? * Could you describe some specific ways in which one could combine autoregressive and masked LMs to truly get the best of both worlds? My (admittedly naïve) ideas about how one would do this all involve some mixture of objectives during pre-training. This seems like it would implicitly end up just giving you the advantages of MLMs, since a lot of their behavior comes from the fact that they receive more information (from the right context) than autoregressive systems. Suggestions/Typos: * L15-16: inter alig -> inter alia * When discussing MLM scoring (Salazar et al., 2020), could you also cite the improved version introduced by Kauf & Ivanovna (2023) here: https://aclanthology.org/2023.acl-short.80/

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The limitations of directly comparing models that perform inference with right context versus without could be more directly discussed. Otherwise, I think the authors have done a good job of discussing limitations.

Summary

This paper investigates whether BERT style masked language models can perform in-context learning in multiple LM benchmarks used in the GPT-3 paper, and specifically compares the results of DeBERTa to the GPT-3 model. The paper shows that in multi-choice Q&A, winogrand style quizzes the DeBERTA model can match (sometimes outperforms) GPT-3 model, whereas DeBERTa down performs in machine translation benchmarks. To be able to evaluate these models on generative tasks, the paper also introduces a simple way to use BERT like models in auto-regressive generation, which is itself a significant challenge in the field.

Strengths

- The paper presents extensive results for MLMs for in-context learning which were missing and highly required in the field. - The paper proposes simple (slightly hacky) procedures for (i) auto-regressive sampling from MLMs, (2) calculating logprobs of a sequence of tokens with MLMs.

Weaknesses

- The comparison of sampling procedures missing. I would like to see what scores you would get if you calculate logprobs by just summing individual pseudo-likelihoods of the tokens without doing +2 additional mask tokens. Similarly, what happens if you use Wang and Cho, 2019 method for text generation? - The few-shot scaling curves are not complete: only provided 0-shot, 1-shot, and the best few-shot result.

Questions

- In machine translation benchmarks, were there any examples above L>512, and what the reported results did for those examples for GPT-3? - In Figure 3, you write “[w]e use an open-source replication of GPT-3, OPT, which should perform similarly on this task”. How did you know this? I would delete this claim unless you discuss the evidence. I think it would be unnecessary to discuss it here. So, I suggest removing that claim. - Figure 4 is a little odd. Why don’t you have an actual scaling curve with the actual number of shots but instead have 0, 1, and few labels?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

As noted in the paper, The sampling procedure is too slow to use in practice since for each token the model needs to process the entire sequence again due to bi-directional attention.

Summary

This work proposes a simple modification on masked LMs such that they can be used as a generative way and conduct experiments as generative models. The claim of this work is that, through this modification, masked LMs can be used as in-context learners as GPT-3, adapting to new tasks with further fine-tuning. The experiments demonstrate the performance improvement with DeBERTa and a few categories of NLP tasks.

Strengths

Overall, this work demonstrate the possibility of using masked LMs in a generative way. Although the modification is simple, the experiment results on some benchmark evaluation tasks are impressive. In addition, the method description with limitations show the full picture of the proposed methods, which gives readers a comprehensive understanding about this work.

Weaknesses

Despite of the impressive results, the work and the experiments are still in the preliminary stage, where more work can be done or needed to be done. For example - The proposed modification may be too simple, and more technical modification is needed to improve the efficiency. For example, in section 2.1, the opportunity of providing a more technically solid work is simplified as “We believe that this limitation can be fixed, …” Intuitively, I agree, although I do expect more technical novelty from this part. - Similar concern also applies to section 2.2, where it says “We improve on this behavior by interpolating between …”. Again, more technical discussion and novelty on this part will be deeply appreciated. Without appropriate technical depth, I am wondering what the technical novelty of this work could be. In addition, the experiment design is less convincing. The title and most of the discussion are about masked LMs, while the experiments are only about DeBERTa. There are also some writing issues about this paper, for example, - Although this paper is about in-context learning, the major modification is about turning masked LMs to generative models. So, it’s not clear to me how this can be particularly tied to in-context learning - What the notation means in equations 1 and 2? - I think there may be some typos in equation 2 About experiments - It would be great to list the inference costs of DeBERTa and GPT-3 - What are the values of $k$ for few-shot learning?

Questions

Please refer to the previous section.

Rating

3

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

The paper has listed the limitations of the proposed methods, for example, how to improve the proposed algorithms. However, given the lack of technical depth, some of the listed limitations should be addressed in this work, instead of future work.

thank you

Thank you I read the rebuttal. I will keep my positive score as is.

Thanks for the additional information, still concerned about technical novelty

As I mentioned in the title, I appreciate the additional clarification and results, some of which directly answer my questions. However, I am still concerned about the technical novelty of this work. Since efficiency is the core of generative models, asking the efficiency question when we test the MLM capacity on ICL is reasonable. Therefore, I don't think the "out of scope" argument makes sense.

Rebuttal revisions

We would like to thank the reviewers for their time and valuable comments. Based on the feedback, we have made several minor revisions to our paper, which are summarized here. We believe the paper greatly benefited from incorporating their feedback. * Reviewers *iPtc* and *7Ges* pointed out that the paper is missing evaluation of different sequence-scoring methods. We included an ablation study in the Appendix that quantitatively backs up our proposed method from Section 2.2. * Similarly, reviewer *ayxu* requested a comparison of different generative methods, we conducted another ablation study that is now included in the Appendix. We think that these two extra evaluations are a great addition to the paper. * We fixed a typo in Equation 2, as pointed out by reviewer *iPtc*. * We included a table with all results, together with the values of *k* (number of shots, as requested by reviewer *iPtc*). * Following the feedback from *ayxu*, a more fine-grained evaluation of few-shot abilities on SuperGLUE is put into the Appendix (similar to Figure 3.8 from [Brown et. al (2020)](https://arxiv.org/abs/2005.14165)). * We cite [Kauf & Ivanovna (2023)](https://aclanthology.org/2023.acl-short.80/), after reading this paper suggested by reviewer *7Ges*. We also plan to evaluate this method and include it in the ablation study mentioned in the first point. * We greatly value that the reviews highlighted some potential misunderstandings, we tried to clarify the wording in the key places of the paper. * We added a section with examples of text generation into the Appendix, to better show the generative ability of masked language models. We again follow the [Brown et. al (2020)](https://arxiv.org/abs/2005.14165) for these qualitative tests and show samples from "learning and using novel words" test and "correcting English grammar" test.