Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models

Summary

The paper proposes a new method named LaPael to enhance knowledge injection for large language models. Different from traditional data-level augmentations or noise-based methods, LaPael operates at the latent level, preserving the semantic integrity of the text while introducing meaningful variability. Experimental results on three question-answering datasets show that the proposed method outperforms the baselines.

Strengths

1. The authors propose a latent perturbation for enhancing knowledge injection of LLMs, and the proposed method outperforms the baselines. 2. The paper is written-well, with a direct motivation and a well organized structure.

Weaknesses

1. The difference between the method proposed in this paper and the previous perturbation or enhancement methods in the feature space is not explained clearly. 2. Why is the output distribution of Transformer assumed to be Gaussian in Training Part of Section 4.2? What is the effect of directly calculating the KL divergence of two distributions without considering the type of distribution? The author's explanation will help readers better understand the method. 3. The authors claim that the paraphrasing method requires a high computational cost, so they should compare the proposed method with the paraphrasing method on computational cost.

Questions

1. In Table 7, why was it not compared with the Fine Tuning (+para) method? 2. Will the proposed knowledge injection method have a negative impact on the knowledge already mastered by LLMs?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Please see above Weaknesses and Questions.

Summary

This paper focusses on the ability of LLMs to learn new knowledge. Previous work has shown that paraphrasing techniques help learning this new knowledge. However, as the authors argue, explicit paraphrasing has a high computational cost, and the paraphrased data is of limited diversity. To circumvent this, this paper introduces latent paraphrasing, by adding a latent paraphrasing layer in the standard transformer module, in between the LayerNorm and the MLP. This layer is trained with paraphrased data, generated by an LLM. The optimization objective is to minimize the KL-divergence between the output of a “standard” transformer model with the paraphrases as input, and the output of a transformer including the paraphrasing layer with the original sentences as input. The authors test their approach, called LaPael, on three datasets: SQuAD, StreamingQA, and ArchivalQA, and compare against multiple fine-tuning baselines. They show that LaPael mostly outperforms the baselines, even in cross-domain transfer experiments.

Strengths

* This paper presents a large number of ablation experiments, that show the benefit of each of the components of the proposed approach. These experiments are very insightful, as at first the method might sound a bit complicated. Moreover, the extensive ablation experiments can help steer future research in this direction. * The proposed approach is effective, and outperforms the baselines with relatively large margins.

Weaknesses

* The objective of the paper is to explicitly focus on fine-tuning approaches, instead of approaches that rely on external memory. In that light, it makes sense to only compare against fine-tuning baselines. However, if one is just interested in what method performs best on the task, comparing against external memory based approaches would have been a good addition to the results section. * As a related point, I am wondering to what extent the used datasets can really evaluate new knowledge injection. For example, SQuAD is based on Wikipedia, which the models have probably seen during pretraining. Also see my related question below. * I also have a few other questions about the approach, which I have added in the question section below.

Questions

* The authors mention that they use Vicuna as the base model. It seems unlikely that the model has never seen the new knowledge represented in D_K during pre-training. At the same time, the authors show that the model without adaptation scores much lower than any of the other baselines. This makes me wonder to what extent the authors think that LaPael measures true new knowledge injection in the LLM? Some clarifying questions: * Line 177: regarding the gold mask for tokens that correspond to the named entity. How is determined what the named entities are in the sentence? Do the authors deploy some sort of NER tagger? * Line 193: mentions “rephrasing each question of D_QA”. Should this be “rephrasing each document”? * Line 217: mentions that “the LLM is fine-tuned on the test set”. To confirm — does this refer to the paraphrased version of the test set, D_K? * Line 238: regarding “Combining LaPael and Paraphrases”. I understand the objective of this experiment, and I agree with the authors that it is insightful. I am not entirely sure if I understand the experimental setup here. Specifically, what does “number of paraphrases” mean in the LaPael setting in Figure 4? Is “ours” the default LaPael setup, and “fine-tuning” LaPael + fine-tuning? * Table 7: Can the authors elaborate on how they adapt the baselines in this real-world data scenario?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are addressed in the appendix.

Summary

This paper presents LaPael, a novel approach to injecting new knowledge into Large Language Models (LLMs). Previous works have shown that fine-tuning the model with data augmented by paraphrasing helps the model learn new knowledge. However, this requires high-quality paraphrased data each time new knowledge is injected, which is costly. The paper proposes a method that introduces a paraphraser layer within the LLMs, which acts as a latent paraphraser that paraphrases the input text in the latent space. The authors train the latent paraphraser using paraphrased data. Once the latent paraphraser is trained, it is used to fine-tune the LLM on the new knowledge. LaPael outperforms standard fine-tuning and paraphrasing approaches, showing improvements on question-answering benchmarks.

Strengths

1. The paraphraser layer within LLMs introduces a novel method for incorporating new knowledge, reducing the need for high-quality paraphrased data and reducing the significant cost of fine-tuning the model. 2. The paper demonstrates that LaPael outperforms the supervised fine-tuning method on question-answering benchmarks.

Weaknesses

**The major weakness of the paper lies in the experiment section and the details provided regarding the experiment. Below are my concerns and questions related to this section:** 1. Line 190: "We use the subset of questions in the test set of each dataset for $D_{QA}$" -> Why is only a subset of questions chosen instead of the entire test set? No details about this are mentioned in the paper. 2. Creating a document out of question-and-answer pairs to inject knowledge into the LLM and then testing it on the same test set is inappropriate for evaluating knowledge injection. It would have been more suitable to use datasets with available documents, inject the knowledge from these documents, and then test the model on the QA. Additionally, the documents used should not have been seen by the pretrained LLM during its pretraining phase to quantify how much knowledge has been injected into the LLM. 3. There are no statistics available for the created documents, such as their size. Creating small paragraphs from question-and-answer pairs, fine-tuning the model on these paragraphs, and then testing it on the same question-and-answer pairs does not constitute a correct and fair evaluation. 4. Line 212: Why are in-context examples used instead of experimenting with a zero-shot setting to assess the extent of the model's learned knowledge? 5. Lines 184-185: It is not clear what the authors meant by "We sample N noise." 6. Line 182: "From Equation 10" should be "from Equation 15." **The writing of the paper is also very weak. Here are a few examples:** a. The acronym "LM" is used in the experiment section without being defined. The paper uses "LLM" for Large Language Models and sometimes "LM." b. The paper uses "$D_{knowledge}$" and "$D_{K}$" interchangeably. Please ensure consistency. c. There are missing details in the experiment section, and no correct reference is provided to indicate where in the appendix these details can be found.

Questions

Please refer to the weaknesses section.

Rating

3

Confidence

5

Soundness

1

Presentation

1

Contribution

2

Limitations

NA

Summary

This paper proposed a latent paraphraser to generate paraphrased data which will be used as augmented data in LLMs' fine-tuning. To tackle the challenges of repetitive external model interventions, the latent paraphraser (LaPael) is trained to add a small perturbation at the latent feature level of the LLM and eliminate the need for users to repeatedly paraphrase using LLMs once latent paraphrasers are trained. Specially, with the paired-data as input, i,e., original and paraphrased sentence, the training objective is to minimise the KL divergence between distributions of original sentence after the paraphraser transformation and the distribution of the paraphrased sentence. The experimental results show that LaPael outperforms existing noise-based paraphraser in generating better augmentation data.

Strengths

1. The paper is well writen and easy-to-follow and the research topic of generating augmentation data is important. 2. The empirical results clearly show the superiority of the proposed method, LaPael.

Weaknesses

1. The cost of training the latent paraphraser is not clear, as expensive training will impedes the efficiency of the proposed method. 2. Lack of qualitative analysis of show how the phrased sentences differ from the given paraphrased sentences from GPT3.5.

Questions

1. In table3, the cross-domain transfer: why the results trained on X evaluated on X, is worse than trained on X but evaluated on Y. 2. Any explainations or case study showing the generated paraphase sentences from LaPael? 3. The computation cost comparsion with noise-based paraphraser.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The trade-off between task performance and computation cost.