Summary
This paper aims to address the retrieval-augmented text generation problem, where the principle of memory retrieval is to select examples similar to the input. The authors are motivated by an observation in their preliminary experiments that the memory examples that better resemble the data distribution during inference come from the model's output, instead of the training data. Therefore, they propose a novel framework, selfmem to address the retrieval-augmented text generation problem. The proposed framework consists of a memory-augmented generator and a memory selector. Differently, the generator can produce multiple candidates to serve as generated memory instead of retrieved memory from the training corpus. The authors experiment on three generation tasks: machine translation, summarization, and dialogue, and demonstrate performance improvement in various evaluation metrics.
Strengths
1. The paper provides sufficient related work, which is helpful for readers to understand the research context.
2. The research motivation is well-supported by preliminary experimental results, which is a good starting point for the research.
3. The paper includes thorough experiments covering representative tasks in the generation field, including machine translation, summarization and dialogue generation, which could be a valuable contribution.
Weaknesses
1. The novelty of the paper is limited. The authors propose a dual structure: Retrieval-augmented Generator and memory selector, each of which is built upon existing paradigms.
2. The overall paper is not well-presented.
(a) Many parts of the paper lack clear descriptions. For example, the setting of the preliminary experimentation is unclear at all.
In L119-120, how does the number 38.89 in Table 1 come from? For example, what does the head of row (Memory, Hypothesis) mean? There is no brief description of this experimental setting in the context. The results in Table 1 are not clear enough to understand the motivation.
(b) Additionally, there are grammar errors and unclear sentence structures in the paper that hinder the reader's comprehension.
In L113-114, "The primary motivation behind our framework stems from the observation that the memory, which is more similar in distribution to the data during inference, is not the training data (38.89 BLEU, as shown in Table 1)." -> "The primary motivation behind our framework stems from the observation that the memory is more similar in distribution to the data during inference, not the training data (38.89 BLEU, as shown in Table 1)."
3. The experimental results are not that convincing as the authors did not provide human evaluation results. Due to the limitations of automatic evaluation metrics, it is usually necessary to conduct human evaluation to assess the quality of generated output in generation tasks.
In this paper, the generation is boosted with the examples that meet the automatic evaluation metrics. This naturally leads to an increase in the numerical values of automatic evaluation metrics. It is still necessary to use human evaluation results to further determine the quality of the generated output.
Questions
Suggestions:
1. In Figure 2, the shading in the square exceeds the boundaries, which needs to be fixed.
2. More captions are needed for Figure 2(b) to better illustrate the content.
Typos & Grammar errors:
1. Please refer to the comments in Weaknesses.
2. In L215, there is an irrelevant word "open" that should be removed.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
I didn't find the relevant discussion is this paper.