Improving Language Models via Plug-and-Play Retrieval Feedback

Response to Reviewer fuhX [1/2]

Dear Reviewer fuhX, Thanks very much for your review! Here are the responses to your questions/weakness. - Q1: Pease explian why you chose text-davinci-003 and Code-Davinci-002 (Codex) as the backbone language models? Backbone language models are limited, and it is difficult to determine whether the effect is due to the approches or the emergent nature of the language model. Thank you for highlighting the importance of backbone model selection. Our initial decision to use Davinci-003 and Codex as backbone models was made to ensure a fair comparison with baseline methods (e.g., GenRead, RePLUG) that we have discussed in the paper. Besides, the ChatGPT and GPT-4 system might involve many user data for instruction tuning, which could have contamination with many academic benchmark dataset, such as NQ and TriviaQA. Lastly, in OpenAI’s announcement that both ChatGPT and GPT-4 will be subject to ongoing updates in their model parameters. These continual modifications would lead to non-reproducible experiments, potentially compromising the reliability of our research outcomes. In response to your concern, we have conducted additional experiments using ChatGPT (gpt-3.5 turbo) and GPT-4, which reinforce the adaptability and effectiveness of our approach across different model architectures. These results are now detailed in the revised version of our paper, providing a comprehensive view of our method's applicability. | Experiment on ChatGPT (gpt-3.5-turbo) | NQ | TriviaQA | HotpotQA | |--------------------------------------------------------|-------------|-------------|-------------| | Closebook question answering (Question -> Answer) | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 | | Retrieve-then-Read (Question -> Retrieval -> Answer) | 34.3 / 41.5 | 58.7 / 63.7 | 31.7 / 33.6 | | ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.5 / 48.1 | 66.3 / 71.1 | 34.1 / 36.0 | | Experiment on GPT4 (gpt-4) | NQ | TriviaQA | HotpotQA | |--------------------------------------------------------|-------------|-------------|-------------| | Closebook question answering (Question -> Answer) | 34.8 / 49.6 | 64.6 / 72.8 | 30.8 / 33.7 | | Retrieve-then-Read (Question -> Retrieval -> Answer) | 32.5 / 46.5 | 59.9 / 67.1 | 31.6 / 37.5 | | ReFeed (Question -> Answer -> Retrieval -> Refinement) | 36.8 / 54.4 | 66.3 / 74.0 | 36.9 / 42.6 | From the above table, we can obserbe: (1) The retrieval-feedback consistently improves the system, compared to both standard closebook QA method, and retrieve-then-read pipeline. (2) The traditional retrieve-then-read system even sometimes hurt by noisy retrieval. On the contrary, our ReFeed could avoid misled issue if the first-round generation is good. - Q2: Can REFEED avoid generating incorrect or hallucinated information? Yes, this is the main contribution of ReFeed, avoiding hallucination via using retrieval feedback to validate the correctness of generated output. The evaluations are mainly based on open-domain factual QA, the improvement reveals the ReFeed could greatly reduce the hallucination / improve factuality. Figure 5 in the paper show some cased scenarios. - Q3: Reproducibility, limitations, and lack of qualitative evaluation do not confirm its validity. Table 1 is inadequate because the settings are not clear. Thank you for your suggestions. We acknowledge these aspects are crucial for confirming the validity of our research. Regarding reproducibility, while this was not included in the main paper, comprehensive details have been provided in Appendices A.1 and A.2. Additionally, to further ensure reproducibility, we will make all codes and data publicly available for sure. Regarding the limitations of our study, we have addressed this by adding a dedicated section on page 9 of our paper. This section aims to provide a transparent overview of the potential constraints and boundaries of our research. Regarding qualitative evaluation, all methods were evaluated in a few-shot QA/generation setting. Although the domains vary, with focusing on commonsense QA [1] and on open-domain QA [2], methods share the objective of using feedback to improve the performance of large language models like GPT-3. Table 1 mainly reveals the main differences in method design and application domain, without mentioning their implementation details. [1] Rethinking with retrieval: Faithful large language model inference. [2] Check your facts and try again: Improving large language models with external knowledge and automated feedback.

Response to Reviewer fuhX [2/2]

- Q4: Related work discussion on Prompt tuning, Chain-of-Thought (CoT), Tree of Thoughts (ToT), and, Retrieval Augmented Generation (RAG). Thank you for incorporating the related work discussion into your paper. Here's a polished summary of the differences between your work and existing approaches: First, in comparison to Retrieval-Augmented Generation (RAG), our \textsc{ReFeed} method also leverages retrieval as a crucial component of our pipeline. However, unlike conventional methods that directly use retrieval to enhance model performance in complex reasoning and factual accuracy, our research introduces a novel application. We demonstrate that retrieved documents can be effectively employed as feedback to refine language model outputs, significantly improving factual accuracy. Second, our approach diverges from recent advancements such as Chain-of-Thought, Tree-of-Thought reasoning, and Self-Refinement. These methods do not utilize external knowledge to augment language model reasoning and factual accuracy. Instead, they concentrate on improving the language model's reasoning abilities through diverse prompt designs.

Response to Reviewer Jo78

Dear Reviewer Jo78, Thanks very much for your review! Here are the responses to your questions/weakness. - Q1: The paper's novelty is in question, as the process of using an initial output for retrieval and then refining it does not appear to be a novel approach, and the absence of citation to related work, such as [1], raises concerns. Additionally, the paper lacks in-depth insights into the understanding of retrieve-then-read pipelines. The paper altogether seems more like a system report that shows the effectiveness of each trick, not an academic research. We acknowledge the concerns regarding the recent work by Jiang et al. on Active Retrieval Augmented Generation (2023). **It's important to note that our paper was published within 10 days on a third-party open-access repository. Due to the anonymity policy, we cannot specify the exact repository and date, but we emphasize that our work was indeed novel at the time of its publication and is contemporary with Jiang et al.'s research.** Despite the similarities in the pipeline (generate-retrieve-regenerate) between Active RAG and our ReFeed approach, we offer distinct contributions. Firstly, one-pass decoding in Active RAG may limit potential matches to diverse relevant information. Our research demonstrates that multi-pass decoding, or diverse generation, can facilitate more comprehensive retrieval feedback based on varying outputs. Secondly, we address a significant issue in the pipeline where retrieved documents might mislead the language model, leading to a correct output being inaccurately revised. We propose an ensemble strategy that allows for the re-evaluation of answer trustworthiness, thereby enhancing the reliability of the output. This aspect of our research presents a novel approach to managing the complexities inherent in retrieve-then-read pipelines. - Q2: How are the two enhanced modules integrated together? The two enhanced modules are built in two different steps. First, the diverse generation is built in the first step when generating initial answer, instead of generating only one answer, the system generates multiple diverse answer via sampling. Then, we use each generated initial answer to retrieve a set of documents and remove the duplicated documents. Lastly, the ensemble method is built in the second step when producing the final answer. The final output will be based on all initial answer likelihood and refined answer likelihood. - Q3: Typos. Thank you for pointing out the typos. We have revised them and updated the paper accordingly.

Response to Reviewer KK6a

Dear Reviewer KK6a, Thanks very much for your review! Here are the responses to your questions/weakness. - Q1: It is unclear whether the proposed approach would work with another backbone than davinci. How would the approach generalize for other models than davinci? Thank you for highlighting the importance of backbone model selection. Our initial decision to use Davinci-003 and Codex as backbone models was made to ensure a fair comparison with baseline methods (e.g., GenRead, RePLUG) that we have discussed in the paper. Besides, the ChatGPT and GPT-4 system might involve many user data for instruction tuning, which could have contamination with many academic benchmark dataset, such as NQ and TriviaQA. Lastly, in OpenAI’s announcement that both ChatGPT and GPT-4 will be subject to ongoing updates in their model parameters. These continual modifications would lead to non-reproducible experiments, potentially compromising the reliability of our research outcomes. In response to your concern, we have conducted additional experiments using ChatGPT (gpt-3.5 turbo) and GPT-4, which reinforce the adaptability and effectiveness of our approach across different model architectures. These results are now detailed in the revised version of our paper, providing a comprehensive view of our method's applicability. | Experiment on ChatGPT (gpt-3.5-turbo) | NQ | TriviaQA | HotpotQA | |--------------------------------------------------------|-------------|-------------|-------------| | Closebook question answering (Question -> Answer) | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 | | Retrieve-then-Read (Question -> Retrieval -> Answer) | 34.3 / 41.5 | 58.7 / 63.7 | 31.7 / 33.6 | | ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.5 / 48.1 | 66.3 / 71.1 | 34.1 / 36.0 | | Experiment on GPT4 (gpt-4) | NQ | TriviaQA | HotpotQA | |--------------------------------------------------------|-------------|-------------|-------------| | Closebook question answering (Question -> Answer) | 34.8 / 49.6 | 64.6 / 72.8 | 30.8 / 33.7 | | Retrieve-then-Read (Question -> Retrieval -> Answer) | 32.5 / 46.5 | 59.9 / 67.1 | 31.6 / 37.5 | | ReFeed (Question -> Answer -> Retrieval -> Refinement) | 36.8 / 54.4 | 66.3 / 74.0 | 36.9 / 42.6 | From the above table, we can obserbe: (1) The retrieval-feedback consistently improves the system, compared to both standard closebook QA method, and retrieve-then-read pipeline. (2) The traditional retrieve-then-read system even sometimes hurt by noisy retrieval. On the contrary, our ReFeed could avoid misled issue if the first-round generation is good. - Q2: I'm skeptical that taking average probabilities of the output before and after the refinement would work in all models. How would verify that the LLM is well-calibrated in order to decide whether a refinement is more plausible or not? If the model is not well-calibrated, how would you proceed? Thanks for your question! Indeed, there is a fundamental limitation in ensuring that the language model is appropriately calibrated. And we admit our method should be based on a relatively well-calibrated language model. If the model is not well-calibrated, one way is to better measure uncertainty using semantic-based metrics [1], instead of using probability only. We add a limitation discussion in the paper to point out the issue and remind our ensemble strategy could only be applied on well-calibrated model. Besides that, our main retrieval feedback framework is not relying on model calibration.

Response to Reviewer N6CA [1/2]

Dear Reviewer N6CA, Thanks very much for your review! Here are the responses to your questions/weakness. - Q1: Following W1, I would encourage the authors to describe the methods with details and motivation, especially when each component is shown effective individually. For instance, I wonder how the de-duplication is done to ensure diversity; and why BM25 is chosen instead of other retrieval methods (of course not just answering "other methods also just used it"). We appreciate your emphasis on the need for detailed methodological descriptions. Regarding the de-duplication process, we first use each generated initial answer to retrieve a set of documents. For each document retrieved, we calculate a corresponding similarity score. Subsequently, we merge all retrieved documents and rank them based on their similarity scores, keeping only the top-k documents. This approach ensures that we maintain diversity in the information we gather and use. There are two main reasons for choosing BM25 as our retrieval method. First, BM25 demonstrates excellent adaptability and generalizability across a variety of contexts. Second, dense retrieval methods, such as DPR, are often trained with QA datasets or synthetic query-document pairs. Therefore, evaluating such dense retrieval methods might disproportionately reflect their ability to provide retrieval feedback, as the retrieval more accurate than BM25. However, demonstrating that BM25 performs well within our retrieval feedback framework suggests that there is potential for significant improvement with more advanced retrieval methods. Lastly, we want to summarize out motivation of each component: (1) General Retrieval Feedback Pipeline: Language models often generate information that is either incorrect or hallucinated. The retrieved information allows the language model to reassess the initial outputs and, if necessary, refine them to produce new answers. (2) The diverse retrieval strategy enhances the diversity of the outputs and enables more comprehensive retrieval feedback based on these varied outputs. (3) Ensemble method is motivated that there are times when the retrieved documents may mislead the language model, leading to a correct output being revised into an incorrect one. So, the ensemble technique takes into account both the initial and refined outputs, ultimately enhancing the overall performance. - Q2: Following W2, I would really like to know how the proposed method can be applied to state-of-the-art models like GPT-4. Although it might not be reproducible for a certain metric number, it can still give some signs about the performance upper bound. Similarly, I wonder if the method can be applied to a conventional smaller neural language model (e.g., BART and T5) since the proposed is a plug-and-play method. Thank you for highlighting the importance of backbone model selection. Our initial decision to use Davinci-003 and Codex as backbone models was made to ensure a fair comparison with baseline methods (e.g., GenRead, RePLUG) that we have discussed in the paper. Besides, the ChatGPT and GPT-4 system might involve many user data for instruction tuning, which could have contamination with many academic benchmark dataset, such as NQ and TriviaQA. Lastly, in OpenAI’s announcement that both ChatGPT and GPT-4 will be subject to ongoing updates in their model parameters. These continual modifications would lead to non-reproducible experiments, potentially compromising the reliability of our research outcomes. In response to your concern, we have conducted additional experiments using ChatGPT (gpt-3.5 turbo) and GPT-4, which reinforce the adaptability and effectiveness of our approach across different model architectures. These results are now detailed in the revised version of our paper, providing a comprehensive view of our method's applicability. | Experiment on ChatGPT (gpt-3.5-turbo) | NQ | TriviaQA | HotpotQA | |--------------------------------------------------------|-------------|-------------|-------------| | Closebook question answering (Question -> Answer) | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 | | Retrieve-then-Read (Question -> Retrieval -> Answer) | 34.3 / 41.5 | 58.7 / 63.7 | 31.7 / 33.6 | | ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.5 / 48.1 | 66.3 / 71.1 | 34.1 / 36.0 | | Experiment on GPT4 (gpt-4) | NQ | TriviaQA | HotpotQA | |--------------------------------------------------------|-------------|-------------|-------------| | Closebook question answering (Question -> Answer) | 34.8 / 49.6 | 64.6 / 72.8 | 30.8 / 33.7 | | Retrieve-then-Read (Question -> Retrieval -> Answer) | 32.5 / 46.5 | 59.9 / 67.1 | 31.6 / 37.5 | | ReFeed (Question -> Answer -> Retrieval -> Refinement) | 36.8 / 54.4 | 66.3 / 74.0 | 36.9 / 42.6 |

Response to Reviewer N6CA [2/2]

From the above table, we can obserbe: (1) The retrieval-feedback consistently improves the system, compared to both standard closebook QA method, and retrieve-then-read pipeline. (2) The traditional retrieve-then-read system even sometimes hurt by noisy retrieval. On the contrary, our ReFeed could avoid misled issue if the first-round generation is good. - Q3: Following W3, there are still several hyper-parameters (e.g., k in the module-1) in the proposed method. I wonder how they affect the performance (instead of "because other papers use this number"). Previous research, as mentioned in papers [1] and [2], indicates that increasing the value of 'k' beyond 10 leads to only marginal improvements while simultaneously contributing to higher complexity. Additionally, as noted in [3], expanding the context length excessively for LLMs can result in “lost in the middle issue”, i.e., LLM ignores the middle part of a long text input. To further investigate the effect of varying 'k', we conducted experiments using ChatGPT (gpt-3.5-turbo) on datasets such as NQ, TriviaQA, and HotpotQA. The results are shown in the following tables (metric: left EM, right accuracy). The experimental results are consistent with findings in [1] [2]. As shown in the Table when ‘k’ is set as 10 performs significantly better than when ‘k’ is set as 1 or 5, however, when ‘k’ is set as 20, the improvement is marginal over when ‘k’ is set as 10. We also observe that when 'k' is set as a small number, such as 1, the language model could be misled because it tends to trust the only one given documents. | K=1 with ChatGPT (gpt-3.5-turbo) | NQ | TriviaQA | HotpotQA | |--------------------------------------------------------|-------------|-------------|-------------| | Closebook question answering (Question -> Answer) | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 | | Retrieve-then-Read (Question -> Retrieval -> Answer) | 23.6 / 30.0 | 40.5 / 44.8 | 23.2 / 24.0 | | ReFeed (Question -> Answer -> Retrieval -> Refinement) | 34.0 / 41.9 | 65.9 / 68.7 | 27.1 / 28.3 | | K=5 with ChatGPT (gpt-3.5-turbo) | NQ | TriviaQA | HotpotQA | |--------------------------------------------------------|-------------|-------------|-------------| | Closebook question answering (Question -> Answer) | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 | | Retrieve-then-Read (Question -> Retrieval -> Answer) | 30.5 / 37.6 | 50.4 / 55.4 | 28.5 / 29.0 | | ReFeed (Question -> Answer -> Retrieval -> Refinement) | 34.3 / 42.1 | 67.5 / 70.8 | 30.0 / 30.4 | | K=10 with ChatGPT (gpt-3.5-turbo) | NQ | TriviaQA | HotpotQA | |--------------------------------------------------------|-------------|-------------|-------------| | Closebook question answering (Question -> Answer) | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 | | Retrieve-then-Read (Question -> Retrieval -> Answer) | 34.3 / 41.5 | 58.7 / 63.7 | 31.7 / 33.6 | | ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.5 / 48.1 | 66.3 / 71.1 | 34.1 / 36.0 | | K=20 with ChatGPT (gpt-3.5-turbo) | NQ | TriviaQA | HotpotQA | |--------------------------------------------------------|-------------|-------------|-------------| | Closebook question answering (Question -> Answer) | 32.3 / 39.9 | 65.6 / 69.5 | 23.5 / 24.0 | | Retrieve-then-Read (Question -> Retrieval -> Answer) | 34.6 / 42.0 | 59.5 / 64.5 | 31.8 / 33.8 | | ReFeed (Question -> Answer -> Retrieval -> Refinement) | 37.8 / 48.2 | 66.5 / 71.2 | 34.4 / 36.5 | [1] Prompting GPT-3 To Be Reliable. ICLR 2023 [2] REPLUG: Retrieval-Augmented Black-Box Language Models. EMNLP 2023. [3] Lost in the Middle: How Language Models Use Long Contexts. TACL 2023.