RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation

Summary

Traditional retrieval-augmented systems often face two main issues: (1) the irrelevant content in retrieved documents may mislead the model and (2) searching with the original query often fails to retrieve adequate information. To this end, this paper introduces a method called Refine Query for Retrieval Augmented Generation (RQ-RAG), which trains a 7B Llama2 model to adaptively rewrite, decompose, and disambiguate the original input query with their collected datasets. They conduct experiments on both single-hop and multi-hop QA datasets. The results show the effectiveness, high potential, and robustness of their method.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

1. Positive experimental results: The method proposed in this paper outperforms existing state-of-the-art retrieval-augmented methods like Self-RAG-7B on multiple datasets, with detailed comparative tests against the baseline of the llama-7B model. 2. Comprehensive experiments: The paper also demonstrates the effectiveness, rationality, and robustness of its method through experiments on the proportion of dataset enhancement, sampling strategies, and the sources of retrieved information.

Reasons to reject

1. The paper proposes three methods of data enhancement: rewrite, decompose, and disambiguate. I am still concerned about how much these three methods actually contribute to performance improvement and whether they are all effective. 2. For the MQA (Multi-Hop QA) task, the paper only compares with the Llama2-7B backbone baseline, which shows significant improvements. However, in the MQA field, knowledge editing is also a widely studied method (e.g., MeLLo [1], PokeMQA [2]). I recommend the authors provide comparisons with these similarly high-performing knowledge editing methods. 3. The refined query method proposed in the paper adds Decomposition and Disambiguation to the queries. Is this effective for every query? Could this approach introduce bias in the processing of some original data? [1] Mquake: Assessing knowledge editing in language models via multi-hop questions [2] PokeMQA: Programmable knowledge editing for Multi-hop Question Answering

Summary

This paper proposes an approach similar to Self-RAG or Self-Ask with specific actions to do multiple steps of retrieval and rewrite queries as needed. This system-like approach is pretty effective on various single-hop and multi-hop QA tasks. Notably, it outperforms Self-RAG on single-hop QA, although the baselines for multihop-QA do not seem particularly strong. The main weakness is the writing is not very clear on the motivations for these actions, nor how their approach is specifically different to previous methods. For example, self-ask is not mentioned as a baseline and already includes multi-step retrieval and question decomposition.

Rating

6

Confidence

5

Ethics flag

1

Reasons to accept

1. The results compared to self-RAG are compelling. Although the multi-hop results seem to be missing key baselines. 2. The ablations are informative for choosing between search engines, although it would be nice to see more discussion about retrieval accuracy across different search engines, such as whether they exhibit similar behavior and prefer different kinds of queries. 3. The method is simple and flexible, depending mostly on synthetic labels. That being said, more could have been discussed about the dataset construction in the main text.

Reasons to reject

1. The paper is not self-contained. Crucial details for the dataset construction are missing or left to the appendix. Details on baselines are not included at all, e.g. how does this differ from Self-RAG? 2. The llama-2-7b baselines for multi-hop QA are not very effective, and it's a bit unclear whether this is because they do not conform to the task well, or they have some lack of capability. 3. The self-ask approach is not discussed, yet it is a well known method for tasks like multi-hop QA with search engines. 4. The Llama-2-7b is a weak model compared to various other models that would have been available at the time, accessible through APIs such as together.ai. For example, llama-2-70b. It is not clear whether their method will still be effective for these stronger options. To make it more clear, the main weakness are the lack of self-ask baseline (and potentially other well known baselines) and general lack of details in the main text. Including those types of details would certainly strengthen the paper, although it may reveal limited novelty or effectiveness of the presented approach.

Questions to authors

Why not try other self-consistency alternatives, such as universal SC? How is PPL different from beam search?

Summary

This paper introduce a new training and inference method for retrieval-augmented generation, which enables an LM learn to retrieve when necessary and re-write compositional and ambiguous queries. The proposed method is based on recent retrieval-augmented instruction-tuning method, particularly Self-RAG (e.g., using special tokens to control model-behaviors). While in Self-RAG, the authors only train models for single-turn settings and reuse the input-output data from existing datasets, this paper re-generate output data based on retrieved documents, which authors find crucial to achieve the strong performance. Experimental results on single-hop and multi-hop queries indicate the effectiveness of proposed method.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- The proposed method is well-motivated and carefully designed. - While there are several work that leverages existing datasets for retrieval-augmented instruction-tuning, the authors empirically explore answer regeneration strategy that regenerates outputs of existing datasets based on the retrieved answers. This may shed light on improved instruction-tuning pipeline for retrieval-augmented LMs. - Experimental results show the effectiveness of proposed method compared with prior method such as SAIL or Self-RAG.

Reasons to reject

- Limited novelties: as also noted by the authors, the proposed method is heavily influenced by prior work such as Self-RAG or SAIL, and the novelties newly introduced by this work may be limited. - The baseline models are somewhat weak — several recent work shows strong performance on the datasets, and also to my understanding, the ChatGPT and GPT-4 baselines do not use retrieval. Although I don’t think the proposed method should outperform proprietary models whose training details are private, claiming superiority over ChatGPT when ChatGPT baseline doesn’t use retrieval seems unfair to me. - Evaluations are mostly done in short-form QA tasks, and it is an open question how well we can apply the proposed setup for long-form generation tasks. For instance, the ensemble method shows the best performance on most of the tasks, but ensembling long-form outputs are not trivial.

Summary

The paper has good originality and high significance in the topic of retrieval-augmented generation. The paper lacks some analyses, causing a possibility that the paper's claim is not verified well, lowering the quality. The clarity is fine. See "Reasons To Accept" and "Reasons To Reject" below for details.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1) The empirical observations support the superiority of the proposed method well. They show that the proposed method outperforms state-of-the-art models in single-hop and multi-hop QA tasks (Self-RAG-7B and GPT-3.5-turbo, respectively). 2) The paper conducts extensive comparison experiments using four datasets for each of single-hop QA and multi-hop QA. 3) Although I see a few typos, the paper is well organized, making it easy to follow the content.

Reasons to reject

1) The contribution by each of learned capabilities (e.g., query decomposition) is not adequately analyzed. In particular, I suspect the contribution by query disambiguation might be minimal because ambiguous QA data (ASQA) occupies only 10% of the constructed dataset, according to Figure 5. The contribution could be more convincingly justified if it showed statistics of how many occurrences of query refinement types exist in the dataset and how many times each of those refinement actions was chosen by LLM during inference. See also the question below. 2) The clarification process by the proposed query disambiguation could be inefficient because a large number of disambiguated queries is potentially possible, especially if a disambiguated aspect is at the instance level. For example, if a question is "What is the length of a U.S. president's term?", the answer depends on a particular U.S. president. Then, the proposed method could end up generating numerous queries "What is the length of the term for the U.S. president Joe Biden?", "What is the length of the term for the U.S. president Donald Trump?", etc. If numerous disambiguated queries are generated, the proposed method will lack runtime efficiency, and LLM will be likely to generate erroneous final answers. 3) No error analysis is provided. The paper would be more insightful for future work if it showed an analysis of what errors are dominantly produced from each of the query refinement actions.

Questions to authors

- Section 2.1: I assume Y_new can significantly (semantically) differ from Y_origin, based on Section 4.4. If so, how does one know when to stop the iterative query refinement annotation process and make Y_new? This point would be better clarified if the paper gave an example and presented details of the answer template shown in Figure 2. - Figure 3: Section 4.3 says the confidence-based sampling strategy generally excels for single-hop QA tasks, but actually Figure 3 shows that the ensemble-based strategy performs best across single-hop and multi-hop tasks, except for OBQA. What is the reason why this anomaly (i.e., the confidence-based strategy outperforming the ensemble-based one) happens with OBQA? - Appendix A.1: More detailed dataset statistics should be provided. It would be interesting to see the number of occurrences of each query refinement and the average number of refined queries in each QA pair. - Section 2.3: However, using different query to -> However, using different *queries* to - Section 2.3: ... which leads to different final answer. -> ... which leads to different final *answers*. - Section 2.3: We ensemble the final results by select the final results ... -> We ensemble the final results by selecting the final results ...

Follow-Up on Rebuttal Response for Paper

Dear Reviewer, Thank you for the time and effort you have put into evaluating my work. As the review process is nearing completion, I wanted to kindly inquire if there are any unresolved issues or concerns that you still have. Should you have any further questions, I am more than happy to address them. Thank you once again for your consideration.

Follow-Up on Rebuttal Response

Dear Reviewer, Thank you for your comments on our paper. I believe that our rebuttal addresses the issues you raised, and the newly provided experimental results further substantiate the novelty of our work. If you have any additional questions or concerns, please feel free to reach out. Thank you once again for your time and consideration.

Re: Rebuttal by Authors

Thank you for the response. It's good to see the action distribution is balanced. I'll keep my original evaluation.