Summary
The paper introduces Corrective Retrieval Augmented Generation (CRAG) to reduce hallucinations in large language models (LLMs). CRAG improves generation robustness by assessing retrieved document quality with a confidence score, triggering corrective actions if needed. It also enhances retrieval by adding large-scale web searches and a decompose-then-recompose algorithm to filter key information from documents. This plug-and-play method boosts RAG-based performance, as shown by experiments across multiple datasets.
Strengths
1. The paper addresses an important research question that is critical. Recognizing the imperfections in existing Retriever technologies, this paper focuses on how to mitigate these issues within the Retrieval-Augmented Generation (RAG) paradigm, thus contributing valuable insights toward developing more robust RAG systems.
2. The paper presents compelling experimental results.
Weaknesses
1. **The experiment in this paper is not sufficient, particularly in the selection of baselines.**
The paper lacks comparison with highly relevant prior works that have proposed corrective strategies for Retrieval-Augmented Generation (RAG). Notably, it does not include baselines such as RARR [1] and DRAGIN [2], which are essential for contextualizing the contribution of this work within the existing literature. Including these baselines would strengthen the evaluation and provide a clearer understanding of the advancements offered by the proposed method.
Furthermore, the baselines used for comparison are inadequate, with a noticeable absence of state-of-the-art (SOTA) baselines from 2024. For a comprehensive and fair evaluation, it is imperative to compare the proposed method against at least two or three contemporary baselines from 2024. This would demonstrate the method’s effectiveness relative to the latest advancements in the field.
2. The ablation results indicate that removing each component of the proposed method has minimal impact on the final results. This suggests that the individual components may not significantly contribute to the overall performance. A deeper analysis is needed to understand the role and necessity of each component within the model.
3. The paper states:
>“To the best of our knowledge, this work makes the first attempt to design corrective strategies for RAG to improve its robustness.”
This claim is inaccurate. Prior works such as RARR [1] and Dragin [2] have already explored corrective strategies for RAG. It is important for the authors to accurately acknowledge existing research to position their contribution correctly within the academic discourse.
4. (Minor) The paper appears to have been prepared in a hurry, there are formatting errors such as the question mark in the citation on Line 98.
5. About methodology
The overall methodology and structure of the paper seem disorganized. The motivation presented focuses on addressing the reliance on the relevance of retrieved documents and correcting instances where the retriever returns incorrect information. However, several components of the proposed solution do not directly address this motivation. For instance, incorporating Web Search when retrieval results are poor raises questions. If a superior retriever (Web Search) is available within the pipeline, it is unclear why it is not utilized from the beginning. Additionally, the inclusion of passage splitting does not seem directly related to correcting retrieval errors, leading to confusion about how these components align with the stated goals. The training methodology and performance metrics of the retrieval evaluator are not adequately described. It is unclear how the evaluator is trained and assessed. The reliance on existing datasets like PopQA for relevance signals may introduce biases, as these datasets might not accurately reflect real-world retrieval tasks. Moreover, utilizing T5-Large, which is an older and less capable model compared to current large language models, may limit the evaluator’s effectiveness, particularly for complex queries.
6. There is a lack of transparency regarding whether the baselines use the Web Search API. If the proposed method leverages advanced commercial search engines while the baselines rely solely on a single retrieval method, the comparison is inherently unfair.
References
[1] RARR: Researching and Revising What Language Models Say, Using Language Models
[2] Dragin: Dynamic Retrieval Augmented Generation Based on the Real-time Information Needs of Large Language Models