ChatQA: Surpassing GPT-4 on Conversational QA and RAG

Summary

This paper proposes a family of fine-tuned models (ChatQA) that surpass GPT-4 on conversational QA and RAG. It introduces a two-stage instruction fine-tuning method to enhance the model’s capability of using the retrieved context for generation. In addition, it shows that fine-tuning a single-turn retriever (Dragon) on human-annotated data can closely match LLM-based query rewrites, thereby eliminating computational and API costs. Lastly, this paper collects a ChatRAG benchmark for evaluating RAG, table-related QA and arithmetic calculations. Experiments on the ChatRAG Bench show that the proposed ChatQA models outperform or closely match the strong generalist model GPT-4.

Strengths

1. Overall, the writing of the paper is good. The experiments and ablation studies are sound and comprehensive, demonstrating the effectiveness of the two-stage instruction fine-tuning method and the data curation recipe. 2. The open-sourced training, the data curation recipes, and the use of the public foundation models (Llama2 and Llama3) could be valuable and beneficial for the community in chasing proprietary models.

Weaknesses

1. Training an open foundation model with curated instruction data is not new. The paper could be improved if it demonstrates why the selected mixture of training data is effective, and training on them could match GPT-4. 2. While the ablation studies show the effectiveness of the collected data in training ChatQA models, I believe more fine-grained data selection and analysis could potentially further improve the performance [1][2]. [1] How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources, 2023\ [2] A Survey on Data Selection for Language Models, 2024

Questions

1. L163, in the stage-2 context-enhanced instruction tuning, it seems that you again use all of the SFT datasets from stage-1. Will it lead to overfitting? If you want to maintain the instruction-following ability, why not merge all data from stage-1 and stage-2 and then do multi-task tuning?

Rating

7

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper explore RAG in conversational QA senarios. It proposes a two-stage instruction tuning method for conversational QA in a RAG manner, accompanied by a comprehensive benchmark. The training of LM involves two stages: (i) SFT on dialogue and QA datasets and (ii) context-enhanced instruction tuning which includes SFT on both synthetic and existing datasets. Similar strategies are applied to tune the retriever for multi-turn QA scenarios. The experiments demonstrate that the proposed ChatQA model surpasses various existing large language models in performance.

Strengths

This paper effectively underscores the significance of RAG in chat-based QA scenarios, a critical area that has been overlooked in previous research. It offers a detailed and concise overview of the current research landscape, scenario positioning, and existing challenges, providing valuable insights into these areas. Additionally, the proposed benchmark presented in the paper could be valuable for advancing research in this domain.

Weaknesses

- There is a lack of technical novelty. The paper focuses on fine-tuning a language model using an existing dialogue dataset and self-created data for this scenario. While the approach is clearly outlined, it would be helpful if the authors could further elaborate on the innovative aspects of their methodology to highlight its novelty. - The paper lacks clear definitions and formalizations of the tasks that need to be addressed, which obscures the distinctions between its concepts and existing works. It took me some time to realize that the "instruction tuning method" pertains solely to the construction of training data and is conducted without retrieval. This leads to questions about the method's relevance to the subsequent RAG framework. The connection between the proposed method, which is tailored for LM in chat-based QA, and the RAG remains unclear and needs better delineation. - Lack of essential ablation studies. Numerous choices regarding training data selection and the construction of synthetic data are only briefly explained. These choices, while seemingly reasonable, necessitate thorough ablation and comparison with standard processes to validate their effectiveness. Additionally, although several strategies for tuning the retriever for this scenario are proposed, there is a lack of analysis and ablation to justify their usefulness.

Questions

1. Is there any difference between the concepts of "multi-turn QA" and "conversational QA" in this work? These terms appear to be used interchangeably. 2. Could you provide justification for the proposed retriever tuning strategies? 3. Does the conversational QA scenario differ from traditional QA primarily because it includes additional context? If so, it would be intriguing to assess the impact on QA accuracy when this context is removed. Furthermore, comparing the importance of this context against the content retrieved in the RAG may provide insightful observations.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

NA

Summary

This paper introduces ChatQA, a suite of models that outperform GPT-4 on retrieval-augmented generation (RAG) and conversational question-answering (QA) tasks. The key contributions of the paper include: - A two-stage instruction tuning method that improves RAG performance. - A dense retriever optimized for conversational QA that performs comparably to state-of-the-art query rewriting models while reducing deployment costs. - CHATRAG BENCH, a comprehensive benchmark comprising 10 conversational QA datasets for evaluating RAG, table-related QA, arithmetic calculations, and unanswerable questions. - ChatQA-1.0-70B, built on Llama2, which slightly outperforms GPT-4-0613 and GPT-4-Turbo on CHATRAG BENCH without relying on synthetic data from OpenAI GPT models. - Llama3-ChatQA-1.5-70B, which surpasses GPT-4-Turbo-2024-04-09 by a good margin. - Good performance on single-turn QA and RAG benchmarks, with Llama3-ChatQA-1.5-70B outperforming existing frontier RAG models like Command R+.

Strengths

The paper demonstrates several notable strengths across several dimensions: - The two-stage instruction tuning method is a novel approach to enhancing RAG performance, combining supervised fine-tuning (SFT) with context-enhanced instruction tuning. - The development of a dense retriever optimized for conversational QA without relying on query rewriting is an innovative solution to a common challenge in multi-turn QA systems. - The creation of CHATRAG BENCH as a comprehensive evaluation suite for conversational QA and RAG tasks is a good contribution to the field. - The empirical results are robust, with the ChatQA models outperforming strong baselines, including GPT-4, across multiple datasets and tasks. - The ablation studies provide valuable insights into the contributions of different components of the proposed method. - The study on handling "unanswerable" scenarios addresses an important challenge in QA systems, contributing to the development of more robust models.

Weaknesses

- The study primarily focuses on Llama2 and Llama3 of sizes 7B/8B, 70B as base models. Including a wider range of base models could provide insights into the generalizability of the proposed methods. Also, a detailed analysis of how performance scales with model size could offer insights into the trade-offs between model size and performance. - The paper doesn't address potential ethical implications or biases that might be present in the developed models or the CHATRAG BENCH. A brief discussion on this would enhance the paper's comprehensiveness. - While the paper provides extensive quantitative results, it doesn't include a detailed qualitative analysis of the types of errors made by the models. This could offer insights into areas for future improvement.

Questions

1. How does the performance of ChatQA models scale with model size? Is there a point of diminishing returns? This analysis could offer insights into the trade-offs between model size and performance. 2. Are there specific limitations of the current approach that you think are most important to address apart from expanding to code-related tasks or math reasoning tasks? 3. What do you see as the most promising avenues for future work based on your findings? 4. Do you have any explanation or intuition as to why incorporating more unanswerable samples beyond 1.5K leads to lower accuracy scores in most of the tasks in Table 11? 5. You mention that fine-tuning a single-turn query retriever performs comparably to state-of-the-art query rewriting. Can you elaborate on the trade-offs between these two approaches beyond computational cost?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

They discuss the limitations at the beginning of the Appendix (Line#908-914).

Summary

The paper introduces ChatQA, a suite of models designed to excel in retrieval-augmented generation (RAG) and conversational question answering (QA). The authors propose a two-stage instruction tuning methodology to bolster generative capabilities and a dense retriever optimized for conversational QA to improve retrieval effectiveness. Notably, the ChatQA models, even when built on foundation models perceived as weaker than GPT-4, demonstrate superior performance on RAG and conversational QA tasks, surpassing GPT-4 in certain benchmarks. The paper further contributes the ChatRAG Bench, a comprehensive benchmark for evaluating RAG and conversational QA models, and releases model weights, training data, the ChatRAG Bench itself, and the retriever to the community. Soundness: The technical claims, experimental methodology, and research design in this paper are well-supported and sound. The central claims are convincingly backed by extensive experimental results and comparisons to established baselines, including GPT-4. The ablation studies further validate the efficacy of the proposed two-stage fine-tuning approach and the significance of the curated datasets. The paper demonstrates a meticulous and rigorous approach, ensuring the reliability and reproducibility of its findings. Presentation: The paper exhibits a clear and well-organized presentation style. The writing is lucid, and the technical concepts are effectively conveyed. The authors adequately contextualize their work within the landscape of prior research, highlighting the novel contributions of their approach. Overall, the paper is well-written and easy to follow, making it accessible to a broad audience. Contribution: The paper makes a significant contribution to the field of conversational QA and RAG. The proposed ChatQA models push the boundaries of performance, even outperforming GPT-4 in certain benchmarks. The open-sourcing of model weights, data, and the ChatRAG Bench fosters transparency and collaboration, promoting further advancements in the research community. The paper's findings challenge prevailing assumptions about the necessity of relying on synthetic data from OpenAI GPT models, opening avenues for innovative training strategies.

Strengths

1. The paper introduces a novel two-stage instruction tuning methodology that significantly enhances the context-aware and RAG-based QA capabilities of LLMs. It also proposes an effective dense retriever optimized for conversational QA. 2. The research is meticulously conducted, with rigorous experiments and comprehensive evaluations. The ablation studies provide valuable insights into the contributions of various components of the proposed approach. 3. The paper is well-written and clearly structured, making it easy to follow the authors' line of reasoning. The technical concepts are presented in an accessible manner. 4. The paper's findings are impactful, demonstrating that state-of-the-art performance in conversational QA and RAG can be achieved without reliance on synthetic data from OpenAI models. The open-sourcing of resources further amplifies the significance of this work.

Weaknesses

The paper primarily focuses on evaluating the "unanswerable" scenario using a small set of samples. A more extensive evaluation involving diverse "unanswerable" scenarios would enhance the robustness of the findings.

Questions

How does the proposed two-stage instruction tuning method compare to other state-of-the-art instruction tuning approaches in terms of efficiency?

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

The paper does not identify any potential negative societal impact, which may be worth exploring in future work, particularly given the potential for misuse of powerful conversational QA models.