Library

MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries

Retrieval-augmented generation (RAG) augments large language models (LLM) by retrieving relevant knowledge, showing promising potential in mitigating LLM hallucinations and enhancing response quality, thereby facilitating the great adoption of LLMs in practice. However, we find that existing RAG systems are inadequate in answering multi-hop queries, which require retrieving and reasoning over multiple pieces of supporting evidence. Furthermore, to our knowledge, no existing RAG benchmarking dataset focuses on multi-hop queries. In this paper, we develop a novel dataset, MultiHop-RAG, which consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence. We detail the procedure of building the dataset, utilizing an English news article dataset as the underlying RAG knowledge base. We demonstrate the benchmarking utility of MultiHop-RAG in two experiments. The first experiment compares different embedding models for retrieving evidence for multi-hop queries. In the second experiment, we examine the capabilities of various state-of-the-art LLMs, including GPT-4, PaLM, and Llama2-70B, in reasoning and answering multi-hop queries given the evidence. Both experiments reveal that existing RAG methods perform unsatisfactorily in retrieving and answering multi-hop queries. We hope MultiHop-RAG will be a valuable resource for the community in developing effective RAG systems, thereby facilitating greater adoption of LLMs in practice. The MultiHop-RAG and implemented RAG system is publicly available at https://github.com/yixuantt/MultiHop-RAG/.

cs.CLComputer Science
Source

Paper

References (23)

01ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation SystemsJon Saad-Falcon, O. Khattab, Christopher Potts et al.2023 · North American Chapter of the Association for Computational Linguistics · 333 citations In Library
02RECALL: A Benchmark for LLMs Robustness against External Counterfactual KnowledgeYi Liu, Lianzhe Huang, Shicheng Li et al.2023 · arXiv.org · 59 citations In Library
03RAGAs: Automated Evaluation of Retrieval Augmented GenerationES Shahul, J. James, Luis Espinosa Anke et al.2023 · Conference of the European Chapter of the Association for Computational Linguistics · 876 citations In Library
04Benchmarking Large Language Models in Retrieval-Augmented GenerationJiawei Chen, Hongyu Lin, Xianpei Han et al.2023 · AAAI Conference on Artificial Intelligence · 606 citations In Library
05Llama 2: Open Foundation and Fine-Tuned Chat ModelsHugo Touvron, Louis Martin, Kevin R. Stone et al.2023 · arXiv.org · 18k citations In Library
06Enabling Large Language Models to Generate Text with CitationsTianyu Gao, Howard Yen, Jiatong Yu et al.2023 · Conference on Empirical Methods in Natural Language Processing · 702 citations In Library
07Evaluating Embedding APIs for Information RetrievalEhsan Kamalloo, Xinyu Crystina Zhang, O. Ogundepo et al.2023 · Annual Meeting of the Association for Computational Linguistics · 28 citations In Library
08One Embedder, Any Task: Instruction-Finetuned Text EmbeddingsHongjin Su, Weijia Shi, Jungo Kasai et al.2022 · Annual Meeting of the Association for Computational Linguistics · 479 citations In Library
09Text Embeddings by Weakly-Supervised Contrastive Pre-trainingLiang Wang, Nan Yang, Xiaolong Huang et al.2022 · arXiv.org · 1.5k citations In Library
10Towards a Unified Multi-Dimensional Evaluator for Text GenerationMing Zhong, Yang Liu, Da Yin et al.2022 · Conference on Empirical Methods in Natural Language Processing · 376 citations In Library
11Improving language models by retrieving from trillions of tokensSebastian Borgeaud, Arthur Mensch, Jordan Hoffmann et al.2021 · International Conference on Machine Learning · 1.8k citations In Library
12Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning StepsXanh Ho, A. Nguyen, Saku Sugawara et al.2020 · International Conference on Computational Linguistics · 1.5k citations In Library

Scroll for more · 11 remaining

Similar papers

Multi-Meta-RAG: Improving RAG for Multi-Hop Queries using Database Filtering with LLM-Extracted MetadataMykhailo Poliakov, Nadiya Shvai2024 · International Conference on Information and Communication Technologies in Education, Research, and Industrial Applications · 32 citations In Library
ConRAG: Consensus-Driven Multi-View Retrieval for Multi-Hop Question AnsweringYikai Zhu, Kun Chen, Qihuang Zhong et al.2026 · arXiv.org · 0 citations In Library
EfficientRAG: Efficient Retriever for Multi-Hop Question AnsweringZiyuan Zhuang, Zhiyang Zhang, Sitao Cheng et al.2024 · Conference on Empirical Methods in Natural Language Processing · 26 citations In Library
Hierarchical Lexical Graph for Enhanced Multi-Hop RetrievalAbdellah Ghassel, Ian Robinson, Gabriel Tanase et al.2025 In Library
Hierarchical Lexical Graph for Enhanced Multi-Hop RetrievalAbdellah Ghassel, Ian Robinson, Gabriel Tanase et al.2025 · Knowledge Discovery and Data Mining · 4 citations In Library
Optimizing Multi-Hop Document Retrieval Through Intermediate RepresentationsJiaen Lin, Jingyu Liu2025 · Annual Meeting of the Association for Computational Linguistics · 5 citations In Library
Evaluating Multi-Hop Reasoning in RAG Systems: A Comparison of LLM-Based Retriever Evaluation StrategiesLorenz Brehme, Thomas Strohle, Ruth Breu2026 · arXiv.org · 0 citations In Library
Dynamic Multi-Hop Retrieval-Augmented Generation Framework for Professional Domain Question AnsweringJingxuan Zhu, Yunxi Tong2026 In Library
Can Large Language Models Perform Retrieval-Augmented Generation as Multi-Hop Reasoning Over Knowledge Graphs?Mohamed Abdel Hamid, Salsabil Amin El-Regaily, Mostafa Aref2025 In Library
FAIR-RAG: Faithful Adaptive Iterative Refinement for Retrieval-Augmented GenerationMohammad Aghajani Asl, Majid Asgari-Bidhendi, Behrooz Minaei-Bidgoli2025 · arXiv.org · 7 citations In Library

Peer review

Acceptcolmweb.org 2024 · via OpenReview
Reviewer rLY33/10 · confidence 5/52024-05-05

Summary

The paper introduces "MultiHop-RAG," a dataset designed to evaluate retrieval-augmented generation (RAG) systems by focusing on multi-hop queries. The author conducts experiments on the various embedding models, testing their efficacy in evidence retrieval for multi-hop queries and, furthermore, evaluates the ability of off-the-shelf LLMs such as GPT-4 and Llama2-70B, indicating that direct RAG methods struggle with the multi-hop query demands.

Rating

3

Confidence

5

Ethics flag

1

Reasons to accept

The paper identifies a gap in current RAG systems, specifically their inefficiency in handling multi-hop queries. By focusing on this underexplored area, the paper employs GPT-4 to develop a benchmark and conducts experiments to evaluate both the retrieval and generation components of the RAG systems. However, there remain key issues that prompt reasons for rejection, see next part.

Reasons to reject

The paper claims to present the first multi-hop RAG (Retrieval-Augmented Generation) dataset. However, this statement overlooks existing datasets such as HotpotQA and 2WikiMultiHop, which also focus on multi-hop reasoning and are cited in the related works section. However, the paper does not adequately differentiate its dataset from these predecessors in the main body of the text. A clarification or rephrasing to accurately position this work within the existing landscape of multi-hop datasets is necessary. The evaluation methodology appears to inadequately address the core concept of multi-hop reasoning. The described approach primarily involves direct prompting to retrieve relevant documents and generate answers, which seems to neglect advanced multi-hop retrieval and reasoning techniques referenced in prior studies [1,2,3]. More rigorous testing focused on multi-hop reasoning capabilities is essential to substantiate the dataset's utility and effectiveness. The construction of the dataset raises several concerns. The method involves using GPT-4 to generate answers, which is followed by a quality assurance process. However, the details of this process are vaguely described, such as the proportion of data points filtered out during quality checks and the accuracy of GPT-4 in generating correct answers. For a project aiming to establish a benchmark, the reliance on LLMs for generating dataset content without thorough validation may undermine the credibility and utility of the dataset. Typically, LLMs might be more suitably used for augmenting training sets rather than creating benchmarks intended for evaluating model capabilities. [1] Yu, W., Zhang, H., Pan, X., Ma, K., Wang, H., & Yu, D. (2023). Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210. [2] Zhang, J., Zhang, H., Zhang, D., Liu, Y., & Huang, S. (2023). Beam retrieval: General end-to-end retrieval for multi-hop question answering. arXiv preprint arXiv:2308.08973. [3] Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022). Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509.

Reviewer zFdY7/10 · confidence 3/52024-05-09

Summary

This paper presents a new dataset to assess multihop RAG, with preliminary performance results by a number of available embeddings and models, with items to test inference, comparison, temporal relations, and a null option to assess hallucination. The dataset is recent (Sept-Dec 2023) so includes data that the tested models were not trained on. The abstract indicates that existing systems perform "unsatisfactorily", and the paper clarifies that conclusion by showing that best performance is 56% or less (table 6). The paper is well-situated and fairly clear. I think it will be useful. One shortcoming is in the quality control (step 5). The paper says "we manually review a subset sample", but we are not told how large that sample is, how it was selected, or whether reviewers agreed with each other. A related problem is the complete lack of individual item analysis in the paper. Accuracy per query type is reported, but across those types, were the same items difficult for all systems? And *those* difficult items are the ones that should be manually reviewed, and discussed as examples. A perhaps easier alternative sanity check could be provided with a quick check of human performance on the task. Another shortcoming is the lack of analysis of which features of existing systems best explain the range of performance results. None of these things are listed among the limitations that could be improved with future research.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

The dataset and presentation address an important aspect of language models that needs improvement.

Reasons to reject

I think some additional attention should be given to quality control and item analysis. If the dataset is widely used, these will be of interest.

Questions to authors

Some informal item analysis must have been done, and could be reported in the quality section (step 5).

Reviewer w5Gn7/10 · confidence 5/52024-05-11

Summary

In this paper, the authors propose a multi-hop queries datasets named MultiHop-RAG for evaluating the LLMs' retrieving and reasoning capabilities. The dataset contains various well-designed types including inference, comparison, temporal and null queries. The detailed data statistics also present the integrity and adequacy of the dataset. From the Table 5 and Table 6, we can observe that the various LLMs can not perform the retrieve and reason well in this difficulty scenario, indicating that the dataset is valuable for the future research.

Rating

7

Confidence

5

Ethics flag

1

Reasons to accept

1. The author propose a novelty and effective problem for RAG scenario with LLMs to further investigate. 2. The dataset MultiHop-RAG is constructed comprehensive and rationality, including various query types and categories. 3. The experiments show the dataset is very challenge for various typical LLMs indicating the potential value of the dataset for future research work.

Reasons to reject

1. The number of query seems a little bit small, the author can collect more in the future. 2. For the Table 6, we can observe that the LLMs can not perform well in this dataset. However, the detailed reasons are not analyzed in the experiment results description.

Reviewer AfhS8/10 · confidence 5/52024-05-13

Summary

The paper introduces a novel dataset named MultiHop-RAG, specifically designed to address the inadequacies of existing Retrieval-Augmented Generation (RAG) systems in managing multi-hop queries. The authors establish that current RAG setups fall short in accurately handling these complex queries and highlight the absence of a dedicated benchmarking framework to assess RAG performance on multi-hop inquiries. The developed dataset, MultiHop-RAG, comprises a knowledge base built from English news articles, numerous multi-hop queries, their correct answers, and identifiable supporting evidence.

Rating

8

Confidence

5

Ethics flag

1

Reasons to accept

The development of MultiHop-RAG addresses a glaring gap in the research on RAG systems which is their inadequacy in handling multi-hop queries. This dataset provides a foundational step towards enhancing RAG performance and extending LLM applicability. By making the dataset and benchmarking code publicly available, the paper offers valuable resources for the wider research community. This openness facilitates replication and further research, potentially accelerating progress in the area.

Reasons to reject

While the paper introduces a new dataset and conducts experiments, it may lack a clear comparison to existing datasets. A comparative analysis demonstrating the unique challenges and benefits of MultiHop-RAG over previous datasets could strengthen the case for its necessity and utility. In addition, the use of an English news article dataset as the sole source for the knowledge base might introduce bias or limit the scope of queries the dataset can effectively support. The paper could benefit from a discussion on these limitations and potential mitigation strategies.

Reviewer rLY32024-06-03

I remain unconvinced by the authors' rebuttal. The novelty of this work is still unclear. The response does not adequately address the fundamental difference between this work and existing multi-hop reasoning benchmarks, aside from the retrieval data source. My initial question was not intended to suggest that Wikipedia should be the data source. Rather, I am concerned with the conceptual differences, particularly given that previous literature has also evaluated retrieval performance and generation accuracy of large language models. If the primary contribution of this work is the introduction of a different data source, then the claim of presenting "the first multi-hop RAG (Retrieval-Augmented Generation) dataset" appears tenuous. Regarding my second concern, the authors did not directly address why they did not evaluate using existing multi-hop reasoning baselines. The baselines I mentioned, along with several other widely used multi-hop reasoning methods(such as chain-of-note[1], react[2]), generate follow-up questions and use these questions to retrieve relevant documents before generating a final answer. This approach would seem to be more appropriate as a baseline for evaluating a multi-hop RAG dataset. The choice of data source should not preclude testing against these established methods. In light of these points, I plan to maintain my score. [1] Yu, W. et.al. (2023). Chain-of-note: Enhancing robustness in retrieval-augmented language models. [2] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.

Authorsrebuttal2024-06-03

Thank you very much for your reply! We appreciate your academic professionalism and willingness to engage in this discussion. Below, we provide our response to each of your comments in this round of discussion. **Q1. The novelty of this work compared to existing multi-hop reasoning benchmarks** In addition to the retrieval data source, we have designed certain questions to assess the reliability of the target RAG systems, particularly in scenarios involving null queries. This type of question lacks a relevant context in the corpus, necessitating the LLM to identify and reject responses rather than make up a wrong answer. This is another important difference. Since LLM struggles to say "I don’t know" [1] and hallucinations exist in the RAG framework [2], it is important to measure the robustness of RAGs. Thus, our dataset proposes a comprehensive testbed for the RAG framework.  **Q2. No evaluation of the advanced framework** We appreciate your insight into the RAG framework and its role in our research. While focusing on dataset construction, we utilized a basic RAG framework to demonstrate the dataset’s potential. We are excited about the more complex frameworks, which could provide further insights, and have acknowledged this as a promising area for future research in the limitation section. And we're still exploring new methods. For your reference, we have tested the performance of advanced RAGs based on generating follow-up questions (query decompose) and using these questions to retrieve relevant documents. We also tested the Hybird search [3]. The baseline in the following table is the basic RAG framework, and the retriever is the best model tested in our work (bge-large-en-v1.5). Here are our results: | mode | chunk size | MRR@10 | MAP@10 | Hits@4 | |:---------:|------------|:------:|:------:|:------:| | baseline | 512 | 0.4975 | 0.2447 | 0.6718 | | hybrid | 512 | 0.7017 | 0.3682 | 0.8421 | | decompose | 512 | 0.508 | 0.2509 | 0.6794 | | baseline | 256 | 0.4322 | 0.2017 | 0.6049 | | hybrid | 256 | 0.6779 | 0.3469 | 0.8018 | | decompose | 256 | 0.4576 | 0.2218 | 0.6222 | | baseline | 128 | 0.3679 | 0.1728 | 0.4958 | | hybrid | 128 | 0.5988 | 0.3016 | 0.7135 | | decompose | 128 | 0.3884 | 0.1923 | 0.5188 | It is evident that the advanced framework's performance on our datasets is suboptimal. As a result, we believe Multihop-RAG can be a useful resource for benchmarking advanced RAG technology, and we are available for further discussion if you have any additional questions or need more clarification. [1] Yin, Zhangyue, et al. Do Large Language Models Know What They Don't Know? [2] Niu, Cheng, et al. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. [3] Gordon V. Cormack, et Reciprocal rank fusion outperforms condorcet and individual rank learning methods.

Authorsrebuttal2024-06-03

Further clarification on the differences between MultiHop-RAG and existing multi-hop datasets

Please allow us to further clarify a key difference between MultiHop-RAG and existing datasets such as HotpotQA and 2WikiMultiHop. We acknowledge the existence of multihop QA datasets such as HotpotQA and 2WikiMultiHop, among others. However, these existing datasets are not designed for the RAG task and do not include a retrieval step. The input used in these datasets is already provided with multiple pieces of evidence, and they primarily examine QA models' performance in multi-hop reasoning. Even if one extends HotpotQA to evaluate retrieval performance by combining the HotpotQA input data (Wikipedia text) as a corpus, this extension does not align with real-world RAG tasks. In real-world RAG scenarios, the dataset is usually internal or proprietary corpus, not a corpus used in LLM training such as Wikipedia. This is why we mentioned the Wikipedia data source issue in our previous response. Our developed dataset, on the other hand, is specifically designed for the RAG task. It can be used to benchmark multi-evidence retrieval capability (given a question that requires multiple pieces of evidence, how to retrieve relevant pieces of evidence from a database) as well as multi-hop evidence generation capability. Our experiments find that an off-the-shelf RAG solution cannot effectively handle multi-evidence retrieval, which subsequently affects downstream generation performance. We believe the focus on the retrieval aspect sets us apart from existing datasets.

Authorsrebuttal2024-06-07

Any further questions?

Dear Reviewer rLY3, We appreciate your constructive feedback. As noted in our engagement, we believe there is a misunderstanding regarding the contribution of our work. We want to reiterate that our developed dataset is specifically for RAG tasks, which differs from previous multi-hop QA datasets that primarily focus on the multi-hop reasoning aspect while overlooking the multi-hop retrieval aspects. In addition to this major difference, there are also other outstanding differences, which we do not repeat here in the interest of your time. If we clear up this misunderstanding, we believe this paper constitutes a valuable resource for benchmarking real-world RAG systems, as most of the human queries in RAG systems involve multi-hop, multi-evidence retrieval, and reasoning, as rightly recognized by other reviewers. We hope we have clarified this point and would appreciate it if you could reconsider your rating. Should you have any further questions, we are very happy to address them. Best, Authors

Reviewer zFdY2024-06-04

Thanks, these statements clarify the quality control tests you did, though the statement of results here is still too brief for me to understand fully. Were per-item correlations with GPT-4 or any other calculated?

Authorsrebuttal2024-06-05

Thanks for your insights! Please allow us to further clarify about the quality control tests. First, the table in Q1 shows the difficulty distribution for each question type based on majority voting from crowdsourced annotations. The difficulty ranges from 1 (easiest) to 5 (most difficult). Scores below 3 are considered easy, while 3 and above are considered difficult. Temporal queries, which involve reasoning about time, are the most challenging for humans, followed by comparison queries that require making comparisons. Null queries, which have no answerable questions, are not evaluated and therefore excluded from this data. The results align with the model answer accuracy shown in Figure 3. As expected, the model exhibits the lowest accuracy on comparison and temporal queries, which are identified as the most difficult for humans. In contrast, the model achieves the highest accuracy on inference questions, which appear to be relatively easier based on the difficulty distribution. Second, motivated by your comments, we reviewed the per-item correlations between human evaluators and model-generated responses. In our understanding, the per-item correlation you referred to may be the model's accuracy in answering each question compared to the human-annotated difficulty levels. If we have misunderstood, please correct us. Table 1 shows the accuracy of different models in answering the questions we used in human annotations. GPT-4 remains the best, followed by Google-PaLM. The results in Table 2 specifically analyze the questions the models answered incorrectly. For each model, the table shows the average difficulty scores of the questions it struggled with, as determined by human annotators. The 'nan' values indicate no incorrect questions. Thus, we can find that GPT-4 and PaLM can handle more challenging questions (failed on a high difficulty score), while open-source models struggle with simpler questions, as indicated by the average difficulty scores of their incorrectly answered questions in Table 2. This observation is consistent with the findings in Table 6 of the paper. Combining all the tables, we can clearly see how the difficulty of different types of questions varies for both humans and models. The stronger the model, the higher its accuracy in answering questions, and it can handle more challenging problems. **Table 1: Generation accuracy of LLMs for the sampled questions.** | Model | Comparison Query | Inference Query | Temporal Query | Avg. | | ------------ | ----------------- | --------------- | -------------- |-------------- | | GPT-4 | 40.00% | 100% | 50.00% | 63.00% | | Google-PaLM | 30.00% | 100% | 50.00% | 60.00% | | ChatGPT | 0.00% | 100% | 40.00% | 46.67% | | Claude-2.1 | 10.00% | 80.00% | 10.00% | 33.34% | | Llama-2-70b-chat-hf | 0.00% | 90.00% | 0.00% | 30.00%| | Mixtral-8x7B-Instruct | 0.00% | 80.00% |0.00% | 26.67%| **Table 2: Average human annotated difficult score of the inaccurate questions from different models.** | Model | Comparison Query | Inference Query | Temporal Query | | ------------ | ----------------- | --------------- | -------------- | | GPT-4 | 2.0 | nan | 4.6 | | Google-PaLM | 2.43 | nan | 4.6 | | ChatGPT | 2.4 | nan | 4.16 | | Claude-2.1 | 2.55 | 2.0 | 4.55 | | Llama-2-70b-chat-hf | 2.4 | 2.0 | 4.4 | | Mixtral-8x7B-Instruct | 2.4 | 2.0 | 4.4 | The findings from our analysis clearly demonstrate that our dataset poses a substantial challenge due to its high difficulty and quality, making it an ideal testbed for RAG systems. We are available for further discussion if you have any more questions or need additional clarification.

Program Chairsdecision2024-07-10

Decision

Accept

© 2026 NYSGPT2525 LLC