Rebuttal by Authors (Continued)
- **W3: The comparison of baselines may not be fair since the parameter sizes of Hippo-RAG are significantly larger than other methods. Contriever and ColBERTv2 achieve comparable results on MuSiQue and HotpotQA, but when combined with HippoRAG, the improvement is insignificant. ColBERTv2 is the second-best model, and the improvement from Hippo-RAG seems incremental on the two datasets.**
In the response below, we present evidence and arguments that we hope will convince the reviewer that our performance improvements are substantial and our experimental setting is sound and fair.
#### **Improvements on MuSiQue**
We first highlight that our method “achieves significant performance improvement from various challenging multi-hop QA benchmarks”, as pointed out by the reviewer when listing our strengths.
HippoRAG improves R@5 on MuSiQue by 5.5% and 2.7% over Contriever and ColBERTv2 respectively, as well as 3.4% F1 score on QA compared to ColBERTv2. Many well-received and highly cited works such as IRCoT [2], Self-Ask [3], ITER-RETGEN [4] and MCR [5] highlight improvements similar to ours.
#### **HotpotQA’s Weaknesses as a Knowledge-Integration Benchmark**
As discussed in subsection “Single-Step Retrieval Results” (Section 4) of our paper, our lower performance on HotpotQA is mainly due to its lower need for knowledge integration due to existing shortcut signals. This HotpotQA limitation is referenced when constructing MuSiQue [6], but also explored more deeply in Appendix B of our paper. In bridge multi-hop questions, the query and second supporting passage should be linked only through a bridge entity. However, as shown in Figure 6, HotpotQA queries are on average as similar to the second supporting passage as to their distractors, instead of less similar like in the other datasets, making that second supporting document easier to detect without knowledge integration.
#### **Measuring the Impact of Improved Knowledge-Integration Directly**
Additionally, when we measure the impact of improved knowledge integration in MuSiQue and 2WikiMultiHopQA, our improvements are even larger than our overall results. In Table 8, we show that HippoRAG’s ability to find all supporting documents is key to its strong performance, obtaining a 6.3% and 38.6% improvement over ColBERTv2 in All-Recall@5, compared to a 2.7% and 21.3% improvement in standard Recall@5.
#### **Our Experimental Setting is Sound and Fair**
Finally, we refer to the reviewer’s concern that our comparison with Contriever and ColBERTv2 might not be fair given their small size compared to LLM. We understand the reviewer’s apprehension, however, we argue that improving the performance of a system using an LLM and comparing it to the original system is a well-accepted paradigm in AI research. As a specific example, many well-received QA works such as IRCoT [2], Self-Ask [3] and RAPTOR [7] leverage a large LLM in different ways and compare with the original pipeline; which in these papers contain various retrieval methods like BM25, DPR [8] and ColBERTv2. We note that HippoRAG follows this same research paradigm, which is related to the large and growing body of recent work that leverages LLMs to challenge prior state of the art based on smaller models.
As a final remark, we also note that our method outperforms RAPTOR and is comparable with IRCoT (and considerably more efficient). Since both of these methods augment the original retrieval process with outputs from an LLM, they can be seen as HippoRAG’s most direct baselines in that sense.
- **Q2: Why does Hippo-RAG perform significantly better on 2Wiki?**
As discussed in subsection “Single-Step Retrieval Results” (Section 4) of our paper, 2WikiMultiHopQA’s construction is more entity-centric than the other two datasets, making it particularly well-suited for HippoRAG’s design.
To provide some extra context, 2WikiMultiHopQA was created by leveraging the Wikidata KG to determine which entities and relations could be found in Wikipedia passages to create compositional, comparison, inference or bridge-comparison multi-hop questions. Due to this construction process, queries in this dataset include at least one named entity, a characteristic that our methodology can leverage given its previously mentioned reliance on NER to link queries to the KG.