Beyond Relevance: Evaluate and Improve Retrievers on Perspective Awareness

Summary

This work introduces an additional intrinsic component—perspective—to the information retrieval process and presents a new benchmark, PIR. It also introduces a zero-shot, projection-based method called PAP. Experimental results show that current retrievers have limited awareness of subtly different perspectives in queries and can also be biased toward certain perspectives. Meanwhile, the method PAP outperforms other baseline methods in multiple settings and can enhance the perspective sensitivity of the retrieval process in a zero-shot manner without additional fine-tuning. Furthermore, perspective-aware retrieval can significantly improve the performance of downstream tasks.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

1.The underlying motivation for this research is compelling and well-founded. 2.The methodology introduced is both straightforward and efficacious. 3.This study establishes a new benchmark that is thoroughly examined and analyzed in detail.

Reasons to reject

1.Certain sections of the text lack clarity, and the logical progression appears flawed. We will address specific questions regarding these issues in the Questions section. 2.While Section 3.4 employs GPT-3.5 to generate perspectives for QA downstream tasks, experiments do not extend to retrieval tasks. It would be advantageous to include retrieval experiments using perspectives generated by GPT-3.5 in Table 4 as well. 3.The manuscript lacks a comparative analysis with existing research on cosine similarity. It would be beneficial for the authors to conduct comparative experiments with the methodologies outlined in https://arxiv.org/abs/2403.05440.

Questions to authors

1. Regarding Weakness 1: -It is unclear whether the terms 'root queries' and 'neutral queries' used in the abstract refer to the same concept. Could you please clarify? -In the concluding sentence of Section 2.1, is the 'q' in "For each q" intended to represent the tuple (p, r)? Does this notation imply both an implicit 'p' within 'q' and an extracted 'p'? -Are the formulations 'q = (p, r)' and 'q = p + r' mentioned in Section 2.1 and Equation 1 consistent with their respective descriptions in Table 4? -What is denoted by 'S' in the final sentence of Section 2.3? -Could you define what is meant by 'knowledge part' and 'gold knowledge' as referenced in the first sentence of the third paragraph of Section 3.3? 2. The study primarily investigates zero-shot information retrieval (IR). Could you speculate on the potential outcomes or adjustments necessary for few-shot IR scenarios?

Summary

The three main contributions of this paper are (1) the introduction of a perspective-aware information retrieval (PIR) benchmark by repurposing existing retrieval benchmarks, (2) a method for improving perspective-aware retrieval by projecting queries (and potential the datastore) into perspective space (PAP), and (3) an empirical analysis of off-the-shelf retrievers on PIR and how to build in PAP.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

* The paper is well-written and the experiments are thorough. * The proposed problem and benchmark are timely, as many works are concurrently identifying shortcomings in retrieval methods. * The perspective projection method is also simple and intuitive.

Reasons to reject

The empirical improvement from incorporating the PAP project method is modest, both for recall and downstream tasks. My stance is that the authors have not found the best downstream task to really highlight their method and that this paper is one or two experiments away from being a strong paper. Perspectrum seems to be on the right track -- I suspect the downstream tasks that will benefit the most from perspective-aware retrieval will be reasoning-intensive (rather than knowledge-intensive) and require comparing multiple perspectives. One could imagine trying to prove a theorem by retrieving lemmas with different perspectives, e.g. geometric or algebraic. I am open to being convinced that the improvement in downstream tasks is significant.

Questions to authors

1. Bottom of page 6: "PAP variances" => variants 2. The intent perspective extraction prompt in appendix A.8 is simple, and I would expect "intent perspective" to be a rare bigram. I am therefore surprised the performance is high for generating perspectives from queries. Do you have any intuition for why this works? 3. The improvement over the no-PAP retrieval baseline in Perspectrum is large, but the improvement over the no retrieval baseline is small. Can you explain this?

Summary

This paper explores a new retrieval setting in which different perspectives of the queries are to be recognized and attended to by the retrieval system. The key requirement is that the retrievers should attempt to address the subtle different perspectives expressed in the queries. A new benchmark for the task (Sec 2) is created, including a new variant of the recall metric (Eq 1) that takes perspectives into account. A new approach is proposed to improve the perspective awareness of a SimCSE baseline, called perspective-aware projection (Sec 3), by performing a perspective/corpus projection over the input embeddings. The main results in Table 5 show that the proposed methods can improve the accuracy and correlation of SimCSE on AmbigQA and Perspectrum datasets respectively by a wide margin. This paper is written up carefully, has a clear storyline regarding a key limitation in mainstream retrieval paradigm about perspective fairness. It shows that while SimCSE as a retriever doesn't always do a good job in a perspective-awareness scenraio (Table 5), this side of effectiveness can be further enhanced by employing a geometric projection approach PAP and PAP+. The overall narratives and the experimental effort are interesting and insightful, the methodology is simple and easy to reproduce. I'm not generally across the fairness line of work for retrieval, but the main point raised in the paper ("improving perspective awareness") seems crucial and timely. Technical novelty aside, all three main claims are reasonably demonstrated and fulfilled in separate sections. I think the paper can meet the bar for acceptance and be included in the proceedings.

Rating

6

Confidence

3

Ethics flag

2

Reasons to accept

- This paper presents a new formulation about perspective awareness and some practical dataset and methodology for evaluation. Given that LLMs are now used widely in all sorts of applications, this type of evaluation effort that strengthens model safety and reliability is particularly timely. - The paper shows that this new perspective-awareness evaluation criterion is not unreasonably difficult. A geometric projection method can be applied to SimCSE and improve the effectiveness.

Reasons to reject

- On the methodology side, Sec 3 only reviews a simplistic geometric approach. I do not believe technical novelty would be a limiting factor for this type of work, but given that ranking/prediction fairness has been studied for many years in NLP & IR there should be some relevant prior art that we can refer to and compare the proposed method with.

Questions to authors

- In Sec 2.2, you write "we use heuristics to ensure that all the queries..." Can you make unpack this mention of heuristics and call out explicitly what you do in the main text? - Eq 1 seems to use a cyclic dependency: $q = p + r$ but then $p$ is to be sampled from a set conditioned on $q$.

Summary

This paper formally introduces the task of perspective-aware retrieval, that conditions retrieval upon a specific perspective (e.g. opposing or supporting a claim). This has important influence over downstream tasks, such as RAG. The paper introduced a benchmark for this new task, and designed a simple method that outperforms various baselines.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

The paper is very clearly written and easy to understand, and has important impact over downstream tasks such as RAG.

Reasons to reject

The mathematics of the projection method seems dubious, see questions below.

Questions to authors

- Section 3.1 the PAP method: the equation provided moves the query vector away from the perspective vector by some amount, it does not project it onto the p space. Additionally, in the term prefixed with the minus sign, there could be an coefficient parameter to tune. Please explain. - Table 4: $\cos(q, c)$ is not a correct notation since $\cos$ takes 1 parameter. Please write $\cos\angle(q, c)$. The `dual-sum` and `tri-sum` lacks intuition to me. - Section 2.5: you argue that SimCSE, not specifically trained with any specific task outperforms. However, you stated that an NLI-fine-tuned SimCSE performs better. Please clarify.

Looking Forward to Further Discussion

Dear reviewer CcDN, We would like to thank you again for the feedback and suggestions. We hope our clarification and results on the setting you mentioned addressed your concerns. We are committed to improving our manuscript and would appreciate it if you find our response adequate and consider raising the score.