Generative Retrieval Meets Multi-Graded Relevance

Summary

The paper targets the problem of Multi-Graded Relevance in a Generative Retrieval setup. Generative Retrieval uses a Seq2Seq model to decode relevant “docids” for a given query. Prior work on Generative Retrieval has focused on tasks where there is only one relevant document per query and the degree of relevance is binary : “relevant” or “not-relevant”. The paper talks about Multi-graded Retrieval tasks, where each query has multiple relevant documents, each having its own degree of relevance. The paper proposes a methodology to tackle the graded retrieval tasks in a Generative Retrieval Framework. It identifies the key challenges : 1. Having distinct but relevant “docids” and 2. Graded Relevance when matching query and documents. It proposes solutions to these challenges in “regularized fusion” and “MGCC” approaches respectively. Additionally the authors also develop and experiment with a pre-training setup for multi-graded relevance retrieval.

Strengths

1. The problem of multi-graded retrieval in Generative Retrieval is relevant/important with no prior work 2. The challenges are well-identified i.e., the goal of having distinct but relevant docids for documents. 3. The solutions proposed are well formulated and intuitive to understand. 4. Empirical Results are strong with thorough ablations and detailed discussions. 5. The authors propose and experiment with a pre-training framework, showing positive results.

Weaknesses

While not a strong weakness, the mathematical notation in the paper can use some work to make the paper an easier read. I found it tough to parse through “Section 3 : Preliminaries”. The paper would benefit from resolving this.

Questions

I just have a few clarifying questions : 1. Are there numbers on Recall? Prior work I believe focuses on MRR and Recall metrics. 2. Why include the last term in Eq2 (similarity in embedding space) when the same is taken care of by Eq1? Is it for training stability? 3. If I understood correctly, then docids are generated by decoding a random sample on the hypersphere (z=e+r). Does that mean if we were to sample multiple times for each document, we would get distinct docids from the randomness in r itself? Do we need e of documents (through QG module) to be “distinct” (i.e. do we need regularized fusion if we have better sampling or more samples of r?)

Rating

7

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Typo : In line 188, “These fixed” needs “T” to be lowercase While not a limitation, I believe the paper would also benefit from a few Qualitative examples showing Multi-graded relevance in practice. An uninitiated reader might not be familiar with Generative Retrieval or Multi-graded relevance and a figure/qualitative example might be useful for them.

Summary

The work deals with generative retrieval for cases where documents have multi-graded relevance. The authors propose GR2 a framework for generative retrieval in such cases by tackling two important challenges. First they optimize for relevance and distinctiveness of document IDs by a regularized fusion approach which comprises a pseudo query generation module followed by an auto-encoding module that reconstructs the document ids. Secondly, the authors introduce a graded, constrained contrastive learning approach which aims to bring together representations of queries with the representations of the relevant docids and push apart irrelevant docids in mini-batch. GR2 shows impressive performance improvements over other approaches and is also relatively efficient.

Strengths

- The work tackles an important task of generative retrieval for cases of documents with multi-graded retrieval. Compared to existing works like [2], the work captures the relationship between labels using a graded contrastive loss leading to impressive gains over existing generative IR models and dense IR models. - The loss functions are well motivated and shows the impact of contrastive learning on generative retrieval. I think further studies on effect of different contrastive losses would be an interesting avenue to explore. - The authors perform extensive experiments and ablations of the proposed method to reinforce the importance of each of the components. [1] Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval LEE et. al. [2] Learning to tokenize for generative retrieval.

Weaknesses

- Some important baselines are missing in the current version of the work. ColBERT (ColBERTV2) in PLAID setup is a powerful dense retriever and would serve as a valuable approach for comparison. Additionally, dense retrieval models like contriever and tas-b have also demonstrated impressive performance on a wide range of benchmarks and are strong dense retrievers due to their training objectives and should be included for comparison. Particularly since GR2 is also used in pre-training setup it is important to compare to retrievers like Contriever. pre-trained using contrastive learning for retrieval. - While there is clear intuition for the distinctiveness, relevance regularization losses and the MGCC loss, further experiments could be carried out with respect to the contrastive learning parts of these approaches to understand the approach better and to help improve or strengthen the results. For instance, negative sampling is an important aspect of contrastive learning and a good mix of hard and soft negatives are crucial for learning useful representations. While in MGCC authors explore mini-beach based negatives, it is also important to explore global negatives similar to works like ANCE [1]. - The paper would benefit from some error analysis. This involves analysis of cases where wrong docids are considered relevant, and an analysis of whether it stems from the stochasticity of generative models or other external factors. The paper does not currently report any list of common errors or a qualitative analysis of results. It is critical to understand where the current approach fails as it would provide avenues for further research. [1] Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval LEE et. al. [2] Learning to tokenize for generative retrieval.

Questions

- Equation 10 gives equal weights to the regularization losses. Have you considered weighting them differently ?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

While authors briefly describe the limitations in conclusion a more detailed explanation would help contextualize the work and understand future directions better.

Summary

The paper proposes a novel QG based docid for generative retrieval, optimising relevance and distinctness of the generated queries jointly. It introduces the MGCC loss with multi-graded labels. Experiments on subsets of Gov2, ClueWeb09-B, Robust04, MS Marco, and NQ with up to 500k documents validate the effectiveness of the methods. Overall, the contributed methods are valuable and inspiring, and the results are convincing.

Strengths

- Jointly optimizing relevance and distinctness with an AE block is a nice idea, effectively balancing both factors as demonstrated by ablation studies. - Creating semi-supervised graded PT data from Wikipedia seems very useful, with remarkable improvements from this additional training. - The paper is well-presented and the experiments are comprehensive.

Weaknesses

- Limited corpus scale remains a concern for various GR works, further discussion on its potential influence on the proposed method is needed. Nonetheless, a 500k corpus is substantial and applicable. - The paper should discuss the existing works more about the use of QG in GR

Questions

- The term "pre-train" is somewhat misleading, as using semi-supervised Wikipedia data for further training differs from typical LLM pre-training. - Some examples for generated queries, both with and without using the RF building block, to better understand the reasons behind its effectiveness.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes

Summary

In this paper, the authors propose a new generative retrieval model, which utilizes multi-grade relevance labels instead of binary relevance. Using graded relevance labels is not well-discussed in previous works. Pros: - The problem itself is interesting and important. Generative retrieval is a hot research topic in the IR community. - The proposed solution is reasonable. Cons: - In the Introduction, the authors introduced the reason why the simple generation likelihood of docids cannot work for graded relevance. It is strange to claim that "Docids commonly exhibit distinct lengths" given the truth that some existing docid schemas (such as semantic id used in DSI, the PQ-based ID, etc.) Even for token-based docID, we can still add a fix-length constraint. It is hard to convince the readers by claiming that "as a fixed length might not adequately encompass diverse document semantics". - Both DocID schema design and learning from multi-grade relevance are considered in the paper. I am quite interested in learning whether the multi-grade relevance part can also work with other DocId schemas. So an ablation study about this (e.g., using semantic string as docids) should be given (note that this is different from the experiments in Section 5.2.2) .

Strengths

- The problem itself is interesting and important. Generative retrieval is a hot research topic in the IR community. - The proposed solution is reasonable.

Weaknesses

Cons: - In the Introduction, the authors introduced the reason why the simple generation likelihood of docids cannot work for graded relevance. It is strange to claim that "Docids commonly exhibit distinct lengths" given the truth that some existing docid schemas (such as semantic id used in DSI, the PQ-based ID, etc.) Even for token-based docID, we can still add a fix-length constraint. It is hard to convince the readers by claiming that "as a fixed length might not adequately encompass diverse document semantics". - Both DocID schema design and learning from multi-grade relevance are considered in the paper. I am quite interested in learning whether the multi-grade relevance part can also work with other DocId schemas. So an ablation study about this (e.g., using semantic string as docids) should be given (note that this is different from the experiments in Section 5.2.2) .

Questions

na

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

na