Self-Retrieval: End-to-End Information Retrieval with One Large Language Model

Summary

This paper introduces Self-Retrieval, an end-to-end IR system driven entirely by a single LLM. This model integrates all essential IR functions—indexing, retrieval, and reranking—into the LLM's architecture. By internalizing the retrieval corpus through self-supervised learning, the model transforms the retrieval process into a sequence of passage generation tasks and conducts self-assessment for reranking. The authors provide experimental evidence showing that Self-Retrieval outperforms traditional sparse, dense, and generative retrieval methods on benchmark datasets like NQ and TriviaQA.

Strengths

1. The integration of all IR functions into a single LLM is a novel contribution that leverages the inherent capabilities of LLMs across the full spectrum of IR tasks, offering a streamlined and potentially more effective approach. 2. The concept of Self-Retrieval is introduced clearly, making it accessible to readers. The detailed explanation of how the LLM handles indexing, retrieval, and reranking provides a good understanding of the system's operation. 3. The paper presents good experimental results that demonstrate significant improvements over existing retrieval methods.

Weaknesses

1. My major concern is that the experimental settings are inconsistent with existing work, making the results unconvincing. Specifically, - Most existing studies conducted experiments on NQ@320k datasets, but the main experiments of this paper are conducted on NQ@40k datasets. It is important to explain the reason of this setting. - According to the statistics in Table 2, each document in the dataset is split into 26~29 passages, which indicates that the passages are quite short. It is known that retrieving short passages are easier for generative retrieval methods. - It is necessary to explain why different models are selected for NQ@40k, TQA@40k, and NQ@320k experiments. - It is better to include more experimental results on the full KILT datasets as many existing studies for a fair comparison. 2. In Section 3.4, the three tasks are learned in a "1+2" manner. Please add more explanations on this design and provide experimental evaluation on other possible strategies (e.g., training the three tasks jointly) 3. Ensuring consistent use of key terms throughout the paper would improve its readability and professionalism. For example, all "large language models" should be written in "LLMs".

Questions

Please see my concerns in weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

n/a

Summary

This paper proposes Self-Retrieval, an LM that retrieve, rerank passages, and generate answers using a single model. For the retrieval task, it adopts generative retrieval and directly generates passage text. For reranking, it utilizes the generation probability as the relevance score. For answer generation, it generates answers based on the top-1 passage. Evaluation shows that Self-Retrieval outperforms previous dense and generative retrievers in retrieval tasks and achieves better EM scores in answer generation tasks.

Strengths

1. Self-Retrieval consolidates the multi-step RAG pipeline with a single model. The concept is novel. 2. Self-Retrieval achieves the best performance for both passage retrieval and answer generation tasks. 3. Self-Retrieval shows promising results when the corpus size scales to 3 million.

Weaknesses

1. Generating passage content is time-consuming. This paper could analyze the latency of Self-Retrieval and compare it to other alternatives, such as generating spans (ref SEAL) or keywords (ref Term-Set Generation). 2. The proposed model includes an in-domain fine-tuned reranker, while the baseline BGE-FT + reader does not have a reranking stage. This may make the comparison unfair since reranking can significantly improve RAG results. 3. The evaluation are all based on Wikipedia. It is unclear whether the model can perform well on a corpus that is not as well-structured as Wikipedia. 4. The different steps in Self-Retrieval are independently optimized, and the `knowledge sharing and deep collaboration` effects of this consolidated model have not been validated.

Questions

1. Compare the efficiency and effectiveness of using shared models versus separate models for the three steps in Self-Retrieval?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations are adequately discussed.

Summary

This paper introduces Self-Retrieval, a new generative retrieval architecture. Self-Retrieval first memorizes the corpus into LLM's parametric knowledge using self-supervised training. Given a query, it generates the target document with constrained decoding, then re-assess document by decoding if the document can answer the query. On subset of NQ and TriviaQA, this approach significantly outperforms existing dual encoders and generative retrieval models.

Strengths

- Intuitive architecture for using LLM for retrieval. The paper presents a self-supervise object to help model memorize the corpus. Then it integrate retrieval and reranking into a single constrained decoding process. The "reranking" process can be viewed as self-critique / chain-of-thoughts, and may potentially unlock deeper integration between retrieval and reasoning. - Strong quality improvements. The paper reports substantial improvements over previous dual-encoder approaches and generative retrieval models. - Ablations shows that the model scales well with model size. Previous dense retrievers often plateau due to the bottleneck layer; the scaling curve of this new architecture is promising.

Weaknesses

- Experiment only used wikipedia-based datasets. However, wikipedia is heavily used in pretraining, so it is unclear if the proposed approach can let model sufficiently memorize other datasets. - More importantly, the method relies on generating the passage title. Unlike wikipedia, many retrieval datasets do not have high-quality, natural language passage titles. It is unclear how the method works on those datasets. It would be nice to test retrieval benchmarks like MS MARCO or BEIR/MTEB. - Missing dense retrieval + cross-attention reranking baselines. Such 2-staged pipeline is standard in IR. Since the proposed method's reranking stage essentially uses cross attention to judge the query and the retrieved candidate passage, the computational cost of the reranking stage is similar to that of a separate cross-attention reranker. It is fair and necessary to compare it with commonly-used rerankers such as MonoT5, RankT5, or BGE reranker. The ablation in Table 3 seems to show that the proposed method's retrieval-alone performance is stronger than most retrieval baselines, but the paper can be more convincing if having e2e comparison to other 2-stage retrieval pipelines like BGE + BGE reranker or GTR + RankT5. - Lacking efficiency discussion.

Questions

- Can the method scale up to the full NQ/TriviaQA? If so, would be nice to see the performance on these more standard setups. If not, what is the main bottleneck for scaling up? - How many candidates were considered in the reranking stage?

Rating

6

Confidence

5

Soundness

3

Presentation

4

Contribution

3

Limitations

NA

Summary

The paper proposes an approach of self-retrieval, which uses the probability of generation of the passage as the ranking criterion. To limit the generation to the existing passages, a trie-structure is used, forcing the generation to produce the existing passages. The experiments compared the method with several existing approaches, including sparse retrieval, dense retrieval and generation-based retrieval. The proposed method outperforms the others.

Strengths

The idea of relying on the generation of an existing passage for ranking is interesting. The use of trie structure to constrain the generation to the existing documents is also nice. The proposed method thus has some novelty compared to the literature. The experimental results are very good, showing improved performance on document retrieval and QA.

Weaknesses

A key idea is the use of trie for passage generation. This is described briefly. More details should be presented. If a whole passage should be generated using trie, then the depth of trie structure will be very large (equivalent to the length of the passage). What about the storage cost of trie? What is the time efficiency?

Questions

Have you evaluated the space and time complexity of the approach, and compare it to the existing methods? How do you deal with long document (in particular for the trie structure)?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes