Summary
As one of the mainstream paradigms for document retrieval, generative approaches have enjoyed a steady growth of interest thanks to the recent thriving of large language models. Document tokenization is a crucial step in generative retrieval, which is rule-based in most existing methods, usually generalizing poorly to unlabeled documents. To this end, the paper presents a novel document tokenization learning framework GENRET that learns to tokenize a document into semantic docids in a discrete auto-encoding scheme. GENRET consists of a shared sequence-to-sequence-based document tokenization model, a generative retrieval model, and a reconstruction model, optimized in an end-to-end fashion using a well-designed progressive training scheme to stabilize the training process. Experimental results show that GENRET significantly improves over the baseline methods, especially in generalizing to unseen documents.
Strengths
1. The authors have identified a significant problem in existing generative retrieval approaches. Conventional practice typically treats document tokenization as a fixed pre-processing step; however, this is not optimal as ad-hoc document tokenization methods often fail to capture the complete semantics of a document and generalize poorly to unlabeled documents. Therefore, the paper proposes for the first time to take document tokenization as a learnable module and introduces a novel framework to represent documents as discrete docids that effectively capture the semantic information of the document.
2. The auto-encoding scheme is not new but reasonably integrated into the proposed framework. The progressive training scheme effectively addresses the challenge of learning docids in an autoregressive fashion and stabilizes the training process. The diverse clustering techniques successfully facilitate the diversity of generated docids.
3. Extensive experiments on various benchmark datasets have demonstrated the superiority of the proposed method against established document retrieval approaches.
Weaknesses
1. I think the presentation of the method description (Sec 3) could be further improved, given that some details are not clearly explained. For instance, according to Eq. 4, the reconstruction model is non-parametric, and the stop gradient operator is applied to $\mathbf{d}_t$ and $\mathbf{d}^*_t$, so which part of the parameters the reconstruction loss is aimed at optimizing? I guess it's probably the parameters of the codebook and the encoder-decoder. However, the authors mentioned that "the gradients to document representation $\mathbf{d}_T$ are defined as $\frac{ \partial \mathcal{L}_r}{ \partial \mathbf{d}_T} \coloneqq \frac{ \partial \mathcal{L}_r}{ \partial \mathbf{z}_T}$ " (in Line 173), but $\mathbf{z}_T$ is actually taken from the codebook based on Eq. 5, so the reconstruction loss $\mathcal{L}\_{rec}$ is to optimize the codebook only? Moreover, does the generative retrieval model $P$ share the same codebook with the document tokenization model $Q$? What about the encoder and decoder? In addition, this section also does not cover the inference phase after the training completion. Finally, I do not understand why the docid re-assignment technique can promote the diversity of generated docids.
2. All experimental evidence shows that the proposed approach is effective on purely quantitative measures. However, I do not see any qualitative experiments presented, e.g., by visualization analysis to demonstrate that the codebook embeddings are uniformly scattered in the document representation space or the diversity of generated docids is better than baselines, etc.
Questions
**Q1:** Please take a look at **Weakness1** for suggestions to improve the presentation.
**Q2:** In Eq. 8, is the final loss directly the sum of reconstruction loss, commitment loss, and retrieval loss? Should we set different weighting coefficients for each loss to balance the effects between them?
**Q3:** For the implementation details (Sec 4.3), the authors say that "the length of the docid $M$ is dependent on the number of documents present. For datasets containing a larger number of candidate documents, a larger value of $M$ is set to ensure that all documents are assigned unique document ids". So what $M$ is set to for each of the three datasets used in this paper? Is it possible to suggest how $M$ should be taken for a given number of documents from the perspective of theoretical analysis?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
As mentioned in the conclusion, the limitations of this work are two-fold. One is the lack of validation of the proposed method on large-scale data, and the other arises from the insufficient generalization of the model to unoptimized document collections in different types or domains.