Model-enhanced Vector Index

Summary

In this study, the authors present a new approach called Model-enhanced Vector Index (MEVI) that combines ideas from both autoregressive sequence-to-sequence models for indexing and dense retrieval models (twin-tower architectures), and includes Residual Quantization. This approach offers significant advantages and proves to be highly effective in real-world applications. MEVI demonstrates nice performance gains in terms of achieving both high recall accuracy and faster retrieval speed, even when dealing with large-scale corpora.

Strengths

* The method introduced combines two of the recent approaches for document retrieval - dense retrieval and the generative neural indexing. It showcases the power of these together and is extensively evaluated and experimented, both in an offline manner as usual, but also online in real-word scenarios (in commercial advertising). They also include a nice latency evaluation. * The paper is well-written and easy to follow in general.

Weaknesses

Major comments: * MSMARCO is the only benchmark which was evaluated in this work. I’d love to see if the method generalizes well across more datasets. This is required in order to convince readers to favour this method. * While an online A/B testing experiment appreciated, there is no guarantee about the quality of such kind of testing, while controlled crowdsourcing does provide an infrastructure for filtering out bad annotations. * L152: I am not convinced that the tree structure is problematic because of decoding time. It can actually alleviate the decoding if using controlled decoding instead of plain beam search. Also, the hierarchical nature of the structure of embeddings proposed in this work is not that different (as stated in L167). * The authors report results using MRR@10 Recall@50 Recall@1000 but it woul’d be interesting to see both more fine-grained retrieval results (Recall@1) following previous works, as well as a figure which states how the method results progress as k is being increased. The training process of the twin-tower model and the sequence-to-sequence model are distinct which may be very problematic. But I do acknowledge this was stated in the limitations section. Minor comments: * Please elaborate more both in the contributions part in the introduction section about and in the introduction itself more about the proposed MEVI method. After reading this, I didn’t have any clue about the method which should be stated much early in the text. * Related work section - please state how your method differs from other recent works. * L146: insert space after T5-ANCE. Similar comment for the paragraph starting in 149 and the rest of the paper. * L157-166: The mathematical notations lack clarity in terms of explicitly defining the dimensions of the vectors/matrices and what they represent. * Table 4: please include detailed caption so the table can be understood standalone. The same applies for the rest of the tables.

Questions

See weaknesses above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes.

Summary

This work introduces residual quantisation codebook into k-mean clustering to generate semantic IDs. These semantic IDs are used to provide initial cluster-level ranking and prune the corpus into document subset with thousand documents. The interpolation between the dual-encoder similarity and a derived cluster score is used as final ranking. The experiments are on msmarco passage dataset.

Strengths

- An novel idea is implemented and examined to exploit the semantic ids from generative retrievers to introduce clusters into dual-encoder - The writing is easy to follow and the method is insightful - Promising results are reported on large in-house dataset Ms Marco and on an online system

Weaknesses

- The cluster-based retrieval has been long studied and this paper should be put into context better, e.g., [1, 2] . Besides, is this the first work to introduce clusters into dual-encoder retriever? - The effectiveness improvement in Table 1 comes from the fine-grained clustering information. Does such boost still exist when using state-of-the-art dual-encoder retrievers, like rocketqav2 and GTR, where the document similarity is with higher quality? Actually, in Table 1, it can be seen that use of HNSW could boos the recall considerably. Also, the results of the online system are also on top of weak dual-encoder models (four layers). Thus, it is beneficial to better understand when and where the clustering information could help. - For the dynamic update of documents, is this the first work that investigates this problem? What are the existing methods tackling this problem and how they design the experiments? The results from Table 2 look promising, but I am not sure how convincing it is to support that claims of dynamic updates. - As an ablation results, Table 3 is hard to read, and the configuration differences among rows are more than one out of three (subgroup, bits, top-k), and it is hard to answer by increasing subgroup or k it helps. - Some formatting issues: a space is required between the text and their follow-up citation, like line 150 1. https://dl.acm.org/doi/pdf/10.1145/253495.253524 2. https://arxiv.org/abs/2008.00150

Questions

- What is the configuration of \alpha and \beta and how sensitive is the performance relative to their configs?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper proposes to improve dense neural IR models by adding a RQ structure (hierarchical clustering based on residuals at each step) before the ANN search. The structure is used to construct a semantic identifier string for each document. The authors thus use a generative approach to retrieval combined with a dense model. This approach has two advantages, firstly to have a faster retrieval (since the ANN search is restricted to a subset of the document), secondly to have a better retrieval (by exploiting a cluster-based score). Experiments conducted on MS Marco (passages) and a proprietary advertisement dataset show improvements over T5-ANCE.

Strengths

The paper is well written and technical details ensuring reproduction are given. The approach is promising as it both allows for more effective and efficient retrieval - which is always a challenge in IR. The proposed approach, based on RQ clusters, is original and combines the strength of generative and dense models in IR. It also shows that adding documents is not as problematic as for other generative approaches (but however the performance is worse than T5-ANCE in that case, for a higher latency).

Weaknesses

- Comparison with state of the art models (A2R/Adversarial Retriever-Ranker for dense models, SPLADE for sparse ones) is missing - it would have been good to see how with better dense models the proposed approach evolves. - With respect to generative models, several approaches have better results than NCI nowadays (e.g. Ultron) or the more recent “Learning to tokenise” (SIGIR’23). At least Ultron should be reported (ArXiv 2022) for comparison purposes. - The authors have chosen to use a proprietary dataset (section 4.7) for the second evaluation. It would have been much wiser to use publicly available ones as nothing can be checked on this one (e.g. BEIR) - reporting proprietary results in the appendix would have been appropriate.

Questions

- what is the performance of ANCE without HNSW? This would have been a proper baseline. - in section 3.2, how are $\alpha$ and $\beta$ set? - What it the advantage of using RQ over hierarchical clustering?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitation section contradicts a bit the initial claims (“… the capacity of the sequence-to-sequence model is still insufficient to cope with large-scale corpus within acceptable inference latency …”), but correctly lists the different limitations of the paper.

Summary

This paper present a new method for ensembling generative retrieval and embedding-based dense retrieval. Prior generative retrieval work that solely relies on a generation model to generate the retrieval document, but it is difficult to scale to large corpus. In contrast, this paper uses the generation model to predict candidate document clusters, but still adopt a dense retriever to produce the final document rankings, where the document from different clusters are weighted according the generation model's output. The authors first evaluated this method on MS MARCO. The proposed method show improvements over a dense retriever T5-ANCE, and also good support to corpus update. The authors also tested this method in a production ad system where improvements were also observed over their production dense retrieval model (a 4-layer transformer dual-encoder).

Strengths

- This paper proposes a simple idea to ensemble generative retrieval model and embedding-based dense retrieval. In the proposed method, the generative retrieval model is essentially predicting document clusters from a query which are then used for reweighting or pruning the dense retrieval results. This aproach is a lot more practical and scalable compared to existing generative retrieval approaches that need to generate the exact document identifier. - Experiments show promising results. - Paper is well written.

Weaknesses

- The authors did not compare to any published neural retrieval baselines on MS MARCO, as T5-ANCE and NCI are both their own implementations. I'm wondering how does the method work with SOTA dense retrievers such as RocketQA v2 or coCondensor, whose results seem stronger than T5-ANCE on MS MARCO. - It would be nice to add another dataset, such as Natural Questions. - It is unclear to me what is the advantage of RQ compared to other hierarchical clustering approaches. It would be nice to add some discussion and ablation studies. - There are some confusions around T5-ANCE. First of all, T5-ANCE does not seem to exactly follow the ANCE recipe which uses dynamic indexing for hard negative mining. Second, it is unclear the model size and hyperparameters for this model. - I think is is an over-claim to say "For the first time, we demonstrate that a novel-designed generation-based model is able to handle a large corpus with millions of documents, reaching high recall performance and low serving latency at the same time." From my understanding, this method mostly rely on dense retrieval and generation-based model is only providing cluster-level weights. To justify this claim, the author should report the performance without ensembling with dense retrieval.

Questions

- How does the model compares or works with SOTA dense retrievers on MS MARCO? - What is model size of the generation model? How does it affect serving latency? - How's the performance with just the generation model?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No

Summary

This research paper introduces a deep text retrieval model that possesses the capability to effectively manage a large corpus comprising millions of documents. The model achieves remarkable recall performance while maintaining relatively low latency. Moreover, in addition to surpassing the performance of existing methods in a document retrieval benchmark, the paper also showcases the successful integration of deep text retrieval models into a commercial advertising system, demonstrating their practical value in an industrial setting.

Strengths

1. S1: They are supposed to be the first to demonstrate the successful implementation of deep text retrieval models in the practical application. 2. S2: They investigate and test the effectiveness of Residual Quantization within the deep text retrieval model. 3. S3: The experiment results on MSMARCO dataset display the effectiveness of the proposed method.

Weaknesses

W1: They conduct experiments on a single dataset for their study. However, it is recommended that their model undergo experiments on commonly used datasets such as Natural Question, which is utilized in both DSI and NCI research. In addition, they point out that the existing methods are limited by unacceptable serving latency but they do not conduct such experiments to prove this. Since their method use Residual Quantization (RQ) as their clustering method, this method may be more time-consuming than other clustering methods during training.

Questions

1. In dynamic update experiments, if more documents come in, their codebook may not be "optimal", so how can the codebook be adjusted to preserve similar effectiveness and robustness? Otherwise, the resulting method may not be good. In addition, their Top-10 results are worse than T5-ANCE. 2. More experiments are needed. They should also compare their method with one another datasets used in DSI or NCI. And the latency experiment should include other baseline models. 3. If RQ is applied to DSI and NSI, does their method achieve better performance or does the utilization of normal k-means in the proposed method potentially diminish its effectiveness to a significant extent?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

The authors pointed out the limitations of their model including not jointly learning between twin-tower model and seq2seq model, and their method is still unacceptable inference latency in the large-scale corpus. They had provided potential solutions to address the above issues.