Recommender Systems with Generative Retrieval

Summary

This paper proposes a generative retrieval approach wherein the model autoregressively decodes the identifiers, called Semantic ID, for the target candidate. They propose a method to generate the semantic ID for items which is a tuple of codewords which are trained from the embeddings of the text component of the items obtained using a standard sentence BERT type of model. Finally, a seq2seq model is trained to predict the semantic ids for the next item given the current item. The authors show that the proposed technique outperforms SOTA techniques on some retrieval datasets. They also claim that the semantic ID trick enhances the generalization property of the seq2seq model especially for cold start cases.

Strengths

Originality: The paper seems to present a novel idea which is original enough. The items are represented in a novel semantic ID sequence form which is then used in a standard seq2seq framework. Clarity: I think the paper is clear enough to understand and reproduce.

Weaknesses

The method itself is and its larger claim of generative in nature is the primary concern I have with the paper. The proposed method tries to generate semantic id sequence for items already seen by the system. Any novel combination of semantic ID will lead to an item which is not part of the retrieval system. This leads to the authors constraining the semantic IDs to just 3 code words and then an ad hoc 4th code word to introduce uniqueness in the codes. This part seems to suit small, well controlled item sets, but the scaling out of such a system to billions of items seems not clearly explained or experimented with. The experimental results seem not consistent with the published paper, especially for P5 paper. The numbers reported in the paper are higher than this paper which are not shown as is, but the authors implement their own P5 and compare results which are way different than the published results. The other comparative results are all copied over from the benchmark mentioned in the footnote ttps://github.com/aHuiWang/CIKM2020-S3Rec.

Questions

I would like to see large code sets for items. This would make the system truly scalable. Otherwise, constraining embeddings to 3 codewords seems a step in the opposite direction which will work with very small item sets in controlled environments. I would also ask the authors to perform comparative experiments with P5, where the parameters do not need to be comparable, but the best performance obtained should be the primary aim.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

1 poor

Limitations

NA

Summary

The paper is trying to approach sequential recommendation task using semantic IDs. By incorporating semantic IDs, the paper shows that the retrieval performance can be improved.

Strengths

1. The ablation study conducted is good. For example, it compares RQ-VAE against LSH, and random. 2. The proposed method and framework is intuitive and makes sense. 3. The conducted experiment seems extensive and the proposed method is shown to outperform than non-semantic-id method like S3-Rec.

Weaknesses

1. The paper title seems to emphasize on Generative Retrieval rather than on ranking, if that is the case, in the experiment section, shouldn't we care more about recall than NDCG? Why proposed method seems to improve NDCG a lot but much less so on recall metrics? 2. The novelty of the proposed method seems limited, the main idea seems to overlap with paper "Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders", and probably should be considered as incremental value added to the previous published paper. Although the method emphasized on generative AI, it is still generative on the item semantic ID level, it would be more significant if the generative mechanism is on the word tokens, in my opinion.

Questions

Besides the questions in the weakness section above, another question is 1. Semantic ID idea seems useful, but would it be nicer if we do not need to use semantic embeddings as middle layer and directly do generative AI on word tokens?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Summary

The paper proposes TIGER, a generative retrieval approach to sequential recommendation. More specifically, TIGER proposes to use semantic IDs of a product (based off RQ-VAE quantization of sentence-T5 product feature embeddings) instead of typically used atomic item-IDs in the sequential recommendation pipeline. Empirical analysis on 3 Amazon subsets show promising results compared to itemID-only and ID+text based existing recommenders.

Strengths

- Interesting design to use quantized embeddings of items (semantic IDs) - Successful application (i.e., outperforming atomic ID-based recommenders) of the generative retrieval paradigm for sequential recommendation - State of the art results on medium-sized academic datasets

Weaknesses

- No ablations, e.g., effect of number of codebooks, prepending user-ID, codebook size, Transformer architecture, etc. - No significance tests while reporting ranking results

Questions

- It seems the text+ID based recommenders are not consistent in what product features are being used. E.g., S3-Rec seems to use category and brand information whereas FDSA seems to use category, brand, and description. Is the improvement in TIGER just because of using extra feature signals like title and price? - Unclear which two semantic IDs are “closer”? I had to assume that since a residual quantization is being performed, being close to the first quanta holds more importance than the last.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

I will use this space to provide some (hopefully constructive) suggestions for this paper: - Best to outline in Table 1 that S3-Rec and FDSA use item-text features whereas all others are ID-only recommenders. - It would be interesting to see the performance on the unprocessed Amazon datasets since arbitrary preprocessing of user-data has been noted to interfere with true performance [1]. - It would be interesting to see a TIGER variant where only three-length semantic IDs are used (i.e., non-unique items would be allowed) as it would become a text-only recommender. - Complementing the zero-shot analysis in the paper with an analysis of TIGER’s performance over the user/item coldness spectrum (in comparison with other models) - Although released after this submission to NeurIPS, [2] might be a good reference to compare TIGER with (this comparison does not affect my review). [1] Sachdeva, N., Wu, C. J., & McAuley, J. On sampling collaborative filtering datasets. WSDM ‘22. [2] Li, J., Wang, M., Li, J., Fu, J., Shen, X., Shang, J., & McAuley, J. Text Is All You Need: Learning Language Representations for Sequential Recommendation. arXiv preprint arXiv:2305.13731.

Summary

This paper proposes a generative retrieval for an efficient recommendation, where the retrieval model autoregressively decodes the identifiers of the target candidates. The evaluation results show the proposed method can perform better than competing baselines on various dataset.

Strengths

1. Generative retrieval-based recommender model with item features 2. Extensive evaluation show the superiority of the proposed algorithm

Weaknesses

1. Limited contributions. This paper claims the first to propose generative retrieval for recommendation systems using Semantic ID representation of items. However, this is not true. RQ-VAE is also simply used for obtaining the semantic ids, and no improvements are made in the paper. 2. The similar idea is mainly used for efficient recommendation, but no efficiency study is performed. 3. The settings of some baselines are not reported. I know that SASRec could be significantly improved using some tricks. It is not clear whether these tricks are used in the experiments. 4. The comparison is not fair. The proposed algorithm contains side information, but some baselines do not take them into account. 5. The experiments are only performed on the Amazon datasets. It is not sufficient to prove the effectiveness of the proposed algorithms.

Questions

see the attachments.

Rating

2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

1 poor

Limitations

N/A

Summary

The paper introduces TIGER (Transformer Index for Generative Recommenders), a novel generative retrieval model for sequential recommender systems. This innovative approach involves generating Semantic IDs—sequences of semantic tokens derived from an item's text features. A hierarchical quantizer (RQ-VAE) is employed on content embeddings to formulate these tokens, thereby allowing the model to predict the Semantic IDs for generative retrieval. The authors claim that TIGER outperforms current state-of-the-art models, while offering a new solution for cold-start recommendations and increasing item diversity by adjusting the temperature parameters. Furthermore, it showcases impressive scalability and the capacity to generalize to novel and unseen items.

Strengths

1. This paper is well-written and technically solidity, effectively combining novel concepts such as Differentiable Search Index (DSI) using Transformer memory and a hierarchical quantizer, RQ-VAE, for generative retrieval in sequential recommendation task. 2. The idea driving the work is both logical and compelling, providing a fresh perspective on generative retrieval tasks in the context of recommender systems. 3. Empirical results show that TIGER not only outperforms the SOTA baselines on the SR task but also exhibits an ability to generalize to unseen items and enhance item recommendation diversity.

Weaknesses

1. The paper could benefit from more detailed information about the text encoder's input, given that quantization is a crucial part of Semantic ID generation. It's unclear whether all components depicted in Figure 2(a) are utilized. Additionally, handling missing information/data in datasets, such as the Amazon Beauty dataset, needs addressing (E.g., in the Amazon Beauty dataset, approximately 50% of the brand information is missing.). The authors should explain how they preprocessed the data. 2. The paper's treatment of related work and baseline comparisons raises some concerns. Specifically, the performance results of the P5 model appear notably lower than what's presented in its original paper. The authors have explained the data reprocessing using a random item ID, but a comparison using the more updated benchmark in [2] - which also uses a random item ID - could offer a more balanced analysis. It would be insightful to understand the reasons behind the observed performance degradation of the P5 model in authors’ setting. 3. The proposed solution to collision problems, appending an extra token at the end of the ordered semantic codes, raises questions about scalability. For instance, if ten items share the Semantic ID (12,24,52), would the model need to add two extra tokens and maintain a new lookup table? 4. The experimental design for cold-start recommendations in section 4.3 could be revisited. The removal of 5% of test items from the training data split might change the original training sequences, potentially leading to shorter sequences and artificially enhancing recall results in the cold-start retrieval setting. 5. A comparison with additional ID encoding methods, similar to those presented in [2], could strengthen the argument for RQ-VAE's superiority and effectiveness. 6. While the authors state that inference cost is not their main contribution, a rough comparison of TIGER's inference cost to LSH and NN-based methods would help to understand this method's trade-offs. References. [1] OpenP5: Benchmarking Foundation Models for Recommendation(https://arxiv.org/pdf/2306.11134.pdf) [2] How to Index Item IDs for Recommendation Foundation Models. (https://arxiv.org/abs/2305.06569)

Questions

Please refer to the weaknesses section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Please refer to the weaknesses section.

Comparison of RQ-VAE with VQ-VAE

> A comparison with additional ID encoding methods, similar to those presented in [2], could strengthen the argument for RQ-VAE's superiority and effectiveness. As requested by the reviewer, we have included an additional ID encoding method VQ-VAE [4] as a baseline for the Beauty dataset. Please note that we have also compared our method with LSH and random-id based encoding schemes in the main paper. A comparison is shown below: | | Recall@5 | NDCG@5 | Recall@10 | NDCG@10 | |---------------------------------------------------|----------|---------|-----------|---------| | Semantic IDs using VQ-VAE | 0.04306 | 0.02994 | 0.06417 | 0.03673 | | Semantic IDs using RQ-VAE (Ours) | 0.0454 | 0.0321 | 0.0648 | 0.0384 | As seen, the RQ-VAE-based item ID encoding performs better than VQ-VAE. We would also like to emphasize that with RQ-VAE the generated Semantic ID tokens are hierarchical which opens possibilities for new capabilities (namely, cold-start recommendations & controllable recommendation diversity) as described in Section 4.3 of the paper. [4]: Oord, Aaron van den, Oriol Vinyals, and Koray Kavukcuoglu. "Neural discrete representation learning." arXiv preprint arXiv:1711.00937 (2017).

Results for additional experiments

> No ablations, e.g., effect of number of codebooks, prepending user-ID, codebook size, Transformer architecture, etc. As requested by the reviewer, we performed some ablation studies, and we see that our technique consistently gives good results across varying setups. Below we present results for the Beauty dataset. Ablation results for transformer architecture: | Layers | Recall@5 | NDCG@5 | Recall@10 | NDCG@10 | |---------------------------|----------|---------|-----------|---------| | 3 | 0.04499 | 0.03062 | 0.06699 | 0.03768 | | 4 (reported in the paper) | 0.0454 | 0.0321 | 0.0648 | 0.0384 | | 5 | 0.04633 | 0.03206 | 0.06596 | 0.03834 | Ablation results for the with and without user information setups: | | Recall@5 | NDCG@5 | Recall@10 | NDCG@10 | |--------------------------------------|----------|--------|-----------|---------| | No user information | 0.04458 | 0.0302 | 0.06479 | 0.0367 | | With user id (reported in the paper) | 0.0454 | 0.0321 | 0.0648 | 0.0384 | >No significance tests while reporting ranking results We computed the standard error for our recall and NDCG metrics for the datasets by running the training and evaluation 3 times using different random seeds. As we can see, the standard errors are quite small as compared to the recall and NDCG metrics. |Mean ± Standard Error | Recall@5 | NDCG@5 | Recall@10 | NDCG@10 | |---------------------|-----------------|------------------|-----------------|-----------------| | Beauty | 0.0441 ± 0.00069 | 0.0309 ± 0.00062 | 0.0642 ± 0.00092 | 0.0374 ± 0.00061 | | Sports and Outdoors | 0.0278 ± 0.00069 | 0.0189 ± 0.00043 | 0.0419 ± 0.0010 | 0.0234 ± 0.00048 | | Toys and Games | 0.0518 ± 0.00064 | 0.0375 ± 0.00039 | 0.0698 ± 0.0013 | 0.0433 ± 0.00047 |

Scalability experiment for Semantic IDs

> Scalability To test the scalability of Semantic IDs, we ran the following experiment: We combined all the three datasets and generated Semantic IDs for the entire set of items from the three datasets. Then, we used these Semantic IDs for the recommendation task on the Beauty dataset. We compare the results from this experiment with the original experiment where the Semantic IDs are generated only from the Beauty dataset. We see that there is only a small decrease in performance here. | | Recall@5 | NDCG@5 | Recall@10 | NDCG@10 | |-------------------------------------|----------|--------|-----------|---------| | Semantic ID generated from combined dataset | 0.04355 | 0.3047 | 0.06314 | 0.03676 | | Semantic ID generated from only the Beauty dataset (reported in the paper) | 0.0454 | 0.0321 | 0.0648 | 0.0384 |

Author response to the reviews

We thank all the reviewers for their time and effort in reviewing our paper and providing constructive feedback. We are glad to know that the reviewers found the proposed technique novel (Reviewers VYRN, 5DSM, 5EdZ), that it outperforms the existing methods (Reviewers VYRN, mHfk, 5DSM, aLHH), that the paper contains extensive evaluation (Reviewers mHfk, aLHH) and is well written (Reviewers VYRN, 5EdZ). We have tried to address the concerns of each reviewer below. We have also added the results for the experiments requested by the reviewers, we request the reviewers to please take a look.

Reply to the rebuttal

> We would like to state again that the focus of this paper was not on efficient recommendation, but rather proposing a new paradigm for recommender systems. The first sentence in the abstract is "Modern recommender systems perform large-scale retrieval by first embedding queries and item candidates in the same unified space, followed by approximate nearest neighbor search to select top candidates given a query embedding. In this paper, we propose a novel generative retrieval approach... I am not convinced by "the focus of this paper was not on efficient recommendation" > It would be helpful for us if you could please provide a link to the prior work that performs generative recommendation using semantic IDs. The related papers I know are as follows: [1] Learning tree-based deep model for recommender systems, KDD 2018. [2] Recommender Forest for Efficient Retrieval, NeurIPS 2022. Note that though the first paper (TDM) is not about using semantic IDs, it can be considered as generative recommendation using beam search. The difference is whether to use Transformer to encode the routing trajectory. In TDM, since each node has its own embedding, using the routing trajectory to represent the node in the tree may have a subtle effect. > It would be very helpful if you could point us to any information regarding that. I just read the code, they use the binary cross-entropy loss in SASRec. In fact, SASRec can be significantly improved by using alternative loss functions, such as the cross-entropy loss or the sample softmax loss.