Authors' response (Part 1)
We thank the reviewer for the insightful and constructive feedback which are helpful in improving our study. We will respond to the highlighted weaknesses and questions in a point-by-point fashion.
> Some of the mistakes make it confused. (1) In Table 1, there is a wrong bold in the second column. (2) “Table 3”in the 7th page is typed as “Figure 3” by mistakes. (3) The citation at the end of the 5th page is not in the right format.
Thank you for catching these mistakes. We will fix them in the revision.
> The evaluation of the prompting strategies could benefit from increased interpretability. For instance, there is no experiment validating whether engagement-guided prompting effectively utilizes neighbor information to guide the LLM. It may seem counterintuitive to assert that important neighbor information is considered merely by including item names without a more detailed description and categorization.
We believe there is a misunderstanding about the engagement-guided prompting. In fact, the engagement-guided prompting strategy does NOT focus on item names, it includes the descriptions of other important neighbors. In Figure 1 and Section 2 (the last sentence of the 4th paragraph), we show that the prompt of the engagement-guided prompting is “Summarize the commonalities among the following descriptions: ‘description’; ‘descriptions of other important neighbors’”. Here the ‘description’ refers to the description of the target item, and ‘descriptions of other important neighbors’ refers to the item descriptions of other important neighbors. It does not represent item names.
> The paper lacks a detailed discussion on the potential limitations and challenges of using LLMs for personalized recommendation, such as computational complexity, model interpretability, and potential biases in the generated prompts.
We appreciate this suggestion and will include a detailed discussion on the potential limitations and challenges of using LLMs for personalized recommendation in Appendix. Here we briefly discuss the potential limitations and challenges:
**Computational Complexity:** The LLM-Rec framework focuses on enhancing input for personalized recommendations through generating item descriptions. This approach addresses the problem of original item descriptions often being incomplete or insufficient for effective recommendations. The primary computational load in LLM-Rec arises during the augmentation phase including the output text length, number of items, and the concatenation of text embeddings.
For output text length, our findings indicate that selecting important words for inclusion, rather than incorporating all response words, can lead to improved recommendation performance, as evidenced in Table 3. Future research could explore the balance between the number of words generated and the resulting performance enhancements.
For the number of items, to mitigate this, we propose potential strategies to reduce the need for augmentation across numerous descriptions. One approach is developing a model that evaluates whether an original item description already possesses sufficient information for recommendations. This avenue warrants further exploration in subsequent studies to decrease computational demands.
For the concatenation of text embeddings, as demonstrated in Table 2 and Figure 6, combining the embeddings of augmented text substantially enhances performance. However, this also increases the input dimension for the recommendation module, potentially raising computational costs. A solution could be to opt for a simpler model structure, such as an MLP, which we have implemented in LLM-Rec, to manage this increase in computational complexity.
**Model Interpretability:** LLM-Rec introduces an innovative approach to model interpretability by enabling direct comparisons between variations in generated prompts and corresponding shifts in recommendation performance. This relationship is illustrated in our research, as showcased in Figures 4 and 5, as well as Tables 3, and 9-16. Nonetheless, we propose that delving into a more granular, word-level analysis could yield deeper insights. By examining both the generated responses and the subsequent recommendation module, we can gain a more nuanced understanding of how specific textual changes influence overall model behavior, enhancing our comprehension of model interpretability.
**Potential Biases in generated responses:** The presence of potential biases in the prompts generated by LLMs is a significant concern, primarily because these biases are deeply ingrained in the data used to train the LLMs. This challenge calls for a multifaceted approach, including algorithmic adjustments, ethical guidelines, and continuous monitoring for bias, which are important future directions.