Graph-enhanced Optimizers for Structure-aware Recommendation Embedding Evolution

Summary

In this paper, the authors propose a novel optimization algorithm that is talored for recommender systems. It incorporates graph structural information into the optimization process, aleviating the burden of performing GNN for RS. The convergence of the algorithm is theoretically demonstrated. Besides, it could be incorporated into existing well-performed optimizers like AdamW. The experiments are conducted to test its effectiveness on different types of recommendation models and consistent performance improvements are observed.

Strengths

1. This paper is innovative in its consideration of utilizing graph information the algorithm level, not model level, for recommender systems. The proposed new algorithm is theoretically guaranteed to converge. 2. The application of the proposed algorithm is widely discussed, including its incorporation into existing popular optimizers and its combination with knowledge distillation for recommender systems. 3. The experiments are extensively conducted, including different size of datasets, different types of recommendation backbones and baselines. The results demonstrate both the effectiveness and efficiency of the proposed algorithm for improving existing recommendation models.

Weaknesses

1. The technique in Section 2.4 is a little too specific for AdamW. 2. Table 5 seems to have an error. In the last row, 0.568 is not the best result. Is it just a typo? Or does +DKD not further improve the performance in this case?

Questions

In Table 6, the proposed optimization algorithm even improves the GNN-based recommendation models. What is the reason for this phenomenon?

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have addressed the limitations in Section 5 and Appendix.

Summary

This paper proposes Structure-aware Embedding Evolution (SEvo) to improve recommender systems by directly integrating graph structural information into embeddings. Unlike traditional methods, authors propose guide embedding update momentum with graph smoothing regularization. The proposed method can be integrated with a wide range of optimizers for neural networks, e.g., AdamW. The proposed method significantly increases performance metrics on several recommender datasets for standard models and outperforms GNNs.

Strengths

* Novel intriguing view on the problem of preserving structural information for recommender systems * Plug-n-play design of the method so that it can be easily transferred to any recommendation architecture * Solid performance gains * Faster than training GNNs

Weaknesses

* The paper is difficult to follow. It would be nice to see a simple summary of the SEvo pipeline in the form of a scheme or algorithm. * The SEvo formulae use the normalized adjacency matrix, so propagation of the gradients over the sampled node neighborhood is required. Suppose we have a large graph with a relatively high degree of each node. Mini-batch may have poorly correlated nodes, so many node embeddings should be updated simultaneously. This can lead to memory consumption issues and a notable increase in training time. * The tables report only the average time across different datasets. The detailed computational (or non-aggregated graphs with time) and space complexity is required to understand the ability of the method to scale * SEvo accelerates momentum only over first-order neighbors. However, for some graph-related tasks, it is critical to handle long-range dependencies.

Questions

* How does SEvo work in mini-batch fashion? Does it require specific batch preparations, or should it be smaller on average? * How does the model training time scale with the size of the graph? Could you provide a graph epoch time vs. graph size e.g. for a SASRec? * Can high-order proximity / long-range dependencies be incorporated using SEvo?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

* The time and space complexity analyses of the method are required to understand its scalability.

Summary

The paper introduces Structure-aware Embedding Evolution (SEvo), a novel embedding update mechanism for recommender systems. SEvo directly integrates graph structural information into embeddings, ensuring that related nodes evolve similarly with minimal computational overhead. This approach differs from traditional Graph Neural Networks (GNNs), which typically serve as intermediate modules. SEvo is designed to enhance existing optimizers, particularly AdamW, to improve recommendation performance by incorporating moment estimate corrections. Theoretical analysis confirms the convergence properties of SEvo, and experiments demonstrate consistent improvements across various models and datasets.

Strengths

1. The paper proposes a new method to enhance over smoothing during the backward pass. 2. The method can be naturally integrated with momentum-based optimizers

Weaknesses

1. Uncleared relationship related to recommender system 2. Key points need further explanation.

Questions

1. I do not see the relationship between your method and recommendation task. Your method aims to change gradient direction based on graph topology. It is more suitable to study it under more general graph datasets. Recommendation task has no relationship with your method. 2. In line 116, the author mentioned “These two criteria inherently conflict to some extent”. Why are structure-aware and direction-aware inherently in conflict? Do you have any explanation on this point? 3. We have already enhanced smoothness during the forward-pass during neighborhood aggregation. Why do authors think it is necessary to further enhance it during the backward pass? Will it lead to more severe over smoothing problem?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

See weakness and questions

Summary

This paper proposes SEvo, an embedding updating mechanism that directly injects the graph information into the optimization process. This paper points out two critical criteria for directly injecting graph structure information into the embedding updating process for recommendation. Based on the proposed two criteria, this paper makes efforts to derive a solution named SEvo for injecting the graph structure information directly. SEvo is model-agnostic and can be implemented in various optimizers. The experiments are detailed, and the algorithm is theoretically guaranteed.

Strengths

1. This paper is well-motivated and well-organized, making this paper easy to understand. This paper first proposes two criteria and derives the final form of SEvo. I appreciate the efforts of the authors to make this process so clear. 2. The experiments are detailed. Experiments on various datasets showcase the effectiveness of SEvo, with detailed ablation studies. 3. The proposed method is easy to implement. SEvo can be integrated into various optimizers without complex modification.

Weaknesses

The reason behind the success of SEvo on large-scale datasets remains unclear. I am extremely curious about this. The improvement is unbelievably huge, making me doubt the reported results. A level of 5% in practice industrial application is extremely huge. However, the experiment results show that SASRec equipped with SEvo performs twice as well as the vanilla SASRec. I believe that if the reported results are true, this performance even exceeds the SOTA method by a large margin since SASRec is still a strong baseline in practice. Has the author carefully tuned the base model?

Questions

I am curious about the differences in gradient descent between SEvo and methods that explicitly model the neighborhood relationship with graph structure (e.g., LightGCN). The modified embedding updating mechanism in SEvo also includes components of the adjacent matrix, which looks similar to the graph propagation mechanism in GNN-based methods. Could you please give an example showcasing the differences in gradient descent (e.g., comparing the gradient descent processes of LightGCN and MF-BPR+SEvo)?

Rating

6

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

See weaknesses and questions.

My concern has been addressed.

Thank the author for addressing my concern. I will keep my rating.

Acknowledgment of Clarifications and Updated Final Rating

Thank you for your detailed and thoughtful rebuttal. I appreciate the time and effort you have taken to address my concerns. Having carefully considered your responses, I am satisfied with the clarifications and additional insights provided. Your explanations have resolved the issues I initially raised, and I now have a clearer understanding of the contributions and significance of your work. I appreciate your efforts and will be revising my final rating to reflect the improvements made.

Keep score unchanged

I have acknowledged the rebuttal from authors and keep my score unchanged.