Safe Collaborative Filtering

Response to Reviewer QW8m

**The authors sincerely appreciate your valuable comments on enhancing the quality of our paper. We hope that we have addressed all your concerns in our response. If there are any points we missed or if you have any further questions, we would be delighted to continue the discussion.** First of all, we appreciate your recognition of the novel aspects of our CVaR minimization approach and the mathematical rigor underpinning our methodology. The emphasis you placed on our comprehensive comparative analysis and the robustness and efficiency of our method is particularly encouraging. These aspects underscore the practical importance of our research for large-scale recommender systems. Next, we also appreciate your suggestion to provide more detailed explanations of specific concepts in our main paper. Owing to page limitations, we directed these explanations to the Appendix in our Supplementary Material. In the revised version, we will clarify these concepts further, as you have suggested. We believe this revision will enhance the paper's readability and appeal to a broader audience. We greatly appreciate your valuable input.

Response to Reviewer RZsQ

**The authors sincerely appreciate your valuable comments on enhancing the quality of our paper. We hope that we have addressed all your concerns in our response. If there are any points we missed or if you have any further questions, we would be delighted to continue the discussion.** ## Questions 1. Can you make comparison to the existing model: e.g. Wen, Hongyi, et al. "Distributionally-robust Recommendations for Improving Worst-case User Experience." Proceedings of the ACM Web Conference 2022. 2. In your paper, major contributions are done by forcing re-formulation of CVaR to be scalable. Can you elaborate, what is the problem of controlling the "tail-risk" with distribution-wise losses like divergences (KL), or Wasserstein metrics? ## Response 1 Wen et al. (2022) utilized the softmax cross-entropy loss and focused on minimizing the average loss by applying weighted averages for known subgroups. While their approach and ours are not mutually exclusive and could be seen as complementary, there are clear distinctions. Firstly, our methodology prioritizes reducing user losses at the extreme ends of the distribution (the tail) by solving a convex problem at each step, derived from a meticulously designed loss function combined with matrix factorization models. This strategy stabilizes the solution path and maintains the necessary convergence speed in practice. In contrast, Wen et al. (2022) aim to minimize average loss across groups, dynamically updating model parameters, group-wise loss estimates, and group weights via stochastic gradient descent/ascent. While their approach offers an interesting model architecture, it faces "a few challenges specific to recommendation" as noted in their Sections 3.2.2 and 4. Their method, similar to Mult-VAE in the use of neural networks and listwise loss functions, differs in the requirement to update loss estimates and weights for groups. As a result, their method is expected to demonstrate a slower convergence speed compared to Mult-VAE, and significantly slower than ours. Secondly, our method operates independently of any prior knowledge about subgroups. However, it is conceptually easy to integrate group-specific information if desired, making it compatible with the algorithmic structure proposed by Wen et al. (2022). It is important to note, though, that their method may introduce computational challenges as mentioned earlier, and combining our approach with theirs could be computationally demanding in practice. Third, our loss function is uniquely designed to address users at the extremities of the loss distribution, rather than centering on the average losses across subgroups. This approach offers planners a more refined and targeted control over the loss metrics they seek to minimize. ## Response 2 The growing body of research on distributionally robust estimation often employs the "Min-Max" principle, where the focus is on minimizing the worst-case performance across a specific class of distributions. Research in this area often restricts the class of distributions using metrics such as KL divergence or Wasserstein metrics, aiming to minimize average loss while maintaining robustness against deviations within these predefined classes. Our method diverges from this norm by directly minimizing the Conditional Value at Risk (CVaR), thereby enhancing user satisfaction at the tail end of the distribution. This approach resonates with a fundamental principle in machine learning, eloquently stated by Vapnik: "When solving a problem of interest, do not solve a more general problem as an intermediate step. Try to get the answer that you really need but not a more general one" (Vapnik, 1995, 2006). In this vein, our direct approach is more fitting compared to methodologies that indirectly constrain CVaR through divergences or Wasserstein metrics. Nevertheless, it's important to acknowledge that a similar objective might be attainable through careful selection of distribution classes, albeit requiring sophisticated knowledge and precise customization. When considering the relationship between CVaR and KL divergence, the potential role of Entropic Value at Risk (EVaR) as an intermediary is often brought to light. EVaR, deriving insights from KL divergence, is designed to tackle the computational challenges in estimating CVaR and also serves as an upper bound for it. Consequently, imposing constraints on EVaR implies a loose bounding of CVaR as well. However, our approach pivots away from this intermediary, focusing instead on directly computing CVaR. This direct methodology not only circumvents the need for EVaR but also proves scalable in large-scale applications such as recommendation systems.

comment on reply

Thanks for your replies and quote from Vapnik "When solving a problem of interest, do not solve a more general problem as an intermediate step. Try to get the answer that you really need but not a more general one" (Vapnik, 1995, 2006)." My questions were aimed at positioning contributions in right way. I am ok with answers. I do not have anything new to add, except to try to add some of this clarifications to a paper.

Response to Reviewer nPPr

**The authors sincerely appreciate your valuable comments on enhancing the quality of our paper. We hope that we have addressed all your concerns in our response. We would be delighted to continue the discussion if there are any points we missed or if you have any further questions.** > As far as I understand, the main motivation of this paper is tail performance (in Abstract). However, I found that the main contribution of this paper focuses on parallel computation and scalability (in the manuscript). The main focus of this work is tail performance optimization, with a particular emphasis on scalability. We believe that recommender systems are particularly critical applications of machine learning where scalability is paramount. Therefore, instead of applying existing CVaR optimization methods (e.g., CVaR-MF), we develop an approach to achieve scalability in large-scale recommender systems. > What if we just compute gradient only with bottom-$\alpha$% losses? We interpreted your comment as focusing on the bottom-$\alpha$% of users rather than individual user-item interactions. The suggested approach is essentially consistent with CVaR-MF, which focuses on the bottom-$\alpha$% users. Hence, **we believe this hypothesis has been thoroughly examined in our manuscript.** > What if we just put some high weight on the tail users? There are various weighting strategies for this approach. In this light, CVaR-MF can be viewed as a naive implementation that equally weights tail users while disregarding others. Our algorithm can also be viewed as one that assigns smooth weights to users,　represented by dual variables $\mathbf{z}$. This leads to an ALS-type algorithm with user reweighting (Section 3.3), which achieves excellent scalability. Therefore, **we believe our manuscript comprehensively examines both of the approaches you've suggested.** > Did you train two SAFER2 models with $\alpha=1.0$ and $\alpha=0.3$? If so, that is somewhat unfair for the baselines (e.g., VAE-CF). **We set $\alpha=0.3$ for both CVaR-MF and SAFER2 throughout all experiments.** Although we have already discussed this in Section 5.1 and Table 3 in the supplementary material, we acknowledge your suggestion and will improve the clarity of this point in the revised manuscript. > When $\alpha=1.0$, there are many recent methods outperforming Multi-VAE. e.g., LightGCN, SGCL. Our evaluation operates within the strong generalization setting. However, **the suggested GNN-based methods cannot be evaluated in this setting.** These models cannot make predictions for new users. In fact, we could find few references that compare such methods in a strong generalization setting; Mao et al. (2021) have attempted to compare their GCN-type method in this protocol with some modifications to their method (see Section 4.8 of Mao et al. (2021)). Finally, we would like to emphasize that we do not intend to claim that SAFER2 is a state-of-the-art method regarding the average ranking quality; in fact, we have reported the superior average performance of Mult-VAE on ML-20M in Table 1. We are confident our experiments sufficiently support our claim that SAFER2 improves tail performance while maintaining scalability and average ranking quality competitive with iALS, which is the fastest method with excellent ranking quality. > When $\alpha=1.0$, in my opinion, the pair-wise and list-wise loss functions outperform the point-wise loss. To our knowledge, **pairwise loss functions are not considered competitive in this setup, and they are not commonly evaluated as baselines**; for example, see Section 7 of Steck et al. (2020), Section 4.4 of Liang et al. (2018), and Table 3 of Rendle et al. (2022). Also noteworthy in this context is the conclusion of Rendle et al. (2022), *"It is striking that iALS achieves competitive or better performance than models learned with ranking losses (LambdaNet, WARP, softmax) which reflect the top-n recommendation task more closely."* Furthermore, **we have already compared listwise loss functions.** The multinomial likelihood of Mult-VAE is essentially a listwise loss (a.k.a. softmax cross-entropy); see Eq. (2) of Liang et al. (2018). > When $\alpha=0.3$, there are many recent methods enhancing the performance of tail users. Thank you for your feedback. We would appreciate it if you could provide references for this. > Equation numbers are needed for each equation when finding the location of the equation by ordering. We will ensure the equation numbers are displayed as you suggested, and we appreciate your suggestion. ## References Mao, Kelong, et al. "SimpleX: A simple and strong baseline for collaborative filtering." CIKM 2021. Steck, Harald, et al. "Admm slim: Sparse recommendations for many users." WSDM 2020. Liang, Dawen, et al. "Variational autoencoders for collaborative filtering." WWW 2018. Rendle, Steffen, et al. "Revisiting the performance of ials on item recommendation benchmarks." RecSys 2022.

Response

Thank you for your thorough response. \ I will raise my score by one step. \ However, I cannot recommend this paper for acceptance. You mentioned that "we have reported the superior average performance of Mult-VAE on ML-20M in Table 1. We are confident our experiments sufficiently support our claim that SAFER2 improves tail performance while maintaining scalability". 1. Comparable performance with Multi-VAE is not sufficient since there are too many outperforming recent methods. Multi-VAE was proposed in 2018. 2. Enhancing recommendation performance for tail users (e.g., cold-start, popularity bias) is a widely studied concept in recent papers. However, the authors do not compare the tail performance with recent methods in the experiment.

Response to Reviewer nPPr: Part 2 [3/4]

### Scalability w.r.t. the number of users and items We also examine the scalability of RecVAE and our methods. Due to RecVAE's alternating SGD algorithm mentioned above, it is not straightforward to compare RecVAE and our method in terms of the runtime per epoch. We hence discuss the relative training speed of each method on the two datasets (i.e., ML-20M and MSD), which are of different sizes; MSD has approximately 4 times the number of users and twice the number of items compared to ML-20M. As in the above experiment, the embedding size of SAFER2 is set to $d=256$ for ML-20M and $d=512$ for MSD. That of SAFER2++ is set to $d=2,048$ for both of the datasets. Table E shows the runtime per epoch of each method on ML-20M and MSD. Despite the SGD-based algorithm, which does not require the separability of the objective function, RecVAE is 9.6 times slower on MSD than on ML-20M. This is due to its $O(|\mathcal{U}||\mathcal{V}|)$ complexity for a single epoch, and therefore, RecVAE is not infeasible in real-world applications or requires some approximation techniques, which would lead to the quality degradation, e.g., sampled softmax (Bengio and Senécal, 2003). By contrast, SAFER2 is very efficient on both ML-20M and MSD, while it takes 16.7 times longer for MSD than ML-20M. The dominant factor in SAFER2's complexity is $\mathcal{O}(|\mathcal{U}|d^3)$, which stems from solving $d \times d$ linear systems for users; since the number of users is much larger than that of items on ML-20M and MSD, the cost for users becomes significant. More precisely, we solved each linear system by using the Cholesky decomposition with the approximated cost of $(1/3)d^3$. Therefore, since we set $d=256$ and $d=512$ for SAFER2, this result aligns with the theoretical complexity, $4 \cdot ((1/3) \cdot 2^3)=10.66... \approx 16.7$, which demonstrates the scalability of SAFER2 with respect to the numbers of users and items. It can also be observed in the result of SAFER2++: when the embedding size is fixed, SAFER2++ takes only 4 times longer for MSD than ML-20M. In summary, our proposed approach is highly scalable while maintaining average and tail performance, whereas RecVAE lacks scalability with respect to the number of users/items in terms of runtime per epoch, in addition to its slow convergence demonstrated in the previous experiment. Table E: Runtime/epoch on ML-20M and MSD. | Method | ML-20M | MSD | MSD / ML-20M | | -------- | -------- | -------- | -------- | | RecVAE | 40.9 sec | 395.0 sec | 9.6 | | SAFER2 | 3.4 sec | 57.0 sec | 16.7 | | SAFER2++ | 40.0 sec | 166.2 sec | 4.1 | #### Supplementary discussion SAFER2++ overcomes the cubic dependency on $d$ in SAFER2; comparing the relative runtime between SAFER2 and SAFER2++, SAFER2++ takes only $40.0 / 3.4 = 11.8 < (1/3)\cdot(2{,}048/256)^3=170.66...$ and $166.2 / 57.0=2.92 < (1/3)\cdot(2{,}048/512)^3=21.33...$ times longer for $d=2,048$ than $d=256,512$ on ML-20M and MSD, respectively. ## References Bengio, Yoshua, and Jean-Sébastien Senécal. "Quick training of probabilistic neural nets by importance sampling." AISTATS 2003.

Response

Thank you for your full response.\ It is really impressive and appreciated. My last major concerns were 1. Comparable performance with Multi-VAE is not sufficient since there are too many outperforming recent methods. Multi-VAE was proposed in 2018. 2. Enhancing recommendation performance for tail users (e.g., cold-start, popularity bias) is a widely studied concept in recent papers. However, the authors do not compare the tail performance with recent methods in the experiment. For those, 1. I did not mean the SOTA VAE-based methods. However, as you already noted, there are tremendous recent methods that outperform VAE-CF. For example, you mentioned GCN-based cannot be evaluated in your setting, however, there are many inductive GCN-based CF methods. In the same manner, I can name at least five recent methods outperforming VAE-CF and able to be evaluated in your setting. The point is, even though you mentioned iALS has been reported to exhibit state-of-the-art comparable performance in the 2022 RecSys reproducible track (Rendle et al., 2022), that is one paper and there are many other papers outperforming iALS and VAE-CF. 2. Enhancing recommendation performance for tail users (e.g., cold-start, popularity bias) is a widely studied concept in recent papers. However, the authors do not compare the tail performance with recent methods in the experiment. In the same manner, there are many recent methods for tail users. For these reasons, I cannot give a strong positive score on this paper. \ However, in light of your impressive response, I will raise my score to 6, which is on the positive side.