Estimating Propensity for Causality-based Recommendation without Exposure Data

Summary

This paper presents a review of a novel method proposed for estimating causal effects in situations where no observation of the treatment variable is available. The authors introduce an innovative approach that utilizes interactions to approximate missing exposure or propensity data. The method relies on several key assumptions, including the significance of "popularity" as a strong confounder of both treatment and outcome variables, as well as the proposition that the outcome variable is a product of the unobserved propensity and relevance (which is a function of historical popularity). Additionally, the paper assumes a monotonic relationship between popularity and exposure (Assumption 1) and utilizes a parametric assumption based on the Beta distribution. The paper does a great job reviewing existing literature. And the simulation section provides a thorough comparison with baselines and oracles that have access to the treatment variable. The ablation study of its own method is also very comprehensive which provided intuition on the most important factors of the proposed model.

Strengths

The proposed method is very novel by leveraging strong intuitions in the domain to find strong confounders that can reliably impute the exposure variable through a learnable objective. The paper presents a fresh new perspective into the causal inference application to Recommendation systems when privacy limits the access to treatment assignments in production applications.

Weaknesses

N/A

Questions

Question 1: PropCare is a solid method when all the assumptions are met: The method assumes `popularity` as a strong confounder of treatment and of Y. I wonder how general the assumptions are when the treatment changes or when applied to other datasets when the treatment does not have a single strong-enough confounder. Question 2: about line 291 (simulation section): Can authors share the distribution of the fitted propensity scores using all the compared methods? Wonder if POP's bad performance is due to a practical violation of the positivity assumption, plus not mitigating it through bounding the fitted value between e.g. [0.02, 0.98]. Since the PropCare method handles the positivity assumption violation in its regularization term, it should be a fair comparison to perform comparable mitigation for the other baselines.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

N/A

Summary

This paper proposes a propensity estimation model for causality-based recommendation without accessing the ground-truth propensity score or exposure data. Prior knowledge about item popularity is utilized to estimate the propensity score. A theoretical analysis is provided to understand the proposed model.

Strengths

- This paper investigates an interesting problem in causality-based recommendation, propensity score estimation, and proposes a model to get rid of the requirements for ground-truth exposure data. - A theoretical analysis is provided to understand the critical factors of the proposed model. - Ablation study is conducted.

Weaknesses

- I have serious concerns about the evaluation framework in this paper. All the recommendation experiments are based on one single DLCE model, and the adopted baselines for comparison are rather weak. I suggest the authors to include more advanced baselines, and more importantly, to compare with a wider range of backbone models. Just to list a few widely acknowledged causal recommendation approaches [1-3] which all do not require exposure data. - The technical contributions of the paper is limited. The proposed pairwise relationship between item popularity and propensity score is similar to the design in [1,2], which also leverage popularity to serve as a *soft* proxy for propensity score. - The adopted three datasets are very small, considering modern recommendation platforms. - In Table 3, the proposed PropCare model does not achieve the best performance with respect to Tau and F1 score, outperformed by both POP and CJBPR. - The case study is hard to follow and not convincing to me. [1] Bonner, Stephen, and Flavian Vasile. "Causal embeddings for recommendation." Proceedings of the 12th ACM conference on recommender systems. 2018. [2] Zheng, Yu, et al. "Disentangling user interest and conformity for recommendation with causal embedding." Proceedings of the Web Conference 2021. 2021. [3] Zhang, Yang, et al. "Causal intervention for leveraging popularity bias in recommendation." Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021.

Questions

1. The authors only experiment with one backbone model, namely DLCE. I suggest the authors to compare with more state-of-the-art causal recommendation approaches. 2. What are the main contributions of the proposed model? What are the differences between the proposed pairwise relationship and existing works such as [1] and [2]. 3. I suggest the authors to conduct experiments on large-scale datasets. 4. Please provide more explanations on the case study.

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

N/A

Summary

This paper proposes a framework for causality-based recommendation system. Different from traditional correlation-based recsys (e.g. collaborative filtering), causality-based recsys makes recommendations based on the causal "uplift". While there are several causal recsys models in the literature, they rely on exposure data and/or propensity scores to be given. In real-world scenarios, exposure data are often unavailable, difficult to obtain or noisy. Hence, in this paper, the authors proposed a propensity estimation framework, in the absence of exposure data which is a more practical setup, to ultimately allow causality-based recommendation. Experimental analysis are conducted with both quantitative results and a case study.

Strengths

S1. The problem studied is of significant research and practical value. In particular, the setup without assuming any exposure data is practical and can be easily deployed on most recsys platforms. Overall, the motivation and challenges are well articulated and convincing. S2. The key assumption in 4.2 is well argued and presented. The empirical validation of the assumption convincingly support the assumption. S3. Experiments are comprehensive with evaluations on causal performance as well as the quality of estimated propensity and exposure. Detailed analysis of results reveal deeper insights, such as the factors influencing casual recommendation. Moreover, the case study is interesting and provide intuitive evidence to the benefit of a causal recsys. S4. Overall the paper is well executed with a well motivated and effective solution.

Weaknesses

W1. Below the theoretical property in 4.5, the authors mentioned that the proposition guides certain design choices, such as regularization of the global distribution in Eq (7). While this is a theoretical implication of the proposition, is there any empirical evidence? The current ablation study does not seem to test the usefulness of the KL regularizer. W2. Data set differences： it is not clear to me the difference between DH_orig and DH_personalized. It was stated DH_personalized has a simulated "personalization factor". What is this factor? how does it work exactly? Minor comments: line 233: "should be avoid" -> avoided Table 1: the last 4 column names, are not clearly explained.

Questions

See Weaknesses.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Summary

The authors propose a propensity estimation/learning method based for unbiased recommendations. The method assumes no external data and only uses the user interaction data for learning. The main idea is well explained in Assumption 1 in the paper, which states that for two items with similar click/interaction probability, the more popular item is likely to be recommended to the user over the other. Authors incorporate this assumption in Eq. 6. The propensities learned via the proposed method perform better as compared to the baselines on several benchmark datasets.

Strengths

- The paper is very well-written and very easy to follow. The math is also intuitive to understand, overall a good job by the authors in writing. - The proposed method (Eq 6) follows intuitively from the main assumption (Assumption 1). - Experiments are performed on multiple benchmark datasets, and the proposed method outperforms the baselines. - Authors use the state-of-the-art causal recommendation method DLCE.

Weaknesses

- The choice of the KL-divergence-based regularization is not super clear. The GNN paper citation (11) does not use beta distribution as a regularizer, but rather as a weight in the GNN aggregation. Also since $Q$ is the empirical distribution of the estimated propensity scores, how is the regularizer used in the training (could be explained via the gradient equation)? - It is not clear how the ground-truth propensities in the ML dataset are used. In the appendix authors very briefly touch upon that (lines 58, 59), but it's not clear how that relates to the ground-truth propensities. MovieLens has explicit ratings, which users self-select, and the bias in the original dataset reflects the user's self-selection bias, not the recommendation bias. Saito et al [1] transform the ratings into binary feedback to model the bias used in the current paper. - The main metrics used for the evaluation (CP, CDCG) are not explained in the main section, but rather in the appendix. Since they are not very well-known metrics, it would be better if the authors include them in the main experimental section. Reference: - [1] Saito, Yuta, et al. "Unbiased recommender learning from missing-not-at-random implicit feedback." Proceedings of the 13th International Conference on Web Search and Data Mining. 2020.

Questions

- What is the effect of the regularization term in propensity training? An ablation experiment could help, additionally, an experiment with varying $\mu$ values could also give some insight into this. - If authors could elaborate more on the ML dataset setup (ground-truth propensity, binary feedback from ratings, etc)? (see point 2 in the weakness section)

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are correctly addressed.

Summary

This paper focuses on causality-based recommendation system, by proposing a PROPCARE method that estimates the propensity score by using its correlation with popularity. The motivation is well stated and related work is well discussed. Through experiments, the proposed method outperforms the baselines.

Strengths

1. The paper is well written and easy to follow. The motivation is clearly stated and the authors did a good survey around related work. 2. The proposed method is sound, with some theoretical analysis. 3. Experiments are conducted carefully with case studies to demonstrate the effectiveness of the proposed method.

Weaknesses

1. In order to model causality, the authors should make clear statement about whether the relationship between popularity and propensity is correlation or causality, which significantly affects the modeling foundation of this work. 2. Technical foundation is limited. From ML perspective, the technical contribution in this work is to combine point-wise and pair-wise modeling by combining the popularity with propensity estimation, which are all well-developed methods. 3. The baselines used in this method are not state-of-the-art, which lacks of confidence to evaluate the contribution of this work. Causality-based recommendation should share the same goal of general recommender systems, which should include the state-of-the-art methods to compare the recommendation accuracy. 4. The datasets used in this work is quite small, it's worthy to check the performance and scalability of the proposed method in large-scale datasets.

Questions

1. How should the author distinguish the relationship between popularity and propensity as correlation or causality? 2. What's the recommendation accuracy comparison between PROPCARE and state-of-the-art recommendation models? 3. How is the model's performance and scalability to large-scale datasets?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See above.

Response

Apologies for the late reply. Thanks for your clarification and response, this is helpful.