Uncertainty-Aware Instance Reweighting for Off-Policy Learning

Summary

This paper delves into the issue of off-policy learning, the objective of which is to devise a new action selection policy based solely on the logged feedback derived from a logging policy. The paper pays particular attention to scenarios in which the logging policy remains unidentified and its estimation proves challenging. Under these conditions, common estimators, like IPS, may lose beneficial attributes such as unbiasedness. To address this complication, the paper introduces a new off-policy learning (OPL) approach called Uncertainty-aware off-policy learning. This new framework aims to optimize the uncertainty-aware objective function by employing a novel weighting scheme that is tuned by minimizing an upper bound of MSE in estimation. A local convergence based on the proposed method is also shown. Experimental results indicate that this proposed framework outperforms a range of benchmark methods on both semi-synthetic and real-world recommendation datasets.

Strengths

- The paper addresses the practically relevant problem of dealing with uncertainty in logging policy estimation in off-policy learning. - The paper proposes a reasonable and conceptually straightforward method to handle the issue of uncertain logging policies, providing theoretical guarantees regarding estimation and local convergence. - The paper presents comprehensive experiments, not just basic performance comparisons, but also experiments on off-policy evaluation (OPE) and critical hyperparameters (some of which are included in the appendix).

Weaknesses

- Given that several papers already exist on the topic of distributionally robust off-policy learning (OPL), as discussed in the paper, the formulation of a problem addressing the uncertainty of logging policies may not be groundbreaking, even though I understand that their motivations differ somewhat. - In the experiments, the issue might also be tackled by simply applying calibration during the estimation of the logging policy, as seen in the following paper: Aniruddh Raghu, Omer Gottesman, Yao Liu, Matthieu Komorowski, Aldo Faisal, Finale Doshi-Velez, and Emma Brunskill. Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters. https://arxiv.org/pdf/1807.01066.pdf - In most of the experiments, CE performs quite well and is not substantially outperformed by UIPS. Therefore, considering the current experiment results, I may not use UIPS in practice and would rather rely on CE, which is much easier to implement (there is no need to estimate the logging policy when using CE), and does not require the tuning of additional hyperparameters as with UIPS. - Related to the previous point, in most experiments, the second-best methods for each metric and dataset perform very similarly to UIPS. I am not sure how essential it truly is to address uncertainty in logging policy estimation. I understand that the results are statistically significant, but results can be deemed significant even with a slight performance difference if the sample size is sufficient. In this context, my focus is on the performance difference.

Questions

- How do baseline methods perform in the experiments when they are combined with a calibrated logging policy estimator? Some additional results about this would be useful. - Could you provide the results relative to the performance of the (true) logging policy? This enables us to see how much improvements the methods bring compared to the logging policy. - When does UIPS become really crucial? That is, are there any situations where UIPS performs well while all other methods do not work satisfactory. In the current experiments, the second-best methods perform very similarly to UIPS in all datasets and metrics. Moreover, CE performs reasonably and stably for a range of metrics and datasets, which makes it a really good choice in practice indeed.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The paper touches on the tightness of the bound as a limitation and future work in the last section.

Summary

The paper considers a scenario in off-policy evaluation where we don't have access to the action probabilities of the logging policy, which we need to compute the propensities in IPS. Prior work would estimate these probabilities from data, but would ignore the uncertainties associated with these estimates. In this work the authors propose to re-weight the propensities based on thess uncertainties. The exact form of the weights is derived by minimizing an upper bound on the MSE. The resulting method consistently beats SOTA baselines on both toy and real datasets.

Strengths

A novel IPS variant that is grounded in and backed by theory (i.e. is derived by minimizing the MSE of the estimator), and is designed to solve a concrete problem in existing methods. Solid experimental methodology, sufficient details on the experimental setup to enable reproducibility. Strong results: the proposed method beats a broad list of SOTA baselines on both toy and real datasets.

Weaknesses

I have doubts re. the importance of the problem that the method is solving: we can simply log the propensities, right? How common is it in practice that these propensities are not available? At the same time, I see that in some experiments the method improves upon using GT propensities. This would be a good selling point of the method, but it goes against the initial motivation. How can a method that takes uncertainty into account be better than having _no_ uncertainty? This warrants more discussion in the paper, and makes me wonder if there are alternative explanations to why the method works. In general, I've found the manuscript to be rather dense and hard to read in places. I would've liked to see more intuition and/or pedagogical examples to understand what the method does exactly. E.g. on line 158: "UIPS assigns them with an increasing weight [..] as uncertainty increases" -- why does it make sense? The quality of the write-up could be improved: a few grammar mistakes, figure captions could be more informative, some opaque references to prior work could be expanded to make the paper more self-contained. The related work could be stronger: lines 301 - 320 largely repeat the introduction. What about other methods in on-policy/off-policy that exploit the uncertainties? UCB in bandits, for example?

Questions

How sensitive is the method to the uncertainty estimation approach? Does something simple like MC-Dropout produce similar results? Also see a few questions in the weaknesses section.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Little discussion of limitations: e.g. what about additional cost due to having to estimate uncertainties? Limited societal impact.

Summary

This paper proposes an Uncertainty-aware Inverse Propensity Score estimator (UIPS) for off-policy learning, taking into account the uncertainty in the estimated logging policy. The authors demonstrate that the commonly used method of estimating the logging policy can lead to biased estimators, particularly for samples with small estimated logging probabilities. UIPS addresses this issue by reweighting the propensity scores based on the uncertainty of the estimated logging policy. The paper provides a theoretical analysis of the convergence properties of UIPS and presents experimental results on synthetic and real-world recommendation datasets, comparing against state-of-the-art baselines.

Strengths

● The paper addresses an important problem in off-policy learning and proposes a novel method, UIPS, to improve the quality of the discovered policy. ● The authors provide a comprehensive theoretical analysis of UIPS, including a convergence guarantee. ● The experimental results demonstrate the effectiveness of UIPS compared to some baselines on both synthetic and real-world datasets.

Weaknesses

● There remain some issues unsolved in the paper, such as the availability of the logging policy. See the questions for details. ● There are some related works that are not mentioned in this paper. In off-policy RL, several papers work on behavior-agnostic instance reweighting [1,2]. They compute the prioritization weight without the need of obtaining a behavior policy. There are also papers that discuss the importance ratio term when applying RL to recommendation systems [3,4]. ● Introducing another neural network to estimate $\beta^*$ will increase the system complexity and the computational cost during training and testing. This may hinder the practical application of the algorithm. ● The synthetic dataset and the offline evaluation can give biased evaluation results of the algorithms. [1] Sinha, Samarth, et al. "Experience replay with likelihood-free importance weights." Learning for Dynamics and Control Conference. PMLR, 2022. [2] Liu, Xu-Hui, et al. "Regret minimization experience replay in off-policy reinforcement learning." Advances in Neural Information Processing Systems 34 (2021). [3] Cai, Qingpeng, et al. "Reinforcing User Retention in a Billion Scale Short Video Recommender System." arXiv preprint arXiv:2302.01724 (2023). [4] Chen, Minmin, et al. "Off-policy actor-critic for recommender systems." Proceedings of the 16th ACM Conference on Recommender Systems. 2022.

Questions

1.How is this paper related to behavior-agnostic methods [1,2,5]? With a GAN-like estimator, these methods no longer reconstruct all those behavior policies. They may also be regarded as baselines to compare with. 2.Why are the probabilities $\beta^*(a|x)$ not recorded in the data? With stochastic logging policies, it is easy to store probabilities together with state and actions when generating data. With deterministic logging policies, a common practice is to sample actions from a Gaussian distribution, with policy output as mean and a certain standard deviation. The action probability will also be available. 3.Why do UIPS-O and UIPS-P lead to poor performance? [5] Nachum, Ofir, et al. "Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections." Advances in neural information processing systems 32 (2019).

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The usage of off-policy correction and uncertainty-based reweighting is limited to policy-based techniques based on the REINFORCE trick. Such techniques can have higher variance than value-based techniques such as TD3 and SAC, and may lead to unstable training.

Summary

This paper proposes UIPS, a method that models the uncertainty of the estimated logging policy to improve off-policy learning. It assigns weights to each observation instance instead of simply dropping those with high uncertainty. The paper deduces the optimal form of weights from minimizing the upper bound of the resulting estimator’s MSE. Then it gets the improved policy by a two-step iterative optimization. This method is evaluated on both synthetic data and three real-world datasets in which UIPS outperforms multiple baselines.

Strengths

The paper addresses an important problem in policy optimization and proposes an innovative way to handle the uncertainty in the logged data. It derives a closed-form solution for the upper bound of MSE and proves the convergence of the method. It also conducts extensive evaluation on synthetic and real-world dataset to demonstrate the effectiveness of UIPS.

Weaknesses

1. For both synthetic and real-world data, although UIPS achieves the best results, it does not outperform the second-best method by a significant extent. It's not clear how this method will be useful in application. 2. The paper lacks a more detailed study and comparison with existing work on handling uncertainty. More explanation is needed on why UIPS is better compared to other methods. 3. Some deduction is not clear. For example, in-between steps are needed to show the "log trick" used in formula (2). The figure illustration is not very clear either. For example, it's confusing to use ``Log(item freq)" on the X-axis. In Tables 3 and 4, the metrics are not very clear.

Questions

1. How many iterations and how much time does the algorithm run in the evaluation? Does the result verify Theorem 3.4? 2. For Figure 1, the conclusion "items with lower frequencies in the logged dataset have lower estimated logging probabilities" is not a universal trend but only applies for frequency less than 7. How do you explain this observation, and does this affect your other results?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors are upfront about their limitations and listed future directions to address them.