Off-Policy Evaluation (OPE) in contextual bandits is crucial for assessing new policies using existing data without costly experimentation. However, current OPE methods, such as Inverse Probability Weighting (IPW) and Doubly Robust (DR) estimators, suffer from high variance, particularly in cases of low overlap between target and behavior policies or large action and context spaces. In this paper, we introduce a new OPE estimator for contextual bandits, the Marginal Ratio (MR) estimator, which focuses on the shift in the marginal distribution of outcomes $Y$ instead of the policies themselves. Through rigorous theoretical analysis, we demonstrate the benefits of the MR estimator compared to conventional methods like IPW and DR in terms of variance reduction. Additionally, we establish a connection between the MR estimator and the state-of-the-art Marginalized Inverse Propensity Score (MIPS) estimator, proving that MR achieves lower variance among a generalized family of MIPS estimators. We further illustrate the utility of the MR estimator in causal inference settings, where it exhibits enhanced performance in estimating Average Treatment Effects (ATE). Our experiments on synthetic and real-world datasets corroborate our theoretical findings and highlight the practical advantages of the MR estimator in OPE for contextual bandits.
Paper
Similar papers
Peer review
Summary
This paper addresses Off-Policy Evaluation (OPE) in contextual bandits, a significant problem in fields such as healthcare and personalized recommendation systems. OPE involves evaluating the performance of new policies using only existing data generated by a current policy, which can pose a challenge due to high variance in estimators, particularly in situations with large action or context spaces. The study proposes a new OPE estimator called the Marginal Ratio (MR) estimator, which mitigates this issue by considering the shift in the marginal distribution of the outcome resulting from the policy shift, rather than the policy shift itself. This approach makes the MR estimator more robust to increasing sizes of action and context spaces than existing methods like Inverse Probability Weighting (IPW) or Doubly Robust (DR).
Strengths
- The paper tackles the practically relevant problem of off-policy evaluation - The paper is well-written and it was easy to follow the main arguments. The intuition statements right after each theorem were also useful - The paper proposes a new estimator called the MR estimator, which defines the weight in terms of the shift in the conditional reward distribution between the logging and target policies and is expected to reduce the variance compared to IPW, DR, and MIPS. In an ideal case, MR can also be unbiased. - Advantages of the proposed method over a range of conventional methods such as IPW, DR, and MIPS are demonstrated in a comprehensive way
Weaknesses
- Eq.(3) may be unstable in cases where the target and behavior policies greatly differ, and there are many actions where the variance of $\rho(x,a)$ becomes very large. - It is intuitive that MR reduces the variance from IPS and MIPS, but it needs to estimate the weight in terms of the marginal reward distributions, which are always unknown. If we can estimate the reward distributions well, then we could simply rely on DM. Therefore, it would be necessary to formalize the advantages of MR over DM in the case of an unknown $w(y)$. - The experiment design could be improved. As I understand it, in the experiments, both $\rho(x,a)$, $p(e|x,a)$, and $w(y)$ are estimated from the logged data. While it's true that $w(y)$ is always unknown, in many industry applications where we can control $\pi_b$, $\rho(x,a)$ of IPW&DR and $p(e|x,a)$ of MIPS are known. Therefore, it would be necessary to compare MR with estimated $\rho(x,a)$ and IPS, DR, and MIPS with their respective true weights. - This is not critical, but there are a few concurrent work that aims to further improve MIPS so would be great to discuss these papers in a revised version: - Jie Peng, Hao Zou, Jiashuo Liu, Shaoming Li, Yibao Jiang, Jian Pei, Peng Cui. Offline Policy Evaluation in Large Action Spaces via Outcome-Oriented Action Grouping. WWW2023. https://dl.acm.org/doi/abs/10.1145/3543507.3583448 - Yuta Saito, Qingyang Ren, Thorsten Joachims. Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling. https://arxiv.org/abs/2305.08062
Questions
- Could you provide some additional experiment results comparing MR with an estimated weight $\hat{w}(y)$ and IPS, DR, and MIPS with the true respective weights? - How does MR perform when $\rho(x,a)$ takes some extreme values, making solving Eq.(3) hard? Specifically, I would love to see experiment results, for example with $n =5,000, n_a = 5000, \alpha = 1.0$, which should produce the situation I am interested in. Figure 8 in the appendix is relevant, but $n$ is too small to observe large empirical $\rho(x,a)$. - What would be the theoretical comparison between DM and MR? Why does MR perform much better than DM, even though both perform OPE by estimating some aspect of the reward distribution? - Can we extend MR to a doubly-robust version? How would it look, and how would its variance compare to that of DR with $\rho(x,a)$?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The limitations of the work are stated in the final section and they look reasonable.
Summary
This paper proposes a marginal OPE estimator that directly corrects the distribution shift wrt rewards instead of correcting the distribution shift wrt actions. The proposed estimator generalizes the idea of marginal IPS (MIPS) and achieves the minimum variance among them when the marginal importance weight is accurately estimated. Experiments on both synthetic and classification-to-bandit datasets demonstrate that the proposed estimator enables more accurate OPE than baseline estimators.
Strengths
1. The idea of applying importance sampling on rewards rather than actions is reasonable from the context. The paper is easy to follow, and the contribution is clearly stated. 2. The paper also provides a data-driven method to estimate the marginal importance weight, which is a simple and straightforward approach. 3. The connection between MIPS, which estimate the expected outcome by applying marginal importance weight wrt action embeddings is discussed, and the theory suggests that the proposed MR achieve minimum variance among marginal estimators.
Weaknesses
1. As author(s) discuss in the limitation, one potential concern lies in the accuracy of the marginal importance weight. In my understanding, the proposed method seems to work well when the reward is discrete, as an adequate amount of data is observed for each reward. However, when the reward is continuous, I guess sometimes only one importance weight is observed for each reward, which leads to the variance in the estimation of the marginal importance weight. This may lead to a similar difficulty as [Kallus&Zhou, 18]. I have several questions regarding this concern, which I list in the following “Questions” section. Nathan Kallus and Angela Zhou. “Policy Evaluation and Optimization with Continuous Treatments” (AISTATS’18) 2. The marginal importance weight on reward has not been discussed in the standard OPE in contextual bandits, but some similar ideas have been discussed in OPE of RL [Rowland et al., 20] and another OPE framework which estimates the cumulative distribution function of a policy [Xu et al., 22] [Wu et al., 23]. As discussions with these methods are important to clarify the novelty of this work, I suggest discussing connections between these papers in related work. Mark Rowland, Anna Harutyunyan, Hado van Hasselt, Diana Borsa, Tom Schaul, Rémi Munos, and Will Dabney. “Conditional Importance Sampling for Off-Policy Learning” (ICML’20) Yang Xu, Chengchun Shi, Shikai Luo, Lan Wang, and Rui Song. “Quantile Off-Policy Evaluation via Deep Conditional Generative Learning” (‘22) Runzhe Wu, Masatoshi Uehara, and Wen Sun. “Distributional Offline Policy Evaluation with Predictive Error Guarantee” (ICML’23)
Questions
The following questions are related to my concern discussed in Weakness 1. 1. In which condition the marginal importance weight is accurately estimated? 2. Could you provide an error analysis on the estimation of the marginal importance weight (especially when the reward is continuous)? 3. How the marginal importance weight is parametrized and estimated? Are there any assumptions or regularization? How difficult the hyperparameter tuning is?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
As discussed in Weakness 1, it would be valuable to discuss the potential difficulty in estimating the marginal importance weight when the reward is continuous.
Summary
This paper introduced a new off-policy evaluation (OPE) method called Marginal Ratio (MR) to contextual bandits problems. Conventional methods such as Inverse Probability Weighting (IPW) and Doubly Robust (DR) rely on estimating policy ratios, which could incur large variances when there is low overlap between the target policy and the behavior policy. Variance and bias analyses of the MR estimator are provided. Specifically, the MR estimator achieves smaller variance compared to both IPW and DR when the important weights $\rho$ and marginal ratio $\omega$ are known. When both of them are unknown, the authors proposed a method that can approximate the weights and showed that with high probability the MR estimator is more favorable than the IPW counterpart. Extensive numerical experiments are provided, where the MR estimator outperformed baselines in terms of the mean square losses while maintained small variances.
Strengths
The paper is overall well-written and easy to follow. The theorems, propositions and comparisons are well-organized. The authors provided a good motivation to study the MR estimator, which addressed the high variance issues when using IPW and DR in practice. The idea is novel and interesting and the simulation results showed great potentials of the proposed estimator.
Weaknesses
The assumptions of proposition 3.3 and 3.4 are quite strong. Though the MR estimator achieves lower variances comparing to IPW and DR estimators, I think the assumption of knowing the ratio $\omega$ is much stronger than knowing the importance weight $\rho$. Thus, it is not a very fair comparison. When the behavior policy and the marginal ratio $\omega$ are known, the authors showed that as the training data size $m$ increases, the biases of MR and IPW estimators are almost the same with high probability and the variance of MR estimator could be smaller than the IPW counterpart. I don't think this is a strong result as the MR estimator depends on the unknown importance ratio $\rho$ and the order term $\mathcal{O}(m^{-1/3})$ in the variance analysis could be very large in practice. Overall, I think the theoretical results the MR estimator are sound yet incremental compared to existing methods.
Questions
1. Can the order in the term $\mathcal{O}(m^{-1/3})$ be improved, e.g., $\mathcal{O}(m^{-1/2})$? 2. I am a bit confused by the connections to MIPS and the application to causal inference. As you mentioned, MIPS is trying to solve large action space problem while in causal inference the action space is usually binary. What is the benefits of using the MR estimator? Is it more applicable to large action space problems or small action space problems? Also, the connection and comparison to MIPS is not very straightforward to me as the final analyses do not explicitly depend on the action space size. Can you provide more explanations here?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
N.A.
Summary
The paper looks at the problem of off-policy evaluation (OPE) in contextual bandits. The problem consists of estimating the expected reward obtained from a policy different from the one that collected the data. The authors first analyze classic OPE estimators used for this problem, such as Inverse Probability Weighting (IPW) and Doubly Robust (DR). The authors notice how such estimators have high variance (especially IPW), and this is due to their focus on policy shift. The authors propose a novel estimator, called Marginal Ratio (MR) estimator, which focuses on the shift of the marginal distribution of the rewards instead. The authors also theoretically show that, whenever the marginal ratio is known exactly, MR has lower or equal variance compared to IPW. The authors also derive a relationship between the variances of MR and DR. In this case, however, they show that it is not true that the variance of MR is always lower or equal to the one of DR. Furthermore, the authors show that MR has lower variance than Marginalized IPS (MIPS, a recently proposed estimator), under the same assumption used by MIPS.
Strengths
- Simple and interesting idea with a novel point of view - Clear presentation - Careful experimental analysis (also considering the appendix) and strong empirical performance
Weaknesses
- Since the problem with IPW and DR is high variance, a really simple way to reduce the variance for such estimators is the Self-Normalization trick [1]. I would have preferred to see the empirical comparison also against the Self-Normalized variant of IPW and DR. - There is no investigation on whether the Self-Normalization trick is applicable to the proposed MR. This could further improve the performance of the proposed estimator (if applicable) - The experimental setting seems to be such that IPW has high variance, which is fine since it is a known characteristic of IPW. However, there are also settings where IPW performs well (see [2] for an in-depth review of the performance of state-of-the-art OPE estimators). It could have been useful to see a comparison between the baseline estimators and MR in such settings, even only in the appendix. [1] A. Swaminathan and T. Joachims, The self-normalized estimator for counterfactual learning, NeurIPS 2015. [2] Saito et al., Evaluating the Robustness of Off-Policy Evaluation, RecSys 2021.
Questions
Taken from the above section: - Do you think that the Self-Normalization trick is applicable to MR?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
The authors have addressed the limitations of their work.
Summary
This paper addresses the problem of off-policy evaluation (OPE) using a variant of the inverse propensity score (IPS) estimator on logged bandit data. The authors specifically aim to reduce the variance of IPS by focusing on the shift in the marginal distribution of rewards instead of the policies themselves, resulting in a new estimator called the Marginal Ratio (MR) estimator. The proposed estimator is extensively analyzed, including comparisons with other related works such as IPS, DR, and MIPS. Furthermore, the authors demonstrate the utility of the MR estimator in causal inference for estimating average treatment effects (ATE). The effectiveness of the MR estimator in OPE is evaluated through a standard experimental setup, involving both synthetic and semi-synthetic data. # POST-REBUTTAL: I have read the author’s rebuttal and the authors addressed all my concerns.
Strengths
The clarity and quality of writing in this paper are solid. The authors not only address the standard contextual bandit setting but also extend their work to causal inference for estimating average treatment effects (ATE). The overall idea of considering the shift in the marginal distribution of outcomes instead of the policies themselves is simple and very interesting. The theoretical concepts are effectively explained, and the proposed estimator is comprehensively compared to standard estimators such as IPS, DR, and MIPS. To the best of my knowledge, the derived guarantees are original contributions. The authors employ a standard experimental setup to validate their claims and showcase the favorable performance of their estimator through an extensive set of experiments.
Weaknesses
**Computational Efficiency:** The proposed MR estimator introduces an additional step of estimating $w(y)$, which may pose computational challenges compared to IPS, where the importance weights $\pi/\pi_0$ are readily accessible. While this might be manageable in the context of off-policy evaluation, where a policy $\pi$ is given and the goal is to estimate its value using $\hat{\theta}$ through IPS, DR, MIPS, or MR, it may become problematic in off-policy learning. In off-policy learning, we optimize $\hat{\theta} = \hat{\theta}(\pi)$ with respect to $\pi$ to find the policy that maximizes the reward. Then, performing a regression to estimate $w(y)$ in each optimization step with respect to $\pi$ can be computationally demanding. This presents a notable challenge. **Experiments:** The experimental setup presented in the paper is solid. However, it would be beneficial to incorporate more complex computer vision datasets such as Fashion-MNIST and CIFAR-100, as seen in recent studies (e.g., [1, 2]). These datasets feature larger action sets and contexts when converted to contextual bandit instances. **Related Work:** While not a weakness, it would be valuable to include citations for two recent papers [1, 2] that propose new corrections for the importance weights to reduce variance. In [1], the authors suggest clipping only the propensity scores $\pi_0$ (as $\rm{max}(\pi_0, 0)$) instead of the importance weight $\pi/\pi_0$. Additionally, [2] introduces a smooth regularization of the importance weights by incorporating $\pi/\pi_0^\alpha$, where $\alpha \in [0, 1]$. While it is not necessary to directly compare the proposed approach with these papers, acknowledging their existence and contributions would be appropriate. [1] Ben London and Ted Sandler. Bayesian counterfactual risk minimization. In International Conference on Machine Learning, pp. 4125–4133. PMLR, 2019. Preprint version arXiv:1806.11500. [2] Imad Aouali, Victor-Emmanuel Brunel, David Rohde, and Anna Korba. Exponential Smoothing for Off-Policy Learning. arXiv preprint arXiv:2305.15877, 2023.
Questions
- In terms of computational efficiency, how does the efficiency of the MR estimator compare to IPS? - Is there an efficient method to optimize the MR estimator with respect to the policy $\pi$ (i.e., in the context of off-policy learning) considering that estimating $w(y)$ is required in each iteration during the optimization process?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
3 good
Contribution
3 good
Limitations
The authors adequately addressed limitations in Section 6.
Thank you!
I appreciate your comprehensive response. My concerns have been addressed. I recommend incorporating a detailed discussion that explores the challenge of applying MR in off-policy learning as it is an important task related to off-policy evaluation, alongside a comparison with the missing related works. Additionally, the inclusion of the suggested experiments and the new CIFAR-100 trials would significantly enrich the paper. __Given your responses and assuming that the paper will be revised accordingly, I am pleased to raise my score to 7.__
Thank you for the feedback!
We are deeply grateful to the reviewer for providing positive feedback and for increasing their score. We also appreciate the useful suggestions which will greatly benefit our work and will make the suggested changes to our paper.
I appreciate the authors' response and would like to inquire about an additional matter. Upon reviewing the references, I observed that paper [26] is listed in the references but not explicitly mentioned in the paper itself. After examining its content, I found it to be both significant and relevant to the topic. Hence, I recommend engaging in a comprehensive discussion that highlights the distinctions between your work and [26]. Could you kindly explain the differences between your work and [26]? [26] Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient off-policy evaluation in Markov decision processes. J. Mach. Learn. Res., 21(1), June 2022. ISSN 1532-4435. Furthermore, I came across the following relevant survey that might be of interest: Uehara, Masatoshi, Chengchun Shi, and Nathan Kallus. "A review of off-policy evaluation in reinforcement learning." arXiv preprint arXiv:2212.06355 (2022).
We thank the reviewer for their question. > Could you kindly explain the differences between your work and [26]? [26] considers the problem of off-policy evaluation in RL setting. Specifically, the authors derive efficient off-policy estimators separately for Markov Decision Process (MDP) and non-Markov Decision Process (NMDP) settings. The OPE estimator proposed in MDP setting utilizes marginalised importance weights however, the main difference between our methodology and that of [26] is that MR uses ratios of marginal densities of rewards $Y$ whereas [26] utilises the ratios of marginal densities of state-action pair $(s_t, a_t)$ at time $t$. As we explain next, this choice of marginal densities of $(s_t, a_t)$ is specific to MDP settings and helps alleviate the problem of 'curse of horizon', a problem specific to reinforcement learning. In a contextual bandits setup (i.e. single decision setting), the ratio of marginal densities of $(s_t, a_t)$ in fact reduces to the policy ratios. In an MDP, the data comprises of trajectories $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \dots, s_t, a_t, r_t)$, where for the sake of simplicity we assume that $a_i \mid s_i \sim \pi^b(\cdot\mid s_i )$. Given a target policy $\pi^*$ (which we assume is stationary for the sake of simplicity again), the traditional IPW estimator considers the ratios of joint distributions of the whole trajectories under target and behaviour policies, which reduces to the product of policy ratios: $$\frac{p_{\pi^*}(\tau)}{p_{\pi^b}(\tau)} = \prod_{i=0}^t \frac{\pi^*(a_i \mid s_i) }{\pi^b(a_i \mid s_i) }$$ However, in an MDP, all relevant information regarding $r_t$ in the past trajectory is captured by the state-action pair $(s_t, a_t).$ Technically, this property can be expressed as $$r_t \perp (s_0, a_0, r_0, s_1, a_1, r_1, \dots, s_{t-1}, a_{t-1}, r_{t-1}) \mid (s_t, a_t)$$ Therefore, when estimating $E_{\pi^*}[r_t]$, instead of weighting the rewards by the joint distribution over the entire trajectory (as in traditional IPW estimator), the methodology in [26] only focuses on the shift in the marginal distribution of $(s_t, a_t)$, which leads to a more efficient off-policy estimator. The MR estimator would instead directly focus on the marginal shift in the rewards $r_t$. To summarise, we would like to reiterate that while [26] also considers marginalized density ratios, the importance ratios considered in [26] are different than the ones used in MR. The methodology considered in [26] is aimed more towards addressing the problem of OPE in RL settings where the trajectories might span a large horizon and the challenges faced are different than the ones in contextual bandits. We hope the above answers the reviewer's question. We will include a more detailed discussion regarding this comparison in our paper. > Furthermore, I came across the following relevant survey that might be of interest: Uehara, Masatoshi, Chengchun Shi, and Nathan Kallus. "A review of off-policy evaluation in reinforcement learning." arXiv preprint arXiv:2212.06355 (2022). We thank the reviewer for this reference and will incorporate this in the updated version of our paper.
I appreciate the authors for answering my question. I recommend including this discussion in either the main paper or the appendix.
I appreciate the authors' clarifications. Most of my main concerns were addressed nicely. I still have minor points as below, but I can increase my score to 5 to indicate that at least I am no longer on the negative side. > However, since Eq. (3) is a simple scalar to scalar regression it is easy to optimise in practice. It is hard to believe that every scalar to scalar regression is easy. I do consider the empirical analysis of the paper that demonstrates the effectiveness of MR with the estimated weights in a range of situations to be convincing, but since the estimation of $w(y)$ is such a crucial step of MR, it might be possible to further improve the paper by, for example, showing the accuracy of this regression problem, including the case where $\rho(a,x)$ has high variance, in the empirical analysis. > In contrast, DM involves estimating the expected conditional reward $\mu(a,x)$, which is obtained by regressing from $(a,x)$ to $y$. This is true if there are only action index, but in practice, there are often many useful action features and it is typical to use them when performing OPL or OPE. This should only increase the accuracy of DM or DR via improving the reward regression accuracy, and thus I am not still very optimistic that the superior empirical performance of MR can be generalized to this practically typical setup without further analysis. > Our setting of unknown importance policy ratios captures a wide variety of real-world applications, ranging form health-care to autonomous driving. This is totally true, but my argument was that there are also many situations especially in the industry that we know the true importance weights for IPS and DR. Thus, the comparison of MR with the estimated weights and IPS/DR/etc. with the true weights should also be the content in the main text, but it was nice to see the authors' additional efforts to perform empirical analysis about this situation.
Thank you!
Firstly, we are deeply thankful to the reviewer for increasing their score. >it might be possible to further improve the paper by, for example, showing the accuracy of this regression problem [...] We take the reviewer's suggestion onboard and will include additional investigation into the accuracy of the regression problem with increasing variance of policy ratios $\rho(a, x)$. >There are often many useful action features and it is typical to use them when performing OPL or OPE. This should only increase the accuracy of DM or DR via improving the reward regression accuracy [...] We agree that in settings where the action features are highly predictive of the outcome $Y$, the reward model $\mu(a, x)$ may be easy to learn and DM may perform well. However, we would like to note that in many settings, such as healthcare (where the outcome $Y$ is often noisy), this may not be the case leading to a high bias in DM due to model misspecification [Saito et al., 2020; Farajtabar et al., 2018]. We will include a more detailed discussion on this in our paper. Lastly, the suggestion to incorporate a comparison of MR with estimated weights and IPS, DR and MIPS with true weights in the main text is duly noted, and we will include the empirical results for this setting in the revised version of our paper. Once again, we appreciate reviewer's thoughtful feedback and are grateful for their reconsideration of our paper's score. Yuta Saito, Aihara Shunsuke, Matsutani Megumi, and Narita Yusuke. Open bandit dataset and pipeline: Towards realistic and reproducible off-policy evaluation. arXiv preprint 474 arXiv:2008.07146, 2020. Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning, volume 80, pages 1447–1456. PMLR, 2018.
Thank you for the detailed answers to the questions, and the clarifications on the experimental settings and results resolved some of my concerns. However, let me clarify my intention in the comment below. > **Q.** when the reward is continuous, I guess sometimes only one importance weight is observed for each reward, which leads to the variance in the estimation of the marginal importance weight. This may lead to a similar difficulty as [Kallus&Zhou, 18]. > **A.** In comparison, the setup considered in [Kallus & Zhou, 18] is different as they consider continuous valued actions (in their setting, the reward may or may not be continuous). Consequently, the challenges posed in their setting are orthogonal to our setting because their methodology involves estimating the density of the continuous valued actions under observational data, which may be a difficult task in general. Our methodology in contrast only involves a simple scalar to scalar regression which is much easier to estimate in general. I do understand that [Kallus & Zhou, 18] consider a different setting where the action space is continuous. What I meant here is that, [Kallus & Zhou, 18] needed to incorporate the nearby actions to deal with the variance issue of IPS caused by sparsely observed actions in the action range of $(-\infty, \infty)$ even when the action space is 1-dimensional. In your setting, a similar situation can happen, because it also considers importance sampling on the 1-dimensional reward space in the range of $(-\infty, \infty)$. For instance, how about the case where the reward function is given as E[r|x, a] = a and Var(r) = 0? In this case, the true reward propensity is equivalent to the action propensity, and thus similar issues to [Kallus & Zhou, 18] can happen with the proposed method.
Thank you for your question!
We thank the reviewer for their insightful questions and continued engagement. Below we address the reviewer's comments by elucidating the similarities and differences between our setup and that of [Kallus & Zhou, 18]. Both, [Kallus & Zhou, 18] and our work address the problem of off-policy evaluation using importance sampling methodologies. The importance ratios in [Kallus & Zhou, 18] are the policy ratios $\rho(a, x) = \pi^*(a\mid x)/\pi^b(a\mid x)$ whereas the importance ratios in our setting is the ratio of marginal distributions $w(y) = p_{\pi^*}(y)/p_{\pi^b}(y)$. [Kallus & Zhou, 18] consider a setting with continuous action space (e.g. Medicine doses) and a target policy $\pi^*(a\mid x)$ which is Dirac delta at a specific action value $t(x)$ (i.e. $\pi^*(a\mid x) = \delta_{t(x)}(a)$). Since the action space is continuous, we have that $p_{\pi^b}(A = t(X)) = 0$ and we can not use the traditional IPW estimator shown below (as it will almost surely be 0): $$ \hat\theta_{IPW}= 1/n\sum_{i=1}^n \frac{\mathbb{1}(a_i = t(x_i))}{\pi^b(a_i \mid x_i)} Y_i. $$ Instead, the authors propose relaxing the unit mass of the Dirac delta function using a kernel function $K(u)$. To summarise, the main challenge being addressed in this work is the estimation of $\rho(a, x)$ in the case when action space is continuous and target policy is deterministic (conditioned on $X$). The authors address this problem by replacing the dirac delta target policy $\pi^*$ with the smoothed out kernel density estimation. In contrast, for MR estimator, given a policy ratio $\rho(a, x)$, we can estimate the marginal ratios $w(y)$ using the regression shown in Eq 3 when the rewards are continuous. In other words, compared to IPW, the MR method only involves the additional step of estimating the marginal ratios $w(y)$, which can be estimated directly using a simple regression and does not involve estimating the marginal distributions $p_{\pi^*}(y)$ or $p_{\pi^b}(y)$ separately. In practice, when we use neural networks (for example) to perform this regression, this imposes an implicit restriction of smoothness on the computed ratios $\hat{w}(y)$. Therefore, if a specific value of reward $y'$ is unobserved in observational data, our trained regression model will infer the value of the ratio $\hat{w}(y')$ by interpolation. >how about the case where the reward function is given as E[r|x, a] = a and Var(r) = 0? We assume that the reviewer means that $Var_{\pi^*}(r) = 0$, (i.e. the target reward distribution is Dirac delta). Given the two conditions above, it can be shown that $Y = A = a'$ almost surely under the target policy $\pi^*$ for some fixed value $a'$, and therefore the target policy is also Dirac delta $\pi^*(a\mid x) = \delta_{a'}(a)$. Hence, in the example given above, the setting is the same as the one considered in [Kallus & Zhou, 18], i.e. the action space is continuous and the target policy is Dirac delta. This means that estimating the policy ratios $\rho(a, x)$ will involve the same challenge as considered in [Kallus & Zhou, 18]. However, we emphasise that this problem does not arise because of continuous reward space, but is more generally present whenever we are dealing with Dirac delta target policies and continuous action spaces, and will also be present if the reward space was discrete in this example. Moreover, this challenge is not specific to the MR estimator, and will also be present in any methodology which uses the policy ratios $\rho(a, x)$ (such as IPW, DR, MIPS, etc). In comparison, specifically for the MR estimator, if we were **given an estimate of the behaviour policy $\rho(a, x)$** (obtained using the methodology in [Kallus & Zhou, 18], for example) then we can simply estimate $\hat{w}(y)$ using a regression, and the rest of our methodology remains the same. In other words, MR does not impose any significant additional complexity compared to IPW as it only involves an additional scalar to scalar regression. We do not need to explicitly incorporate any smoothing methodologies when estimating $\hat{w}$, as the regression implicitly performs this smoothing in practice. We hope that the above addresses the reviewer's question adequately.
I appreciate the authors' additional clarification. Now I have resolved my questions and concerns, and given that the authors have promised to enhance the related work, I am happy to raise my score to 7. The idea of applying importance sampling directly to the reward would be worth sharing with the community.
Thank you for your detailed responses and clarifications. I believe that the additional experiments mentioned in your comment will greatly benefit the paper. I remain convinced that this paper is deserving of acceptance, and thus, I will maintain my score of 7.
Thanks for the response. The authors addressed my concerns and I raised the score.
Decision
Accept (poster)