Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies

Response to Reviewer $\text{\textcolor{red}{pubB}}$

Thank you for taking the time to provide thoughtful feedback on our work. Please let us know if any of our responses below need further clarification. $ $ **W1. Novelty of our work** Performing in-sample off-policy evaluation (OPE) of a deterministic policy is challenging because in-sample OPE necessitates importance sampling (IS), while the IS ratios are zeros for a deterministic target policy. To address this issue, we utilized kernel relaxation and metric learning, which have not been used for OPE in MDP settings. Our novelty lies in the integration of kernel relaxation, metric learning, and in-sample OPE to address the issue. To integrate the components, we derived the MSE of the in-sample estimated Q update vector regarding kernel relaxation and metric learning in MDP settings. To the best of our knowledge, our method is the first in-sample OPE algorithm for evaluating deterministic target policies. We believe our work’s novelty is also in creating the theoretical guarantee of the proposed method. $ $ **Q1. How to extend our method to policy learning** Extending our method to policy improvement (PI) is our future work. Naive application of our work on PI will result in an actor-critic algorithm for offline RL where we evaluate the target policy with the critic learned by our method and improve policy by some PI methods such as the one used by TD3+BC [Fujimoto et al., 2021]. The reviewer referred to the papers that have not been published yet. However, in considering how to extend our work to PI regarding those works, work by Jin et al. [Jin et al., 2023] is an offline contextual bandit policy learning algorithm that works on discrete action spaces and tries to learn the policy that has the highest Q-estimation while avoiding learning policy that has high uncertainty (or variance) in the Q-estimation. The variance comes from a large IS ratio due to small behavior policy density values. Because our method also uses IS, we may adopt their method when we expand our work to offline RL. The work by Chen et al. [Chen et al., 2023] is an offline policy iteration algorithm that works on infinite horizon MDPs with continuous states and actions. Upon analysis of an OPE error, they decomposed the error into two parts using Lebesgue’s decomposition theorem: 1) absolutely continuous part w.r.t. data distribution where importance sampling can be applied, 2) singular part (e.g., Dirac measure, deterministic target policy). For the singular part, they used maximum mean discrepancy to make the upper bound of the OPE error. Since our method is developed for OPE of deterministic policies, our method may be integrated into the singular part and be used for PI. $ $ [Fujimoto et al., 2021] Fujimoto et al. “A minimalist approach to offline reinforcement learning” NeurIPS 2021. [Jin et al., 2023] Jin et al. “Policy learning “without” overlap: Pessimism and generalized empirical Bernstein’s inequality” Arxiv 2023. [Chen et al., 2023] Chen et al. “STEEL: Singularity-aware reinforcement learning”. ArXiv 2023. $ $ **Q2. The detailed procedure on metric learning** To clear up how metric learning is done in our method, we explicitly state in the last paragraph of section 4.2 that the pseudo-code of our algorithm in Algorithm 1 in Appendix B, and modified Algorithm 1 to contain more details on metric learning.

Response to Reviewer $\text{\textcolor{blue}{bgFP}}$

Thank you for taking the time to provide valuable feedback on our work. Please let us know if any of our responses below need further clarification. $ $ **Q1. The practicality of evaluating a deterministic policy and its example** Please note that our algorithm is not an offline RL algorithm that can choose the form of the target policy (stochastic/deterministic) but an algorithm for evaluating a given deterministic policy. We focused on evaluating deterministic target policies since there are many cases where the evaluation of deterministic policies is needed, e.g., safety-critical systems such as industrial robot control and drug prescription, where the sequences of actions need to be precisely and consistently controlled without introducing variability [Silver et al., 2014][Kallus and Zhou, 2018]. In these situations, one may want to evaluate the performance of the deterministic policies based on offline data when the robot control or drug prescription policy is updated before deployment. We included an explanation of the practicality of our problem setting and an example in the second paragraph of the introduction section. $ $ [Silver et al., 2014] Silver et al. “Deterministic Policy Gradient Algorithms” ICML 2014. [Kallus and Zhou, 2018] Kallus and Zhou. "Policy Evaluation and Optimization with Continuous Treatments.." AISTATS 2018. $ $ **Q2. How in-sample learning is conducted in the experiments?** Please note that in-sample learning does not mean on-policy learning. In-sample learning denotes the method that learns the value function by only using actions in the (off-policy) dataset [Xu et al., 2023][Kostrikov et al., 2022]. Even though the stochastic behavior policy and the deterministic target policy differ, it is in-sample learning if the actions used in the Q updates are in the dataset. Our method updates the Q-function with the update vector estimated only using the samples in the data following Eq.(6). Detailed procedure of our algorithm used in the experiments is presented in Algorithm 1 in Appendix B. Our algorithm works when the support of a deterministic target policy relaxed by a kernel is in the support of a stochastic behavior policy, and the experiment setting is made to satisfy the condition. To clear up how our algorithm does in-sample learning, we explicitly state in the main text that the pseudo-code of our algorithm is in Algorithm 1 in Appendix B. Furthermore, we modified Algorithm 1 to explain how the samples in the data are used to learn Q-function. $ $ [Xu et al., 2023] Xu et al., "Offline RL with no OOD actions: In-sample learning via implicit value regularization." ICLR 2023. [Kostrikov et al., 2022] Kostrikov et al., "Offline reinforcement learning with implicit q-learning." ICLR 2022.

Response to Reviewer $\text{\textcolor{green}{PFzG}}$

Thank you for taking the time to provide detailed feedback on our work. Please let us know if any of our responses below need further clarification. $ $ **W1. Practicality of evaluating a deterministic policy using data generated by a stochastic policy and its example** There are many cases where the evaluation of deterministic policies is needed, e.g., safety-critical systems such as industrial robot control and drug prescription, where the sequences of actions need to be precisely and consistently controlled without introducing variability [Silver et al., 2014][Kallus and Zhou, 2018]. In these situations, one may want to evaluate the performance of the deterministic policies based on offline data when the robot control or drug prescription policy is updated before deployment. For the offline data, if the data is collected from the policies used in the past, the data can be viewed as sampled from a single stochastic behavior policy. We included an explanation of the practicality of our problem setting and an example in the second paragraph of the introduction section. $ $ [Silver et al., 2014] Silver et al. “Deterministic Policy Gradient Algorithms” ICML 2014. [Kallus and Zhou, 2018] Kallus and Zhou. "Policy Evaluation and Optimization with Continuous Treatments.." AISTATS 2018. $ $ **W2. Parts of the algorithm that are learned and not learned** We learn the kernel metric composed of $h$ (scale of the metric, referred to as bandwidth) and $A$ (shape of the metric, referred to as metric) as well as $Q$ function, using Eq. (10), Eq. (13), and Eq. (6) respectively. The pseudo-code of our algorithm is provided in Algorithm 1 in Appendix B, where we use the learned bandwidth $h$. We modified the paper's main text to guide readers to our pseudocode in Algorithm 1 in Appendix B and clear up what is learned and given in our algorithm. Furthermore, we modified Algorithm 1 to contain more details. We also modified the caption of Table 1 to explicitly mention what is learned in our algorithm in the experiment. $ $ **W3. Hyperparameter selection** For the hyperparameters in our method, bandwidth $h$ in our method is learned by Eq. (10). As for the IS ratio clipping range, we selected the range by grid search on Hopper-v2 domain from the cartesian product of minimum IS ratio clip values {1e-5, 1e-3, 1e-1} and maximum IS ratio clip values {1, 2, 10} and applied same hyperparameters on the other environments. For the hyperparameters in the baselines, we mostly followed the settings in SR-DICE as they experimented on the same or similar environments. We added a brief explanation of the hyperparameter settings in the experiment section and details in Appendix C.1, “Hyperparameters” section. $ $ **S1. Intuitive explanation of the theorems and propositions** Thank you for your suggestion. We added intuitive explanations of the theorems and propositions in the main text. For Theorem1, we included an intuitive analysis of the leading-order bias in the paragraph after Theorem 1. For Proposition 3, we included an intuitive explanation of how the metric measures similarity between actions and how it is reflected in the importance resampling in the second paragraph after Proposition 3. $ $ **S2. Separation of section 4.2 into Optimal bandith/metric section and error bound analysis section** Thank you for your suggestion. The contents related to the error bound are put in a separate subsection “4.3 Error Bound Analysis”