State-Action Similarity-Based Representations for Off-Policy Evaluation

Summary

This paper introduces a new diffuse-metric for measuring behavioral similarity between state-action pairs for OPE, named ROPE. ROPE is used to learn state-action representations using available offline data. Theoretically, this metric can bound the OPE error. Empirically, ROPE boosts the data-efficiency of FQE and achieves lower OPE error than other OPE-based representation learning algorithms. It is claimed that this work is the first that successfully uses representation learning to improve the data-efficiency of OPE.

Strengths

(1) There is a certain degree of application innovation in this work. It is claimed that this work is the first that successfully uses representation learning to improve the data-efficiency of OPE. (2) The research on related works is relatively comprehensive. The techniques involved in the proof are interesting and completely non-trivial. The maths overall seem correct and fully rigorous. (3) The main body of the paper is well-written and easy to follow. (4) This work enhances the data-efficiency of OPE methods through representation learning, which is of great significance for OPE methods.

Weaknesses

(1) While the main body of the paper is well-written, there is space for improvement. I defer some of my issues in the appendix to "Questions". (2) Plots can be improved by: Improve colour-scheme by taking into consideration colorblindness. For instance, avoid red-green-blue combination (see e.g. https://davidmathlogic.com/colorblind/#%23D81B60-%231E88E5-%23FFC107-%23004D40 for more details). (3) There are a few misprints/suggestions in the text of the main body that I spotted: Line 142: “s1`, s2`\~P,a1`,a2`\~pi_e”should be ” s1`~P, s2`\~P, a1`~pi_e, a2`\~pi_e “ or “s1`,s2`\~P; a1`,a2`\~pi_e”. Line 167: “d_pi_e(s1,a1 , ;s2,a2)” should be “d_pi_e(s1,a1;s2,a2)”.

Questions

(1) What is the meaning of △ in line 56? There seems to be no note. (2) This work proposed a State-Action Behavioral Similarity Metric, of which the core is to learn the State-Action Representations. Can this State-Action Representations only be used for Off-Policy Evaluation？What is the effect of using this method to fit Q-function for general reinforcement learning algorithm. (3) In lemma2 and theorem 2, why can the existence of the difference upper-bounds demonstrate the learned representations can help FQE estimateρ(pi_e) ? (4) The answers of the three questions proposed in Empirical Study is not obvious. Which sections correspond to the answers? (5) In Figure 1(c), what is the meaning of 0.7 circled in red? Is it the reference state-action (s*, a*)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Limitations are explicitly discussed by the paper and the authors have partially addressed them. As far as I can see, has no potential negative societal impact.

Summary

This paper introduces an OPE-tailored state-action behavioral similarity metric that acts as a new loss for representation learning that can be used to learn a encoder for the state-action features in place of the original features.

Strengths

-- Very well written paper -- Interesting contribution to OPE

Weaknesses

-- Little discussion about why state-representation then OPE is an easier task than OPE itself

Questions

(1) It is clear that the distance metric is derived from triangle inequality on the difference in action-value functions. Can you theoretically justify why learning the representation and then plugging into FQE is an 'easier' task than just doing FQE? If it is not easier, why should we prefer it? (2) Sometimes FQE (which i assume is the "identity" in your plots) performs competitively with (or outperforms) ROPE. How can we anticipate this? I really want the OPE community to think about robustness of algorithms.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

The paper introduces a method to enhance the data-efficiency of the fitted q-evaluation (FQE) algorithm in off-policy evaluation (OPE) for reinforcement learning. They propose using a learned encoder and an OPE-tailored state-action behavioral similarity metric to transform the fixed dataset, improving the representation learning process. Theoretical bounds on OPE error are derived, and empirical results demonstrate the effectiveness of the proposed method in improving data-efficiency and reducing OPE error compared to other approaches.

Strengths

The paper demonstrates several strengths: 1. The paper addresses an important research direction in the field of off-policy evaluation (OPE) by focusing on enhancing data-efficiency through representation learning. This contributes to the advancement of OPE methods. 2. By learning representations based on a behavioral metric, the proposed approach avoids the direct use of importance sampling, which can introduce large variance in OPE. This innovative technique improves the stability and reliability of the OPE process. 3. The paper provides theoretical analysis, demonstrating the effectiveness of the proposed algorithm. This contributes to the understanding of the underlying principles and supports the validity of the approach. 4. The paper presents numerous experimental results, validating the effectiveness of the proposed method. These empirical findings provide strong evidence of the improvements achieved in terms of data-efficiency and OPE error reduction.

Weaknesses

1. My major concern is that the paper lacks a clear and intuitive explanation or discussion on why learning state-action representations improves data-efficiency in OPE. Providing a more intuitive explanation or discussing the underlying reasons for this benefit would enhance the clarity and understanding of the proposed approach. 2. It would be beneficial to include an illustration and an algorithm for the proposed paradigm in the main text. These visual aids would help readers grasp the key concepts and the implementation details more easily. 3. The paper overlooks a related work [1] that focuses on learning pseudometric-based behavioral representations for offline RL. Including a discussion of this work would enhance the completeness of the literature review and provide a more comprehensive understanding of the research landscape. [1] Learning pseudometric-based action representations for offline reinforcement learning. ICML 2022.

Questions

Considering the mentioned weaknesses, the paper raises the following questions: Can the authors provide a more intuitive explanation, discussion or visualizations regarding the benefits of learning state-action representations for data-efficiency in OPE? Addressing this question would enhance the clarity and understanding of the proposed approach. Furthermore, I would like to emphasize that addressing these concerns would significantly contribute to the improvement of my evaluation and, potentially, my overall score for the paper.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

limitations has been discussed by the authors.

Summary

Towards enhanced data-efficiency of the fitted q-evaluation (FQE) method, this work first proposes an OPE-tailored state-action behavioral similarity metric and then uses this metric and the fixed dataset to learn an encoder, which is used to transform the fixed dataset. Experiments on the OPE tasks illustrate that the proposed method improves the data-efficiency of FQE and obtains a lower OPE error compared to other OPE-based representation learning methods.

Strengths

1. The motivation of this work is clear, which is a problem worth studying under off-policy evaluation topic. 2. Various experiments as well as relevant analyses are performed in this work, which illustrates the efficacy of the method.

Weaknesses

1. Some details of this work are not clear enough and some analysis is too superficial. 2. Although the approach is superior to the other baseline, the advantage seems not obvious enough, except in the setting of $\mathcal{D}_{100}^{off}$.

Questions

There are some questions about this paper: 1. In Section 3.1, to learn a state-action representation, this work follows the method form [1]. I recommend that the authors give a supplementary explanation of this method. In addition, I wonder what is the difference between the proposed method, ROPE and the work [1]. In other words, why do the authors consider state-action representations? OPE-tailored behavioral similarity metric is unclear, and why is this metric strongly correlated with OPE? 2. In Section 4.2.2, the authors consider three other OPE-based representation learning methods, I recommend that the authors introduce these methods in a more intuitive way, such as figures. The experimental results of each method should be analyzed, for example, why the target-phi-sa performs poorly. The current version is not clear. 3. From Figure 3(b), the reward-only is significantly better than FQE and slightly worse than the rope. What are the details of the reward-only method? Does it involve learning state-action representations? This makes me more concerned about the need for learning state-action representations. 4. This work mainly considers the FQE approach, and I am more concerned about the generalizability of the approach. [1] Castro P S, Kastner T, Panangaden P, et al. MICo: Improved representations via sampling-based state similarity for Markov decision processes[J]. Advances in Neural Information Processing Systems, 2021, 34: 30113-30126.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

This paper discusses the limitations of the proposed algorithm and future work.