Offline reinforcement learning endeavors to leverage offline datasets to craft effective agent policy without online interaction, which imposes proper conservative constraints with the support of behavior policies to tackle the out-of-distribution problem. However, existing works often suffer from the constraint conflict issue when offline datasets are collected from multiple behavior policies, i.e., different behavior policies may exhibit inconsistent actions with distinct returns across the state space. To remedy this issue, recent advantage-weighted methods prioritize samples with high advantage values for agent training while inevitably ignoring the diversity of behavior policy. In this paper, we introduce a novel Advantage-Aware Policy Optimization (A2PO) method to explicitly construct advantage-aware policy constraints for offline learning under mixed-quality datasets. Specifically, A2PO employs a conditional variational auto-encoder to disentangle the action distributions of intertwined behavior policies by modeling the advantage values of all training data as conditional variables. Then the agent can follow such disentangled action distribution constraints to optimize the advantage-aware policy towards high advantage values. Extensive experiments conducted on both the single-quality and mixed-quality datasets of the D4RL benchmark demonstrate that A2PO yields results superior to the counterparts. Our code is available at https://github.com/Plankson/A2PO
Paper
Similar papers
Peer review
Summary
This paper proposes an offline reinforcement learning method called A2PO, which aims to solve the problem of constraint conflicts in mixed-quality datasets collected from multiple behavior policies. A2PO optimizes offline learning by explicitly constructing advantage-aware policy constraints, especially in the presence of data of variable quality. Specifically, A2PO employs a CVAE to disentangle the distribution of actions under different behavior policies, modeling the advantage values of all training data as conditional variables. The agent can then optimize the advantage-aware policies based on these disentangled action distribution constraints to achieve high advantage values. Through extensive experiments on single- and mixed-quality datasets from the D4RL benchmark, A2PO demonstrates its superior performance over other offline RL baselines and advantage-weighted competitors.
Strengths
1. The authors introduce CVAE to deal with the problem of disentangling behavior policies in mixed-quality datasets, which is novel to me. 2. The paper is well-structured and clearly written. 3. Several analytical experiments are also provided to help understand the method
Weaknesses
See questions.
Questions
1. Although this paper presents a new approach, the comparison with existing techniques may not be in depth enough and lacks an analysis of why the algorithm works. In addition, could the authors further discuss the conditions under which A2PO may fail? 2. This paper focuses on the analysis and comparison of advantage-weighted offline RL methods, ignoring recent work on policy constraints in the same relative as A2PO, such as PRDC: Policy Regularization with Dataset Constraint for Offline. It would be more convincing if the authors could add research and experimental comparisons of related work to demonstrate the advantages of A2PO among its relative methods (even SOTA methods). 3. Regarding implementation details, how can we ensure a sufficient number of high-quality samples in the dataset to support training when selecting ξ = 1? Additionally, how can we guarantee the accuracy of the introduced advantage information? In general, this work demonstrates commendable quality and I would be willing to raise my score if my concerns could be addressed.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
None
Experiment results for Q1 and Q3
Table R1. Test returns of our A2PO and return-conditioned and data rebalancing baselines. **Bold** indicates the best performance among the two algorithms. *Italic* indicates that the score are directly taken from the original paper. For the newly-constructed dataset, we rerun %BC, DT with the official code from [1]. As ReD[2] is a method without publicly available source code, we reimplement it ourselves. | Env | %BC | DT | CQL+ReD | TD3BC+ReD | IQL+DeReD stage 1 | IQL+DeReD stage 2 | IQL+ReD | A2PO | | -------------------------------- | ------------- | ------------- | ---------- | ---------- | ----------------- | ----------------- | ------------- | ----------------- | | halfcheetah-medium | *42.5* | *42.6* | 48.2 | ***48.5*** | *47.5* | *47.6* | *47.6* | 47.1$\pm$0.2 | | hopper-medium | *56.9* | *67.6* | *69.4* | *59.3* | *65.5* | *65.1* | *66.0* | **80.3**$\pm$4.0 | | walker2d-medium | *75.0* | *74.0* | *83.5* | *83.7* | *74.5* | *81.9* | *78.6* | **84.9**$\pm$0.2 | | Halfcheetah-medium-replay | *40.6* | *36.6* | ***46.3*** | *44.7* | *43.9* | *43.4* | *44.3* | 44.8$\pm$0.2 | | hopper-medium-replay| *75.9*| *82.7* | *98.6* | *77.4* | *92.8* | *100.1* | *101.0* | **101.6**$\pm$1.3 | | walker2d-medium-replay| *62.5*| *66.6* | ***86.7*** | *82.3* | *72.9* | *77.0* | *79.5* | 82.8$\pm$1.7 | | Halfcheetah-medium-expert| *92.9* | *86.8* | *81.6* | *93.2* | *87.9*| *91.8*| *92.6*| **95.6**$\pm$0.5 | | hopper-medium-expert| *110.9* | *107.6* | *95.0* | *106.2* | *89.3*| *104.7*| *106.1*| **113.4**$\pm$0.5 | | walker2d-medium-expert| *109.0* | *108.1* | *110.0* | *110.0* | *110.1*| *110.5*| *110.5*| **112.1**$\pm$0.2 | | halfcheetah-random-medium| 40.1$\pm$0.3 | 42.0$\pm$1.3 | -| -| -| -| 42.0$\pm$3.0 | **48.5**$\pm$0.3 | | hopper-random-medium| 21.0$\pm$5.9 | 3.1$\pm$0.0 | -| -| -| -| 6.7$\pm$2.9 | **62.1**$\pm$2.8 | | walker2d-random-medium| 32.0$\pm$8.1 | 66.9$\pm$8.4 | -| -| -| -| 60.0$\pm$2.6 | **82.3**$\pm$0.4 | | halfcheetah-random-expert| 7.7$\pm$2.7 | 10.3$\pm$7.4 | -| -| -| -| 42.4$\pm$26.1 | **90.3**$\pm$1.6 | | hopper-random-expert| 2.4$\pm$0.1 | 90.2$\pm$8.5 | -| -| -| -| 16.7$\pm$6.4 | **112.5**$\pm$1.3 | | walker2d-random-expert| 53.4$\pm$14.5 | 103.4$\pm$7.9 | -| -| -| -| 93.2$\pm$29.1 | **109.1**$\pm$1.4 | | halfcheetah-random-medium-expert | 29.1$\pm$5.2 | 42.6$\pm$0.6 | -| -| -| -| 39.1$\pm$21.6 | **90.6**$\pm$1.6 | | hopper-random-medium-expert | 62.0$\pm$18.3 | 46.1$\pm$1.9 | -| -| -| -| 31.3$\pm$6.6 | **107.8**$\pm$0.4 | | walker2d-random-medium-expert | 10.6$\pm$4.1 | 78.8$\pm$3.0 | -| -| -| -| 52.0$\pm$10.9 | **97.7**$\pm$6.7 | Table R2. Test returns of recent policy constraints Offline RL method PRDC and our A2PO. | Env| PRDC| A2PO| | -------------------------------- | ------------------- | ------------------ | | halfcheetah-medium| ***63.5**$\pm$0.9* | 47.1$\pm$0.2 | | hopper-medium| ***100.3**$\pm$0.2* | 80.3$\pm$4.0 | | walker2d-medium| ***85.2**$\pm$0.4* | **84.9**$\pm$0.2 | | Halfcheetah-medium-replay| ***55.0**$\pm$1.1* | 44.8$\pm$0.2 | | hopper-medium-replay| *100.1$\pm$1.6* | **101.6**$\pm$1.3 | | walker2d-medium-replay| ***92.0**$\pm$1.6* | 82.8$\pm$1.7 | | Halfcheetah-medium-expert| *94.5$\pm$0.5* | **95.6**$\pm$0.5 | | hopper-medium-expert| *109.2$\pm$4.0* | **113.4**$\pm$0.5 | | walker2d-medium-expert| *111.2$\pm$0.6* | **112.1**$\pm$0.2 | | halfcheetah-random-medium| **56.5**$\pm$2.6 | 48.5$\pm$0.3 | | hopper-random-medium| 5.5$\pm$0.4 | **62.1**$\pm$2.8 | | walker2d-random-medium| 5.5$\pm$0.8 | **82.3**$\pm$0.4 | | halfcheetah-random-expert| 1.3$\pm$0.5 | **90.3**$\pm$1.6 | | hopper-random-expert| 24.8$\pm$14.6 | **112.5**$\pm$1.3 | | walker2d-random-expert| 1.1$\pm$0.7 | **109.1**$\pm$1.4 | | halfcheetah-random-medium-expert | 10.5$\pm$2.8 | **90.6**$\pm$1.6 | | hopper-random-medium-expert| 88.5$\pm$15.2| **107.8**$\pm$0.4 | | walker2d-random-medium-expert| 4.87$\pm$3.2| **97.7**$\pm$6.7 | | maze2d-umaze| 127.4$\pm$23.4| **133.3**$\pm$9.6 | | maze2d-medium| 60.0$\pm$5.3| **114.9**$\pm$12.9 | | maze2d-large| 151.6$\pm$16.0| **156.4**$\pm$5.8 | | antmaze-umaze-diverse-v2| *90.0$\pm$6.8* | **93.3**$\pm$4.7 | | antmaze-medium-diverse-v2| *78.8$\pm$6.9* | **86.7**$\pm$9.4 | | antmaze-large-diverse-v2| *50.0$\pm$5.4* | **53.3**$\pm$4,7 |
Summary
The authors propose an advantage aware offline RL algorithm for datasets consisting of data from multiple behavior policies. The algorithm consists of two steps, (1) behavior policy disentangling: Wherein a CVAE is trained to output actions conditioned on normalized advantage and state. (2) Policy optimization, where the policy is optimized based on the actions generated by the CVAE. The authors demonstrate the performance of the algorithms on standard benchmarks
Strengths
1. The paper is well written and easy to follow 2. The experimental section is extensive
Weaknesses
1. **The Agent policy optimization is unclear** 1. Why is a behavior regularization term (2nd term in (8)) needed when you have a generative model trained on the dataset? 2. Why not condition the 2nd in eq 8 term on c*? 2. **Missing some return conditioned baselines** 1. Comparison with return conditioned approaches such as %BC and Decision Transformers are needed to understand the advantage of the proposed approach 2. Comparisons with data rebalancing techniques [1] can help understand methods with mixed datasets [1] Yue, Yang, et al. "Boosting offline reinforcement learning via data rebalancing." arXiv preprint arXiv:2210.09241 (2022).
Questions
See weaknesses
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors address limitations
Experiment results for Q2
Table R2. Test returns of our A2PO and other return conditioned and data rebalancing baselines. *Italic* indicates that the scores are directly taken from the original paper. For the newly constructed dataset, we rerun %BC and DT with the official code from [2]. As ReD[3] is a method without publicly available source code, we reimplement it ourselves. | Env | %BC | DT | CQL+ReD | TD3BC+ReD | IQL+DeReD stage 1 | IQL+DeReD stage 2 | IQL+ReD | A2PO | | -------------------------------- | ------------- | ------------- | ---------- | ---------- | ----------------- | ----------------- | ------------- | ----------------- | | halfcheetah-medium | *42.5* | *42.6* | 48.2 | ***48.5*** | *47.5* | *47.6* | *47.6* | 47.1$\pm$0.2 | | hopper-medium | *56.9* | *67.6* | *69.4* | *59.3* | *65.5* | *65.1* | *66.0* | **80.3**$\pm$4.0 | | walker2d-medium | *75.0* | *74.0* | *83.5* | *83.7* | *74.5* | *81.9* | *78.6* | **84.9**$\pm$0.2 | | Halfcheetah-medium-replay | *40.6* | *36.6* | ***46.3*** | *44.7* | *43.9* | *43.4* | *44.3* | 44.8$\pm$0.2 | | hopper-medium-replay | *75.9* | *82.7* | *98.6* | *77.4* | *92.8* | *100.1* | *101.0* | **101.6**$\pm$1.3 | | walker2d-medium-replay | *62.5* | *66.6* | ***86.7*** | *82.3* | *72.9* | *77.0* | *79.5* | 82.8$\pm$1.7 | | Halfcheetah-medium-expert | *92.9* | *86.8* | *81.6* | *93.2* | *87.9* | *91.8* | *92.6* | **95.6**$\pm$0.5 | | hopper-medium-expert | *110.9* | *107.6* | *95.0* | *106.2* | *89.3* | *104.7* | *106.1* | **113.4**$\pm$0.5 | | walker2d-medium-expert | *109.0* | *108.1* | *110.0* | *110.0* | *110.1* | *110.5* | *110.5* | **112.1**$\pm$0.2 | | halfcheetah-random-medium | 40.1$\pm$0.3 | 42.0$\pm$1.3 | - | - | - | - | 42.0$\pm$3.0 | **48.5**$\pm$0.3 | | hopper-random-medium | 21.0$\pm$5.9 | 3.1$\pm$0.0 | - | - | - | - | 6.7$\pm$2.9 | **62.1**$\pm$2.8 | | walker2d-random-medium | 32.0$\pm$8.1 | 66.9$\pm$8.4 | - | - | - | - | 60.0$\pm$2.6 | **82.3**$\pm$0.4 | | halfcheetah-random-expert | 7.7$\pm$2.7 | 10.3$\pm$7.4 | - | - | - | - | 42.4$\pm$26.1 | **90.3**$\pm$1.6 | | hopper-random-expert | 2.4$\pm$0.1 | 90.2$\pm$8.5 | - | - | - | - | 16.7$\pm$6.4 | **112.5**$\pm$1.3 | | walker2d-random-expert | 53.4$\pm$14.5 | 103.4$\pm$7.9 | - | - | - | - | 93.2$\pm$29.1 | **109.1**$\pm$1.4 | | halfcheetah-random-medium-expert | 29.1$\pm$5.2 | 42.6$\pm$0.6 | - | - | - | - | 39.1$\pm$21.6 | **90.6**$\pm$1.6 | | hopper-random-medium-expert | 62.0$\pm$18.3 | 46.1$\pm$1.9 | - | - | - | - | 31.3$\pm$6.6 | **107.8**$\pm$0.4 | | walker2d-random-medium-expert | 10.6$\pm$4.1 | 78.8$\pm$3.0 | - | - | - | - | 52.0$\pm$10.9 | **97.7**$\pm$6.7 |
I would like to thank the authors for their detailed response. I have updated my score accordingly. I wish the authors the best!
We sincerely appreciate your positive feedback, the time and effort you have put into reviewing our responses and clarification. It is immensely gratifying to learn that our clarifications have helped address your main concerns. Thank you once again for your constructive feedback and support.
Summary
The paper proposes using CVAE to train an agent and using a decoder, which conditions on state and advantage value, to generate actions. The Q function and V function are trained simultaneously, and the policy is trained in a TD3-BC style. Experiments show that the method can outperform baselines in most tasks.
Strengths
The paper is easy to follow, and the motivation and intuition for the algorithm design are reasonable. The experiments are comprehensive and justify the advantages of the method claimed by the authors. Conditioning on the advantage value, instead of using it as a coefficient, is interesting, allowing for sampling to make the advantage value extreme ($\xi=1$) to get the optimal action.
Weaknesses
I found some results in the experimental section problematic. More details are in the Questions section.
Questions
1. In line 169, what prior distribution of $p(z)$ did you choose? 2. When training the Q and V functions (Eq(7)), the authors claimed that training simultaneously can stabilize training. Do the authors have any intuition, theory, or literature to back up such a design? I ask because, in my memory, most methods that need to estimate the V function train Q and V separately, such as IQL[1] and SAC[2]. Simultaneous training may cause instability, similar to training GANs. 3. In Table 1, I found some baseline results inconsistent with the original paper. For example, hopper-medium-v2 is 90.5 for Diffusion-QL in the original paper, but 70.3 in this paper; antmaze-umaze-diverse is 66.2 for Diffusion-QL in the original paper, but 24 in this paper. Most of the results for Diffusion-QL are not consistent with the original paper, while in line 234, the authors claim their results are from the original paper. Can the authors explain the mismatch? 4. In line 463, the authors state they use the v2 version for antmaze tasks, while most other baselines, including IQL and Diffusion-QL, use the v0 version. In my experience, the v2 version dataset can always provide a higher score than the v0 version when using the same algorithm. It is unfair to compare A2PO's v2 results with the v0 results of other baselines. 5. I found that the training curve of hopper random-medium-expert in Figure 4 is unstable. Can the authors provide more training curves for other environments? [1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021). [2]Haarnoja, Tuomas, et al. "Soft actor-critic algorithms and applications." arXiv preprint arXiv:1812.05905 (2018).
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
My main concern is about the empirical experiments. It seems the algorithm doesn't have competitive performance compared to diffusion-based policy algorithms.
Experiment results for Q3 and Q4
Table R1. Test returns of A2PO and other baselines in gym tasks. **Bold** indicates the best performance among these algorithms. | | CQL | EQL | Diffusion-QL | CQL+AW | A2PO | | ------------------------- | ------- | ------- | ------------ | ------ | ----------------- | | halfcheetah-medium | *44.0* | *47.2* | *51.1* | *49* | 47.1$\pm$0.2 | | hopper-medium | *58.5* | *70.6* | ***90.5*** | *71* | 80.3$\pm$4.0 | | walker2d-medium | *72.5* | *83.2* | ***87.0*** | *83* | 84.9$\pm$0.2 | | Halfcheetah-medium-replay | *45.5* | *44.5* | ***47.8*** | *47* | 44.8$\pm$0.2 | | hopper-medium-replay | *95.0* | *98.1* | *101.3* | *99* | **101.6**$\pm$1.3 | | walker2d-medium-replay | *77.2* | *81.6* | ***95.5*** | *87* | 82.8$\pm$1.7 | | Halfcheetah-medium-expert | *91.6* | *94.6* | ***96.8*** | *84* | 95.6$\pm$0.5 | | hopper-medium-expert | *105.4* | *111.5* | *111.1* | *91* | **113.4**$\pm$0.5 | | walker2d-medium-expert | *108.8* | *110.2* | *110.1* | *109* | **112.1**$\pm$0.2 | Table R2. Test returns of A2PO and other baselines in antmaze v0 tasks. | Env | BC | BCQ | TD3+BC | CQL | EQL | Diffusion-QL | AWAC | IQL | CQL+AW | LAPO | A2PO | | ------------------------: | ------ | ------ | ------ | ---------- | ------------- | ------------ | ------ | ------ | ------ | ------------ | ---------------- | | antmaze-umaze-diverse-v0 | *45.6* | *55.0* | *71.4* | ***84.0*** | 50.8$\pm$11.6 | *66.2* | *49.3* | *62.2* | *54* | 0.0$\pm$0.0 | 72.6$\pm$10.2 | | antmaze-medium-diverse-v0 | *0.0* | *0.0* | *3.0* | *53.7* | 62.2$\pm$6.7 | *78.6* | *0.7* | 70.0 | *24* | 30.2$\pm$9.4 | **80.2**$\pm$4.0 | | antmaze-large-diverse-v0 | *0.0* | *2.2* | *0.0* | *14.9* | 38.0$\pm$5.5 | ***56.6*** | *1.0* | *47.5* | *40* | 22.3$\pm$4.8 | 52.1$\pm$7.9 |
Summary
This paper presents A2PO. A2PO is an offline RL method for learning with datasets that was collected by a diverse set of poliices. The aim of the method is to disentangle the data being collected by each policy by using advantage calculation and then using this information to better learn policies from the dataset. Specifically, they use a VAE and condition it on advantage and state and then train in latent states. They perform a thorough set of experiments on benchmarks to compare to baseline methods. They also perform an ablation study to find the important qualities in their method.
Strengths
Originality. Average strength. The originalty of this work is what I would expect. They have with a novel idea of disentangling the policies using advantage estimation which builds upon the previous idea of advantage estimation. I think this work is of solid novelty. Quality. Major strength. The emperical analysis in this work is done very well. They compare to many baselines and show their method is clearly superior in many of the domains. They also perform a very good ablation study that shows the quality of their choises. I think this is a high point of the paper. Clarity. Major strength. The paper is very well written, provides excellent figures and is easy to follow. Great job here. Significance. Average strength. The work here seems like it takes a good step forward in the domain of offline RL with mixed quality datasets. The significance here is solid due to their impressive results but no more than I would expect from a good paper.
Weaknesses
I don't have any major comments here. I believe this paper is complete, written well, has impressive results and is easy to follow.
Questions
None.
Rating
8
Confidence
3
Soundness
4
Presentation
4
Contribution
3
Limitations
The limitation they address is that the method is slower than others, but no slower than some baselines they provide. I believe this is addressed well enough and improving upon the computation speed is clearly out of scope.
Summary
The paper introduces A2PO, a novel approach to offline reinforcement learning that addresses the constraint conflict issue in mixed-quality datasets by using a Conditional Variational Auto-Encoder to disentangle action distributions and optimize policies towards high advantage values. A2PO demonstrates superior performance over existing methods on the D4RL benchmark, showcasing its effectiveness in leveraging diverse offline datasets for more robust policy learning.
Strengths
see questions part
Weaknesses
see questions part
Questions
This paper focuses on the constraint conflict issue in offline reinforcement learning when dealing with mixed-quality datasets collected from multiple behavior policies. By using a Conditional Variational Auto-Encoder (CVAE), A2PO disentangles the action distributions of different behavior policies present in the mixed-quality dataset. The CVAE models the advantage values as conditional variables, allowing it to generate action distributions that reflect the behavior policies' distinct characteristics. In general, this paper is interesting and well-written. One question is that since we can directly select the good trajectories according to the rewards, is there any ablation study to show that CVAE can do better than this simple method?
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
see questions part
Thank you for the authors' detailed response. My concerns have been addressed, and as a result, I have increased my score to 6. I do have a final question for discussion, which will not affect my score. Why is an additional actor $\pi$ needed to learn the latent variable $\tilde{z}^*$ (Eq 8)? What would happen if we simply input a random vector instead? Since $\xi$ is conditioned during the training of the CVAE, I wonder if directly inputting $\xi^*$ and a random $z$ would also produce a high Q-value action.
We sincerely appreciate your positive feedback, the time and effort you have put into reviewing our responses and clarification. We will address your doubt here. **[Q1: Why is an additional actor π needed to learn the latent variable z~∗ (Eq 8)? What would happen if we simply input a random vector instead?]** The random latent-input policy mentioned is the CVAE policy outlined in our manuscript. Specifically, the CVAE policy directly takes the state-advantage $c^*=s||\xi^*$ and random noise $z_0 \sim \mathcal{N}(0,I)$ as input to the CVAE decoder to get action output. Although the advantage-aware CVAE,can effectively generate actions that adhere to the corresponding in-sample distribution, the A2PO actor $\pi_\omega$ provides a more precise mechanism for action selection by directly outputting a discrete $\tilde{z}^*$ within the latent space. Meanwhile, our experimental findings, detailed in Appendix E, demonstrate a consistent performance advantage of the A2PO policy over the CVAE policy across the majority of scenarios. This performance disparity is particularly pronounced in the newly constructed mixed-quality datasets, such as the random-expert and random-medium-expert scenarios. These results indicate that A2PO successfully achieves well-disentangled behavior policies and an optimal agent policy, thereby exceeding the capabilities of CVAE-reconstructed behavior policies.
We sincerely appreciate your positive feedback, the time and effort you have put into reviewing our responses and clarification. It is immensely gratifying to learn that our clarifications have helped address your main concerns. Thank you once again for your constructive feedback and support.
We thank the reviewers for the feedback!. Since the discussion period is ending soon, we give a brief summarization here, including the comments from the reviewers and the future revision according to the reviewers's opinion. - Reviewer's comments - We are glad that all reviewer appreciates our work contribution, and we are encouraged by the positive comments on the novelties (Reviewer 3MzD & xTq4& e3Zq & W3XG), adequate experiment settings (All Reviewer), and good paper quality (Reviewer e3Zq & xTq4 & M64c & W3XG). Meanwhile, in the individual replies, we have already addressed other comments. - Future revision - We have addressed the reviewer's concerns in the point-to-point response. Below are the revisions we will make based on the reviewer's feedback. - Additional baselines comparison According to reviewers M64c and W3XG, we will add experiments comparing A2PO with the return-conditioned methods DT and %BC, data rebalance methods ReD, policy constraint methods PRDC methods. - Additional ablation study According to reviewer M64c, we will add an ablation study on A2PO regularization term in actor loss. - Modification of the presentation According to reviewer xTq4, we will update the description of the policy evaluation paragraph and the corresponding formula. We will correct the incorrect score reports related to baselines in the gym task and update the score reports in the antmaze v0 version environment.
Thank you for the reply! My concern is solved and I would like to improve my rating to 7.
Decision
Accept (poster)