Library

Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL

Offline reinforcement learning (RL) offers an appealing approach to real-world tasks by learning policies from pre-collected datasets without interacting with the environment. However, the performance of existing offline RL algorithms heavily depends on the scale and state-action space coverage of datasets. Real-world data collection is often expensive and uncontrollable, leading to small and narrowly covered datasets and posing significant challenges for practical deployments of offline RL. In this paper, we provide a new insight that leveraging the fundamental symmetry of system dynamics can substantially enhance offline RL performance under small datasets. Specifically, we propose a Time-reversal symmetry (T-symmetry) enforced Dynamics Model (TDM), which establishes consistency between a pair of forward and reverse latent dynamics. TDM provides both well-behaved representations for small datasets and a new reliability measure for OOD samples based on compliance with the T-symmetry. These can be readily used to construct a new offline RL algorithm (TSRL) with less conservative policy constraints and a reliable latent space data augmentation procedure. Based on extensive experiments, we find TSRL achieves great performance on small benchmark datasets with as few as 1% of the original samples, which significantly outperforms the recent offline RL algorithms in terms of data efficiency and generalizability.Code is available at: https://github.com/pcheng2/TSRL

cs.LGcs.AINeural Networks and Reservoir ComputingReinforcement Learning in RoboticsModel Reduction and Neural NetworksComputer Science
Source

Paper

References (65)

01Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value RegularizationXiangsen Wang, Haoran Xu, Yinan Zheng et al.2023 · Neural Information Processing Systems · 55 citations In Library
02Offline RL with No OOD Actions: In-Sample Learning via Implicit Value RegularizationHaoran Xu, Li Jiang, Jianxiong Li et al.2023 · International Conference on Learning Representations · 134 citations In Library
03A Policy-Guided Imitation Approach for Offline Reinforcement LearningHaoran Xu, Li Jiang, Jianxiong Li et al.2022 · Neural Information Processing Systems · 83 citations In Library
04When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement LearningHaoyi Niu, Shubham Sharma, Yiwen Qiu et al.2022 · Neural Information Processing Systems · 77 citations In Library
05Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based ImaginationJiafei Lyu, Xiu Li, Zongqing Lu2022 · Neural Information Processing Systems · 34 citations In Library
06When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement LearningJianxiong Li, Xianyuan Zhan, Haoran Xu et al.2022 · International Conference on Learning Representations · 39 citations In Library
07Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement LearningChenjia Bai, Lingxiao Wang, Zhuoran Yang et al.2022 · International Conference on Learning Representations · 183 citations In Library
08Learning State Representations via Retracing in Reinforcement LearningChangmin Yu, Dong Li, Jianye Hao et al.2021 · International Conference on Learning Representations · 9 citations In Library
09Koopman Q-learning: Offline Reinforcement Learning via Symmetries of DynamicsM. Weissenbacher, Samarth Sinha, Animesh Garg et al.2021 · International Conference on Machine Learning · 36 citations In Library
10Offline Reinforcement Learning with Soft Behavior RegularizationHaoran Xu, Xianyuan Zhan, Jianxiong Li et al.2021 · arXiv.org · 36 citations In Library
11Offline Reinforcement Learning with Implicit Q-LearningIlya Kostrikov, Ashvin Nair, S. Levine2021 · International Conference on Learning Representations · 1.6k citations In Library
12Representation Learning for Online and Offline RL in Low-rank MDPsMasatoshi Uehara, Xuezhou Zhang, Wen Sun2021 · International Conference on Learning Representations · 142 citations In Library

Scroll for more · 38 remaining

Similar papers

Efficient Online Reinforcement Learning with Offline DataPhilip Ball, Laura Smith, Ilya Kostrikov et al.2023 · arXiv (Cornell University) · 401 citations In Library
Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced DatasetsZhang-Wei Hong, Aviral Kumar, Sathwik Karnik et al.2023 · arXiv (Cornell University) · 32 citations In Library
Koopman Q-learning: Offline Reinforcement Learning via Symmetries of DynamicsM. Weissenbacher, Samarth Sinha, Animesh Garg et al.2021 · International Conference on Machine Learning · 35 citations In Library
Improving the Offline Dataset on Offline Reinforcement LearningHuihui Zhang, Guoyin Chen2026 · IEEE Transactions on Neural Networks and Learning Systems · 0 citations In Library
Beyond Static Datasets: A Unified Framework for Online-Offline Policy SynthesisRevista, Zen, IA, 102025 · Zenodo (CERN European Organization for Nuclear Research) In Library
Beyond Static Datasets: A Unified Framework for Online-Offline Policy SynthesisRevista, Zen, IA, 102025 · Zenodo (CERN European Organization for Nuclear Research) In Library
Sample Efficient Offline-to-Online Reinforcement LearningSiyuan Guo, Lixin Zou, Hechang Chen et al.2023 · IEEE Transactions on Knowledge and Data Engineering · 30 citations In Library
Boosting Offline Reinforcement Learning via Data RebalancingYue Yang, Bingyi Kang, Xiao Ma et al.2022 · arXiv (Cornell University) · 29 citations In Library
A Minimalist Approach to Offline Reinforcement LearningScott Fujimoto, Shixiang Gu2021 · Neural Information Processing Systems In Library
Towards Data-Driven Offline Simulations for Online Reinforcement LearningShengpu Tang, Felipe Vieira Frujeri, Dipendra Misra et al.2022 · arXiv (Cornell University) · 1 citation In Library

Peer review

Accept (poster)NeurIPS 2023 · via OpenReview
Reviewer xNUN5/10 · confidence 3/52023-07-04

Summary

This paper proposes a physics-informed dynamics model TDM and a new offline RL algorithm TSRL, which exploit the fundamental symmetries in the system dynamics for sample-efficient offline policy learning, embedding and enforcing T-symmetry between a pair of latent forward, and reversing ODE dynamics to learn fundamental dynamics patterns in data. Empirical results on D4RL benchmark datasets validate the good generalization ability of TSRL.

Strengths

- The idea is interesting and makes intuitive sense. - The paper is overall well-written. - Authors do comprehensive experiments on D4RL tasks to evaluate the generalization ability of the new method.

Weaknesses

- The performance of TSRL is not comparable with other baselines in most Adroit human and cloned tasks. - No comparison of other offline reinforcement learning methods using data augmentation. - No experiments to validate TSRL alleviates the problem of over-conservatism.

Questions

- Explain the reason that the performance of TSRL is not comparable with other baselines in most Adroit human and cloned tasks. - Add comparisons of other offline reinforcement learning methods using data augmentation. - Add related work of the data augmentation methods using offline RL in Sec. 6.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The limitations of the paper are properly addressed by the authors, but the societal impacts of the paper are not discussed.

Reviewer dZPC5/10 · confidence 4/52023-07-05

Summary

The current offline RL algorithm requires a large amount of offline dataset training and has poor performance on small datasets. This article proposes a framework to address this issue. By learning a T-symmetry enhanced dynamic model, capture more fundamental dynamic relationships. Afterward, the article applies T-symmetry to offline RL, uses a model to regularize constraints in the latent action space, and uses a model to filter data for data augmentation. The experimental results showed improvement in small datasets.

Strengths

1. The key idea is novelty. This paper explores symmetries to enhance the performance of offline RL with small datasets. The proposed method has a new technical insight. 2. The article is clearly structured. 3. The proposed method improves performance on offline RL with small datasets.

Weaknesses

1. This article chooses the backward model because of irreversible action, but this does not quite meet the definition of time-reversal symmetry. Other articles (https://arxiv.org/abs/2111.12600) respectively choose the inverse model to get the reversible action, and then process the irreversible action, rather than simply giving up. 2. The proposed method has a great improvement on small datasets, but there is still a lot of gap from the performance of complete data.

Questions

1. Compared with other methods, since t-symmetry can learn better representation and dynamics, why is there a great improvement in small datasets, but not in full datasets? 2. Why choose to perturb the latent space and then filter the data, rather than perturbing directly on the state space, and then filtering? Is this part of the performance improvement caused by the latent space, the filtered data, or both?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Reviewer 5ovg5/10 · confidence 4/52023-07-06

Summary

This work introduced a Time-reversal symmetry enforced dynamics model, which leverages the consistency between a pair of forward and reverse latent dynamics for improving the sample efficiency of offline RL algorithms. Conducted experiments demonstrate the effectiveness of the proposed method.

Strengths

- The proposed method makes sense in improving the sample efficiency of offline RL algorithms. - This work is clearly presented so that it is easy for readers to catch up with the main ideas.

Weaknesses

- I suggest authors to strengthen the analysis on the rationale, i.e., why can the proposed method improve the sample efficiency of off-line RL algorithms. This will make the paper more insightful. - This work lacks sufficient discussion on the relation/comparison with related works, especially for those also exploiting the consistency, e.g., PlayVirtual [1] and some others cited by [1]. [1] Yu, Tao, et al. "Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 5276-5289.

Questions

1. Why can the proposed method improve the sample efficiency of off-line RL algorithms? Pls give in-depth and convincing analysis. 2. What are the relations between this work and those metioned in the weakness part? 3.An open question: Are there any ideas to extend the core idea of this work to Online RL algorithms?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Pls see the weakness and questions parts.

Reviewer P7X77/10 · confidence 4/52023-07-31

Summary

The paper investigates the time-reversal symmetry of forward and reverse dynamics in reinforcement learning (RL). The authors propose a Time-reversal symmetry enforced Dynamics Model (TDM) that models the consistency between forward and reverse dynamics. Using the TDM, they further propose an offline RL algorithm that leverages the learned TDM in three ways: 1. Using the representation from TDM for value function learning; 2. Using TDM to penalize OOD samples; 3. Using TDM to moderate useful data augmentation. Extensive experiments demonstrate that the proposed method outperforms a number of baselines, especially, the proposed method can learn a better policy with significantly fewer samples.

Strengths

1. The proposed method is novel and backed by convincing experimental results. 2. The concept of time-reversal symmetry could potentially be widely applied in RL, as it represents a fundamental structure in many RL problems.

Weaknesses

1. The proposed method leverages a learned dynamic model for offline RL, whereas most of the baselines are model-free (except for MOPO), which seems somewhat unfair. In particular, the proposed method appears somewhat related to the Dreamer [A, B] method, which also leverages a dynamic model for data augmentation. I am curious about the authors' thoughts on comparing with Dreamer. 2. The claim of leveraging time-reversal symmetry is slightly concerning as the method essentially learns the forward and reverse dynamic models instead of leveraging the time-reversal symmetric property of the MDP (e.g., P(s, a, s')=P(s', a, s) in certain scenarios).  [A] Hafner, Danijar, et al. "Dream to Control: Learning Behaviors by Latent Imagination." International Conference on Learning Representations. 2019. [B] Hafner, Danijar, et al. "Mastering Atari with Discrete World Models." International Conference on Learning Representations. 2020.

Questions

1. I wonder how the authors ensure the fairness of their experiments, given that the proposed method appears more complex than the baselines. Do all the baselines have a similar amount of trainable parameters as the proposed method? 2. Instead of ensuring the T-symmetry using an extra loss term, is it possible to constrain the neural network architecture so that the model respects the T-symmetry by definition (like an equivariant neural network architecture)? 3. The proposed method doesn't seem to be specific to offline RL. Have the authors tried to use it in online RL?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors do address the limitations of their work. However, the discussion could be more comprehensive. For example, under what circumstances would the T-symmetry break? Are there scenarios where the proposed method might fail?

Reviewer P7X72023-08-11

The reviewer thanks the authors for their thoughtful rebuttal. Most of my concerns are addressed, but I would like to clarify on W2. I apologize for mistyping the equation in my review, what I meant was $P(s, a, s')=P(s', \mathbf{-a}, s)$, which is similar to the original T-symmetry $F(s, a)=\dot{s}=-G(s', a')$. I understand that the proposed approach is not limited to irreversible actions, but the original T-symmetry is. However, would the original T-symmetry be more efficient when the assumption of action being reversible is satisfied? Essentially, the original T-symmetry has the potential to automatically generalize to reversed transitions, but the proposed method cannot. This is by no means criticizing the proposed approach, instead, I acknowledge that the proposed method resolves the reversible assumption. My point is that, first, as mentioned above, though the original T-symmetry is more constrained, does it have advantages when the assumption is satisfied? Second, IMO the word `T-symmetry` implies the original T-symmetry where the transition is symmetric when the time is reversed. I am slightly concerned about the wrong implications it might have for the readers.

Authorsrebuttal2023-08-13

Response to the Reviewer P7X7 Comments (1/2)

We really appreciate the reviewer for the thoughtful comment. Regarding the new comments on T-symmetry: - The original T-symmetry for physical systems is actually defined on state measures $\mathbf{x}\in\Omega$ of dynamical systems (i.e., $d \Gamma(x)/dt=-F(\Gamma(x))$, $\Gamma$ is the time invertible transformation), rather than state-action pairs $(s,a)$ for typical control problems. In these systems, the evolution on $\mathbf{x}$ is determined by some underlying physical laws. While for control problem, we have an external control policy to influence state evolution. Hence there has to make some adaptation to the original T-symmetry in order to make it usable in control problems, especially for the MDP setting. - In our adapted *extended T-symmetry* for the MDP setting, we preserved most of the characteristics of the original T-symmetry definition, such as using ODE forward and reverse dynamics, and time-reversal on states (i.e., $F(s,a)=\dot{s}=-G(s',a)$). We model the reverse dynamics as $G(s',a)$ rather than $G(s',a')$ as it can overcome the irrevsible action issue while still roughly following the core idea of T-symmetry. - Moreover, in our abstract, introduction, preliminaries as well as method sections in our paper, we only refer to our treatment as the "extended T-symmetry" rather than "T-symmetry" to demonstrate that it is an adaptation on the original T-symmetry. - Finally, in the early development stage of our work, we actually tested using $G(s',a')$ as the latent reverse dynamics, but it leads to inferior performance as compared to our final form $G(s',a)$, probably due to the inability to capture the irreversible actions. We will reproduce these results and reply in a follow-up post in the next 2 days. We hope this can address your remaining concerns about our method.

Authorsrebuttal2023-08-13

Response to the Reviewer P7X7 Comments (2/2)

Dear reviewer, we have finished the experiments by only changing the latent ODE reverse dynamics $g(z_{s'},z_a)$ to $g(z_{s'},z_{a'})$ in TDM (i.e., using $G(s',a')$ rather than $G(s',a)$), all other configurations in the TDM model are kept the same. The final performance of TSRL on the 10k D4RL reduced datasets are provided in the following table: **Table 1: Performance of TSRL with different reverse dynamic models with 10k data, each result is generated by 3 random seeds** | **D4RL tasks**| **$g(z_{s'},z_{a'})$**| **$g(z_{s'},z_a)$**| | -------|:--------:|:--------:| |Hopper-m| 50.5 $\pm$ 11.6| **62.0 $\pm$ 3.7** |Hopper-mr|11.4 $\pm$ 7.3|**21.8 $\pm$ 8.2** |Hopper-me|45.8 $\pm$ 10.7|**50.9 $\pm$ 8.6** |Halfcheetah-m|36.3 $\pm$ 5.8|**38.4 $\pm$ 3.1** |Halfcheetah-mr| 26.2 $\pm$ 7.7|**28.1 $\pm$ 3.5** |Halfcheetah-me| 37.6 $\pm$ 4.8|**39.9 $\pm$ 21.1** |Walker2d-m|38.1 $\pm$ 7.5|**49.7 $\pm$ 10.6** |Walker2d-mr|20.8 $\pm$ 16.2|**26.0 $\pm$ 11.3** |Walker2d-me|25.2 $\pm$ 20.9|**46.7 $\pm$ 17.4** - As we can observe in the above results, using $G(s',a')$ will lead to inferior performances in all tasks. We think the performance degradation may be due to the following reasons: - As we have described in our previous discussion, using $G(s',a')$ cannot capture the irreversible actions. Our design of using $G(s',a)$ on the other hand provides a much more generic form for the MDP settings. - Second, if we use both $a$ and $a'$ to train a dynamics model with the forward and reverse dynamics $F(s,a)$ and $G(s',a')$, as $a'$ is from the behavior policy $\pi_{\beta}(s')$ in the dataset, then the learned dynamics model will implicitly capture the characteristics of the behavior policy. This is not desirable, as we want the dynamics model to only capture fundamental system transition dynamics $P(s'|s,a)$. Including information related to the behavior policy will introduce undesirable spurious correlations between $s'$ and $a'$. By contrast, if we model TDM only using $F(s,a)$ and $G(s',a)$, then it basically uses the same information as in the transition dynamics $P(s'|s,a)$, which can help learn a more fundamental dynamics model.

Reviewer P7X72023-08-15

Thank you for the discussion and the new experiment. My concerns are resolved and I would like to increase my evaluation to 7. The new experiments clearly shows that the extended T-symmetry outperforms the original T-symmetry, but I am curious that if the environment's action is fully reversible, would the extended T-symmetry still outperform? This is probably out of the scope of the paper, and I am just mentioning this for the purpose of discussion rather than proposing new experiments. Moreover, it just came to me that there is another paper that used the learned dynamic model to improve RL [A]. This is probably not super related because they were doing transfer learning, but I just feel that both works might share a high-level idea of using dynamic model to regularize RL. [A] Sun, Yanchao, et al. "Transfer RL across Observation Feature Spaces via Model-Based Regularization." International Conference on Learning Representations. 2021.

Authorsrebuttal2023-08-16

Thanks for raising the score

We want to express our sincere gratitude to the reviewer for increasing the score. In response to the comment, we will explore more task environments with different dynamical properties in the future to further examine the performance of our method. We appreciate the reference provided by the reviewer and will add it to our final paper.

Reviewer dZPC2023-08-18

Thank you very much for the responses which address and answer my questions. After reading the rebuttal and reviews of other reviewers, I decide to keep my score.

Reviewer 5ovg2023-08-21

Thanks for your responses.

I confirm that I read your responses, and suggest you to add the contents in your rebuttal to the revised paper.

Area Chair LfMZ2023-08-21

Reviewer, please submit your response to author's rebuttal

Please read the rebuttal of the authors and respond.

Area Chair LfMZ2023-08-21

Reviewer, please respond to the rebuttal

Reviewer, please respond to the rebuttal.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC