Library

Zero-Shot Reinforcement Learning from Low Quality Data

Zero-shot reinforcement learning (RL) promises to provide agents that can perform any task in an environment after an offline, reward-free pre-training phase. Methods leveraging successor measures and successor features have shown strong performance in this setting, but require access to large heterogenous datasets for pre-training which cannot be expected for most real problems. Here, we explore how the performance of zero-shot RL methods degrades when trained on small homogeneous datasets, and propose fixes inspired by conservatism, a well-established feature of performant single-task offline RL algorithms. We evaluate our proposals across various datasets, domains and tasks, and show that conservative zero-shot RL algorithms outperform their non-conservative counterparts on low quality datasets, and perform no worse on high quality datasets. Somewhat surprisingly, our proposals also outperform baselines that get to see the task during training. Our code is available via https://enjeeneer.io/projects/zero-shot-rl/ .

cs.LGcs.AIReinforcement Learning in RoboticsDomain Adaptation and Few-Shot LearningAdversarial Robustness in Machine LearningComputer Science
Source

Paper

References (100)

01Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-FunctionsYevgen Chebotar, Q. Vuong, A. Irpan et al.2023 · Conference on Robot Learning · 169 citations In Library
02HIQL: Offline Goal-Conditioned RL with Latent States as ActionsSeohong Park, Dibya Ghosh, Benjamin Eysenbach et al.2023 · Neural Information Processing Systems · 149 citations In Library
03Weighted Policy Constraints for Offline Reinforcement LearningZhiyong Peng, Changlin Han, Yadong Liu et al.2023 · AAAI Conference on Artificial Intelligence · 18 citations
04For SALE: State-Action Representation Learning for Deep Reinforcement LearningScott Fujimoto, Wei-Di Chang, Edward James Smith et al.2023 · Neural Information Processing Systems · 122 citations In Library
05What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?Rui Yang, Yong Lin, Xiaoteng Ma et al.2023 · International Conference on Machine Learning · 38 citations In Library
06Optimal Goal-Reaching Reinforcement Learning via Quasimetric LearningTongzhou Wang, A. Torralba, Phillip Isola et al.2023 · International Conference on Machine Learning · 115 citations In Library
07Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-TuningMitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh et al.2023 · Neural Information Processing Systems · 269 citations In Library
08Extreme Q-Learning: MaxEnt RL without EntropyDivyansh Garg, Joey Hejna, M. Geist et al.2023 · International Conference on Learning Representations · 135 citations In Library
09Offline Q-Learning on Diverse Multi-Task Data Both Scales And GeneralizesAviral Kumar, Rishabh Agarwal, Xinyang Geng et al.2022 · International Conference on Learning Representations · 80 citations In Library
10How Crucial is Transformer in Decision Transformer?Max Siebenborn, B. Belousov, Junning Huang et al.2022 · arXiv.org · 18 citations In Library
11Does Zero-Shot Reinforcement Learning Exist?Ahmed Touati, J. Rapin, Y. Ollivier2022 · International Conference on Learning Representations · 89 citations In Library
12Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLTaku Yamagata, Ahmed Khalil, R. Santos-Rodríguez2022 · International Conference on Machine Learning · 131 citations In Library

Scroll for more · 38 remaining

Similar papers

On Zero-Shot Reinforcement LearningScott Jeen2024 · arXiv (Cornell University) · 0 citations In Library
Tackling the Zero-Shot Reinforcement Learning Loss DirectlyYann Ollivier2025 · arXiv.org · 0 citations In Library
Does Zero-Shot Reinforcement Learning Exist?Touati, Ahmed, Jérémy Rapin, Yann Ollivier2022 · arXiv (Cornell University) · 89 citations In Library
A Unified Framework for Zero-Shot Reinforcement LearningJacopo Di Ventura, Jan Felix Kleuker, Aske Plaat et al.2026 In Library
A Unified Framework for Zero-Shot Reinforcement LearningJacopo Di Ventura, Jan Felix Kleuker, A. Plaat et al.2025 · Proceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems · 0 citations In Library
Improving Zero-Shot Offline RL via Behavioral Task SamplingNazim Bendib, Nicolas Perrin-Gilbert, Olivier Sigaud2026 · arXiv.org · 0 citations In Library
Zero-Shot Reinforcement Learning Under Partial ObservabilityScott Jeen, Tom Bewley, Jonathan M. Cullen2025 · arXiv.org · 7 citations In Library
Provable Zero-Shot Generalization in Offline Reinforcement LearningZhiyong Wang, Chen Yang, John C. S. Lui et al.2025 · International Conference on Machine Learning · 3 citations In Library
A Survey of Zero-shot Generalisation in Deep Reinforcement LearningRobert Kirk, Amy Zhang, Edward Grefenstette et al.2023 · Journal of Artificial Intelligence Research In Library
Reducing Conservativeness Oriented Offline Reinforcement LearningHongchang Zhang, Jianzhun Shao, Yuhang Jiang et al.2021 · arXiv (Cornell University) · 6 citations In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer ZxeR3/10 · confidence 2/52024-07-11

Summary

The work investigates methods for zero-shot reinforcement learning (RL) that can be trained on small, homogeneous datasets. This research is driven by the need to make zero-shot RL practical when large, heterogeneous datasets are unavailable. The authors identify the limitations of existing methods that overestimate out-of-distribution (OOD) state-action values when trained on low-quality datasets. They propose incorporating conservatism into zero-shot RL algorithms to mitigate these issues. Their experimental results demonstrate that conservative zero-shot RL methods outperform their non-conservative counterparts on low-quality datasets while maintaining competitive performance on high-quality datasets.

Strengths

* Addressing Significant Issue: The paper attempts to address a significant gap in zero-shot RL by focusing on the challenges of using small, homogeneous datasets, which are more common in real-world applications. * Improved Performance: The proposed conservative zero-shot RL methods consistently outperform non-conservative counterparts on low-quality datasets and do not degrade performance on high-quality datasets.

Weaknesses

1. **The problem definition**: I do not think the settings of this work is so-called Zero-shot RL since the work introduces another dataset. And the code link is invalid. 2. **Complexity of Implementation**: The introduction of conservatism increases the complexity of the algorithms, which may pose implementation challenges for practitioners. 3. **Limited Real-World Validation**: While the experiments are thorough, there is limited validation of the proposed methods in real-world settings, which could affect their applicability. 4. **Comparative Analysis**: The paper could benefit from a more in-depth comparative analysis with other state-of-the-art methods to highlight specific strengths and weaknesses.

Questions

See in weakness.

Rating

3

Confidence

2

Soundness

2

Presentation

2

Contribution

2

Limitations

See in weakness.

Reviewer ZC956/10 · confidence 3/52024-07-12

Summary

This work addresses zero-shot reinforcement learning (RL), focusing on training agents to perform tasks without explicit rewards during pre-training. The authors investigate the performance degradation that occurs with small, homogeneous datasets and propose conservative zero-shot RL algorithms inspired by single-task offline RL approaches. Experimental results demonstrate that the proposed methods improve performance on sub-optimal datasets without compromising effectiveness on large, diverse datasets.

Strengths

* Zero-shot pretraining for RL is a crucial problem that can enhance generalization in downstream tasks. * The proposed method is well-motivated, and the writing is clear. * The experiments are thorough and effectively validate the proposed method.

Weaknesses

* One weakness of the paper is the lack of a clear definition for "low-quality data." As shown in Figure 8, it might be more accurately described as "coverage" rather than quality. Providing a more explicit definition is recommended. * Additionally, the comparison of baselines is incomplete, as it lacks an offline goal-conditioned baseline, which is highly relevant in the considered tasks. Including a representative offline GCRL baseline, such as GC-IQL, would strengthen the comparison. * Some related works are missing, particularly in the offline GCRL domain. References [1][2][3][4] are pertinent as they also focus on learning from offline datasets without rewards and deploying for multiple goals. The didactic example in this paper is very similar to Figure 1 in a prior paper [3], which also convey similar information during zero-shot deployment for multiple goals. A discussion with the prior results would be beneficial. Furthermore, a very relevant offline zero-shot RL method [5] is neither cited nor discussed. *References* [1] Yang R, Lu Y, Li W, et al. Rethinking goal-conditioned supervised learning and its connection to offline rl. ICLR, 2022. [2] Park S, Ghosh D, Eysenbach B, et al. Hiql: Offline goal-conditioned rl with latent states as actions. Advances in Neural Information Processing Systems, 2024. [3] Yang R, Yong L, Ma X, et al. What is essential for unseen goal generalization of offline goal-conditioned rl?[C]//International Conference on Machine Learning. PMLR, 2023. [4] Eysenbach B, Zhang T, Levine S, et al. Contrastive learning as goal-conditioned reinforcement learning[J]. Advances in Neural Information Processing Systems, 2022. [5] Frans K, Park S, Abbeel P, et al. Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings[J]. arXiv preprint arXiv:2402.17135, 2024.

Questions

* Could the authors provide a more explicit definition of "low-quality data"? * It is recommended to compare the proposed method with a representative offline GCRL baseline. * Including related works discussed above would make this study more comprehensive. * Can this method also be applied to approaches based on successor features?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

One limitation is that VC-FB and MC-FB do not outperform CQL on the D4RL benchmark; addressing this issue is left for future work.

Authorsrebuttal2024-08-13

Response by authors

Thanks for pointing us toward this work. We currently cite [2], and use their proxy for dataset quality (downstream performance on certain tasks) to rank our datasets. However, we were unaware of [1], and the metrics they propose are helpful, although not directly applicable to our setting as we’ll explain, but we’re keen to try to use these ideas. In [1] they propose metrics for measuring the *exploitation (TQ)* and *exploration (SACo)* of the behaviour policy that creates the dataset. *TQ* is estimated as the mean reward of trajectories in the dataset w.r.t. a single downstream task of interest. Since we’re interested in generalising to any downstream task, in principle we’d need to calculate TQ for an infinite set of reward functions as specified by our task sampling distribution $\mathcal{Z}$ which is clearly intractable. As an alternative, we could calculate TQ w.r.t our test reward functions only, but our datasets are made up of state-action-next-state transition samples, not of full trajectories, so we can’t do this either. Consequently, TQ doesn’t quite fit our problem setting. However, we can measure *SACo;* the ratio of unique state-action pairs contained in a dataset w.r.t. some reference dataset. We used their codebase to do so w.r.t. an idealised dataset of 100k unique state-action transitions. See the metrics below, note we had to vary the number of discretisation bins to to get meaningful results across environments with different sizes of state and action spaces. | | Walker (5 bins) | Point-mass Maze (25 bins) | Quadruped (3 bins) | Jaco (4 bins) | | --- | --- | --- | --- | --- | | RND-100k | 0.993 | 0.557 | 0.657 | 0.981 | | DIAYN-100k | 0.432 | 0.395 | 0.587 | 0.983 | | Random-100k | 0.477 | 0.036 | 0.526 | 0.801 | | | | | | | Though this is a step in the right direction it is still not a complete picture. For that, we would need a TQ-like metric that is applicable to our setting. This has raised an interesting discussion which we’ll add to the paper to highlight the limitations of these existing approaches. We will add this table to Appendix B (Datasets), cite this work, update text in the main body to mention this method and to discuss its limitations.

Reviewer 5D6P7/10 · confidence 2/52024-07-12

Summary

This paper identifies an overestimation problem of zero-shot RL algorithms, particularly in low data or low data quality settings. To address this, they propose to use a value conservative method to mitigate the overestimation (from CQL). They showcase that this effectively allows for reducing the overestimation, then empirically show performance improvements on low or bad quality data regimes, without impacting performance on usual regimes.

Strengths

I am not an expert of zero short RL, but the paper seems sound and is mostly clearly written. The problem seems relevant as zero short RL from low quality data is relevant for robotic applications, as underlined by the authors. I like the first figure, that provides intuition, similar to the ones in DDQN's paper. The method is clearly motivated and explained, the experimental evaluation is clear, the figure and caption allow assessing the defined precise scientific questions. I do not see much flaw in this paper presentation and method, but my qualification is not too high.

Weaknesses

**Provide a bit more background and intuition about zero-shot RL**. This is not really mandatory, but it would help a broader audience. **Introduce the RANDOM and RND datasets**. Maybe an introduction of these datasets can help better grasp Figure 2 and 3, and would help the previous point.

Questions

* What is the intuition behind the overestimation reduction of CQL ?

Rating

7

Confidence

2

Soundness

3

Presentation

4

Contribution

4

Limitations

There is a dedicated limitation section that correctly addresses the limitations of the method.

Reviewer qeSX7/10 · confidence 3/52024-07-12

Summary

This work proposes modifications to Forward-Backward representations method. The modifications are aimed to improve the method's robustness to dataset quality. The vanilla FB suffers when the dataset does not cover the state space well, and overestimates the quality of actions. The authors show this with a simple example on point-mass. Two modifications are proposed, VC-FB and MC-FB, which take inspiration from CQL and make sure the FB representations are not too optimistic on the transitions not seen in the dataset. VC-FB and MC-FB are then thoroughly tested on ExORL and D4RL. The authors show that VC-FB offers the best improvement on ExORL, although none of the proposed modifications beat CQL on D4RL benchmark. Nevertheless, in all settings the modifications greatly improve over vanilla FB.

Strengths

- The paper is clearly written. - The proposed modifications have clear mathematical motivation, similar to CQL - Both the modifications improve over the vanilla FB representation, and even outperform CQL on ExORL - The experiments are thorough and clear

Weaknesses

- The novelty is limited: the authors take CQL's method for correcting overly optimal predictions and apply them to FB.

Questions

- How do the proposed modifications affect the performance of the model when the dataset is of good quality? Does it hurt the performance?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have adequately described limitations and potential negative societal impact of their work.

Reviewer ZC952024-08-09

Thank you for the responses. While most issues have been resolved, I think the first question remains unaddressed. The authors claim that "quality" is composed of "diversity" and "data size," which I disagree with. Does this imply that data with low diversity but large size is considered high quality? Additionally, "coverage" or "diversity" has been studied in previous works such as [1][2], whereas "quality" is often referred to as the average return of the trajectories. I suggest the authors reconsider their definition or terminology to ensure rigorousness of a good work. [1] Schweighofer K, Radler A, Dinu M C, et al. A Dataset Perspective on Offline Reinforcement Learning[J]. arXiv preprint arXiv:2111.04714, 2021. [2] Yarats D, Brandfonbrener D, Liu H, et al. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning[J]. arXiv preprint arXiv:2201.13425, 2022.

Reviewer qeSX2024-08-09

Thank you for your response. My question regarding "good quality" dataset was rather about what would happen if the dataset you're pre-training on actually contains trajectories that solve the downstream task. In section 4.1 you say that you use offline TD3 performance on a given dataset as a proxy for dataset quality. My question is basically what will happen if you push the quality (as measured by TD3) even higher than you have so far. For example, you could take online TD3 replay buffer. Since there's not much time for you to collect the data and train on it, I'm just asking for you intuition about this.

Authorsrebuttal2024-08-11

Response from authors

Thanks for clarifying your question. We hint at the setting you describe when we train on the medium-expert dataset in our D4RL experiments (a dataset containing a mix of ~optimal and suboptimal trajectories for the downstream task). We report those results in Table 10 Appendix C. We found VC-FB and MC-FB were performant in this setting, unlike FB which fails catastrophically, but didn’t match the performance of CQL. Our intuition is that the success of our methods in this setting depends closely on the choice of $\tau$ which reflects how “conservative” our methods are w.r.t. OOD state-actions. If $\tau$ is too high, the methods are not conservative w.r.t. OOD state-actions and we get catastrophic failures akin to FB. If $\tau$ is too low the methods are too conservative about OOD state-actions and the policy struggles if it deviates from the ~optimal trajectories described by the dataset. Similar findings are discussed in the original CQL paper in Appendix F. Of course, this is far from perfect, and we’d like to look at more robust methods in future work.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC