Survival Instinct in Offline Reinforcement Learning

Summary

The authors make an interesting and important observation about offline Reinforcement Learning. They note that an agent can learn from offline data even when the reward signal is not the same one as that used to train the online agent. Moreover it can be vastly different, or even with no reward signal at all, within certain types of task and with certain datasets. In particular, when a dataset has limited and biased data coverage, the aim of an offline agent to stay within the data distribution means that it has an equivalent of a survival instinct. This point is noted first empirically and then proven formally as well as being shown in an extensive set off environments and settings. The conditions necessary of the training data for an agent to learn a near-optimal policy are identified. Finally, a perspective on offline RL is given which means that the reward signal can be essentially ignored, so long as the appropriately biased dataset is provided.

Strengths

The originality and quality of the paper are both very strong. The combination of formal mathematical proofs of the insensitivity to the rewards and the experiments which show this to be true are married well together. The paper as a whole is very well written, and the appendix is in particular very thorough and shows a good understanding of the need for transparency and clarity in experimental settings. The amount of formality within the bulk of the paper is also appropriate, giving enough to understand the results, but not to swamp them. The nature of the findings are themselves original and are likely to have impact in the field of offline RL.

Weaknesses

I see few weaknesses overall, although there are a few typos throughout: In most of the figures, the number of seeds is provided, though in figure 1 it is not. This should be added. Line 120: "worst case a necessary" Line 143: "In following" Line 178: "s, a" may be clearer as (s,a). Line 214: "This is contrast" Line 227: "Positive data bias assumption" Footnote on page 5 has an additional bracket at the end. Line 232: "benefits"->"benefit" Line 245: "states have a safe action" Line 247: "is also mild" Footnote on page 6: "we include a ablation" Line 290: "with wrong can" Line 295 needs to be rewritten. Line 310: "with provably pessimism" Line 315: "A implicit" Line 318: "due the" Line 332: "the end of episode" Line 334: "there is no timeouts" Line 915 in the appendix: "the followings are true" Line 1025: "first we the" Line 1089: "and let the goal state to be"

Questions

I think that a discussion under which circumstances an adversarial reward signal could be designed to attack an offline agent may be interesting. While it is clear that with the appropriately biased data, a random reward signal has little effect, there may be circumstances where a strong enough adversarial reward signal and an inappropriately biased dataset may not be sufficient to lead to near optimal performance. Is it possible to make claims in the multi-agent offline setting similar to those in the single agent setting?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Limitations of the work are given, but none of the potential societal impact of this work has been discussed. Given that Offline RL may well play a large role in ML systems going forward, this should be mentioned.

Summary

This work displays a new observation that offline reinforcement learning (RL) algorithms can develop efficient policies, even when trained with incorrect reward labels. The authors attribute this resilience to the pessimistic nature of offline RL algorithms and the inherent biases in data collection processes. The findings introduce a ‘survival instinct’ in the systems that inform how we understand offline RL benchmarks and create future ones. The study, therefore, recommends a new strategy for offline RL that promotes learning of desired behavior through flawed rewards but intentional bias in data coverage.

Strengths

1. A good warm-up experiment to help readers understand the motivation. 2. Sound theoretical analysis to show the property of ‘survival instinct’ in current offline RL algorithms. 3. Comprehensive experiments on current D4RL and meta-world benchmarks with different offline RL algorithms.

Weaknesses

1. There exists a gap between the intuition (longer trajectories in the data have a smaller optimality gap) and the definition of the Positive Data Bias. On the other hand, Positive Data Bias does not have a quantitative value for each dataset so that the reader can clearly see the difference between the different datasets. 2. Authors claim that offline RL is intrinsically safe. However, missing relevant experiments to validate their claims.

Questions

3. Regarding the intuition about the positive data bias, I am pretty interested in the experiments in the Antmaze dataset from D4RL, where it is a multi-goal task and there does not exist positive data bias claimed by the authors. 4. A detailed analysis of the influence of the three types of “wrong” reward labels can make this work more complete and robust. 5. More experiments about safe RL are highly appreciated [1]. 6. In line 247 about the safe RL. "This condition can be easily satisfied by filtering out unsafe states in post processing." From the previous work [2], one biggest reasons why we prefer offline RL over BC is that offline RL can learn that those states are dangerous. What does the author think about this? [1] Liu Z, Guo Z, Lin H, et al. Datasets and Benchmarks for Offline Safe Reinforcement Learning[J]. arXiv preprint arXiv:2306.09303, 2023. [2] Kumar A, Hong J, Singh A, et al. When should we prefer offline reinforcement learning over behavioral cloning?[J]. arXiv preprint arXiv:2204.05618, 2022.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

see the above

Summary

This paper reports a very interesting new phenomenon that would be interesting to the offline RL community. It demonstrates that even when trajectories have the wrong reward labels, offline RL can learn good policies. The paper's experiments attempt to dissect the reasons for why this surprising phenomenon emerges, and argues that special to offline RL algorithms, pessimism endows the agent with survival instinct to stay within the data support, so that safe policies can be learned within these constraints.

Strengths

The phenomenon that has been identified is surprising; the experiments done to dissect this phenomenon are clear and solid. Overall a solid paper.

Weaknesses

I have only one concern, and welcome further clarification by the authors. The authors claim that their algorithm demonstrates this robustness phenomenon arises due to the survival instinct of offline RL algorithm. But it seems from the paper that the circumstance of this only arises in the situation that the data collected for offline RL has long timescale trajectories. Is this the case? For instance, it does not arise in other circumstance investigated, like when there are multiple sources of data, or other circumstances that deviate from this central feature. If this is the case, then the authors should state this clearly. This is still very much an interesting phenomenon, but is slightly more limited than the first impression one gets when reading the abstract and introduction that this is a potentially quite a universal phenomenon in offline RL. If long trajectories is not the only circumstance in which this phenomenon, the authors can make the paper even more clearer by listing all the circumstnaces in which this phenomenon arise, which would be very helpful for the reader. Regardless, as long as the authors match their conclusions to their demonstrations, this paper demonstrates indeed a very interesting phenomenon arising in offline RL that is worthy of publication.

Questions

See questions present in the above sections. And one more question, about the suggestion at the final part of the abstract (and conclusion): - if the implicit bias for long trajectories in offline data is the sole feature that is responsible for this wrong labels phenomenon, then is it still reasonable to suggest that "whereby an agent is “nudged” to learn a desirable behavior with 20 imperfect reward but purposely biased data coverage"? In other words, does the "survival instinct" as discussed in the paper really have the sensitivity to make such a (next step) hypothesis reasonable?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have a limitations section, and are fair. They also nicely clarify that of course, offline RL does not always learn from wrong rewards.

Summary

In this paper, the authors show that offline RL algorithms have an implicit survival instinct that often allows them to learn good policies with incorrect rewards. The authors argue that this is due to the data being positively biased and the pessimism that constrains offline RL algorithms to the data. This argument is supported by theory saying that when these conditions are met, the learned policy is close to the optimal policy and meets certain safety guarantees. They show several ways in which data bias can be introduced that benefit offline RL algorithms, such as length bias. Lastly, this phenomenon is shown empirically across several offline RL algorithms and benchmarks.

Strengths

* Overall, I really enjoyed this paper. While pessimism is known to play an important role in offline RL, I think that this paper does a good job of illuminating how important pessimism is in these offline benchmarks. I think this will be an important paper for the offline RL community, especially when considering these benchmark tasks. * The paper is very well written. * The paper is very comprehensive with a strong Appendix section detailing experiment details and additional results. * The paper has a lot of theory explaining the survival instinct phenomenon and showing how algorithms in the literature have this survival instinct.

Weaknesses

* I still do not fully understand the argument around length bias (see Questions), and I am not convinced that this is a main cause of the phenomenon shown in Figure 3. In particular, how do we know specifically the length of the trajectories is the important property? I would assume that the medium and medium-expert trajectories datasets just have more support over actions that prevent the agent from falling down. The length of the trajectory just seems like a byproduct of these good actions. More discussion or evidence here would be appreciated. * The main paper makes it unclear how data was generated for the grid world. While the appendix does have some more information about that, a little more should be explained here to make the example more understandable. * The offline agent is used on the “wrong reward” but what is the wrong reward? It seems like in the appendix you consider three wrong rewards, but which one is used? It seems like different rewards here could change the outcome drastically. * (Line 219) The authors claim that a dataset generated by the optimal policy is infinitely positively biased regardless of the reward. This seems like a bold claim to make for all MDPs and all reward functions without a proof. * What happens with expert D4RL datasets? It would be interesting to see what performance is on this, especially since the claim is that datasets from the optimal policy are infinitely positively biased regardless of the reward. * Potential typo on Line 1415: The agent starts deterministically in the upper left it seems, not right.

Questions

* Line 217 you say that any policy is infinitely positively biased for any rewards resulting from potential-based reward shaping. Isn’t positive bias a property of the offline dataset and not the reward? This didn’t make sense to me. * Do you have any hypotheses for why some algorithms have different levels of survival instincts? Is it because of the amount of pessimism in the algorithms? * I still have several questions about length bias: * I understand that often in benchmarks longer trajectories correspond to higher returns, but it is not clear to me why this always helps. For instance, suppose we have a dataset with trajectories where half the time the robot fell and half the time it did not. If we label all rewards with -1, wouldn’t the agent be incentivized to fall as fast as possible to incur less negative rewards. * Perhaps I am misunderstanding the assumptions in Proposition 5, but it seems like there is no assumption on how likely long trajectories are to appear in the dataset, only on the returns of long trajectories. I would think there has to be some assumption on the former.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

Although I do not think this work necessarily needs to address these limitations, it would be beneficial to have some discussion about them in my opinion: * While the paper does cover many relevant offline RL benchmarks, it does not explore all types of datasets. In particular, it does not appear that the authors explored cases where there are only sub trajectories of good behavior that need to be stitched together. * The assumption of positively biased data in Theorem 1 seems strong. Since the definition of positively biased data is about the value of the delta-optimal policy set of the CMDP, there seems to be no intuition for when Theorem 1 will hold when looking at properties of the offline dataset only. In other words, it does not appear that there are any methods that practitioners could use to estimate the role of survival instincts for any arbitrary dataset. * While there are theoretical results for offline model-based RL works, the paper’s experiments focus on model-free offline RL algorithms only.

Official Reply from Reviewer k1TU

I appreciate the authors' clarifications, and most of my concerns have been addressed. At the same time, I am impressed by the experiments in [1] which show that offline RL methods with naïve data filtering achieve comparable performance to state-of-the-art offline safe RL algorithms. I believe this work and the d4rl Mujoco dataset will attract the attention of the offline RL community. The previous work [2] has mentioned the phenomenon of learning with incorrect reward signals, but the strength of this work lies in its exhaustive experimentation and analysis of this phenomenon. I will maintain my rating, and I suggest citing the work of Shin D et al. [2] in this paper. [2] Shin D, Dragan A D, Brown D S. Benchmarks and algorithms for offline preference-based reward learning[J]. arXiv preprint arXiv:2301.01392, 2023.

Thank you for reviewing and discussing with us

Dear reviewers, We would like to thank all reviewers again for their time and positive feedback on our work. We will incorporate the reviewers' suggestions in the final revision. It is glad to see that we all think that the findings presented in the paper bring out many new possibilities in RL. We are looking forward to them. Best, Authors