Is Value Learning Really the Main Bottleneck in Offline RL?

Summary

The paper presents an empirical analysis to determine the main challenge in offline RL for control among value function learning, policy extraction, and policy generalization to test-time states. With various deep learning-based experiments, it reaches the conclusion that policy extraction and policy generalization are the main bottlenecks instead of value function learning.

Strengths

1. The paper is systematic on specifically isolating the three components such as by using decoupled RL algorithms (having value function learning and policy learning phase separately). This is important to avoid confounding factors. 2. The paper covers results across various data dimensions such as coverage, sub-optimality, and amount. 3. The paper has clear takeaways, which are helpful in determining actionable advice. 4. The paper provides an alternative direction (e.g. better policy extraction algorithms instead of value function learning algorithms) for researchers to pursue in trying to improve offline RL.

Weaknesses

1. It is a bit concerning that the main results in Figures 1, 2, 6 are over only 4 seeds. In my experience, offline RL algorithms can be quite fragile especially with so much variation in coverage, dataset amount etc, that 4 seems quite limited. Given that most of the environments are not image-based, more than 4 seeds seems reasonable. 2. Related to above, the same figures do not report any information on variance of performance, which makes it difficult to determine if the reported means have been accurately reported. I would suggest reviewing some guidelines [1] 3. While the takeaways are nice, the claim to “always use” (above Section 5) is too strong, in my opinion. 4. In section 4.4, it's unclear if the first reason is a good explanation for better performance. While the actions of DDPG + BC are more extreme, I don't think extreme actions are what we should be aiming for. The experiments are presented as though these extreme actions lead to good performance (even if this was not the intention). While it may be true in the evaluated domains, it is unclear if this can be a general finding. I will be willing to increase the score if the issues above are addressed, especially 1 and 2. [1] Empirical Design in Reinforcement Learning. Patterson et al. 2023.

Questions

1. There seems like there is a relation between DDPG + BC’s focus on limited exploration (some improvement but staying close to behavior policy) and test-time generalization. More specifically, by encouraging a policy to be close to the behavior policy, the test-time generalization should naturally be better since it will not deviate much from those states. Section 4 does not comment on this relation, what are the authors thoughts on this? 2. Regarding the limitations mentioned at the end, the authors may be interested in this paper [1] as well which discusses how some value-based objectives may be unreliable for performance. [1] does not give an suggestions, but does report a similar finding for another use-case. [1] Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error. Fujimoto et al. 2022.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes the authors report on the limitations in Appendix A.

Summary

This paper attempts to understand the relative importance of policy learning and value learning in offline reinforcement learning. The analysis is broken into two parts: (1) when decoupling the policy and value learning steps the authors test the relative data efficiency of the two steps, and (2) the authors test how well the policy generalizes at test time. Experiments are conducted on a wide variety of domains and the authors argue that they show that policy learning is often the main bottleneck of offline RL.

Strengths

1. The high level experimental methodology of comparing the relative importance of policy learning and value learning by varying the dataset size in decoupled algorithms is an interesting idea. It could be useful to guide future work to know if there is more upside to improving on policy learning or value learning. 2. The result that IQL+DDPG outperforms IQL+AWR (which was used in the original IQL paper) is an interesting standalone result. 3. While somewhat preliminary and cursory in the paper, the idea of OPEX and TTT to update the policy at test time with a fixed value is interesting as a way to show that the value function has more information in it than the policy extracts.

Weaknesses

1. The results are very heuristic and often unclear. The definitions of "policy bound" and "value bound" in terms of color gradients is vague and not very informative. Visually, it is hard to get much out of the figures which just throw a ton of data at the reader without a very intuitive way to interpret it. This could maybe be resolved by creating a more clear single metric that indicates the tradeoffs or some summary statistics that show aggregate trends averaging across methods or datasets. Currently, the results are pretty hard to parse and not very convincing as a result. 2. The empirical results are not as conclusive as the analysis/text suggests. For example, the main claim of section 4 that "policy learning is often the main bottleneck of offline RL" does not seem to be the right takeaway or emphasis of the results. Instead the results in both figure 1 and 2 indicate that sometimes policy learning is the bottleneck and sometimes value learning is the bottleneck. 3. The methodology in section 5 is a not clear. It seems that the authors run offline-to-online IQL (line 304), but this would be using AWR, which the previous section suggests not to do. Moreover, the online algorithm updates not just the policy, but also the value. The paper does not consider the generalization of the value function, but only the policy. Perhaps a cleaner way to test interesting out of distribution generalization would be to consider the state distribution of the optimal policy? This would of course not test near optimal states (so maybe some noise could be added or something as in later experiments), but could be conceptually cleaner than fully changing the setting to online. 4. It is not clear why there is such focus on the generalization of the policies and not the values. They seem to both matter, especially for things like OPEX or TTT to work. It would be interesting to see how well the value functions are generalizing as well as the policies to go to the papers main claims to be comparing the relative importance of these two steps. 5. In general, the paper tries to cram in too many semi-related results. I would encourage the authors to focus the story a bit more clearly and maybe split some parts into the appendix or into a separate paper.

Questions

See weaknesses

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

Yes, but could do more to address how the results are not always clear cut.

Summary

This paper empirically analyzes the bottlenecks in offline RL from three aspects: value learning, policy extraction, and policy generalization at evaluation time. Through the empirical evaluation, two observations were made: 1) the policy extraction algorithms affect the performance of offline RL significantly, often more than its underlying value learning objective. 2) the sub-optimal performance of offline RL agents is mainly attributed to a lack of generalization of the unseen state during evaluation instead of accuracy in the training distribution.

Strengths

**Originality**: Good. This paper tries to analyze the bottleneck of offline RL methods and includes some novel and interesting observations that were not systematically discussed before. **Clarity**: Good. The paper is well-structured and easy to follow. The takeaway after the empirical analysis helps the reader understand the experiment's results. **Significance**: This is important. The reason behind offline RL's underperformance is an important topic for the future development of offline RL methods. **Technical Accuracy**: 1. Throughout evaluation with a decent amount of experiments 2. The experiment design are motivated and backed by hypothesizes

Weaknesses

1. There is a lack of variance measures, such as standard deviation (std) or confidence intervals (CI), for the data-scaling matrices in Figures 1, 2, and 6, making the numerical results less plausible. 2. For some experiments in empirical analysis 1, an increase in data leads to a decrease in performance. For instance, in Figure 1 (gc-antmaze-large), both IQL+AWR and IQL+DDPG show that the 10-10 configuration performs worse than the 1-10 and 10-1 configurations, which seems to contradict to the assumption, more data leads to better value/policy approximation. These observations lack an adequate explanation. 3. I am concern that one of the main question (as outlined in the title as well) “is value function learning the main bottleneck of offline RL” is not sufficiently examined in the empirical experiments. While the amount of data used for training can be an implicit indicator of how well the value function is trained, it does not directly tell us the distribution of approximation errors (e.g., overestimation). Overestimation could still be a significant issue in offline reinforcement learning. It would strengthen the augment if an direct comparison between the predict value and true value for can be conducted, just like in [1] and [2]. [1] Van Hasselt, H., Guez, A. and Silver, D., 2016, March. Deep reinforcement learning with double q-learning. In *Proceedings of the AAAI conference on artificial intelligence* (Vol. 30, No. 1). [2] Fujimoto, S., Hoof, H. and Meger, D., 2018, July. Addressing function approximation error in actor-critic methods. In *International conference on machine learning* (pp. 1587-1596). PMLR.

Questions

1. For empirical analysis 2, I am wondering if the online data mainly improves value learning, policy learning, or both of them. The current evaluation only shows the improvement of evaluation MSE in actions but I'm wondering if the Q-value estimation loss would follow the same pattern, maybe the author could provide more insights on this?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations were not discussed in the paper.

Summary

This paper addresses the question of why offline RL often underperforms imitation learning. They formalize the question they choose to ask, " is the bottleneck in learning the value function, the policy, or something else? What is the best way to improve performance given the bottleneck?", and provide three potential explanations: imperfect value function estimation, imperfect, policy extraction from value function, and imperfect policy generalization to novel states during evaluation. They next perform a series of experiments to test each hypothesis, and conclude two main reasons for offline RL's underperformance: policy extraction from value functions, and test-time policy generalization. They use these observations to highlight important takeaways for RL practitioner and RL researchers.

Strengths

I think that this is an exceptionally well-written paper. They make it very clear what their hypotheses are, how they test for each, and what the reader should take away from each subsection. I further think that this paper addresses important questions in offline RL, and brings to light important directions for future work, which is a valuable contribution. The full experimental details as well as code are provided, which is very useful for the community in reproducing the results and building off of this work.

Weaknesses

At a high level, there are not many weaknesses I can name in this work. One is perhaps that their contributions are mainly empirical observations, and it would be nice to support these with theoretical results (even in very simple settings full of necessary assumptions), but I believe that even without theory this paper is very strong.

Questions

From what I can tell, most of the empirical results are in environments with continuous action spaces. Do you expect the results to carry over to environments with discrete action spaces?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors discuss and address limitations of their work.