Doubly Mild Generalization for Offline Reinforcement Learning

Summary

The paper proposes an algorithm to allow for some mild generalization to OOD actions in offline RL by proposing constraints on which actions to generalize on and by constraining bootstrapping signal to avoid overestimation. It includes results and ablations on common offline RL benchmarks.

Strengths

1. The idea is very simple (which is a good thing!), intuitive, and can be easily integrated into existing algorithms. I think it provides nice insight into how to think about generalization and overestimation which can be useful for others. 2. The paper does a nice job of explaining the intuition behind the method and faults with previous methods. I appreciated Table 1. 3. The paper includes results on downstream performance metrics as well as ablation studies.

Weaknesses

1. It is a bit concerning that the main results are over only 5 seeds. While it is commonly done in prior work, I don’t think that means it's correct. I’d refer authors to [1]. I think it would be nice to increase the number of seeds. 2. Related to above, the variations seem to be based on standard errors (for example, Table 2) but it has typically been documented that these are unreliable and should be avoided [1] in RL research. Instead opting for some high-confidence confidence interval would be better. I would be willing to increase the score if the results were still true with more seeds. [1] Empirical Design in Reinforcement Learning. Patterson et al. 2023.

Questions

1. How do the authors think the insights in this paper can be used for off-policy evaluation where the behavior policy is unknown and we want to evaluate a fixed target policy? The control setting allows some flexibility in say , constraining what the optimal policy can be, but with evaluation this seems trickier. 2. The method seems to rely on using the empirical behavior policy to constrain the policy improvement. This seems reasonable, but the paper does not study this deeply. While this question may seem separate from the question investigated in this paper, in offline RL, it is typical for multiple policies to generate the data, which means the empirical policy is a multi-modal policy. I am curious how sensitive the proposed method would be to bad estimates in the empirical policy? At the very least I think the paper should discuss this point since the algorithm makes the assumption that such an empirical policy is easily accessible.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes, the paper addresses this in Section 6.

Summary

The paper introduces a novel approach called Doubly Mild Generalization (DMG) to tackle the well-known issue of over-generalization in Offline Reinforcement Learning (RL). Offline RL often suffers from extrapolation errors and value overestimation due to the generalization of value functions or policies towards out-of-distribution (OOD) actions. To address these challenges, the authors propose DMG, which comprises two key components: (i) mild action generalization and (ii) mild generalization propagation. Mild action generalization involves selecting actions in the close neighborhood of the dataset to maximize the Q values, while mild generalization propagation mitigates the propagation of potential erroneous generalization through bootstrapping without impeding the propagation of RL learning signals. Theoretically, DMG ensures better performance than in-sample optimal policies under oracle generalization scenarios and controls value overestimation even in worst-case generalization scenarios. Empirically, DMG achieves state-of-the-art performance on standard offline RL benchmarks, including Gym-MuJoCo locomotion tasks and challenging AntMaze tasks, and demonstrates strong online fine-tuning performance.

Strengths

- **Innovative Approach**: The concept of doubly mild generalization is novel and addresses a significant challenge in offline RL by balancing the need for generalization with the risk of over-generalization. - **Theoretical Rigor**: The paper provides a thorough theoretical analysis of DMG, proving its advantages in both oracle and worst-case generalization scenarios. The theoretical guarantees add robustness to the proposed method. - **Empirical Performance**: DMG demonstrates superior empirical results, outperforming existing methods on multiple benchmarks. This includes both standard locomotion tasks and more complex navigation tasks, showcasing the versatility and effectiveness of the approach. - **Seamless Transition to Online Learning**: The method's ability to smoothly transition from offline to online learning, achieving strong fine-tuning performance, is a notable strength, making it practical for real-world applications where a mixture of offline data and online interactions is common.

Weaknesses

- **Dependence on Continuity Assumptions**: The theoretical analysis relies on certain continuity assumptions about the Q function and transition dynamics. While these assumptions are standard, they may not hold in all practical scenarios, potentially limiting the applicability of the theoretical guarantees. - **Potential Sensitivity to Hyperparameters**: The performance of DMG might be sensitive to the choice of hyperparameters such as the mixture coefficient (λ) and penalty coefficient (ν). A more detailed analysis of hyperparameter sensitivity would be beneficial to understand the robustness of the method.

Questions

- Can you explain why using max and min in Eq.5? It’s a bit confusing as the policy is normally a Gaussian or deterministic policy. - What about directly regularizing the KL between $\pi$ and $\hat \beta$ in Eq 13? - Comparison with more SOTA algorithms such as STR is suggested. - What does the symbol × mean in Figure 1 - It seems the performance of DMG when $\lambda=1$ in Figure 1 is inferior to that of TD3BC. I’m curious what’s the potential reason for this phenomenon as it should be better or similar to TD3BC when the in-sample term is removed.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have discussed the limitations in the draft.

Summary

The offline RL community has recently shown a surge of interest in in-sample offline reinforcement learning algorithms. However, these algorithms can sometimes be too restrictive, unable to leverage the generalization capability of deep neural networks. As a remedy, the authors proposed an algorithm called Doubly Mild Generalization that uses the weighted mean of the in-sample maximum and mildly generalizable maximum Q-value. Experiments conducted on various offline RL benchmarks manifest the effectiveness of the proposed algorithm.

Strengths

The authors provide rigorous mathematical proofs for most of their arguments. They also provide a practical version of their algorithm, which performs very well on multiple offline RL benchmarks.

Weaknesses

1. The constants $C_1$ and $C_2$ of Theorem 1 are state-dependent. 2. Lines 120-121 are difficult to understand. How does (4) relate to the generalizability of Q functions? 3. The meaning of Assumption 1 is unclear at first glance. An explanation of what might prevent $\mathcal{T}_{\textrm{DMG}}$ from being well-defined would be helpful for the readers. Also, I think the term "well-defined" is misleading. There's no problem with "defining" the DMG operator for state-action pairs outside the dataset; it's just that they would be incorrect and not be very meaningful. ### Minor comments 1. $\alpha$ is missing from Eq. (21) 2. $k=\infty$ → $k\to\infty$ in Equations (48) and (49)

Questions

Please refer to the **Weaknesses** section.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors adequately addressed the limitations and potential negative societal impact of their work.

Summary

This paper studies the problem of extrapolation error and value overestimation in Reinforcement Learning (RL). The authors exploit generalization in offline RL by using mild action generalization and mild generalization propagation.

Strengths

1. The problem studied is on interest : offline RL typically suffers from study of generalization beyond datapoints seen in the dataset. 2. Strong theoretical results complimented with rigorous experiments. I'm impressed with the number of baselines the authors have tried out.

Weaknesses

Assumption 11 looks very restrictive. The scenarios to which the approach can be applied gets significantly reduced if the transition needs to be lipschitz in the action space (for ever state, state next pair). For example for robotics tasks, where transitions are determinisitc, the transition matrix will take values 0-1, mnaking the Lipschitz constant very large in this scenario.

Questions

The idea of generalization to stata-action pairs in the neighborhood of those seen in the dataset has been studied in https://arxiv.org/pdf/2402.12570 and I urge the authors to cite and state the differences to the approach in this paper.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

This paper has no significant limitations.