Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL

The divergence of the Q-value estimation has been a prominent issue in offline RL, where the agent has no access to real dynamics. Traditional beliefs attribute this instability to querying out-of-distribution actions when bootstrapping value targets. Though this issue can be alleviated with policy constraints or conservative Q estimation, a theoretical understanding of the underlying mechanism causing the divergence has been absent. In this work, we aim to thoroughly comprehend this mechanism and attain an improved solution. We first identify a fundamental pattern, self-excitation, as the primary cause of Q-value estimation divergence in offline RL. Then, we propose a novel Self-Excite Eigenvalue Measure (SEEM) metric based on Neural Tangent Kernel (NTK) to measure the evolving property of Q-network at training, which provides an intriguing explanation of the emergence of divergence. For the first time, our theory can reliably decide whether the training will diverge at an early stage, and even predict the order of the growth for the estimated Q-value, the model's norm, and the crashing step when an SGD optimizer is used. The experiments demonstrate perfect alignment with this theoretic analysis. Building on our insights, we propose to resolve divergence from a novel perspective, namely improving the model's architecture for better extrapolating behavior. Through extensive empirical studies, we identify LayerNorm as a good solution to effectively avoid divergence without introducing detrimental bias, leading to superior performance. Experimental results prove that it can still work in some most challenging settings, i.e. using only 1 transitions of the dataset, where all previous methods fail. Moreover, it can be easily plugged into modern offline RL methods and achieve SOTA results on many challenging tasks. We also give unique insights into its effectiveness.

Paper

References (61)

Scroll for more · 38 remaining

Similar papers

Peer review

Reviewer 4Xya7/10 · confidence 4/52023-06-07

Summary

This paper studies the divergence phenomenon in Q-value iteration methods (e.g. Q-learning), especially focusing on the offline RL scenario. They introduce a theoretical framework for studying this issue, predicting divergence and even the training step at which it is likely to happen. Such analysis, which is based on the NTK (Neural Tangent Kernel), fully contains the linear case study (i.e. deadly triad analysis in linear settings) and extends it to non-linear value approximation. The insights they draw allow them to show that reducing a new metric, SEEM which measures self-excitation using NTK, reduces divergence by regularising the network. Regularising the generalisation of Q-networks as opposed to the dominant view of constraining the policy results in a new viewpoint in handling divergence in Q estimation and could yield many improvements in this domain in the future. They experiment with various regularisation methods from the deep learning toolbox, including dropout, BatchNorm, and LayerNorm, showing that LayerNorm is best able to stabilise training (i.e. avoid blowup in Q estimates) and achieve more homogenous and lower SEEM. They show that BC (behaviour cloning) also achieves a low SEEM but at the cost of bias due to policy constraints, which does not allow it to find the best solution or perform well in more challenging domains.

Strengths

- The regularisation viewpoint on this phenomenon is a new one (as far as I know) yet a natural one. - The regularisation viewpoint frees us from introducing policy constraints (which introduce bias) and allows us to simply solve the problem using off-the-shelf methods from the deep-learning toolbox. - The theorems (if proofs are correct, I couldn't verify them personally with certainty) bring highly beneficial theoretical insights to the problem. - The experiments are remarkable at showing that: (1) SEEM is a powerful metric in signifying divergence, (2) LayerNorm reduces SEEM and more homogeneously does so during training than other regularisation methods, resulting in stable Q estimation, (3) LayerNorm combined with SOTA offline RL methods results in significant improvements, especially under scarce access to offline data.

Weaknesses

- I believe a short description of offline RL methods used could enhance the exposition of the ideas: policy constraint is not formally introduced because the methods incorporating it are not discussed in any depth.

Questions

1. Given that this paper does not consider EMA or frozen targets (as in DQN), I am wondering how much using such gradual target updates contributes to alleviating divergence in Q estimation. In relation to the above question, I'm also uncertain how much using double Q-learning alleviates such issues in the offline RL setting. Wouldn't it be useful to have an ablation study to see, e.g., how much adding each of these in combination with each other helps performance and how much each of these help in isolation? Say, use **DDPG w/o EMA** vs. **DDPG w EMA** vs. **DDPG w EMA and Double Q-learning** vs. **DDPG w EMA and Double Q-learning and LayerNorm**, showing that additional advancement improves performance. Or compare in isolation, e.g.: **DDPG w/o EMA w LayerNorm** vs. **DDPG w EMA w/o LayerNorm**. 2. Are the results in Fig. 9 in combination with EMA? If not, what if you use EMA to show how the insights from the scenario w/o EMA carry over to the with-EMA case? 3. Does using LayerNorm as in your solution have the potential of allowing online DQN-style methods to step away from EMA/frozen target approaches altogether towards using the same online and target networks? 4. Does your solution allow DQN to solve Baird’s Counterexample (which was first used to show the potential divergence of Q-learning with function approximation)? 5. Does the analyses apply also to discrete-action pure critic settings such as DQN? **Minor:** - Can you elaborate on the way BC introduce explicit bias? - I assume from the size of the $d_0$ layer that the architecture of the Q-function is action-in, correct? Line 82: don’t -> do not Line 119: To summary -> In summary Line 123: explains -> explain

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

1. Analysis does not directly apply to the standard practice of using Exponential Moving Average (EMA; as in DDPG) or target-network freezing (as in DQN). 2. It is not clear how much of the contributions would carry over to the online learning setting and what would be the implications of using LayerNorm in online RL. 3. The analysis seems to apply to MLP architectures. It is not directly discussed how much of the contributions would apply to commonly used architectures such as ConvNets, ResNets, etc.

Reviewer T4o65/10 · confidence 4/52023-06-14

Summary

The paper theoretically investigates the problem of value function overestimation in offline RL through the lens of neural tangent kernel (NTK). Additionally, the paper presents empirical findings that validate the effectiveness of incorporating LayerNorm before each activation function in mitigating value network divergence. The paper conducts extensive experiments on D4RL AntMaze and D4RL MuJoCo Gym while varying the dataset size and demonstrates that LayerNorm ensures stable value convergence and leads to state-of-the-art performance even with significantly small datasets.

Strengths

- The paper is well written and organized. The paper provides empirical evidences to support its theoretical claims (Figures 3, 4). - The proposed method can be easily integrates with existing offline RL algorithms (e.g., CQL, IQL, TD3+BC) and consistently improves performance.

Weaknesses

- It is worth noting that DR3 has already investigated the dynamics of Q-learning in offline RL using NTK [1]. It is crucial for the author to properly reference this prior work and to establish a clear connection between the two studies. The author should explicitly highlight the similarities and differences between the proposed analysis and the findings presented in DR3. - The idea of applying LayerNorm (or GroupNorm) has already been proposed in Scaled QL [2]. - There should be a proper reference to support the claim that the neural network becomes a linear function when the input’s norm is too large (line 234). - It is unclear why the linearity of the neural network leads to an improperly large kernel value (line 234). - The experimental setup in Section 5.2 is very similar to DR3 [1]. [1] Aviral Kumar et al., DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization, ICLR 2022. \ [2] Aviral Kumar et al., Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes, ICLR 2023.

Questions

Please see the weaknesses above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

N/A

Reviewer WzD57/10 · confidence 4/52023-07-05

Summary

This paper analyzes Q-value divergence in Offline-RL by considering a neural tangent kernel for the value function. They show that consideration of this kernel is predictive of Q-value divergence. This analysis further leads to the observation that using a LayerNorm yields a kernel that behaves more like one would hope -- with nearer values being more impacted than farther ones. Their empirical tests on D4RL benchmarks similarly show the benefits of LayerNorm.

Strengths

I like the insights in this paper and suspect that its publication will lead others to further engineering that will yield even better performing adjustments than the simple addition of the LayerNorm. Figure 5 suggests there is more to be done. Why does the figure on the left indicate almost the opposite of what one would hope for. At a minimum, should the value right at x_0 be red?

Weaknesses

The paper needs a number of editing improvements. First, it needs an English grammar checker. The errors mostly don't lead to difficulties in understanding, but, the paper should not be published with the current level of English quality. In section 5.1, the BC term should be explained even though the reference is given for it. The reference to a score of 81.4 in that section does not appear to match the numbers in Table 1. The legends in the figures need to use a larger font size.

Questions

None

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

N/A

Reviewer T4o62023-08-11

Thank you for your explanation on the differences between the proposed analysis and that of DR3. While Theorem 3 in your paper and Theorem 3.1 in DR3 arrive at similar conclusions, DR3 adopts more assumptions. I find this quite intriguing. Have you verified the validity of the assumptions and lemmas you made? Specifically, regarding Lemma 1, it is not common that an MLP with ReLU activation is a homogeneous function in the parameter space.

Authorsrebuttal2023-08-11

Thank you for your time and reply. Here is our further explanation, we hope this can help to resolve your confusion. # About Theorem 3 and Theorem 3.1 in DR3 Yes, these two results do seem to be similar. But they are actually derived from quite different foundations. Our results convey novel implications by capturing a mechanism that is more realistic, so that it requires fewer assumptions to reach a similar conclusion. Specifically, DR3 Theorem 3.1 analyzes the condition for $\theta^*$ to be the stable fixed point of TD-update, namely $Q_{\theta^*}\left(s_i, a_i\right)=r_i+\gamma Q_{\theta^*}\left(s_i', a_i' \right)$ for every $(s_i,a_i,s_i')$. This means DR3's analyzing framework **focuses on the setting where the model perfectly fits the offline training data**, and the SGD noise plays a role as implicit regularization. This noise continues to increase the Q-value for the unseen data point $Q_{\theta^*}(s_i',a_i')$ in the null space of $G$, while preserving its fixed-point property in training data at the same time. As a consequence, all the dataset points' value inflates to infinity. Our key difference and advancement is that, we found such divergence does not happen in the parameter subspace where the Q-network perfectly fits the training data's Q-value Bellman equation. Actually, the model is "chasing its shadow" rather than staying at the stable fixed point of Bellman update. Once the pathological extrapolation of MLP starts the so-called self-excitation procedure, **the training loss will never become near-zero, but keep exploding**. The underlying mechanism is explained in our initial rebuttal response from step 1 to 4. Each TD update step on $\theta$ tries to let $Q_{\theta}(s_i, a_i)$ get closer to $r_i+\gamma Q_{\theta}\left(s_i', a_i'\right)$, but increase the latter bootstrpped target even more due to improper generalization. In conclusion, our theory for explaining the divergence mechanism differs at the very beginning. It captures a mechanism that has yet to be fully discovered before. Therefore, being more fundamental, our analysis is able to reach a conclusion without assumptions like zero training loss or assumptions on SGD noise's covariance, etc. # Homogeneity of ReLU-activate MLP Indeed, ReLU-activated MLP **with bias** is not rigorously homogeneous in the entire parameter space. But bias-free ReLU activated MLP, namely function $f(x)=W_L \sigma(W_{L-1} \sigma (\cdots \sigma(W_1 x)))$ is homogeneous with respect to $\theta=(W_1, W_2, \cdots, W_L)$. You can verify this from simple fact that $W_i x$ and ReLU activation $\sigma(\cdot)$ are homogeneous functions ($\max(kx,0)=k\max(x,0)$), and the composition of homogeneous functions is homogeneous. However, **it is important to point out that our analysis also applies to ReLU-activated MLP with bias in each layer.** The main reason is a little subtle. We empirically found that such homogeneity still holds with high precision for MLP with bias, at least good enough in signifying the value divergence and giving correct asymptotic order. We hypothesize that when value divergence is going to happen, the product dot $W_i z_i$ is relatively large compared to the scalar $b_i$, and the effect of bias in output is negligible. We also run experiments to validate the homogeneity. We define a family of 3-layer ReLU-activated MLPs (with bias term) with different scaling $\lambda$ of the same network parameter $\theta$ (from D4RL training checkpoints when value divergence happens), and feed these networks with the same input. We show one example in the below table, which confirms that the $L$-degree homogeneity of output, gradient and NTK is valid at high precision, the NTK is almost parallel, too. The larger the scaling factor is, the more accurate $k^L$ increasing pattern is for output ( $k^{L-1}$ for gradient and $k^{2(L-1)}$ for NTK), as described in Lemma 1. Further, we empirically validated the $L$-degree homogeneity of NTK holds for all checkpoints in D4RL experiments where divergence happens. | $\lambda$ | 5 | 10 | | ---- | ---- | ---- | | $f_{\lambda \theta}(x)/f_{\theta}(x)$ | 124.99 ($5^3$) | 999.78 ($10^3$) | | grad scale | 25.001 ($5^2$) | 100.001 ($10^2$) | | NTK scale | 624.99($5^4$) | 9999.84 ($10^4$) | | NTK cos | $>1-10^{-6}$ | $>1-10^{-6}$ |

Reviewer T4o62023-08-12

Please check the lemma

I appreciate your response. It is now evident that Lemma 1 relies on empirical observations rather than being theoretically accurate. Now, let's look at Theorem 1. Is it appropriate to approximate $X_{t+1}^* \approx X_t^* $, knowing that $X_t^*$ contains the $\arg\max$ function? It is important to carefully examine the proofs of the lemmas and theorems you proposed, considering their informal nature.

Authorsrebuttal2023-08-12

Clarifying and Validating Assumptions

Thank you for your appreciation. We highly value your comments and believe that they have allowed us to further refine our work. Firstly, we would like to clarify the expression regarding our paper's reliance on empirical observations rather than being theoretically accurate. **Our approach aims to be theoretically accurate and rigorous under the realistic assumption**. This assumption that the bias term has a negligible effect on output and NTK in Q-value divergence scenarios is backed by empirical validation, as we have shown. **Adopting realistic assumptions does not undermine the value and accuracy of theoretical studies.** For example, DR3 also adopts a zero-critic loss assumption, which might not always hold true in real offline RL scenarios. Regarding the equation $X^*_{t+1} \approx X^*_t$, we agree that validating this assumption is crucial. Indeed, we were intrigued when we first discovered this relationship. **We have verified it in all offline environments we studied, and showcase one of them in our draft**. You can refer to Fig 1 in our draft, where the second column "action similarity" measures the cosine similarity between each step's policy action and the final policy, remaining 1 for most of the time. Our attachment to the global response also shows that policy actions become extreme-point actions once Q-values diverge. These two experimental results support our assumptions. Your concerns were helpful for identifying unclear justifications for assumptions and improving the clarity of our paper. We will revise our paper accordingly to make the validation of assumptions more clear. We're grateful for your reviews.

Authorsrebuttal2023-08-17

Enhancing Precision of Assumption Statement

After carefully considering your concerns regarding our assumptions, such as $X_{t+1}^* \approx X_{t}^*$ and the homogeneity of the ReLU-activated MLP, we realized the potential ambiguity introduced by our informal notation "$\approx$". This could potentially compromise the rigor of our theoretical deductions. While our assumptions have been empirically validated in pratical experiments, we understand your concern about the rigor of the assumption and theoretical preciseness. We agree that **using precise quantification such as** $|X_{t}^* - X_{t+1}^*|=o(\eta)$ **rather than** $X_{t+1}^* \approx X_{t}^*$ **would provide a clearer description of our assumptions**. Similarly for the definition of near-homogeneity of ReLU MLP. Adopting such formal notation will refine our proofs and make our study more precise. We will revise our theory statement in a more rigorous manner. We really appreciate your constructive feedback about the rigorous assumption definition. Indeed, a formal description is essential for enhancing a theorem's accuracy. However, we believe this lack of formalism doesn't invalidate the correctness of our theorem or diminish the overall contribution of our work. The identified issues related to these assumptions are technical in nature, rather than fundamental flaws. Since the assumptions are aligned with empirical observations and the subsequent deductions are carried out with rigor, our results rest on a solid, realistic foundation. We have verified that all theoretical results can also hold with a rigorous assumption description, (e.g. o(·), O(·)) without essential difficulty. We believe that the current limitations in the formal rigor of our assumptions will not detract from our vital contributions, specifically the novel insights into Q-value divergence and its underlying mechanism. Once again, thank you for your invaluable insights which have guided us to refine our theoretical presentation. We welcome any further discussions or clarifications.

Reviewer T4o62023-08-19

Thank you

Apologies for the delayed response. Many of my concerns have been addressed. However, it is not accurate to claim that $|X_{t+1}^* - X_{t}^*| = o(\eta)$ without any underlying assumptions, especially given that $\arg\max$ is not Lipschitz continuous. Upon reviewing both the main text and the supplementary materials, I was unable to locate any assumptions pertaining to Theorem 1. In the revised version, please explicitly specify the assumptions you've employed to validate the theorems. I raise my score from 3 to 5.

Authorsrebuttal2023-08-19

Thanks for your response

Thank you for for taking the time to thoroughly review both our main text and supplementary materials. We greatly appreciate your expert suggestions, which indeed have helped us improve the rigor and clearness of our paper. We will ensure to explicitly specify these assumptions pertaining to theorems in the revised version of the manuscript. We're also grateful for your increased score and your overall appreciation of our paper.

Reviewer 4Xya2023-08-19

Thank you for your response, thoroughly addressing my questions/concerns. I also read through the discussion with reviewer T4o6 and believe that the reviewer's concerns are being addressed. I have now raised my confidence score and would be glad to see the paper accepted.

Authorsrebuttal2023-08-19

Thanks for taking the time to review our paper and for your prompt response. We also appreciate your constructive suggestions, which help us more clearly identify LayerNorm's effectiveness. We are honored to have received your support for its acceptance.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC