Thank you for your time and reply. Here is our further explanation, we hope this can help to resolve your confusion.
# About Theorem 3 and Theorem 3.1 in DR3
Yes, these two results do seem to be similar. But they are actually derived from quite different foundations. Our results convey novel implications by capturing a mechanism that is more realistic, so that it requires fewer assumptions to reach a similar conclusion.
Specifically, DR3 Theorem 3.1 analyzes the condition for $\theta^*$ to be the stable fixed point of TD-update, namely $Q_{\theta^*}\left(s_i, a_i\right)=r_i+\gamma Q_{\theta^*}\left(s_i', a_i' \right)$ for every $(s_i,a_i,s_i')$. This means DR3's analyzing framework **focuses on the setting where the model perfectly fits the offline training data**, and the SGD noise plays a role as implicit regularization. This noise continues to increase the Q-value for the unseen data point $Q_{\theta^*}(s_i',a_i')$ in the null space of $G$, while preserving its fixed-point property in training data at the same time. As a consequence, all the dataset points' value inflates to infinity.
Our key difference and advancement is that, we found such divergence does not happen in the parameter subspace where the Q-network perfectly fits the training data's Q-value Bellman equation. Actually, the model is "chasing its shadow" rather than staying at the stable fixed point of Bellman update. Once the pathological extrapolation of MLP starts the so-called self-excitation procedure, **the training loss will never become near-zero, but keep exploding**. The underlying mechanism is explained in our initial rebuttal response from step 1 to 4. Each TD update step on $\theta$ tries to let $Q_{\theta}(s_i, a_i)$ get closer to $r_i+\gamma Q_{\theta}\left(s_i', a_i'\right)$, but increase the latter bootstrpped target even more due to improper generalization.
In conclusion, our theory for explaining the divergence mechanism differs at the very beginning. It captures a mechanism that has yet to be fully discovered before. Therefore, being more fundamental, our analysis is able to reach a conclusion without assumptions like zero training loss or assumptions on SGD noise's covariance, etc.
# Homogeneity of ReLU-activate MLP
Indeed, ReLU-activated MLP **with bias** is not rigorously homogeneous in the entire parameter space. But bias-free ReLU activated MLP, namely function $f(x)=W_L \sigma(W_{L-1} \sigma (\cdots \sigma(W_1 x)))$ is homogeneous with respect to $\theta=(W_1, W_2, \cdots, W_L)$. You can verify this from simple fact that $W_i x$ and ReLU activation $\sigma(\cdot)$ are homogeneous functions ($\max(kx,0)=k\max(x,0)$), and the composition of homogeneous functions is homogeneous.
However, **it is important to point out that our analysis also applies to ReLU-activated MLP with bias in each layer.** The main reason is a little subtle. We empirically found that such homogeneity still holds with high precision for MLP with bias, at least good enough in signifying the value divergence and giving correct asymptotic order. We hypothesize that when value divergence is going to happen, the product dot $W_i z_i$ is relatively large compared to the scalar $b_i$, and the effect of bias in output is negligible.
We also run experiments to validate the homogeneity. We define a family of 3-layer ReLU-activated MLPs (with bias term) with different scaling $\lambda$ of the same network parameter $\theta$ (from D4RL training checkpoints when value divergence happens), and feed these networks with the same input. We show one example in the below table, which confirms that the $L$-degree homogeneity of output, gradient and NTK is valid at high precision, the NTK is almost parallel, too. The larger the scaling factor is, the more accurate $k^L$ increasing pattern is for output ( $k^{L-1}$ for gradient and $k^{2(L-1)}$ for NTK), as described in Lemma 1. Further, we empirically validated the $L$-degree homogeneity of NTK holds for all checkpoints in D4RL experiments where divergence happens.
| $\lambda$ | 5 | 10 |
| ---- | ---- | ---- |
| $f_{\lambda \theta}(x)/f_{\theta}(x)$ | 124.99 ($5^3$) | 999.78 ($10^3$) |
| grad scale | 25.001 ($5^2$) | 100.001 ($10^2$) |
| NTK scale | 624.99($5^4$) | 9999.84 ($10^4$) |
| NTK cos | $>1-10^{-6}$ | $>1-10^{-6}$ |