Thank you for your reply. It appears that there may be a misunderstanding. The reviewer seems to believe that the instability in Recurrent off-policy RL is due to numerical explosion or instability of RNN gradients, and that addressing the gradient issues alone would resolve the overall training stability. This understanding aligns with the traditional issues of RNNs and corresponding solutions. However, there will be different issues in off-policy RL.
Our experimental results (as shown by the red line in Fig. R3) indicate that the gradient explosion problem is not significant, as advanced RNNs (such as GRU, Mamba, etc.) have already addressed gradient instability quite effectively. If the gradients were indeed growing exponentially with the number of steps, merely reducing the RNN learning rate by 30 times would not be sufficient to counteract this exponential divergence, whereas gradient clipping would effectively mitigate such issues. Contrary to this, the experimental results show that our method is significantly more stable than gradient clipping.
This is because the fundamental cause of instability in Recurrent off-policy RL is not gradient instability, but rather excessively large output variations between consecutive updates of the RNN. Even if the gradients are not large (on the same order of magnitude as those in MLPs), the outputs of the recurrent policy/value function can vary significantly between updates. These large output changes introduce instability into RL. For instance, in value function training, the (simplified) optimization target is $||Q(s,a)-r(s,a)-\gamma\hat{Q}(s',a')||_2^2$, where $\hat{Q}$ and $a'$ are the target value function network and the outputs of the current policy on $s'$, respectively. If the outputs of the value function network and policy network vary significantly, $a'$ will change greatly, and $\hat{Q}(s',a')$ will also fluctuate substantially. This results in large shifts in the optimization target of the value function after each update, leading to training instability. Similar instabilities occur in policy training, contributing to overall RL training instability. This is why, as shown in Fig. R4, even after clipping the gradients, the value loss remains unstable. **To the best of our knowledge, this finding has not been reported in previous work.** We will include these discussions in the revised version of our paper.
Our theoretical results demonstrate that output variation does not increase exponentially with the number of steps but converges to a certain value. Therefore, we can balance the output variations, amplified by time steps, with a small RNN-specific learning rate. In contrast, if gradient clipping or truncation is used to keep the gradients within a normal range (similar to MLPs), the RNN output variations will still be large, and RL training will remain unstable.
The results of Gradient truncation with 4 or 8 truncation steps are shown in the table below. In both variants, the algorithm produced infinite outputs before 250 iterations, leading to early stopping.
| | $LR_{CE}$=$10^{-5}$ | $LR_{CE}$=$3\times10^{-4}$, grad-step-truncation-4 | $LR_{CE}$=$3\times10^{-4}$, grad-step-truncation-8 | $LR_{CE}$=$3\times10^{-4}$, grad-step-truncation-32 |
| :-----------------: | :-------------------------------------: | :------------------------------------------------: | :------------------------------------------------: | :-------------------------------------------------: |
| AntBLT-V-v0 | $ \mathbf{1986} \pm\mathbf{73} ^\star$ | $408\pm 74$ | $269\pm 88$ | $499±55$ |
| HalfCheetahBLT-V-v0 | $ \mathbf{2679} \pm\mathbf{176} ^\star$ | $-107\pm 320$ | $-703\pm 330$ | $−458±244$ |