Summary
This paper considers the problem of a noisy signal affecting the policy updates when using actor critic algorithms for policy improvement. It takes inspiration from the idea of symmetric cross-entropy loss, which has been shown to mitigate a similar problem caused by noisy labels in supervised classification tasks. The paper proposes to modify the policy gradient updates for A2C and PPO by creating corresponding reverse losses for each of these algorithms, which when incorporated into the regular loss, gives us the "symmetric" A2C and PPO losses. Gradient analysis is done on these new losses to show that their introduction does not interfere with the gradients of the original losses. Finally, experiments are done in tasks with discrete action spaces, continuous action spaces, and with LLM training, to show that these symmetric losses improve performance compared to the baselines. Specifically, this paper shows that the symmetric PPO loss improves performance over the regular PPO loss, probably because it mitigates the effects of the small batch, advantage normalization, and the progressively off-policy updates that take place in PPO.
Strengths
* The paper takes an approach useful for classification with noisy labels and successfully modifies it for policy gradient approaches. The idea itself is intriguing and needs to be delved into deeper to better characterize how estimation errors due to function approximation or noisy rewards can be mitigated via this scheme.
* The gradient analysis in Section 4.3 gives some confidence that the proposed approach is not harming learning.
* Experiments successfully degrade performance of A2C and PPO, and show that the symmetric versions better deal with the noisy rewards.
* The hypothesized reasons for why SPPO improves over PPO make intuitive sense.
* Figure 2 is a great addition to the paper. It clearly shows that even though advantage normalization is necessary in PPO, that normalization can potentially change whether an action is encouraged or discouraged based on data in the mini-batch.
Weaknesses
* This paper seems to have the seeds of a great idea. Some deeper investigation and perhaps an attempt at generalizing the proposed approach beyond two specific loss functions might be useful.
* I could see the parallels between the RCE loss and the proposed reverse A2C and reverse PPO losses. But a first principles derivation of the loss could perhaps be more convincing and lead to a more general loss. As it stands, the loss seems slightly shoe-horned, and I'm not convinced it is the correct drop in for a reverse cross-entropy loss. I see the advantage value $A(x, k)$ as the equivalent of $q(k|x)$. That cal also be connected to the noisy labels and the error in advantage estimation. But the proposed loss seems to consider the action taken by the agent to be $q(k|x)$. Could the authors explain the justification for this choice? One suggestion to instead use the advantages is to map the advantages to a simplex and turn them into probabilities. The RCE loss can then be directly adapted. Could that perhaps be a better approach?
* Perhaps this is a repeat of the previous point, but in equation 8 the clipping used in PPO is not present. Also since the PPO loss does not follow the CE loss structure of having a log p, the presence of Z in the reverse seems incongruous.
* Instead of more specific weaknesses for the work presented in the paper, I have questions for why certain approaches are not considered, or suggestions to strengthen the paper. I include these in the next section. The rest of this section I will point out minor typos and edits.
* On line 53, perhaps cite [1] as an example of ensembles being useful for better value prediction
* On like 54, perhaps cite [2] or some other paper for an example of normalization helping.
* Lines 68-70 seem like a generalization. A2C does not do advantage normalization
* The paper calls its solution the "symmetric RL loss" but the loss pertains to policy gradient, or even more specifically to actor-critic, methods. Perhaps call it the symmetric policy gradient loss?
* Line 177-178. The advantage function is slightly misconstrued here. The advantage function estimates how much better it would be to deterministically take action $a$ instead of following the current policy $\pi$.
* Line 178: " In the approach section" is an awkward phrasing. Consider "in the next section"
* Line 189: " ... also consider incorporates ..." is wrong grammatically. Perhaps drop "consider"
* Line 213 claims: `A highly engineered reward function is required to eliminate errors, ...`. A reference to back this claim up would be helpful.
* Line 215: "Has model errors", can be better expressed as "has estimation errors" or "approximation errors".
* Line 248: Perhaps the reference is meant for Equations 7 and 9, instead of 4 and 9?
* Line 399: Better to cite Bellemare et al. [3], for the arcade learning environment.
* Line 403 posits that the Atari environments only give rewards of 0 or 1. That does not seem to be correct. Is the paper focusing on a particular setup where the rewards are clipped?
* Line 457: Extra "is": `Note that the open-source GPT-J model "is" often outputs empty summarizations for most evaluation data`
* The table in Appendix 14 does not clarify whether the "performance increase" is perplexity or reward.
* Line 721, citation for Wang et al. can instead be for the published paper [4]
* Last few lines of page 15: it is helpful to review if you specify what operations you did at each step here.
References:
[1] Fujimoto, S., Hoof, H. and Meger, D., 2018, July. Addressing function approximation error in actor-critic methods. In International conference on machine learning (pp. 1587-1596). PMLR.
[2] Yue, Y., Lu, R., Kang, B., Song, S. and Huang, G., 2024. Understanding, predicting and better resolving Q-value divergence in offline-RL. Advances in Neural Information Processing Systems, 36.
[3] Bellemare, M.G., Naddaf, Y., Veness, J. and Bowling, M., 2013. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47, pp.253-279.
[4] Wang, Y., Ma, X., Chen, Z., Luo, Y., Yi, J. and Bailey, J., 2019. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 322-330).
Questions
* Can a distributional critic, like in QR-SAC [1], be an alternative to this symmetric loss? The distributional critic can better model noisy rewards, and as long as the noise does not obscure the signal, it should be able to estimate the right thing to do.
* Equation 7 looks a little bit like the policy gradient loss for the actions not taken as seen in expected policy gradients [2, 3]. There are obvious differences, but could you elaborate on why the reverse RL loss is different? Would expected policy gradients be something worth comparing to?
* For the RLHF tasks, is the reward model used for training the same one used for evaluation? That seems like it could be prone to overfitting.
* Appendix A.1. Is this analysis dependent on a softmax parameterization of the policy? It would be helpful for readers if that was made clear here.
* In table 11, Performance of SPPO with noise seems better than noiseless atari with or without SPPO on the following games: Centipede, Gopher, StarGunner, VideoPinball, WizardofWor. This result seems very surprising to me. Could you speculate on what the reason for this improved performance when the signal is obscured might be?
References:
[1] Wurman, P.R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T.J., Capobianco, R., Devlic, A., Eckert, F., Fuchs, F. and Gilpin, L., 2022. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature, 602(7896), pp.223-228.
[2] Ciosek, K. and Whiteson, S., 2018, April. Expected policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1).
[3] Allen, C., Asadi, K., Roderick, M., Mohamed, A.R., Konidaris, G. and Littman, M., 2017. Mean actor critic. arXiv preprint arXiv:1709.00503.