Summary
This paper introduces an original approach named Stochastic Gradient Feature Decorrelation (SGFD), which aims to amplify the generalization capabilities of reinforcement learning (RL) agents across various environmental variations. These variations can encompass task-irrelevant visual attributes such as backgrounds, as well as task-related factors like physical configurations. The authors propose achieving this through a decorrelation of features, executed via a resampling technique intended to minimize the Frobenius norm of the cross-covariance matrix, derived from Random Fourier features. Recognizing the inherent challenges of complete decorrelation, the authors shift focus towards effectively decorrelating the most variable features. This is facilitated by leveraging the saliencies from an environment classification model, which, under ideal circumstances, makes its decisions based on distinct features that are not common to different environments.
Strengths
The paper is commendably well-written and coherent, effectively explaining complex ideas in an accessible manner. The authors demonstrate a strong theoretical grounding, with well-motivated intuitions supporting their methodology. Their saliency-guided optimization is an interesting approach, backed by a robust ablation study. SGFD successfully tackles generalization issues relating to both task-irrelevant and task-relevant features, demonstrating a broad scope of applicability. The proposed method do present noticeable enhancements in generalization performance, more so in the case of task-relevant features.
Weaknesses
Despite the paper's strengths, there are some areas where it could be improved. Firstly, the methodology requires several environments with variations to train their environment classifier, introducing an element of manual supervision into the learning process, which may not be ideal in all scenarios. Furthermore, the full algorithm can be challenging to comprehend without first referring to Appendix A, suggesting that the main body of the text might benefit from additional clarification. It might also be beneficial to introduce the general objective - namely, the reweighting of the batch sampled from the buffer - earlier in the paper to give readers a clearer understanding of the process. Lastly, while the authors promote their method as an improvement over Soft Actor-Critic (SAC), the achieved results actually utilize the same updates for the encoder as employed by Adaptive Meta-learner of Behavioral Similarities (AMBS). This could lead to misunderstandings, as readers may infer that the saliency-guided resampling alone yields these performances. It would therefore be beneficial for clarity and fairness if the authors explicitly acknowledged this in the experimental section.
Questions
* Could the authors elaborate on the inference procedure employed during testing? This could help elucidate the practical applicability of the methodology.
* Does the encoder also benefit from the gradients produced through resampling? Understanding this aspect could contribute to a more comprehensive understanding of the process.
* Given the necessity to decorrelate features that vary across environments from those that remain constant — and assuming that the variable features have a higher saliency — it appears that the $p\left(\mathbf{Z}_i\right) p\left(\mathbf{Z}_j\right)$ in equation 7 tends to assign more importance to tuples of varying features. Would it potentially be more efficient to replace it with $|p\left(\mathbf{Z}_i\right) - p\left(\mathbf{Z}_j\right)|$to accentuate the decorrelation between the varying and consistent features?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
While the authors have reasonably addressed the methodological limitations of their approach, they assume no potential negative societal impacts stemming from their method.