Summary
The paper addresses the novel problem of view generalization in reinforcement learning, where an RL agent is trained on an environment with a fixed view and then evaluated on a test environment having the exact same dynamics but observed from a different perspective. In order to address this issue, the authors introduce an innovative method that permits test-time adaptation to the new view.
The authors integrate a learnable spatial transformer network (STN) into the feature extraction component of the agent. This augmented encoder is then fine-tuned at test time to generate features that enable a frozen latent dynamic model to predict future state representations. An essential aspect of the authors' approach is that this adaptation does not necessitate any reward signal, which is a key advantage.
Experimental results demonstrate a significant reduction in the generalization gap presented by a new view across several heterogeneous benchmarks.
Strengths
The paper is well-structured and easy to follow, providing clear and detailed explanations that make believe that I could readily reproduce the method based on the descriptions provided.
The authors introduce a new problem in the field that could stimulate further research.
The proposed method's strength lies in its test-time adaptation capability, which eliminates the need for any reward signal, making it highly applicable to real-world scenarios. This characteristic also suggests that the approach could be integrated into any model-based method, enhancing its universality.
Another advantage is the apparent minimal interaction required with the testing environment, promoting efficiency.
The method is evaluated across various environments, with the results showcasing impressive improvements compared to non-adaptive methods.
The paper's ablation study effectively illustrates the contribution of each component of the method, highlighting the importance of every aspect in achieving the observed results. Additionally, the annex offers excellent visualizations demonstrating the impact of the spatial transformer on the feature map, providing intuitive understanding of the method's mechanics.
Weaknesses
Despite the many strengths of the paper, a few areas could benefit from further development and clarification.
1. It would be beneficial to see a comparison of the performance decrease relative to the original view. The lack of this data makes it challenging to truly gauge the significance of the generalization gap and the efficacy of the proposed method.
2. The scope of comparative studies is somewhat limited. The paper primarily compares with methods not designed for visual adaptation, which may not provide the most insightful comparison. It would have been advantageous to see how the proposed approach stacks up against other strategies specifically intended for visual adaptation.
3. An exploration of whether the proposed method can also benefit model-free algorithms such as Soft Actor-Critic (SAC) is missing. This could broaden the applicability of the findings and provide additional insights into the method's versatility.
4. It remains unclear why the proposed method underperforms with the Inverse Dynamics Model (IDM). A deeper exploration of this anomaly could bolster the robustness and reliability of the approach.
5. Finally, the similarity between the proposed method and the PAD method, which performs test-time domain adaptation in model-free RL, raises questions. The key differences are the use of a Spatial Transformer Network and the leveraging of the latent dynamics model of model-based methods instead of adding it as an auxiliary component. However, these differences, while noteworthy, do not necessarily represent a substantial departure from the PAD method. This similarity begs the question of whether the presented method offers significant novelty or if it is essentially an adjustment of existing approaches.
Questions
1. To better understand the importance of the generalization gap, could you provide performance data related to the original view? This information would help quantify the performance decrease when the agent is switched to a different view.
2. The use of inverse dynamics models with spatial transformer networks in your method isn't entirely clear.
Does IDM completely replace the DM in the TD-MPC algorithm?
Or does it just used as a side network during TD-MPC training and used for test adaptation only?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors mentioned a limitation of this work: the proposed method has not been tested in real-world scenarios, such as with physical robots, which may limit its immediate applicability.
Another potential limitation could be that the method has only been tested on model-based reinforcement learning (RL), whereas it would be relatively straightforward to evaluate its effectiveness in model-free RL.