Summary
The paper introduces a novel black-box tuning approach, focusing on optimizing prompts for the Decision Transformer (DT) within the context of offline meta-reinforcement learning. Drawing inspiration from prompt learning methodologies prevalent in Natural Language Processing (NLP), the strategy involves fine-tuning the prompt segment specifically. This fine-tuning utilizes task-specific offline data, typically scarce compared to conventional training datasets, aiming to achieve specific goals: minimizing behavior cloning loss in offline settings or maximizing returns in online contexts. Even when the tuning process is confined to the minimal parameters essential for prompt adjustment, the proposed method demonstrates performance on par with comprehensive model fine-tuning in scenarios with limited data availability (few-shot settings).
Strengths
**Strength 1: Plausible Application of Prompt-Tuning in RL**
The introduction of prompt-tuning to the realm of Reinforcement Learning (RL) and specifically its integration with the Decision Transformer stands out as a formidable strength of this work. The use of a black-box tuning methodology, coupled with a ranking optimization framework for prompt tuning, is particularly noteworthy. This approach avoids the necessity for a comprehensive, end-to-end gradient update of the entire model, presenting a novel, convincing, and computationally efficient strategy. Also the empirical result with random prompt initialization suffices to prove the efficacy of the proposed method.
**Strength 2: Comprehensive Ablation Studies**
The paper presents thorough and insightful ablation studies, offering a detailed analysis that contributes significantly to understanding the proposed method's effectiveness. A standout aspect is the content presented in Figure 5 of the appendix, which emphasizes the critical role and impact of the prompt-tuning process.
Weaknesses
**Weakness 1: Need for Greater Clarity in Problem Formulation**
This work delves into a crucial application within the domain of Reinforcement Learning (RL), yet there is room for enhancing the clarity of the problem formulation, specifically concerning the concepts of 'meta' and 'offline' in the context of offline meta-RL.
- **Meta Aspect**: The manuscript appears to presuppose the availability of an appropriate, task-specific prompt at the outset of each task. However, the essence of meta-RL traditionally entails a mechanism to infer the dynamics of the current task [1]. This distinction is crucial, as the described setup leans more toward a 'multi-task setup,' assuming a known index for every task (May be the random prompt initialization does this work?). While prior works such as MACAW [2] and Prompt-DT [3] also frame their contributions in the context of 'meta-learning,' there seems to be a slight deviation from standard expectations. There's a risk of misinterpretation, potentially leading readers to assume that the prompt evolves independently from a task-independent prompt, given an unidentified task index.
- **Offline Aspect**: The scope of the research extends beyond purely offline methodologies, especially considering the inclusion of reward-based tuning as seen in Eq. 7. Consequently, it would be appropriate to broaden the scope of the related work section to encompass studies that bridge both online and offline setups, including references such as [4,5]. While the current method maintains considerable benefits due to its limited use of steps and focus on prompt tuning, acknowledging and discussing related works would help situate this study within the broader scholarly discourse. It is essential for readers to understand the specific advantages and novel positioning of this work compared to these related methodologies.
**Weakness 2: Enhancements Needed in Empirical Validation**
While this research primarily adopts an empirical approach, there are areas where the robustness of empirical validation could be fortified. The absence of source code is a noticeable limitation that adds a layer of complexity in grasping the finer nuances of the implementation, particularly the operational specifics of the prompt training and initialization. The details in the appendix could benefit from further expansion to alleviate this concern.
Moreover, in the realm of meta-reinforcement learning, the consistency and reliability of results are paramount. The current practice of utilizing three random seeds for experiments, although commonly accepted, may not suffice to capture the full variability and confirm the stability of the proposed method. It is advisable for the authors to consider expanding the number of random seeds to reinforce the statistical significance of the results, enhancing confidence in the findings' replicability and robustness.
Also there are some questions regarding the experiments (see the Questions below)
(Minor Points)
In Equation 1, maybe $a_t$ should be removed.
For Figure 3, increasing the font size would greatly enhance readability and comprehension.
Certain notational choices in the paper could benefit from reconsideration to enhance clarity. For instance, using "\mu" to signify the standard deviation may lead to confusion, as this symbol often denotes the mean or a behavior policy in statistical contexts. Additionally, the use of "||" for concatenation in Equation 5 may be misleading, given its common interpretation as a norm notation in mathematical literature.
[1] Beck et al., A Survey of Meta-Reinforcement Learning, 2023.
[2] Mitchell et al., Offline Meta-Reinforcement Learning with Advantage Weighting, 2021.
[3] Xu et al., Prompting Decision Transformer for Few-Shot Policy Generalization, 2022.
[4] Lee et al., Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble, 2021.
[5] Zheng et al., Online Decision Transformer, 2022.
Questions
Question 1:
The observation that PTDT-online does not surpass its offline counterpart, despite utilizing additional online samples, raises questions regarding the underlying factors. Could the discrepancy in performance be attributed to the distinct objectives outlined in equations 6 and 7? If this is the case, the rationale behind proposing PTDT-online warrants further clarification, as one might expect the incorporation of online data to enhance, rather than diminish, the model's efficacy.
Question 2:
Table 1 presents an intriguing scenario where PTDT-offline outperforms PDT with an expert prompt in nearly all environments, with the exception of cheetah-vel. This outcome prompts several questions. Is it possible that PTDT, even when initiated with a random prompt, evolves to generate a prompt on par with or superior to that constructed by an expert? Alternatively, could this suggest that the supposed advantage of an expert prompt does not universally translate across all environments? Understanding the dynamics that led to these results could provide valuable insights into the adaptability and utility of prompts in varying contexts.
Rating
5: marginally below the acceptance threshold
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.