We present a new algorithm, Cross-Episodic Curriculum (CEC), to boost the learning efficiency and generalization of Transformer agents. Central to CEC is the placement of cross-episodic experiences into a Transformer's context, which forms the basis of a curriculum. By sequentially structuring online learning trials and mixed-quality demonstrations, CEC constructs curricula that encapsulate learning progression and proficiency increase across episodes. Such synergy combined with the potent pattern recognition capabilities of Transformer models delivers a powerful cross-episodic attention mechanism. The effectiveness of CEC is demonstrated under two representative scenarios: one involving multi-task reinforcement learning with discrete control, such as in DeepMind Lab, where the curriculum captures the learning progression in both individual and progressively complex settings; and the other involving imitation learning with mixed-quality data for continuous control, as seen in RoboMimic, where the curriculum captures the improvement in demonstrators' expertise. In all instances, policies resulting from CEC exhibit superior performance and strong generalization. Code is open-sourced at https://cec-agent.github.io/ to facilitate research on Transformer agent learning.
Paper
Similar papers
Peer review
Summary
This paper presents Cross-Episodic Curriculum (CEC) to boost the learning efficiency and generalization of Transformer agents. Specifically, CEC places the cross-episodic experiences into a Transformer’s context, which forms the basis of a curriculum. The authors also provide three concrete curriculum implementations: Learning-progress-based curriculum, Task-difficulty-based curriculum and Expertise-based curriculum. Experiments on discrete control, such as in DeepMind Lab, and continuous control, as seen in RoboMimic, show the superior performance and strong generalization of CEC.
Strengths
1. Simple but effective idea for Cross-Episodic Curriculum. 2. Three concrete curriculum implementations for different settings (RL or IL). 3. Good paper writting. It is easy to follow the methods and experiments.
Weaknesses
1. My major concern is the baselines used in the experiments (and also the related works). Specifically, the authors only compare the Transformer-based BC agent, which is a weak baseline. The reviewer will encourage the authors to compare with more recent methods like Algorithm Distillation [1] and TIT [2], or discuss the reason why these methods are not suitable for comparision. (The reviewer guesses that BCQ and CQL are based on MLP?) 2. Some important details are missed in the experiments. For example, in the ablation study, the reviewer (and most readers I guess) would like to know how to implement the "without cross-episodic attention" and "Curriculum granularity", but the authors put these important information in the Appendix. [1] In-context Reinforcement Learning with Algorithm Distillation. ICLR 2023. [2] Transformer in Transformer as Backbone for Deep Reinforcement Learning. 2022.
Questions
1. Could the authors compare with more recent methods like Algorithm Distillation [1] and TIT [2], or discuss the reason why these methods are not suitable for comparision? 2. The key insight of CEC is that sequential cross-episodic data manifest useful learning signals that do not easily appear in any separated training episodes. Besides the better performance, could the authors provide more evidence for this insight? 3. Why Learning-progress-based curriculum is worse thanTask-difficulty-based curriculum in most evaluation environments?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
It will be better to discuss which curriculum is suitable for which environment setting?
Summary
This work proposes a new method, CEC, to boost the learning efficiency and generalization capability of the agent by structuring multiple episodes for deploying the transformer’s pattern, and sequence recognition capability. The proposed method shows improved performance compared to the baseline and shows generalization capability.
Strengths
1. The authors propose a new approach for the transformer-based agent that leverages the information from the various types of trajectories by deploying the representation capability of the transformer architecture. 2. The proposed method shows knowledge distillation capability through several evaluation and generalization experiments quantitatively.
Questions
1. If we have to manually set a pair of trajectories and their corresponding curriculum levels, it means we know the quality or progress, or expertise of the obtained trajectories in the online RL process (e.g. Table 1, Eq (1)). I think this is a quite strong assumption or extra burden since the RL framework itself should enable the agent to automatically learn how to solve the problem without external intervention. It is not clear whether given prior curriculum sequence is a fair assumption. 2. In line 107, it is not clear the meaning of the notation “n” and “t”. if “n” is the number of entire episodes with various types and “t” is the timestep, then does the transformer not refer to the full horizon of the episode when computing the log-likelihood of the policy? 3. Also, I was wondering how the transformer distinguishes each trajectory’s level or expertise. I think the transformer cannot distinguish it without a learnable token or expertise embedding which works similarly to the position embedding in NLP. If we assume the level is given for embedding, it would be burdensome to collect the accurate level for each trajectory in practice. If the transformer is implemented without these considerations, how can we ensure the transformer leverage the information from various level of trajectories? I think it would be proper to compare with the transformer-based offline RL models such as Decision Transformer [1] or other recently proposed SOTA models to check whether the proposed method just memorizes every sequence pattern in the trajectories or infers something helpful from the curricular data. [1] Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." *Advances in neural information processing systems* 34 (2021): 15084-15097.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
The authors do not discuss the potential negative societal impact of their work and do not explicitly discuss the limitations of their method in the manuscript. I do not anticipate ethical issues.
Follow-up on our response
Dear reviewer, As the discussion stage is ending soon, we wonder if our response answers your questions and our extra experiments address your concerns? If yes, would you kindly consider raising the score? Thanks again for your very constructive and insightful feedback!
Summary
This paper introduces a novel algorithm, referred to as Cross-Episodic Curriculum (CEC), which aims to improve the learning efficiency and generalization capabilities of Transformer agents in multi-task RL settings. The algorithm has been specifically developed to exploit the limited availability of sub-optimal data in environments with a scarcity of data, such as robot learning. The researchers examine the efficacy of Cooperative Evolutionary Computation (CEC) in two distinct and representative scenarios: online RL using 3D maze environments in DeepMind Lab, and imitation learning from human demonstrations of varying quality in RoboMimic. The findings indicate that visuomotor policies, which were trained using the expertise-based curriculum, exhibit the capability to surpass established baselines and achieve superior performance on simulated robotic manipulation tasks. Furthermore, these policies demonstrate significantly better performance compared to agents trained using offline RL algorithms. The researchers reach the conclusion that Cross-Entropy Method (CEC) offers a potentially effective strategy for leveraging restricted yet sub-optimal data in contexts characterized by a scarcity of data, such as robot learning.
Strengths
1. The authors conduct an empirical assessment to examine the efficacy of CEC in two distinct and representative scenarios: online reinforcement learning using 3D maze environments from DeepMind Lab, and imitation learning from human demonstrations of varying quality in RoboMimic. 2. Comparative analysis: The findings indicate that visuomotor policies, which were trained using the expertise-based curriculum, exhibit the capability to equal or surpass established baselines in simulated robotic manipulation tasks. Furthermore, these policies demonstrate significantly superior performance compared to agents trained using offline reinforcement learning algorithms. 3. Extensive applicability: The researchers reach the conclusion that CEC (Contextual Embedding Clustering) presents a promising strategy for leveraging limited yet sub-optimal data in scenarios characterized by a scarcity of data, such as robot learning. Moreover, they assert that CEC is both effective and widely applicable in diverse problem contexts. 4. The open-source code pertaining to the algorithm is provided as supplementary materials and will be publicly accessible to facilitate further research on the learning of Transformer agents.
Weaknesses
1. Restricted comparison: The study presents a comparison between the proposed algorithm and a small number of established baselines. However, it does not encompass a comprehensive range of contemporary algorithms, thereby constraining the breadth of the comparison(e.g.[1], [2]). 2. Inadequate consideration of limitations: The manuscript fails to adequately address the limitations associated with the proposed algorithm or the conducted experiments. This omission may impede the comprehension of the potential drawbacks and challenges that may arise when employing the algorithm in real-world scenarios. 3. How does the proposed algorithm scale to larger and more complex environments? The paper only considers relatively small and simple environments, and it is unclear how well the algorithm would perform in larger and more complex environments, such as those encountered in real-world robotics applications. [1] Piotr Mirowsk, Razvan Pascanu, Fabio Viola, Learning to Navigate in Complex Environments. [2] Siyuan Li, Rui Wang, Minxue Tang, Chongjie Zhang, Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards.
Questions
1. What distinguishes CEC from hierarchical RL? 2. The experimental results presented in section 4 raise two inquiries: (1) What are the reasons behind the comparatively inferior performance of CEC in comparison to vanilla RL when applied to relatively uncomplicated tasks? (2) What factors contribute to the subpar performance of Learning progress? 3. The fundamental concept of CEC, as I comprehend it, is to facilitate the exchange of experiences derived from diverse environments, tasks, and episodes in order to enhance the performance of RL. The aforementioned viewpoint aligns with the fundamental principles of various curriculum RL approaches. However, extensive research has demonstrated that measuring learning progress is a highly efficacious metric. In contrast, the findings presented in this article deviate from this prevailing notion. The authors do not expound upon the specific definition of learning progress, which may be a crucial factor and potentially the primary underlying cause of Q.3. 4. The experimental control group setting described earlier in this paper is deemed insufficient. In addition, it is requested that the authors incorporate a comparative analysis of the curriculum pertaining to environmental change, e.g. PAIRED (Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design).
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors do not discuss the limitations.
Follow-up on our response
Dear reviewer, As the discussion stage is ending soon, we wonder if our response answers your questions and our extra experiments address your concerns? If yes, would you kindly consider raising the score? Thanks again for your very constructive and insightful feedback!
Summary
This work aims to study mechanisms of cross-episode attention to effectively learn to improve polices by training on contexts containing gradually improving trajectories.
Strengths
On the whole, this paper is well written. The topic of transformers in-context learning as an approach to planing and RL is of increasing importance. This work provides another important datapoint in that space for a technique which has scientific value to try.
Weaknesses
The main weakness with this work is its lack of comparison to existing approaches in the space. The empirical results are only compared against approaches that are not intending to take advantage of in-context learning. This looses most of the scientific value of the work when several other prominent works have used the in-context learning capabilities of transformers in various ways to great success. It is known that in-context learning can help with these sorts of tasks, the question which this paper is in the position to answer, is if this way of applying in-context learning works better or worse than the others which have been tried. Even if this approach is importantly distinct from those, it is important to compare against the other natural approaches that have already been tried, such as Algorithmic Distillation or AdA, to understand if this approach to applying in-context learning works better or worse than other approaches. On a similar note, the paper is often vague about how their approach is different from other approaches in the literature. Specifically the distinction between "test time meta-learning" and "generalization across varying test configurations in each episode" is difficult to understand.
Questions
How does your RL approach in the DM labs differ from the general approach from Adaptive Agents Team et al?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
3 good
Limitations
The paper could include more discussion of the feasibility of getting data with the requisite ordering of poor data first, better data next, best data last. It could also be useful for the paper to include some mention of the limitations of the imitation paradigm.
Follow-up on our response
Dear reviewer, As the discussion stage is ending soon, we wonder if our response answers your questions and our extra experiments address your concerns? If yes, would you kindly consider raising the score? Thanks again for your very constructive and insightful feedback!
Thanks for the feedback
I appreciate the authors' feedback. It addresses most of my concerns. Although I like the ideas proposed in this paper, it is more important to figure out which scenarios are these methods/curriculums applicable to. Currently, the authors only give a general description for this question. So I would keep my score.
Dear reviewer, We are glad to know that our reply addressed your concerns. Due to the length limit, we discussed at a high level the last question about which curriculum is suitable for which task setting. In this reply, we will provide more detailed explanations, and welcome your suggestions and feedback on new experiments that can address your remaining concern. In this work, we introduced three curricula: Learning-progress-based curriculum (L), Task-difficulty-based curriculum (T), and Expertise-based curriculum (E). L and T are particularly suitable for RL, while E is suitable for IL. In RL, as what we did in DMLab tasks Goal Maze and Watermaze, we recommend starting with L. Agents resulting from this curriculum learn better visuomotor policies that outperform the DT baseline as well as the BC baseline trained on expert data. However, if the task itself is too challenging such that source algorithms barely make progress, we recommend T. This is justified by our experiment on the DMLab task Irreversible Path. Concretely, the RL agent directly trained on the test difficulty completely failed, which results in the unsatisfactory performance of the agent trained with the L curriculum. However, the agent resulting from the T curriculum learns critical skills from easier settings and applies them to the hardest setting, and hence significantly outperforms the L-curriculum variant, both DT and AT baselines, as well as RL oracles. This is further qualitatively investigated and justified by rollout visualization provided [here](https://tinyurl.com/yypwytt). Feasibility of obtaining each curriculum is extensively discussed in [global response](https://tinyurl.com/4bc2469z). Furthermore, we would like to highlight that similar to many established works that aim for developing novel algorithms for learning visuomotor policies, our discussion on suitability is neither too narrow to lose generality, nor too vague to be hardly practical. For example, authors of [1] also recommend their new method at an abstraction level that is similar to ours, as quoted below. > Recommendations. In general, we recommend starting with the CNN-based diffusion policy implementation as the first attempt at a new task. If performance is low due to task complexity or high-rate action changes, then the Time-series Diffusion Transformer formulation can be used to potentially improve performance at the cost of additional tuning. Nevertheless, we agree with the reviewer that it is important to figure out which scenarios these curricula are applicable to. Since our investigation in RL setting is already comprehensive, **we would like to further investigate the IL setting by also applying the learning-progress-based curriculum on RoboMimic tasks**, due to the fact that it also provides trajectory data generated by RL algorithms with labeled reward. We will update with new results once they are ready. Meanwhile, we welcome the reviewer to provide suggestions and feedback on new experiments that we can further conduct to fully eliminate the last piece of concern. **References** - [1] Chi et al., Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023.
Dear reviewer, Following your suggestion, we further investigate the suitability of different curricula under IL setting (detailed discussion in RL setting can be found in the previous reply). We conduct new experiments to apply learning-progress-based curriculum to RoboMimic tasks considered in this work. Results are summarized in the table below. We control the total number of training trajectories to be the same used in expertise-based-curriculum (i.e., 270 trajectories for each task). All other factors are also identical. Training data are subset of machine-generated dataset provided by RoboMimic [1]. | | Expertise-Based Curriculum | Learning-Progress-Based Curriculum | CQL | |:---:|:---:|:---:|:---:| | Lift | 100.0 ± 0.0 | 32.0 ± 17.0 | 2.7 ± 0.9 | | Can | 100.0 ± 0.0 | 30.0 ± 2.8 | 0.0 ± 0.0 | | Average | **100.0** | 31.0 | 1.4 | We would like to highlight two findings. - First, to complete your curiosity about curriculum's suitability for different environments in IL setting, if human demonstrations are available, even if they are generated to be heterogeneous in quality, we would still recommend to use expertise-based curriculum. This is well supported by its significantly better performance over learning-progress-based curriculum. - However, our learning-progress-based curriculum is not completely without merits. In the absence of expert demonstrations and only with the access to machine-generated data, we would still recommend our learning-progress-based curriculum because it achieves non-trivial performance and significantly outperforms offline RL methods such as CQL [2]. More importantly, the reported CQL is trained on full machine-generated dataset provided by RoboMimic, i.e., 1500 trajectories for task Lift and 3900 trajectories for task Can. Please don't feel hesitate should you have further questions. If we have fully addressed your concerns, would you kindly consider raising the score? Thanks again for your very constructive and insightful feedback! **References** - [1] Mandlekar et al., What Matters in Learning from Offline Human Demonstrations for Robot Manipulation, CoRL 2021. - [2] Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning, NeurIPS 2020.
I appreciate your response. After checking the other reviews and responses, the clarifications have made some points clearer to me. However, I maintain my initial score as my overall evaluation of strengths and weaknesses remains consistent.
Acknowledgement of the rebuttal
Thank you for your response, which addressed my concerns. These clarifications should be included in the next version. I will raise my score to 6.
Thank you for the response. While additional results attached in the pdf are informative, the major concern (first question) is still not alleviated. I read the global response and the author's response is about the feasibility of the data with the curriculum. But my concern is given "prior curriculum sequence" itself. As mentioned in my question, in online RL, the agent should learn how to solve the problem without external intervention. If we have access to the pair of trajectories and their corresponding curriculum levels, then it would be appropriate to experiment with imitation learning and offline RL setting where an assumption of the pre-collected dataset is valid. Thus, I would decide to keep the score.
Dear reviewer, Thank you for your reply and explanation of your concern. We believe that there exists a misunderstanding of our work’s setting. We apologize for that and would like to clarify. Regarding our RL experiments on DMLab, similar to previous work [1], training data are collected by source RL agents during their **online** learning. Once the dataset is obtained, our method is trained **offline** in a purely supervised manner. We notice that this setting is common [1-4] and the assumption of pre-collected datasets is mild. We indeed apologize that this subtlety is not clearly explained in our initial submission and may lead to misperception of our work. We devote ourselves to clarify in our final version. However, we would like to highlight that this should not diminish the merits of our work. Please don't hesitate to let us know if you have further questions. **References** - [1] Laskin et al., In-context Reinforcement Learning with Algorithm Distillation, ICLR 2022. - [2] Chen et al., Decision Transformer: Reinforcement Learning via Sequence Modeling, NeurIPS 2021. - [3] Reed et al., A Generalist Agent, TMLR 2022. - [4] Lee et al., Supervised Pretraining Can Learn In-Context Reinforcement Learning, Workshop on New Frontiers in Learning, Control, and Dynamical Systems, ICML 2023.
Response to Authors
Thank you for your rebuttal. You have partially addressed my concerns on the novelty of this work. However, it is still problematic that this is largely written as if the in-context learning ability of transformers has not been used in RL when there have been many notable successful applications. As such I find it difficult to raise my score above a 5 (borderline accept) without a more narrow and clear framing of the contributions relative to other methods of achieving in-context learning in RL.
Decision
Accept (poster)