Incremental Learning of Retrievable Skills For Efficient Continual Task Adaptation

Summary

This paper tackles the problem of continual imitation learning, where an agent needs to continually adapt to new tasks through imitation learning. The paper proposed IsCiL, a method that utilizes prototype-based skill incremental learning to gradually grow a repository of skill prototypes that can be retrieved and adapted to new tasks. Evaluated on Franka-kitchen and meta world, IsCil showed superior performance over previous CiL methods.

Strengths

The problem of continual imitation learning is very important. The overall idea of utilizing skills to achieve continual imitation learning is promising. The experimental results of IsCiL are well-presented, and thoroughly conducted.

Weaknesses

Overall the contribution seems incremental. The idea of prototype-based skill learning, parameter-efficient adaptation via LoRA, as well as the idea of tackling lifelong learning through skills have already been explored by previous works, and it is not clear what the key innovation of this work is. Some of the important concepts in this paper, such as skill prototype, base, and skill adaptor, are not properly defined, making it hard to comprehend how they are generated and used. For example, how exactly are the pre-trained base model and the adaptor weights combined? Where does the pre-trained base model come from? How are the skill prototypes initilized? This paper assumes access to a set of subgoals. In practice, this seems like a strong assumption, as these subgoals essentially segment the demonstration trajectories and implicitly define a set of short-horizon skills. Considering that these sub-goals are also used for generating and selecting skill prototypes, it raises the concern that these sub-goal specifications are doing the heavy lifting.

Questions

Although not necessary, I’d be interested to see the performance of IsCiL on more challenging benchmarks such as LIBERO. Also, given that the metrics measured by this work are intuitively visible from the training curve, it would be great to show some training curves of IsCiL and baseline methods. Minor questions: - How many skills are updated for each demonstration transition? It seems that only the retrieved skill is updated for each transition. Wouldn’t this be an inefficient use of data? - How exactly does IsCiL determine when to add new skills? Do we need a human to manually identify new skills?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The limitations is adequately addressed.

Summary

The paper presents IsCiL, an approach to continual learning that addresses the limitations of knowledge sharing in traditional Continual Imitation Learning methods. IsCiL uses a prototype-based skill incremental method where each skill is represented by prototype embeddings and skill adapter parameters for LoRA adaptation. The state encoder encodes observations and subgoals into state embeddings, while the skill retriever matches these to existing skill prototypes during inference. The method is evaluated in environments like Franka-Kitchen and Meta-World, demonstrating its ability to learn and adapt without needing complete expert demonstrations. The results show IsCiL's performance in task adaptation and its ability to perform task unlearning for privacy concerns.

Strengths

The paper introduces a novel prototype-based skill retrieval mechanism that effectively learns skill prototypes and adapters. The proposed method is evaluated in environments like Franka-Kitchen and Meta-World, showing improvements over TAIL when knowledge sharing across demonstrations is important (semi and incomplete settings). While skill adapters and prototypes themselves are not particularly novel, the authors' contribution lies in the innovative method of continual learning through on-the-fly adapter retrieval. The authors additionally show that the method can enable skill unlearning without significant degradation in performance. The paper is well-structured, with clear explanations of the state encoder, skill retriever, and skill decoder components.

Weaknesses

1. The retrieval and adaptation processes at every time step might lead to increased inference time and resources, which is not thoroughly analyzed in the paper. 2. Handling overlapping skills appears to not be handled or need manual intervention to unlearn, which is not ideal for maintaining performance across an increasing number of tasks. 3. The paper lacks a detailed analysis of the computational overheads and scalability issues associated with maintaining a prototype-based memory and multiple adapters, which is important for real-world applications. 4. A major assumption in the paper is the high costs and inefficiencies associated with comprehensive expert demonstrations. It is not well motivated why obtaining a single comprehensive demonstration would be more cost-effective than multiple incomplete demonstrations. This should be further motivated.

Questions

1. Does retrieval and adaptation occur at every time step? How does this affect inference time and compute demands? 2. The method seems to require manual selection of the task identifier for task unlearning. How does the method handle conflicting skills that need to override or combine with existing prototypes? How are skills consolidated as the number and complexity of skills learns grows? 3. Can the unlearning process be applied at different granularities (e.g., specific sub-tasks or stages within a task), or is it only applicable at the (sub-)task level?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors do address limitations but could include more transparency on the inference time and resource use, which the paper does not thoroughly analyze. Secondly, handling a large amount of skill prototypes without additional consolidation.

Summary

The paper introduces a new adapter-based method for continual imitation learning that avoids episodic replay and exhibits better forward and backward transfer and overall performance as compared to prior work. The authors compare their method against baselines on a couple of simulation benchmarks and also provide ablation studies to motivate their design choices.

Strengths

- The paper addresses an important problem of doing continual learning without having to store all the data seen in the past. - The method is described in detail with experiments provided on a variety of baselines. IsCIL seems to outperform baselines on both simulated benchmarks. - The authors include ablation studies in the paper to promote their design choices.

Weaknesses

- The paper is a little hard to follow in certain parts. Figure 1 could be made clearer. - The method assumes access to datasets labeled with sub-goals. This must be added to the limitations. - Line 140 mentions that a fixed function f is used to encode the observation and goal into a state embedding. Is this fixed function either obtained from the pretraining phase or is a pre-trained encoder of some sort? How does this encoder deal with changes in a non-stationary environment that it might not have been trained on? - It might be useful to also evaluate IsCiL on the LIBERO benchmark which is developed for such continual learning studies and also provides human-collected demonstrations. This would help highlight the efficacy of the proposed method further. - Is the choice of using a diffusion base policy for a specific reason? - The exact lifelong setting for experiments in Table 1 is unclear. From what I understand, the base model is pre-trained on a subset of tasks/objects and new tasks/objects are introduced during the lifelong learning stage. Assuming the table reports multitask performance for all baselines, this raises two questions - (1) Since FWT and BWT are only reported with a single value, does this mean that the training is only done in two stages - pretraining with limited objects and all new objects introduced together? In case the new objects are introduced incrementally, should these numbers be computed for each stage where a new object/task is introduced? (2) How does varying the task order and the initial pretraining set of tasks/objects affect the final performance? - Lines 233-234 mention that IsCiL exhibits performance ranging between 84.5% and 97.2%. However, in Table 1, I do not see any number as high as 97% and I can see IsCiL performance as low as 68.9% in certain settings. Also, IsCiL does not seem to be “surpassing the oracle baseline for Multi-task learning” as mentioned for any of these cases. Some clarification about how to interpret these results would be helpful.

Questions

It would be great if the authors could address the “Weaknesses” listed above. Also, is there an expanded version of the name of the method - IsCiL? I am willing to increase my score once these questions have been addressed.

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors have addressed the limitations. I have suggested adding the requirement of sub-goal labeled datasets as a limitation under “Weaknesses”.

Summary

Learn a two-layer hierarchy from a sequence of datasets, where the low-level skills are represented by a discrete set of prototypes: vectors that can be mapped to repeated patterns of actions represented by basis functions. The basis function parameters are then passed into a decoder function which takes actions based on the observation and goal. The prototypes are recovered by performing k-means to cluster the data of a particular skill. The frequency of skill selection overall is a score, and the decoder is trained with imitation learning.

Strengths

Introduces a complex but clearly effective system for skill learning. Shows good results in an important setting of imitation learning from multiple datasets. Demonstrates unlearning capability, which is useful in some contexts.

Weaknesses

It is not obvious how the skills might be entangled together post-hoc, since the reusability of a skill across tasks might make its unlearning impossible. The experimental results also seem cherry-picked to ensure that this is not an issue, which is probably disingenuous to the actual cause of privacy: whether a component be relearned without any of the information from a particular source. The experiments appear to be convincing only in the semi and incomplete settings, but it is not entirely clear what the semi or incomplete settings are. Without a clear picture of how these components are defined, it is not clear whether the empirical results actually support the claims made in the introduction.

Questions

Why is Continual Imitation Learning abbreviated CiL? It seems like it should be CIL. Are there clear ablations on how the many components contribute to the overall performance? What metric can be used to evaluate the unlearning capability in the context of privacy? Can this be used to ensure particular data is not used? Was this evaluated?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

see above.

Thank you for the rebuttal

I thank the authors for the detailed clarifications and additional experiments. Taking the rebuttal into account, I am raising my score to 6.

Response to Authors

I appreciate the clarifications and believe that the additions will strengthen the paper. I am happy to raise my score.

Thank you for the comments

Thank you for your comments and for providing the additional analysis and clarifications on efficiency and scalability. As a result, I have increased my score.