Summary
This paper proposes an approach to transfer and adapt robotic skills to unseen tasks. The method decomposes the ‘transfer gap’ between a source and target task into a sequence of intermediary tasks, and employ a differentiable simulator as a model by which to determine how to transform the actions along the sequence of intermediary tasks.
The paper addresses an interesting problem and the theoretical development is clear, but I have concerns about the assumptions and applicability of the problem setting, and also feel that the experimental justification is lacking.
Questions
**Problem setting and assumptions**
The proposed method assumes that there is a successful solution providing actions for the source task, and that the difference in the actions between two adjacent intermediary tasks is small throughout the sequence. The TransferStep algorithm adapts from one task to the next by iteratively (i) rolling out the current actions in the environment; (ii) computing the error in the object displacement; (iii) updating the actions by propagating gradients through the rollout.
Assuming I have understood this correctly, I have a few questions and comments:
1) How robust is the approach to different kinds of transfer gap? Transferring from a pushing task to another (or opening/closing different amounts, or moving an object to a different configuration) may be reasonable, but what about more challenging cases, where the behaviour or affordance might change, or the task might reuse multiple existing skills? Eg. placing an object → inserting an object somewhere else, reaching and grasping an object → lifting an object.
2) If the method is not intended to transfer to different behaviours (only different object displacements), why not just perform planning on the target task directly? TransferStep itself appears to be a form of trajectory optimisation at each step, and at least for the tasks presented in the experiments (which are all relatively simple single-stage manipulation tasks with the object pose changing), I would expect both source and target task to be solvable by planning with the differentiable simulator.
**Experiments**
The authors claim a “rigorous experimental framework” and an “exhaustive evaluation”, but the only evaluation is with 4 source/target tasks and weaker baselines (see my comment below), with a single ablation for the different planning methods used to determine intermediate goals.
The comparison to baselines is not compelling in my opinion, as I would expect them to not succeed in the applied setting.
For example, MAML is a meta-learning approach that learns a good parameter initialisation for adaptation by learning-to-adapt on a very large distribution of tasks. As such, I don’t find it surprising that it fails when training on 4 source tasks (I would expect it to only be useful here if trained on many variants of the same task and adapted to another variant). It feels like the wrong fit for a baseline, and something like finetuning via RL or planning would make more sense to me.
Similarly the direct transfer approach applies the same TransferStep algorithm but without the intermediary goals - as such it is more of an ablation without the planning stage.
**Minor comments**
Figure 1 is confusing. I think it means “to select the best candidate for the (i+1)th subtask…”. The current wording makes it seem like the output of the q function is something else, which ‘generates’ the task.
**Summary**
While I was intrigued by the theoretical development, I think there unfortunately needs to be more evidence that this method could be useful for challenging tasks, along with more clarity around why the proposed problem setting and solution could be useful versus eg. planning in the target domain.