Reply to reviewer Zre9
We thank the reviewer for engaging and asking clarifying questions. We provide additional clarification below.
**[Weakness 1 & 2] Then how does the current task know whether to use go-forward or go-backward predictive loss when the downstream task is using the current pre-trained model?**
Thank you for pointing this out. In short, we use a **one-hot task label** to modulate the video model for the desired task, when the video model is trained on diverse data.
We would like to clarify four points:
* In our experiments, the video model training data always includes the expert target task data, except for cross-embodiment generalization experiments.
* The video model can be trained on the target task-only data (Figure D). This is the single-task imitation learning setting.
* Using multi-task data can further improve the video model with **one-hot task labels**. During RL policy training, we condition the video model on the target task one-hot vector (Figure 3). This is the multi-task imitation learning setting.
* Training the video model with multi-task and multi-agent data enables cross-embodiment generalization (Figure 8 and Figure B). Here, we also use a one-hot task label to specify the target task.
Sections 4.1 and 4.3 elaborate the training details of our video model but we understand that it is not easy to parse these points. We will revise our paper to make the aforementioned points clear.
**[Weakness 1 & 2] Secondly, how can we ensure that the previous (large amount of) expert behaviors are relevant to the expert behavior of the current task?**
Similar to the previous answer, the training dataset includes the expert behavior of the current task, and we specify the target task by feeding a one-hot task label to the video model.
As can be seen in cross-embodiment experiments (Section 4.3) and OOD generalization analysis (Figure B), using data other than the current task helps generalization capability of our video model.
**[Q2 & Q3] Why isn't there an overestimation problem, which is common in offline RL?**
Thank you for raising this question. Offline RL suffers from the overestimation problem mainly due to **bootstrapping** from out-of-distribution actions. Here, only one high value prediction from out-of-distribution actions can be bootstrapped and affect the overall performance significantly.
As mentioned by Reviewer Zre9, VIPER also has the similar risk that an RL agent exploits out-of-distribution states with high log-probabilities. However, since VIPER measures a probability of a trajectory not a single state, even though one OOD state luckily gets a high reward, it is much less likely to happen throughout the entire trajectory. Thus, our RL agent can learn to follow the distribution of expert trajectories. We will include this discussion in our updated version of the paper.
**[Q2 & Q3] The authors state that "we find that VIPER assigns higher rewards to expert behavior on OOD tasks than to suboptimal and random behaviors on in-distribution tasks". Is it because the model is overfitting? If so, then the intrinsic rewards provided are very sparse.**
In Figure B, VIPER assigns higher rewards to expert behaviors on OOD tasks, which haven’t been seen during the video model training (it’s from OOD tasks). Thus, we can conclude that VIPER does not overfit to training data. Instead, it shows a strong generalization capability and thus, provides dense and informative rewards. Additionally, the trained video model achieves a test set loss similar to the train set loss; another indicator that the video model is not overfitting.
—
We would like to thank Reviewer Zre9 again for engaging in discussion and providing constructive feedback. We tried our best to address the reviewer’s concerns:
- Conducted additional experiments in Figure D to address the concern “The effectiveness of the proposed method may be limited if the expert data is scarce”. Figure D demonstrates VIPER achieves high performance even with only 10 expert videos (10% of the original dataset size), and learns a less efficient but still meaningful behavior even with only 1 expert video.
- Included additional baselines (GAIfO and BCO), which are shown in Figure A
- Performed additional analysis evaluating the OOD performance of VIPER (Figure B). Notably, we find that VIPER assigns higher rewards to expert behavior on OOD tasks than to suboptimal and random behaviors on in-distribution tasks. The reward histograms show that VIPER reflects different levels of task performance well, even out of distribution.
- Addressed additional concerns regarding VIPER achieving expert-level control and performance compared to baselines (Q4 & Q5).
Please let us know if you have any concerns and questions!