Vision Language Models are In-Context Value Learners

Dear reviewer nm81, Thank you for your thoughtful comments and feedback! Here, we respond to specific questions and concerns the reviewer raises. --------------------------------------- **Question/Comment 1**: Method’s applicability to online RL. **Response 1**: Yes, GVL is applicable to online RL. Many commonly used online RL algorithms, such as SAC [1], maintain a replay buffer for trajectories the policy has encountered during exploration. Recent practical methods for online RL also bootstrap from offline datasets (e.g., [2, 3]). In all these settings, GVL can be applied the same way. However, we do note that in a pure online setting where we need value estimates for every step as soon as the step is executed, then we would need to continuously apply GVL to partial trajectories, which can be slow and expensive. But we believe that this may not be a practical concern given the limited use cases of such pure online settings in practical robotics applications. Nevertheless, we have added a discussion on this topic in our limitation and discussion section. [1] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018. [2] Ball, Philip J., et al. "Efficient online reinforcement learning with offline data." International Conference on Machine Learning. PMLR, 2023. [3] Nakamoto, Mitsuhiko, et al. "Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning." Advances in Neural Information Processing Systems 36 (2024). --------------------------------------- **Question/Comment 2**: Dataset Quality Estimation Section 4.3 can be strengthened with empirical results. **Response 2**: Thanks for this suggestion! We have added one more real-world experiment during the rebuttal period. Details are in Appendix F of the updated manuscript. Given that we find DROID to be a dataset that has low VOCs in Section 4.3, we investigate whether this finding allows us to train better policies in the setting of co-training on the DROID dataset, which was how the dataset was used for real-world imitation learning in the original paper. Specifically, we study whether GVL’s data quality estimation capability can be used to discover better co-training datasets from a raw, unfiltered diverse dataset like DROID. In Table 7 in Appendix F, we find that using GVL filtering can enable diffusion policies to (1) match the performance when co-trained on bigger datasets while converging faster, and (2) significantly outperform a baseline where the co-training dataset size is kept the same as the GVL filtered dataset but randomly selected. --------------------------------------- **Question/Comment3**: Figure 1 and typos should be fixed. **Response 3**: Thanks for catching these issues. We have proofread and revised our manuscript to improve its presentation. We thank the reviewer again for their time and effort helping us improve our paper! Please let us know if we can provide additional clarifications to improve our score.

Dear reviewer tm1A, Thank you for your thoughtful comments and feedback! Here, we respond to specific questions and concerns the reviewer raises. --------------------------------------- **Question/Comment 1**: More real-world results showing how the value estimations of GVL help with downstream policy learning can strengthen the experiment section. **Response 1**: Thanks for this suggestion! We have added one more real-world experiment during the rebuttal period. Details are in Appendix F of the updated manuscript. Given that we find DROID to be a dataset that has low VOCs in Section 4.3, we investigate whether this finding allows us to train better policies in the setting of co-training on the DROID dataset, which was how the dataset was used for real-world imitation learning in the original paper. Specifically, we study whether GVL’s data quality estimation capability can be used to discover better co-training datasets from a raw, unfiltered diverse dataset like DROID. In Table 7 in Appendix F, we find that using GVL filtering can enable diffusion policies to (1) match the performance when co-trained on bigger datasets while converging faster, and (2) significantly outperform a baseline where the co-training dataset size is kept the same as the GVL filtered dataset but randomly selected. --------------------------------------- **Question/Comment 2**: Is GVL applicable to online RL? **Response 2**: Yes, GVL is applicable to online RL. Many commonly used online RL algorithms, such as SAC [1], maintain a replay buffer for trajectories the policy has encountered during exploration. Recent practical methods for online RL also bootstrap from offline datasets (e.g., [2, 3]). In all these settings, GVL can be applied the same way. However, we do note that in a pure online setting where we need value estimates for every step as soon as the step is executed, then we would need to continuously apply GVL to partial trajectories, which can be slow and expensive. But we believe that this may not be a practical concern given the limited use cases of such pure online settings in practical robotics applications. Nevertheless, we have added a discussion on this topic in our limitation and discussion section. [1] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018. [2] Ball, Philip J., et al. "Efficient online reinforcement learning with offline data." International Conference on Machine Learning. PMLR, 2023. [3] Nakamoto, Mitsuhiko, et al. "Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning." Advances in Neural Information Processing Systems 36 (2024). We thank the reviewer again for their time and effort helping us improve our paper! Please let us know if we can provide additional clarifications to improve our score.

Dear reviewer 1PKJ, Thank you for your thoughtful comments and feedback! Here, we respond to specific questions and concerns the reviewer raises. --------------------------------------- **Question/Comment 1**: The block diagram should include a language block. **Response 1**: Thank you for this suggestion. We have revised Figure 1 accordingly in our updated manuscript. Please let us know whether the clarity of our concept diagram can be further improved. --------------------------------------- **Question/Comment 2**: Comparison against representation learning methods, including LIV, is missing for visuomotor control tasks. **Response 2**: We’d like to clarify that the purpose of our visuomotor control experiments is to compare different dataset compositions generated by GVL and lack thereof, instead of comparing different choices of pre-trained visual representations. In all our visuomotor control experiments, we fix the choice of initial representations and believe our experimental findings hold independent of that choice. Specifically, we use state-of-art algorithms such as Action-Chunking Transformers (ACT) [1] or Diffusion Policy (DP) [2], and compare variants that differ in the training datasets; the vision encoders for both algorithms are Resnet-18, and they are continuously fine-tuned during imitation learning. Please let us know whether you have any remaining questions on this topic, and we are happy to provide additional clarifications during the rebuttal period. [1] Zhao, Tony Z., et al. "Learning fine-grained bimanual manipulation with low-cost hardware." arXiv preprint arXiv:2304.13705 (2023). [2] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." The International Journal of Robotics Research (2023): 02783649241273668. --------------------------------------- **Question/Comment 3**: Additional experiments can strengthen the paper. **Response 3**: Thanks for this suggestion! We have added one more real-world experiment during the rebuttal period; details are in Appendix F of the updated manuscript. Given that we find DROID to be a dataset that has low VOCs in Section 4.3, we investigate whether this finding allows us to train better policies in the setting of co-training on the DROID dataset, which was how the DROID dataset was used for real-world imitation learning in the original paper. Specifically, we study whether GVL’s data quality estimation capability can be used to discover better co-training datasets from a raw, unfiltered diverse dataset like DROID. In Table 7 in Appendix F, we find that using GVL filtering can enable diffusion policies to (1) match the performance when co-trained on bigger datasets while converging faster, and (2) significantly outperform a baseline where the co-training dataset size is kept the same as the GVL filtered dataset but randomly selected. --------------------------------------- **Question/Comment 4**: Is there a plot/table that shows how much incorporating observation shuffling helps or using autoregressive value prediction helps than the traditional bellman returns? I think this is important to understand the efficacies of the 2 proposed incorporations **Response 4**: Yes, we have compared to the ablation that does not incorporate autoregressive value prediction in Section 4.4; in our updated manuscript, Table 4 displays the result. In short, we find removing autoregressive value predictions substantially hamper VLMs’ value prediction capability, reducing it to a random level performance; this is precisely due to the fact that VLMs can no longer maintain any consistency among its task progress predictions. Comparison to no-shuffling ablation is presented in Figure 5, and again qualitatively in Figure 10 in Appendix; we find that removing shuffling makes the model insensitive to the quality of the input video and output monotonic values even when the video fails at solving the language task description. In terms of comparison to traditional bellman returns, we have compared to LIV [1], which uses a variant of the traditional bellman return to learn value functions. We have shown that GVL significantly outperforms LIV on all evaluation datasets. Note that we are not able to fine-tune a pre-trained VLM, such as Gemini or GPT-4o, using bellman-based objectives; however, given that GVL is able to attain satisfactory results without doing any fine-tuning, we see that as an advantage of the proposed approach. [1] Ma, Yecheng Jason, et al. "Liv: Language-image representations and rewards for robotic control." International Conference on Machine Learning. PMLR, 2023. We thank the reviewer again for their time and effort helping us improve our paper! Please let us know if we can provide additional clarifications to improve our score.

Dear reviewer RQdG, Thank you for your thoughtful comments and feedback! Here, we respond to specific questions and concerns the reviewer raises. --------------------------------------- **Question/Comment 1**: It would be nice to see LIV performance on the ALOHA dataset. **Response 1**: Thanks for this suggestion. We have now added an updated Figure 3 to include comparison between GVL (Zero-Shot) vs. LIV on our ALOHA-250 dataset. As shown, LIV’s qualitative behavior is similar to the one on OXE – the VOCs are close to a uniform distribution. In conclusion, GVL still produces task progress estimates that are higher quality than LIV on ALOHA-250, though the gap has become smaller compared to the results on OXE since the ALOHA dataset is sufficiently challenging for GVL too. --------------------------------------- **Question/Comment 2**: The use of VOC on the OXE evaluation is somewhat circular. **Response 2**: Thanks for this suggestion. To break up the circularity, we need to manually delegate a few datasets to be of “high-quality” to use VOC as a metric. To this end, we have selected RT-1 [1], Bridge [2, 3], DOBBE [4], and BC-Z [5] datasets because these datasets have successfully been used in prior works for large-scale imitation learning. The result is illustrated in Figure 10, Appendix F of our updated manuscript. As seen, on this subset of high-quality and diverse OXE datasets, GVL consistently generates highly positive VOCs; in contrast, LIV generates VOCs close to a uniform distribution, indicating that it struggles to accurately predict language-conditioned task progress on unseen robot videos. These results validate our experimental protocol of using VOCs as a metric for universal value functions. In addition, we have also improved the writing in Section 4.1 to make the distinction clearer. [1] Brohan, Anthony, et al. "Rt-1: Robotics transformer for real-world control at scale." arXiv preprint arXiv:2212.06817 (2022). [2] Ebert, Frederik, et al. "Bridge data: Boosting generalization of robotic skills with cross-domain datasets." arXiv preprint arXiv:2109.13396 (2021). [3] Walke, Homer Rich, et al. "Bridgedata v2: A dataset for robot learning at scale." Conference on Robot Learning. PMLR, 2023. [4] Shafiullah, Nur Muhammad Mahi, et al. "On bringing robots home." arXiv preprint arXiv:2311.16098 (2023). [5] Jang, Eric, et al. "Bc-z: Zero-shot task generalization with robotic imitation learning." Conference on Robot Learning. PMLR, 2022. --------------------------------------- **Question/Comment 3**: more analysis regarding GVL’s ability to accurately estimate values in videos where repetition is present. **Response 3**: Thanks for this suggestion! We have included two videos in the updated supplementary material in which repetition or failure is present. As shown, in both cases, when the robot gripper pulls back from the object of task interest, the GVL task progress estimates decrease, and when the gripper recovers and makes progress again, GVL estimates increase. These results demonstrate GVL’s ability to accurately estimate values in videos when repetition is present. --------------------------------------- **Question/Comment 4**: ALOHA-13 should be clarified. **Response 4**: Thank you for this suggestion. We have added a sentence in Section 4.2 clarifying. In short, ALOHA-13 is a subset of ALOHA-250 for which we have collected additional demonstration data. For example, instead of having only 2-3 demonstrations per task in ALOHA-250, we have 500+ Demos per task in ALOHA-13. This allows us to obtain statistically significant results on a task level for the in-context value learning experiments. We thank the reviewer again for their time and effort helping us improve our paper! Please let us know if we can provide additional clarifications to improve our score.

Thank you to the authors for their detailed response! For the reasons discussed below, I will be further raising my score. **Question/Comment 1: It would be nice to see LIV performance on the ALOHA dataset.** Thank you for making the additional effort to gather these results. I also appreciate the honest discussion from the authors regarding the performance of GVL and LIV on ALOHA-250: "GVL and LIV exhibit similar qualitative trends, indicating that both methods now struggle with the complexity of this dataset." That being said, GVL is able to augment its performance via in-context learning. In contrast (at least to my current understanding, however it would be helpful if the authors could verify), LIV cannot in-context learn due to its architecture: it computes values using cosine similarity in the embedding space rather than using a generative method. Due to the performance benefits of in-context learning, which only GVL can achieve, GVL seems to have a clear edge. **Questions/Comments 2-3: The use of VOC on the OXE evaluation is somewhat circular. More analysis regarding GVL’s ability to accurately estimate values in videos where repetition is present.** Thank you for addressing these concerns. I appreciate the isolation of specific "high-quality" datasets from OXE; doing so circumvents the issue with circularity and provides solid evidence of GVL's performance. That being said, it crossed my mind that perhaps GVL has simply learned to output a linear, sequential ordering (from initially shuffled data) rather than truly evaluate observations. I don't, however, believe that this is the case. Your two case-studies are helpful for intuitively visualizing what is going on, but my primary reason for believing that GVL works is GVL-SD. Since GVL is able to more effectively filter datapoints than the previous baseline (SuccessVQA), it seems clear to me that GVL can effectively assign values to "bad" trajectories (which would result in low VOCs). Thus, the success of GVL-SD seems to indicate GVL's ability to properly evaluate trajectories rather than simply output linear, sequential orderings. For future work, I think it would be good to gather a larger dataset of ground truth "bad" trajectories (i.e., ones that contain repetition or reverse progress) for a more systematic evaluation. Right now GVL's ability needs to be implied from the success of GVL-SD and a few case-studies. However I think this inference is reasonable and makes sense for the scope of this paper. **Question/Comment 4: ALOHA-13 should be clarified.** Thank you to the authors for clarifying this and for adding more description to the paper.