Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors

Summary

This paper explores a crucial aspect of robotic agents’ performance -- ensuring that while they achieve task goals, they also adhere to a broader range of constraints and preferences that go beyond mere task completion. To this end, the authors seek to improve the performance of robots in daily tasks by using large vision and language models (VLMs) to catch their undesirable behaviours and then refine them.

Rating

6

Confidence

5

Ethics flag

1

Reasons to accept

1. The paper introduces a novel use of video-language models (VLMs) as behavior critics in the context of robotics. 2. The proposed benchmark consisting of diverse scenarios where robots achieve goals but may exhibit undesirable behaviors is valuable. It provides a structured way to assess and improve robot behavior, offering a tool that other researchers can use to benchmark their systems.

Reasons to reject

1. Precision issues. The VLMs demonstrate lower precision after incorporating visual grounding. This indicates a potential reliability issue in the critiques provided by the models, which could lead to incorrect assessments of robot behavior. 2. Generalization concerns. The paper does not explicitly address how well the proposed system generalizes across different types of robots or environments. 3. Dependence on quality of training data. The effectiveness of the VLMs heavily depends on the quality and variety of the training data. Any biases or limitations in the video dataset might lead to suboptimal behavior critiques, impacting the overall system performance. 4. Lack of integration details. More information might be needed on how the behavior critiques are integrated back into the robotic systems for improving behavior.

Questions to authors

1. Besides recall and precision, are there any other metrics or methods used to evaluate the effectiveness of the VLMs in identifying undesirable behaviors? 2. Can the authors discuss how the proposed system performs across various types of robots and different operational environments? What steps are taken to ensure the system’s robustness and generalization? 3. What specific challenges do the proposed method face that result in lower precision after adding grounding part? 4. How are the critiques generated by the VLMs integrated back into the robotic systems to improve behavior? How to ensure the critiques from VLMs are correct? What mechanisms are in place to ensure that these critiques lead to meaningful behavioral adjustments?

Summary

### Problem addressed When employing large-scale generative models in robotics and embodied AI, despite their capability to generate meaningful solutions, they frequently fall short of producing desired outputs on the first try. To address this challenge, the paper investigates the potential of utilizing large vision and language models (VLMs) as scalable Behavior Critics for identifying undesirable robot behaviors in videos. ### Main contributions A comprehensive evaluation of VLM critics in real-world household environments, showcasing their strengths and failure modes, and providing guidelines on their effective utilization in an iterative process of policy refinement. ### Methodology - VLMs as behavior critics: Frames from a video for a policy for a short-horizon manipulation task (GPT-4V and Gemini Pro) are provided along with an instruction prompt that asks the VLM to identify any undesirable, potentially harmful, or risky behaviors in the video. - Benchmarking VLMs behavior critics and analysis of failure modes: 114 test cases are prepared, each containing 2 negative videos and 1 positive video of different policies attempting a task, along with human annotations. A taxonomy of 4 types of errors is proposed to study the failure modes of the VLM critics. - In addition, it is experimented to have a critic compare pairs of videos and provide pairwise preference. - Augmenting behavior critics with grounding feedback: Grounding feedbacks calling out hallucinated events are provided to the VLM critic to improve the accuracy of the critics, which significantly improves the precision rate. - Integrating critiques into a closed-loop system: A viable closed-loop system is demonstrated, where Code-as-Policies (CaP) is employed to control the robot, receiving the feedback from a VLM critic regarding undesirable behaviors, and accordingly adjust its policy. ### General evaluation - Quality: The methodology, experiments, and analysis are well-conceived and well-presented. - Clarity: The paper is well-written and easy to follow, with clear explanations of methods and logic. - Originality: The work is practical yet innovative. - Significance: This paper presents a solid and valuable study on the interesting application of using VLMs as critics evaluating robotics policies with respect to constraints that are hard to quantitatively and exhaustively delineate. The direction is promising given the fast advancement of VLM models, and holds the potential of boosting the field of embodied AI tremendously.

Rating

8

Confidence

3

Ethics flag

1

Reasons to accept

- The topic is very interesting and relevant, and the paper is a solid piece of work that establishes a framework for evaluating and understanding VLMs as policy critics.

Reasons to reject

- The benchmarking dataset could be further expanded both in size as well as in the coverage of scenarios. - The taxonomy proposed is not fully justified.

Questions to authors

1. Will the curated benchmarking dataset be released? 2. Could authors provide more detailed statistics of the collected data? For example, concrete task names and the number of suboptimal / optimal policy videos collected and annotated for each task. 3. How exactly are the 114 test cases, each containing 3 videos, constructed with the 175 videos? Apparently there are videos reused. Could such reuse introduce biases to the evaluation metrics? This also makes the detailed data statistics more necessary to report. 1. A related question, on page 8, in section 5.2, it is mentioned that there are 262 critiques. Are these 262 critiques made on all 175 videos? What is the distribution of number-of-critiques-per-video? 4. The taxonomy seems to be focused on false alarms. Could the errors where a VLM critic *misses* undesirable behaviors (measured by recall) be divided to categories and analyzed as well? 5. On page 5, in section 4.2, the "Evaluation metrics" part says > “precision” should permit the critics to suggest valid critiques even when they do not associate with any human-annotated negative event. while on page 8, in section 5.1, the second paragraph says > Note that the recall-precision evaluation is conducted solely with videos containing negative events It is now unclear to me whether "positive videos" are considered when evaluating precision. 6. Could the "critique operationalizability" be improved by prompt engineering or few-shot in-context examples? When the VLM critic can correctly identify an undesirable behavior, it seems possible that its critiques can be more operationalizable if guided / prompted accordingly. 7. Why is the taxonomy designed such that the error types are not mutually exclusive? Is the "causal understanding errors" vs "visual grounding errors" the only overlapping case?

Summary

Generative models often overlook task constraints and user preferences, and instead focus solely on achieving task success To address this, the authors propose using video-language models (VLMs) as behavior critics to catch undesirable agent behaviors in videos. The authors conduct a study on using VLMs to evaluate robot behaviors and find that they can identify a significant portion of undesirable behaviors. The authors also demonstrate a framework for integrating VLM critiques into a closed-loop policy-generation system, which can refine policies according to the critiques.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

The framework includes a benchmark dataset of 175 videos, a set of evaluation metrics. This framework provides a robust and systematic way to evaluate the performance of VLMs in this context. This approach addresses the limitations of VLMs in visual grounding and provides a way to refine the outputs of VLM critics. The authors also demonstrate the effectiveness of this approach in achieving a high precision rate. The authors discuss the potential of integrating VLM critiques with closed-loop policy-generation systems, such as Code-as-Policies.

Reasons to reject

The evaluation is conducted on a relatively small dataset, which may not capture the full range of possible errors or scenarios. The experiment relies on perfect grounding feedback to improve the performance of the VLM critic. However, in real-world scenarios, feedback may be imperfect, random and noisy. Also, how does the VLM critic handle hazzard during robot execution? The experiment focuses primarily on GPT-4V and Gemini-Pro. I would like to know more results about other (weaker) VLMs' (e.g. Instruct GPT, Kosmos, Llava) performance in order to justify the contribution of the proposed methodology.

Summary

The paper addresses the automatic verification of robot behaviors. This paper investigates using large vision and language models (VLMs) for automated verification of robot behavior. The authors empirically evaluate the ability of VLMs to detect undesirable robot actions given short videos and prompts. They find that GPT-4V, the VLM they use, tends to produce false positives (incorrectly identifying desirable actions as undesirable) when evaluating manipulation videos in which the grabber tool was primarily controlled by humans. In contrast, Gemini Pro tends to produce false negatives (missing inappropriate actions).

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

* The application of VLMs to detect inappropriate robot behaviors is an interesting and promising area of research with the potential to enhance real-world robot applications. * The paper identifies the main challenge, Visual Grounding, in using current VLMs to detect undesirable behaviors. Furthermore, it demonstrates that humans can address this challenge to improve detection accuracy, providing insights into potential limitations and guiding future research. * The proposed method is experimentally evaluated on a real robot's behavior policy.

Reasons to reject

* The authors evaluate Recall and Precision using data that includes only inappropriate behaviors, potentially overestimating Precision. Since practical robot policies should minimize undesirable actions, evaluating these metrics on data that includes appropriate behaviors is essential to predict real-world performance. Additionally, Table 1 presents Recall and Precision for data with undesirable behaviors alongside Accuracy for data with both appropriate and inappropriate behaviors, which could be misleading. * The lack of data availability hinders future research. * The experimental results indicate a significant gap between the demonstrated accuracy and realistic applications. The authors should consider limiting the environment and tasks to more focused scenarios with higher applicability, given the diverse nature of the household tasks considered in the study.

Questions to authors

* Why were Recall and Precision evaluated using data that includes only undesirable behaviors? * What is the estimated frequency of undesirable behaviors in real-world applications? While the occurrence rate depends on the robot's policy, some assumptions are necessary for designing evaluation methods. * Was the prompt template tuned for Gemini Pro? The readers may wonder whether prompt templates exist that could improve Gemini Pro's Recall.

Please see reviewer question

Hi Authors, You might have missed the questions from the reviewer. It would be great if you could respond. Thanks, AC