Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning

Summary

The paper introduces a algorithmic framework that fine-tunes VLM using RL to enhance their performance in multi-step, goal-directed decision-making tasks. The authors highlight the limitations of traditional visual instruction tuning, which relies on pre-collected datasets and may not effectively train VLMs for interactive decision-making scenarios. Their proposed framework addresses this by providing task descriptions to the VLMs, prompting them to generate chain-of-thought (CoT) reasoning, and translating these into executable actions within an environment. Empirical results show that this method significantly improves the decision-making capabilities of VLMs, enabling them to outperform commercial models like GPT4-V and Gemini.

Strengths

1. The paper is well-written and clear, making it accessible and easy to follow. 2. The proposal of an alternative to current visual instruction tuning for decision-making is intriguing, with well-justified design choices that demonstrate a thoughtful approach. 3. The execution is particularly noteworthy and well-designed. For instance, the method of balancing the influence of chain-of-thought (CoT) output and action output is elegantly handled and demonstrates innovative thinking.

Weaknesses

The paper is good overall. There are a few areas that could benefit from improvement: 1. The current implementation requires fine-tuning for each task individually. However, large Vision-Language Models (VLMs) are generally capable of handling multiple tasks such as Visual Question Answering (VQA) and Optical Character Recognition (OCR). 2. The trade-off between the CoT output and action output is manually set. This means that the balance must be adjusted for different tasks depending on their nature. It is unclear how this parameter can be set for training a multi-task policy. 3. Despite the use of Low-Rank Adaptation (LoRA) for fine-tuning, the computational cost remains high. 4. The framework shows promise, but the evaluation tasks do not fully convey the benefits of using a multi-modal large language model (MLLM). Specifically, the model's ability to reason or the potential to explicitly correct its own behavior in scenarios where this is crucial is not thoroughly demonstrated.

Questions

It would be interesting to explore how visual prompting could enhance the framework and improve performance.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

yes.

Summary

This paper provides a framework that fine-tunes a large vision-language model (VLM) with reinforcement learning (RL) for decision-making tasks requiring vision and language understanding. In the framework, the VLM takes as input a state s_t that contains a visual observation o_t and input prompt v_t^{in}. The input prompt v_t^{in} contains a task description, legal action space, and desired output. Then, the VLM generates text v_t^{out} that contains CoT reasoning and an open-ended text action. The framework applies a post-processing function f on the open-ended text action, and obtain a legal action a_t. Then, the framework inputs the legal action a_t to the environment and obtains reward r(s_t, a_t) and the next visual observation o_{t+1}. Based on the reward r(s_t, a_t), the framework fine-tunes the VLM with RL (i.e., PPO). The proposed framework is evaluated on two environments: gym_cards and ALFWorld. The gym_cards environment contains four tasks like NumberLine, EZPoints, Points24, and Blackjack, and these tasks require an ability to recognize numbers in figures for arithmetic reasoning. This paper empirically demonstrates that the LLaVA-7B model fine-tuned by the proposed framework can outperform close-source LLMs such as GPT4-V and Gemini on these environments.

Strengths

S1. This paper provides a method that can fine-tune VLMs with RL for decision-making tasks. The overall problem setting seems novel and interesting. S2. Technically, the method consists of (1) prompt design for domain-specific outputs, (2) post-processing open-ended text for legal actions, and (3) estimating action probabilities of VLM policies. Among them, the third technique is very interesting. To generate a better text action, the VLM generates CoT reasoning text before it. To estimate action probabilities of VLM policies, this paper proposes to split the probability of generating CoT reasoning and the probability of generating a text action. S3. This paper empirically demonstrates that LLaVA-7B model fine-tuned with RL can outperform GPT-4V and Gemini on the gym-card environment that requires vision and language understanding.

Weaknesses

W1. Even though this paper demonstrates that LLaVA-7B fine-tuned with RL outperforms GPT4-V on the gym-card environment, according to Table 2, on ALFWorld, the performance gain does not seem significant (GPT4-V 19.4 vs Ours 21.7).

Questions

Q1. Regarding the weaknesses W1 above, is there any reason the performance gain does not be significant on ALFWorld?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

L1. The authors provide limitations in the Section 7 (i.e., Conclusions, Limitations, and Future Directions).

Summary

- This paper studies the training of vision-language models for decision-making tasks via reinforcement learning. - The authors train a 7B parameter Llava model and a baseline CNN model with proximal policy optimization (PPO) on the alfworld and `gym_cards` environments. - Additionally, the authors study the impact of also generating and reinforcing CoT rationales produced by the VLM. - They find that RL is often able to substantially improve the performance of a fine-tuned VLM, and that producing CoT tokens significantly helps during exploration.

Strengths

- This is a useful empirical paper that serves as a sanity check for using RL on multimodal tasks. - I was surprised to see the impact that reinforcing CoT tokens had. I didn't expect this. This could be a useful signal for the community. - The experimental setup is sound and well-designed. I especially liked the carefully prepared Llava-sft baseline. - The paper makes a convincing case that RL adds something beyond SFT, at least for `gym_cards`.

Weaknesses

- Only one VLM. I would like to have seen whether the conclusions change (even on a single environment) with a different VLM or different sizes of VLM.

Questions

- I would have liked to see more of an explanation of why RL does not outperform SFT across the board on Alfworld.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Not applicable.

Summary

The paper proposes an algorithmic framework for fine-tuning large vision-language models (VLMs) using reinforcement learning (RL) for multi-step decision-making tasks. The framework enhances VLMs' reasoning and decision-making capabilities by incorporating chain-of-thought (CoT) reasoning. The empirical results demonstrate that the proposed method outperforms existing models like GPT-4V and Gemini in various decision-making tasks, highlighting the significance of CoT reasoning.

Strengths

1. The topic is interesting and valuable to the community. 2. The presentation is clear, and the paper is easy to follow. 3. The methodology is general sound.

Weaknesses

My primary concerns lie with the experimental results and methodology. Please refer to the questions below for detailed points.

Questions

1. In the appendix, it is observed that CoT not only provides thought processes but also manually encoded key states, such as the “current number” and “target number” in the NumberLine task (Figure 9). This crucial information is hidden in the main text (See Figure 3). The authors need to clearly differentiate the source of CoT's effectiveness in improving reasoning—whether it stems from manually labeled key states or the textual thought process itself. This distinction should be experimentally validated and presented in the main text rather than being relegated to the appendix. 2. Moreover, regarding the CoT issue, Figures 11 and 13 show that CoT encodes not only the key states used for decision-making but also information "directly obtained from the formula in the image." There seems to be no reasonable justification for this, as theoretically, this information can be derivable from the images. An effective RL method should be capable of optimizing to identify this critical information through rewards. The inclusion of this manually encoded information makes the experiment tricky because it allows any difficult image understanding problem to be reduced to a manual extraction process. Authors should not manually encode such information to boost the final performance of their method. 3. Did your baselines (GPT4-V and Gemini) include CoT information? 4. To validate the effectiveness of the proposed RL algorithm, it would be beneficial to compare it with other RL methods, such as "OFFLINE RL FOR NATURAL LANGUAGE GENERATION WITH IMPLICIT LANGUAGE Q LEARNING" and "ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL." If using the authors' SFT model with your CoT, can we replace the proposed RL algorithm with these RL methods to achieve optimization?

Rating

3

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

NA