Guide Your Agent with Adaptive Multimodal Rewards

Summary

The paper proposes an interesting and novel idea of using large pretrained mulitmodal models to compute image-text alignment score to use as a reward to train return-conditioned policies using a Decision Transformer (DT). They propose a novel imitation learning framework called Multimodal Reward Decision Transformer (MRDT) which trains a return-conditioned policy using adaptive reward signals from image-text multimodal rewards. The authors also propose a fine-tuning scheme that uses VIP and IDM objective to fine-tune using CLIP-Adapter using data from in-doman demonstrations to improve performance. The paper also shows MRDT successfully improves generalization to unseen levels and unseen goals on 3 environments from OpenAI procgen benchmark. Authors also present an analysis on how MRDT helps mitigate goal misgeneralization and present a set of metrics to evaluate the quality of multimodal rewards.

Strengths

1. Paper is well-written and easy to follow 2. The proposed approach is interesting and is shown to be effective on the OpenAI Procgen benchmark when generalizing to unseen levels and unseen goals. It outperforms prior work InstructRL on all 3 environments by a decent margin. The results also show that finetuning using proposed fine-tuning scheme further improves the results. 3. The analysis on quality of multimodal rewards is quite informative. It gives a clear insight on how the multimodal rewards differ for different cases . The analysis shows finetuned multimodal rewards better capture distinction, distance (S) and distance(F). 4. The analysis in figure 8 shows the rewards generated with instructions aligned with the goal state gets higher reward compared to a random instructions. This demonstrations a pretrained multimodal model like CLIP can give good reward signal to the policy.

Weaknesses

1. The paper proposes an interesting idea but the experiment section is lacking breadth. I'd appreciate if authors consider evaluating their method on multiple benchmarks which have multi-modal input. Similar to InstructRL paper authors can consider a subset of tasks from RLBench for evaluation of the proposed method. The environments in ProcGen are quite simple and the generalization is only being tested for unseen instantiations of a single task (for 3 environments). Does MRDT work well if we want a single policy that can achive different goals? One example is Object Navigation problem (but in simpler grid world environments), where the agent has to navigate to single instance of one of the n target object categories i.e. Find a chair/Find a sofa. It'd be nice if authors can add more experiments from different benchmarks to the paper. 2. It seems like for all 3 environments in ProcGen benchmark a better reward can be cosine between image embeddings of current state vs the expected goal state. Have authors tried using this simple baseline? Does this lead to better or worse results than MRDT? The concern I have is that these environments are quite simple and might not require multi-modal rewards. 3. Does using multimodal rewards from CLIP like model leads to goal misrepresentation problem? For example, let's consider a augmented version of Maze II environment. If in addition to the yellow gem and diagonal line there was a straight red/yellow line and the task was ""Navigate a maze to collect the diagonal line"" or ""Navigate a maze to collect straight line"", how would all the baselines perform? Are these multi-modal rewards capable of clearly distinguishing these similar looking goals? or this leads to agents learning a average policy where it sometimes confuses the straight line with diagonal line because of misrepresented reward signal? 4. The qualitative results in appendix B are quite interesting and I appreciate authors added these results in the supplementary. In Figure 9, for the coinrun task the Fine-tuned CLIP rewards are almost as high as the reward that agent achieves at the goal state. This seems concerning and hints towards multi-modal rewards not being able to distinguish goal state from a random subset of states. This could lead to policy learning arbitary behaviors or not learning the task at all if trained/finetuned using RL instead of imitation learning. Do authors have any insights on why the finetuned reward model show unexpected behavior? And how can this be mitigated so that we can successfully use these rewards for online training/finetuning

Questions

1. The proposed method is interesting and a novel but intuitive idea to leverage pre-trained multi-modal models as rewards for learning a policy. The idea is novel and seems promising, the main concern I have is on the breadth of the experiment section. It'd be nice if authors can show results on slightly more complex environments to support the effectiveness of MRDT. I'd be open to updating my rating if authors can demonstrate the effectiveness of MRDT on different benchmarks/tasks. 2. As outlined in the weakness section, it would be great if authors can discuss more about why the reward spikes initially for finetuned CLIP model as shown in the analysis in appendix B. 3. I'd appreciate if the authors can also discuss the goal misrepresentation problem I describe in point 3 of the weakness section.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

1. The experiments section seems lacking at the moment and could benefit from evaluation on different benchmarks 2. Multi-modal rewards post finetuning exhibit unexpected behavior which could prove to be a big problem when finetuning policies using these rewards. These rewards can sometimes misrepresent the value of the state agent is in. 3. It is not clear how well the approach would work in more complex tasks with partial observability and non-markovian states.

Summary

The authors propose a formulation of the reward-conditioned decision transformer but do so using a learned reward function estimated by a pretrained CLIP style model which maps expert demonstrations and textual descriptions of these demonstrations into the same domain. They evaluate their method on the coinrun and Maze navigation domains.

Strengths

1. The method is principled and the intuitive: The authors describe how to finetune existing language and vision models and use alignment between the observation and the text a reward for a decision transformer. 2. The method figures are clear and easy to understand (except Fig1 see weaknesses).

Weaknesses

1. Example poorly motivated: How is goal misgeneralization any different from policies overfitting to training environments? Figure 1 needs improvement since the whole paper is motivated by it and currently it is impossible to tell what is happening in the 2 panes. 2. Comparison with baselines: The idea of using CLIP for reward generation is not entirely new and Zest https://arxiv.org/pdf/2204.11134.pdf is a very close related work published about a year ago. Comparing to this paper is a natural baseline. 3. Generalization across language commands : Does the method enable generalization using language commands? The only experiments are demonstrated are domain generalization (unclear how OOD the test tasks are in comparison to the train). 4. Unclear how the method prevents goal misgeneralization: While the introduction motivates the work from this problem, the experimental section does not lend any credence to the fact that this method would prevent such a problem. Experiment

Questions

Please help me understand how to distinguish goal misgeneralization is different from overfitting to training tasks. Further, it's not clear to me how adding language prevents overfitting. Consider an extreme case, where the dataset of demonstrations only contains trajectories moving towards the coin placed in the exact same location. How does adding language prevent overfitting here? The language label here will still be the same across trajectories. It's unclear to me why language would prevent overfitting.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The paper evaluation is a bit tenuous and I would encourage expanding the scope of the experiments. Currently, a lot of the experiments are geared towards evaluating the reward. Since a pretrained model, i.e., CLIP is being used to generate the rewards, it would be useful for readers to know how much of a domain alignment is needed for this work and to what extent the method breaks when trained on vastly OOD environments.

Summary

This paper presents a framework called Multimodal Reward Decision Transformer (MRDT) that uses the visual-text alignment score from pre-trained vision-language models (after careful fine-tuning using the in-domain data) as the reward signals in visual-based reinforcement learning. Specifically, the authors propose to train a return-conditioned policy based on Decision Transformer (DT) with the improved reward signals learned by the tuned VLMs. The method is shown to generalize better than without the multimodal reward strategy in environments with unseen goals partly due to the knowledge captured in pre-trained models.

Strengths

+ The proposed method leverage pre-trained VLMs in a simple yet effective way for reward learning. + This is an important direction to study foundational models for decision-making. Compared to directly using pre-trained visual embedding models, this strategy of reward learning is more promising so far, IMO.

Weaknesses

- Limited experiments. The authors only perform their studies on three tasks in ProcGen, far from most published work on this subject. - Due to the limited experiments, it might require higher technical novelties. However, the novelty is also limited since the proposed method is built on top of several existing methods (VIP, IDM, CLIP-Adapter, etc.) I am open to raising my score given more experimental evidence.

Questions

Need to also discuss some other existing relevant literature on this subject, e.g., [1] Reward Design with Language Models [2] LIV: Language-Image Representations and Rewards for Robotic Control

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The major limitation is the lack of adequate evaluations. The approach is an otherwise promising one.

Summary

This work tackles the goal misgeneralization problem in goal-conditioned RL agent setups. They propose to leverage pre-trained multimodal models, i.e., CLIP, to serve as a reward function that simultaneously estimate both text and image modalities for deciding an engineered reward for each time step. The authors train a transformer-based model to take a sequence of encoded state representation, estimated reward from CLIP, and historical actions, to predict the next reward and state representations for learning a return-conditioned policy. Additionally, to encourage smoother and more robust reward functions, they adopt value implicit pre-training and inverse dynamic model to incentivize better adapted rewards. The experimental results on OpenAI Procgen benchmarks demonstrate better generalization abilities over baselines without the proposed reward engineering scheme.

Strengths

- Utilizing CLIP as an adaptive reward function is sound and easy to implement. - The proposed idea is neat and should be transferable or up-scalable to more complex tasks and environments. - The adoptions of both VIP and IDM are to the point for the proposed return-conditioned policy training. - The reward evaluation protocols are well-motivated and justified.

Weaknesses

- While the experimental setups are solid, the adopted environments (testbed tasks) are far from real-world imageries, and the adaptation may not utilize very well (and much) what CLIP has learned during the pre-training. More challenging environments are needed to justify whether the multimodal rewards that coming out from a pre-trained models are indeed useful. I.e., a valid baseline is to just train a small sizable multimodal transformer (with both VIP and IDM applied) and see if the pre-training alignment is indeed that important/useful. - To really (and effectively) showcase the generalization ability, I suggest experiments that are conducted on generalizing to unseen (but perhaps visually similar) tasks as a testbed for the multimodal rewards. Using the testbeds this work adopt, for example, Maze I transferring to Maze II would be an interesting and insightful experimental setup. - Literature reviews: using CLIP scores or CLIP-based perceptual loss [1] has been quite popular these days. The author should provide more in-depth discussion on related works along using pre-trained multimodal models as a strong supervision signal. [1] Vinker, Yael, et al. "Clipasso: Semantically-aware object sketching." ACM Transactions on Graphics (TOG) 41.4 (2022): 1-11.

Questions

- Why don’t you choose tasks and/or domains that are closer to CLIP’s training distribution (i.e., more real-world imageries), such as some egocentric embodied tasks or navigational agent? - Do you think your method, or perhaps, CLIP-based rewards, are trustworthy enough to generalize well and replace the actual environmental rewards? I.e., can we assume we can simply adopt the engineered rewards and disregard the environmental success signals to train an agent and it still performs (maybe almost) on par with that using the environment reward. (I guess one will need to further train a termination signal from CLIP, too.)

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

- The authors did not explicitly point out the limitations of this work. I suggest the authors discuss more on this matter in both their proposed ideas and the execution of the experiments.

Response to Rebuttal

Thanks for your experiments and rebuttal. I still have some concerns which I list below. W1. What you have described is overfitting. And if it is indeed overfitting, then are a slew of works that address overfitting in offline RL. Can you please clarify if you mean something else? If not, then there is no point in adding more jargon to a field. W2. Thanks for the experiment. W3. To clarify: does the training data only contain yellow cheese? If so, I am worried this experiment does not really tell us much about the role of language conditioning in generalization as the agent may only attend to the "yellow" portion of the language command. Could you perhaps run inference only on another object with a different color?

Response to Rebuttal

Thank for the responses. W1, L1: I appreciate the additional experiments. The results on RLBench are promising, please include these results in the final manuscript. W2: Can you clarify what the input to the MRDT baseline for these experiments were? Was it a simple language instruction like "find red diagonal line" or "find yellow cheese"? And these goals used during test time are all already seen during training? If yes, would it be possible to just evaluate GC-DT and MRDT on ProcGen environments using an unseen goal? For example: evaluate MRDT trained to navigate to yellow cheese to go to a red diagonal line and similarly evaluate GC-DT with an image goal of red diagonal line in test environment? Such an evaluation will be more robust compared to evaluating MRDT with already seen goals vs GC-DT with unseen goals i.e. image goals from the test environment. I understand the short time left in the discussion period, in case authors can't provide results I'd appreciate some discussion on the evaluation setup of GC-DT vs MRDT. W3, Q3, L2: I appreciate the experiments. It'd be nice to include these experiments in the appendix of manuscript as well. I think most of my concerns have been addressed apart from a discussion on evaluation setup on W2. I am updating my score to reflect this. I'd appreciate if authors can respond to my questions on evaluation setup on W2.

Reviewer Response

Thanks for the rebuttal. [W2] addresses my concerns while the added experiment in [W1] partially does so. I still believe it requires more tasks to better reflect the efficacy of this approach. I would like to raise my score to the borderline, slightly leaning toward rejection, though.