Summary
The authors introduce Language Feedback Models (LFM), a method filtering out desirable transitions (i.e. transitions considered as helping to solve a task) collected by an agent in an environment to then improve this agent by doing Imitation Learning on these transitions. The LFM method has three phases: 1) First, a fairly large number of transitions are collected by the initial policy, and GPT4 is used to determine which transitions are desirable. Then, 2) given this dataset of desirable transitions, a smaller LLM is trained to reproduce GPT4 in saying these transitions are desirable. Finally, 3) the policy is used to collect new transitions, which are then filtered out by the smaller LLM, taking only the ones considered as desirable, and the policy maximizes the likelihood of the actions chosen in these transitions.
Experiments in two TextWorlds and one visual navigation environment are performed, showing that LFM successfully improves the initial policies (learned with Behavioral Cloning from expert demonstrations) and leads to better results than using GPT4 to directly propose the best action to perform and imitate it. Additionally, experiments show that the learned feedback model can be used to improve the policy in unseen test tasks which further increases the policy's performance even though the feedback model has not been trained on the test tasks. The authors also show that one can go further and not only train the smaller LLM to select desirable transitions but also to explain why these transitions are desirable (also by imitating GPT4), leading to better explainability.
Strengths
The method relies on distilling the feedback ability from GPT4 to a much smaller LLM (Flan-T5 770M) to reduce the cost of asking for feedback at every step. The authors show that they successfully transferred this ability and that the obtained feedback model generalizes to unseen tasks. The experiments also show that their method is not only less compute-effective but also much more efficient than directly imitating GPT4 used as the policy.
The authors also show the method can be trained to provide explanations of the feedback given, leading to better explainability of the method.
Weaknesses
One of the main weaknesses I see is the lack of baselines to properly assess the efficiency of the method. The LFM method acts like a rejection sampling algorithm where only transitions considered desirable by the feedback model are kept to fine-tune the policy. Similarly, If all environments provide a reward if would have been interesting to have a baseline relying on this information for the rejection sampling. If the reward is sparse, one could wait for the end of the episode and only keep desirable trajectories (e.g. whose episodic reward is above some threshold or if the goal has been reached).
The authors also say "In this work, we consider long-horizon settings with only sparse and delayed task-completion rewards. Consequently, we focus on imitation learning from demonstrations as opposed to reinforcement learning from rewards." It could also be interesting to see how RL performs with the same amount of collected data.
Finally, it seems the authors use a dataset of expert demonstrations to first train the initial policy with BC before applying their method and the compared baselines. However, they do not provide any insights on how important this preliminary phase is. Indeed, on key advantage of LFM is that it relies on BC while not requiring an expert policy. However, results from Table 3 show that LFM mostly seems to substantially improve the initial policy learned with BC from expert demonstrations.
Questions
- In Table 3, BC results are different from "Prev SOTA" on ALFWorld and Touchdown. Why is this the case, and how did the authors collect expert demonstrations that led to a policy better than SOTA?
- When describing the ACTPRED baseline l237, it is mentioned "First, we execute k steps of the base policy, then query the LLM for the next action". What are these k first steps and why are they necessary?
- The authors computed the perplexity averaged over tokens of each possible action with the policy (that is an LLM) and selected the action with the minimum perplexity. Why did the authors not use the (log) probability of the sequence of tokens as in SayCan (Ahn et. al, 2022), GLAM (Carta et. al 2023) or TWOSOME (Tan et. al 2024)?
- Also, computing the perplexity (or probability) of each action to follow the prompt can be very extensive when the action space grows, given that the policy is still a fairly large model (770M). Did the authors use anything to make this fast enough, especially in ScienceWorld where the set of possible actions can be very large?
- In LFM, an initial phase uses the policy to collect trajectories that are then given to GPT4 in order to obtain the dataset to finetune the feedback model. In the experiments, 10k trajectories of 20 steps have been collected if I understood well. Were these trajectories also appended to the dataset used to improve the policy later on?
- The authors showed an example of an explanation provided by the feedback model (based on Flan-T5 770M) in Table 5. Given the limited abilities in text generation of Flan-T5 770M, did the authors perform a deeper study of the explanations provided by the model, especially on the unseen test tasks?
Limitations
The authors discuss the limitations of the introduced method in the appendices. They identify that, while LFM improves compute efficiency by distilling feedback abilities from GPT4 into a smaller LLM, Flan-T5 770M is still a fairly large model to be called at every step, both for the policy and the feedback.
Also, Appendix G shows that using Llama 2 70B instead of GPT4 can significantly impact the results. This is not a strong limitation, in my opinion, given that GPT4 is only used in the initial phase to train the smaller LLM.