We propose and deploy an approach to continually train an instruction-following agent from feedback provided by users during collaborative interactions. During interaction, human users instruct an agent using natural language, and provide realtime binary feedback as they observe the agent following their instructions. We design a contextual bandit learning approach, converting user feedback to immediate reward. We evaluate through thousands of human-agent interactions, demonstrating 15.4% absolute improvement in instruction execution accuracy over time. We also show our approach is robust to several design variations, and that the feedback signal is roughly equivalent to the learning signal of supervised demonstration data.
Paper
Similar papers
Peer review
Summary
Training of a continuously learning, instruction following agent from feedback provided by users during collaborative interactions. The problem is that humans often give noisy reward and at irregular intervals. The method formulates the learning scenario as a contextual bandit problem and alternates between training and deployment stages. The main issues addressed is the irregular timed reward and credit assignment problem. The method addresses this using heuristics and demonstrated its effectiveness in the CREALBAR environment using human evaluation.
Strengths
The paper is straightforward to follow. The topic is very relevant. The evaluation and improvement throughout the interaction rounds are significant in many aspects. The experiments conducted are very thorough and complete and have convincing results to highlight the continual learning nature of the agent.
Weaknesses
- The assumption of feedback is matched with an action or close by actions (heuristics presented in the paper) is somewhat restrictive - transitions without rewards are discarded and not fully utilised, quite a waste of resources - requires expensive labelling process
Questions
- if I understand correctly, since the method formalizes itself as a contextual bandit problem, maximizing the immediate expected reward, Line 77 says "Feedback signals are given in realtime, and are not timed to coincide with a specific action" will not be favorable for training since the corresponding action and reward is not even matching up? - isn't the assumption of giving a binary reward and optimizing for the contextual bandit objective amounts to gradient descent/ascent on the right/wrong actions (with noise, since humans give reward quite randomly), which is almost equivalent to supervised learning? If this is the case, I feel like the continual learning part is just iteratively collecting more supervised learning data, which performance improvement is quite obvious
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Limitations and broader impacts are complete and included in the appendix
Summary
The authors propose a method for online continually training an instruction-following agent based on user realtime feedback gathered for a collaborative game CerealBAR. The agent need to follow the human's instructions and complete the task. The paper utilizes the contextual bandit learning approach, with immediate rewards computed from user binary feedback. Through the evaluation of thousands of human-agent interactions, they demonstrate a significant improvement in instruction execution accuracy over time. They also conduct experiments with multiple training scenarios and verify the robustness of their approach.
Strengths
* The authors propose and deploy the methods in the collaborative human game. They collected agent-human interaction data will be helpful for future agent design. * It is a pilot work to demonstrate the strength of continual learning from human feedback for instruction-follow agent. The authors shows that after training the agent from human-human dataset, the policy can be greatly improved from the online off-policy learning from human binary feedback. * The authors conduct comprehensive online experiment with variants of learning choice and human post-interaction evaluation. The variants of learning choice show the robustness of the framework and the human post-interaction evaluation shows empirical analysis on the agent behavior improved after learning from human.
Weaknesses
* The framework deign is not very clear. The authors do not mention much the design of the policy model, which transform from human instruction and observation into action (Model part in section 4). This paper mainly discusses about how to define user feedback and how to train with continual learning given the feedback. * The framework uses binary signal from the user feedback, which is limited to represent the human satisfaction or feedback. It can be difficult to generalize to other tasks with complex instructions and observations. * The framework is relatively simple with REINFORCE algorithm. It would be more convincing if the authors can conduct more experiments on other standard off-policy learning algorithm for the robustness of the framework.
Questions
Although the model can achieve 15.4% absolute improvement through continual learning over time from continual learning, it is hard to gain insight of how good the an instruction-following agent is. In order to show benefit of off-policy learning in this framework, have we tried some baseline models for comparison, such as offline RL, due to the difficulty of additional online data collection?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The experiments is difficult to reproduce. Training policy with from human realtime feedback can lead to significantly higher cost compared with offline RL or user simulator, which is widely used in dialogue agent design.
Summary
This work demonstrated a simple yet effective framework for continual learning in instruction following task utilizing human feedback. Using CEREALBAR as testbed, this work demonstrated the framework in abundant details, and show effectiveness through experiment results. This work also conducts various analysis related to human feedback during and post-hoc the interactions. Finally, various decision choices are compared.
Strengths
The method is new and well supported by the framework description, the experiment results, and the analysis in the paper. The writing is well organized and clearly written in general. This work address embodied ai problem from a relatively new perspective. It demonstrates effectiveness on task performance as well as benefits on reducing annotation effort. This work also provides promising future directions centering around the human-agent dynamics like the form of human feedback, the communication style between human-agent.
Weaknesses
I don't find any noticeable weakness of this paper.
Questions
1. Figure 3: left, the x axis range indicates number of turk workers right? aren't they 108 in total? 2. Maybe I missed something, but how do you annotated the demonstration data? Also, since reward propagation is one of the main design choice, does the gold data's feedback distribution similar as those annotated by mturkers? 3. Do you have any clue on how user adaptation contribute to specific error decreasing? (figure 3, right) 4. line 271: not sure what "we deploy a single agent for all but FEWERDEMO in the first round" means. Can you explain more? 5. (Figure 4, right)There seems to be a golden ratio between positive and negative feedback emerging. What do you think of that?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Limitations are adequately addressed in supplementary material.
Summary
This strong work presents a systems contribution in a fully-fledged system for continual learning from language feedback, in the context of situated human-to-robot instruction following tasks. Using the CerealBar environment (roughly inspired the card game SET, with an embodied flair), this work starts by learning an agent's instruction following policy from a set of offline demonstration data. This agent is then deployed against a crowd of *real human users* who work together with the agent to complete more and more task instances in the CerealBar environment, with humans providing (sporadic) binary feedback in real-time. After each learning round (a fixed number of episodes), the agent policy is updated from the real-time reward data, where the authors define a heuristic to assign the real-time feedback to individual (state, action) pairs taken by the robot; this heuristic is derived from empirical "delay in human response" data, as well as from the traditional RL literature for assigning credit to actions that aren't explicitly provided feedback. The agent update is a contextual bandit style update, using a variation of the simple REINFORCE policy gradient update (training effectively a 1-step RL policy). Across 11 rounds of updates from real-world human data, the proposed system obtains impressive results, showing a constantly growing trend in both individual instruction execution accuracy, as well as total CerealBar score (with the steepest increase in performance happening across the earlier rounds). The work also has a series of systematic ablations (comparing the binary feedback based approach with "full supervision" - showing that the proposed bandit approach can almost match the same performance), as well as qualitative Likert scale results from the real crowd users actually interacting with this system.
Strengths
This is a well-written and extremely well-executed systems contribution - the task of learning to improve situated agents from language and binary feedback is incredibly timely, and every part of the proposed system is implemented carefully and studied thoroughly. The evaluation is strong (both qualitative and quantiative), and definitely show that not only is the agent able to effectively improve with feedback, but it's able to improve *efficientyly* and actually grow its capabilities over time. I am a huge fan of this work. Separately, I think the dedication of this paper to open-sourcing all parts of the pipeline, including the multi-round crowdsourced interaction data is admirable, and will be an incredible resource to the community.
Weaknesses
In a paper such as this, I can understand it's very hard (costly + time-consuming + introduces a lot of confounds) to evaluate multiple different learning approaches at the full-scale of the real-world evaluation (though I really appreciate the experiments in section 5.2 that does this for a limited number of rounds). That being said, I would love to look at other agent learning paradigms, and justify the use of the 1-step contextual bandit style reward, vs. a multi-step RL approach that automatically learns how to perform credit assignment given the sporadic binary feedback. Especially using some of the more recent tools in the RL toolbox (e.g., learning a value function/advantage function, PPO-style clipped updates, or even off-policy/offline RL methods). A general worry I have is that much of this paper hinges off the design of the reward/utility function, which starts from the principled binary feedback provided by users, but is then further processed through a series of heuristics that may or may not really capture what's going on (for example, how the current heuristic labels transitions when there isn't an explicit binary reward tied to that timestep). I would love for the authors to address these choices in a bit more detail.
Questions
What motivated the choice of using clipped IPS coefficient for the policy gradient update vs. a PPO-style update (with a learned one-step value function estimator for debiasing); given the choice of using REINFORCE policy gradient anyway for the 1-step update, this would've been an easy addition, and possibly more stable? I'd be curious to see a more detailed breakdown of the types of language instructions (and *when* in an episode a user usually provides feedback) across the different rounds; are there more complex instructions/abstractions learned over time? Do users intervene differently in different rounds? What are the failure modes?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
The paper provides a thorough limitations section (in the appendix) that is well thought out and clearly states weaknesses in the current approach. It would be nice to move some of the punchlines to the main body in the final version of the paper though!
Summary
The authors work on the CEREALBAR setting, where two agents (one human and one computer) cooperate using natural language to achieve a shared goal. Specifically, the authors propose a new setting where the human agent can provide binary feedback to the computer agent. In their new setting, the authors follow the contextual bandit setting and use a REINFORCE-like algorithm to train the agent. Overall, experiments show that the agent achieved an improvement by learning from human signals.
Strengths
1. The authors explore a setting that involves real-time human feedback, which may be under-explored. 2. The experiments involve significant human labour and may be beneficial to the community if the authors share their data.
Weaknesses
1. Limited contribution. The authors simply applied existing algorithms to their settings. 2. Experiments show little insight. Their experiments demonstrate that the agent improves by receiving more human feedback, which is expected. 3. (Minor) Lack of concrete numbers. Results are mostly shown in figures, and it's difficult to see the real performance numbers.
Questions
1. Why do you use a contextual bandit rather than the full RL setting? This is a bit unintuitive since you have a sequential decision-making process. 2. In Figure 3, it seems the performance is the highest at around round 5. Is there a reason for this? 3. Do you plan to release the collected human feedback data? I checked the data folder in the supplementary file but didn't see them. 4. What models are you using? The paper mentions neural networks, but could you please give more details?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
Limitations are discussed in the Appendix. For example, the authors openly disclose that they did not use a more modern architecture for their agent.
Rebuttal Response
Thanks for responding to my review! I think this is a very strong paper, and highly encourage the other reviewers to engage with the authors and consider raising their scores as well!
Reply to the author
Thank you for the response. I really appreciate it! As for contribution, I think the dataset can be beneficial for the community. However, I feel more analysis can be really helpful if the entire learning process is to be claimed as a major contribution. Currently, the experiments show that more human data improves performance, but I think that is not surprising and doesn't fully justify the approach. I feel more baseline approaches can help improve the paper. My question was about the left-hand side of Figure 3, where "strongly agree" usually peaks at around round 5.
We conduct analysis via deployment experiments comparing variants of our approach (finding it robust to several modifications, that using negative feedback helps, and that we can roughly match performance of a learning approach that requires significantly more expensive training data), analysis of how user adaptation occurs and the extent of its influence on perceived model performance, as well as analysis of model errors and how they change over time through continual learning. We are happy to add additional analyses that the reviewers consider as important to demonstrate the effectiveness of our approach, and provide further insight into our experiments. With respect to Figure 3, the broader trend of agreement with the statements (agree + strongly agree) are increasing consistently over the rounds (likewise, rates of disagreement are also consistently decreasing over time). It’s critical to consider the full range of ratings for a complete view of trends.
Thank you for the rebuttal and discussions! To Reviewer zvAL: It seems the remaining concerns are regarding more analysis and more baselines. Can you please elaborate? Thanks. AC
Reply to the Authors
I am happy to provide suggestions. These experiments may contain flaws, but I hope they can convey my concerns. 1. Using offline human-agent interactions to provide a performance upper bound. Currently, while the performance increases with more human interactions, it is unclear how effective the proposed methods are since the authors mainly compared with variants of their approach. 2. REINFORCE as a baseline. The authors commented in the rebuttal that it is generally hard to get it working. However, I still think this can be an important baseline, even if it doesn't work. This is especially so because it naturally addresses (or suffers from) the sparse reward problem that the authors were trying to resolve (with the reward propagation). 3. Even simpler reward mapping. In the previous comment, the authors mention mapping real-time feedback is one of the key and novel contributions as part of their whole learning process. Therefore, I believe it's important to analyze this more. Currently, the proposed mapping variants can only assign rewards to 60% (simple) and 80% (propagated) of the actions, which is a also concern for reviewer 4ZhD. A simpler method could be assuming positive unless negative. This assigns reward to all actions, and it seems more natural to the +1 and -1 feedback setting, which is more laborious for the human annotators anyways. 4. Data efficiency analysis. Please correct me if I am wrong, but the SUPONLY baseline seems to match the number of interactions instead of the number of data points. Although the authors show that it's similar to supervised learning, it's possible that learning with human feedback requires significantly more data samples, and further analysis may be helpful. Overall, I believe that the proposed learning process is not particularly novel, and Figure 4 suggests that the various proposed designs make little difference to learning outcomes and that the key is indeed to get more data. Therefore, I maintain my current evaluation.
Reply to the reviewer:
We are quickly responding here per the points raised by the reviewer. We rushed this response, so we apologize for any typos or grammar errors. We thank the reviewer and the AC for the discussion. 1. If we understand the reviewer’s suggestion, it is to collect the same overall number of interactions with the initial system, and do a single round of training. This approach misses on earlier (in the sense of using fewer interactions) opportunities to learn, which means improving earlier, and providing later users a better experience, the way our more rapid improvement cycle does. Technically, this manifests in much higher regret over the lifetime of the system. This is because the executions in this data will be of relatively lower quality, per the performance of the initial system. Therefore, it’s not clear to us how this is an upper bound. The reviewer might mean just using gold-standard demonstrations (as we derive from human-human interactions). We conduct this experiment (SupOnly), showing our much cheaper data (because it requires no annotation or human-human WoZ interactions) is at least as good. 2. Our objective becomes identical to REINFORCE if you remove the clipped inverse propensity scoring (IPS) coefficient. However, removing the IPS term with negative rewards immediately reveals the problem with the objective, because you will have terms with negative sign, which means they have unbounded value as we minimize the loss (as probability of examples with negative reward goes to zero, the log probability goes down to $-\infty$). This completely breaks down the optimization. IPS fixes this by weighing down such terms to 0, as their probability decreases. 3. We actually found densifying the rewards is not critical. Our heuristics are more complex than suggested, and have little noise (<10%). The proposed approach (assuming positive unless negative) is simpler, but will also generate much more noise. While we didn’t experiment with this approach, our experiments clearly suggest it will be weaker. Can the reviewer explain why this approach is more natural? Studies in human feedback (outside of CS) usually rely on signals in both directions (e.g., Krauss et al. 1966). 4. The number of data points in SupOnly is actually higher (we rushed this response, so couldn’t compute exact statistics) than in RewardProp. The reason for that is that human-human games have a much higher score, and the number of instructions correlates well with the score. So human-human games are longer, giving more instructions. We choose to maintain the number of interactions equal to be the most strict towards our approach.
1. My original idea was that you would collect the dataset as per the current approach for 11 rounds. After the interactions, further, finetune the model using all collected data. My thought was to compare continual learning and offline learning. However, after taking a closer look at the algorithm, it appears that it is already taking the union of all the previous data, and it essentially re-trains the model using full data in every iteration. I feel this is not exactly true to the continual learning setting, where keeping all previous data is usually not feasible. I would be interested in seeing how the algorithm performs if it only retains, for example, one iteration of the training data. 2. Even if it does not work, I think showing that will be beneficial, as it further strengthens the author's decision to stick with the contextual bandit setting. An alternative would be PPO-style clipping, which I believe is also suggested by other reviewers. 3. My suggestion does not contradict the studies on human feedback that the authors cited. In both my suggested setting and the author's setting, there are positive and negative rewards. It's more about the design of the setting. Asking the annotators to give both positive and negative feedback may cause inconsistencies among them, and this is partially shown as only 60% of the actions have rewards in the simple setting. In fact, another analysis could be consistency analysis between annotators in terms of when they provide feedback. On the other hand, the suggested new setting could alleviate this, making human annotation simpler and more consistent. Densifying reward being not critical feels somewhat contrary to the paper. For example, the authors proposed a Heuristically Propagated Reward to improve reward density from 60% to 80%. 4. Thank you for the clarification. I feel this is a surprising result that gold instruction is as effective as binary feedback. It's possible that, as reviewer 4ZhD suggested, it's very much like supervised learning, perhaps because the environment is too simple with only a few possible actions. This somehow invalidates the learning-from-feedback setting, and a more reasonable result would be binary feedback being less effective. I feel a detailed analysis here may be beneficial for the paper. Again, I'd like to thank the authors for the discussion and their detailed responses.
Quick followup -- thank you for the discussion!
Thank you for the discussion. Unfortunately, the lead author is traveling, so we will respond when they return, if the discussion is still open. We just didn't expect the discussion to continue beyond the original Aug 16 deadline (although we appreciate the opportunity). At the meta level, I think there are a few points important to consider. We all seem excited about the prospect of adaptive systems that improve over time from interaction with their users, as we do in this paper. This is extremely understudied (in a transparent way), especially when considering actually building and deploying such systems to interact with humans. Does our paper answer all possible questions in this space? Does it attempt to say the final word? Of course not. This is not possible for almost any interesting ML problem. This is even more challenging when we consider the cost of experiments, both time and money. It's not like doing another run on GPUs that we already have in the lab. We invested > 30k USD in the experiments we show, and > 20k USD in pilot studies that went nowhere. (Yes, we tried more complex algorithms, including variants of PPO, but getting them to work required more careful tuning, which is costly with humans in the loop -- the results are too rudimentary to report with authority, so we can't say anything final). There's also the time issue. Working with MTurk is really hard, and creating a pool of high quality workers to do this relatively complex and time-consuming crowdsourcing task has been an immense challenge. We don't have an org with internal annotators at our disposal. So, our work shows a working method, which is pretty much a first in this space. Will others identify other approaches that do better? I am pretty certain. But, setting such a fertile stage for future work, well, I think it's a good thing. Are there more experiments we can run? Of course (and much of it forms exciting directions for future work), but there's a cost-benefit calculation that is very different that when just doing yet another run on your GPUs. As a community, we need an avenue to do this type of research that is not closed forever in the labs of billion-dollar companies. Will each such paper answer all possible questions? I don't think that's possible. But, can we learn a lot from such papers? Definitely! I think there's a ton to learn from our paper, and this is what drives whatever it's the community builds in the coming years. I can't wait to see it.
I also appreciate the authors for the in-depth discussions. Thank you! I agree that the authors are working on an interesting problem, and I fully acknowledge that the experiments are difficult and costly. (I also mentioned this as one of the strengths of the paper.) However, I believe this perhaps makes it even more crucial to have rigorous studies, because a potential mistake may lead to a significant amount of money wasted for people that follow this work. This is especially the case for the sample efficiency analysis for example, because it does not require additional human annotation. Overall, I think I could have been a bit harsh on the details and would like to raise my score from borderline reject to borderline accept. I hope the lead author can enjoy a well-deserved vacation for completing this work :)
Thanks for addressing my questions! It is a very interesting work and the effort you put in the work is respectable. I will update my score to 8
Thank you for elaborating! The overall design is reasonable and the online continually learning study is valuable for future research. I updated my score to 7.
Decision
Accept (spotlight)