The recent rapid advancement of machine learning has been driven by increasingly powerful models with the growing availability of training data and computational resources. However, real-time decision-making tasks with limited time and sparse learning signals remain challenging. One way of improving the learning speed and performance of these agents is to leverage human guidance. In this work, we introduce GUIDE, a framework for real-time human-guided reinforcement learning by enabling continuous human feedback and grounding such feedback into dense rewards to accelerate policy learning. Additionally, our method features a simulated feedback module that learns and replicates human feedback patterns in an online fashion, effectively reducing the need for human input while allowing continual training. We demonstrate the performance of our framework on challenging tasks with sparse rewards and visual observations. Our human study involving 50 subjects offers strong quantitative and qualitative evidence of the effectiveness of our approach. With only 10 minutes of human feedback, our algorithm achieves up to 30% increase in success rate compared to its RL baseline.
Paper
Similar papers
Peer review
Summary
The paper introduce GUIDE, a RLHF framework for real-time RLHF with online and continous human feedback. GUIDE translates the human feedback to dense reward. Addtionally, GUIDE includes a parallel training model that learns a simulated human feedback. By involving 50 participants annotation, GUIDE solves three typical challenging sparse reward environments.
Strengths
1. The paper is easy to read. 2. GUIDE firstly proposed novel continous human feedback and is also evaluated human annotators. 3. GUIDE demonstrated improvement compared to baselines in 3 environments, and analyzed cognitive tests and analyses.
Weaknesses
1. My biggest concern comes from the practicality of GUIDE. From both theoretical and experimental perspectives, I find it hard to believe that such a simple continuous feedback model can be applied to real-world scenarios. For example, the paper states in line 38 that "Current research has demonstrated success primarily in simple, low-dimensional tasks with limited solution spaces." However, the experiments conducted in the paper also involve environments where baseline algorithms like DDPG or SAC can converge with a good reward function after only about **10 minutes of training**. Moreover, according to the experimental results, GUIDE, which incurs a high cost of human feedback, does not outperform manually designed simple rewards (such as the distance to the target, I think it is not hard to design it). Therefore, despite the fact that the environments used do have continuous actions and image inputs, I believe these environments are not suitable for validating RLHF algorithms because the reward functions are easy to design and the tasks themselves are simple. 2. The core argument of the paper is that continuous real-time feedback is extremely difficult to implement in practice. It requires annotators to constantly provide scalar rewards without pause, and such absolute value annotations are more susceptible to biases from different individuals. Pair-wise annotation is much easier than absolute value annotation and can be conducted asynchronously with the training process. If an AI agent needs to be trained for several days, the cost will be unacceptable. 3. Although the paper suggests using model predictions to synthesize feedback, such a simple supervised learning regression objective is unlikely to accurately model the complex reward distribution. My reasoning is that predicting the relative goodness of A and B is easier than predicting scalar reward values, but there will still be many prediction biases. 4. The definitions of various symbols in the paper are imprecise and confusing, for example: - What is the meaning of A(s, a) in Equation 1? Also, A(s) = a in the line 123, is it the same? - difference of Q(s, a) and q? - How to get the r^hf? These typos make it very difficult to understand the details of the paper. 5. There is a lack of discussion on recent related works in RLHF, such as: - [1] White D, Wu M, Novoseller E, et al. Rating-based reinforcement learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(9): 10207-10215. - [2] Yuan Y, Hao J, Ma Y, et al. Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback[J]. ICLR2024. - [3] Guan L, Verma M, Guo S S, et al. Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation[J]. Advances in Neural Information Processing Systems, 2021, 34: 21885-21897. - [4] Guan L, Valmeekam K, Kambhampati S. Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences[J]. ICLR2023.
Questions
Can you provide more details about the annotator's annotations, such as the actual interface and the frequency of operations?
Rating
3
Confidence
4
Soundness
3
Presentation
2
Contribution
2
Limitations
Yes, and yes.
Dear Reviewer, Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above. We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts! Best regards, Authors
Thank you for your reply, I still think that the GUIDE method is difficult to apply to reinforcement learning agent that require long term training (e.g., a few days, atari/procegen/minecraft), and even if it could be used, sustained manual loads for a few days would still be excessively costly and inconsistent. I have some concerns about GUIDE's usability for real-world tasks, and a few of the toy experiments in the paper don't support the author's point of view. Thanks a lot, I will clearly keep my score.
Dear reviewer, thank you for your response. We believe there is clearly a misunderstanding of our work. >“I still think that the GUIDE method is difficult to apply to reinforcement learning agent that require long term training (e.g., a few days, atari/procegen/minecraft),” >“I have some concerns about GUIDE’s usability for real-world tasks” We find these points raised by the reviewer to be puzzling and self-conflicting. As you have mentioned your concern for real-world applicability, but the environments mentioned by you (e.g., atari/procegen/minecraft) are not real-world tasks. In fact, our bowling task is one of the challenging ones in atari. Both of our find treasures and hide-and-seek are similar to minecraft sub-tasks. Given our setup with partial visual observation, long-horizon planning, and difficult-to-design reward functions without extra info, we believe our tasks are not toy examples. **As stated above, this is evidenced by our results in Fig. 5 where SOTA RL struggles to converge while GUIDE improves them by a large margin.** We believe that this shows the exact potential of GUIDE. It is infeasible to train regular RL on an unrealistic amount of experience, this is exactly the purpose of using real-time human guidance for accelerating agent learning. As shown in Fig 5 of our paper, **untrained human participants using GUIDE are able to reach the same level of performance as the RL baseline within half the training time.** We believe these are encouraging results and it shows the potential of real-time human guidance. >“sustained manual loads for a few days would still be excessively costly and inconsistent” We agree that human feedback is expensive and effectiveness varies across individuals. However, we addressed both of these issues by introducing a simulated feedback provider and conducting the largest scale analysis existing on the effect of individual differences on AI guidance. This has not been done before. >“and a few of the toy experiments in the paper don’t support the author’s point of view.” We would like to emphasize that RL baselines still struggle on the tasks that the reviewer categorized as “toy examples.” Again, shown in Fig 5, GUIDE surpassed the baseline by a large margin. **Overall, we don’t consider devaluing our contribution simply based on a gap already pointed out by our experiments and us showing strong results in scalability to be appropriate for scientific advancements.** We do not feel the reviewer’s points directly relate to our work.
I believe the author misunderstood my point. My core opinion is that real-world tasks are usually very complex and cannot be learned in just 10 minutes. Therefore, longer training times and more complex environments pose significant challenges to the GUIDE method mentioned in the paper. Many RLHF methods use environments like Atari or Minecraft for training to demonstrate their effectiveness in complex settings. However, the author has not proven this point, and the simulated feedback reward section lacks sufficient and comprehensive experimental evidence. I am not fully convinced that GUIDE can promote the development of the RLHF community. The feedback method of GUIDE is somewhat novel but useless, it requires a high level of annotator focus and a considerable tolerance for errors, making it infeasible for long-term annotation tasks.
Thank you for your response. We would like to address all of your concerns and questions below. ---- >“I believe the author misunderstood my point. My core opinion is that real-world tasks are usually very complex and cannot be learned in just 10 minutes. Therefore, longer training times and more complex environments pose significant challenges to the GUIDE method mentioned in the paper. Many RLHF methods use environments like Atari or Minecraft for training to demonstrate their effectiveness in complex settings. However, the author has not proven this point, and the simulated feedback reward section lacks sufficient and comprehensive experimental evidence. I am not fully convinced that GUIDE can promote the development of the RLHF community. “ We would like to ask for specific clarifications from the reviewer. Though we have emphasized that one of our tasks is a challenging Atari game, and the other two tasks are similar to sub-tasks in Minecraft, the reviewer kept referring to “many RLHF methods use environments like Atari or Minecraft.” To our knowledge, and as stated in our paper, state-of-the-art algorithms with the same settings as ours (real-time human-guided RL) use much simpler environments than ours. In fact, ours is the most challenging benchmark experiment. For instance, Deep Tamer uses bowling, which is included in our experiment, and Deep COACH[1] uses Minecraft (a drastically simplified version that is simply goal navigation in a 10 x 10 grid world with discrete actions). Ours involve continuous control from partial visual observations and competitive settings. If the reviewer is referring to other work that uses Atari or Minecraft but does not fall into our same problem setting, such as preference-based offline setting or imitation learning, we would like to argue that this is not a fair request to ask us to compare or use the same tasks. Our setting is much more challenging – 1) we do not assume any expert level of task execution as most imitation learning algorithms do; 2) our setup aims to tackle novel problems in real-time instead of assuming plenty of time and many offline rollouts or replay buffers for humans to provide feedback. Therefore, we kindly ask the reviewer to point at specific literature that uses the same setting but uses Atari or Minecraft as their comparisons. >“The feedback method of GUIDE is somewhat novel but useless, it requires a high level of annotator focus and a considerable tolerance for errors, making it infeasible for long-term annotation tasks.” We disagree with the reviewer’s statement. We would like to point out that the reviewer’s comments are based on conjecture without evidence. As stated above, from our results, there is no evidence showing that a continuous feedback interface poses a higher cognitive load than discrete feedback since both require the users to watch the agent continuously, while GUIDE does not require constant switching and clicking on different buttons, nor evidence showing that continuous feedback requires a considerable tolerance for errors. From our 50 human subject studies as the experiment with the largest scale, our results show the opposite. Our continuous feedback mechanism still performs stronger than the discrete feedback baselines by a large margin on all 50 subjects, while the discrete feedback is less robust against individual differences. We have demonstrated the effectiveness of our feedback mechanism in our experiment results, as pointed out above, contrary to the reviewer’s comments that our feedback method is “useless”. ---- *references* [1] Arumugam, Dilip, et al. "Deep reinforcement learning from policy-dependent human feedback." arXiv preprint arXiv:1902.04257 (2019).
Summary
The paper proposes a new approach to reinforcement learning with human feedback in simple video games. The method relies on continuous human feedback that is provided by the human observer hovering their mouse over a window with a spectrum of positive and negative rewards. Unlike prior approaches, this method converts human feedback directly into a reinforcement learning reward with an added time delay. Moreover, the method includes a model that regresses states into observed human feedback, which allows for simulated human feedback. The effectiveness of the method is demonstrated in three simple games in a human study with 50 participants.
Strengths
1. The authors propose a simple way to incorporate continuous human feedback as a reinforcement learning reward with a constant time delay. The environment reward and human feedback are simply added to form the final reward function. 2. It is demonstrated that human preference can be directly regressed from states and actions to provide simulated human feedback. 3. The authors perform an extensive human study showing the effectiveness of their method. They also correlate the subject’s performance in a cognitive test with their ability to guide an RL agent.
Weaknesses
There are three major unstated assumptions: 1. The delay between an event appearing on the screen and the change in human feedback is constant (Question 1). The authors tune this constant for each environment. But, more complex environments might induce different delays as the human observer might need to think about what they saw. 2. People are able to provide constant feedback (Question 2). This might not be true for more complex environments where certain states might have ambiguous values. 3. The human feedback is Markov (Question 3). This might not be true in more complex games. ## Detailed comments: * Equation 1 should have Q instead of A. Unless you want to define an advantage function A. * Equation 2 should have an upper-case Q. * The term $R_{t+k+1}$ in Equation 2 is not very clear. * The meaning of “We follow recent advancements in neural architectures and hyperparameter designs” on line 125 is not clear. * The rest of the paragraph on lines 125 - 127 is superfluous. ## Minor comments: * Inconsistent spacing between text and citations. * Calling this approach a “computational framework” might be a bit redundant given the context of the conference.
Questions
1. Is it reasonable to assume constant reaction time of the human observer? 2. How would your method deal with games with more ambiguous state values, such as Montezuma's Revenge? 3. The human feedback regressor $\bar{H}(s, a)$ uses the Markov assumption, but the human feedback might be dependent on their memory of prior states. What if the human feedback is non-Markov?
Rating
5
Confidence
3
Soundness
3
Presentation
2
Contribution
3
Limitations
Limitations are partially addressed.
Dear Reviewer, Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above. We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts! Best regards, Authors
Summary
The paper proposes a new framework for human-in-the-loop reinforcement learning, where the human and provide real-time and continuous feedback, and an algorithm where the learning agents uses the human feedback to accelerate policy learning. The paper conducted a user study of 50 subjects to demonstrate the effectiveness of the proposed framework in accelerating policy learning and improving success rates over RL and human-in-the-loop RL baselines. Optionally, a human feedback simulator can also be trained to mimic human feedback after a certain amount of time, reducing the amount of human input.
Strengths
- The paper proposes human-in-the-loop RL where the human can provide real-time continuous feedback, which is a novel paradigm compared to mainstream existing work which focus on discrete feedback signals. - This work conducted a user study of 50 subjects, which is the largest among relevant works. This is a great contribution in assessing the effectiveness of human-in-the-loop RL. - The evaluation is done on three challenging tasks, and GUIDE outperform all the baselines by a large margin on the "find treasure" and "high and seek" tasks. - The paper provides a detailed individual difference characterization by conducting a series of human cognitive tests. Analysis of the human cognitive test data provides meaningful insights. These data can also be very useful in future work.
Weaknesses
- The baselines are generally quite weak. Based on the experiment results, it is unclear whether real-time continuous feedback is necessarily the best way for humans to guide the policy learning. There might be intermediate points on the spectrum of conventional discrete feedback and full continuous feedback that provides the best tradeoff between amount of human input and effectiveness of guiding policy learning. - Whether the simulated human feedback is helpful is unclear. In both the "bowling" the "find treasure" task, the score does not increase much after switching to simulated feedback. It might be the case that the simulated human feedback only works for tasks where it is straightforward to model the reward.
Questions
- How does human-in-the-loop compare to BC or reward engineering for the same amount of human input? - How are the three tasks for evaluation chosen, and what are the motivations for such choices? One motivation for doing human-in-the-loop RL is for tasks where designing a reward function or providing demonstrations are difficult, such as the backflip task. However, for the tasks examined in this paper, there seems to be other simple ways for the RL agent to learn from human input.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
2
Limitations
The paper focused on the final success rate of policy learning, but did not provide sufficient data from the user's perspective. For example, the user study failed to include a survey regarding whether the real-time feedback system feels easier to use than the discrete feedback system.
Dear Reviewer, Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above. We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts! Best regards, Authors
Summary
This paper proposes a new framework, GUIDE, for learning from continuous human feedback in complex decision making domains with continuous action spaces. By framing human feedback as a state-action value function, the framework proposes to learn this function and to combine it additively with the (generally sparse) reward coming from the environment. The feedback is collected in a continuous fashion by asking participants to move their mouse up or down to indicate higher or lower feedback values. In a user study, the paper finds that training agents with this type of feedback yields better performing agents than baselines. After assessing participants in a suite of cognitive tests, it finds that participants that score higher on the cognitive tests trained better agents.
Strengths
The paper introduces an interesting new way of collecting continuous human feedback, and shows significant improvement in two out of three tasks considered. The use of cognitive tests as part of the user study is interesting, and uncovers insightful correlation between subject performance and their cognitive test scores.
Weaknesses
The assumptions regarding what human feedback represents do not seem consistent between section 3 and 4 (see Questions). Further, the treatment of the feedback collection is rather simple (added to environment reward function) and, especially if it does represent a signal regarding the future value of state-action pairs, heuristic. Relative to Tamer and Deep Tamer, which treated human feedback more consistently by using it directly as a proper state-action value function, this paper feels like a regression on that front. The implementation of the c-DeepTamer baseline raise a number of questions (see Questions), which shake my confidence in it as a baseline, or as a proper representative for how well Deep Tamer should perform here.
Questions
In 3.2 you propose to extend Deep Tamer by using the human feedback estimator F(s,a) as a critic within an actor-critic framework. Thus, you treat human feedback here as an estimate of the future value of each state-action pair (a Q-value). In 4.3, you propose to use the human feedback signal as a reward, to be added to the environment reward. Does this mean you make a different assumption regarding human feedback in 4.3. than in 3.2? Or do you propose to use feedback on future value as a reward? I am confused by your claim that Deep Tamer uses DQN. Looking at algorithm 1 in the Deep Tamer paper, it’s policy greedily chooses the action that maximizes the learnt human value function H(s,a). I do not see any use of DQN. Did your c-DeepTamer baseline receive the same (sparse) environment reward as GUIDE? And if so, how did you use this reward signal in conjunction with a critic that is trained from human feedback only? Why was the training of c-DeepTamer stopped after 5/10 minutes? Would it not be possible to continue training it using simulated human feedback (similar to GUIDE) or without human feedback? Within the user study, did participants get time to familiarize themselves with each environment before data recording began? I imagine participants would have needed some time to figure out how to solve each task. Is equation (3) correct? What is the purpose of he additional $- F(s, A(s))$ term?
Rating
4
Confidence
2
Soundness
2
Presentation
3
Contribution
2
Limitations
Limitations are discussed in the conclusion.
Dear Reviewer, Thank you again for your detailed review of our paper. We aim to try our best to address all your concerns with our point responses above. We truly value your feedback. As the end of the rebuttal period is approaching, please feel free to let us know if you have any additional questions or comments. We would be happy to answer them. We look forward to hearing your future thoughts! Best regards, Authors
Thank you for your detailed response. Unfortunately I continue to see two weaknesses in the paper. I like the proposed mechanism for giving feedback. It is elegant and to my knowledge novel. Reviewer JtyH has a point by saying that complex RL tasks will need more significant amounts of feedback, but in my view any interactive reward learning method would. This could have been explored more in the paper. The idea of combining environment reward with human feedback, however, cannot be counted as a novel contribution considering earlier work in this area [A,B]. Overall, the technical contribution is thus limited. A very solid set of human subject experiments would have been able to compensate for this, but in my view the experiments presented at this point are not quite complete enough. The c-DeepTamer baseline is artificially limited. My best understanding is that each bit of feedback was used only once in a gradient update, contrary to standard ML practice (please correct me if I'm wrong). Further, c-DeepTamer is not able to make use of the environment's reward, raising questions about whether it is a fair baseline in the first place. Why not use a baseline like the [A,B] that could use the environment's reward? The paper could also have been made much stronger by comparing to alternative interactive methods for learning from humans, including learning from preferences [C] and learning from demonstrations (e.g. [B]). I appreciate that the authors have clarified some of my concerns, and will raise my score accordingly, but I still feel that the paper falls short of the acceptance threshold. [A] Xiao, Baicen, et al. "FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback." Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems. 2020. [B] Brys, Tim, et al. "Reinforcement learning from demonstration through shaping." Proceedings of the 24th International Conference on Artificial Intelligence. 2015. [C] Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in neural information processing systems 30 (2017).
Thank you for your response. We are glad that the reviewer found the feedback mechanism to be elegant and novel. We would like to address all of your concerns and questions below. ---- > “I like the proposed mechanism for giving feedback. It is elegant and to my knowledge novel. Reviewer JtyH has a point by saying that complex RL tasks will need more significant amounts of feedback, but in my view any interactive reward learning method would. The idea of combining environment reward with human feedback, however, cannot be counted as a novel contribution considering earlier work in this area [A,B]. Overall, the technical contribution is thus limited.” We agree that prior work has also used human feedback for reward shaping. However, we believe that our main contribution is using such a method to ground real-time dense continuous feedback and conduct large-scale human experiments to verify it. We will add relevant discussion of this literature in our revised manuscript. > “in my view the experiments presented at this point are not quite complete enough. The c-DeepTamer baseline is artificially limited. My best understanding is that each bit of feedback was used only once in a gradient update, contrary to standard ML practice (please correct me if I'm wrong).” To maintain a fair comparison, c-DeepTamer strictly follows Deep TAMER in the way gradients are updated. As described in the original Deep TAMER paper Algorithm 1, the gradient updates happen: (a) whenever the human provides new feedback, using the new feedback information as the data sample; and (b) at a fixed rate, using data sampled from the replay buffer. We believe that a stronger solution would be able to update the network as standard ML practice, which is exactly one of our contributions beyond the original DeepTamer work. However, if we modify the original DeepTamer work to include our contribution, our experiments will become an ablation study of our own method instead of comparing it with prior work. Considering the expensive 50 human studies, which already cost 2 hours per subject and $1,000, running ablations is unfortunately not feasible. Our choice is to focus on comparing the previous literature in the same setting as our problem. > “Further, c-DeepTamer is not able to make use of the environment's reward, raising questions about whether it is a fair baseline in the first place. Why not use a baseline like the [A,B] that could use the environment's reward? The paper could also have been made much stronger by comparing to alternative interactive methods for learning from humans, including learning from preferences [C] and learning from demonstrations (e.g. [B]).” We agree that additional baselines would be interesting to compare with. However, experiments involving real-time human interactions are costly to run. Our current 50 human experiments include three environments, two algorithms, cognitive tests, and initial and ending setup, as traditional human experiments already cost nearly 2 hours. The total cost of the experiments is $1,000. Longer experiment sessions will not only be more costly but will also affect human feedback quality. Under the time constraint, we chose the baseline that had the closest setting to ours. Deep TAMER is the state-of-the-art in real-time human-guided RL through scalar feedback of state-action pairs. [A] is not real-time, as the human operator provides feedback to trajectories sampled from a replay buffer. They also only provide feedback to actions or states instead of state-action pairs. As the reviewer has mentioned, [B] learns from demonstration, and [C] learns from preference, which are different settings from learning from feedback. We would like to clarify that our work did not argue that real-time continuous human feedback is the optimal modality to provide feedback. In particular, we did not argue that this modality is better or can replace demonstration or preference learning. For demonstration, it has a different assumption that humans may be experts in the tasks, and this does not fall into our problem setting. In fact, we observe that humans typically have difficulty performing very well in our challenging hide-and-seek task where things change very fast, and the observations are partial. Preference learning is offline, which assumes multiple rollouts are provided, and humans do not need to react in real-time. It is often applied to the whole trajectories instead of state-action pairs. Our real-time continuous human feedback proposes a different problem setup and can be an alternative modality. However, we leave the exploration of multiple modalities as future work. Therefore, from both the perspective of financial, time and human costs and the problem-setting perspective, we did not compare the baselines from different modalities or problem settings. Our experiments indeed show significant improvement over real-time discrete feedback, which supports our claims.
I thank the authors for addressing my questions. I increased the confidence score to 4. Considering the overall contributions and limitations of the paper, I will keep my other ratings.
Decision
Accept (poster)