Learning from Active Human Involvement through Proxy Value Propagation

Summary

The work introduces Proxy Value Propagation (PVP), an approach to reinforcement-learning without ground-truth rewards by instead using human intervention as a signal of quality. The value of a state-action pair is determined by a human intervention, where interventions signal that the agent's behavior was bad (and we assume that the human's action was good). Compared to naive RL approaches or other human-intervention methods, such as HG-DAgger, PVP learns self-driving policies much faster and with fewer crashes than all other approaches. The work is also tested with a user-study, gathering demonstration data from real humans.

Strengths

Technical contribution: * The proposed method works very well empirically, outperforming baselines with and without human intervention. * The relative contribution of different components of the PVP approach are tested with an ablation study in the CARLA domain. * Experiments cover several domains including continuous/discrete actions, different input mechanisms, and different state observations. * The proposed approach specifically considers balance between human/agent labels, making sure to consider human preferences even when there are relatively few demonstrations/corrections. Experiments: * The experiments are clearly described and appropriately test the proposed PVP method against baselines. * Human users are recruited and the human-intervention method is tested with real users, not synthetic/heuristic labeling. Novelty: * The paper compares well against prior work and the related works section covers most important previous approaches to the problem. * The experiments compare against several baselines in prior work. Clarity: * The method is intuitive and easy to follow. Figures and tables clearly communicate the results. * The supplementary material provides additional detail where useful (hyper-parameters, control schemes, etc.). Reproducibility: * Code is provided to reproduce all experiments.

Weaknesses

Technical contribution: * Prior work in human-labeling and intervention has found that human labelers are slow to react, and this time-delay must be accounted for (e.g. in [1], the authors needed to use eligibility traces to account for _which_ agent action was incorrect). Strange that this did not surface as a problem in these studies. * The proposed treatment of agent vs. human Q-values could result is serious policy degradation if the human labeler makes mistakes or if the demonstrator is sub-optimal. * Fusion of PVP with environment reward resulted in _worse_ performance. * Removing epsilon greedy exploration from DQN effectively forces humans to do the exploration for PVP, which seems to work well in the driving simulator, but could be an issue in other domains. User study: * Information on the study is lacking. How many participants? What about background in computer science or driving? Mean age? What sorts of instructions were provided? Were participants allowed to practice first? These details can be in the supplement if they are not critical to the analysis of the algorithm, but they should be included for a user study. Novelty: * The approach seems very similar to [1], which treats human labels as indications of advantage instead of simply as reward (as in DeepTAMER, which is cited in the paper). While comparison to [1] is perhaps not necessary (as the paper compares thoroughly to several state-of-the-art approaches), [1] should at least be mentioned. * Should be aware of other approaches to human-intervention labeling (e.g., [2], which also considered driving feedback, or [3] which performs BC and then RL) [1] James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, and Michael L. Littman. 2017. Interactive learning from policy-dependent human feedback. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (ICML'17). JMLR.org, 2285–2294. [2] Mariah L. Schrum, Erin Hedlund-Botti, Nina Moorman, and Matthew C. Gombolay. 2022. MIND MELD: Personalized Meta-Learning for Robot-Centric Imitation Learning. In Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction (HRI '22). IEEE Press, 157–165. [3] Cheng, Ching-An, Xinyan Yan, Nolan Wagener and Byron Boots. “Fast Policy Learning through Imitation and Reinforcement.” Conference on Uncertainty in Artificial Intelligence (2018).

Questions

1. Is there any drawback to the proposed approach to action labeling when compared to something like COACH [1]? Should PVP be considering the time-delay of human feedback? 2. Line 187-188 states that the novice-policy (agent behavior) contains information of the forward dynamics. What does this mean? 3. The addition to the CQL loss is likened to an L2 regularization on the Q-values from the agent, but later experiments adding this L2 penalty to CQL seem to perform much worse than PVP. Why is there a discrepancy between the two? Shouldn't CQL + L2 == PVP, as in Equation 6? 4. It seems that PVP relies heavily on humans doing the initial exploration. Would something like LOKI [3] perform comparably, or is the value propagation part of the key to success? 5. Please address the questions listed above concerning the user study. Additional detail should be provided.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

The authors list and acknowledge several limitations. Mitigations to these are left to be the subject of future work. Societal impact of the work is not explicitly addressed, though an ethics statement does appropriately address IRB for the study.

Summary

The paper proposes an approach for learning from human interventions based on offline RL with pseudo-rewards. The paper assumes a shared autonomy setting where a human monitors an agents rollouts and can intervene whenever deemed necessary and provide corrective demonstrations. The algorithm relabels those interventions with positive Q value targets and the faulty policy actions that led to intervention with negative Q targets. It then trains the policy on these pseudo-rewards with offline RL. In extensive experiments on multiple driving simulators with differing human input devices, the approach leads to faster and more successful learning of desired behaviors than prior RL and learning-from-human-intervention approaches.

Strengths

- the paper's approach is elegant and simple, it leverages common offline RL techniques in combination with pseudo-reward labels and, due to it's simplicity, seems easily applicable to other domains - the empirical evaluation of the approach is comprehensive, with numerous baselines spanning the relevant related works, multiple environments and many meaningful ablations - the paper is well-written and easy to understand. it has intuitive visualizations for illustrating the approach, evaluation environments and qualitative results.

Weaknesses

(A) **Concerns about training objective**: the training objective sets the **Q-value targets** of the human intervention trajectories to +1 and the Q-value targets of the policy actions in those intervention trajectories to -1. I have a few concerns about this objective: (1) by setting the Q-value target instead of setting the reward, the objective does not allow reward propagation through these updates, e.g. from post-intervention states. At the same time the algorithm runs "regular" Q-updates with full propagation and adds both objectives. Why not set **rewards** instead of Q-targets to indicate good and bad behavior and only run the regular Q-update on all data samples? That seems more elegant than the current two-part objective (2) the current objective assigns low Q-value to all policy actions during human intervention. However, the human intervention only provides a signal that the sequence of policy actions **before** the intervention were bad. It would for example be possible that the human "over-intervenes" and keeps providing demonstrations in states in which the policy's actions are already good again. In these cases the current objective would penalize good policy actions (B) **Only navigation environments**: while the experimental evaluations are comprehensive, they are limited to navigation environments (driving and grid world navigation). It would strengthen the paper to show the same algorithm can work for teaching robot manipulation tasks, e.g. by performing teleoperation in a simulated robotic environment. (C) **Unclear human user experiments**: all experimental evaluations are performed with real humans in the pipeline. While this is generally positive, it means special care needs to be taken to ensure fair comparability between all methods, since different users may interact with systems differently, they learn and adapt to systems over time etc. Thus, human user studies require great care in execution and documentation, eg ensuring that the same users interact with all methods and baselines (to account for differences between individual users), the order of methods is randomized and all users are given some time to practice with the system (to account for adaptation effects) etc. The paper does not provide any details in this regard for the quantitative evaluations in Tables 1, 2 and 4. It would be important to provide more details on the design of all experiments involving human users to ensure fair comparison.

Questions

(Q1): How could the proposed method be combined with pre-trained policies or ground truth rewards? --> the current method requires lots of human teaching, it would be beneficial if human teaching was only used to *supplement* autonomous agent learning from some external source of supervision. The one experiment in the paper that does combine it with external rewards seems unsuccessful -- how can this be fixed? (Q2): How can the proposed method be extended to situations where humans cannot directly provide demonstrations? --> eg for an ant robot it is impossible for humans to provide demonstrations (the authors mention this short-coming of the current approach in the limitations section). I am wondering whether you have any ideas how this could be extended to situations like the ant robot?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

- add more details about human experiment design - add a (simulated) robotics manipulation environment - clarify concerns about the proposed objective ## Summary of Review Overall, I think this is a strong paper. It combines a novel, yet simple and elegant approach with an extensive evaluation and comparison to numerous prior works. While I would like the authors to address the points raised in my review, I am happy to accept the submission.

Summary

This paper focuses on human-in-the-loop for reward-free policy learning, wherein a human has the option to override the policy by taking over control when the agent attempts to perform risk behaviors. To solve this problem, this work presents Proxy Value Propagation (PVP), which directly assigns positive values (+1) to the actions induced by the human interruption and negative values (-1) to the actions the agent takes in specific states. By manipulating the values, PVP encourages the agent to approximate human behaviors while avoiding the actions intervened by human.

Strengths

- The problem was well-motivated and the setting is realistic. - The empirical results suggest PVP outperforms a set of baselines involving imitation learning and other human-in-the-loop methods in terms of multiple measurements. Furthermore, the experiments with human subjects are convincing and extensive. - The proposed method is simple and efficient.

Weaknesses

One major concern is that the technique of directly manipulating the value seems to be not well-motivated. As claimed in lines 178-180, the optimal policy should approximate the behaviors of human subjects while avoiding performing the intervened actions. Starting from this motivation, two natural ways are 1) assigning high rewards to human actions and low rewards to intervened policy actions, and 2) constraining the policy by manipulating the likelihood of the specific actions. Since the value function induced by TD learning measures the long-term performances starting from specific state-action pairs, directly assigning +1 or -1 to the value function seems to tell the agent that it tends to succeed or fail starting from the s-a pairs. However, human might only take control at some discrete time steps, and the subsequent behaviors after the interruption are unpredictable. Thus, the natural design is to assign manufactured rewards (+1/-1) to the specific transitions from my perspective. Though I believe that directly manipulating the value and propagating through TD-learning might be a better choice, it is worth comparing with the naive baseline mentioned above and further discussing the advantages of PVP. Though learning with online human interruptions is quite an interesting setting, it is pretty ideal to me as online learning is risky and expensive even with a human guardian in several domains (e.g., robotics). In terms of the driving case investigated in the paper, I believe that offline learning with human interruption datasets might be more realistic and interesting given numerous in-the-wild datasets. However, I would agree the investigation is out of scope concerning the current paper.

Questions

- If the reward function is set to zero, why $Q=\sum r_t$ can be set to $+1$ or $-1$? - For two adjacent time steps in an episode, if both $Q(s_1,a_1)$ and $Q(s_2,a_2)$ are set to 1, how can it satisfy Bellman equation as $Q(s_1,a_1) = \gamma max_a Q(s_2,a)$?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper introduces a reward-free approach termed Proxy Value Propagation (PVP) to facilitate safe and human-aligned reinforcement learning (RL). PVP assigns high Q values to human actions (the state-action pairs in the human demonstration) and low Q values to agent actions that necessitate human intervention. The authors conduct experiments across diverse environments, offering a comparison with existing RL and human-in-the-loop (HL) techniques, as well as the results of user studies.

Strengths

1. This paper is well-written and readable, and provides detailed appendices and codes to supplement the paper's algorithms, experiments, and implementation details. 2. This paper presents a reward-free approach to improve learning from human active involvement, effectively utilizing all data generated during the human intervention, including both agent and human-generated data. 3. The proposed method can be extended to various value-based RL Methods.

Weaknesses

The training process of PVP seems to rely heavily on human involvement, whether through observation or intervention, which can be quite costly, especially when scaling up to more complex scenarios. However, this may not be an issue if the cost is lower than designing safe rewards and the performance is superior to basic RL methods. Nonetheless, the experimental results and settings in the paper leave me puzzled, and I am unable to determine if the results sufficiently support the conclusions. 1. I am unsure if the experimental setup for Base RL Methods is fair. Why is the Episodic Return so high, but the Success Rate is lower than that of humans? For instance, GT Sophy [1], a race car AI trained using Deep RL by Wurman, Peter R., et al., has already surpassed human champions. Additionally, Imamura, Ryuji, et al. [2] mention that "all agents learn the policy in approximately 400 epochs" and "its score places it among the top 10% approximately 28,000 human players." Can the authors provide more detailed descriptions of the rewards for Base RL Methods? Although the reward settings are briefly described in the appendix, the calculation methods and weights for each reward are not listed. Unreasonable rewards may affect the performance of Base RL Methods and the fairness of comparisons. 2. PVP can be considered as providing dense training signals to the agent, as the agent's training relies on human judgment for every (s, a) pair. Even if the action is not intervened, it still depends on human judgment, thus providing a positive and accurate training signal, leading to faster convergence. For Base RL Methods, can designing more reasonable dense rewards and properly adjusting the weight of crash penalties achieve the same effect? Based on the evaluation metrics, human preference seems to be focused on reducing crash. Can manually designed rewards also express human intentions and preferences? The author should compare the cost of manually crafting the reward with the cost of human intervention, which can highlight the advantages of PVP. 3. I also have doubts about the experimental results for BC and GAIL. What was their training data? Why do human demos have a 97% success rate, while BC and GAIL have success rates of less than 1%? Are there any errors in the experiments? 4. I am also puzzled about the user study results in Table 3. According to the appendix, the scores can only be 1, 2, 3, 4, or 5, with an upper limit of 5 points, and PVP has an average score of 4.8. Why is there a standard deviation of 0.5 when the scores are so high? Are there cases with very low scores, and did the authors analyze these low-score cases in detail? Bad cases can be potential safety hazards. 5. The statement about PVP with reward results is unevidenced: "which might be caused by the fact that the native reward function might not be aligned with human preference." Success rate is a metric of agent capability, so why can't an agent's capability be improved if it is not aligned with human preferences? Is there evidence to prove that RL cannot surpass humans in this task [1, 2]? 6. Additionally, the description of human subjects is insufficient. What is the proficiency of the human subjects in the environments? Were they all novices or professionals? Was a standard test specification or guide provided before the test to ensure consistency in test objectives? 7. Have the authors compared the repetition rates of (s, a) and (s, an) in the Novice Buffer and Human Buffer? This might be closely related to the reliability of human subjects, and learning may not be meaningful for unreliable humans. 8. How much variation is there in (s, an) among different human subjects? When participants have different styles, how should this be handled? For example, if some participants think the current (s, a) is reasonable, but others think it needs intervention, how should this sample be treated? [1] Wurman P R, Barrett S, Kawamoto K, et al. Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature, 2022 [2] Imamura R, Seno T, Kawamoto K, et al. Expert Human-Level Driving in Gran Turismo Sport Using Deep Reinforcement Learning with Image-based Representation, 2021

Questions

See the revision recommendations in the "Weaknesses" section. The authors are encouraged to address the aforementioned concerns, provide more detailed experimental information, and incorporate these insights into subsequent revisions.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

This paper partially discussed the limitations, but there may still be some ethical concerns. It is worth considering whether the proposed method could be potentially abused by malicious human subjects, leading the AI to learn in an incorrect manner. Can malicious human preferences be detected in a timely manner, and the data be promptly removed?

Summary

The authors present Proxy Value Propagation for policy optimization. They model a proxy value function so that human intents receive high value (i.e. actions for which the human has inputted as corrective actions over a policy) while on-policy agents have low values for actions that caused an intervention. When optimized via TD-learning they are able to stitch together labeled values of demonstrated state-action pairs with unlabeled data from agent exploration (claimed). They execute the proposed algorithm on a number of environments (e.g. MiniGrid, MetaDrive, CARLA and the Grand Theft Auto 5 driving task) and show significant improvements over baselines.

Strengths

- Human-in-the-loop (with active human involvement) is one of the few tried-and-tested methods to fine tune imitation learning / RL methods on complex real-world domains. Improvements in this field are actually quite impactful to practitioners deploying these systems in production (unlike some other methods that rarely see use outside of the academic setting). To this end the problem is well posed and of interest to the community. - PVP is a conceptually simple algorithm. I'm pretty sure this paper is self contained and I have everything I need to re implement it. - The workflow of the algorithm is also practical. I can imagine using this algorithm in a real system with the scale properties such a system would require.

Weaknesses

- nit: "It is also compatible with different forms of human control devices, including gamepad, driving wheel, and keyboard" - True of baselines (and many other methods) as well? Or alternatively, this seems like an obvious point. - nit: line 128: "It is unrealistic to invite a real human subject to involve in such training." There are many real-world uses of Dagger. Be careful with over generalization. - I'm not sure the claim that unlabeled (s, a) pairs are stitched together with human expert data is all that well experimentally justified. Or put another way, if the formulation has some equivalency to CQL (with an additional L2 regularization term), in the limit that these regularizes are extremely strong, the method will just look like behavioral cloning. To claim that there is value propagation to unlabeled states I would want to see experiments to confirm this (because in my experience CQL does not actually stitch together sub-optimal trajectories as is often claimed). - Figure 2 results are quite compelling. Although you're comparing a random-exploration based RL method (and pretty vanilla one at that) against a human-in-the-loop method. This is apples to oranges. I'd also be curious to see what performance looks like against state-of-the-art offline RL or supervised imitation learning in these domains (there's some of this in Table 1, but not in this earlier teaser image).

Questions

- A collection of human demonstrations is assumed (line 143), however it's still not overly clear to me how this is used. Maybe this is already clear in the text but which Q function is actually pre-trained with this data? Presumably there needs to be some overlap between the agent and human preferences initially otherwise every state would induce a human intervention? Maybe spill out this warm up procedure in more detail. - It's also not entirely clear under what conditions the human is told to intervene? In continuous and high rate action and state spaces (particularly in driving applications), there is not one specific action that causes failure. Are humans instructed to let the agent diverge from an optimal trajectory by a specific metric amount (e.g. half a lane in the case of self driving)? Or is it up to the human operator under what conditions would they intervene? Is the method sensitive to such human ambiguity?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

First NeurIPS paper where I can comfortably say the limitations section is well done :-) I have nothing else to add. A societal impact statement doesn't seem necessary for this application / paper.

Possible typo in the reward function

It seems the collision term of the reward function is positive reward with a positive multiple (+1 for collision, multiple of 5). Is one of these terms supposed to be negative (either the reward or the scaling factor)? Otherwise it seems the car would be encouraged to collide with obstacles.

Thank you for your rebuttal!

Thank you for your response. I am generally happy with how it addresses my comments. I appreciate the explanations for my concern (A2) and the inclusion of details regarding the human user studies! I am still a bit unsure about question (A1), ie reward vs value targets. How is this different from off-policy RL where we need to learn from transitions collected with a different policy? Additionally, the failure case mentioned seems like it could be avoided by giving some strong penalty for human involvement that outweighs the benefits of the following good actions? This would encourage the policy to avoid interventions but still follow the good actions after? I would appreciate the authors insight on this question, but am in general happy to recommend acceptance for the submission.

Response

Given the response of the authors, I do think this paper is promising and reasonable. Nevertheless, I also think the mathematical interpretation of this method should be improved in future work.

Acknowledgement of rebuttal

Thanks for the detailed response and clarifying any misunderstandings I may have had. I also read through the responses to other reviews. I still think this is a very strong paper and will keep my score at Strong Accept.