Learning to Influence Human Behavior with Offline Reinforcement Learning

Summary

This paper presents an investigation for learning to influence suboptimal human opponents in agent-human interactions. It claims that by performing offline RL on human-human interaction data, an agent can learn to influence its opponent's actions and latent strategies, even if there is no explicit information of influence in the data. In the first set of experiments, the authors showed that CQL achieved higher test returns when paring with human opponents, and in the second set of experiments, they showed that decoding latent representation for states performed better than two other opponent modeling technique.

Strengths

1. The paper is well-written. Materials are organized in a way that facilitates reading, and the connection to existing literature is clearly stated. 2. Learning to influence human opponents, especially those suboptimal ones, is an important problem for human-agent interactions. 3. The idea of learning influence strategies from human-human data is interesting and intuitive.

Weaknesses

1. There lacks explicit empirical evidence to support the claims made in this paper. The authors claimed that: (a) agents can learn to influence their opponents' actions using offline RL (the CQL algorithm), and (b) agents can learn to influence their opponents' latent strategies. However, there is no direct quantitative evidence of the increase in such influence. The reported improvement in test return can be explained by many factors. For example, rather than influencing its opponent, an agent may proactively solve the task itself. The only evidence is four states of a trajectory, yet it is not sufficiently annotated for readers. 2. The presentation for modeling latent strategies is confusing. This paper does not offer a tangible description of the "latent strategies". From eq (1), the "latent strategies" seem to be vectors that improve the prediction of the next states. But eq (1) seems not characterize the relationship between these vectors and the actions of the opponent. So I agree that they are latent vectors that improve policy learning, but I cannot see why they are strategies of human opponents.

Questions

1. Since this paper considers a multi-agent setting, I wonder how the MDP formulation in sec 3 covers the information of the opponents. In particular, do you consider the opponent's actions as part of the states or part of the actions? 2. Improvements in test return can be explained by multiple factors. To explicitly show that offline RL shapes human opponents' actions or strategies, more explicit evidence is necessary. For example, what about reporting the action frequencies of humans?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

1 poor

Presentation

3 good

Contribution

2 fair

Limitations

Yes, they have.

Summary

This paper presents an investigation into the application of offline RL on a dataset of human-human interactions to develop a policy capable of influencing human behaviors towards desired outcomes. The authors demonstrate that the learned policy not only affects immediate behavior but also influences the long-term strategies and preferences of the human partner. Experimental results provide strong evidence for the effectiveness of offline RL in learning to influence humans and achieving desirable rewards.

Strengths

1. This paper is well-written and easy to follow. 2. The utilization of offline RL on an existing dataset of human interactions is a straightforward yet effective approach. 3. The problem of learning to influence human behaviors from existing datasets is both interesting and important for the field

Weaknesses

1. The experiment solely employs offline RL methods as baselines. It would be beneficial to compare offline RL methods with approaches that leverage both offline data and online interactions. For example, one potential baseline could involve first acquiring a learned human model via behavior cloning and subsequently learning an adaptive policy to cooperate with the learned human model in the simulator, as demonstrated in [1]. 2. The evaluation appears to be limited to assessments involving human participants. It would be valuable to evaluate the learned agent against a learned human model or scripted policies, as demonstrated in [1] [2]. 3. The claim that there is no evidence of influence between humans, coupled with the assertion that the learned policy can induce long-term changes in human strategy, may be overly conclusive and requires further substantiation. Reference: 1. Strouse, D. J., et al. "Collaborating with humans without human data." Advances in Neural Information Processing Systems 34 (2021): 14502-14515. 2. Yu, Chao, et al. "Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased. ICLR 2023.

Questions

1. The authors state that the dataset contains no evidence of influence. However, how can one differentiate between behaviors resulting from mistakes and those intentionally performed by the human player to influence their partner? 2. The data collection process outlined in Section 5 involves various approaches. What impact does the dataset have on the learned policy? Would the learned policy differ significantly if only the first or second set of instructions were used for training? 3. Section 6 claims that the learned policy can induce long-term influence on the latent strategies of the human partner. Besides the average episode reward, is there any additional evidence supporting the assertion that the learned policy genuinely influences humans in the long term? The case study presented in Figure 6 and Lines 390-393 demonstrates short-term behavior influence, where the agent blocks humans from picking up onions to encourage tomato selection. My question is whether this influence is truly long-term, meaning, would the human player exhibit a stronger preference for picking tomatoes after repeated instances of being blocked from picking onions by the learned agent? Clarity issues: 1. In Line 208, the term "naive offline RL" is not immediately clear. Please provide a more precise explanation. 2. Regarding Figure 5 and Figure 6, it would be better to mark which chef is controlled by the human player and which chef is controlled by the learned policy.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

This work currently focuses on relatively simple environments. It would be promising to scale the method to more complicated systems.

Summary

This work aims to provide some evidence that offline reinforcement learning techniques can be used in the field of human-agent collaboration to influence or improve the behavior and underlying strategies of humans. The authors first verified that agents trained by CQL can influence the behavior of human players in some scenarios. Then, by simply modifying the CQL by conditioning on human latent strategy representations, the agents can learn to adapt to changes in human behaviors.

Strengths

The research purpose of this work is an important topic in the field of AI, how to improve human performance in human-agent collaboration. The author's attempt to verify the effectiveness of offline reinforcement learning techniques in human-agent collaboration scenarios is commendable.

Weaknesses

1. After carefully reviewing the manuscript, I personally think the current content (methods and experimental results) does not effectively demonstrate the thesis that offline RL can learn to guide humans toward better performance by combining human latent policy representations. I agree that the agent may learn to adapt to changes in human behavior through offline RL, but it remains unclear how offline RL can tangibly influence and improve human performance. Furthermore, there is a wealth of research in the field of human-agent collaboration, such as Du, et al.[1] and Alamdari, et al.[2], focus on developing assistive agents to improve human performance. It would be advantageous to discuss this research in the paper to provide a more comprehensive understanding of the field. In the experimental part, the existing experimental results (improved team rewards & few examples) cannot support this assertion either. I suggest that the authors provide some objective metrics of human participant performance to make the conclusion more solid. 2. User Study is crucial in human-agent collaboration research, and I’d suggest the authors provide additional information, including: - Ethical review: Did participants provide informed consent for their involvement in this research and for the use of their data toward this project? Were they fully informed about the purpose of the research, did they confirm their approval for involvement, were they told about how to withdraw their data? - Test settings: What is the proficiency of the participants in the game? Were they all novices or professionals? Whether to provide a standard test specification or guide before the test to ensure the consistency of the test purpose? 3. Recent research in the field of human-agent interaction has highlighted the significance of evaluating agents using both objective and subjective metrics. See Strouse, et al.[3], and McKee, et al.[4]. Incorporating subjective metrics into the evaluation of agents can provide a more holistic understanding of their impact on human performance and well-being. For example, did participants prefer playing with the trained agent over other baseline agents? During the collaboration, did the participants perceive their actions to be influenced by the agent's behaviors? If yes, to what extent? Did all participants perform better as a result? 4. This work attempts to verify the effectiveness of existing offline RL techniques in some human-agent collaboration scenarios. Due to the lack of comparisons with state-of-the-art methods in the field of human-agent collaboration, such as Strouse, et al.[3], Yu, et al.[5], etc., it remains unclear where the boundary of offline RL in human-agent collaboration is. 5. There are some unevidenced and unsubstantiated claims in the manuscript. For example, - In line 208, "It is evident that naive offline RL cannot learn adaptive policies." - In line 179, "humans are passive and will often only respond to what their partner is doing" 6. Minor Concerns: - Many references are not uniformly formatted, such as [27] "nature" and "Science" , whether conference abbreviations, e.g.: "(ICLR)", are reserved. - In line 369, "Thereofre" should be "Therefore" - In line 412, "simulataneously" should be "simultaneously" [1] Du, et al. Ave: Assistance via empowerment. 2020. [2] Alamdari, et al. Be considerate: Avoiding negative side effects in reinforcement learning. 2022. [3] Strouse, et al. Collaborating with Humans without Human Data. 2021. [4] McKee, et al. Warmth and competence in human-agent cooperation. 2022. [5] Yu, et al. Learning zero-shot cooperation with humans, assuming humans are biased. 2023.

Questions

See the revision recommendations in the "Weaknesses" section.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

As discussed above, this work lacks the comparison of SOTA methods, and the user studies are not sufficient, which would limit its reliability.

Summary

This paper proposes a novel framework to empower an intelligent agent with the capability to effectively influence suboptimal human behavior during interactions. The primary objectives revolve around tackling two key challenges: (1) deducing a new strategy to influence human action and (2) learning to influence the human's long-term latent strategy. To evaluate the proposed framework's efficacy, a set of collaborative tasks (i.e., Overcooked environment) are employed. The experimental results demonstrate that the agent can successfully learn to influence human behavior based on a human-human interaction dataset, even though no instances of successful influence were present.

Strengths

Generally, I find this paper to be well-motivated, accompanied by clear and easily understandable examples. The organization and presentation of the content are well-structured, making it easy to read and comprehend. The focus of the paper lies in addressing the critical issue of enabling intelligent agent to influence suboptimal human behavior which is expected to be a topic of significant interest in the human-AI interaction research field. The proposed approach to employing offline reinforcement learning to tackle the problem is innovative such that it successfully extends the application of offline reinforcement learning in a novel context. The evaluation section is comprehensive and thorough, providing compelling evidence that supports the efficacy of the proposed method.

Weaknesses

First, for the example in line 66, if we consider the real world scenarios, such behavior (i.e., repeatedly blocking the human from doing something) without providing any explanation would largely affect the human trust in the robot. This example appears to assume that the human will naturally respond by adjusting their strategy to avoid that ingredient in the future. However, this implicit assumption is quite strong and may not hold true in real-world situations. As the authors mention in the last section, the experiments were conducted on simple simulated game-like environments rather than real world scenarios. It would be intriguing to see how the proposed approach performs when applied to more complex real-world tasks involving human-AI interactions. Furthermore, when we talk about human-AI interaction, it includes both collaborative and competitive interactions. However, the scope of the paper is focused solely on collaborative settings, without investigating or evaluating the approach in competitive scenarios. Expanding the research to encompass competitive interactions may provide a more comprehensive understanding of the proposed method's versatility and potential applications. Finally, in the evaluation section, the authors compare the performance between BC and offline RL methods. I feel that the paper could be further strengthened by incorporating a comparison with prior works that approximate human decision-making models and subsequently learn to influence human behavior using the learned model. This broader comparison would enhance the paper's context and shed light on the relative advantages and contributions of the proposed approach in the context of existing research.

Questions

1. When the agent tries to influence human behavior, is it possible that the human gets annoyed, particularly if the influence becomes excessively insistent, dominating, or aggressive? Such a scenario could potentially lead to negative reactions from the human and undermine the effectiveness of the interaction. 2. In the provided example, the authors mention the human goes to plate when he sees the plate is right next to him. This situation seems to assume that the human has a myopic perspective. I am wondering whether the proposed approach is generalizable if the human suboptimal behavior can be attributed to other various possible reasons. 3. The agent's objective is to elicit the expected human behavior. However, the agent may generate positive action (e.g., pass the plate) or negative action (e.g., block the way). These actions may significantly impact human perception and trust. How can you ensure that the agent influences human behavior effectively without causing harm to the team? 4. Does the agent's active influence imply a leadership role, as it takes on a guiding role in shaping the collaborative decision-making process?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

I appreciate the authors for the limitation and ethical implications section in the paper. Additionally, given the paper's objective of enabling the agent to influence human behavior and long-term strategies, it is essential to consider potential impacts on human trust and the possibility of unintentionally biasing the human towards a worse policy. These factors warrant serious consideration to ensure that the proposed approach fosters responsible and ethical human-AI interactions. Addressing these concerns would contribute to a more comprehensive understanding of the potential implications of the proposed method.