Summary
This paper studies the strategic multi-agent problem setting of two LLM Players interacting in a negotiation (or bargaining) game and proposes to use feedback from an LLM Critic to improve each Player’s expected behavior and performance in the game. Importantly, the paper aims to study how AI Feedback can enable Player improvement when playing competitive games under well-defined rules. The proposed method is to use the player dialog history and critic feedback as “in-context demonstrations” for players to self-improve over the course of the game. Experiments are conducted on a bargaining game instance negotiating the price of a balloon, investigating several LLMs as base models for the Players (and their Critics).
Strengths
Identified strengths of paper:
- I really like the premise of this work: that agents playing in a strategic game may each be able to benefit and improve their negotiation strategies, not only by observing their opponents, but additionally by heeding the advice of personalized critics whose aim is to help (cooperate with) them. Like other AI feedback methods, this type of approach has the potential to scale better than using Human feedback for the same type of assistance, presuming feedback is helpful and thus desirable for this negotiation task.
- The topic is timely and very relevant to the multi-agent and game theoretic communities, as it takes a long-existing and well-studied problem (multi-agent negotiation) and investigates how rational player strategies can be improved using AI feedback and LLMs.
Weaknesses
Identified concerns and suggestions to improve the paper submission:
- It would be extremely helpful to see examples of what the *instructions* and *in-context learning* looks like for the Critic agents? Examples of their few-shot prompts (or instruction data for Finetuning) are important, as this is a key contribution of the work (adding AI Feedback to strategic reasoning/negotiation settings). Notably, where are the Buyer/Seller Critics getting the suggested negotiation strategies in Figure 1B from (e.g. the flinch technique, the anchoring technique, etc)? They seem to be good, general strategies to have in the player’s toolbox, but require the Critic to have knowledge of effective negotiations and how/when to employ them. Thus, how are the Critics coming up with general negotiation strategies and reasoning about when to apply them? This is important for opening the black box of the AI Feedback component of the proposed method.
- While I like the overall premise, the novel technical contribution of this work seems tenuous. With that, the problem setting is strongly inspired by AlphaGo Zero, but is there any traditional learning in this setting (i.e. any updates to the agent policies based upon the AI feedback given)? That is still unclear to me. Currently the paper seems to simply add a small amount of additional context in the prompt for the Player LLMs. If there is no traditional learning, this setting is also critically different from AlphaGo Zero in that the proposed feedback signal (from Critic Agents) is **not** used to update the Player policies. This should be explicitly clarified. Also, if there are model updates, it would be useful to see the learning update rules. If not, why not do any finetuning of the model? At least as an experimental condition to compare against and understand if in-context learning is "sufficient". Perhaps more motivation and context for this design decision would be helpful.
- This work explores a potentially interesting research direction (negotiation games or strategic gameplay more generally + AI Feedback for improved player strategies) but it’s not clear to me how well motivated AI feedback is for this negotiation game or how interesting the problem is in its current instantiation. In particular, it would be helpful if the paper provided some type of performance analysis to show that the negotiation game used (an instance of balloon purchasing) is interesting and challenging to solve *before* ever adding in LLMs or a Critic. How would existing multi-agent negotiation/bargaining approaches solve this game? What equilibria do the Players generally converge to without LLMs? How and why does using LLMs change the *expected* solution? Given two Player LLMs, why is a Critic *necessary* to have? In other words, if the Critic is being prompted with examples of bargaining conversations, could the Buyer/Seller agents not simply see those same examples and then converge upon the same solution (they are currently finding) *without* a Critic? Or is there some value that the Critic role *uniquely* adds (e.g. more efficient or robust convergence on an equilibrium in this negotiation game)? Motivating the game selected as an interesting and challenging problem in its own right is critical.
- Furthermore, I still question the generality and significance of the paper’s empirical findings. **RE Significance:** How significant and meaningful are the differences in Seller performance in Table 1? The numbers don’t seem that different to me, but perhaps with more context, the significance becomes more clear. With that, I see a distributional shift in Figure 4, but how meaningful is this shift? What are the implications of it, regarding how good or bad these solutions/equilibria are before and after receiving AI feedback? Do the solutions to the negotiation game simply change a small amount but are comparable in terms of how preferable/desirable they are or do they get qualitatively better in some way? **RE Generality:** The paper seems to use only one evaluation domain (balloon purchasing). It is not clear to me which empirical findings/trends from the use of AI Feedback on this one, seemingly simple bargaining domain, are expected to generalize. In particular, regarding the effectiveness and impact of AI Feedback in other multi-agent negotiation domains and settings. This limited evaluation seems more like an interesting case study than general findings that transfer.
- In Subsection 4.3, the paper provides analysis of each of the LLM models (gpt-3.5, gpt-4, claude-v1.0, etc) versus a fixed gpt-3.5-turbo opponent. However, a more thorough investigation examining the cross product of each LLM model as Buyer against every other LLM model as Seller (a *complete* Payoff matrix with all pairwise model comparison) could potentially provide more general insights. Why was this not done? Can it still be done to show a resulting Payoff Matrix? Not doing a pairwise comparison of *all* pairs of models might unnecessarily constrain the results and thus the insights that can be extracted.
Questions
Please see Weaknesses section.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Please see Weaknesses section.