LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play

Summary

The authors present an LLM discussion framework for improving the creativity of LLM outputs by using multiple LLM agents with diversified role-playing instructions in a 3-step pipeline.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

The idea of giving different role-playing instructions to different agents seems intuitive and well-motivated, and the method appears empirically effective at increasing creativity. Additionally, although they are evaluating creativity which (I think) should be a fairly difficult thing to nail down evaluation-wise, the experimental setup and analysis seem reasonable and sound, which I think is a solid contribution in its own right (and could perhaps be highlighted more if the authors want to).

Reasons to reject

While the authors have shown that their method can improve the creativity of LLM outputs, it does appear to come at some cost to fluency. In LM generation one can typically increase diversity trivially at the cost of fluency by e.g. increasing the temperature. It's not immediately obvious to me that the proposed method is not just a fancy way of doing the same tradeoff, rather than "pushing the Pareto boundary" in some sense. While the current experimental results are fine, in my opinion this paper would be substantially stronger if the authors showed results on e.g., something like a creative problem solving task where the additional creativity actually results in something like higher success rate or accuracy, rather than solely improving creativity for its own sake.

Questions to authors

- Why does your method often perform worse on fluency compared to some baselines? Is there a tradeoff for fluency vs diversity (similar to how you make this tradeoff when you change the temperature of generation)? - Why is flexibility not a main metric? Isn't coming up with more different answers a pretty reasonable thing to evaluate? Or is the problem just that you don't have a good way to "deduplicate" different answers? Not sure if I'm misunderstanding what is meant by flexibility in your setup. - Why are all the metrics 0 when you have 3 agents in fig 6?

Summary

The paper presents a method called LLM Discussion, which is a multi-turn multi-LLM method to attempt to extract more creative responses out of LLMs on common creativity "brainteasers" such as Alternative Uses. The work experiments on four types of tasks, and evaluates different conversational methods to extract creative answers. The LLM Discussion method is found to be more creative both through a robust human evaluation, and using an LLM as a judge, which is found to correlate satisfactorily with human judgments.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

- The work is thorough, considers prior both in education and recent CS work seriously, and discusses the implications and limitations in a nuanced and realistic way. - The LLM Discussion framework -- although somewhat evident -- is well constructed, particularly with the concept of the three phases of discussion: initiation, discuss & convergence. - The human annotation seems thorough, and the release of the collection of judgments might be useful to others studying creativity in similar settings. - The paper is generally well-written, and although some details are lacking, these can easily be resolved in a round of revisions.

Reasons to reject

- Nothing serious, overall the work is creative and rigorous. - The four tasks studied are known to be somewhat artificial. As the authors point out, the tests to evaluate creativity in humans might not be adequate to evaluate LLMs, since the latter can be infinitely verbose at very low cost. The authors point this out and attenuate the importance of certain TTCW metrics, yet the tasks themselves remain tasks that were created for humans, not LLMs. - There is a strong correlation between response length and perceived creativity, both in human & LLM evals. The work did not directly control for length of response (which seems like it would have been very easy to do...) and the examples shown in Figure 4 unfortunately show that LLM Discussion outputs are much longer. Are the promising results therefore only reflecting the underlying signal that responses in that condition are significantly longer on average?

Questions to authors

Q1: Can you report the average length of response for each task and each approach? This would help explain to the reader whether length explains away the creativity difference or not. Why did you not include in your prompt an expected response length, to normalize for this important effect? Q2: Besides the tasks you've collected, whose intended audience are humans under a time constraint, do you have a sense of what tasks could be more appropriate to evaluate LLMs' creativity abilities? Do you believe that the "number of turns" reflects a similar effect to human test-takers having "limited time" in TTCT? Q3: Two of the core contributions in LLM Discussion seem to be: (1) the three phases, (2) the role-playing. Yet, neither of these two elements seems to have been ablated. Do you have experimental evidence showing that such elements indeed lead to improved creativity in the responses? Q4: It is not 100% clear in the paper which LLM is used in experimentation. The draft clearly states which GPT model is used for auto-eval, but not for generation. Are you using GPT4? Are the different roles all played by the same underlying LLM, or different LLMs? Does that make a difference? Q5: It is not 100% clear in the paper how the "Convergence Phase" is conducted. It seems logical that only one LLM would perform the final agglomeration based on the conversation so far. How do you pick which "role" gets to do the final response? (e.g., do you get the final response from the environmentalist or the social entrepreneur). Q6: Temperature plays a strong role on the diverging nature of an answer. What temperature setting did you set the generation models with, and do you expect increased temperature to further improve creativity scores? Minor notes: - It seems a bit of a stretch to call tau correlations at ~0.3 "strong correlation" - This reviewer appreciated the pun in the title of Section 6, kudos :)

Summary

The presented approach gets much better results on key measures of creativity compared to SOTA. This paper is a well-written presentation of the novel LLM Discussion approach to generating creative responses on established creativity and scientific innovation benchmark tests. The approach is a highly original simulation of "collaborative discussions with diversified peers" that involves (i) a three-phase, initiation-discussion-convergence framework for multi-turn dialogues designed to elicit diverse and creative responses from (ii) LLM-based agents, each of what are assigned distinct roles and are directed to generate answers building on the others' responses. This framework, (i), is compared against single-LLM and multi-LLM approaches and outperforms these previous methods on several creativity metrics. The role-playing LLM-based agents, (ii), cleverly leverages the ability of LLMs to imitate very different personalities in order to generate more diversity of ideas.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

* Ambitious objective to simulate "collaborative discussions with diversified peers" with novel design of 3-phase framework for discussions by role-playing LLM agents. * Exciting results with strong gains over previous methods of generating creative answers. * Clear delineation of where this approach contrasts with state of the art. * Solid empirical evaluation methodology with established tests and metrics, engaging both automated and human judgements, as well as documented ablation tests, explaining development of specialized prompts, # discussion rounds and # LLM agents

Reasons to reject

* The authors do not discuss going beyond the role-playing agents as artificial people, and address other ways of coming up with better answers,. * The paper does not address what is actually happening inside the model, that would also give deeper insights.

Questions to authors

* The space of possible prompts and ways of combining the results of prompts is so broad that it seems unlikely these results are anywhere near the best that could be obtained with present models. Q1: Why, for instance, are the particular roles of "visionary millionaire, startup founder, social entrepreneur, creative professional, customer, environmentalist, digital nomad, industry insider, futurist" (which sounds like a list of people you might encounter at a party in Silicon Valley) chosen instead of, say, homemaker, mathematician, janitor, and professional actor? Or, say, all 16 Meyers-Briggs personality types? Or people from 20 diverse countries or time periods? There's no justification for this and no way of knowing what would be best a priori. At the moment there is so much low-hanging fruit that it is easy to come up with new approaches that get big gains, but it would be helpful to find some automated way of searching the space of possibilities more thoroughly. * Q2: What might help explain why fluency and flexibility scores are lower for LLM Discussion than LLM Debate, even though these matter less than originality and elaboration as measures of creativity. Perhaps there would be some simple way to alter the prompts to boost these factors as well? * Q3: how is the prompt in the convergence phase leading to "convergence" as opposed to simply enumerating answers from prior phase, given instruction to "present a list"? Did you experiment with an explicit prompt to create a summary or draw a conclusion?

Summary

This paper proposes an approach to enhancing the creativity of large language models (LLMs) by facilitating multi-agent discussions and incorporating role-playing elements. The key ideas of a three-phase discussion framework and role assignment are well-motivated and explained. The authors conduct evaluations across 4 creativity benchmarks, comparing their proposed method against baselines. The results demonstrate improvements in some metrics of TTCT like originality and elaboration, suggesting that the proposed approach can indeed induce more creative and novel responses from LLMs. Overall, this work presents a relatively convincing approach to an important problem in AI, with findings that are likely to stimulate further research in this area.

Rating

6

Confidence

5

Ethics flag

1

Reasons to accept

1) Clever use of multi agent discussion and roleplaying in open domain NLG task 2) Good evaluation with LLM and humans and also showing correlation between each

Reasons to reject

While the paper presents an interesting approach and promising results, there are a few key concerns that should be addressed before it is ready for publication 1) The role-playing component is a novel contribution, but the paper lacks deeper analysis on the specific roles used, how they were chosen/generated, and whether these roles may introduce unintended biases or perspectives that could skew the results.The role-playing aims to induce diverse perspectives, but the paper does not discuss how the chosen roles represent or account for diversity across different demographics, cultures, domains, etc. A more critical examination of this aspect is needed. 2) The task chosen are relatively old and sentence level so not very clear if this approach would be generalizable for other creative tasks such as design, ideation, creative writing, generating metaphors etc 3) The results are primarily quantitative, but a deeper qualitative analysis examining the generated responses, the nature of conversations/discussions, and specific strengths/weaknesses could provide valuable insights. 4) Theoretical grounding: While the motivations are clear, the paper could benefit from stronger theoretical foundations explaining why multi-agent discussion and role-playing are expected to enhance LLM creativity from a cognitive/psychological perspective.

Questions to authors

Why AUT ? There is mounting evidence that GPT4 is good at this so to benchmark you should focus on complex creative tasks https://arxiv.org/abs/2303.12003 https://news.uark.edu/articles/69688/ai-outperforms-humans-in-standardized-tests-of-creative-potential

Thanks for your response

Thanks for the extra analysis on fluency vs diversity. I'll maintain my score of 7 and recommend acceptance.