Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs

Summary

This paper presents a series of experiments designed to probe the ability of LLMs to form conceptual pacts / to act on precedent in repeated reference games. Given that LLMs have been tested on reference games before, this is a natural next step to test, and it is good to see it realised. The authors find that there is very little evidence of LLMs treating the conversational context in the same way as the presence of conversational (and hence shared) context would be by natural interlocutors. Only when coaxed through very suggestive prompting, some small effect can be found. The paper is well written and easy to follow, and could be useful if only to introduce the relevant literature to a field where talk about "mental models" and "theory of mind" is often very cheap.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

- Very timely experiment, well conducted - Well and very clearly written - Good command of the relevant literature

Reasons to reject

- Discussion somewhat disappointing. I think the results are stronger than the authors let on: The fact that the models fail so completely to exhibit precedent effects indicates that they, despite what many people seem to hope, have not transcended beyond being text completion models and towards being agent models. The models simply have no reason to communicate efficiently -- among other things, because they are not communicating. - Related to that, I found the experiments on using prompts to induce the desired behaviour a bit unnecessary. With these prompts, the focus moves from investigating what kind of entity has been learned to less interesting engineering questions. (Although I do understand the rationale that users of these models might even be confused if apparent conceptual pacts are broken.)

Questions to authors

- Can you say more about why you engineered promts for each model? This is a question that arises in all kinds of between-model evaluations, and people come to very different conclusions. But it seems to me that most often, people decide on "no prompt engineering is fairest". - Not a question, but a comment: You could mention the PhotoBook dataset in the related work, which was built to study exactly your main phenomenon. (Haber et al 2019) - Not a question, but a comment, part II: You could mention as related the model-model self-play frameworks that came out last year, which pioneered the setup that you are also using. One of which, (Chalamalasetti et al 2023) even includes a reference game as one of the test dialogue games. (But they are not focussing on repeated episodes.) Kranti Chalamalasetti, Jana Götze, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, and David Schlangen. 2023. clembench: Using Game Play to Evaluate Chat-Optimized Language Models as Conversational Agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11174–11219, Singapore. Association for Computational Linguistics. Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. 2019. The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1895–1910, Florence, Italy. Association for Computational Linguistics.

Summary

The paper proposes a framework to evaluate the conversational adaptation of MLLMs. The paper then describes evaluations of a number of MLLMs. The results show that the generations do not become more efficient over time, as would be desirable. However, more careful prompting somewhat mitigates this. These findings have implications for the development of MLLMs.

Rating

9

Confidence

4

Ethics flag

1

Reasons to accept

This is an important problem, and the paper describes a valuable contribution to addressing it. The setup is simple and clear, which I see as a strength. I also found the introduction well-motivated and the background sections very detailed.

Reasons to reject

Some limitations and issues I wanted to raise include: -The contribution appears somewhat iterative compared to existing setups and measurements by Hawkins et al (2020ab), extending to evaluating MLLMs. The novelty is not very clearly communicated to readers. -The paper is very relevant to, but does not engage with the concepts and findings on conversational grounding and building common ground through dialogue. Please see related work [1-4] on some relevant seeds for background and existing studies regarding evaluating and improving conversational adaptation of LLMs (especially 3 and 4, which characterize similar limitations in similar settings). -Anthropomorphizing language should be avoided, especially when the claims are unfounded ("they (MLLMs) may understand the increasingly efficient language of their interlocutor"; this study doesn't aim to measure understanding); "MLLMs may passively understand" (see [5,6]) 1. Clark, Herbert H., and Susan E. Brennan. "Grounding in communication." (1991). 2. Andukuri, Chinmaya, et al. "Star-gate: Teaching language models to ask clarifying questions." arXiv preprint arXiv:2403.19154 (2024). 3. Shaikh, Omar, et al. "Grounding Gaps in Language Model Generations" NAACL 2024. 4. Testoni, Alberto, and Raquel Fernández. "Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions." EACL 2024. 5. Inie, Nanna, et al. "From" AI" to Probabilistic Automation: How Does Anthropomorphization of Technical Systems Descriptions Influence Trust?." arXiv preprint arXiv:2404.16047 (2024). 6. Abercrombie, Gavin, et al. "Mirages: On anthropomorphism in dialogue systems." arXiv preprint arXiv:2305.09800 (2023).

Ethics concerns details

N/A

Summary

The manuscript explores the ability of MLLMs to engage in reference games, which involve a duo of a speaker and a listener examining similar images. The speaker's task is to choose and describe one image for the listener to pinpoint. While simple for humans, the game's purpose is to uncover tactics to reduce the length of descriptions needed over time through implicit agreements among the participants. The core experiment involves MLLMs playing these games to see if they can evolve such communicative strategies. The MLLM prompts are specifically designed to encourage this behavior. The study notes that the state-of-the-art MLLM tested did not effectively develop these strategies for more efficient communication, although it could comprehend them when presented.

Rating

4

Confidence

5

Ethics flag

1

Reasons to accept

Overall the paper present a decent piece of work. The presentation is clear and the concepts well introduced and recalled. The study can be a first step on the line to train MLLM with better capacity of adapting or forming ad-hoc conventions when communicating with humans.

Reasons to reject

The title is misleading: the term adaption is way too vague to describe the phenomenon taking place during reference games. And even more it can let think that the LLMs may be adapted to reach such specific objectives. But the study boils down to pure observation of MLLMs' behaviors when used in reference games. Proposing different prompts with customized instructions ("Gricean" etc) is a good start, but one may straightforwardly thing of means to accompany these prompts and re-enforce their effects (for instance depending on the type of exemples proposed in the context, or using MLLMs in several epochs; including explicit summarisation/simplification taks). Here's what we might like to read.

Summary

In order to understand the Multi-modal LLM (MLLM)'s capability of In-context Conversational Adaption (ICCA). This paper defined ICCA as a framework to evaluate. ICCA includes (1) A human-human interaction dataset that focus on repeated reference game interaction (2) A few success metrics including accuracy, message length, and Word Novelty Distance (WND) (3) Two types of set up: model-as-speaker and model-as-listener with different types of prompt settings With such a framework, a series of experiments has been conducted and the results suggested the SOTA MLLMs are not good at in-context adaptation yet. The paper is structured well, and quite easy to follow. The experiment design looks good to me. The result itself isn't surprising. Thus I am on the borderline leaning towards accepting the paper.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

**Novelty** Building metrics to evaluate LLMs on in-context adaptation is a nice thing to have. When LLMs are more capable of getting answers right, the next step might be getting the communication more efficient during multimodal settings. **Clarity** The paper is clear about what it does and how things are being done. **Sounds Approach** The approach of the paper is sound. Defining

Reasons to reject

**Significance** The finding of the paper isn't surprising. Establishing the metrics and building a baseline about MLLMs on one task itself is a contribution, I don't feel excited about the result and the work.

read

Thank you for your reply. I am still convinced that this paper would make a valuable addition to the conference and the wider literature, and will keep my score.