How Susceptible are LLMs to Influence in Prompts?

Summary

This paper investigates the issue of the influence of augmented prompts on the accuracy of LLMs in a question answering setting, with the augmented input being provided by another advocate model (in this case, LLama-2). The authors test three different LLMs (Mixtral, Llama-2 and Falcon), the so-called judges, and analyze the impact on their performance of several factors, including confidence and authority of the source. They found that susceptibility to influence is generally higher when the model is less certain about the correct answers (= when the model generally has a lower dataset accuracy) and when the explanations about the possible answers come from a source with a higher level of expertise (this is simulated by manipulating the root prompt of the LLMs, the test is carried out at six different levels of expertise of the "persona" of the model).

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

+ clear and exhaustive analysis of the impact of influence on LLM prompts, taking into account factors such as model confidence and source authority, and investigation of the possible mitigation strategies + narrative is generally easy to follow and experiments are well structured

Reasons to reject

- a possible issue is that the results all refer to a scenario in which the Llama-2 model is used as the advocate, and we cannot be sure about the generalizability of the findings (how would that change if a less powerful model was used? How about a more powerful model like GPT-4)? Also, in page 4 the authors should explain why they specifically chose to use Llama-2 as the advocate.

Questions to authors

TYPOS AND MINOR ISSUES: - Section 3: "We do note... partially suppress the influence provided": this sentence is not very clear. Please, reformulate it. - Section 3: aanchoring --> anchoring - Section 3: This result, validates reported sycophantic behavior --> This result validates reported sycophantic behavior (but maybe in that context better "confirms the reported sycophantic behavior"?) - Section 3.1: resolve around prompting --> revolve around prompting - Section 3.1: byl the advocate --> by the advocate - Section 4: in princliple --> in principle

Summary

The paper presents a study on the susceptibility of LLMs to the influence from additional prompts and information sources. The paper focuses largely on analyzing three LLMs (Llama2, Mixtral, and Falcon) and how adding other model generated answers to the prompt can influence their own answers. Nevertheless, there is no specific method proposed based on the analysis to mitigate the issue that authors point out. The findings on LLMs’ sensitivity to augmented inputs are interesting, but to my current understanding not as surprising. The paper presents various figures to help illustrate its analysis and the writing is largely clear. Small typos need to be fixed, and the authors may want to make their summary or takeaways of each paragraph clearer.

Rating

5

Confidence

4

Ethics flag

1

Reasons to accept

- The paper presents an overall interesting analysis on LLMs’ susceptibility to prompts. - The paper presents various analysis and visualizations to help illustrate the findings, which are quite helpful.

Reasons to reject

- It does not seem surprising that LLMs are sensitive to prompts, and a prompt stating that a certain option is the correct answer would almost surely have an impact on LLMs. Since they are autoregressive models and the next token generation depends on the previous tokens $p(x_{t+1}|x_1,\ldots, x_t)$, changing the context/prompts will cause differences in the model generations, without explicitly fine-tuning the model to disregard irrelevant or harmful contexts. - Following my previous point, there have also been works that suggest LLMs tend to change their answers with external feedbacks (e.g. when the input states that one option is probably the correct answer), as the authors also listed some in the related works. Moreover, LLMs can generate persuasive adversarial prompts, causing other LLMs to produce incorrect or even unsafe answers (jailbreaking) [1]. Researchers have been studying the attack and defense for LLMs based on the fact that LLMs are highly susceptible to prompts without specific strategy/fine-tuning to make it more robust. Meanwhile, through the presentation of this work, I could not find a dedicated paragraph discussing the major difference in findings between this work and the previous works that discovered similar things. - There are assumptions made in the paper that I feel is not well-supported (e.g. for PIQA, SIQA and CSQA, authors expect the model to have a strong bias and be strongly grounded?) - Some part of the experiment setting is not well-explained. It is not clear that whether the authors used model-generated answer (and explanation) to augment the prompt, or pre-select an answer randomly and let model generate explanations to augment the prompt. (i.e. using model answers or random answers?) - The last statement about having multiple advocates would improve the accuracy is reasonable. Meanwhile, it is highly similar to the concept of consistency and there are more works to leverage several LLM answers/reasonings for a more robust final answer. - In terms of writing, I feel that it would be more helpful to have the takeaways summarized and highlighted in each section/paragraph that presents different findings. [1] How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs

Questions to authors

- To mitigate the issue of LLMs relying more on an answer/explanation in prompts that may even hurts its performance, have the authors tried fine-tuning the LLM on a training set of, say, CSQA with the augmented prompts? I would tend to believe that a small-scale fine-tuning (e.g. LoRA on 7B models) would largely solve this specific issue. - It seems that LLama2 is used as the advocate. What would be the reason to make such specific choice? Would there be difference in terms of persuasiveness for different LLMs that serve as advocates?

Summary

This paper studies the sensitivity of LLMs’ responses to the injected context in the prompt. It examines how the output response of an LLM will be swayed by the prediction and explanations provided by an external source. It shows that models are generally easily swayed by the explanations, regardless of their quality. And when the input is presented as being authoritative or confident, the model is more easily swayed.

Rating

7

Confidence

3

Ethics flag

1

Reasons to accept

This paper studies an important problem in LLMs: the influence of the external knowledge sources (or context) on the LLM responses. Such a problem is particularly important in retrieval augmented generation settings, where there is potential conflict between the external context and the internal knowledge. Examining the model’s robustness against authoritative advocacy and possibly erroneous retrieved context is important to build an LLM system that is trustworthy. This paper carries out a comprehensive study of the problem and shows that LLMs are easily swayed by the injected context from an advocating model, especially when explanations are present there. In particular, the paper examines the influence of context injection from different aspects, such as the confidence of the advocacy and the confidence of the judge model. In addition, it further shows that different prompting methods have limited effect in mitigating the issue and calls for further measures for solving the problem.

Reasons to reject

This paper mainly examines the influence of injected context on QA tasks. Advocacy would more influence the potential factuality. It would be interesting to see how such advocating context would influence reasoning capabilities of LLMs. In particular, how would it influence the factuality and correctness of the step-by-step reasoning processes. How resilient the reasoning process is against such injection? It also shows that critical reasoning ability is insufficient to address the problem. It is also interesting to see whether a model with stronger general logic reasoning capability would be more robust against such advocacy? The three models used in the paper would be close in their capabilities. It would be interesting to see how the advocating context would influence a wider spectrum of models from weak reasoning capabilities (7B or even smaller models) to much stronger reasoning capabilities (e.g., Llama3-70B). Also, it would be interesting to see whether a stronger model would have much stronger influence on the judge model and vice versa. Overall, a “influence matrix” that characterizes different levels of advocating model vs different levels of judge model would help draw a more complete picture of the problem. Currently, it is mainly through prompting the model to be different levels. However, using models with different levels of actual capabilities would be more interesting.

Questions to authors

See the above section.