Measuring and Controlling Instruction (In)Stability in Language Model Dialogs

Summary

The paper proposes a method, namely split-softmax, for avoiding the instruction drift present in conversations as the context increases. First, authors evaluate the instruction drift, through five categories: multi-choice response, answer character, answer-string format pattern, memorization of certain facts and languages the agent speaks. They simulated 200 conversations by prompting large language models (LLMs) for playing the user and the system. They experimented with GPT3.5-turbo and LLaMA2-Chat-70B They implemented a function in python to compute the instruction stability. Second, they observed stability decay in as the number of turns increase. Third, authors hypothesize this is due to the attention decay and they propose the split-softmax method to mitigate the instruction drift: a power-law scaling of attention. They compare split-softmax against two baselines: prompt-repetition and classifier-free guidance. They show split-softmax is more stable as the number of turns increase than classifier-free. However, it has similar performance than prompt-repetition although it requires less context.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

The paper clearly explains the instruction drift problem and it's cause. Authors design a way to evaluate instruction drift on 5 types of instructions: multi-choice response, answer character, answer-string format pattern, memorization of certain facts and languages the agent speaks. The do show a decrease of stability as the conversation gets longer. Authors hypothesize this is due to the attention decay and they propose the split-softmax method to mitigate the instruction drift. They show split-softmax is more stable as the number of turns increase than classifier-free guidance. However, it has similar performance than prompt-repetition although it requires less context.

Reasons to reject

It is not clear how the instruction stability function was implemented for each of the five categories: multi-choice response, answer character, answer-string format pattern, memorization of certain facts and languages the agent speaks. Authors limited to say that the function should return a number between 0 and 1 (however it is computed deterministically). Fig 4, It would have been interesting to depict the stability decay in all the attention heads or in average. Fig 6 shows that split-softmax is not better than system prompt repetition when the dialogue length increases, however it requires less context.

Questions to authors

1) How was the instruction stability function implemented per each of the five categories: multi-choice response, answer character, answer-string format pattern, memorization of certain facts and languages the agent speaks? Which libraries were used for each case? 2) Is the LLMs who plays the user more unstable that the one who plays the system according to Figure 3? 3) Fig 4. Is the behaviour of all the heads similar to the 11th head? I suggest to move the Theorem A1 to the main paper. Missing citations: How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings Kawin Ethayarajh. A Geometric Analysis of Neural Collapse with Unconstrained Features Part of Advances in Neural Information Processing Systems 34 (NeurIPS 2021)

Summary

This paper presents a simple approach to addressing instruction drift in language model dialogs, particularly focusing on system-prompted chatbots. The authors introduce a quantitative benchmark for measuring instruction stability and propose a novel method, split-softmax, to mitigate attention decay, which they identify as a key factor contributing to instruction drift. The paper demonstrates through empirical testing on models like LLaMA2-chat-70B and GPT-3.5 that attention to initial system prompts decays over the course of long dialog exchanges. The split-softmax method, which requires no additional training or parameter adjustments, effectively enhances the model's focus on initial prompts, thereby improving instruction adherence over extended interactions.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- The paper introduces an original method for quantifying and addressing instruction drift in chatbots, which is a significant contribution to the field of dialog systems and AI safety. - The empirical results are compelling, showing clear evidence of attention decay and its impact on instruction drift, along with a detailed comparison of the split-softmax method against strong baselines. - Addressing instruction drift is crucial for the safe deployment of AI systems in real-world applications, making this work highly relevant and timely. - I like this paper for its intuitive understanding of the attention decay phenomenon and backs this intuition with solid theoretical explanations.

Reasons to reject

- The discussion could benefit from a broader analysis of how these findings might influence other aspects of language model behavior and deployment in more varied contexts. - While the paper compares the proposed method against strong baselines, including more diverse techniques could strengthen the claims.

Questions to authors

- Could you elaborate on how the split-softmax method might affect other aspects of model performance, such as response diversity or susceptibility to adversarial attacks? - In extended dialog scenarios beyond the tested lengths, how does the split-softmax method perform in maintaining instruction stability?

Summary

* The paper proposes a benchmark to measure the ability of language models to follow the system instruction over the course of a conversation. * Using LLama2-70B and gpt-3.5-turbo, the paper shows that language models often suffer from significant instruction drift within 8 rounds of conversation. * This decline in performance can be attributed to attention decay over the course of the conversation - with each turn, the model starts paying less attention to the system prompt. * The paper proposes a method called split-softmax to combat this issue and improve adherence to the system prompt - this method performs better than simple, but intuitive baselines like repeating the system prompt at regular intervals and using classifier-free guidance.

Rating

8

Confidence

4

Ethics flag

1

Reasons to accept

* Very relevant and important direction of research - implications are useful for all fine-grained instruction following tasks that a language model assistant would be expected to perform (function calling, tool usage, staying in character) * Excellent empirical analysis of instruction instability in language models and identification of the reason behind why that might be the case. * The proposed method - split soft max - is very intuitive and easy to implement in practice.

Reasons to reject

* Would like to see some measure of helpfulness like MT-Bench to evaluate model behavior, instead of relying only on benchmarks like MMLU to quantify the impact of the intervention. * There are not many details on the benchmark dataset or how the system prompts are chosen in the paper - would be helpful if more details are added.

Questions to authors

* It seems like choice of what type of data is used to construct the alignment blends/what goes into SFT & RLHF would be very important to counter this kind of behavior. Would fine-tuning on role-playing data/alignment blends help improve instruction stability? * For the graph in appendix C, what would it look like if the instruction was in the first user turn instead of the system prompt? * How are the probing questions created? Are they designed to be adversarial in nature?