StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses

Summary

Standard Large Language Models (LLMs) struggle with handling dialogues with long contexts due to efficiency and consistency issues. This paper finds that the structure of the dialogue context is consistent, and special tokens may aggregate information. Therefore, this paper aims to use special tokens to encode dialogue history information to reduce inference costs and enhance the ability of LLMs to handle long dialogues. To achieve this, it proposes two information reconstruction functions to improve the information aggregation capability of special tokens. Experiments have shown that the article has achieved its intended purpose.

Strengths

1. This paper finds that the separator tokens, namely conversational attention sinks, generally aggregates more attention than other words and tokens. Therefore, this paper proposes StreamingDialogue, which utilizes there special tokens to enhance the capability of LLMs to handle long-context dialogue. 2. StreamingDialogue has achieved good results on multiple dialogue datasets. Analysis experiments also demonstrate that this method is capable of handling long-context dialogues and can reduce inference latency and memory usage.

Weaknesses

1. The training method introduced by StreamingDialogue is overly complex, resulting in the model performing multiple forward passes on the same sample during training, leading to more than three times the training cost. 2. The comparison of different methods is unfair. In the main experiment, StreamingDialogue is fine-tuned on specific datasets, making it evident that it can surpass the non-training method StreamingLLM [1]. Although the authors also explored the non-training setting of the proposed method in Section 4.6, they only conducted experiments on Llama2-7B-chat and evaluated it using only 1-gram and 2-gram metrics. Therefore, the effectiveness and generalizability of the method are not verified. [1] Xiao et al. Efficient Streaming Language Models with Attention Sinks. ICLR 2024.

Questions

1. In line 127-128, the authors mention that by caching only the corresponding conv-attn sinks, the time complexity of attention computation can be reduced from $O(T^2L^2)$ to $O(T^2)$. However, have the authors considered that if each utterance contains $L$ tokens, the time complexity should be $\mathbf{O(T^2L)}$? 2. For the same sample, it would be best for the authors to use the same notation on line 157 and line 175 to avoid confusion.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors adequately address the limitations and, if applicable, potential negative societal impact of their work.

Summary

This paper tackles the challenge of long context dependencies of LLMs in dialogue settings. The authors first posit that end of utterance tokens like "\n" and </s> could conceivably summarize the information in the utterance, and propose to attend to such sinks rather than entire utterances to allow LLMs to perform long-context tasks with lower computational costs. Two learning strategies, SMR and LMR are introduced to encourage EoU tokens to carry key information from preceding utterances, and remember key information from previous EoU tokens. The authors evaluate on multiple dialog datasets, and demonstrate that the proposed approach can significantly improve computational complexity and make it possible to operate on longer dialogs.

Strengths

1. The problem that the paper is tackling is important for the community at large, and the proposed approach seems intuitive and logical. 2. The paper is original, well structured and written, and experiments appear apt to substantiate the authors claims.

Weaknesses

1. The authors do not compare against plausible alternative approaches like infinity former or the compressive transformer, which are also memory based approaches. Comparisons to position interpolation based approaches with RoPE would also be interesting to see.

Questions

1. Line 111: Yes, position interpolation based approaches may not provide for infinitely long sequences, but is that required for dialogue tasks? Perhaps this could be rephrased to make the authors point. 2. Line 119 makes a leap in logic that is perhaps not fully intuitive. The authors claim that higher attention on EoU tokens suggests that these aggregate information. Did the authors test whether the responses of the LLM to conversations implied that high attentions on EoU tokens could possibly capture information from the preceding utterances ? 3. Line 160: Did the authors consider restricting the attention for u' tokens to just the sink token, rather than sink tokens and previous tokens? The current phrasing seems to indicate the former over the latter.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors are requested to add notes on limitations and potential negative social impacts. The additional training methods proposed may incur computational cost, and I encourage the authors to comment on this.

Summary

The paper introduces a novel approach for encoding long conversations. Motivated by the results of StreamingLLM, the authors observe that the end-of-utterance (EOU) or separator tokens aggregate more attention than other tokens in a dialogue generation task. The authors refer to the EOU tokens as conv-att sinks (conversational attention sinks). Based on this observation, the authors propose to attend and cache only the conv-att-sinks of the past utterances to represent the dialogue history, thereby making the space complexity of the attention mechanism linear to the number of turns in a conversation. To learn quality embeddings for conv-attn sinks, the authors propose two auxiliary tasks, SMR (a response reconstruction task) and LMR (a response recall task). The proposed StreamingDialogue encoding strategy achieves comparable performance to dense attention (attention on all previous tokens) and outperforms memory-efficient baselines on Persona-Chat and MSC datasets. The method also exhibits 4× speedup while reducing memory usage by 18× compared to the dense attention strategy.

Strengths

1. The paper is well-written and easy to understand. The proposed method is well-motivated and addresses an important problem of encoding long dialogue contexts. 2. The proposed attention strategy of utilizing only the conv-attn sinks is simple and effective. StreamingDialogue shows better performance than memory-efficient baselines on both automated and human evaluation. The method also shows performance comparable to that of the dense attention strategy. 3. Results suggest that the SMR and LMR help to learn a rich representation of the conv-attn sinks. The authors also show evidence (Fig. 7) that the model can recollect/generate past information from the previous conv-attn sinks. 4. The method is cost-effective and can achieve significant speed-up compared to the dense attention strategy.

Weaknesses

1. Although the results shown in Table 1 are positive, the metrics are not well-suited for the open-domain dialogue generation task. The authors have shown their results on two additional metrics (USL-H and Dial-M) in Table 6 for the MSC dataset. USL-H and Dial-M have been shown to be better metrics than BLEU, ROUGE, Distinct, and perplexity, especially for persona-grounded datasets like Persona-Chat. However, Table 1 does not show the results with USL-H and Dial-M. Also, in the case of MSC, StreamingDialogue performs better than StreamingLLM on the USL-H metric but not on the Dial-M metric. This is why I think that although the method is appealing, the results are not strong enough to support it. 2. There is no information about inter-annotator agreement for the human evaluation. 3. The experimental setup with the Persona-Chat and MSC dataset is not clear. The authors have not mentioned whether they used the persona profiles to generate the responses for the result of Table 1. 4. The use of the BLEU metric is not consistent. The authors use average BLEU in Table 1 and Table 2. However, BLEU-1 and BLEU-2 are shown in Table 3, whereas only BLEU-1 is shown in Table 4. It would be better to show all three variations of BLEU in all the result tables. 5. The results are shown only on two dialogue datasets. There are other datasets with long dialogue contexts like Topical-chat. MultiWOZ is another dataset for task-oriented dialogue systems, which includes lots of conversations where the user utterance directly refers to past dialogue history.

Questions

1. Why did the authors not use USL-H and Dial-M for Table 1? 2. Did the authors use the persona profiles to generate the responses? If yes, how was it included in the context? If not, is it fair to compare BigBird and StreamingLLM with StreamingDialogue? 3. Explain the inconsistent use of the BLEU metric. 4. Why are Layer 0 and Layer 1 shown in Fig. 1a, whereas Layer 28 is shown in Fig. 1b? 5. Did the authors analyze the conv-attn sinks for the example in Fig. 7?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Looking forward to your feedback

We highly appreciate your valuable time spent in reviewing our work. The insights and contributions you have made to improve the quality of our submission are sincerely acknowledged. We would like to inquire whether our response adequately addressed your questions. Your feedback holds immense value to us, and we eagerly await your reply.