With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation

Summary

This paper introduce a simple and effective framework Temp-Lora to extend the context window for LLMs for long text generation. The author argues that traditional methods (length extrapolation & context window extension) demand significant hardware resources during training / inference, so Temp-lora is proposed to store context information in model parameters instead of KC cache. The parameters of Temp-Lora is training on the fly using previously-generated tokens, and discarded when the inference is completed. Experiment on PG19 data-set shows a significant improvement in PPL/BLEU, and a large amount of memory reduction and latency decrease.

Rating

5

Confidence

5

Ethics flag

1

Reasons to accept

The motivation of this paper is solid, focusing resolving the context limitation for long text generation. The proposed method is simple and effective, the paper is easy to understand. And the experimental results seems promising, achieving PPL reduction, memory reduction & latency reduction at the same time.

Reasons to reject

1. I think the related work comparison is weak. In the intro, the author strongly claims that "Temp-Lora is compatible with and enhances most existing long text generation methods." But I don't see enough evidences in the experiment part to support this claim. So maybe the author can elaborate on this point. 2. It's exciting to see good experimental results (achieving PPL reduction, memory reduction & latency reduction at the same time), but on the other side of the coin, I think it's also partially because the baseline is relatively weak, maybe adding more comparison system with the length extrapolation and context window extension methods will make the results more convincing. 3. One of the main advantage of this method is preserving the hardware resources, I assume it also includes computation cost, and the on-the-fly training of Temp-lora requires additional GPU during inference. So maybe the author could add more detail on this direction, and list out computation overhead for a better presentation of this paper.

Questions to authors

see weakness part. Also in Figure 1, there is a missing arrow to point to Chunk1 to "Generate"

Summary

This paper propose a method called Temp-Lora, which embeds the context information into a Lora module for each chunk to enable long text generation. The proposed method is implemented based on Llama2 and evaluated on two benchmark datasets. The performance on PPL, memory usage and latency get improved compared with Llama2.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

The proposed method is simple and it shows improved generation performance and decreased inference time compared with Llama2.

Reasons to reject

1. The proposed method has only been evaluated on Llama2, leading to limited generalization. 2. Temp-Lora performs better than Llama2. This is good. But the paper only shows the performance of Llama2, lacking some strong baselines such as flashattention, pagedattention. Therefore, we can't know how good Temp-lora is compared with some strong models.

Questions to authors

Can the author show some cases generated by the proposed method? Only providing the PPL can't intuitively show the generation performance of the proposed method.

Summary

This paper proposes that a Low Rank Adaptation module (Temp-Lora) that is temporarily invoked and discarded during the generation of long text. The results of applying this model to text generation tasks appear to be substantial, with improvements in perplexity accompanied by reductions in memory usage and inference latency.

Rating

8

Confidence

3

Ethics flag

1

Reasons to accept

1. Technological impact: The paper proposes replacing the widely used KV cache with a mechanism that stores context information in model parameters during training and generation and with some apparent savings of computational cost. This reviewer, at least, found the approach interesting, and potentially useful. The Temp-Lora module only exists during long-text generation, and therefore requires a mechanism to determine when then long-text generation is being performed. At some leve, this proposal is a technically-sophisticated hack—an interesting one, but nevertheless one whose functionality might eventually be more organically folded into the model itself. In its current form, therefore, the proposal is probably not a long-term solution, but one that allows the fields to move ahead while defining a performance space that a longer-term, more-organic solution would need to cover. The definition of this achievable performance space is probably the primary reason that this paper should be accepted. 2. Evaluation: The paper presents extensive ablation studies showing consistently improved results at least on the metrics and datasets employed. 3. Clarity: The paper should be largely clear to someone who is well accomplished in language modelling. For the most part it is readable and interesting, with good documentation of results and valuable discussion.

Reasons to reject

Evaluation: Although the evaluation is extensive, it has limitations. The only comparisons made in Table 1 are with models in the llama family. The paper would be more convincing if they had shown benefits using other Open Source models, e.g., Phi-2 or Mistral, as well. (Does the efficacy of the approach depend on the model?) it is a pity, perhaps, that the paper does not attempt to compare performance with one of more of the approaches noted in the second paragraph of the introduction. The perplexity and BLEU improvements seem large, but the reader does not get to see how the approach compares with some of the alternative approaches available. Overreliance on a single dataset (PG19) may be an issue. The authors used a random sample of 40 books with high perplexity from within this data set. Though the reasons for this manner of selection are stated in the paper, given that the models may have seen this dataset during training, the high starting perplexity is likely to bias the results. It may be easier to obtain strong percentage-wise reductions from a higher starting point than a lower one.

Questions to authors

Acronyms should be spelled out when they are introduced in the abstract and introduction. The manner in which the paper introduces the terms KV cache and Lora imposes unnecessary presuppositional burdens on the reader. The KV cache is not really an organic part of a language model, but a heuristic designed to make inference more efficient. Similarly Lora. These issues should be easily remediable before the final version is ready.

Summary

In this paper, the authos proposed Temp-Lora, an alternative way for efficient long text generation. Its essence lies in training during the inference process using the generated output. It enables the storage of nearly infinite context information directly within the model’s parameters, marking a distinct difference from existing attention weights-based techniques. The experimental results across various applications demonstrated the profound impact of Temp-Lora.

Rating

5

Confidence

5

Ethics flag

1

Reasons to accept

1. The proposed method is interesting and has the potential to significantly enhance both effectiveness and efficiency in the domain of long-context generation. 2. This paper is well-written, with a clear and comprehensible description of the method.

Reasons to reject

1. The experimental results presented are insufficient. The authors must first conduct a comparative analysis with existing methods for long context generation. Additionally, it is imperative to validate the proposed method using established long-text benchmarks to ensure its robustness and applicability. [1] Dynamic-NTK: https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamically_scaled_rope_further_increases/ [2] LongBench: https://github.com/THUDM/LongBench 2. The primary advantage of the proposed method is its ability to operate without the need for saving the KV Cache. The authors should provide a comprehensive demonstration of the improvements in overall efficiency across various settings and scenarios to substantiate this claim, e.g., throughput in batch decoding.

Questions to authors

1. I am interested in understanding the performance of the proposed method within the context of batch decoding. Could the authors provide a detailed analysis of its efficiency and effectiveness in such scenarios? 2. In Table 1, why does the perplexity (PPL) decrease as the range extends further back? Additionally, how is the data distributed across the different intervals? A thorough explanation of these observations would be beneficial.

A Discussion about some common questions

Dear reviewers! Thank you for your hard work. During the rebuttal phase, we conducted many experiments and addressed most of your questions in detail: 1) Comparison of Temp-Lora with other existing long-text generation methods like Dynamic-NTK (Please refer to **Response to concern 1 of Reviewer 3Zws**); 2) The effect of Temp-Lora on other open-source models (Please refer to **Response to concern 1 of Reviewer Sdmf**); 3) Why we don't use benchmarks like LongBench (Please refer to **Response to concern 2 of Reviewer 3Zws**). We are confident that after reading the rebuttal, you will have a new and more comprehensive understanding of the effectiveness of Temp-Lora. In fact, we have already applied Temp-Lora technology in many real-world applications, such as: 1) In our chat-style assistant, users can upload infinitely long documents as background information, and we will use these documents to initialize a Temp-Lora module and continue training it with subsequent user conversations. 2) We have developed some lifelong game NPCs that need to accompany users for a very long time and gradually have "unique memories" between them. These memories are implemented with Temp-Lora. 3) We also have a translation tool specifically for professional translators. They can update the Temp-Lora module with their translation history when translating long documents (such as novels) to maintain better consistency in subsequent translations (this scenario is very similar to our second experiment). We believe that Temp-Lora represents a new paradigm for storing long-context information: in addition to being stored in the context window and external databases, model parameters can also effectively store historical knowledge. This is also consistent with human nature: people do not re-attend all context information every time they make a decision, most of the time we rely on the memory in our brains (model parameters) to make decisions, only when we need detailed information and can't remember it, will we review past records (KV cache and RAG) to help make decisions. If you have any other questions, please send them before the end of the discussion period, and we will answer them as soon as possible.

Reply to authors

Thanks for the authors' response! I think most of my concerns have been addressed. Therefore, I increase my score to 6. By the way, I feel it's not a good idea to direct one reviewer to read the responses addressed to another reviewer.