Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline

Summary

The authors propose a simple method to perceive the length of an LLM response by asking the LLM. Then, the authors propose to groups queries with similar response lengths into micro-batches, which are then allocated to different GPU nodes and processed in parallel . The authors show empirical gain in terms of throughput. This approach is also orthogonal to other inference acceleration approaches.

Strengths

1. The proposed method is simple and exact and does not sacrifice response quality while achieving speedup. 2. The authors show impressive speedup in terms of throughput. 3. The speed up the author achieve can be applied on top of other inference acceleration techniques.

Weaknesses

1. One fundamental weakness of the paper lies in the assumption that requests can be reordered, which may not hold in production. The author did not show any fairness metric which may help readers understand how their method affect each individual request in practice. 2. The author did not show any fine-grained ablation studies examining how often and how often the requests have been reordered and how it affects inference latency. I would happily raise my rating if the authors can present more thorough ablation studies.

Questions

I have listed my concerns above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors adequately addressed the limitations and potential negative societal impact of their work.

Summary

>**Rebuttal:** The provided details satisfy my concerns. I think this paper should be accepted after applying the agreed changes. >**TL;DR:** The paper presents a new technique to reduce the inference time of LLMs under intensive usage. This is an important problem that can reduce wasteful computations. However, the paper is missing some key comparisons and the experimental methodology is lacking justifications. Addressing my concerns and questions would improve my score. This paper proposes a new to reduce the inference time of LLMs under intensive usage. The technique save wasteful computations by predicting the response length and aggregating similar predicted lengths together. The paper presents an inference pipeline, which is composed of response length prediction, failure collection and recomputation (FCR), and variable batch size (VBS). Experimental results on real-world instruction datasets using the Vicuna-7B model demonstrated an 86% improvement in throughput without sacrificing performance quality. The datasets include the Instruction-in-Wild and Alpaca. The proposed technique is compared to previous works and outperforms them on both datasets.

Strengths

* **S.1.** The proposed technique can reduce inference time of LLMs and gain a 86% performance improvement. * **S.2.** The paper tackles an important problem of reducing the LLM inference time and wasteful computations. * **S.3.** The proposed technique outperforms previous compared works on two datasets. * **S.4.** Reproduction code is provided as part of the submission.

Weaknesses

* **W.1.** Th paper lacks comparison to existing LLM inference works such as [1][2]. * **W.2.** Some the key technique attributes are not well justified. For example, the target length prediction is four times the actual length. No experiments or ablations are provided to justify the "four". Another example is the FCR mechanism, which immediately stops the generation process when the maximum predicted length has been surpassed. There might cases where the generation process might only need a few more tokens to complete the task. * **W.3.** The proposed technique relies on very short inputs in order to be effective. This is rarely the case for chat-bot, which require support for multi-turn conversations and are given as the examples for LLM usage. Furthermore, the proposed technique relies on high LLm usage to create batches of similar predicted lengths. This high usage is typically found in chat-bots. [1] Yu, Gyeong-In, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim, and Byung-Gon Chun. "Orca: A distributed serving system for {Transformer-Based} generative models." In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538. 2022. [2] Pope, Reiner, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. "Efficiently scaling transformer inference." Proceedings of Machine Learning and Systems 5 (2023).

Questions

* **Q.1.** The paper describes several LLM use cases such as ChatGPT, Bard, and Claude. These chat-bots require support for multi-turn conversations, which expands the conversation history. How would the proposed technique work in such cases?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The limitations of the proposed technique are described throughout the paper. The limitations include overhead of length prediction and poor compatibility with long inputs. However, the latency effects of waiting for the aggregation of batches is not discussed or evaluated.

Summary

This paper comes up with the technique of using LLM to help LLMs’ inference to be more efficient. It predicts the queries’ response length, and group the those with similar response length into the same micro-batch, so that the inference efficiency can be effectively improved.

Strengths

In the experiments, the proposed method gains significant improvement in terms of inference speed. It is reasonable that the token redundancy leads to inefficient inference in batch. It is effective and easy to implement for existing LLMs.

Weaknesses

The metric of horizontal axis in Fig. 2 (a) is missing.

Questions

Imaging a scenario that a user’s query has extreme long response length predicted, would this user wait long time for the response? How about to use a small model to learn to predict the response length?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

None.

Summary

The authors propose to improve the throughput of the LLM inference systems by correctly predicting the length of the response. Method summary: 1. Predict the length of the response (Binning length for prediction modules to learn better) 2. Use the prediction to batch the queries with similar prediction to improve throughput (use variable batch size to leverage GPUs while managing memory requirements) 3. Failure collection module to cut-off mispredicted batch evaluation. To ensure that this module is not triggered too often, it is advisable to over-estimate length of the prediction

Strengths

1. The paper tackles an important problem, provides a simple recipe for the solution and works reasonably well. 2. The evaluation is to the point. Answers all the natural questions that might arise.

Weaknesses

The writing can be better. Some details in the questions section.

Questions

Comments: 1. For table two to make sense, it would be useful to have some data statistics like distribution of response lengths on those 175 instructions. 2. Table 3 caption can be improved a lot. It was not clear what i am looking at first. The table talks about vicuna model inference and various prediction methods for that. Elaborate on the caption. Questions: 1. In table 4. why is the vanilla avg length significantly different from other lengths including ground-truth predictor?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Response to the authors

Thank you for the clarifications. Please make the discussed changes to the manuscript. Hope it gets in!

Response to Rebuttal

Thank you for the detailed answers and results. This solves some of my concerns. However, I'm not convinced regarding W.3 and partially W.1. * **W.1.** I'm not fully convinced that there is not a single existing algorithm that can be compared to. Adding a detailed explanation including orthogonal compatible approaches should be sufficient. * **W.3.** This is my main open concern. The provided Fig. 4 is computed on instruction data. The input of instruction data is usually short, and the generated outputs are usually longer that the input. This is typical for instruction based datasets, but not for real world applications. In cases where the inputs are long, the Response Length Prediction time would take a large portion of the computation time. For example, let's say the task is to give a score to a very long passages. In this example, the inputs are very long and the outputs are short (a single integer). The Response Length Prediction model would take just as long as the actual inference and thus would almost increase the total inference time by two times. This is of course an edge case, but it is an actual limitation of the paper which is not explored. Addressing my concerns would improve my score.