SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices

Summary

This paper presents a novel speculation decoding method called SpecExec, designed to improve the performance of large language model (LLM) inference on consumer-level GPUs with offloading scenario. The main contribution is the application of speculative execution, a technique from CPU architecture, to the speculation decoding of LLMs. SpecExec leverages a draft model to construct a large draft tree of potential continuations, which are then verified in parallel by the target model. This approach aims to mitigate the inefficiencies of existing speculative decoding methods, particularly when dealing with a large number of draft tokens. The experiments result demonstrate that SpecExec achieves up to 18.7x speedup.

Strengths

- This paper tackles specific and clear setup. Since offloading is the main bottleneck of LLM inference on consumer-level GPUs, using speculative decoding with many draft tokens is much more preferable. - SpecExec is robust for non-deterministic request (i.e., t = 0.6, top_p = 0.9) - As the SSSP problem is one of the well-studied problems in the CS domain, it may be easy to adopt other variants of SSSP to generate different draft trees (extendability).

Weaknesses

- Although the paper targets consumer-level GPUs, the experiment results show that SpecExec achieves the highest speedup with the A100 GPU, one of the most popular datacenter-level GPUs. - Token throughput is still slow because of the offloading. For A100 GPU, using 4-bit quantized 70B model (AWQ, GPTQ, ...) and without speculative decoding will be more practical. - SpecExec seems plausible only for single request with single user.

Questions

- How much speedup does SpecExec gain without RAM offloading? For example, Llama2-70b chat GPTQ model on an A100 GPU. I am just curious - So, if I understood correctly, sampling parameters such as t = 0.6, p = 0.9 do not affect the draft tree, right? (deterministic tree) - In Figure 3 and 4, random sampling results are comparable with greedy sampling on the MTBench dataset while the C4 dataset shows lower hit ratio on random sampling. Does this difference come solely from the difference between datasets? - Can SpecExec achieve high performance gain on token penalty scenarios? For example, repetition penalty, presence penalty, and frequency penalty.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

No additional limitations regarding societal impact. For technical limitations, I stated in the weakness section.

Summary

In this paper, the authors propose a speculative decoding technique, SPECEXEC (Speculative Execution), that can generate up to 20 tokens per iteration for the LLaMA-2 70B model. SPECEXEC enables the LLaMA-2 70B model to run on a consumer GPU by a parallel decoding strategy. The offloading scheme can process thousands of tokens using the same latency as that of one token. Therefore, the offloading scheme can be naturally used in supporting speculative decoding. However, prior speculative decoding techniques do not scale with a large batch of draft tokens. SPECEXEC uses a smaller draft model to built a tree of best tokens by single-source shortest path (SSSP) searches, and then processes best tokens with the target model by offloading. SPECEXEC also includes a new algorithm to determine which tokens are accepted.

Strengths

1. The paper works on an important topic. 2. The paper includes enough supporting data.

Weaknesses

1. The paper is not well-written. SPECEXEC is built on a important observation that the offloading of the target model can process thousands of tokens using the same latency as that of one token. SPECEXEC suffers from a lower acceptance rate for individual tokens. For a small number of draft tokens, SpecExec is less efficient than speculative decoding. However, for a large enough number of draft tokens, SpecExec obtains a better tree and increases the accept rate. SPECEXEC adopts the offloading technique to mitigate the processing of the large enough number of draft tokens. Unfortunately, the authors did not justify the applications or user cases that must use the offloading schemes.

Questions

1. What type of applications are more suitable for SpecExec? The applications have not enough server-level GPUs?

Rating

5

Confidence

3

Soundness

3

Presentation

1

Contribution

3

Limitations

no limitation.

Summary

The authors present a method to improve the efficiency of speculative decoding on consumer-grade hardware. The technique addresses the inefficiencies of existing speculative decoding approaches when applied to devices with limited GPU memory, necessitating parameter offloading to RAM or SSD. SpecExec leverages a powerful draft model to create a speculative tree of potential token continuations, which the target model then verifies. This approach enables the processing of multiple tokens in parallel, significantly improving the inference speed on consumer GPUs.

Strengths

The paper provides a solid empirical analysis of key inefficiencies of speculative decoding and presents a well-founded solution. The paper is written clearly with solid motivation and method sections. The empirical results that the authors provide are impressive, showing consistent speedup over SpecInfer on the number of generated tokens per step. In addition to results on generated tokens per step, the authors also present an impressive speedup on tokens per second, showing that the improvement achieved at generated tokens per step can be translated to real wall-clock speedup.

Weaknesses

1. Some experiment results are a bit confusing. For instance, the captions in Figure 3/4 say Generation rate vs draft size for Llama 2-7B/70B models, but it's not clear where the 7B model numbers are. The main text states the plots are for 70B models. 2. The authors claim they have results on both base and chat models. But it's not clear from the tables and figures, which are chat models and which are base models. And further, whether this would lead to different results/conclusions.

Questions

My questions are mainly regarding the experiment details discussed above in the weakness section.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors adequately addressed the limitations and, if applicable, potential negative societal impact of their work.

Summary

This work introduces SpecExec, an improved speculative decoding method. By constructing a better draft token tree and refining the process of verifying tokens, SpecExec significantly increases the number of tokens accepted in a single verification while producing exactly the same outputs as sequential sampling from the target model. Additionally, SpecExec leverages the capability of modern offloading engines to dispatch model parameters in parallel with computation, successfully achieving effective acceleration under offloading configurations. Experiments demonstrate that SpecExec can run a 70B model on consumer-grade GPUs and achieve up to 10.6 times acceleration.

Strengths

1. SpecExec uses a modified version of Dijkstra’s algorithm to create better token trees, significantly increasing the number of tokens accepted in a single verification. 2. SpecExec views the process of parallel verification of candidate tokens as looking them up in a speculative “cache”, ensuring that the output is identical to that of sequential sampling, while speculative decoding only guarantees correct overall sampling probabilities. 3. SpecExec has demonstrated excellent acceleration in experiments, significantly outperforming SpecInfer and showcasing superior performance.

Weaknesses

1. While SpecExec achieves significant performance acceleration in systems with offloading, it is unclear how this method performs in terms of acceleration in speculative inference systems that do not have offloading.

Questions

1. In Line 14 of Algorithm 1, "next_probs := FORWARD(τ, θ_target)", how is forward inference performed on the draft tree? Is it the same method used in SpecInfer? 2. The paper lacks clear illustrations of the working process of SpecExec. Could additional diagrams be added to provide a more vivid explanation? 3. The paper mentions several articles related to speculative decoding but lacks an introduction to these works and does not explain the differences between them. Adding this background could help readers better understand this paper and it's contribution.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

4

Limitations

The authors have not adequately discussed the limitations of this paper.