Dear Reviewer F2bw,
Thanks for your detailed and valuable feedback. Here is our response.
Q1.1: Limited improvement over previous methods.
Our approach has key advantages beyond speedups, like memory saving and cheaper training cost as mentioned in Figure 7a and Table 2\. More free memory enables the use of larger batch sizes, resulting in the potential of improved throughput. Memory efficiency is also vital for long-context tasks, where a large KV cache size could otherwise lead to OOM errors. Additionally, reducing training time significantly lowers training costs, making PPD more accessible for local deployments. Faster training is particularly valuable in online learning scenarios, where the target model distribution may shift over time, requiring frequent retraining.
Q2: Further evaluation on larger models
First, we would like to emphasize that our primary focus, as highlighted in the paper, is on edge and mobile settings. The design of our method, particularly features such as memory savings, is specifically tailored to address the constraints and requirements inherent to these scenarios.
Second, the decrease in acceptance length observed in Table 1 as model sizes increase is attributable to the smaller optimal tree size, rather than a decline in the capability of PPD. In fact, as demonstrated in Figure 6a, the prediction accuracy of PPD improves for larger models due to their greater depth and wider token embeddings, which enhance its overall performance.
Q3: Performance under high GPU utilisation
First, given that our primary use case is the mobile/edge setting (e.g., AIPC), a single-batch scenario is common. In such settings, idle compute resources are often present due to the memory-bound nature of LLM inference, making our method particularly efficient.
Second, in high-utilization scenarios (e.g., large batch sizes), memory availability becomes a critical concern, as highlighted by [FlexGen](https://arxiv.org/pdf/2303.06865). Unlike previous speculative/parallel decoding methods, PPD incurs negligible memory overhead, which helps mitigate these memory constraints and supports efficient inference in such scenarios.
Lastly, we provide analysis of the effect of batch size on the speedup ratio in Appendix E, showing that significant speedups can still be achieved with increased batch sizes. Furthermore, our method is orthogonal to recent advancements in batched speculative decoding [optimization](https://arxiv.org/abs/2406.14066), making it compatible with such approaches.
Q4: Unnecessary computation for incorrect guess tokens.
It is important to note that PPD adopts the same guess-and-verification procedure as previous parallel decoding methods such as Medusa, LookAhead Decoding, and Blockwise Parallel Decoding. Therefore, the additional computation observed is not introduced by our proposed method.
Q5: Details on EPT training and usage.
Please refer to Appendix D for the ablation studies on the design choices of EPT, including the attention mechanism, hyperparameters, and other relevant factors.
In summary, each EPT is associated with its own trainable embedding, which is optimized via backpropagation during training. This ensures an effective initialization that supports accurate multi-token generation. A single prompt token is composed of multiple EPTs. To produce a single output token prediction, the logits generated by these multiple EPTs are averaged, functioning similarly to an ensemble approach. For example, if three EPTs are used per prompt token, the next-next word prediction is obtained by averaging the logits produced by the three EPTs.
Q6: Explanation on x-axis in Figure 8\.
It refers to the tree size as discussed in the paper. The acceptance ratio of 2.6 for a tree size of 500 is reasonable, given that only 3 prompt tokens are used, which effectively bounds the acceptance ratio to 3\.
Q7: How to combine PPD with speculative decoding.
We apply PPD to the draft model to accelerate its draft generation, with the draft model running on the same GPU. In this setup, there exist 2 tiers of guess\&verification processes. The guess-and-verification process is combined for PPD acceleration of the draft mode, while, at the top level, the guess and verification processes are separated between the draft and target models.