We are sincerely grateful for your effort and time in reviewing our rebuttal and for your valuable questions.
---
**Q5.** IntactKV pre-saves all system prompts as pivot tokens for Vicuna models, which includes all attention spikes, similar to QFeP.
**A5.**
The discussion continues from our previous response to your Q2, which highlighted the differences between QFeP and IntactKV [1].
Notably, IntactKV selects different pivot tokens depending on the type of LLMs.
- For pre-trained LLMs (e.g., LLaMA), IntactKV selects only the [BOS] token as a pivot token.
- For supervised fine-tuned LLMs (e.g., Vicuna), IntactKV stores all system prompts as pivot tokens.
We denote these methods as IntactKV[B] and IntactKV[P], respectively, according to [1].
In our previous rebuttal, we compared QFeP with IntactKV[B] due to the absence of system prompts for pre-trained LLMs.
When quantizing activations for pre-trained LLMs, QFeP's ability to address activation spikes more effectively than IntactKV becomes evident, as discussed in the previous response.
Nevertheless, further investigation into activation spikes for supervised fine-tuned (or instruction fine-tuned) LLMs would be valuable.
Previous works have found that pivot tokens persist after instruction fine-tuning [1, 2].
This finding implies that activation spikes are transferable and that our methods are also effective.
Indeed, we observed equivalent activation spikes in both LLaMA-2 and its fine-tuned models, such as Vicuna-v1.5 and LLaMA-2-Chat.
While IntactKV[P] may store long system prompts for pivot tokens, QFeP saves a more compact prefix with a length of 3.
Furthermore, QFeP is explicit and prompt-agnostic, providing flexibility for system prompts.
Finally, **QFeP can be applicable to more generalized and effective solutions for addressing activation spikes when applying activation quantization, regardless of whether the LLM is pre-trained or fine-tuned.**
[1] Liu, Ruikang, et al. "IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact." arXiv preprint arXiv:2403.01241 (2024).
[2] Sun, Mingjie, et al. "Massive activations in large language models." arXiv preprint arXiv:2402.17762 (2024).
---
**Q6.** I am curious as to why QFeP does not improve the performance of the 4-bit LLaMA2-13B model in Appendix 3. Could this be due to the influence of an additional prefix affecting the model's performance?
**A6.**
During the preparation of the rebuttal to reviewer geCr, we confirmed that introducing only an additional prefix of QFeP slightly degrades the performance of the FP16 model.
Based on this observation, we hypothesize the following regarding QFeP's contribution to the 4-bit LLaMA-2-13B:
- Case 1 (Atom [3] + QFeP): Given that quantization errors have been minimized via the fine-grained group quantization scheme of Atom, the prepended prefix of QFeP has the possibility to slightly degrade performance. Note that the fine-grained group quantization utilizes a finer granularity than per-token quantization.
- Case 2 (OmniQuant [4] + QFeP): At present, we are unable to determine the specific factor that degrades the perplexity for WikiText-2, although QFeP achieved a performance gain for C4.
To provide further evaluation results, we assessed four zero-shot tasks, as shown in the table below.
The results indicate that QFeP improves the OmniQuant baseline for three tasks, specifically the WinoGrande task.
This is similar to the results of W4A6 LLaMA-2-13B in Table 2 of the attached PDF.
While there are numerous potential factors that could explain the degradation (e.g., trainable components of OmniQuant or softmax quantization in low-bit), we are making an effort to include an ablation study, similar to Figure 6, to clarify the influence of an additional prefix. Thanks again for your questions.
| Model | #Bits | Method | PIQA($\uparrow$) | LAMBADA($\uparrow$) | HellaSwag($\uparrow$) | WinoGrande($\uparrow$) | Avg($\uparrow$) |
|---|---|---|:---:|:---:|:---:|:---:|:---:|
|LLaMA-2-13B|FP16| - | 79.49% | 76.54% | 60.20% | 72.38% | 72.15% |
|LLaMA-2-13B|W4A4|Atom|78.02%|75.80%|58.48%|70.56%|70.72%|
|LLaMA-2-13B|W4A4|$\quad$+QFeM|78.07%|75.43%|**58.49%**|70.40%|70.60%|
|LLaMA-2-13B|W4A4|$\quad$+QFeP|76.99%|**76.19%**|58.22%|70.80%|70.55%|
|LLaMA-2-13B|W4A4|$\quad$+QFeM+QFeP|**78.24%**|75.61%|58.34%|**71.11%**|**70.83%**|
|LLaMA-2-13B|W4A4|OmniQuant|67.25%|41.63%|44.95%|52.57%|51.60%|
|LLaMA-2-13B|W4A4|$\quad$+QFeM|**71.11%**|**48.01%**|46.86%|57.14%|**55.78%**|
|LLaMA-2-13B|W4A4|$\quad$+QFeP|68.93%|38.99%|45.95%|**58.48%**|53.09%|
|LLaMA-2-13B|W4A4|$\quad$+QFeM+QFeP|70.02%|44.89%|**46.93%**|57.46%|54.82%|
[3] Zhao, Yilong, et al. "Atom: Low-bit quantization for efficient and accurate llm serving." Proceedings of Machine Learning and Systems 6 (2024): 196-209.
[4] Shao, Wenqi, et al. "Omniquant: Omnidirectionally calibrated quantization for large language models." arXiv preprint arXiv:2308.13137 (2023).