We thank the reviewer for the valuable feedback on improving this paper! Please find below our response to the reviewer’s questions.
> [1] ran an extensive comparison across jailbreaking techniques, and found that [2] and [3] are the strongest attack techniques when comparing to all the techniques mentioned in this paper. Either a larger scope of attacks should be defended against using RobustKV, or proper analysis of patterns within the studied jailbreaks should be provided. Given the evolving offense-defense balance, defense techniques cannot be measured against a select set of attacks without a proper argument for it.
Following the reviewer's feedback, in addition to the jailbreak attacks in Section 5, we further consider more recent attacks. Under the default setting, we evaluate RobustKV against PAP (Zeng et al., 2024), PAIR (Chao et al., 2023), and many-shot jailbreaking (Anthropic, 2024), on Llama2-7B (with more details in Appendix C4).
| KV Eviction Rate | 0% | 15% | 20% | 30% |
| :-----: | :-----: | :-----: | :-----: | :-----: |
| ASR | 46% | 24% | 14% | 8% |
The table above demonstrates RobustKV's defense capabilities against PAP. When tested against the authors' released adversarial prompts, RobustKV reduces PAP's attack success rate from 46% to 14% with a 20% token eviction rate. While PAP employs an interpretable persuasive adversarial prompt that distributes harmful queries throughout the prompt structure, RobustKV maintains its effectiveness by identifying harmful content through token importance scores (rather than their positions). This approach enables successful detection and mitigation regardless of where malicious content is embedded within the prompt.
| KV Eviction Rate | 0% | 20% |
| :-----: | :-----: | :-----: |
| ASR | 2% | 0% |
The table above shows RobustKV's effectiveness in defending against PAIR under the setting of "--attack-model gpt-4 --target-model llama2 --judge-model gpt-4 --n-iterations 20 --n-streams 5 --max-n-attack-attempts 5". RobustKV reduces PAIR's ASR from 2% to 0% when using a 20% eviction rate. Notably, PAIR is a black-box jailbreak attack. While PAIR employs prompt-level perturbation, in contrast to token-level attacks like GCG, its resulting prompts can still be decomposed into two distinct components: the harmful query and the jailbreak prompt. RobustKV successfully identifies and mitigates the harmful query component through its importance score-based analysis.
> [4] relies on in-context learning, which may not be defended against RobustKV. In security, in order to provide adequate evidence of your hypothesis, please seek out counterexamples that might be most challenging.
Following the reviewer's feedback, we extend our evaluation to assess RobustKV's performance against many-shot jailbreak attacks. We first evaluate it on Llama2-7B. The token window limitations on Llama2 (up to 4096 tokens) restrict our analysis to a maximum of $2^5$ shots. Across all tested configurations, from $2^0$ to $2^5$ shots, the ASR consistently remains at zero, aligning with findings reported in Anthropic (2024).
| \# Shot | $2^0$ | $2^1$ | $2^2$ | $2^3$ | $2^4$ | $2^5$ |
| :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |
| Undefended | 58.4% | 80.3% | 85.1% | 92.0% | 91.9% | 93.8%
| RobustKV | 46.0% | 64.5% | 67.8% | 84.5% | 83.3% | 83.5%
We further test RobustKV's performance (using a fixed 20% eviction rate) against many-shot attacks on Mistral, a weakly-aligned LLM. As shown in the table above, while RobustKV shows reduced effectiveness against in-context learning-based attacks, compared to other attacks (cf. Table 1 in the paper), it still attains a substantial reduction in the ASR of many-shot attacks. We acknowledge the limitation of RobustKV in defending against jailbreak attacks using alternative learning paradigms (e.g., in-context learning) and identify the enhancement of RobustKV against many-shot attacks as a key direction for our ongoing research.
> Generating uninformative responses may be a significant challenge to any model developers implementing such methods.
Our objective is to generate uninformative responses to jailbreak prompts while preserving high-quality responses to benign queries. Prior work (e.g., H2O) suggests that LLMs can maintain good performance even with 95% KV cache eviction. As our method requires evicting around 20% of the KV cache, we believe it is practical to maintain model performance. Further, RobustKV's implementation is compatible with any Transformer-based LLMs, as it only requires collecting token importance scores across attention layers and removing low-ranked tokens. Upon completion of the review process, we will release all code and data used in this research to facilitate the reproduction and deployment of our work.