Summary
This paper proposes a defense against jailbreak attacks on large language models (LLMs) using the principle of information bottleneck. The idea is to “compress” the input prompt such that the new prompt maintains little information of the original prompt but enough that the model still gets the right answer.
Strengths
### Significance
The problem of stopping jailbreak attacks is well-motivated, timely, and will have an impact on the progress of AI development, both in the industry and in the academia.
### Originality
I believe there is novelty in the approach taken in this paper. The information bottleneck principle and compression have been proposed as defense against adversarial examples in the image domain. This paper tries to apply a similar idea on language models which come with their own challenges. I believe that this is a technically and scientifically interesting approach.
### Quality
Apart from the points that I will touch on in the Weaknesses section, I believe that the proposed method is technically solid. Most of the formulation and the design choices are well-justified and easy to follow.
Weaknesses
### 1. Experiment design
My most critical comment is on the main result (Section 5.2) and the experiment methodology. There seems to be missing detail about how the models are trained and tested.
1. **Training set of the baselines.** Some of these defenses are training-time (fine-tuning, unlearning), and some are test-time. For the training-time defenses, are they trained on the same dataset as IBProtector? This is an important question because, based on Appendix D.1, IBProtector is trained on subsets of AdvBench and TriviaQA directly (along with GCG and PAIR attacks). **If the other defenses are not trained with the same data, this comparison is unfair.** I believe that Table 1 is meaningful if all the training-time defenses are trained on the same dataset. It is also a good idea to separate training-time and test-time defenses.
2. **All the defenses are not tested against white-box attacks.** Please correct me if I misunderstand this. Based on Appendix D.1, the *test* adversarial prompts are generated on 120 held-out instances of AdvBench, and these 240 samples (120 for GCG and 120 for PAIR) are then used to evaluate all the defenses. Is this the correct understanding? If so, what is the target model for these 240 samples? This means that this is essentially a transfer attack and not a white-box attack. This is essentially even a weaker attack than Section 5.3 where the attacks are unseen. The results from Section 5.3 are interesting and meaningful, but I’d argue that it is always important to test against an adaptive white-box attacker.
### 2. Modeling design decision
1. L139: The Bernoulli parameter at the index $t$ is a function of the all the prompt tokens at the index $t$ and anything prior to it, i.e., $\pi_t = p_\phi(X_{\le t}$). Is this a deliberate design choice? I would think that letting $\pi_t$ depends on the entire input, i.e., $\pi_t = p_\phi(X_{1:T}$), yields a better result. Or, is this more like because $\phi$ is an autoregressive model?
2. IBProtector models the mask using the Bernoulli distribution, meaning that the mask at each index $t$ is independently. This does seem suboptimal. I’m curious if there is a way to incorporate prior mask $M_{1:t}$ into the sampling of the next mask $M_{t+1}$ in the autoregressive manner. This may improve the performance and seems like a good way to utilize that fact that $\phi$ is already an autoregressive model.
3. There are many approximations and heuristics (Eq. (3), (5), (7)) introduced into the original formulation. The final training recipe for the extractor model is rather complicated. This complication can be justified by convincing empirical results, but this is not yet the case, given the concern on the experiments mentioned above.
### 3. Information bottleneck concept
I have several questions and comments on this aspect. Please correct me if I’m mistaken in these aspects.
1. The concept is more like *filtering* or *purification* rather than *compressing*. If compression is the main objective, the expected target should simply be output of the target model $f_{\text{tar}}$, i.e., $f_{\text{tar}}(X_{\text{sub}}) \approx f_{\text{tar}}(X)$ . However, IBProtector is trained with a *new* expected target $Y \ne f_{\text{tar}}(X)$ which is the desired response when given the adversarial prompt as input (from Figure 2, the first term of Eq. (7)). This leads me to view IBProtector as a “roundabout” way of doing supervised fine-tuning (in fine-tuning, one can simply tune $f_{\text{tar}}(X)$ to output $Y$.
2. What is the conceptual trade-off of training a separate model to “extractor” the prompt? Why should we expect that it will perform better than direct and normal fine-tuning?
3. Presumably there are various way to get $X_{\text{sub}}$ from $X$ (e.g., paraphrasing, or compressing in a continuous space). Why does IBProtector use the masking technique? What is the intuition behind this and what are the trade-offs? Please support and motivate this design choice.