Summary
This paper proposes a mitigation strategy called "pure tuning, safe testing" to mitigate harmful finetuning issues for LLMs. The strategy is very simple, basically to use a safety system prompt for inference and do finetuning without such a prompt. The core philosophy is that harmful knowledge in the finetuning stage is learned without a safety prompt, but in inference time the the added safety prompt is used and therefore harmful knowledge will not be activated.
Strengths
1. The studied problem -- harmful finetuning for LLMs by itself is important and has raised widespread public interest among the community, and this paper is one of the early batches of papers to propose a timely analysis and mitigation strategy for the problem.
2. Comprehensive evaluation is conducted to show the effectiveness of the method.
3. The paper is well-written, and the proposed strategy is simple enough to understand, which I think may raise the common interest among the community.
Weaknesses
1. The core issue of PTST is that: given that the system prompt has changed between finetuning/testing, it does not make sense to me that why the helpfulness is not degraded while the harmfulness is lowered. Both benign helpful knowledge/harmful knowledge are learned with the finetuning system prompt, changing the template in the inference time will simultaneously lower helpfulness/harmfulness in my view. However, this is not the case in Table 2 (a) and Table 3(a), which indicates that changing the template will not always lower helpfulness (sometimes even increase helpfulness, e.g., CA->CL). I conjecture the reason is that the length of CL prompt is longer, which elicits better helpfulness performance. An explanation for this phenomenon will be appreciated.
2. The observation in Section 4 that mixing safety data can reduce ASR is available in Vlguard Zong et al. [2024]. I understand that this is a concurrent finding, but it would be nice if the authors could mention and discuss this in Section 4.
3. The experimental results are not intuitive enough. Particularly, I think it is not ideal to use so many prompts (e.g., TV, TA,CV,CA,CL) for comparison. When I am reading Table 2, I am confused about which one is a safety prompt and which one is not a safety prompt, and therefore, I cannot immediately get the intuition shown by the results.
4. The literature review seems to be comprehensive, but there are a few related works missing. Since (Qi et al, 2024), there are a few mitigation solutions proposed to address the same challenges. I would appreciate it if the authors could appropriately cite and discuss these literature:
------------------Before NeurIPS review cycle------------------
[1] Fine-tuning can cripple your foundation model; preserving features may be the solution https://openreview.net/forum?id=VQ7Q6qdp0P (ICLR2024 template)
[2] Immunization against harmful fine-tuning attacks https://arxiv.org/pdf/2402.16382 （ICLR2024 workshop template）
------------------concurrent------------------
[3] Representation noising effectively prevents harmful fine-tuning on LLMs https://arxiv.org/pdf/2405.14577 （NeurIPS2024 template）
[4] Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning httpsImmunization against harmful fine-tuning attacks https://arxiv.org/abs/2405.18641 （NeurIPS2024 template）
[5] No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks https://arxiv.org/pdf/2405.16229 （NeurIPS2024 template）
[6] Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models https://arxiv.org/pdf/2405.16833v1 （NeurIPS2024 template）
[7] A safety realignment framework via subspace-oriented model fusion for large language models https://arxiv.org/pdf/2405.09055 （Elsivier Journal template, first available May, 2024）
[8] Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models https://arxiv.org/abs/2405.17374 （NeurIPS2024 template）
I am aware that some of the listed work is concurrent work (e.g., con-current submissions to NeurIPS 2024). However, it is encouraged to also cite and discuss them, because that will be beneficial for the development of the research field (but the authors should at least cite those existing works that appeared before the NeurIPS2024 review cycle).
5. Baselines for comparison are lacking. As there are already a few mitigation strategies for harmful finetuning issues, I suggest the authors add one or two baselines, e.g., Vaccine[Huang et al., 2024] for comparison.
Limitations
The authors have discussed the limitations, but I suggest the authors discuss the potential impact that the helpfulness will be lower with changing the template (see the first weakness), although this is not that apparent per the authors' experimental results.
Overall, I believe that the idea of changing the template in finetuning/testing should reduce the risk of harmful finetuning, but of course, should come with the potentially negative impact that downgrades the finetune performance. I am willing to increase the score, as long as my detailed questions are answered and the limitation is clearly discussed in the paper.