Response to Reviewer FapY (1/2)
Thank you for your acknowledgement and valuable suggestions. Here we address your questions one by one.
**W1-3: Justification about the generalization ability of LLMs aligned with mistake analysis.**
**(1) Generalization beyond training distributions.** We acknowledge that no alignment method, including ours, can fully address all out-of-distribution scenarios. Nonetheless, our model exhibits enhanced generalization capabilities by learning from a wide array of mistakes and their analyses, rather than solely relying on human-labeled instruction-response pairs. As depicted in Figure 5, our model effectively rejects unsafe user instructions in "Goal Hijacking" scenarios for topics not in the training data, a feat not achieved by methods like SFT. This demonstrates our model’s advanced understanding of such complicated mistakes and its improved generalization ability.
**(2) Addressing diverse failure modes and mistakes of alignment models.** Admittedly, even meticulously aligned LLMs face challenges in completely avoiding unsafe responses in novel or unanticipated situations. This necessitates an adaptable "debugging" methodology for addressing numerous corner cases. Section 5.2 details our approach’s efficacy in defending against new-coming instructional attacks. After applying our method, the safety-aligned ChatGLM model demonstrates significant post-alignment improvement, with its Harmless Rate in resisting "Goal Hijacking" attacks increasing from 68.4% to 85.3%. This positions it as the second rank only under the OpenAI's GPT-3.5, as per the SAFETYPROMPTS benchmark [1, 2], while maintaining its helpfulness performance.
**W4 & Q4: Theoretical analysis on how mistake analysis improves alignment.**
In Section 4, Equation (2), we explore the role of mistake analysis in improving LLM alignment from a Bayesian perspective, considering that LLMs operate as conditional generators. Mistake analysis involves examining the model's harmful responses to identify patterns leading to such errors. To train the model in this way, we construct triplets of the instruction $X$, the response $Y$, and the mistake analysis $T$ indicating whether the response is harmful or harmless. Such data refines the model's understanding of the conditional probabilities $p(T|Y, X)$, *i.e.*, the likelihood of a response being harmful or harmless given a certain instruction. Subsequently, we enhance its ability to generate appropriate and harmless responses $p(Y|X, T)$ through guided inference, where the model is required to produce harmless responses during inference (see Figure 1). By focusing on optimizing conditional probabilities through the analysis of mistakes, we enhance the model's ability to generate safe, coherent, and contextually appropriate responses. This is why mistake analysis benefits model alignment. See Section 4.2 and Appendix D for more details.
**Q1: Resilience against adversarial instruction attacks.**
Recent studies [3, 4] demonstrate that even well-aligned models, such as OpenAI's GPT-4 and Anthropic's Claude2, are vulnerable to adversarial instruction attacks, presenting a challenge distinct from aligning with human values. Our research primarily focuses on aligning LLM with human values. We have observed the effectiveness of our method in defending against delicately designed instruction attacks, such as "Goal Hijacking" with superior generalization ability (see Section 5.2). This suggests the potential effectiveness of our method in enhancing resilience against adversarial attacks. Further investigation will be considered in our future work.
**Q2-3: The quality or diversity of mistake analysis for alignment improvements and more ablation study.**
**(1) Quality of mistake analysis.** Yes, the quality of mistake analysis significantly impacts alignment performance. As Table 1 shows, using GPT-3.5-turbo for mistake analysis results in better alignment than using Alpaca-7B itself (see rows 7 and 8). Additionally, guided analysis outperforms unguided analysis, as shown in Table 3 (rows 3 and 4).
**(2) More ablation study on mistake analysis.** In Section 5.3, we present a comprehensive ablation study on the essential components of mistake analysis, including (i) quality of mistake analysis (ii) quantity of mistake analysis and (iii) the source of responses for constructing analysis data. We find that high-quality analysis, whether guided or from advanced models like GPT, leads to better alignment. On the other hand, increasing the volume of analysis data does not always improve results. We observe a decrease in performance when multiple analyses are applied to the same instructions, possibly due to conflicting interpretations in the analyses of the same instruction. Furthermore, we observe that the alignment effectiveness is maximized when the bad cases are generated by the model itself. See Section 5.3 for detailed experimental results.