Safety and trustworthiness are indispensable requirements for real-world applications of AI systems using large language models (LLMs). This paper formulates human value alignment as an optimization problem of the language model policy to maximize reward under a safety constraint, and then proposes an algorithm, Stepwise Alignment for Constrained Policy Optimization (SACPO). One key idea behind SACPO, supported by theory, is that the optimal policy incorporating reward and safety can be directly obtained from a reward-aligned policy. Building on this key idea, SACPO aligns LLMs step-wise with each metric while leveraging simple yet powerful alignment algorithms such as direct preference optimization (DPO). SACPO offers several advantages, including simplicity, stability, computational efficiency, and flexibility of algorithms and datasets. Under mild assumptions, our theoretical analysis provides the upper bounds on optimality and safety constraint violation. Our experimental results show that SACPO can fine-tune Alpaca-7B better than the state-of-the-art method in terms of both helpfulness and harmlessness.
Paper
References (59)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
The paper studies constrained policy optimization for the language model alignment problem. The authors propose a stepwise alignment method that involves two separate steps for fine-tuning a language model: first with a reward and second with constraints. Several advantages of the proposed method are illustrated compared to existing methods, such as simplicity, efficiency, and flexibility. In theory, the authors prove that the reward optimality gap and constraint violation are bounded, assuming linear reward/constraint functions. In the experiment, the authors demonstrate a practical implementation of the proposed method and show better performance than several existing algorithms.
Strengths
- The authors characterize the optimization properties of the safe RLHF, such as strong duality. Although these properties are from the CMDP literature, they are particularly useful for understanding safe RLHF problems. - The authors exploit the constrained RLHF problem structure to show that an optimal policy can be obtained in two steps. This is a useful property, as it allows us to improve existing models with safety constraints by using standard unconstrained RLHF algorithms (e.g., DPO, KTO). - The authors also provide a theory of optimality for the proposed method. Although the linear function approximation assumption is restrictive, this appears to be the first theoretical characterization of safe RLHF. - Despite the proposed algorithm being ideal, the authors provide a practical implementation and test its performance in several variations. Better practical performance is demonstrated through comparison with existing safe RLHF methods.
Weaknesses
- The motivation relies on the existence of a safety model that can be constrained. However, in practice, how to set a constraint on the safety model is not discussed. This can be challenging since a safety model is often inaccurate, and the safety threshold is unknown. - The multiplicative structure of the optimal policy in Theorem 1 assumes an optimal Lagrange multiplier. However, the analysis of the optimal Lagrange multiplier is not provided. - The proposed stepwise method uses any Lagrange multiplier, raising a question: if an approximation of the optimal Lagrange multiplier is used, should we expect to achieve a similar near-optimal policy? - The provided analysis implicitly assumes that an offline dataset can be represented by reward/safety models. However, it is challenging to verify the quality of offline data in practice. What happens if the models can't be represented by the data? - The provided theory assumes linear reward and constraint functions, which can be restrictive in practice. - The choice of an optimal Lagrange multiplier is heuristic in implementation, which is not characterized in theory.
Questions
Below are some other questions for improvement. - What is the meaning of *unparied* in line 110 and line 145? How are paired vs unpaired datasets determined in experiment? - A mismatch in Eq (10): $\pi$ vs $\pi_\theta$. Do reward and safety models share the same parameter in line 120? - What is the difference between Algorithm 1 and the multi-objective alignment $(r,g)$? Pre-selected $\lambda$ works as a preference. - What other preference optimization algorithms can be used in Algorithm 1? - What are some scenarios that involve multiple constraints? Can the authors provide experiments to illustrate these scenarios? - How is the weighting scheme implemented for finding an optimal $\lambda$? What is convergence?
Rating
6
Confidence
4
Soundness
2
Presentation
3
Contribution
3
Limitations
No
Summary
The paper presents the SACPO, a method that optimizes LM policies by sequentially aligning them to maximize helpfulness and harmlessness in either order. By selecting appropriate hyperparameters, the method enables balancing these criteria according to contextual needs. The authors leverage DPO and KTO in various experimental settings to demonstrate the effectiveness of their method, outperforming the prior SafeRLHF approach. SACPO is backed up with strong theoretical validations.
Strengths
S1. SACPO is grounded in strong theoretical foundations, ensuring that the final policy is as effective as if it were optimized simultaneously for both objectives (Theorem 1). The use of the $\delta$-uncertainty quantifier to statistically bound errors in estimation adds a layer of reliability and predictability. By establishing that the error between the estimated and true functions is statistically bounded, SACPO provides a robust framework reliable within known limits. This is useful for safety guarantees of LM harmfulness. S2. While the authors primarily focus on a single safety function to constrain LM harmfulness and do not empirically analyze scenarios with multiple safety constraints, they provide a theoretical framework that outlines how SACPO can be extended to accommodate multiple safety constraints. This facilitates further extensions of this work. S3. Not explicitly mentioned by the authors, but after both optimization stages, if the necessity arises to further fine-tune the language model in either direction of helpfulness or harmfulness, then this option is available. For instance, if further data becomes available, the model is deployed in different contexts, or new requirements for helpfulness or harmfulness arise which can be incorporated into the prompt, then the optimization can continue in one of the directions to ensure that it remains effective and relevant. S4. While the authors do not explicitly address further optimization after the initial stages, SACPO has the potential to facilitate further fine-tuning of the language model as needed. For example, if new data becomes available, the model is deployed in varying contexts, or evolving requirements for helpfulness or harmlessness emerge, the model can be further optimized. S5. SACPO supports combining multiple algorithms for maximizing helpfulness and minimizing harmfulness. While they currently utilize only DPO and KTO, more powerful optimization techniques could be adopted as they are developed. This flexibility enhances the potential to further improve the effectiveness of SACPO. S6. The flexibility granted by selecting an appropriate Lagrangian multiplier $\lambda$, KL penalty $\beta$, as well as the mixing ratio $q$ for P-SACPO enables effective balancing of helpfulness and harmlessness to meet different contextual needs and specific requirements. Fixing $\lambda$ eliminates the need for iterative adjustments and thereby adds stability by avoiding the oscillations and instability encountered with dynamically optimizing primal-dual methods.
Weaknesses
W1. The extent of the evaluation is not clearly defined, particularly in terms of the number and variety of prompts used for testing. This makes it challenging to assess the generalizability of SACPO. W2. I don’t think the evaluation of helpfulness and harmfulness should uniquely be separated to different prompts and responses. Surely, the prompts used in SafeRLHF [1] are specifically selected to ‘trigger’ harmful responses from LMs with higher likelihood, and this is important to detect. However, the goal, ultimately, is to have the model generate responses that are simultaneously helpful and harmless. Therefore, it makes more sense to evaluate these two criteria in parallel on the same prompts and responses. Otherwise, the policy might learn to simply detect prompts that contain ‘triggering’ clauses, and proceed to output very safe but less helpful answers. Contrarily, if the prompt appears safe, the policy can freely generate a maximally helpful answer, while having very low potential of being ranked as harmful. W3. The description of the training protocol involving the PKU-SafeRLHF dataset in the optimization of the base SFT policy using DPO/KTO methods lacks clarity. Specifically, it is not clear whether the same data from this dataset is employed across both optimization stages for helpfulness and harmfulness. Additionally, the duration of the training process is not mentioned. W4. Although used in SafeRLHF [1], I don’t think that solely relying on LLMs to evaluate helpfulness and harmfulness on question-answering problems is a reliable or objective method. Note, that SafeRLHF also incorporated other methods of evaluation. W5. The method introduces several new hyperparameters that necessitate tuning or heuristic selection when trained for other contexts. The choice of optimization sequence, the KL penalty scaler $\beta$, the Lagrangian multiplier $\lambda$, and the mixing ratio $q$ complicate the setup. [1] Dai, Josef, et al. "Safe rlhf: Safe reinforcement learning from human feedback." *arXiv preprint arXiv:2310.12773* (2023).
Questions
Q1. What extra insight do the ELO scores provide? ELO would only be a meaningful metric if methods would be compared with each other, not if they are compared pair-wise with a single baseline. Q2. The main results are presented using a win-rate metric in comparison to the SFT baseline, yet GPT-4 is employed to rate responses on a scale from 1-10. Why is the numerical scoring necessary if the primary evaluation criterion is simply determining whether the new policy outperforms the base one. Why not have GPT-4 directly select the better response from a pairwise comparison, rather than assigning individual scores? Q3. Is the same set of data used for both stages of training? Q4. How extensive and general is the evaluation?
Rating
7
Confidence
4
Soundness
4
Presentation
4
Contribution
3
Limitations
The authors have briefly addressed the limitations of their work. I have pointed out further limitations in the weaknesses section of this review. My main concerns are W2 and W4.
First of all, we would like to emphasize that **our paper mainly focuses on a problem with a single safety constraint**. In relation to multiple safety constraints, **we have shared the same perspective as the initial comments in Strength by Reviewer w7hx.** > S2. While the authors primarily focus on a single safety function to constrain LM harmfulness and do not empirically analyze scenarios with multiple safety constraints, they provide a theoretical framework that outlines how SACPO can be extended to accommodate multiple safety constraints. This facilitates further extensions of this work. Please note that Additional Experiment 2 was conducted **not to support the main claims of our paper but as a supplementary effort** to address the queries raised by Reviewer VoMk and Reviewer 4XGF. > Reviewer VoMk: Can SACPO be generalized to the setting of multiple safety signals? I.e., when using multiple metrics (toxicity, bias, privacy, ...) to measure the harmlessness instead of one single cost. > Reviewer 4XGF: What are some scenarios that involve multiple constraints? Can the authors provide experiments to illustrate these scenarios? **Responses on Additional Experiments 2.** We will now address the questions and comments regarding Additional Experiment 2. Given the limited time frame to complete the experiment, we focused on verbosity bias as an example. This metric was selected because it can be easily evaluated by simply counting the number of words, unlike metrics such as helpfulness or harmlessness, which require pairwise comparison. Our interpretation of the queries from Reviewers VoMk and 4XGF was that they were inquiring whether our SACPO framework can handle multiple safety metrics. Therefore, we decided to prioritize DPO (H) $\rightarrow$ DPO (S) $\rightarrow$ DPO (V) while deprioritizing experiments on alignment orders to ensure that all additional experiments, including Additional Experiments 1 and the one for Reviewer z9np, were completed within the given timeframe. Regarding the evaluation, while several papers (e.g., Dubois+, Saito+) have recently pointed out the verbosity bias in LLMs, to the best of our knowledge, they do not claim that GPT-4 evaluates helpfulness *solely* on generation length. - Dubois et al. "Length-controlled alpacaeval: A simple way to debias automatic evaluators." arXiv preprint arXiv:2404.04475 (2024). - Saito et al. "Verbosity bias in preference labeling by large language models." arXiv preprint arXiv:2310.10076 (2023). Based on these existing research findings, we believe that it is more reasonable to regard that GPT-4 does not evaluate helpfulness solely based on generation length. Additionally, longer responses may be perceived as more helpful since they can contain more information, and humans often prefer more detailed answers. Finally, we emphasize that the verbosity bias is not a primary focus of this paper; it serves merely as an example of a further extension of our work. **W4.** As mentioned during this rebuttal period, the evaluation by GPT-4 is a popular and standard approach, which was especially true at the time of NeurIPS submission. GPT-4 has been shown to be well-aligned with and is a good proxy for human evaluation. For example, in Section 6.4 of the DPO paper (NeurIPS version), they mentioned that “GPT-4 tends to agree with humans about as often as humans agree with each other, suggesting that GPT-4 is a reasonable proxy for human evaluations.” We can also find a similar discussion in Section 4.2.1 of the Safe-RLHF paper, which we used the same evaluation: “When compared to Alpaca-7B, the Beaver-v3 model demonstrated an increase in the Elo score for helpfulness (GPT-4: +244.91, Human: +363.86) and for harmlessness (GPT-4: +268.31, Human: +237.98). Comparatively, the evaluations by GPT-4 and human evaluators are almost consistent.” Based on such previous work on LLM-as-a-Judge or our statistical test, we consider our evaluation sufficiently reliable. That said, as the reviewer mentions, it will be more reliable and unbiased to evaluate using multiple LLMs. Thank you for your valuable comments. Finally, we would be grateful if you could look again at not only the weaknesses of our paper but also the strengths or concerns we have addressed during the rebuttal period.
> Please note that Additional Experiment 2 was conducted not to support the main claims of our paper but as a supplementary effort to address the queries raised by Reviewer VoMk and Reviewer 4XGF. Thank you for highlighting this. I do not consider it as a weakness. > ...they do not claim that GPT-4 evaluates helpfulness solely on generation length... Although not explicitly claimed by prior works, my concern still remains that the response length may have a strong impact on the evaluation of helpfulness. > longer responses may be perceived as more helpful since they can contain more information This is definitely true, however it is not guaranteed that a long response can also simply reiterate the core helpful parts, and is thus perceived by GPT-4 as more helpful. **W4**. Thank you for bringing these references to my attention. They do justify the evaluation strategy more. Although GPT-4: +244.91 - Human: +363.86 seems like quite a substantial gap. Considering the other reviews, the authors' responses to them, and all of my concerns almost fully addressed, I have decided to **increase the score**.
We would like to express our sincere gratitude to Reviewer w7hx for the insightful comments and constructive suggestions. Based on the valuable feedback received during the initial review and discussion period, we will complete the camera-ready paper, ensuring that the information is accurately conveyed to our readers. Your thoughtful remarks and feedback have greatly contributed to enhancing the quality of our manuscript, and we are truly appreciative of your efforts.
Summary
From the perspective of safe reinforcement learning, the author formulates human value alignment as an optimization problem of the LM policy to maximize reward under a safety constraint, and then proposes an algorithm, Stepwise Alignment for Constrained Policy Optimization (SACPO).
Strengths
The author introduces SACPO, an algorithm that effectively enhances the safety of LLMs. The theoretical derivations are solid, and the algorithm's effectiveness is demonstrated through extensive experimental settings.
Weaknesses
1. Although the author included some descriptions of related work in the Preliminaries section, I did not find a dedicated Related Work section in the main paper. This omission significantly impairs readability. It is unclear why the author chose to exclude this section from the main paper. Given that the paper explores the safe alignment of llms from the perspective of safe reinforcement learning, two highly relevant areas of related work would be Safe Reinforcement Learning and Safety Alignment of LLMs. 2. Considering the point 1, I am curious about the relationship between the proposed SACPO algorithm and traditional safe reinforcement learning algorithms. I noticed that the author describes SACPO's two phases in Algorithm 1 as reward alignment and safety alignment. In traditional safe reinforcement learning, such as in the Constrained Update Projection Approach to Safe Policy Optimization, the algorithm's update logic is similarly divided into Reward Improvement and Projection (safety satisfaction). What are the connections between the proposed algorithm and traditional reinforcement learning algorithms, or the challenges and difficulties in extending traditional reinforcement learning algorithms to the LLMs setting? From a reviewer’s perspective, these points are worth including in the main text. 3. I am also working on LLMs Safety Alignment and appreciate the motivation behind this work. I observed that in the experimental section, the author aligns SACPO with DPO(H) and DPO(S), where the latter two are trained separately using the Helpful and Harmless dimensions from Beavertails. Since Beavertails' preference annotations are decoupled—helpfulness is annotated without considering safety—DPO models trained separately are naturally deficient in the corresponding dimensions. I would be interested to see a comparison between DPO models trained with a trade-off between helpfulness and harmlessness and SACPO. For example, how do they perform on datasets like PKU-SafeRLHF and PKU-SafeRLHF-single-dimension? >https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF-single-dimension
Questions
see above.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
see above.
Summary
This paper proposes a new alignment algorithm SACPO to improve the both helpfulness and safety (harmlessness) of language model. SACPO separates the two objectives into two alignment steps. The second step of optimization is equivalent to be an optimization with the policy from first step as the reference policy. Meanwhile, each step can be implemented with reward-free alignment methods (e.g., DPO, KTO). The authors compare the proposed method with SFT base model and show it can achieve higher win rate in terms of harmlessness and helpfulness.
Strengths
- Overall, the paper is clearly written and easy to follow. - The two steps optimization of SACPO can be achieved by reward-free alignment algorithm, which reduces the requirement of computation source and dataset (e.g., it also works for dataset which is not constructed by preference data). - The empirical results show that proposed method exceeds the baseline on helpfulness and harmlessness.
Weaknesses
- Although the authors use P-SACPO as a remedy, it is still questionable that the proposed method cannot get the correct $\lambda^*$. For example, how to step a conservative starting point $\bar\lambda$, how to set the linear interpolation coefficient $q$. - As a general alignment algorithm, the authors should test the performance on other datasets or different models.
Questions
- Can SACPO be generalized to the setting of multiple safety signals? I.e., when using multiple metrics (toxicity, bias, privacy, ...) to measure the harmlessness instead of one single cost. - What are the differences of three subfigures in fig.2? Are they using the same measurements (i.e., are the points in different subfigures comparable)? - Could you compare the win rate of SACPO and baseline against stronger model (e.g., compare it to gpt 3.5 and gpt 4.0 as AlpacaEval does)? - As Remark 2 indicates, the order of two-step alignment will not influence the final policy in theory. However, in fig.2(b), the performances of DPO(H->S) and DPO(S->H) for $\beta/\lambda=0.1$ are very different. Do you have any explanation for that?
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors have mentioned several limitations of this work in Sec.8. One another limitation is that the proposed method only works for the KL divergence regularizer ($D_{KL}[\pi(\cdot|x)\|\pi_{ref}(\cdot|x)]$) as discussed in Remark 1.
Summary
The paper addresses the challenge of fine-tuning a language model (LM) policy to maximize reward while adhering to safety constraints. Building on the concept of Safe RLHF, which introduces a constrained safe RL paradigm for aligning LLMs, the authors propose a novel approach: Stepwise Alignment for Constrained Policy Optimization (SACPO). Unlike the traditional method that simultaneously balances reward and safety optimization, SACPO adopts a stepwise approach, first aligning the LLM for reward and then for safety, or vice versa. They also present a practical variant called P-SACPO, which leverages model merging techniques. Empirical results demonstrate the superiority of SACPO over baseline methods such as Supervised fine tuning (SFT) and Safe RLHF.
Strengths
Merites of the proposed method (SACPO): (1) simple, stable, and computationally efficient compared to Safe RLHF; (2) compatible with different alignment algorithms (DPO, KTO, and IPO) and datasets; (3) it has solid theoretical grounding.
Weaknesses
Experiments on additional datasets could strengthen the paper's findings.
Questions
In the experiments, you have tested only four combinations (DPO (H) → DPO (S), DPO (H) → KTO (S), KTO (H) → DPO (S), and DPO (S) → DPO (H)). Why other potential variants, such as KTO (H) → KTO (S), were not considered?
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
In the conclusion, the authors acknowledge the limitations of their work and the potential societal impact.
**Q1**. Thanks for the clarification. If the comparison is indeed done across all methods, then the use of ELO is meaningful and a good solution. However, this is not clear from the explanation in the appendix. I suggest including how these ELO scores are obtained i.e., 1) across how many trials each pair of methods is compared (is it the full 129+83 prompts for each pair?), 2) what is the order in which the pairs are compared. ELO scores highly depend on these factors, especially if there is such a low number of methods in the comparison. **W2**. I appreciate the response, but I don’t think you’ve fully addressed my concerns. I agree that most prompts in LLM applications are benign and using benign prompts from the AlpacaEval dataset is useful and meaningful for assessing helpfulness, as it is rather unlikely for the model to output anything harmful to benign prompts. My concern is about using red-teaming prompts to only assess harmlessness. I don’t think helpfulness and harmlessness are mutually exclusive. When presented with a ‘harmful’ prompt, the model should foremost not output a harmful response, but meanwhile remain helpful, by explaining relevant nuances about the subject matter. Here’s an example to illustrate my point: if a model learns to detect every red-teaming prompt as harmful, then it can simply give a blank response. Under your separate evaluation approach of only assessing harmfulness, this model would be 100% harmless. This means that the best-performing model only needs to detect when it's being red-teamed and remain silent, and in the meantime provide maximally helpful answers for general benign prompts. Given the example responses from your methods in the appendix, this is, of course, not the case, but the principled point stands regarding the chosen decoupled evaluation approach. Hence, I still believe both criteria should be considered. “First, when evaluations of helpfulness and harmfulness are coupled, safe models are likely to be evaluated as helpful. This means that safety-aligned models potentially obtain an unreasonably high evaluation regarding helpfulness. This is based on our observations in early experiments that DPO (S) were valued as more helpful than we humans thought.” All of this seems more like a problem of a lack of a reliable evaluation method, which further justifies my concern raised in W4. **W3 & Q3**. Thank you for mentioning the training time and dataset usage. Let me further elaborate on my concerns regarding this. Although the wall-clock time, which you provided, is definitely a useful indicator, what I am after here is a more rigorous overview i.e., does one stage of training last until a) the model reaches convergence? b) a fixed number of policy updates have been carried out? c) every datapoint has been incorporated into the training? This is currently left up for interpretation. These details are vital for reproducibility and facilitating a fair comparison for future works. For instance, adding training curves in the main paper or appendix would already greatly remedy this issue. In the main paper, all I can find is “We utilize the PKU-SafeRLHF preference dataset with more than 30,000 expert evaluations”. This alone is not very indicative. Appendix G3 solely presents the hyperparameters which was not really what I addressed in my previous inquiry. I can see that each stage was trained for 3 epochs, but there’s no telling what constitutes an epoch. Although, of course, it is great that the source code is accessible, I don’t think it should be expected for readers to go digging in the code for these important details. **W4**. Thanks for including the reference. It does make it more justifiable since it has already been widely used. However, it would require additional analysis to establish LLMs as competent evaluators for safety alignment. For instance, I am curious whether evaluating the same answer 10 times would yield the same helpfulness/safety score from the LLM. I acknowledge that human evaluators are also imperfect in this matter, as they might assign a different score to the same problem under different internal or external circumstances. However, it is unclear how drastic this variance would be for GPT-4. I don’t see any guarantee why GPT-4 wouldn’t potentially flip its preference between two presented answers on every consecutive trial. Or, for instance, whether the LLM would suffer from anchorship bias and grant one answer a very high score, because the other is comparatively much worse, although both answers are objectively very unhelpful. Note that my concern here is not how much SACPO outperforms SafeRLHF or SFT, but that evaluation solely relying on LLMs (especially only a single model and a single trial) is not a reliable metric (at least not yet). As the core of my main concerns (W2 & W4) remain unaddressed, **I am currently unwilling to raise the score**.
We would like to express our sincere gratitude to Reviewer w7hx who read through our responses. **Q1.** Thank you for your helpful suggestions! In our early experiments, we also noticed that ELO scores highly depend on the comparison order and implemented our evaluation protocol to address this. For each ELO score, we conduct 50 evaluation trials. The final score is the average of these trials. To address the inconsistency caused by comparison order, we shuffle the order in each trial. We used all 128 benign prompts for each pair when evaluating helpfulness and all 83 red-teaming prompts for each pair when evaluating harmlessness. Furthermore, to increase the consistency in each trial, we also duplicated these evaluation results 10 times before computing each trial's ELO score. We will include the details on how we obtained the ELO scores in the camera-ready version. **W2.** We have conducted additional helpfulness evaluation using red-teaming prompts. We make GPT-4 evaluate each response pair three times and take the mean score. | Model | Helpfulness win-rate (red-teaming prompts) | Harmlessness win-rate | |-------|--------------------------------------------|-----------------------| | alpaca-sft | 0.50 | 0.50 | | beaver-v1 | 0.76 | 0.70 | | DPO(H) | 0.62 | 0.49 | | SACPO: DPO(H) -> DPO(S) 0.05 | 0.74 | 0.70 | | SACPO: DPO(H) -> DPO(S) 0.025 | 0.84 | 0.81 | | P-SACPO: q=0.75 | 0.84 | 0.77 | In this evaluation setting, SACPO and P-SACPO still outperform alpaca-sft and beaver-v1, showing that the models trained by SACPO and P-SACPO can simultaneously provide helpful and harmless responses versus harmful prompts. Another important observation is that the helpfulness score of DPO(H) is comparatively low. We observed that GPT-4 often gives low helpfulness scores for the 'informative but harmful' responses produced by DPO(H). We hope these results can elaborate on the concerns and explain why we use benign prompts for evaluating helpfulness. On the other hand, we agree that such evaluation should be considered, and we will include this evaluation result in the camera-ready version. **W3 and Q3.** Thank you for your constructive feedback! In each stage, we trained each model for a fixed number of epochs, as shown in Appendix G.3. An epoch means one complete pass through the entire training dataset. We also confirmed that our models converged with the chosen epoch numbers. We will add these important details in the camera-ready version. **W4.** We have taken much care of the variance of evaluations by GPT-4. In our paper (Appendix G.6), we have included the experimental results of significance tests by running each evaluation three times. We observed and confirmed that the evaluations by GPT-4 are fairly consistent in our experimental settings. We appreciate your valuable suggestions and feedback for improving the quality of our manuscript.
This is not one of my initial concerns, but there are several issues **and a potential major flaw** with **Additional Experiment 2: SACPO with multiple safety signals** in the global response: 1. I am willing to accept the definition that harmlessness is considered as safety, but to correlate verbosity with safety is simply not accurate. Note, that I don’t mind that verbosity is chosen as an additional constraint, but that it is interpreted as a safety signal. I would redefine it as a multi-constraint or double-constraint alignment problem. 2. To make this experiment fully credible, it is also necessary to show other alignment sequences, not only DPO → (H) → DPO(S) → DPO(V). One of the claims of the paper is that SACPO facilitates any order of alignment and can be combined with any optimization algorithm. Therefore, this should equally apply to the double-constraint scenario. 3. The results are presented improperly. If helpfulness and safety are evaluated in a pair-wise comparison, why is verbosity an exception? It would be more coherent to maintain the same evaluation strategy. Additionally, this would allow for a 3D plot for visualization, although I am not expecting this to fit in the single-page additional results PDF. 4. **GPT-4 seems to assess helpfulness solely based on the word count.** From the results in Table 1, we can clearly observe that the helpfulness achieved with SACPO starts to closely match that of the base SFT model as the generation lengths become more equal. This further diminishes the credibility of the chosen evaluation metric and begs the question of whether the helpfulness alignment with SACPO actually leads to more helpful answers not just lengthier ones. As the authors themselves stated: > Although we observed a slight reduction in helpfulness, this can be related to the issue that GPT-4 evaluation tends to prefer longer answers. **W2**. Thank you for providing the extended results. This sufficiently addresses my concern. I believe it is important to demonstrate that SACPO can remain helpful while dealing with harmful prompts. **W3**. I appreciate the clarification. This satisfies my concern. **W4**. Thank you for pointing out the significance testing experiments. This indeed resolves the concern regarding the evaluation stability. However, in addition to my initial concerns about using a single LLM as the sole evaluator, the problems keep re-emerging, especially the one I addressed above. Moreover, in your response to W2, it was mentioned that > GPT-4 often gives low helpfulness scores for the 'informative but harmful' responses. This is not an issue with the dataset nor the strategy of evaluating helpfulness and harmlessness together, but once again that solely relying on an LLM for evaluation is inadequate. The deficient evaluation may jeopardize the results and the conclusions drawn from them. I still don’t think that GPT-4 is a fully quantitative and reliable metric. This substantially decreases the quality of the paper. **I urge the authors to show that GPT-4 does not evaluate helpfulness solely based on the output length of the model or I may be inclined to lower the score.**
Thank you for clarifying the choice of $\lambda^\star$. I have a follow-up question: when the order of your stepwise alignment is changed, how do you use the same averaging scheme? how to set $\bar{\lambda}$? The optimal policy may have different sensitivities to reward and safety objectives.
We would like to express our sincere gratitude to Reviewer 4XGF who read through our responses. We consider that there are two approaches when the order of our stepwise alignment is reversed. The first approach is to regard the KL penalty coefficient for safety as $\beta$ and that for reward as $\beta/\bar{\lambda}$. In this context, our SACPO is effectively optimizing an LM policy with respect to $g + \bar{\lambda} \times r$. Hence, by setting $\bar{\lambda}$ a sufficiently large scalar, reward realignment will lead to a model emphasizing reward (i.e., helpfulness). In this setting, we can directly use P-SACPO as presented in Section 6. While we recommend the first approach, the second approach would be to implement P-SACPO using negation rather than addition. This idea is associated with a notion of task vectors, and negation leads to forgetting or unlearning (for more details, we would like the reviewer to see Figure 1 in [Ref]). Even when the order of the stepwise alignment is changed, we can use a similar $\bar{\lambda}$. Then we apply our stepwise alignment in the order from safety to reward. We now have two models with - Model A: Conservative safety - Model B: Conservative safety + Helpfulness. However, both two models are excessively conservative. Hence, we now apply the negation of Model A from Model B with some coefficient $q$; that is, Model B - $q \times$ (Model A - SFT model). By properly tuning $q$, we can obtain a model that balances helpfulness (i.e., reward) and safety. We appreciate the valuable questions raised by Reviewer 4XGF for improving the quality of our manuscript. [Ref] Ilharco, Gabriel, et al. "Editing models with task arithmetic." The Eleventh International Conference on Learning Representations.
Thank you for your clarification. As shown in Figure 2(b), different alignment orders lead to varying performance gaps, which can be quite large. It seems the authors do not have a complete answer for this. I expect that this issue will become more severe in the multi-constraint case, as different constraints introduce different sensitivities through the Lagrangian multipliers. I am wondering how challenging to set a large vector $\bar{\lambda}$ for multiple constraints. I am not sure how the stepwise method scales up to multiple constraints.
We deeply appreciate Reviewer 4XGF for the insightful comments and constructive suggestions. As the reviewer mentions, different alignment orders lead to varying performance gaps. We hypothesize that the poor representation ability of the LLMs or optimization error regarding DPO might contribute to this phenomenon, though we do not have a definitive explanation. This represents an interesting direction for future research to analyze the gap between theory and practice. A similar comment was provided by VoMk, and we now concur with Reviewers 4XGF and VoMk that this is an important point. We will discuss this in the Limitation section of the camera-ready version. Regarding the questions about multiple safety functions, we currently do not have clear answers, and we sincerely acknowledge this as our honest response. Future work will be necessary to investigate the challenges of setting a large vector for multiple constraints or how the stepwise method scales up to multiple constraints. Finally, we would like to clarify that **our paper primarily focuses on a problem with a single safety constraint**. The theoretical results concerning multiple safety constraints presented in Appendix B are intended to facilitate further extensions of this work. Generally, extending an algorithm designed for a single constraint to one that handles multiple constraints is not straightforward. For instance, this complexity is evident in safe (or constrained) RL algorithms, where many algorithms inherently struggle to manage multiple safety constraints both theoretically and empirically. Based on the reviews, additional experiments, and discussion with reviewers, we now believe more strongly that our SACPO can serve as a solid foundation for future research due to the following reasons: - SACPO enjoys theoretical justification for the extension to multiple safety constraints. - We were able to conduct an additional experiment with two safety constraints within the short rebuttal period, demonstrating SACPO's stability and computational efficiency. We are grateful for your valuable suggestions and feedback, which have significantly contributed to improving the quality of our manuscript.
We are glad to hear that Reviewer VoMk's concerns have been addressed. We would like to express our sincere gratitude to the reviewer who read through our responses and other reviews.
We are delighted to hear that Reviewer KBZp's concerns have been addressed! We extend our heartfelt gratitude to the reviewer for taking the time to consider our responses carefully!
We extend our heartfelt gratitude to Reviewer 4XGF for taking the time to consider our responses and other reviews carefully. Thank you for your constructive review and discussion!
Decision
Accept (poster)