Rule Based Rewards for Language Model Safety

Summary

This paper proposes an innovation to reward modeling, where rule-based rewards are added to a helpful-only reward model. These rule-based rewards have only a small number of parameters that can be fit with a small amount of data. This is meant to help with fast-changing behavior policies for LLMs, to update the LLM faster and with less pain. Additionally, it is meant to lead to more accurate rewards than could be achieved by rankings with human evaluators, especially if there are many rules to follow.

Strengths

Overall, the paper seems highly original, has signs of high quality (lots of thought seems to have gone into their terminology, experimental designs, ablations, etc.), and significance (the work helps to automate part of the feedback to LLMs that are hard for humans to produce, thus making it easier to create nuanced and complex behavior policies for LLMs to follow).

Weaknesses

The main weaknesses concern the clarity of the submission. I am basically sold on the paper, in the sense that it contained many soft signs that made me update toward "this is an important paper", thus leading to an accept score. However, it is only a *weak accept*, ultimately because I found parts of the submission quite hard to understand. **1. Terminology:** The paper has a large amount of terminology with well-known words that have a very specific meaning in the context of this paper. I found it hard to keep it all together in my head. Examples of terminology are: - content policy - content areas - content categories - behavior policy - completion type - hard refusal, safe refusal, comply - safety data - rule-based rewards - proposition - feature - rules I appreciate that the authors highlight new terminology by using boldface. It might help to additionally create a diagram or other means to make the exact relationship of these apparent and to make it easy to keep it in ones working memory. Potentially the writing style could also help with this, by making the language less abstract or periodically using quick examples for reminders. **2. Clarity in feature extraction:** It is a bit unclear to me whether the feature extraction only extracts features in completions, or also features to categorize the data into content categories. To explain my confusion: Ultimately, the features are proposition-truthiness values. So I looked up the propositions, and found the tables 5 and 6. These only contain propositions for completions, and nothing about the content categories! However, this confuses me: The rule-based reward is just a linear combination of features. But how are you supposed to compute a proper reward when the features do not specify the content category, meaning it is not clear (from the features alone) how the model should respond in the first place? **3. Clarity on ranking of completions:** In Section 4.2, to write down the loss function, it is essential to know how the completions are ranked relative to each other. It is unclear to me how this is done. The authors write: *first, use the content and behavior policy rules to determine rankings among completions based on their proposition values.* So I assume what's happening is roughly the following: First, the feature extractor is used to determine the proposition values. If the content category is also known (possibly also through the feature extractors? This was my previous confusion, see above), then the behavior policy specifies a completion type, which comes, according to table 6, with a rule (i.e., composition of true/false values for features in the completion) that should be present. This is similar in type to Figure 2, but much more complex. Now, what's unclear to me: If you compare to completions, and they have the correct truth-values for some of the propositions but not for others, how exactly do you rank the completions relative to each other? One hypothesis I have is that you somewhere (but not in the paper, I think) have a mapping of combinations of proposition-truthiness values to the categories "ideal", "less_good", and "unaccaptable" that are in Figure 2, and then you only rank based on these three tiers (presumably with equality in the ranking if two completions are in the same tier? How is this handled in the loss then?), but I am not sure. An explanation would be helpful.

Questions

**a.** I wonder whether this work could additionally help to alleviate the problems with distributional shifts in classical reward modeling: usually, you have the problem that when the LLM is optimized against the reward model, the LLM output-data is less and less like the data the reward model was trained on, which can lead to reward overoptimization. I think this problem should be less severe in your work since I would guess the feature extractors to be more robust to distributional shifts than reward models. If the weights in the RBR are interpretable, then this should lead to more overall robustness. What do the authors think about this point? **b.** What's the connection between the content and behavior policies to [OpenAI's model spec](https://cdn.openai.com/spec/model-spec-2024-05-08.html)? They seem obviously strongly related. **c.** The authors write "The design of good prompts for feature extraction is nontrivial, and requires some iteration." Could you say more on how the prompts are designed, also for reproducibility?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

4

Limitations

Throughout the paper, the authors address and acknowledge limitations. Additionally, they have a dedicated limitations section.

Summary

The paper presents a novel approach to enhancing the safety behavior of Large Language Models (LLMs) through Rule-Based Rewards (RBRs). The authors argue that fine-tuning LLMs with human preferences can improve their capabilities and safety but may lead to unintended behaviors due to underspecified instructions. To address this, they propose a system where rules defining desired or undesired behaviors are scored by an LLM grader and used as additional reward signals during reinforcement learning training. The paper demonstrates that this method is effective in achieving safety performance comparable to human feedback while reducing over-refusals. The authors also discuss the impact of different design considerations, such as grader model size and the integration of RBRs with the reward model.

Strengths

Strengths: The paper introduces a unique approach by combining rule-based systems with reinforcement learning for enhancing LLM safety. The submission is technically sound, with claims supported by empirical results. The paper is well-organized and clearly written. The findings are likely to be valuable to both researchers and practitioners.

Weaknesses

While the integration of rule-based systems with reinforcement learning is novel, the paper could further emphasize how this approach can be generalized to other types of large language models beyond those tested. Additionally, future work should address the alignment issues associated with Chain-of-Thought (CoT) prompting (e.g. problems and methods mentioned in [1-3]), exploring methods to ensure that CoT frameworks do not inadvertently propagate or amplify biases and errors. [1] Turpin, M., Michael, J., Perez, E., & Bowman, S. (2024). Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36. [2] Feng, K., Li, C., Zhang, X., Zhou, J., Yuan, Y., & Wang, G. (2024). Keypoint-based Progressive Chain-of-Thought Distillation for LLMs. arXiv preprint arXiv:2405.16064. [3] Yona, G., Aharoni, R., & Geva, M. (2024). Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?. arXiv preprint arXiv:2405.16908.

Questions

1. How do the authors ensure that the rules used in RBRs cover a comprehensive range of safety concerns without becoming overly complex or cumbersome to manage? 2. How does the model handle conflicting rules?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

The paper proposes a method for quickly incorporating safety mechanisms (i.e., refusal to obey the query) in LLMs. Given that production-use cases of LLMs can result in rapidly evolving safety requirements, the paper proposes a mechanism to incorporate a safety-based reward function into the LLM's RLHF / RLAIF step. The key idea includes defining the safety requirements via a set of binary propositions and logical compositions of these, called a rule, for each type of desired LLM response (e.g., one rule for situations where the LLM should refuse to comply, another rule for where it should refuse to comply with an apology, etc.). Given these rules, a set of LLM-based binary feature extractors can be defined as a text classification task. After sourcing completion preference (ranking) data for the updated safety requirements, either real or synthetic, a simple linear model can be fitted, reducing data needs. This now serves as a safety-based reward model. Incorporating the safety-based reward alongside the regular RLHF reward during LLM training allows the LLM to satisfy safe user requests at a higher rate, while properly handling unsafe user requests. This is demonstrated via experiments on safety datasets (XSTest, WildChat) as well as other capability (task) datasets (MMLU, GPQA, etc.).

Strengths

+ The paper proposes an elegant, human-friendly mechanism to quickly incorporate evolving safety requirements into LLMs. This is now a large application area and safety remains a serious issue. Smaller organizations are often under serious time and budget pressures so anything that makes it easier to incorporate safety requirements is likely to have a large impact. + The primary contributions of the paper are algorithmic and empirical. The proposed algorithm for incorporating safety via an additional reward model optimized for a ranking objective is elegant and intuitively clear. The overall approach seems to have sufficient novelty (although a bit more clarity here would be nice). + The results indicate that the method seems to work well at improving safety without losing performance or refusing valid requests, while reducing data needs.

Weaknesses

- The proposition-based RBR (safety reward model) has some novelty but also a lot of overlap with Sparrow. The other aspects of LLM training (SFT, RLHF, PPO, etc.) are also not novel. A more detailed description of the novel contributions of this work compared to prior work, both algorithmic and experimental, would help the reader place this work in the larger, fast-moving prior work involving LLM training and safety. - The experimental section leaves a few important questions unaddressed. These include the error rate of the grader LLMs when scoring propositions, the data quality of the synthetic preference data and root causes for over-refusals. Given that this is related to safety, a careful error analysis would strengthen the paper. Important implementation details (Grader LLM, computational details) to aid reproducibility seem to be missing. - Starting in Section 6, the description starts to get a bit cluttered in places. There's a lot going on in terms of datasets (Human, Synthetic, Helpful, Safety, etc.) and the differences in training and data between the resulting experimental variants (e.g., Table 3 in the appendix) got a bit confusing at times. A table summarizing each variant's key details (datasets, model training) would have made the paper easier to read.

Questions

- How often do the grader LLMs make serious errors? Please characterize the grader LLM quality (feature noise) and implementations (e.g., prompts, if any) in more detail. - What's the data quality of the synthetic preference data? How does it compare with the human preference data? - What's the impact of using the restricted linear RM vs a bigger, non-linear RM? Does it make sense to consider a larger RM given the lower cost of synthetic preference data? - What explains the higher over-refusal rate involving Human-PPO and Human-matchRBR-PPO?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Re. author response

I thank the authors for their detailed response. After reading the response, other reviews and comments, I've increased my score to "Weak Accept". The paper could be much clearer than it is currently and the authors do commit to fixing this. However, as Reviewer QvaG noted, this is a rather large delta from the current version. This prevents me from increasing my score further.