Summary
The paper studies how to handle conflicting alignment targets (e.g., helpfulness and harmlessness) in LLM alignment.
* A metric is first introduced to measure to what extent different alignment targets conflict with each other. With this metric, the authors analyzed and compared the conflicting degree of some existing alignment dataset (Anthropic HH and UltraFeedback).
* Secondly, the authors use GPT-4 to re-label selected examples from the UltraFeedback dataset, categorizing them into six new alignment objectives. The resulting new dataset is called Hummer, and a filtered version of Hummer is also proposed named Hummer-F.
* Thirdly, the authors introduced a dynamic sampling method, which dynamically decides the proportion of samples from different alignment objectives, aiming to minimize the performance gap among the alignment objectives.
Empirical results show that the newly proposed datasets (Hummer and Hummer-F) have substantially lower conflicting levels between their alignment objectives. Applying the dynamic sampling method on the new datasets yield much more balanced results in the jailbreak tests.
Reasons to accept
* LLM alignment is a highly important task in LLM development, and how to trade off between conflicting alignment objectives is a major problem faced in LLM alignment.
* A new metric is proposed to measure the conflicting degrees of alignment objectives. This metric allows the community to better understand the relations between the alignment objectives.
* New datasets and sampling methods are proposed to balance the Reward Models’ performance across different alignment objectives. Empirical studies show promising results.
Reasons to reject
* Notations and mathematical definitions can be significantly improved. The main definitions of the metric, presented in Section 4.1, is quite confusing and hard to follow. For example:
* $D_n^p = {d_1, …, d_n}$, but it’s unclear what is n and what is each element d_1. Intuitively, readers may expect each d_i is a tuple (prompt, winning response, losing response). However, from later descriptions, it seems that each d_i is actually a set of tuples, and hence $D_n^p$ is a set of sets. Also, what is $n$ is not described in Section 4.1? Until very late it became clear that $n$ is the number of alignment objectives.
* In Definition 1 (Alignment Dimension Conflict), it is unclear how the expectation is defined: what is the set of elements the expectation operator operates over, and what is the probability of each element? Also for the numerator $(\bar{u}_k - u_k)^2 _{-}$, it is unclear the square operator and the minus operator which has a a higher precedence: if the minus operator has a higher precedence, then it means $(\bar{u}_k - u_k) _{-} * (\bar{u}_k - u_k) _{-}$; however, if the square operator has a higher precedence, it means $[(\bar{u}_k - u_k)^2] _{-}$.
* When defining ADC-B, a new subscript $m4 is introduced, but it’s unclear what is the relation between $m$ and $n$: are they completely independent, or are they the same?
* All the above unclarities hinder the reader to fully understand the newly proposed metric.
* Ablations are needed. Promising results are presented in Table 1 and 2, but it is unclear which contributes more to the improvements, the new dataset or the new dynamic sampling method. It would be great if the authors can present two results for each dataset, one with the naive sampling strategy, and one with the new dynamic sampling strategy. It would be even better to use other sampling/integration strategies (discussed in Section 1, the last paragraph in Page 1) on each dataset, and compare the performance with the newly proposed sampling strategy.
Overall, I find the paper very interesting and it studies a highly important topic in LLM alignment, but the current paper lacks clarity and should be significantly improved before being published. I may recommend a ‘conditional acceptance’, requesting the authors to substantially improve the definitions before finally accepting the paper.