Impact of Preference Noise on the Alignment Performance of Generative Language Models

Summary

The paper studies a problem of noise in the preference alignment data for language models and its effect on the LLMs performance on 2 tasks, summarization and dialogue response generation. The authors use a controlled environment where they control the amount and the type of noise injected in the preference data and study the corresponding models' performance in terms of win rate of the aligned model vs the SFT model. The authors work with 3 types of noise: Random noise oracle (uniformly distributed noise across the dataset with a given ratio), Stochastic noise oracle (proportional to the reward difference between the preferred and dispreferred example) and Gaussian noise oracle (modelling the approximated rewards as stochastic gaussian variables). Having found that noise in preference data does impact the alignment performance (but aligning even on the highly noisy data is beneficial for the model), the authors also conduct a study of performance degradation mitigation strategies in the settings of noisy preferences. Out of the two techniques they try, KL-based regularization and confidence-based data filtering, the latter results in superior robustness while not weakening the learning performance.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

A comprehensive focused study of the effect of preference noise on the LM performance is conducted An interesting and useful result is presented on mitigating the harm of preference noise: while regularization does not always help and decreases the alignment efficiency, data selection based on confidence addresses most of the noise types studied

Reasons to reject

The paper has contributions on the analysis side (the experimental studies themselves), modeling side (model training technique with confidence data selection), and resource side (noise datasets/framework). Out of the 3, I see the analysis contribution the most substantial - if analysis papers are welcomed at the conference, that works as a reason to accept, if not - this is a reason to reject. Another thing that would add much substance to this work and is missing from the paper in my understanding is trying the confidence-based data filtering on a real unaugmented dataset with a known presence of preference noise and evaluating the resulting performance difference with a human study.

Summary

The paper investigates the impact of preference noise on the alignment of GLMs and proposes confidence-based data filtering to mitigate this noise. In controlled experiments, three types of artificial noise are inserted on preferences in the tasks of summarisation and dialogue generation. The authors find that 1) even high noise rates an still be beneficial for alignment and 2) that, contrary to regularisation, confidence-based data filtering improves performance. The paper presents a small but focused contribution, with a firmly-states research question. It doesn't only investigate noise, but proposes mitigation solutions. Even though the scope is limited to two tasks, one model, and one noise type at a time, it can provide initial insights into the effect of preference noise and its mitigation.

Rating

7

Confidence

2

Ethics flag

1

Reasons to accept

- The paper is clearly-written, with a small but novel, focused contribution - Controlled experiments with different noise types sufficiently answer the research question

Reasons to reject

- None, since the limitations are acknowledged in the paper.

Questions to authors

- Please change the line format in the graphs so that the different factor can be discerned by the colour blind and when printing in grey scale.

Summary

This paper investigates the impact of noise in preference data on the alignment performance of Generative Language Models (GLMs) and proposes methods to mitigate this impact. The authors provide a systematic study across two tasks—summarization and dialogue generation—highlighting how even slight increases in noise can significantly degrade performance. They introduce a novel framework for injecting and controlling noise types and levels in training data, leading to a detailed empirical analysis.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

（1）The paper is well-structured, and the topic is highly relevant given the increasing reliance on GLMs for various applications. （2）The paper introduces a novel framework for simulating preference noise and rigorously evaluates the impact using well-defined metrics and experimental setups. （3）The results clearly demonstrate the sensitivity of GLMs to preference noise and offer practical solutions for mitigating these effects, providing valuable insights.

Reasons to reject

(1) The experiments are limited to two specific tasks and one type of GLM (T5). Additional experiments with other models and tasks could help generalize the findings more broadly. (2) While the paper presents data filtering as an effective mitigation strategy, the exploration of regularization methods could be expanded, and other potential strategies could be considered.

Questions to authors

(1) The paper handles single types of noise separately. How might the proposed methods perform under conditions of mixed noise types, which are more likely in real-world settings? (2) While data filtering improves alignment performance, could it also potentially lead to overfitting or reduced model robustness due to reduced training data variability? (3) Could the authors elaborate on why increased regularization was not effective? Are there other regularization techniques that could potentially yield better results in combating preference noise?

Summary

Authors present an empirical investigation about (1) the influence of noise in preference data on the model performance after alignment fine-tuning, and (2) what strategies could alleviate the degradation. First they describe three ways of injecting noise into preference data produced by their gold reward model such that those ways align with real world noise we may find in datasets we use. Second, they present an analysis of performance degradation as the amount of noise go up across multiple tasks and noise schedules. Finally, they investigate training regularization as well as data filtering and conclude that some methods might alleviate degradation but not completely. However, they highlight that even noisy preferences may still improve model's alignment compared to the reference model.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

* this is a valuable analysis that is helpful for LLM community working on alignment. Provided filtering technique might be adopted and extended in future work. * paper has a clear presentation and is easy to read and understand

Reasons to reject

* GLM model authors consider is pretty small (<1B) w.r.t. the parameters count. It feels that authors chose T5 family of models for some reason for both RM and GLM. Experimental results could have larger scope if e.g a llama3 8B or llama2 7B (if larger models are infeasible to train for authors) have been considered. * general instruction following benchmarks such as alpaca eval is not considered in the experiments while it would be interesting to see the trends on winrates when numbers would correspond to some leaderboard.

Questions to authors

Following on rejection reasons: - why did you choose T5 770M model as the main GLM model ? Especially given other way more up-to-date LLMs available. - why did you choose reddit TLDR instead of some more up to date tasks / benchmarks e.g. open assistant 2? Especially since alignment is tightly connected to instruction following tasks rather than summarization.

Authors responded to my concerns

The authors have responded to my concerns sufficiently. All in all, this is a suitable paper for the conference, which presents a small focused contribution.