Summary
The motivation behind this paper is to address the critical challenges of finetuning language models (LMs) from pairwise preferences due to the underspecified nature of natural language. Direct preference feedback is often uninterpretable, inconsistent, and difficult to provide, especially when multidimensional criteria are involved. These issues arise from incomplete instructions or the diverse backgrounds of the individuals providing the feedback. To tackle these challenges, the authors propose a two-step preference modeling approach: first, selecting a context to resolve under-specification, and second, evaluating preference with respect to the chosen context.
There are mainly 4 contributions the authors have made:
**Decomposition of Reward Modeling Error**: The paper introduces a method to decompose reward modeling error into two components: context inference error and context-specific reward modeling error. This decomposition supports the idea that supervising both context and context-specific preference could align models more effectively with diverse human preferences.
**Context-Conditioned Preference Datasets**: The authors contribute several novel datasets designed to investigate the ability of LMs to evaluate context-specific preferences. These datasets, termed "preference reversal" datasets, isolate context-specific capabilities by disentangling them from general preferences.
**Context-Aware Reward Model**: The paper demonstrates the development and finetuning of a context-aware reward model, showing that this model achieves performance comparable to or exceeding that of state-of-the-art models like GPT-4 and Llama 3 70B. The context-aware model also shows improved performance in context-specific tasks.
**Experiments and Benchmarking**: The authors conduct experiments to benchmark the context-specific performance of various models, highlighting that current models benefit from additional context but often fail to fully utilize it. Finetuning with the preference reversal datasets significantly enhances the models' context-specific performance.
Strengths
This paper presents a novel approach to preference modeling in LMs by integrating context-specific evaluation. The key contributions include the introduction of context-conditioned preference datasets, the decomposition of reward modeling error, the development of a context-aware reward model, and comprehensive experiments to validate the approach.
Weaknesses
## Under-supported claims
In section 3.3, the author included various claims that I believe to be interesting but only so when they are better-substantiated.
**Assumption about Cardinality** The hypothesis that the cardinality of the space of contexts is smaller than that of the space of possible completions given a prompt is stated without empirical or theoretical justification. This assumption is critical to the argument about data efficiency in context annotation versus preference annotation. Without evidence or a rationale to support this assumption, the argument remains speculative.
## Details of the dataset curation
The reviewer appreciate the author's efforts in curating the dataset.
Section 4 briefly mentions that the dataset was generated using GPT-4 Turbo with a series of prompts designed to maximize validity. However, it lacks specific details about the prompts, the criteria for selection, and the methodology for ensuring the validity of the samples. This ambiguity makes it difficult to assess the soundness of the curation process.
## Clarity
While the paper presents interesting ideas, there are some areas in writing and presentation that could benefit from improvement:
- The introduction and related work sections are detailed, but they could be more concise to better highlight the key contributions and their significance.
- The figures and tables, although informative, could be optimized to more effectively convey the main points. Some elements might appear redundant or unclear.
- Enhancing the narrative flow would help in making the logical progression of the arguments clearer and more accessible to the reader.
Questions
## Annotation and intent inference
I particularly do not understand the assumption related to Bernoulli distribution:
Section 3.1 states that an annotator implicitly infers intent i from prompt x and samples a preference from a Bernoulli distribution. This process of intent inference by annotators is critical yet highly under-explained. There is no empirical evidence or studies cited that demonstrate annotators' ability to accurately infer intents from prompts. Additionally, the variability in annotators' interpretations and the potential biases they introduce are not addressed.
## Intent distribution
The paper asserts that both users and annotators may possess or infer a distribution of intents and that annotation for most preference queries involves a distribution rather than a specific intent. This is a substantial claim that needs empirical support.
## Data curation process
I would appreciate it if the authors can provide further details on data curation. In particular, how the quality of the synthetic data is maintained.
## Human validation
What is the scale of this validation, and how are the human validators selected? Is the sample size sufficient to generalize the results to the entire dataset?