Carrot and Stick: Eliciting Comparison Data and Beyond

Comparison data elicited from people are fundamental to many machine learning tasks, including reinforcement learning from human feedback for large language models and estimating ranking models. They are typically subjective and not directly verifiable. How to truthfully elicit such comparison data from rational individuals? We design peer prediction mechanisms for eliciting comparison data using a bonus-penalty payment. Our design leverages on the strong stochastic transitivity for comparison data to create symmetrically strongly truthful mechanisms such that truth-telling 1) forms a strict Bayesian Nash equilibrium, and 2) yields the highest payment among all symmetric equilibria. Each individual only needs to evaluate one pair of items and report her comparison in our mechanism. We further extend the bonus-penalty payment concept to eliciting networked data, designing a symmetrically strongly truthful mechanism when agents' private signals are sampled according to the Ising models. We provide the necessary and sufficient conditions for our bonus-penalty payment to have truth-telling as a strict Bayesian Nash equilibrium. Experiments on two real-world datasets further support our theoretical discoveries.

Paper

Similar papers

Peer review

Reviewer A4wM6/10 · confidence 2/52024-06-18

Summary

This paper studies elicting comparison data with truthfulness guarantees, specifically strict Bayesian Nash equilbrium. The authors utilize strong stochastic transitivity to define a Bayesian strong stochastic transitivity model and generalize several existing models. Under this model (i.e., problem setting), the authors propose a peer prediction mechanism based on bonus-penalty and show that it achieves symmetrically strong truthfulness, for both comparison data. Moreover, the authors also identify the conditions (in theoretical results) and the corresponding mechanism that achieves symmetric strong truthfulness for networked data and the general setting. Empirical results on two real-world datasets show that the proposed mechanism achieves better truthfulness or incentive, by showing that being truthful leads to a higher payment.

Strengths

- The studied problem is important and relevant. - The theoretical rigor is appreciated. - The proposed approach seems sound, and the definition of Bayesian SST to generalize several existing models is interesting.

Weaknesses

- An important requirement or assumption is the admissible condition to ensure that the assignment contains the necessary triplets. While the authors describe that one can create a superset, with a bounded size, to ensure the admissible condition. It appears that in doing so, one may need the agents to observe and report additional pairs of items (correct me if I am wrong). If so, in practice, one may not always be able to ask the agents to make addtional observations on required pairs of items (e.g., customers' reviews of products of services). - The abstract and introduction motivates the importance of eliciting comparison data by citing several applications and use cases. However, the covered use cases by the empirical results seem to be less extensive than these.

Questions

In lines 132-133, > ... where the randomness of $S(\cdot, \cdot)$ comes from both $\theta$ and $T_{\theta}$ The randomness of $\theta$ is due to $\theta \sim P_{\Theta}$. What is the randomness of $T_{\theta}$? How should one interpret this, and its implications? What other families of strategies are useful? Can strategically mis-behaving agents be accounted for, for instance to exploit this system to minimize someone's expected payment or intentionally aiming to obfuscate identification of the true comparison or ranking? In other words, are these considered families of strategies sufficient to consider some robustness to mis-behaving agents (and which types)? In lines 173-174, > ... she would expect that others will ... prefer $a$ over $a''$ What is the intuition behind this? Why would $a$ be preferred over $a''$ when the agent itself only has information that $a$ is preferred over $a'$? In line 190-191, > Hence, theorem 3.1 implies that agents’ manipulations can only decrease the probability of transitivity among their reports. What is the implication of "decreasing the probability of transitivity among their reports"?

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

The method requires a specific problem setting or model called the Bayesian strong stochastic transitivity. But it is shown to generalize several other existing models.

Reviewer oRu47/10 · confidence 5/52024-07-12

Summary

This paper considers the problem of eliciting pairwise comparisons truthfully from strategic agents in the absence of ground truth. It considers the popular framework of peer prediction which is a class of mechanisms where reports from "peer" agents are used as a proxy for ground truth in order to design a payment scheme. The problem with using existing peer prediction mechanisms is that they mainly work for settings where tasks are drawn from the same distribution. However, this does not apply to the case of pairwise comparisons, as the two items being compared can be very different. This paper designs a new peer prediction mechanism that is based on the Dasgupta-Ghosh mechanism where the agents also suffer a penalty in addition to rewards so as to avoid spurious "agree to agree" equilibria. The main novelty of the paper comes from the connection it makes to the well-known strong stochastic transitivity (SST) condition for pairwise comparison data. Under the assumption that pairwise comparisons satisfy SST condition, the mechanism cleverly utilizes this condition to ensure that the reward term exceeds the penalty term for truthful agents which is one of the main requirements for truthfulness. The paper also considers other elicitation tasks beyond pairwise comparisons, such as eliciting networked data, and give a general framework for extending.

Strengths

In my opinion, the paper makes a good contribution to the literature on peer prediction. While the proposed mechanism is mainly based on the Dasgupta-Ghosh mechanism, it is a principled way of extending the Dasgupta-Ghosh mechanism to heterogenous tasks and utilizing task specific properties (such as SST) for incentive design. I really like that the paper connects the peer prediction literature to the ranking literature in a meaningful manner.

Weaknesses

I am only concerned about the relevance of this paper to NeurIPS as this is mainly a mechanism design paper. But given the usefulness of pairwise comparisons in training LLMs, there is definitely need for truthful elicitation of comparison data. The results on networked elicitation seem a bit disconnected from the results on elicitation of comparison data.

Questions

Perhaps the framework also works for weak stochastic transitivity?

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

4

Limitations

I did not find adequate discussion about limitations. Perhaps a section on limitations of the mechanism and peer prediction in-general might be useful.

Reviewer c3bN6/10 · confidence 3/52024-07-12

Summary

* This paper proposes a peer prediction mechanism for elicitation of comparison data. * In the model, there is a collection of items $A$, and a set of agents $N$ which privately observe noisy comparisons between items. Comparisons are characterized by a stochastic comparison function $T_\\theta$, parameterized by $\\theta \\in \\Theta$ and drawn from a common prior $P_\\Theta$. * A peer prediction mechanism assigns a pair of items for each agent $i$ to compare. Each agent then observes a noisy comparison signal $s_i$ and strategically reports a possibly different value $\\hat{s}_i$ to maximize their ex-ante payment $\\mathbb{E} [M_i (\\hat{\\mathbf{s}})]$. The goal is to design a mechanism $M$ which elicits truthful reporting. * The authors present Mechanism 1 for elicitation of comparison data, and prove that it is symmetrically strongly truthful (Theorem 3.1). The mechanism is based on a bonus-penalty payment function, which for agent $i$ awards agreement with some agent $j$, and penalizes agreement with some agent $k$. * Section 5.1 presents a generalization of the mechanism to networked data, proving that it is symmetrically strongly truthful for signals sampled from the graph Ising model (Theorem 5.1). Section 5.2 further generalizes by showing a mechanism which is symmetrically strongly truthful for data with uniform dominance. * Finally, empirical evaluation is performed on two datasets (Sushi preferences and Last.fm). Results seem to show stochastic dominance of payout in truthful reporting vs. random reporting.

Strengths

* Presentation is clear and concise. Intuition is given for formal proofs. * The concept of uniform dominance is interesting and seems to be generalizable. * Theoretical results seem to be quite general, and applicable to many pairwise choice models.

Weaknesses

* Empirical evaluation compares truthful reporting against benchmarks which seem relatively “simple” - uninformed random reporting, as opposed to more sophisticated forms of strategic behavior. * Empirical evaluation procedure is not sufficiently clear. Method of constructing the graphs in the empirical section is not presented sufficiently formally. Graph legends and captions are uninformative. * Implications of assumptions are not discussed in detail. Examples for such assumptions - assuming symmetric deviation, the ability to penalize users, assuming that items are a priori similar but ex-post distinct. See below.

Questions

* What is the relation between Mechanism 3 and Mechanisms 1,2? Is it possible to think about Theorem 3.1 and Theorem 5.1 as corollaries of Theorem 5.2? If not, how do they fundamentally differ? * “We assume items are a priori similar but ex-post distinct” L106 - What are the implications of this assumption? When does it apply, and does it restrict generality? * For the given mechanism, is it possible to have a better BNE which is not symmetric? (e.g, a beneficial equilibrium where only some agents report truthfully?) * What would be the implications of agents having limited liability? (e.g if the agents can choose not to participate in the game when they may get negative reward) * Possible typo in L318: “ The figure shows that only about 50% of the agents using an uninformed random strategy receive positive payments, while over 75% of the users in the original dataset receive positive payments” - Percentages seem to add to more than 100%.

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

Results seem to rely on relatively strong assumptions, such as symmetric deviation, and the ability to penalize users. Despite that, such limitations don’t seem to be discussed in detail.

Reviewer A4wM2024-08-09

Post rebuttal

I thank the authors for their prepared response. Most of my questions are clarified and I am happy to maintain my current recommendation of weak accept.

Authorsrebuttal2024-08-10

We thank the reviewer for acknowledging this!

Reviewer c3bN2024-08-14

Thank you for the response! The rebuttal seems to address my main concerns. I increase my score to 6.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC