Summary
This paper studies elicting comparison data with truthfulness guarantees, specifically strict Bayesian Nash equilbrium. The authors utilize strong stochastic transitivity to define a Bayesian strong stochastic transitivity model and generalize several existing models. Under this model (i.e., problem setting), the authors propose a peer prediction mechanism based on bonus-penalty and show that it achieves symmetrically strong truthfulness, for both comparison data. Moreover, the authors also identify the conditions (in theoretical results) and the corresponding mechanism that achieves symmetric strong truthfulness for networked data and the general setting. Empirical results on two real-world datasets show that the proposed mechanism achieves better truthfulness or incentive, by showing that being truthful leads to a higher payment.
Strengths
- The studied problem is important and relevant.
- The theoretical rigor is appreciated.
- The proposed approach seems sound, and the definition of Bayesian SST to generalize several existing models is interesting.
Weaknesses
- An important requirement or assumption is the admissible condition to ensure that the assignment contains the necessary triplets. While the authors describe that one can create a superset, with a bounded size, to ensure the admissible condition. It appears that in doing so, one may need the agents to observe and report additional pairs of items (correct me if I am wrong). If so, in practice, one may not always be able to ask the agents to make addtional observations on required pairs of items (e.g., customers' reviews of products of services).
- The abstract and introduction motivates the importance of eliciting comparison data by citing several applications and use cases. However, the covered use cases by the empirical results seem to be less extensive than these.
Questions
In lines 132-133,
> ... where the randomness of $S(\cdot, \cdot)$ comes from both $\theta$ and $T_{\theta}$
The randomness of $\theta$ is due to $\theta \sim P_{\Theta}$. What is the randomness of $T_{\theta}$? How should one interpret this, and its implications?
What other families of strategies are useful? Can strategically mis-behaving agents be accounted for, for instance to exploit this system to minimize someone's expected payment or intentionally aiming to obfuscate identification of the true comparison or ranking? In other words, are these considered families of strategies sufficient to consider some robustness to mis-behaving agents (and which types)?
In lines 173-174,
> ... she would expect that others will ... prefer $a$ over $a''$
What is the intuition behind this? Why would $a$ be preferred over $a''$ when the agent itself only has information that $a$ is preferred over $a'$?
In line 190-191,
> Hence, theorem 3.1 implies that agents’ manipulations can only decrease the probability of transitivity among their reports.
What is the implication of "decreasing the probability of transitivity among their reports"?
Limitations
The method requires a specific problem setting or model called the Bayesian strong stochastic transitivity. But it is shown to generalize several other existing models.