Calibrating "Cheap Signals" in Peer Review without a Prior

Summary

The paper tackles a problem in peer review where reviewers may provide noisy/biased ratings for papers. The paper investigate a "one-shot" scoring process inspired by the "Surprisingly Popular" method, that can rank papers by their true quality without any prior knowledge, even if different papers have different noise levels and biases. Their method relies on eliciting reviewers’ predictions for a random reviewer’s signal and using this for calibration. Specifically, their model assumes that there is a prior distribution over the states (where each paper's state in the "clean" setting is the distribution of reviewers’ signals) and that reviewers are perfect Bayesians who can update their beliefs based on their private signals. In the "noisy" setting, reviewers can only observe noisy versions of their signals, determined by a random mapping that depends on the noise level and the bias vector. Then, the paper defines the true quality of a paper as the expected score in the clean setting, and aims to design a method that can rank papers by their true quality with high probability, even if they have different noises/biases. For this, they introduce a surprisal vector that measures how "surprising" a paper’s state is compared to the prior distribution, and is normalized by the correlation between reviewers’ signals. The key result is that the surprisal vector is noise-invariant, i.e., the surprisal vector is the same in the clean and the noisy setting, regardless of the noise model. Then, by computing an empirical surprise-based score for each paper, the mechanism papers can be compared in a noise-invariant way and are consistent with their true qualities.

Strengths

- The paper is tackling an important problem, which is the question of how to de-bias and de-noise reviewers to obtain a clearer view of paper quality. - The observation that the surprisal vector is invariant under noise is insightful and interesting, as was the idea to look to the surprisingly popular mechanism.

Weaknesses

- The presentation of the paper could be improved. The intro feels a bit unfocused, there are a number of grammatical errors, and the paper is a bit notation heavy. It might help to move the notation table from the appendix to the main body, and also expand on the "high level ideas" to more provide a clearer overview of the proposed method. It took a couple reads to understand exactly what was being proposed. - The assumption of calibrated agent predictions is not clearly justifiable in a context where agent signals are assumed to be potentially biased. - In line 162, the result cited by Kong et al. (2018) is outdated, suggest updating with more recent results that improve on this: Schoenebeck, Grant, and Fang-Yi Yu. "Two strongly truthful mechanisms for three heterogeneous agents answering one question." ACM Transactions on Economics and Computation 10.4 (2023): 1-26. Srinivasan, Siddarth, and Jamie Morgenstern. "Auctions and prediction markets for scientific peer review." arXiv preprint arXiv:2109.00923 (2021).

Questions

- How do you handle det(U) being negative in practice? - Can you give some intuition for why is the normalization of the surprisal vector important? - Do the exponential bounds in the general case not depend on the cardinality of the signal space? Can you give some intuition for why this is the case? - Are the results robust if noise in agents' predictions of other agents' signals correlated with the noise in their own signals?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

The main limitation I see is that it is unclear what to do if some of the computed statistics (like det(U)) violate the non-negativity requirement. The authors are up front about other limitations. These aren't ideal, but may be OK for the goals of this paper. - Authors assume no incentive issues, i.e., that all agents report truthfully. However, this is unlikely to be the case in practice, especially if participants know that there is - The authors assume that agents are calibrated in their predictions, even if their signals may be biased. This does feel like a bit of an odd assumption, and is unlikely to be true. Why would signals be biased, but predictions of others' signals be calibrated?

Summary

This paper proposes a method to calculate peer-reviews scores for papers in the presence of systematically biased noise, such that the score of a paper with a higher expected score in a noise-free regime is higher than the score of a paper with a lower expected score in a noise-free regime with high probability (as the number of peer reviewers grows large). The method does not use reviewers' historical scores to compute their priors. Instead, the method asks reviewers to predict the scores of other reviewers (similar to the Bayesian truth serum). Numerical experiments confirm the theoretical results that the proposed score is better able to distinguish between "good" and "bad" papers under systematically biased noise compared to a baseline that just averages the uncalibrated peer review scores. **Post-rebuttal:** I am raising my presentation score to 4 (excellent).

Strengths

The paper is clear and precise, and spends sufficient time setting up the problem and building intuition. The proposed method adapts a well-known idea (Bayesian truth serum) to the setting of systematically-biased noise in peer review in a principled manner; this is novel to the best of my knowledge. It is also non-trivial, and provides a useful way to construct peer-review scores without needing historical data for each peer reviewer.

Weaknesses

The generality of the exposition also makes it a bit difficult to follow. I think just limiting the main body of the paper to the binary decision case (accept, reject) and adjusting the notation for this (eg. a for accept, r for reject) would make it easier to follow the main argument of the paper. The generalization to multi-valued decisions (eg. review scores) could be relegated to an appendix or to just one theorem. The numerical experiments do not consider SP as a baseline (possible with some modifications since it cannot be directly applied). A naive application of SP would be the most related method in prior work for the problem considered in this paper. Hence, it would help to see how this would perform.

Questions

Q. Could the authors report numerical results from a naive application of SP as a baseline?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

The authors have noted the limitations of their work.

Summary

The paper considers the problem of comparing two papers based on noisy ratings, where the noise can be arbitrarily biased for different papers. The paper elicits from each reviewer both a rating and a distribution of predicted ratings from other reviewers’ (based on a Bayesian update of the common prior, known to the reviewers). The paper proposes a scoring method that recovers the correct comparison between papers from the noisy reports, which essentially corrects the reported ratings using the reported prediction distributions. The authors theoretically show a bound on the error probability of a comparison based on their scoring method, and experimentally evaluate the comparison accuracy.

Strengths

- The problem considered by the paper, calibrating biased noise in a one-shot comparison between items, is conceptually interesting. The authors motivate this problem well in the peer review setting with examples of cases where multiple reviewers may have the same bias. - The authors prove a theoretical bound on the convergence rate of their method’s error probability to 0. The experimental settings considered are thorough (and demonstrate aspects of the theoretical result).

Weaknesses

- While the problem itself is interesting, the assumption that all agents share and know a common noisy prior is very strong; e.g., in the peer review examples in the introduction, while the noise may be biased in the same direction for all reviewers, the noisy prior may not even be similar for all reviewers. Given this assumption, I’m unsure about the significance of the proposed method, which essentially recovers this prior (as in past work) and uses it to standardize the reports. - I found the clarity of the writing to be generally poor, leading to some significant confusion at points (particularly in Sections 1 and 2). For example, in Section 1, I did not understand the claim following Example 2 (that a higher noise level would result in a lower expected score). In Section 2, it was difficult to follow which aspects are observable by agents and by the mechanism.

Questions

- Could the authors expand on what makes the setting of this work unique as compared to the settings of the past works referenced in Section 1.1 (Lines 123-129)? Are none of these works applicable as a baseline for comparison? - The invariant surprisal vector concept is given a lot of focus (Section 3), when the purpose of this concept seems to only be that it’s used to prove Corollary 1. Since the surprisal-based score (in Section 4) is the main contribution, I might suggest making it more clear in Section 3 what the purpose of the invariant surprisal vector is, in order to keep the focus on the main contributions. - Could the authors clarify the aspects of Section 1 and 2 referenced above?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

1 poor

Contribution

2 fair

Limitations

The authors adequately address the limitations of the assumptions made (perfect Bayesian and identical reviewers).

Summary

This paper aims to detect and correct bias in Peer Review. They propose a one-shot noise calibration process without any prior information. Experiments are conducted on the binary case to show the effectiveness of the proposed method.

Strengths

1. The studied problem is important. 2. Theoretical guarantee is provided for the proposed calibrated score. 3. The organization of this paper is clear.

Weaknesses

1. The rationality of the proposed surprisal scores needs further support. For example, in Figure 1, it seems that the left paper with lower negative reviewers’ prediction is the same as the right paper with higher negative reviewers’ prediction, while the left paper receives 1 accept and 2 reject while the right paper receives 2 accept and 1 reject. This result is confusing since the reviewers may give low P_0,1 simply due to the poor quality of the left paper. 2. The experiments are weak in demonstrating the generality of the proposed method. For example, in Sec 6, the authors only conduct experiments on the binary case, lacking the more general settings with more types of possible signals. 3. The theoretical guarantees lack a clear explanation. For example, in theorem 2, why there is 1/2 in the error probability (Pr[S(A) > S(B)|w1^A,w2^B]+1/2 Pr[S(A)=S(B)|w1^A，w1^B])? The guarantee of such an error probability may still remain a gap in the performance guarantee of the proposed method. 4. Some assumption or claims lacks further support and needs to offer more clarification or explanation. For example, in Sec 2.2, why “only consider the noise where M is invertible?” What will be the situation when M is non-invertible? Why can construct the vector q and the joint distribution matrix U from the prediction matrix P as stated in claim 1? Is it universally applicable to assume that 'each individual receives the clean signal with probability 1-\lambda’?

Questions

1. In theorem 2, why there is 1/2 in the error probability (Pr[S(A) > S(B)|w1^A,w2^B]+1/2 Pr[S(A)=S(B)|w1^A，w1^B])? The guarantee of such an error probability may still remain a gap in the performance guarantee of the proposed method. 2. In Sec 2.2, why “only consider the noise where M is invertible”? What will be the situation when M is non-invertible? 3. Why can construct the vector q and the joint distribution matrix U from the prediction matrix P as stated in claim 1? 4. Is it universally applicable to assume that 'each individual receives the clean signal with probability 1-\lambda’?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The authors have discussed the limitations of this paper in Sec 7.

Summary

The paper tries to solve famous problem of removing bias and noise during peer-review process. The authors address the problem in one-shot setting (without historical data) and propose a novel approach that allows to remove influence of bias and noise (under some assumptions on bias and noise) on the ranking of assessed elements (reviewed papers). The approach uses scoring adjustment (used for ranking) and requires additional signal (action) from each reviewer. The authors provide theoretical guarantees and small synthetic experiments.

Strengths

- Overall good organization of the paper storyline (very good Intro!) - More or less clear statements and easy to follow - Synthetic experiments

Weaknesses

- Some drawbacks in clarity related to assumptions and limitations of the setup - Seem limited practical effect - The contribution does not look very enough for NeurIPS - Lack of comparison with alternative ways to improve scoring (e.g., non-1-shot ones, etc)

Questions

1. It would be nice to understand whether the assumptions are realistic (for instance, Lines 59-60 “the correct answer is positively correlated to each agent’s signal” ; Lines 82 83 “assume that the noise is positively correlated .. ”; and the other). Right now the assumptions look theoretical with no clear understanding, e,g., how frequent they are in practice. 2. The specific assumption about particular cases of noises (Lines 203-205) looks very important while being deferred to Sec3 as Definition 1 (instead of being kept in Section 2 where environment of the problem is described). It makes feeling of overclaiming in Intro, where our Results (Lines 85-91) are stated without assumption of considering invertible noise (while other assumptions and limitations are addressed before Line 85). It would be nice to have some presentation improvements to address this gap. 3. The authors claim that it is nice to have the result about best ranking when the number of reviewers goes to infinity. However, I believe it is not so important result to initially stated problem, where the limitations on the number of reviewers is crucial part of the problem (in most peer review practice we deal with 3-5 reviewers, e.g., see the case of NeurIPS). So, OK to have such theorem, but this result seems not so important and relevant to the problem taken for the research. It would be nice, if the authors focus at least the presentation more on how their results help in real practical cases like n = 3-5. (Yes, I see that the authors are aware of such numbers by providing experiments with such numbers. But (a) it might be just because of simplification of experimentation; and (b) the main contribution lies in the theoretical part of the work). 4. While reading from the first page till the end, I had observed changing perception (for sure, mine, subjective) of the studied setting. From the beginning of the paper till Page 6 (Lines 189-190), I thought that the setup assumes that all reviewers assess all papers, while, in practice, each agent review few numbers of papers. Sections 2.1 2.2 have made me sure that the setup is so strange. Only, Lines 189+ reverted my perception. So, I would strongly recommend to somehow help such readers as I figure out this earlier than Page 6. 5.The statement in Lines 168-169 «we focus on homogeneous noise setting where the noise is the same for all reviewers and will discuss the heterogeneous setting in Section 7" makes feeling of covering “heterogeneous setting” by this work. After reaching Section 7, I have found that Section 7 is Conclusion section and heterogeneous setting are discussed as some future work... Missed expectation ======== AFTER REBUTTAL I thank the authors for answering the questions. I hope that the answers will be reflected and fully addressed in the new revision of the work.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Some presentation improvements may address limitations better See points 1, 2 and 3 in the field “Questions”.

Response to Authors

Thank you for your thorough response and clarifications. - The example showing the failure of the SP mechanism was helpful. Since this mechanism is applicable here, I think it would be good to add SP to the experimental comparison as an alternative baseline (or to explain why such a comparison wouldn't be useful). - Regarding example 2: My understanding of the example is still that that bias in the noise is the issue ("conservative noise"): i.e., if the paper with non-expert reviewers has additional zero-mean noise in the ratings, this wouldn't change the expected fraction of accepts. In the paper, only the term "more noisy ratings" is used without mentioning the bias, which I think is more commonly interpreted as zero-mean noise. This is particularly confusing since the following example 3 specifically focuses on biased noise. - I agree that moving the symbol table to the main paper would help improve the clarity. After reading the other reviews and the author response, I will improve my score from a 4 to a 5 for the time being. The contextualization of the work within the literature was helpful, and I hope that the proposed changes will improve the clarity. Despite this, my main concern about the presentation of the work still stands, and I still remain somewhat unsure about the significance of the results.