Counterfactual Evaluation of Peer-Review Assignment Policies

Summary

In this paper, the authors study the counterfactual evaluation of peer-review assignment strategies with randomness. The authors adapt existing off-policy evaluation methods for the specific problem to handle non-positivity, missing reviews and attribution. A framework of off-policy evaluation with different imputation choices are provided. Empirical evaluations are carried out on two datasets to study (1) effect of randomness and (2) weights on different similarity components.

Strengths

1. The paper studied an important problem as online A/B experiments for evaluating peer-review assignments are very expensive and hard to carry out. 2. The authors propose a comprehensive framework based on off-policy evaluation with different choices of imputation methods. Moreover, the authors adapt the existing method to specific cases for paper review. 3. The authors carry out two case studies using the proposed method on two real-world peer review datasets.

Weaknesses

One major concern is that there is no evaluation on the quality for the counterfactual evaluation. It would be much more convincing if some evaluation can be carried out with the help of random A/B experiments or results from existing random experiments. Different methods provide quite different bounds. It is hard to pick the proper method without a ground-truth provided by A/B experiment.

Questions

Please see above section.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

This paper leverages recently proposed strategies that introduce randomness in peer-review assignment—in order to mitigate fraud—as a valuable opportunity to evaluate counterfactual assignment strategies. This paper introduces novel methods for partial identification based on monotonicity and Lipschitz continuity assumptions for the mapping between reviewer-paper covariates and outcomes. The proposed methods are applied to peer-review data from two computer science venues, the authors find that placing higher weight on text similarity results in higher review quality and that introducing randomization in the reviewer-paper assignment only marginally reduces review quality.

Strengths

1. This paper proposes off-policy evaluation as a less costly alternative that exploits existing randomness to enable the comparison of many alternative policies, compared with A/B tests. 2. The paper is well written and organized. 3. The experiments are sufficient and convincing for the conclusion.

Weaknesses

It would be better to compare the proposed off-policy evaluation method with previous commonly used evaluation methods, and analyze the effectiveness of this method.

Questions

1. Why the proposed off-policy evaluation is effective? 2. Compared with other evaluations, can the proposed evaluation method find a better assignment policy?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors provide the limitations in appendix K. 1. The method requires making assumptions: monotonicity and Lipschitz continuity. 2. The method considers estimates of the average review quality across all assigned reviewer-paper pairs, analogous to the common sum-of-similarities objective used for review-paper assignment, the analysis does not consider the impact on individual reviewers on papers.

Summary

The authors consider the problem of evaluating alternative reviewer matching algorithms. They motivate this problem as the cost of running A/B tests. They propose to use off-policy evaluation by exploiting the randomness in review assignments introduced by a fraud-mitigating scheme in recent years. They tackle several main challenges, including positivity and attrition, using imputation, Manski bounds, and improved bounds under monotonicity and continuity assumptions. They apply their method to two CS venues and their findings suggest that randomization does not significantly hurt review quality and higher review quality can be obtained by paying more attention to text similarity.

Strengths

- Sections 1 and 2 are beautifully written. - The authors have a clear set of challenges that they seek to addresse. - The authors run comprehensive experiment that yield interesting takeaways.

Weaknesses

- I would suggest stating the assumptions clear and upfront. For example, should state Y_i is not a function of the assignment and no interference assumption. - The challenges subsection of Section 3 is a little messy compared to the rest of the writing. One way the authors could improve this is by introducing an explicit problem statement. At the moment, it's a mix of discussion, problem statement, and assumptions. - Section 4 could be improve organizationally. The authors should clearly delineate the proposed approaches & assumptions that go with each. - The experiment section is strong but could be improved in terms of (i) explanation and (ii) organization of takeaways. For example, the figure is not explained in the text, and it takes some work to understand/parse the notation.

Questions

- The authors consider self reported expertise and confidence as measures of review quality - I could see these as good, but also noisy and limiting proxies. What do the authors expect are the limitations of this approach? - I'm having trouble telling whether the results given in Section 5 are verified? That is, can we tell whether the proposed approaches improve estimation (beyond the fact that the authors' approaches do what they say they will do, e.g., that the assumptions result in tighter bounds)?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors discuss a possible limitation in their conclusion. They also discuss their assumptions in-line and refer readers to Appendix G for a discussion on assumption suitability. I do think the authors would benefit from a greater discussion of limitations (particularly if the answer to the second question above is no) and their assumptions in one place.

Summary

This work uses the randomness of a recently-implemented peer review paper assignment algorithm in order to perform off-policy evaluation of other (nearby) randomized assignment strategies. It uses multiple methods for imputing missing values and estimating the errors in the estimators of alternative policies' relative value, including fitting proxy values to the data which obey two plausible structural assumptions. The authors perform this evaluation on paper-reviewer matching data from two conferences, and identify practical and actionable recommendations for modifications to the similarity score calculation and assignment algorithm as currently implemented.

Strengths

This paper has a clear premise, purpose, and scope, and it is well executed. It introduces structural assumptions into the this off-policy evaluation framework which are technically and theoretically interesting and apparently novel. It provides actionable recommendations for improving the implementation of paper-reviewer matching algorithms in a major conference, and a framework for performing similar evaluations going forward.

Weaknesses

It seems hard to say how far these recommendations generalize beyond the conferences which are included in the analysis. It would have been nice to have slightly more explicit discussion of how the various uncertainty estimates are generated in the main body of the work. It would have been interesting to see this methodology used to compare the current method to a wider range of others---even deterministic similarity-based approaches---which have seen prior use.

Questions

In your discussion of review quality, you mention competing definitions of/proxies for review quality. How does your choice (self-reported expertise and confidence) compare against the other metrics (eg auto-generated and meta-reviewer-evaluated), and would you expect your results to be robust to this choice? The randomized paper assignment scheme of [17] which you take as the logging policy generates a fractional assignment based on the q constraints and then performs dependent rounding in order to generate an assignment. Since your estimators are linear in these assignments, I presume their means are unaffected by which rounding scheme is used. But what about the variances? (What rounding scheme is used in [17]?) Can you comment on how the choice of rounding scheme might affect the uncertainties as well as your uncertainty bounds, and whether this plays a role in your analysis? Minor comments (no response expected): You might make the definition of review quality which you work with more prominent. The choice of T_i for surrogate quality as well as text weight was confusing (unless these were surrogate text values, though that was not my understanding).

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

It would have been great if this work had included more data for analysis, especially from a second large conference. The scope of the recommendations offered seem somewhat limited by conferences which are a part of the analysis. It would be especially interesting to see this analysis performed on multiple years of AAAI data (provided there is a consistent on-policy used) simultaneously.

Thank you for the rebuttal

Thanks for the response. I have read the rebuttal and remain a borderline 5/6 score. As a note: While I understand that it is difficult to evaluate the review setting given the limited metrics available, there are clear limitations to using self-reported expertise and confidence as proxies for review quality. I believe that the authors should clearly acknowledge these limitations in the work and possibly even account for how they could change the findings. I agree that the setting they examine is difficult given data (un)availability and that the framework is general, yet the authors should acknowledge the limitations in the work beyond stating that it is hard to get good data. Similarly, I appreciate that verification is difficult in this setting; please state this and why your analysis is sufficient upfront in Section 5. In general, my main feedback is to be upfront with limitations and assumptions. Doing so makes a work stronger, not weaker. This was a trend in the paper (see my initial review and the paragraph directly above).

Thank you for the rebuttal

Thanks for the response. I have read the rebuttal and remain a borderline 4/5 score. I agree with the authors that it is hard to carry out but still desired as a gold standard to validate the methodology.