Large conferences such as NeurIPS and AAAI serve as crossroads of various AI fields, since they attract submissions from a vast number of communities. However, in some cases, this has resulted in a poor reviewing experience for some communities, whose submissions get assigned to less qualified reviewers outside of their communities. An often-advocated solution is to break up any such large conference into smaller conferences, but this can lead to isolation of communities and harm interdisciplinary research. We tackle this challenge by introducing a notion of group fairness, called the core, which requires that every possible community (subset of researchers) to be treated in a way that prevents them from unilaterally benefiting by withdrawing from a large conference. We study a simple peer review model, prove that it always admits a reviewing assignment in the core, and design an efficient algorithm to find one such assignment. We use real data from CVPR and ICLR conferences to compare our algorithm to existing reviewing assignment algorithms on a number of metrics.
Paper
Similar papers
Peer review
Summary
The paper describes a new algorithm for assigning papers to peer reviewers with the goal of ensuring that review assignments are in the core. Thie motivation behind this is that large conferences benefit from joining many different communities of research and thereby enabling more interdisciplinary cooperation. If review assignments are in the core then subgroups will have no incentive to break off and form smaller, more siloed conferences. An algorithm is described that, under certain assumptions, creates assignments between papers and authors that are within the core and claims to run in polynomial time. This algorithm is compared with two commonly used algorithms on a number of metrics. While CoBRA, the new algorithm, does not provide as much utility as the TPMS and PR4A algorithms it is shown that the other algorithms almost always produce outcomes that are not in the core. The paper concludes by suggesting that some middle ground between these approaches may be a useful path forward.
Strengths
While the problem described by the paper has received a fair bit of research the application of the core appears to be a novel approach to the review assignment problem. The problem is not of critical importance but is one that has room for improvement and can relatively easily benefit from algorithmic enhancements. Overall the paper provides clear high-level descriptions of each step and the motivation behind why the authors believe the core is a useful idea for review assignment. The authors are also clear about the limitations of their work and where it does and does not improve upon the status quo. The experiment section was a useful addition. Despite the results not dominating other algorithms I am glad there is some evaluation of CoBRA.
Weaknesses
While the high level components of the paper are explained quite clearly I found the specific details in need of clarification. A number of grammatical issues exist (often across longer sentences; some issues on lines 8, 209, 226-229, 232, 256) Attempting to reformulate some technical ideas and write with simple, shorter sentences might ease reader comprehension. Your motivation is clearly described but I am not wholly convinced that smaller, more focused venues are generally worse. While they may be less naturally suited for interdisciplinary work they also (intuitively, to me) increase the chance that reviewers and researchers come across work more connected to the areas they are working in. While you claim there are only mild assumptions of order separability and consistency there appear to be a number of modeling assumptions that may not be realistic. You say that each author must serve as a reviewer, which (as you do mention) is not possible to guarantee. You also say that CoBRA "takes as input only the preference ranking of each author over individual potential reviewers" (line 58/59). Perhaps this is common phrasing in for reviewer assignment work but *as phrased* this seems like an extremely strong requirement. Being more clear that you probably mean something like a similarity score would be an improvement.
Questions
Can you clarify what the X/Y matching is that you are referring to in the Datasets section? Does this indicate that you are giving as input to the algorithm only X out of Y papers from a conference? How large do the deviating groups for TPMS and PR4A tend to be? If they are extremely tiny or nearly the entire group I would expect that to be a less meaningful deviation than if they are some distinct subfield of research within the conference.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
The authors have done a very good job of discussing their limitations.
Summary
The authors consider a problem of finding reviewer assignments for conference peer review. Specifically, they aim to find a valid reviewer assignment subject to the constraint that no group of authors can achieve a (strictly) preferred reviewer assignment among themselves and a subset of their authored papers. This can be seen as both a fairness constraint and an incentive-compatibility constraint. The authors provide an algorithm to find such a reviewer assignment and theoretically prove that the returned assignment satisfies the desired properties. They then empirically compare the proposed algorithm to several baselines on subsampled real conference data, and show that it achieves reasonable total welfare while other baselines often violate the core constraint.
Strengths
- The problem of finding a paper assignment in the core within the peer review context is interesting and new (to my knowledge). I think the core constraint is well-motivated as a group fairness constraint and is substantively different from the other notions of fairness considered in prior work. - The writing is generally very clear and the paper is easy to read. - The author preference model used by the paper is significantly more general than the standard additive utility assumed in other works. - The empirical results that standard TPMS assignments have significant numbers of core violations on real conference datasets are of practical interest.
Weaknesses
- The setting assumed by the paper is highly simplified compared to the standard peer review setting: all papers are authored by a single author, and no conflicts-of-interest other than authorship are considered. This hinders the practical relevance of the algorithm. - While the proposed algorithm is claimed to be polynomial-time (Theorem 1), this claim does not seem to be proved and no analysis of the time complexity is provided. - The empirical results are constructed by subsampling small sets of reviewers and papers. While this is done for computational reasons when evaluating core violations, I’m not sure why the social welfare results were also approximated.
Questions
- Could the authors clarify the time complexity of the algorithm? - I assume that the USW results in Table 1 are average assigned similarity, not total similarity as defined.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
The authors clearly stated the limitations of their algorithm, noting that their contribution was primarily conceptual and not practical.
Summary
The authors propose to use core concept in the context of peer review system. Potentially decreasing an overall welfare, the new paradigm offers a fairer treatment of small sub-communities, erasing an incentive to create an independent venue. The paper provides an algorithm for assigning the reviewers on a restrictive case of single author submission. Experimental support their claims.
Strengths
The authors are trying to address a crucial problem in the modern era ML community---how to assign papers fairly and keep more communities happy. Overall I enjoyed reading the paper, found the notion of core interesting and the proposed methodology sound.
Weaknesses
The authors are pretty up-front with listing the weaknesses of the work. The major one, of course, being the single-author case.
Questions
It would be nice if the authors could write a more general model and definition (not only single author) for future references, while then saying that for now they only treat a particular case as a proof of concept.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
Section 5 on limitations is pretty clearly written and the list seems exhaustive. Having purely technical background, I do not feel qualified to comment on the main premise of the work: make smaller communities stay within gigantic conferences. At least in TCS and theoretical ML there are some great smaller conferences (SoCG, SOSA, COLT, ALT, FaccT to name a few). I am not sure that these communities would benefit from being incentivised to stay within say NeurIPS. In particular, it seems to me that fairness and accountability community has greatly benefited from the creation of FaccT (the most recent example I know). So, it would be nice to hear an opinion from people that submit regularly to smaller conferences. That being said, I am positive about this submission as it tries to tackle a rather complicated, ill posed and timely problem.
Summary
This paper proposed an approach that lets the assignment of the peer review model satisfy the core, a fairness requirement over groups of authors, so as to prevent small research communities from having the incentive to deviate and set up their own separate conferences. Through theoretical analysis, the authors found that the proposed method, CoBRA, would return a valid assignment in the core if agent preferences are order separable and consistent. Experimental results show that the proposed CoBRA can produce much fair assignments compared to the baseline approaches.
Strengths
1. This paper addressed an important problem, ensuring group fairness in the peer review process. 2. The proposed CoBRA is theoretically and technically sound. 3. As a theoretical paper, it is not difficult to follow. Several examples and cases are given for the audience to better understand the theoretical analysis as well as the proposed models.
Weaknesses
1. The necessity of achieving fairness in peer review has not yet been well motivated. It is unclear why achieving group fairness will result in the improvement of satisfaction for various communities. 2. The proof of Theorem 1 is not complete. Lemma 1 only validates CoBRA can return the valid assignment in the core but the theoretical analysis on the time complexity of CoBRA to generate the assignment is not complete. 3. The optimality of the valid assignment is not proven. 4. The flexibility of putting the CoBRA in practical use has not been fully discussed. I suggest the authors discuss how CoBRA could deal with the imbalance submission when putting in practical use. 5. It is not easy for non-expert audience to understand this paper, especially why satisfying core is good enough for the review assignments. Some minor suggestions: 1. The proposed methods can be extended to scheduling tasks in many other application scenarios such as task scheduling in collaborative edge computing, federated learning, etc. Merely producing a fair per-review assignment may limit the contributions of the CoBRA. 2. The technical challenges can move to the introduction sections.
Questions
1. How optimal is CoBRA's assignment? 2. How does each author's paper submission amounts affect the fair peer review assignment made by CoBRA? Some areas will receive a huge number of submissions but the other areas may not. The number of submissions by different authors also varied a lot. How does CoBRA address such data imbalance issues and guarantee group fairness? 3. How to determine groups in the experiment?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The limitations have been adequately discussed.
Summary
The authors frame their investigation into group fairness in the peer review setting in the context of large conferences (i.e. NeurIPS, AAAI) by considering a simplified peer review model that enforces the existence of a valid reviewer assignment. Within this framework, the authors apply the fairness notion of "the core" to this setting and present an efficient graph-based algorithm that always returns a valid assignment in the core under minor conditions based on author preferences. The authors empirically validate their method using real data from CVPR and ICLR, and evaluate the cost of utilizing this algorithm (in terms of lost utilitarian and egalitarian welfare) in order to satisfy the fairness notion of "the core" and prevent the incentive for any community to establish their own separate smaller conference (which can lead to research topic insularity and harm interdisciplinary research areas).
Strengths
- While there is a good bit of notation in the model presented, it is kept fairly simple and is clearly described/understandable (to me). This is significant because the paper's primary contributions are methodological and theoretical. - The authors do well in presenting mathematical definitions will more intuitive descriptions, lowering the cognitive barrier on the reader. - The proof provided for main theoretical finding (Theorem 1, along with subsequent lemmas and prop) is very rigorous, and thorough. - Toy example illustrating the execution of the algorithm introduced by the authors (CoBRA) is helpful for building intuition of the method (adding a graph-based visualization of the two companion graphs would be a bonus). - The empirical results and interpretation are well-described in words and may benefit from modified format to enabled more direct comparison between methods and highlight potential performance trade-offs (please refer to suggestion in "Weaknesses" section on elaboration). - The authors humbly acknowledge the limitations of their work before it is ready to be used in practice and highlight some potential follow-up directions for investigation. - Overall comment: much of the content is clearly described, but the presentation structure adds cognitive overhead on the reader and makes it more challenging to understand and utilize the provided information. There are clear strengths in this category, but also real room for improvement. [I struggled to determine the rating in this category, perceived to be between a 2-3 overall. Please refer to "Weaknesses" and "Questions" for more details.]
Weaknesses
Primarily, feedback is around presentation and experimental results/details. Some of these are more significant than others. Overall, the paper is fairly well-written but appears very cramped; the structure and presentation could be improved to support understandability and readability. - Additional details on empirical setup and experiments is necessary to support reproducibility. (Note that there is a line in the Appendix outlining the computation resources and some core code files are included in the supplementary.) - Aligned with the comment under "Strengths", there would be benefit in modifying/better aligning the quanititative results with the discussion/explanation (in Section 4 -> Results). The content is good, but presentation is a little hard for me to follow and has room for improvement (perhaps by highlighting best performance for each measure/column in Table 1?). When page limit is not as tight, it'd be helpful to reorganize Section 4 into subsections (and similarly in Section 4). - It's worth noting that _what_ the authors present is fairly clear, but a clearer motivation of the open problem they are working to address is lacking. (Follow-up question in "Questions" section.)
Questions
1. You mention that theoretically, the CoBRA algorithm is efficient (polynomial time). In terms of actual compute time, how does this scale as a function of the dataset size? Can you provide estimates for the wallclock compute time for the empirical analysis? 2. Why is the introduction of the notion of the core meaningful for the peer reviewer assignment problem space? The Related Work included highlights some relevant work in this area, but does not (from my perspective) very clearly or succinctly motivate the open problem you're addressing in a compelling way. This makes it challenging to determine the potential impact of this work. 3. [A "nice to have" suggestion around presentation] While _technically_ the mathematical definitions of measures computed in empirical evaluation are included in Section 4, they are buried within the main text body; these may be easily missed by the reader or make it harder to understand the paper (it appears that main paper real estate was scarce). I suggest that the authors make room in the main paper to describe the measures used mathematically and verbally (at least prior to publishing) and/or expand upon these along with additional rationale/significance in the Appendix.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
Practical limitations are well outlined in the final section of the paper, along with directions for follow-up work to build upon this theory-forward paper.
1. CoBRA can only produce assignments in the core. Are those assignments good enough for practical use? I don't think the authors discuss this point. It is not easy for the nonexpert audience to understand why satisfying the core is enough for making the assignment. 2. The authors claim that "a group that consists of very few people but generates a large number of submissions may not be offered a strong guarantee because such a group, on its own, would not be able to generate high-quality reviews for so many submissions of its own". However, this is the situation that a conference may face and cannot be handled by CoBRA because "the fairness guarantee offered by the core to every group depends on what the group can achieve on its own".
Thank you for putting in further thought and effort in our paper. We appreciate your comments. Our view on these issues is as follows. **Regarding comment 1:** Being "good for practical use" has two parts. Is it necessary in practice? Is it sufficient in practice? For the first part (necessity), we provide motivations for needing the core in practice in the submission, and elaborate on it further in our common response. For the second part (sufficiency), we believe that one should look towards the algorithm, not the core, which is only a minimum requirement. An algorithm can have aspects other than satisfying the core which can lead to reasonable assignments in practice. This is why we conducted experiments with real data to test CoBRA against state-of-the-art algorithms (TPMS and PR4A) on core and social welfare metrics. We find that CoBRA provides an interesting tradeoff: while CoBRA suffers from a small but not insignificant welfare loss, TPMS and PR4A incentivize realistic deviations (see also our response to Reviewer wnxf regarding the sizes of such coalitions), which CoBRA prevents. In that sense, if TPMS/PR4A are on one end of a spectrum which optimizes welfare, CoBRA is on the other end which focuses on fairness. As the very first paper on the subject, we certainly do not claim that CoBRA is ready for practical deployment. We hope that future work can build on ours to design better algorithms that strike a balance between the two extremes to find assignments with better welfare while preventing most, if not all, realistic deviations. We hope that answers your question. **Regarding comment 2:** We fully agree that a conference may include a community that is small but generates a disproportionately large number of submissions. To be clear, we are not saying that CoBRA will necessarily treat such a community poorly, only that it cannot *guarantee* good treatment to such a community even in the worst case regardless of the problem instance. But no algorithm can do so because, for example, the problem instance may consist solely of such communities and no assignment treating them all well may be feasible (due to the shortage of reviewers). Once again, we hope that future work can build on ours to find instance-dependent bounds, where better guarantees for such communities with reviewing deficits can be provided in cases where other communities exist which contribute reviewing surpluses.
Thank you very much for the rebuttal. I appreciate that the CoBRA is theoretically and technically sound. When the assignments satisfy the core, group-level fairness will be reached for the possible group identified by CoBRA. The problem is that CoBRA works under the assumption that the identified group should have the capacity to handle the number of submissions on their own, which is too ideal. The reality is the number of submissions is far more than the number of qualified reviewers who can deliver high-quality review comments. Many authors will submit more than one paper, and some may even submit ten papers to the top conference like NeurIPS, CVPR, AAAI, etc. Some hot topic areas will receive a huge number of submissions but some may just have a few. Since the research areas are different, assigning the researchers from the areas will a few submissions to review the papers in the hot topic areas hardly produce high-quality reviews. Such imbalance issues cause the fairness problem of the assignment and CoBRA may fail to handle it. That's why I am worried about CoBRA's flexibility in practical use. I suggest the authors include a detailed discussion in the revised paper. Overall, I believe it is a good paper and I am willing to raise my score to 5. Thank you very much.
Response to Authors
Thank you for your thorough response and clarifications. I do think it's useful to include the welfare results on the full dataset, or at least to acknowledge that the ranking of the algorithms is not affected by the subsampling. After reading the other reviews and the author response, I will revise my score from a 6 to a 7: despite the practical limitations of the work (most importantly the single-author assumption), I see the conceptual contribution as interesting and valuable.
Thank you for taking the time to read through our response and revising your score. We will do both: acknowledge the fact that the ranking of the algorithms is not affected by subsampling in the main text and include the welfare results for the full dataset in the appendix.
Thank you for the response to my questions. I do agree that groups of 6-15% of the conference size (or even slightly smaller) can certainly represent coherent subcommunities. I believe some mention of this would be a useful addition to the paper. While the paper is compelling I will leave my review as is based on the limitations that CoBRA must overcome before seeing practical usage.
Decision
Accept (spotlight)