Summary
EDIT: Due to author responses below, I have increased my score from a 6 to a 7.
This work proposes a generalized method for parameterizing the aggregation operation of a message passing neural network (i.e. a graph neural network, GNN). This relies on a modification of the generalized $f$-mean that adds two hyperparameters, $\alpha$ and $\beta$. The authors describe how their method, GenAgg, can approximate commonly used GNN aggregators by varying the hyperparameters and the definition of the function $f$. To overcome issues with defining analytical forms of $f^{-1}$, the authors propose to implement $f$ and $f^{-1}$ with two separate MLPs that learn the inversion.
Then they evaluate GenAgg with three sets of experiments. One tests how well their GenAgg method can approximate common aggregation functions. Then they insert GenAgg into a GNN to see if these results hold true when the aggregator is inside a GNN architecture. Finally, they evaluate a GNN with GenAgg on widely-used benchmarks for GNNs.
Strengths
- The described method and parameterized implementation appear sound and practical for researchers optimizing GNN model designs.
- In particular, the formulation of the f-mean to overcome various constraints and the demonstration of stable training behavior make this method attractive for GNN practitioners.
- The proposed GenAgg method clearly and stably outperforms the other aggregator methods evaluated in experiments presented in the main text.
- The experiments are well-designed and adequately thorough for this venue.
- The analysis of their experimental results is interesting and insightful (e.g. for the Aggregator regressions, the authors discuss where PowerAgg fails and why theory deviates from practice)
- The authors share their code. Upon first inspection the release appears to be comprehensive, and providing a ready implementation is a useful contribution.
I cannot comment on the novelty because I am not familiar with other recently proposed methods for aggregation in GNNs. While I checked the math throughout the main text, I did not review all the parameter settings for the aggregation functions in Table 1.
Weaknesses
- Some claims are made about the sample efficiency of other methods (e.g. lines 47, 194) that should be supported by citations or data.
- While I do appreciate the inclusion of the data in Appendix D, there should be a more in-depth discussion of the method's limitations in the main text.
- I am not fully convinced of the overfitting claim, especially because the pattern of results on PUBMED does not differ much from those presented in Figure 3. If overfitting is the reason that GenAgg does not show strong performance benefits on the smaller datasets in Appendix D, we should see a decrease in its benefit when reducing the size of a dataset in the main text (e.g. PATTERN).
Questions
- On lines 145-146, it says that "we found mapping to a higher intermediate dimension can sometimes improve performance over a scalar-valued $f$" with a reference to Appendix E. This claim is not mentioned or alluded to at all in Appendix E.
- Line 151: is $c$ a scalar?
- Section 6 claims that PowerAgg and SoftmaxAgg training curves plateau at least once (lines 326 - 328). However, I do not notice a difference between these training curves and the others. Can you show this more clearly?
- The plots in Fig 3 are hard to read because the lines and error bars often overlap a lot.
- Please add standard deviation to the table as well, to match the error bars in the plots.
- I also recommend putting a color coded label on the right side of the plot to indicate the ordering of the final test accuracy (e.g. from top to bottom for CLUSTER: GenAgg, Softmax-agg, Powermean-agg, mean, max, ...). While this information is in the table, it would make the plots easier to read.
Minor typos:
- "the a" - line 71
- "are a quantitative measures" (line 110) -- "a" should be dropped
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The authors discuss some limitations of the method in Section 6. However, as noted above, including some of the discussion from Appendix D in the main text would be more appropriate.