Sequential Signal Mixing Aggregation for Message Passing Graph Neural Networks

Summary

This paper focuses on the aggregation module in Message Passing Graph Neural Networks (MPGNNs). It tackles the problem that sum-based aggregators, even though widely used, fail to 'mix' features belonging to distinct neighbors, preventing them from succeeding at the downstream tasks. Accordingly, the authors introduce a novel plug-and-play aggregation for MPGNNs, denoted as Sequential Signal Mixing Aggregation (SSMA). SSMA treats the neighbor features as 2D discrete signals and sequentially convolves them with circular convolution by applying 2D Fast Fourier Transform (FFT). The authors also propose several designs to guarantee the practical use of the proposed methods. In the empirical experiments, SSMA successfully improved the performance of several MPGNNs.

Strengths

1. This work has a good motivation and a solid mathematical foundation driving the proposed methods. 2. This paper is well-written. 3. The empirical performance can well support the statement of the paper. 4. According to the best of my knowledge, this is the first work to enhance the aggregation module via circular convolution.

Weaknesses

1. Even though with good theoretical computational complexity analysis, it will be good to have the empirical run-time comparison between the MPGNN and MPGNN+SSMA. 2. According to my knowledge, there are many works using pre-computed PE with conditional aggregation to improve the distinguishability of the neighbors during the 'mixing' of them, (e.g., [1], GraphGPS). They require extra one-time computation on PE but will not increase the complexity of the model inference. There are no comparisons to the line of work. (I am not sure about the exact GraphGPS configuration in the experiments. I have some concerns about it which are listed in "questions") - [1] Dwivedi, Vijay Prakash, et al. "Graph Neural Networks with Learnable Structural and Positional Representations." International Conference on Learning Representations.

Questions

1. What is the configuration of GraphGPS in the experiment? As in the paper, several configurations are provided. 2. In eq (8), the number of separators $m=n+1$ depends on the neighbor sizes; $\mathbf{h}: \mathbb{R} \to \mathbb{R}^m$ is an affine map. How this affine map is determined? Is it learnable? In graph learning, the $n$ is usually assumed unknown and varying. (the same question applies to vector-feature cases as well) 2. Following up on the Weakness 2. The experiment on ZINC follows the 100k parameter setting as ESAN, while other baselines are reproduced. Compared to the 500K parameter setting in GraphGPS, there are interesting points. Regular MPNNs, e.g., GCN, GAT, GIN, PNA, do not show significant improvement on the 500K setting, while GraphGPS is significantly improved. Based on Table 3 and Table B.1 in GraphGPS, GatedGCN ($\sim 0.090$), GraphGPS($\sim0.070$), GraphGPS+NoPE ($\sim0.110$), GINE+RWSE($\sim0.070$). - I am interested to know the improvement of GRASS on GraphGPS on the 500K setting - Can GRASS improve MPNNs with PE?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitation on computational complexity is adequately addressed by the proposed technique.s No potential negative societal impact aware.

Summary

The paper analyzes the discrepancy between the theoretical guarantees of sum-based aggregators and their practical performance, which makes more complex aggregators preferred in practice. They define the notion of neighbor-mixing to explain this gap, and propose a novel aggrgeation module, named SSMA, which builds upon deepsets polynomial.

Strengths

The theoretical analysis is sound and rigorous. The experimental section shows promising performance in practice, and I particularly appreciated that the authors have adjusted the number of parameters in the augmented model to match that of the original, to ensure a fair comparison.

Weaknesses

The paper is hard to follow, and I think adding some intuition on the results would significantly strengthen the paper. For example, while I would have liked to understand the intuition why sum-based aggregators have small neighbor-mixing (which is very briefly touched in lines 116 to 118, but, in my opinion, not sufficiently to have any intuition, and it seems more like a statement that it is the case). Section 4 is generally hard because, while it is supposed to be a way of constructing the model, it ends up being a theoretical digression until 4.3, where then the model becomes clear.

Questions

Can you expand on the intuition of sum-based aggregators lacking mixing abilities?

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

Addressed

Summary

This paper introduces Sequential Signal Mixing Aggregation (SSMA) for Message Passing Graph Neural Networks (MPGNNs), addressing the limitations of traditional sum-based aggregation methods. SSMA enhances the mixing of features from distinct neighbors by treating neighbor features as 2D discrete signals and applying sequential convolution.

Strengths

Treating neighbor features as 2D discrete signals and using sequential convolution is innovative. This approach allows for more effective mixing of features from different neighbors, addressing a significant limitation in traditional sum-based aggregators. By revealing that sum-based aggregators cannot adequately "mix" neighbor features, the paper paves the way for more aggregation techniques. Introducing a convolution-based aggregation module is an advancement in the field. The theoretical foundation of the proposed SSMA method is robust, with well-detailed mathematical formulations and proofs. The paper derives the key equations and supports them with solid theoretical arguments. The practical aspects, including learnable affine transformations, low-rank compression, and normalization, are well thought out and enhance the applicability and stability of SSMA in real-world scenarios.

Weaknesses

- Limited Comparisons: The paper only compares SSMA with one other aggregation method in the appendix. Including comparisons with more diverse and state-of-the-art aggregation methods would strengthen the evaluation and provide a more comprehensive understanding of SSMA's advantages and limitations. Recent advancements such as Generalized f-Mean Aggregation, Principal Neighbourhood Aggregation (PNA), Hybrid Aggregation for Heterogeneous Graph Neural Networks (HAGNN), Robust Graph Neural Networks via Unbiased Aggregation, and GNN-VPA: A Variance-Preserving Aggregation Strategy for Graph Neural Networks could provide valuable benchmarks for comparison​ (ar5iv)​​ (ar5iv)​​ (ar5iv)​​ (ar5iv)​​ (ar5iv)​​ (ar5iv)​​. - Computational Complexity: The paper would benefit from discussing SSMA's computational complexity compared to other aggregation method s. Currently, the paper only addresses SSMA's time complexity. It would be better if it included a comparison with the time complexities of similar aggregation methods from related work. This would provide a clearer context for evaluating SSMA's efficiency and practicality.

Questions

- The authors conducted extensive tests combining different MPGNN architectures with SSMA in the experiments. However, why is there only a comparison of SSMA with one other aggregator in the appendix? Could the authors provide more experimental results comparing SSMA with other aggregators to demonstrate its advantages better? - Could the authors provide direct training time comparisons between MPGNNs using SSMA and other aggregators? This would help to illustrate the time cost associated with adding the SSMA module more clearly. - In Figure 1, SSMA is demonstrated, but the nodes u, v, and w appear more like a set rather than having a sequential relationship. Does "sequential" refer to the pipeline being sequential, or is there a sequential relationship among these graph nodes? - Are there specific types of graphs or tasks where SSMA may not perform as well as other methods?

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

3

Limitations

N/A

Summary

This paper introduces Sequential Signal Mixing Aggregation (SSMA), a novel aggregation method for Message Passing Graph Neural Networks (MPGNNs). SSMA addresses the shortcomings of traditional sum-based aggregators, which often struggle to effectively "mix" features from distinct neighbors. By treating neighbor features as 2D discrete signals and sequentially convolving them, SSMA significantly enhances feature mixing from distinct neighbors. Experimental results show that integrating SSMA into existing MPGNN architectures markedly improves performance across various benchmarks, achieving state-of-the-art results.

Strengths

1. The explanation of the limitations of sum-based aggregators is compelling and insightful, offering a fresh perspective on the problem and effectively motivating the proposed method. 2. SSMA introduces an innovative approach to feature aggregation in MPGNNs, which can be efficiently implemented and scaled to accommodate larger graphs. 3. The experimental results on both node and graph-level tasks clearly demonstrate the effectiveness of the proposed method.

Weaknesses

1. Other aggregation methods, such as LSTM[1] and DIFFPOOL[2], also effectively mix neighbors’ features. These methods should be included in the related work and experimental comparisons. 2. The inductive setting mentioned in the experimental setup section is missing from the experiments section. [1] Inductive Representation Learning on Large Graphs. NeurIPS 2017. [2] Hierarchical Graph Representation Learning with Differentiable Pooling. NeurIPS 2018

Questions

Please refer to the weaknesses.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The limitations have been demonstrated in the paper.

Official Comment by Reviewer QvY7

Thank you for the detailed response. I raise my score to 6.