Contextualized Messages Boost Graph Representations

Rebuttal by Authors

We thank the reviewer for the constructive feedback! 1. We would like to clarify that both SIR-GCN models in Tables 4 and 5 largely employ the same architectural design with only a single GNN layer, as noted in Line 485 for Table 4 and Line 512 for Table 5. Appendix B2 presents a more detailed description of the architecture where the key difference between the two models lies in Table 5 including a graph readout function (since ogbg-molhiv is a graph property prediction task) whereas Table 4 does not (since ogbn-arxiv is a node property prediction task). The large discrepancy in parameter counts is primarily due to the input and output dimensions of each dataset. For ogbn-arxiv in Table 4, the input node features are 768-dimensional with a 40-dimensional output, while for ogbg-molhiv in Table 5, the input node features are only 174-dimensional with a scalar output. This difference in the input node feature dimension, stated in Lines 800 and 809, naturally affects the parameter count due to the Linear layer multiplying the input dimension by the hidden state dimension, even if they have the same architectural design. Specifically, a single Linear layer in ogbn-arxiv requires ~200,000 (768 x 256) parameters while a single Linear layer in ogbg-molhiv only requires ~53,000 (174 x 300) parameters. We also emphasize that, given the distinct nature and domains of these datasets (citation network vs molecular network), a comparison between parameter counts is not meaningful for fairness. Instead, we highlight that SIR-GCN can outperform larger models using fewer parameters in both datasets, particularly with LGGNN in Table 4 and GSN in Table 5. This highlights the utility and expressivity of SIR-GCN. 2. We have performed additional runtime analysis based on the reviewer's suggestion. The models below follow a similar architecture as reported in Appendix B2 for SIR-GCN using the same hyperparameters. The results still underscore how SIR-GCN effectively balances computational efficiency and model expressivity, particularly when compared against PNA which is also designed for uncountable node features. Notably, PNA resulted in an out-of-memory (OOM) error for ogbn-arxiv as it requires $O(|\mathcal{E}| \times d_\text{in})$ memory, where $|\mathcal{E}|$ is significantly larger for this graph and $d_\text{in} = 768$ as noted above in 1. On the other hand, PNA requires significantly longer (more than 50%) runtime compared to the other models in ogbg-molhiv, consistent with the results in Tables 6 and 7. Meanwhile, SIR-GCN requires approximately the same runtime as the other models across both datasets. | Model | ogbn-arxiv | ogbg-molhiv | | :-------: | :----------------: | :----------------: | | GCN | 0.0729s ± 0.0008s | 0.8029s ± 0.1171s | | GraphSAGE | 0.1161s ± 0.0039s | 0.8062s ± 0.1166s | | GATv2 | 0.1006s ± 0.0022s | 0.6835s ± 0.1620s | | GIN | 0.0830s ± 0.0015s | 0.7574s ± 0.1620s | | PNA | OOM | 1.2786s ± 0.0664s | | SIR-GCN | 0.1265s ± 0.0039s | 0.7345s ± 0.1300s |

Rebuttal by Authors (cont.)

3. We further highlight that the runtime analyses on synthetic datasets in Tables 6 and 7 provide a more controlled assessment of computational complexity, illustrating changes in model runtime as model complexity and problem size increase (*i.e.*, as $n$ and $c$ increase). This comparison allows for more meaningful insights into the scalability of each model in isolation. To further strengthen our claim, we have included an additional comparison of the asymptotic complexities of each model in Appendix C and in Line 825 to complement the results in Tables 6 and 7, specifically with regards to PNA which is also designed for uncountable node features but requires significantly longer runtime. The table further highlights how SIR-GCN has a computational complexity comparable to GCN, GraphSAGE, GAT, GATv2, and GIN while outperforming these models across all benchmarks. Notably, SIR-GCN also demonstrates a lower complexity than PNA, yet delivers superior performance across all datasets. These additional analyses underscore how SIR-GCN effectively balances computational efficiency and model expressivity, further demonstrating its novelty and practical utility. | Model | Model Complexity | | ----------- | :--------------------------------------------------------------------------------------------------------------------------: | | GCN | $O(V \times d_\text{out} \times d_\text{in} + E \times d_\text{out})$ | | GraphSAGE | $O(V \times d_\text{out} \times d_\text{in} + E \times d_\text{out})$ | | GAT / GATv2 | $O(V \times d_\text{out} \times d_\text{in} + E \times d_\text{out})$ | | GIN | $O(E \times d_\text{in} + V \times \texttt{MLP})$ | | PNA | $O(E \times d_\text{in}^2 + E \times d_\text{in} \times k + V \times d_\text{out} \times d_\text{in} \times k)$ | | SIR-GCN | $O(V \times d_\text{hidden} \times d_\text{in} + E \times d_\text{hidden} + V \times d_\text{out} \times d_\text{hidden})$ |

Rebuttal by Authors

We thank the reviewer for the positive feedback! 1. The key novelty of SIR-GCN lies in its *anisotropic* and *dynamic* message function, allowing it to better handle uncountable node features. Unlike other GNNs which only employ *isotropic* or monotonic/linear transformations to neighborhood features, SIR-GCN is among the first MPNN instance to introduce *dynamic* transformations to neighborhood features that also account for the center node features. This design specifically enables SIR-GCN to learn the complex, nuanced relationships between pairs of neighboring nodes before aggregating them into a single feature. This is also key to how SIR-GCN handles uncountable node features, as explained in Line 527 of the conclusion. An illustration of this feature is further elaborated in 3 below. 2. Similar to the results presented in Corso et al. (2020), Table 3 also directly presents the available results from Dwivedi et al. (2023), Corso et al. (2020), and Tailor et al. (2021). The missing values indicate that these works have not considered the particular dataset for their model. This is added as a footnote in Table 3. We also emphasize that in addition to the datasets considered in Corso et al. (2020) for PNA, our paper further includes the WikiCS, PATTERN, and CLUSTER datasets from the Benchmarking GNNs (Dwivedi et al., 2023) as an additional evaluation. 3. We thank the reviewer for pointing us to the work of Mao et al. (2024). While the study does analyze real-world graphs with varying degrees of heterophily, the graphs considered are specific to node property prediction tasks. In contrast, GraphHeterophily is designed specifically for graph property prediction which makes the graphs considered in Mao et al. (2024) not directly compatible. Nonetheless, we emphasize that the directed graphs in GraphHeterophily are uniformly generated using DGL's `rand_graph` function, with class labels also uniformly assigned using PyTorch's `randint` function. These measures, highlighted in Line 775, ensure that the graphs are sufficiently diverse in terms of graph structure and heterophily degrees, supporting the robustness of the results. Furthermore, we would also like to highlight that while other GNNs fail in the simple task of GraphHeterophily, SIR-GCN excels in the task. Its remarkable performance is nevertheless expected, as explained in Line 430, since if node features are one-hot encodings of class labels, SIR-GCN with sum aggregation, $\boldsymbol{W_Q} = \boldsymbol{I}$, $\boldsymbol{W_K} = - \boldsymbol{I}$, $\sigma = \text{ReLU}$, and $\boldsymbol{W_R} = \boldsymbol{1}^\top$ can consistently produce accurate outputs, regardless of graph structure or heterophily degrees. This success may be attributed to the ReLU activation applied along edges, enabling SIR-GCN to "reason" along edges, based on the labels of pairs of neighboring nodes, and produce a *dynamic* message accounting for this learned relationship. This illustration further highlights the significance of *anisotropic* and *dynamic* message functions, underscoring the novelty of SIR-GCN as the first MPNN instance to satisfy this requirement. 4. The attention mechanism of the original GAT is *anisotropic* but not *dynamic* as highlighted by Brody et al. (2021). GATv2 addresses this limitation by making the attention mechanism *dynamic*. Nevertheless, both GAT and GATv2 still only linearly transform the messages (neighborhood features) as stated in Line 305. Thus, the features of the center node only affect the aggregated neighborhood features by determining the degree of contribution through the scalar attentional weight. In contrast, SIR-GCN is the first MPNN instance to leverage this idea and make the actual messages *anisotropic* and *dynamic*, rigorously grounded in theory. The experimental results, particularly in Table 3 where SIR-GCN significantly outperforms GAT across all datasets with the same parameter budget, underscore the significance of this idea and the novelty of SIR-GCN. References: Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2105.14491, 2021. Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lio, and Petar Velickovic. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 33:13260-13271, 2020. Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research, 24 (43):1-48, 2023. Haitao Mao, Zhikai Chen, Wei Jin, Haoyu Han, Yao Ma, Tong Zhao, Neil Shah, Jiliang Tang. Demystifying structural disparity in graph neural networks: Can one size fit all?. Advances in neural information processing systems, 2024. Shyam A Tailor, Felix L Opolka, Pietro Lio, and Nicholas D Lane. Do we need anisotropic graph neural networks? arXiv preprint arXiv:2104.01481, 2021.

Rebuttal by Authors

We thank the reviewer for the constructive insights! 1. We would first like to clarify that our work is not an extension of Corso et al. (2020) but rather a novel approach to handling uncountable node features within the MPNN framework. Unlike PNA whose key features are multiple aggregators and scalers, SIR-GCN only employs a single aggregator. Consequently, it is also computationally efficient, requiring only an activation function (linear complexity) along edges, unlike PNA, which requires a full linear layer (quadratic complexity) as shown in Table 8 of Appendix C. Despite its lower computational requirement, SIR-GCN still achieves superior performance, consistently outperforming PNA across all benchmarks spanning both countable and uncountable node features, highlighting its efficiency, expressivity, and practical utility. We further note that unlike PNA whose *anisotropic* nature is simply a direct consequence of applying its main theoretical result of multiple aggregators and scalers within the MPNN framework (Section 2.3 of Corso et al., 2020), our work presents a rigorous theoretical foundation for the *anisotropic* nature of SIR-GCN in Line 216, highlighting its novel contribution to GNN research. Moreover, while Brody et al. (2021) explored *anisotropic* and *dynamic* functions in the context of GAT attention mechanisms, our work is the first to apply these principles specifically to message functions within the broader MPNN framework. This work contributes to GNN literature by providing a rigorous theoretical foundation (Lines 192 and 216) and extensive empirical results on how this key innovation enables SIR-GCN to capture complex, nuanced relationships between pairs of neighboring nodes prior to aggregation, allowing it to better handle uncountable node features. The significance of this contribution is evident in the experimental results, particularly in Table 3 where SIR-GCN consistently outperforms all other GNNs (including GAT) with the same parameter budget. This is also evident in Table 2 where SIR-GCN successfully accomplished the task when all other GNNs failed, as elaborated in 3 below. Overall, our work offers a novel perspective into the representational capability of GNNs, distinct from Corso et al. (2020) and Brody et al. (2021), especially in the problem of uncountable node features, by demonstrating that using only a single aggregator can already substantially improve the representational capability of GNNs. The SIR-GCN, through its use of *anisotropic* and *dynamic* message functions within the MPNN framework, introduces a fundamentally novel approach to this problem, distinguishing it from existing models like PNA and GATv2 whose specific utility and novelty differs from our work. Notably, the unique design of SIR-GCN directly addresses several limitations of existing GNNs, as demonstrated by illustrative examples on synthetic datasets and model performance on benchmark datasets. These findings establish SIR-GCN as a significant and novel contribution to advancing GNN research. 2. The connection between the motivation and the proposed SIR-GCN is firmly established through the theoretical foundation provided by *pseudometrics* and *soft-injective* functions. Corollary 1 guarantees the existence of a *soft-injective* hash function $G$ and *soft-injective* feature map $g$ given a *pseudometric* $d$ on $\mathcal{H}$ and *pseudometric* $D$ on bounded equinumerous *multisets* of $\mathcal{H}$. From this result, two necessary properties of the *soft-injective* message function $g$ emerge, as elaborated in Line 189 onward, which directly inform the design of SIR-GCN. As clarified in Line 192, for arbitrary *pseudometrics* $d$, the corresponding *soft-injective* message function $g$ must be *dynamic* or non-linear. This insight motivates the use of an MLP to model the *dynamic* nature of $g$, similar to Brody et al. (2021). From Line 216, we further show that the *soft-injective* message function $g$ must also adapt to each node independently. This insight then motivates integrating the features of the center node into $g$, consequently making it *anisotropic*, to avoid the impracticality of designing distinct message functions for each node in large graphs. Combining these two properties, SIR-GCN is the first MPNN instance to utilize a *soft-injective* message function that is both *anisotropic* and *dynamic*. This connection between the theoretical motivation and the architectural design ensures that SIR-GCN is not just empirically performant but also rigorously grounded in theory, demonstrating the significance and novelty of our approach. References: Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2105.14491, 2021. Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lio, and Petar Velickovic. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 33:13260-13271, 2020.

Rebuttal by Authors (cont.)

3. The GraphHeterophily dataset is designed to test the ability of models to reason about heterophilous relationships in directed graphs. The graphs are uniformly generated using DGL's `rand_graph` function with each node uniformly assigned one of $c$ class labels using PyTorch's `randint` function. This approach ensures diversity in graph structures and degrees of heterophily, making the dataset robust for evaluation. Detailed descriptions are provided in Appendix B1. The task is then to count the total number of directed edges in each graph connecting nodes with different class labels, as clarified in Line 417. A sample graph is illustrated in Fig. 5, where four nodes (labeled A or B) are connected by six directed edges. Models must then correctly identify/count the four edges (highlighted in blue) that connect nodes with distinct labels. The results in Table 2 show that SIR-GCN consistently achieves near-zero MSE loss for this simple illustrative task while other models (including PNA and GATv2) obtained large losses. This performance is nevertheless expected, as explained in Line 431, since if node features are one-hot encodings of class labels, SIR-GCN with sum aggregation, $\boldsymbol{W_Q} = \boldsymbol{I}$, $\boldsymbol{W_K} = - \boldsymbol{I}$, $\sigma = \text{ReLU}$, and $\boldsymbol{W_R} = \boldsymbol{1}^\top$ can consistently produce accurate outputs, regardless of graph structure or heterophily degrees. The key to this success lies in the *anisotropic* and *dynamic* message function enabled by the ReLU activation along edges, which allows SIR-GCN to "reason" based on pairs of neighboring node labels and generate nuanced, context-aware messages. It is worth noting that while GATv2 can also somewhat "reason" along edges due to its *dynamic* attention mechanism, its reliance on attentional (softmax) aggregation, which fails to make sharp decisions (Velickovic, 2024) and preserve graph structure as noted in Line 433, hinders its performance. This distinction further highlights the flexibility of SIR-GCN in handling such challenges. Overall, this task underscores the importance of *anisotropic* and *dynamic* message functions (in contrast to attention mechanism), demonstrating SIR-GCN as the first MPNN instance to meet these requirements, further solidifying its contribution and novelty with respect to existing GNNs. 4. We thank the reviewer for pointing out the lack of definition for $\boldsymbol{A}$. In response, we have changed $\boldsymbol{W_R} = \boldsymbol{a_\text{GAT}^\top}$ for clarity. To clarify, Eq. 16 simply demonstrates how the unnormalized attention mechanism in Eq. 15 for GATv2 may be interpreted as the contextualized message in the SIR-GCN model, as mentioned in Line 313. We do not intend to suggest that Eq. 16 is equivalent to GAT. Instead, the illustration shows how the concept of *anisotropic* and *dynamic* functions in the attention mechanism of GATv2 are adapted to message functions in SIR-GCN. This aligns with the explanation above in 1. Furthermore, we also emphasize that only the attention weights in GATv2 are *anisotropic* and *dynamic*, its message functions are still only linearly transformed, which can limit its expressivity, as discussed in Line 305. In contrast, SIR-GCN introduces a contextualized, non-linear transformation to the message functions, improving its ability to capture complex relationships. The significance of this contribution is evident in the experimental results, particularly in Table 3 where SIR-GCN significantly outperforms GAT across all datasets with the same parameter budget. Nevertheless, in Line 314, we clarify how GATv2, up to a normalizing constant, can be obtained from SIR-GCN by selecting the appropriate parameters: $\boldsymbol{W_Q} = \begin{bmatrix} \boldsymbol{W_{Q,\text{GAT}}} \\\\ \boldsymbol{0} \end{bmatrix}$, $\boldsymbol{W_K} = \begin{bmatrix} \boldsymbol{W_{K,\text{GAT}}} \\\\ \boldsymbol{W_{K,\text{GAT}}} \end{bmatrix}$, $\sigma\left(\begin{bmatrix} \boldsymbol{h_1} \\\\ \boldsymbol{h_2} \end{bmatrix}\right) = \exp\left(\boldsymbol{a_\text{GAT}^\top} ~ \text{LeakyReLU}\left(\boldsymbol{h_1}\right)\right) \cdot \boldsymbol{h_2}$, and $\boldsymbol{W_R} = \boldsymbol{I}$. References: Petar Velickovic, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. softmax is not enough (for sharp out-of-distribution). arXiv preprint arXiv:2410.01104, 2024.

Rebuttal by Authors (cont.)

5. The horizontal and vertical axes of Fig. 2 represent the scalar neighborhood node features $\boldsymbol{h_{v_1}}$ and $\boldsymbol{h_{v_2}}$ of node $u$, where $v_1$ and $v_2$ are its neighbors. Line 193 provides context where these features represent zero-mean scores for anomaly detection. Fig. 2 illustrates how these features are transformed and aggregated using different *soft-injective* message functions. As clarified in Line 192, for arbitrary *pseudometrics* $d$, the corresponding *soft-injective* message function $g$ must be *dynamic* or non-linear. This insight motivates the use of an MLP to model the *dynamic* nature of the *soft-injective* message function, similar to Brody et al. (2021). Figs. 2(c) and 2(d) then highlight how MLPs may be used to model *anisotropic* and *dynamic* (*soft-injective*) message functions within the SIR-GCN model, aligning with the theoretical motivation discussed in the paper. 6. The results from the DictionaryLookup and GraphHeterophily synthetic datasets serve as key illustrative examples of the practical utility and novelty of SIR-GCN. In the DictionaryLookup task, both GATv2 and SIR-GCN achieved perfect accuracy, highlighting the utility of a *dynamic* attentional or relational mechanism in capturing the relationships between *query* and *key* nodes, as explained in Line 413. In contrast, the GraphHeterophily dataset clearly exposes the limitations of existing GNNs (including GATv2), as only SIR-GCN achieved near-zero MSE loss while all other GNNs obtained large errors, as detailed above in 3. This dataset thus underscores the utility and novelty of SIR-GCN with its *anisotropic* and *dynamic* message functions (in contrast to attention mechanism). Furthermore, the results on benchmark datasets further highlight the superior performance of SIR-GCN over existing GNNs in more complex, real-world problems across various domains. In these evaluations, the SIR-GCN models implemented follow standard model design for GNNs, as outlined in Appendix B2. Specifically, we directly replace the GNN component (from GCN, GraphSAGE, GAT, GIN, and PNA) with SIR-GCN. Since we focused on this single replacement, performing an ablation study is neither appropriate nor feasible, similar to how no ablation study was conducted by Brody et al. (2021) when GAT was replaced with GATv2. Nevertheless, Section 4 provides a detailed mathematical discussion of how modifying specific parameters in SIR-GCN may recover conventional GNNs, whose performance is already included in our results. Hence, the reported results and performance improvements are solely attributed to the novel aspects of SIR-GCN. We are open to suggestions for further improving the results section. Finally, we emphasize that SIR-GCN can be easily integrated into more complex frameworks, such as grouped reversible residual connections (Li et al., 2021) and graph stochastic attention (Miao et al., 2022), to further improve performance. Previous works, particularly in Li et al. (2021) and Miao et al. (2022), have demonstrated that any GNN backbone, such as GCN, GraphSAGE, GAT, GIN, and PNA, can be seamlessly and easily incorporated into their frameworks to employ additional/advanced techniques for enhanced performance. While this also holds for SIR-GCN, exploring such integrations with SIR-GCN is beyond the scope of this paper and left for future works as explained in the conclusion, as it primarily focuses on introducing the key contributions and foundations of SIR-GCN. References: Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2105.14491, 2021. Guohao Li, Matthias Muller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In International Conference on Machine Learning, pp. 6437-6449. PMLR, 2021. Siqi Miao, Mia Liu, and Pan Li. Interpretable and generalizable graph learning via stochastic attention mechanism. In International Conference on Machine Learning, pp. 15524-15543. PMLR, 2022.

Additional Clarifications

We thank the reviewer for the follow-up feedback. 1. We would like to clarify that the novelty of our work is **two-fold**. First, we are the first to provide a **novel theoretical framework** based on *pseudometrics* and *soft-injective* functions to enhance the representational capability of GNNs. This theoretical contribution addresses a critical area in GNN research on uncountable node features, providing an **alternative perspective from the multiple aggregators of PNA**. Second, we further provide a detailed discussion bridging this novel theoretical framework into the MPNN framework, resulting in the SIR-GCN, which is **empirically demonstrated to outperform conventional GNNs across several synthetic and benchmark datasets**. 2. While GATv2 explores the *anisotropic* and *dynamic* properties of GAT **attention weights**, its original motivation for using attention mechanisms are nevertheless **heuristic**. The authors **do not provide a discussion on the representational capability of GATv2**. Moreover, we also highlight the **limitations of both GAT and GATv2**, attributed to their **linear transformation** of neighborhood features and use of **softmax** (Velickovic et al., 2024), which limits their expressivity. In contrast, our work fundamentally differs in context. In particular, our focus is on applying *anisotropic* and *dynamic* properties directly to the **message function**. This allows SIR-GCN to **effectively handle uncountable node features**, as highlighted in Line 531. Critically, we are the first to provide a **rigorous theoretical justification for this approach**, specifically to **enhance GNN representational capability**. This is complemented by **extensive empirical results** showing the contribution of this novel idea. 3. We respectfully disagree with the assessment that performance improvements on real-world datasets are incremental. **SIR-GCN consistently outperforms conventional GNNs** (including GCN, GraphSAGE, GAT, and GIN) by a **significant margin**, as shown in Tables 3, 4, and 5. To further put this into perspective, **SIR-GCN also achieves substantial improvements over PNA** in real-world datasets like MNIST, CIFAR10, ZINC, and ogbg-molhiv while employing a **simpler and computationally efficient design**, as highlighted in Appendix C, **further highlighting the utility and novelty of SIR-GCN**. 4. We would like to emphasize that the complex frameworks, such as grouped reversible residual connections (Li et al., 2021) and graph stochastic attention (Miao et al., 2022), are introduced simply to provide **concrete steps for future works** to build upon our theoretical and empirical results. This analysis is beyond the scope of the current work, similar to **foundational works** such as GATv2 and PNA, since our primary objective is to **demonstrate the core contributions of our proposed SIR-GCN** without employing additional tricks or techniques, as explicitly stated in Line 381. This ensures a fair evaluation where performance gains are **solely attributed to the key features of SIR-GCN**, in line with the reviewer's earlier remark of **justifying the effect of SIR-GCN**. 5. Overall, we firmly believe that our work makes a **significant and novel contribution to GNN research** by providing a **comprehensive theoretical perspective for handling uncountable node features**, supported by empirical validation, and a **practical and efficient SIR-GCN that consistently outperforms conventional GNNs**. Together, these contributions provide a ***new, relevant, and impactful* advancement toward understanding GNN representational capabilities**. References: Guohao Li, Matthias Muller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In International Conference on Machine Learning, pp. 6437-6449. PMLR, 2021. Siqi Miao, Mia Liu, and Pan Li. Interpretable and generalizable graph learning via stochastic attention mechanism. In International Conference on Machine Learning, pp. 15524-15543. PMLR, 2022. Petar Velickovic, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. softmax is not enough (for sharp out-of-distribution). arXiv preprint arXiv:2410.01104, 2024.

Alignment with Prominent Foundational Works in GNN

We thank the reviewer for the additional feedback. We would like to clarify how our work aligns with prominent foundational works in GNN. 1. The theoretical discussions by Brody et al. (2021) only focus on the **representational capability of the GAT attention mechanism**. Specifically, they only analyzed the ***difference* in representational capability between *static* (Theorem 1) and *dynamic* (Theorem 2) GAT attention**, as explicitly written in their theorem statements. In contrast, our theoretical discussions focus on the **broader representational capability of GNNs with a specific focus on uncountable node features** which is **not addressed by Brody et al. (2021)**. 2. In **Xu et al. (2018), Corso et al. (2020), Brody et al. (2021)**, and other prominent foundational works in GNN, they only compared the performance of their proposed *foundational models* (GIN, PNA, GATv2) to **other *foundational GNNs*** (GCN, GraphSAGE, GAT). Notably, these ***foundational models* are not designed to explicitly achieve state-of-the-art (SOTA) performance**. Hence, these works do not compare the performance of their model to SOTA models to ensure a **fair evaluation** where performance improvements are solely attributed to the **key features of the message-passing architecture**. Following these *foundational works*, our results also compare the performance of our proposed *foundational model* SIR-GCN only to *foundational GNNs* (GCN, GraphSAGE, GAT, GATv2, GIN, PNA) to highlight the **performance improvement attributed to the key message-passing features of SIR-GCN**. These results also underscore the practical utility of ***replacing conventional foundational GNNs* with SIR-GCN in existing models**. Furthermore, given the ***foundational* nature of SIR-GCN which is (similar to other *foundational models*) not designed to explicitly achieve SOTA**, a direct comparison of performance relative to SOTA would **not be meaningful nor comparable** and would **require *significant* model architecture tuning** since these models employ **several additional techniques that significantly increase their performance**. 3. **Xu et al. (2018), Corso et al. (2020), Brody et al. (2021)**, and other prominent foundational works in GNN only provide a **theoretical analysis of their key contributions based on the MPNN framework**. It is ***subsequent and separate* works** that extended their results to other complex frameworks as this **required additional analysis that is *beyond the scope of the original paper***. Similar to these *foundational works*, our *foundational paper* also presents a **comprehensive theoretical framework based on MPNN**. Extending our results to other frameworks also requires ***significant* additional analysis** that is ***beyond the scope of our paper*** and warrants a **comprehensive discussion in a *separate* and dedicated work**. Nevertheless, our novel theoretical framework based on MPNN already provides a ***new, relevant, and impactful* advancement toward understanding GNN representational capabilities**. 4. Overall, our work, *in its current form*, already provides a **significant and novel contribution to GNN literature** on the problem of uncountable node features. Furthermore, as a ***foundational paper*** introducing the *foundational GNN model* SIR-GCN, it also already **includes the standard and key discussions and comparisons** in prominent foundational works in GNN. References: Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2105.14491, 2021. Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lio, and Petar Velickovic. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 33:13260–13271, 2020. Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826, 2018.