Summary
The authors propose a theoretical analysis of the initialization of GATs and their impact on the performance of such networks, focusing on the performance vs depth aspect.
The authors propose an initialization algorithm, that starts from a random initialization and modifies the initial random weights to adhere to the findings of the theoretical analysis.
The authors then show the impact of their proposed initialization on several node classification datasets. Specifically, they show that by initializing GATs with the proposed algorithm, deep GATs can be trained to achieve better than standard (Xavier) initialized GATs.
Strengths
The paper addresses a real issue with GATs - the degradation of performance when more layers are added.
The theoretical analysis seems correct to me and is supported with experimental results as well as the inspection of actual training artifacts and details (e.g., the gradients of the GAT layers).
Weaknesses
The paper can be slightly better written, in terms of organization. I think that adding more paragraphs/subsections to better separate between the parts of the paper can help to ease the reader.
The paper lacks a few relevant citations that also consider graph neural networks as gradient flow (see [1][2][3]). However, their focus is not on the initialization of GATs, and therefore the paper here is novel on its own.
While the experimental results are compelling and show the benefit of the proposed method, I think that the authors should also include comparisons with other methods.
[1] GRAND: Graph Neural Diffusion
[2] Understanding convolution on graphs via energies
[3] Improving Graph Neural Networks with Learnable Propagation Operators
Questions
1. Can the authors add results with more layers? For instance 64,80 layers? does the method still work with very deep network?
2. Regarding the initialization of graph neural networks, it is discussed in [3][4] that the GNN weights are initialized with identity matrices. Can the authors comment on this point? is it related to the proposed method?
3. Can the proposed initialization scheme be applied to other graph attention layers, such as superGAT[5] ?
[4] PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations
[5] How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.