Summary
The paper introduces a universal prompt-based tuning method called Graph Prompt Feature (GPF) and its variation (GPF-plus) for pre-trained Graph Neural Network (GNN) models. GPF is a universal method that can be applied to any pre-trained GNN model under any pre-training strategy. It operates on the input graph's feature space and can achieve an equivalent effect to any form of prompting function.
GPF introduces a learnable vector $p$ of dimension $F$, which is added to the node features, where $F$ corresponds to the dimensionality of the node features. The authors also purpose GPF-plus, which assigns an independent learnable vector $p_i$ to each node $u_i$ in the graph, instead of a single vector $p$. Experimental results show that GPF outperforms fine-tuning, with an average improvement of about 1.4% in full-shot scenarios and 3.2% in few-shot scenarios.
Strengths
1) Theoretical Guarantees: The authors provide rigorous derivations and theoretical analyses to demonstrate the universality and effectiveness of GPF. They prove that GPF can achieve an equivalent effect to any form of prompting function and can outperform fine-tuning in certain cases. This theoretical foundation strengthens the credibility of their proposed method.
2) Experimental Validation - Reproducibility: The paper includes extensive experiments conducted under various pre-training strategies, including both full-shot and few-shot scenarios. The experimental results consistently show that GPF outperforms fine-tuning, achieving an average improvement of about 1.4% in full-shot scenarios and 3.2% in few-shot scenarios. Furthermore, GPF surpasses existing specialized prompt-based tuning methods designed for specific pre-training strategies. Moreover, the availability of the source code enhances the reproducibility of the experiments.
Weaknesses
1) A weakness of the paper is the lack of a clear explanation or motivation regarding why the addition of the learnable vector p to the input features in the GPF model leads to better results compared to linear probing with a trainable layer in the final layer. While the Appendix demonstrates the superior performance of GPF, the paper does not provide a comprehensive analysis or reasoning behind this improvement. The absence of a clear explanation may leave readers questioning the underlying mechanisms and factors that contribute to the observed performance gain. Without a proper understanding of the motivations and justifications for the proposed approach, it becomes challenging to assess the significance and generalizability of the findings.
2) Theorem 2 assumes a simple linear 1-layer GNN (without activations). While the theorem provides theoretical insights into the convergence properties of this specific type of GNN, it may not accurately capture the behavior of more complex GNN architectures commonly used in practice. In real-world applications, GNNs often incorporate activation functions to introduce non-linearity and capture more intricate patterns in graph data. By focusing solely on a linear 1-layer GNN without activations, the theorem may limit the generalizability of its conclusions and overlook important aspects of GNN models commonly used in practical scenarios. Therefore, the applicability of the theorem's findings to more sophisticated GNN architectures with non-linear activations remains uncertain.
3) One weakness of the paper is the exclusive use of Graph Isomorphism Network (GIN) as the backbone GNN for finetuning. While GIN is a widely used GNN architecture, it may not be the most expressive or optimal choice for every graph-related task[1,2]. The paper does not provide a comprehensive exploration or comparison of different backbone GNN architectures in the finetuning process. Moreover, maybe a more powerful GNN would be able to achieve better performance in the fine-tuning stage. Therefore, it would be interesting for the authors to examine if their approach leads also to better results when a more powerful GNN is used.
[1] Frasca, Fabrizio, et al. "Understanding and extending subgraph gnns by rethinking their symmetries." Advances in Neural Information Processing Systems 35 (2022): 31376-31390.
[2] Morris, Christopher, et al. "Weisfeiler and leman go neural: Higher-order graph neural networks." Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.
Questions
1) Explanation of the relationship between GPF and linear probing: The paper compares GPF with linear probing, which involves a trainable layer in the final layer. It would be beneficial if the authors could discuss the relationship between these two approaches and explain why GPF, with its learnable vector added to the input features, achieves superior performance compared to linear probing. Is there any inherent advantage or characteristic of GPF that enables it to outperform linear probing?
2) Generalizability of Theorem 2: The paper presents Theorem 2, which focuses on a simple linear 1-layer GNN without activations. However, many practical GNN architectures incorporate activation functions to introduce non-linearity. It would be helpful if the authors could discuss the generalizability of the theorem's conclusions to more complex GNN models commonly used in real-world applications. Can the findings of Theorem 2 be extended to GNNs with non-linear activations? If not, what are the limitations or implications of the theorem in the context of practical GNN architectures?
3) Have you considered or experimented with using more powerful GNN architectures, such as subgraph GNNs or higher order GNNs, as the backbone GNN for the finetuning stage? If so, how does the performance of GPF and GPF-plus compare when utilizing these alternative GNN architectures?
I will be happy to increase my score for the paper, if the authors adequately address the mentioned weaknesses and provide satisfactory explanations and improvements in response to the questions and suggestions raised.
*** After rebuttal increased score to "Borderline Accept" **
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.