DiGRAF: Diffeomorphic Graph-Adaptive Activation Function

Summary

Inspired by the continuous piecewise-affine based transformation, this paper argued that the activation function should not be an uniform selection for different nodes, and developed a learning activation function method, that can take the graph structure into account to activate node-specific features. The experimental results support their proposed model's effectiveness through comparing with existing activation functions in several datasets.

Strengths

This paper provides a new method to learn the activation function, where instead of activating every nodes (representations) uniformly, this method helps generate a node-specific function. The experimental results prove that this method can significantly increase the model performance.

Weaknesses

Many formula seem to be repeated, such as equations in 3.2 and 4.1. The complexity is a bit high compared with an existing activation function.

Questions

1. How will you ensure the parameter \theta you learn from the training can return you a diffeomorphism activation function? It is not clearly stated in your experimental settings, such as how you define transformation T and space \Omega. 2. The method used a GNN to learn the activation function, is there an activation function in the GNN_act? 3. In the node classification experiments, there are some significant improvement in some tasks, while some of them have a minor improvement. Have you tried to figure out the reasons?

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

There is no potential negative societal impact of this work.

Summary

This paper introduces a new activation function, DIGRAF, specifically tailored for graph data in Graph Neural Networks (GNNs). The approach is based on Continuous Piecewise-Affine Based transformation (CPAB). The authors demonstrate that DIGRAF possesses the desired properties highlighted in existing literature and provide a thorough analysis of these properties. Extensive experiments conducted on diverse datasets across various tasks show that DIGRAF outperforms three different types of baselines in downstream performance.

Strengths

1. This paper devise an activation function based on CPAB, which possess many desirable properties of activation functions (e.g. zero-centered, permutation equivariant, etc.) and has solid theoretical guarantees. It's an ingenious idea and seems that this method has never been adopted before in the activation function of GNN. 2. The variables and equations are well-defined and thoroughly explained. The experiments are extensive and robust, effectively supporting the corresponding claims with comprehensive data. 3. The paper addresses the most critical questions about the effectiveness of the proposed method. Each question is answered with extensive experiments and explicit explanations, providing clear evidence for the method’s efficacy. The figures are well-designed, enhancing understanding and readability. Overall, the paper is smoothly written and easy to follow.

Weaknesses

1. Apart from the adoption of the CPAB approach, the primary contribution appears to be the introduction of GNN_act. However, the discussion on why GNN_act is effective is relatively brief. Including more detailed discussions or theoretical justifications would help in understanding its advantages. 2. How do the properties of DIGRAF influence the convergence? It would be nice if it could be discussed which property the gain comes from. 3. As I mentioned above, more ablation studies should be done to reveal how each property and design contribute to the performance gain. Otherwise, DIGRAF (w/o adap.) is good and simple enough. Maybe we can save the budget of GNN_act for a larger GNN_layer.

Questions

1. The size of GNN_act seems to have a minor influence (non-monotonic) on performance according to Appendix F.2. What if you increase the size of GNN_layer a little bit (add the parameter of GNN_act to GNN_layer) and compare DIGRAF (w/o adap. & larger GNN_layer) with the current DIGRAF? I believe this could provide a better justification for the effectiveness of GNN_act. 2. What are the properties that DIGRAF has but general-purpose activation functions or existing graph activation functions do not have? Can you put up a table to clarify it? 3. How do the properties of DIGRAF influence the convergence and performance? 4. The tessellation size decides the degree of freedom of DIGRAF, right? Why does increasing the tessellation size result in marginal changes (non-monotonic)?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations.

Summary

This paper introduces DIGRAF, Diffeomorphic Graph-Adaptive Activation Function, which is a novel activation function adaptive to graph data. DIGRAF leverages Continuous Piecewise-Affine Based transformations, possesses several necessary properties of a good activation function, such as differentiability, zero-centering and permutation equivariance. The author allows DIGRAF adapt to graph data by using an additional GNN to learn the final activation function formula. Adequate experiments which take use of DIGRAF in GNNs proved the high efficiency of this activation function.

Strengths

1. The activation function DIGRAF is a good choice for GNN model design, more effective than the common activation function such as ReLU. 2. The soundness of this paper is wonderful. The theoretical proof of DIGRAF’s properties is detailed and experiments in this paper is convictive, which contain node classification, graph classification and graph regression in task level.

Weaknesses

There should be an ablation study (can be set in the supplementary part) about the hyper-parameters and training strategy in the process of leaning DIGRAF.

Questions

1. In the experiments, why was the sigmoid function not used as an activation function? 2. Are there any differences in the DIGRAF learning process for GCN and GIN?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

1. As the author mentioned, the current DIGRAF may not be optimal since it was learned only for simple tasks, but it can be improved in future work. 2. The application of DIGRAF has not been discussed or implemented in this research. However, using DIGRAF as a novel activation function in some current models might improve their efficiency.

Summary

The paper “DiGRAF: Diffeomorphic Graph Activation Functions” introduces DiGRAF, a novel graph neural network (GNN) activation function based on diffeomorphisms. DiGRAF adapts to graph structures and tasks by learning transformation parameters, enhancing performance across various GNN scenarios. The method demonstrates its effectiveness through extensive experiments on node classification, graph classification, and regression tasks, showing significant improvements over standard and graph-specific activation functions.

Strengths

1. Power of DiGRAF: The paper effectively demonstrates the superiority of DiGRAF in various GNN scenarios, showcasing its adaptability and performance improvements. 2. Extensive Experiment Analysis: The experiments are comprehensive, covering multiple datasets and comparing them with various baseline activation functions. 3. Learnable Activation Function: DiGRAF uses a flexible, learnable activation function that is differentiable and bounded, providing desirable characteristics for machine learning applications.

Weaknesses

1. Limited Contribution: The paper appears to be more of an application of CPAB. The primary contribution of the analysis of the connection between CPAB and GNNs is not thoroughly explored. 2. Performance and Gradient Optimization: The paper lacks a detailed analysis of performance and gradient optimization aspects. 3. Clarification on GNN_act: It needs to be clearer how GNN_act helps boost performance. Is there a way to analyze the new source of information it brings? 4. Proofs for GNN: Most analyses focus on the boundedness of the activation function without specifics tailored to GNNs, making it unclear why this approach is particularly suited for GNNs. 5. Edge Classification Performance: The report on edge classification performance is missing. 6. Hyper-Parameter Effects: It is not clear what the effect of each hyper-parameter is on the performance of the activation function. 7. Source Code: The source code is not available.

Questions

How are the parameters {b_j} (orthonormal basis of the space of velocity fields V) calculated? 2. What would be the activation function of GNN_act if the method’s purpose was to design a new activation function? 3. Are there results for other GNN structures beyond GCN? For instance, GReLU reports results for different GNN structures. 4. Since DiGRAF is also applicable to non-graph datasets, would it be beneficial to see its results on other machine learning methods? 5. What is the closed-form solution of the integral in Equation (2) for 1-dimensional vectors? 6. Is there any information on the learned \theta parameters in terms of sparsity or graph structure?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Inductive Learning: Since the method is task-dependent, it loses its inductive capabilities and cannot be used for cases it did not see. 2. Hyperparameter Choice: There is a dependency on hyperparameter tuning, which may affect generalizability. 3. Time Complexity: DiGRAF has higher time complexity compared to standard activation functions. 4. Decentralized Tasks: The approach is not ideal for decentralized tasks where there is no central data.