Gradient Rewiring for Editable Graph Neural Network Training

Summary

Model editing involved fine-tuning a pre-trained model specifically on training examples where it cannot predict the correct output in order to correct these errors. While model training has been explored for CV and NLP models, GNNs pose a unique challenge due to the unordered data type and node-level classification problem. This paper attempts to rectify this error by proposing a Gradient Rewiring+ (GRE+) method for Editable graph neural network training while preserving the original train/test set performance. GRE+ is evaluated on several small and large classification datasets.

Strengths

- The proposed method aims to rectify errors that still persist after training, without compromising the original performance. To quantify this, the authors report three metrics: The accuracy after model editing has been performed, the DropDown metric measuring how this impacts the original performance and the success rate of the method at actually rectifying the error. The three of these provide a clear, multi-faceted view of the effect of the proposed method against the baselines. - In most situations, the proposed GRE+ proves to be the best method. - The introduction and motivation of the paper is generally well-written and easy to follow.

Weaknesses

- Biggest difference is the presentation and decision to treat GRE and GRE+ as different methods to compare to baseline approaches. This is because the experimental results in Tables 1 and 2 show that GRE on its own is not really noteworthy as it fails to consistently rank as the 2nd best behind GRE+ and is outperformed by GD and ENN on numerous occassions. - This hampers the writing in the method which focuses on what GRE does first, before expanding to talk about GRE+ with more specifics. In contrast to this writing decision, it would be better if GRE and GRE+ were not treated separately, GRE meant GRE+ and instead of having 2 rows for your method in Tabs 1/2, you just had GRE(+), then later did an ablation study (where one option is the GRE config as mentioned in the submitted paper) to show the efficacy/necessity of the whole method. - Line 243 This is the first time the word "Transformers" is mentioned in the paper and its jarringly brought up out of the blue. The introduction broadly mentioned that model editing exists for CNNs and NLP tasks but not specifically the Transformer architecture. This prose should be deeply revised before the paper is publication ready. - There is no analysis/study of the cost/complexity of the model editing methods which is a major weakness. Looking at Fig 3 while it is true that GRE+ improves little/none over GRE, in Tab 1/2 the story is much different. There does not seem to be a situation where GRE is better so I must question why it is presented as a distinct approach itself compared to GRE+. Is it because it is computationally less expensive? - Some statements are not properly substantiated with citations: - L105 "It is well-known that model editing incurs training performance degredation" - L167 "Since shallow GNNs model performs well in practice" seems like a handwave. Also, spelling/grammar check. - Other nitpicks: - Tab 1 caption mentions "OOM" for some methods but that is only used in entries for Tab - Fig 2 should have larger fonts it is very hard to read. - L242, should read ", respectively." when enumerating like that.

Questions

- Fig 3 where are results on ENN?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

A limitations section is provided in the supplementary material.

Summary

The authors propose a model editing technique motivated by an observed inconsistency between the gradient of target and training nodes' cross-entropy losses. Their proposed method, Gradient Rewiring for Editable Graph Neural Networks (GRE), stores the anchor of the gradients of the training nodes and uses the anchors during editing to preserve performance. Finally, the authors empirically evaluate their proposed method on a collection of real-world datasets.

Strengths

- I believe the problem to be an important practical problem and that the paper is well-motivated. - The main observation of the paper, i.e., that the gradients of the target and the training nodes are inconsistent, is interesting and novel in the context of Graph Neural Networks. - The proposed method seems to perform well consistently. The authors have performed a comprehensive empirical evaluation, and I believe the results are solid. - The authors are overcoming the quadratic problem in the number of model parameters by solving a dual problem with significantly fewer variables.

Weaknesses

- The method requires storing the gradients of the training dataset. This would raise the memory consumption. It would be great if the authors could include a table with the memory requirements for different models. - There are a lot of writing/grammar issues. In the following, I will give a (non-exhaustive) list of examples of missing articles and text where the writing could generally be improved. This would generally not be a big deal, but in this case, the writing makes the text very hard to follow and comprehend. I would encourage the authors to do a few revision rounds before updating the manuscript. In my personal experience, free tools such as Grammarly also help with identifying weird formulations or grammar mistakes: - 16-17: the gradient of *the* loss - Line 21: "interpreting the features and topology of graph data" - *interpreting* is a somewhat unusual way to describe learning representations on graphs; please consider rephrasing. - Line 42: "(...) intricate editing for GNNs through the lens of landscape." - this phrasing is very confusing; what landscape? I suggest the authors rephrase to something more akin to "through the lens of the loss landscape of the Kullback-Lieber divergence between the initial node features and the final node embeddings" since this seems to be the technique that [[1]] is proposing. - Line 46: "(...) perspective, and is compatible with existing work" - please consider rephrasing to something similar to "(...) perspective, which is compatible with existing work". - Line 49: "(...) can lead to a deterioration in the performance *of* the training nodes" -> "(...) can lead to a deterioration in the performance *on* the training nodes" - the performance is of the model on the training nodes - Line 55: similar, "performance *of* the training nodes" -> "performance *on* the training nodes" - The caption of Figure 1 is somewhat confusing; please consider rephrasing. - Line 93-94: Somewhat confusing: "motivation to rewire gradients *of* model editing" or "motivation to rewire gradients *for* model editing"? - Line 95: "and advanced version (GRE+)" -> "and *an* advanced version (GRE+)" - Lines 98-99: "we pre-train (...) on (...) datasets" -> "we pre-train (...) on *the* (...) datasets" - Lines 101-102: please consider rephrasing to something more akin to "we fine-tune the well-trained model using the cross-entropy loss of the target sample via gradient descent" - Lines 106-108: "we investigate performance degradation from model gradient perspective" -> "we investigate *the* performance degradation from *a* model gradient perspective"; "we further define training loss" -> "we further define *the* training loss"; "where ... is prediction model ... CE is cross-entropy loss" -> "where ... is *a* prediction model ... CE is *the* cross-entropy loss". - Footnote 3 is very confusing; please consider rephrasing the entire footnote. - Lines 131-147 are also somewhat confusing and hard to comprehend due to language. Please consider rephrasing all of the paragraphs. - Line 148: "where gradient for model prediction is defines as" -> "where *the* gradient for *the* model prediction is *defined* as". Also, the inline *\frac* is not aesthetically pleasing, but that is a very minor complaint. - Line 161: Please consider either removing "*that*": "it is easy to obtain *that* the optimal dual variable v=..." -> "it is easy to obtain the optimal dual variable $v*$=..." or rephrasing to something more akin to "it is easy to see that the optimal dual variable is $v*$=..." - Please also consider rephrasing the next lines, from 162 to 164. - Missing commas on line 172: "the training loss for the whole training dataset, after model editing, is on par with (...)" - Please consider re-phrasing lines 199-200 to something more akin to "We randomly select a node from the validation set on which the well-trained model makes a wrong prediction." - Both Tables 1 and 2 have results with different decimal precision for the standard deviation. For instance, GraphSAGE/GRE/CORA/DD has a single decimal ($3.36 \pm 0.2$) while GraphSAGE/GRE+/CORA/DD contains two decimals ($0.41\pm 0.07$). Please consider using the same amount of decimals for all of the results. It makes the table more aesthetically pleasing and easier to read. - "OOM" is mentioned in the caption of Table 1 as being an out-of-memory error, but there are no OOMs in Table 1. However, Table 2 contains OOMs but the caption does not explain what OOM means. - Please consider increasing the font of the text for both Figures 1 and 2. - Figure 4, first row, first column: the figure covers half the "N" of "GCN". The plots are also not vertically aligned on the two columns. Overall, I believe that the proposed method is solid due to the very good empirical results. However, the writing is lacking to a degree that makes the text very hard to follow and comprehend. I would encourage the authors to revise the text significantly, with multiple rounds of proofreading. Due to concerns about the clarity and writing, I currently recommend rejection. However, I will revise my score if the authors significantly improve the writing during the rebuttal. I think that the overall work could potentially be impactful and useful from a practitioner's perspective. [1]: https://arxiv.org/pdf/2305.15529

Questions

Please see the weaknesses above.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors are discussing limitations in the appendix.

Summary

The work introduces a novel method called Gradient Rewiring (GRE) to address the challenge of editable training in graph neural networks. Traditional fine-tuning approaches often struggle with maintaining performance for both target and training nodes. GRE aims to overcome this limitation by rewiring gradients in a way that preserves locality, leading to improved performance for both types of nodes. The method is designed to enhance the training process in graph neural networks by effectively updating node representations while maintaining the network's overall structure.

Strengths

1. The method of the paper is clear, intuitive, and easy to implement. 2. The writing is clear, making it easy to understand and read.

Weaknesses

1. Although the experiments were conducted on graph datasets, the proposed method is not specifically designed for graphs. This general approach can be tested on various tasks. 2. There are noticeable formatting errors and typos in the paper.

Questions

1. It would be beneficial to provide a more detailed explanation of the experimental process and results in Figure 1. In the first row of Figure 1, the images indicate that the gradients for testing and training become very similar, suggesting that the test loss and train loss should decrease simultaneously. However, in the second and third rows, the test loss decreases while the training loss increases. Please further explain this observation. 2. The training + rewriting pipeline is similar to curriculum learning, where the network is trained in two phases: first on simple samples, then on more difficult ones. I recommend the authors conceptually compare their method with curriculum learning. Additionally, I suggest experimenting with batch data rewriting, which might be more practical compared to rewriting individual samples or sample sequences.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The limitations has been discussed in the Appendix.

Summary

This paper tackles the challenge of editing GNNs. The authors highlight a key issue in GNN editing: the gradient inconsistency between target and training nodes, which can degrade performance when the model is fine-tuned using only the target node’s loss. To address this, they introduce the Gradient Rewiring (GRE) method, which preserves the performance of training nodes by storing an anchor gradient and rewiring the target node’s loss gradient accordingly. The efficacy of GRE is validated through experiments across various GNN architectures and graph datasets, demonstrating its potential to enhance model adaptability without compromising existing accuracies.

Strengths

- This paper is well-motivated and well-organized, addressing a research question that remains to be further investigated. The provided visualization also aids in understanding the motivations behind the study. - The authors offer a detailed derivation process, making the methodology easy to follow. - The method was tested on large-scale datasets, demonstrating the model's scalability.

Weaknesses

- The significance of editing graph neural networks remains unclear. Editing seems to lead to significant performance degradation, especially on large-scale datasets. The authors should discuss why such significant efforts are warranted to modify predictions on individual samples. - I recognize that considering graph neural network editing from the perspective of gradient rewriting is novel, but the authors have only discussed model editing in related work. The lack of gradient modification literature makes it hard to position this paper. - Some experimental details are missing, such as dataset splitting. These details are crucial for evaluating the proposed method. - The time complexity is not provided. - Some minor issues (e.g., a superscript linking to nothing in the caption of Figure 1).

Questions

- In what scenarios would people be willing to accept a substantial overall performance drop in order to modify predictions for individual samples? - Are there any methods developed for editing neural networks on non-graph data based on gradient modification? Or are there any works on gradient modification that are relevant to the method in this paper?

Rating

6

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

As the authors discussed in Appendix B.