Library

DFA-GNN: Forward Learning of Graph Neural Networks by Direct Feedback Alignment

Graph neural networks are recognized for their strong performance across various applications, with the backpropagation algorithm playing a central role in the development of most GNN models. However, despite its effectiveness, BP has limitations that challenge its biological plausibility and affect the efficiency, scalability and parallelism of training neural networks for graph-based tasks. While several non-BP training algorithms, such as the direct feedback alignment, have been successfully applied to fully-connected and convolutional network components for handling Euclidean data, directly adapting these non-BP frameworks to manage non-Euclidean graph data in GNN models presents significant challenges. These challenges primarily arise from the violation of the i.i.d. assumption in graph data and the difficulty in accessing prediction errors for all samples (nodes) within the graph. To overcome these obstacles, in this paper we propose DFA-GNN, a novel forward learning framework tailored for GNNs with a case study of semi-supervised learning. The proposed method breaks the limitations of BP by using a dedicated forward training mechanism. Specifically, DFA-GNN extends the principles of DFA to adapt to graph data and unique architecture of GNNs, which incorporates the information of graph topology into the feedback links to accommodate the non-Euclidean characteristics of graph data. Additionally, for semi-supervised graph learning tasks, we developed a pseudo error generator that spreads residual errors from training data to create a pseudo error for each unlabeled node. These pseudo errors are then utilized to train GNNs using DFA. Extensive experiments on 10 public benchmarks reveal that our learning framework outperforms not only previous non-BP methods but also the standard BP methods, and it exhibits excellent robustness against various types of noise and attacks.

cs.LGcs.AIAdvanced Graph Neural NetworksNeural Networks and ApplicationsMachine Learning and ELMComputer Science
View graphSource

Paper

Similar papers

Forward Learning of Graph Neural NetworksNamyong Park, Xing Wang, Antoine Simoulin et al.2024 · arXiv (Cornell University) · 2 citations In Library
GANN: Graph Alignment Neural Network for Semi-Supervised LearningLinxuan Song, Wenxuan Tu, Sihang Zhou et al.2023 · arXiv (Cornell University) · 13 citations In Library
Graph Neural Networks Go Forward-ForwardDaniele Paliotta, Mathieu Alain, Bálint Máté et al.2023 · arXiv (Cornell University) · 8 citations In Library
AdaGNN: Adaptive-Boosting Graph Neural Networks for Node ClassificationKai-Hsiang Fan, C Li2026 · IEEE Transactions on Computational Social Systems In Library
Local Learning with Boosting-based Backpropagation-Free Graph Neural NetworksLuca Pasa, Paolo Frazzetto, Nicolò Navarin et al.2026 · Machine Learning · 0 citations In Library
Reverse Graph Learning for Graph Neural NetworkLiang Peng, Rongyao Hu, Fei Kong et al.2022 · IEEE Transactions on Neural Networks and Learning Systems · 71 citations In Library
Direct Multi-hop Attention based Graph Neural NetworkGuangtao Wang, Rex Ying, Jing Huang et al.2020 · arXiv (Cornell University) In Library
Neighbor-Anchoring Adversarial Graph Neural NetworksZemin Liu, Yuan Fang, Yong Liu et al.2021 · IEEE Transactions on Knowledge and Data Engineering · 14 citations In Library
Graph Deep Learning: Methods and ApplicationsMuhan Zhang2020 · Open Scholarship Institutional Repository (Washington University in St. Louis) · 2 citations In Library
Adaptive Self-Supervised Graph Representation LearningYunchi Gong2022 · 2022 International Conference on Information Networking (ICOIN) · 0 citations In Library

Peer review

Accept (poster)NeurIPS 2024 · via OpenReview
Reviewer mUbj4/10 · confidence 4/52024-07-12

Summary

The paper introduces DFA-GNN, a new framework designed to train Graph Neural Networks (GNNs) using Direct Feedback Alignment (DFA). Traditional methods like backpropagation (BP), though effective, have several limitations, including inefficiency, scalability issues, and a lack of biological plausibility. DFA-GNN aims to address these problems by utilizing a forward training mechanism specifically tailored for the unique challenges posed by graph data.

Strengths

The paper is well written and easy to follow. Performance: DFA-GNN achieves better performance compared to other non-BP methods and even standard BP methods. Robustness: The framework exhibits resilience to noise and different types of attacks, ensuring its reliability in various scenarios. Efficiency: By enabling parallel gradient computation, DFA-GNN improves the efficiency of the training process.

Weaknesses

The novelty of this paper is somewhat limited as it primarily extends the principles of DFA to GNNs. The core ideas are based on the foundations laid out in the Shalizi et al. (2013) paper. The paper lacks a discussion on the computational complexity of the proposed algorithm. There is a notable variation in the performance improvements observed across different datasets. For instance, DFA-GNN shows a significant accuracy improvement on the Texas and Cornell datasets, while the improvements on other datasets are relatively modest. The paper does not adequately explain the reasons behind this variation in performance. The dataset used for experiments is quite small.

Questions

Why the proposed method doesnot outperform on the computer and chameleon dataset. How does DFA-GNN scale when applied to larger datasets e.g. flicker, Reddit, OBGN ? What is the time complexity of the proposed method?

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

No societal impacts.

Authorsrebuttal2024-08-13

Thank you once again for your valuable feedback on our paper. 1. The time comparison of our method is presented in Table 6 of Appendix A.6 in the original manuscript (for your convenience, we have displayed the table below). Here, we compare the training time per epoch of our method with BP and other state-of-the-art non-BP methods. For the semi-supervised node classification task, all methods are trained for 1000 epochs. During training, we calculate the prediction accuracy on the validation nodes after each epoch and save the model that achieves the highest validation accuracy. This saved model is then used to make predictions on the test nodes. This approach is standard in semi-supervised graph learning. Therefore, the overall training cost is simply one thousand times the numbers shown in Table 6. 2. As noted in our response to W2 and Q3, both BP and the DFA in our method have a complexity of $\mathcal{O}(n^{2})$. Besides, our method introduces an additional complexity of $\mathcal{O}(n^{2})$ due to the pseudo-error generator. As a result, the overall complexity for both our method and BP remains the same $\mathcal{O}(n^{2})$. This explains why our method consistently takes five to ten times longer per epoch compared with BP as shown in Table 6, rather than showing an exponential difference, regardless of dataset size. While BP benefits from decades of research and strong software and hardware support, our method has not yet reached comparable time efficiency of BP. However, our method demonstrates superior time efficiency compared with most non-BP state-of-the-art methods, and the current time cost difference compared with BP is not substantial. Additionally, the parallel update strategy employed by our method for each layer offers considerable potential for parallel computing, which could be further explored to enhance time efficiency. 3. We theoretically derive formulas to integrate the direct feedback alignment mechanism into GNNs, as DFA for fully connected layers is not directly applicable to graph data. Additionally, we provide theoretical proof of the convergence of our proposed method (as shown in Section 4.3 and Appendix A.3). Our work highlights the promising potential of non-BP training in graph deep learning and opens up avenues for tackling some challenges within graphs, which merit further exploration in future research. If our rebuttal has satisfactorily addressed your concerns, we would greatly appreciate it if you could consider reevaluating the score of our paper. Regardless of your decision, we are truly grateful for your guidance and the time you have invested in reviewing our work. Thank you again for your attention and support. Best regards, Authors $\newline$ **Table 6: Average running time per epoch (s). For layer-wise training methods like PEPITA, CaFo, FF, and SF, the total time taken by each layer per epoch is reported.** | Datasets | BP | PEPITA | CaFo+CE | FF+LA | FF+VN | SF | ours | |----------|---------|---------|---------|--------|--------|---------|---------| | Cora | 7.56e-3 | 8.73e-3 | 7.61e-1 | 3.14e-1 | 2.83e-1 | 5.49e-2 | 5.66e-2 | | CiteSeer | 1.06e-2 | 1.11e-2 | 7.68e-1 | 2.59e-1 | 2.61e-1 | 6.88e-2 | 5.68e-2 | | PubMed | 1.07e-2 | 1.07e-2 | 8.24e-1 | 6.94e-1 | 7.61e-1 | 5.34e-1 | 6.76e-2 | | Photo | 8.74e-3 | 1.03e-2 | 7.98e-1 | 2.11 | 1.91 | 4.87e-1 | 5.81e-2 | | Computer | 1.08e-2 | 1.05e-2 | 7.80e-1 | 4.82 | 4.14 | 7.61e-1 | 6.29e-2 | | Texas | 6.13e-3 | 1.07e-2 | 8.05e-1 | 1.47e-1 | 1.56e-1 | 6.88e-2 | 5.60e-2 | | Cornell | 5.42e-3 | 1.06e-2 | 7.46e-1 | 1.51e-1 | 1.24e-1 | 3.59e-2 | 5.53e-2 | | Actor | 9.45e-3 | 1.03e-2 | 7.83e-1 | 6.84e-1 | 6.71e-1 | 2.80e-1 | 5.80e-2 | | Chameleon| 6.24e-3 | 1.13e-2 | 7.97e-1 | 2.28e-1 | 2.09e-1 | 6.88e-2 | 5.61e-2 | | Squirrel | 7.77e-3 | 1.20e-2 | 7.78e-1 | 5.82e-1 | 5.05e-1 | 1.21e-1 | 5.79e-2 |

Reviewer mUbj2024-08-13

Response to Author's Rebuttal

According to this statement although both the method have quadratic complexity, "our method make parallel update for each layer while BP has to update layer by layer". Therefore, the proposed method should theoretically be faster than BP. However, this is not evident from the results in Table 6 of the appendix. Could you please clarify this discrepancy?

Authorsrebuttal2024-08-14

Dear Reviewer, Thank you for your valuable feedback on our paper. We have carefully addressed your comments in our response and submitted it for your review. As the deadline for responses is approaching, we wanted to kindly inquire if there are any additional questions or points of clarification you would like us to address. If any questions are not answered or our response is unclear, we would appreciate the opportunity to communicate further with you. We sincerely appreciate your time and effort in reviewing this manuscript. Best regards, Authors

Authorsrebuttal2024-08-13

We apologize for the confusing expression. The standalone direct feedback alignment (DFA, $\mathcal{O}(n^{2})$) as formulated in Eqs. 5 and 6 of the manuscript can potentially be faster than backpropagation (BP, $\mathcal{O}(n^{2})$) since DFA directly calculates the parameter updates for each layer from the loss, allowing for improved training efficiency through parallelization. However, to generalize DFA for graph data, we introduced the pseudo-error generator as detailed in Sec. 4.2. Consequently, the main time cost of our method arises not only from DFA but also from the pseudo-error generation (Eqs. 7, 8) and node filtering (Eq. 9). According to Eqs. 7 and 8, the time complexity of these two components is $\mathcal{O}(tcn^{2})$ and $\mathcal{O}(cn)$, respectively, where $t$, $c$, and $n$ denote the iteration epochs for error spreading, the number of categories, and the number of nodes, respectively (In our earlier analysis, we approximated the complexity of these components as $\mathcal{O}(n^{2})$ and $\mathcal{O}(n)$ to highlight that our method maintains the same quadratic complexity with respect to $n$, the most significant factor). When these two components are added, our method consistently incurs a time cost that is 5 to 10 times larger than that of BP even though they have the same order of complexity. Although these additional components increase the time cost, they are crucial for adapting DFA to GNNs and handling graph-structured data. The ablation study in Tab. 2 highlights the substantial contribution of these components. Although our method has achieved significant practical time efficiency advantages over existing non-BP methods, we will continue to explore ways to further improve the time efficiency of our training algorithm in future work to bring it closer to that of BP, and we will revise the related sections of the article to reflect these nuances more accurately. Thank you again for your attention and support. Best regards, Authors

Reviewer buRu6/10 · confidence 4/52024-07-13

Summary

The authors propose to apply the Direct Feedback Alignment (DFA) algorithm for backpropagation-free training to graph neural networks. The DFA algorithm is combined with a pseudo-error generation mechanism to provide additional error signals for missing targets in the setting of semi-supervised node classification. The experimental results with a 3-layer GCN model over 10 commonly adopted benchmarks show the proposed model performing generally better than backpropagation and other training algorithms. The ablation analysis confirms that improvements by the proposed DFA-GNN approach can be ascribed to the pseudo-error generation mechanism.

Strengths

**Originality** The application of DFA to GNNs was already introduced in *J. Launay, I. Poli, F. Boniface, and F. Krzakala, “Direct feedback alignment scales to modern deep learning tasks and architectures,” NeurIPS 2020*. However, the addition of pseudo-error generation and the extensive experiments are entirely novel contributions. **Quality** The submission is technically sound and its claims are well supported. **Clarity** The paper is clearly written and well organized. **Significance** The results of this paper can contribute to the research on addressing the training challenges of deep graph neural networks.

Weaknesses

1. The computational cost per epoch of the proposed algorithm is 5 to 10 times larger than backpropagation. 2. No comparison of the overall training costs, including the different number of epochs required for training convergence among the different methods. 3. The ablation indicates that the accuracy improvements are ascribed to the pseudo-error propagation mechanism. As the latter could be applied also to e.g. backpropagation, such experimental evaluation would have provided further interesting insights. 4. In the experiments the model is limited to 3 layers; deeper models would have provided more interesting insights, as depth increases the issues connected with gradient backpropagation. (Long-range graph benchmarks such as those from *V. P. Dwivedi et al., “Long Range Graph Benchmark,” in NeurIPS 2022 Track on Datasets and Benchmarks* could be adopted in this case.) 5. Only node classification tasks are considered, no graph-level or edge-level tasks.

Questions

1. Why is pseudo-error generation spreading, which must be done at teach gradient update iteration, instead of augmenting training labels via label spreading (plus the masking of eq. (9)), which would be only required once? 2. How would the other training methods such as backpropagation perform with pseudo-error generation spreading? 3. How many epochs are required for training convergence by DFA-GNN? How do they compare with the number of epochs required by the other training algorithms?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The limitations of this submission have not been properly discussed. For example, in lines 278-280, it is stated that the "method has a higher training time consumption compared with BP, primarily due to the additional time needed for generating pseudo errors and filtering masks". However, in the conclusions, it is claimed that *efficiency* is one of the advantages of the proposed method.

Reviewer k6XA6/10 · confidence 2/52024-07-16

Summary

Recently, some studies have been exploring new optimization methods that can replace backpropagation, and one of the popular methods is direct feedback alignment (DFA). This paper adapts DFA to graph neural networks (GNNs). It replaces the gradient with a randomly initialized matrix multiplying the prediction error, so that we do not need to use backpropagation to update the parameters. We can directly update the model parameters based on the prediction error. Besides, for semi-supervised graphs where the label is scarce, we only have prediction errors for a few nodes, so the parameter update is not that effective. To overcome this drawback, the paper assumes that nearby nodes have similar prediction errors, so it uses a smoothing technique to generate pseudo prediction errors for unlabeled nodes.

Strengths

(1) The studied problem is very interesting. The non-backpropagation optimization method is a very new field. (2) The method design is good. Though it is an adaptation of direct feedback alignment (DFA) which has been used for Euclidean data, since this area has been rarely studied, I think the adaptation is also a contribution. (3) There are some theories supporting the method. (4) Extensive experiments verify its effectiveness, and it can outperform backpropagation.

Weaknesses

(1) In Formula (4), it ignores the activation function when calculating the parameter update. Since the activation function is ReLU, ignoring it might not cause a big problem. However, when we apply other activation functions, or when we use a complex GNN architecture, I am afraid this method might not work as well. (2) From Table 1, we can see that DFA achieves the best performance, while backpropagation is also very good on some datasets. I encourage the authors to add some discussions about when the proposed method is more useful than backpropagation, on what kinds of datasets, and what kinds of GNN models. Besides, the authors explained why DFA is better than previous non-backpropagation methods in Lines 254-274, but it would be better if the authors could explain why DFA is better than backpropagation. (3) The authors can consider introducing a little more about the limitations of backpropagation and the non-backpropagation optimization methods in the introduction or related work. Since most readers are not familiar with non-backpropagation optimization methods, it deserves more introduction. (4) The authors can improve some notations and make them clearer. For example, in Formula (5), the multiplication of $B^{(2)}$ and $B^{(1)}$ is written as $B^{(1)}$.

Questions

(1) What is $W$ in Formula (7)? (2) In Section 4.3, the authors point out that W will converge to be similar to B. Since B is randomly initialized, if B is at a high point in the loss landscape, then will the optimization fail? Is the randomness of B important to the performance? (3) I might have not fully understood the intuition behind DFA. I am confused about why DFA performs better than backpropagation. I look forward to intuitive explanations. (4) Table 6 shows that all the non-backpropagation methods require more time than backpropagation. If I have correctly understood, DFA can directly calculate the parameter update for every layer from the loss, so it does not need to calculate the parameter update layer by layer as in backpropagation, then why does it need more time than backpropagation?

Rating

6

Confidence

2

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations. I encourage the authors to further explain what kinds of datasets and what kinds of GNN models DFA is most suitable for.

Authorsrebuttal2024-08-12

Thank you for your insightful feedback and for recognizing our efforts. We appreciate your suggestion to further discuss the method's limitations, particularly regarding its suitable scenarios and training efficiency. During the rebuttal period, we conducted additional experiments to address these concerns, focusing on the method's suitability for various activation functions, its performance on large-scale graphs, training time, and convergence behavior. These insights have provided a more comprehensive understanding of the method's capabilities and constraints. In the final version of the paper, we will include a dedicated section discussing these aspects, offering a balanced view of the method's strengths and areas for future improvement. We respect your decision and are grateful for your constructive critique. Your feedback has been instrumental in refining our work, and we thank you once again for your thorough and thoughtful review.

Reviewer k6XA2024-08-12

Reply to the rebuttal

Thank you for your reply! Your responses have addressed most of my concerns. The authors conducted a lot of experiments to verify the proposed method from various perspectives. Nonetheless, it seems that this method has some limitations regarding its suitable scenario and training efficiency that are worth further discussion in the paper. It would be better if the authors could include more discussions about the insights and limitations in the paper. I would like to maintain my score.

Reviewer buRu2024-08-12

Thanks to the authors for their response. I leave my score unchanged.

Reviewer mUbj2024-08-13

Response to Author's Rebuttal

I would like to thank the authors for your response. However, I still have concerns in these points: 1.I could not find a table comparing the time efficiency of your method with state of the art method. 2. I have concerns about the time complexity. The additional step also contribute to an O(n²) complexity? How much it isfaster than standard BP methiod? 3. The novelty of your approach seems limited, as it primarily involves integrating DFA into GNNs.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC