Diffusion-Assisted Distillation for Self-Supervised Graph Representation Learning with MLPs

For large-scale applications, there is growing interest in replacing graph neural networks (GNNs) with lightweight multilayer perceptrons (MLPs) via knowledge distillation. However, distilling GNNs for self-supervised graph representation learning (SSGRL) into MLPs is more challenging. This is because the performance of self-supervised learning is more related to the model’s inductive bias than supervised learning. This motivates us to design a new distillation method to bridge a huge capacity gap between GNNs and MLPs in SSGRL. In this article, we propose diffusion-assisted distillation for self-supervised graph representation learning with MLPs (DAD-SGMs). The proposed method employs a denoising diffusion model (DDM) as a teacher assistant to better distill the knowledge from the teacher GNN into the student MLP. This approach enhances the generalizability and robustness of MLPs in SSGRL. Extensive experiments demonstrate that DAD-SGM effectively distills the knowledge of self-supervised GNNs compared to state-of-the-art GNN-to-MLP distillation methods.