Summary
TempME is an inductive explainer for temporal graph neural networks (TGNNs) over link prediction tasks. It explains TGNNs using temporal motifs, guaranteeing temporally proximate and spatially adjacent explanations, hence more human interpretable. TempME’s pipeline can be broken down into three parts: sampling, embedding, and explaining. Given a link prediction instance, candidate motifs are sampled starting at either end of the edge. The motifs go through a series of steps to generate rich motif embeddings that encode the event's spatial and temporal roles. An MLP learns a Bernoulli distribution over the embeddings. The distribution is used to mask out the best explanation motifs from the candidates. The MLP is trained using the information bottleneck principle to keep the explanations succinct. The mutual information between the predicted label and the explanation is maximized, while the mutual information between the explanation motifs and the set of candidate motifs is minimized. TempME can also boost a model’s performance. The authors show improved results across datasets by concatenating the aggregation of the motif embeddings around a node of interest prior to a model’s final MLP layer.
Strengths
1. Unlike prior work that measures the impact of singular events, TempME measures the combined effect of events on the black box prediction.
2. Cohesive explanations are more human-interpretable than non-cohesive ones.
3. It has significantly less computational cost compared to previous TGNN explainers.
4. It can generalize to unseen nodes, which is highly desirable.
5. The identified motifs can be utilized during training to boost model performance.
Weaknesses
1. The authors have not experimented with synthetic datasets even though they are available. In graphs, the ground truth explanation is often unknown. Hence, synthetic graphs are vital to ascertain that the explanations comply with the ground truth. Please refer to the following for the synthetic datasets and their case study: *Xia, Wenwen, et al. "Explaining temporal graph models through an explorer-navigator framework." The Eleventh International Conference on Learning Representations. 2022.*
2. The authors have cited the following paper as prior work but have not used it as a baseline. *Wenchong He, Minh N Vu, Zhe Jiang, and My T Thai. An explainer for temporal graph neural networks. In GLOBECOM 2022-2022 IEEE Global Communications Conference, pages 462 6384–6389. IEEE, 2022.* Please either compare or provide a justification for not including as a baseline.
3. The authors have compared with GNNExplainer and PGExplainer for comparison with static graph explainers. However, these are no longer state-of-the-art (SOTA) explainers for static graphs. Table 1 and Fig 3 show that GNNExaplainer and PGExplainer are competitive. Using SOTA static graph explainers might have yielded even better results for static explainers and led to different insights. A comparison with the following will be more fruitful:
* *Tan, Juntao, et al. "Learning and evaluating graph neural network explanations based on counterfactual and factual reasoning." Proceedings of the ACM Web Conference 2022. 2022.*
* *Yaochen Xie, Sumeet Katariya, Xianfeng Tang, Edward Huang, Nikhil Rao, Karthik Subbian and Shuiwang Ji. Task-agnostic graph explanations. NeurIPS, 2022.*
4. Please provide the hardware details of the experimental setup.
5. Please provide TempME’s training time as well. It is good to have fast inference, and at the end of the day, inference is what matters, but a user should also have an idea of how long it takes to train.
6. The reference for GINE is correct; however, it does not explicitly use the term GINE in it, which may make it difficult for future readers to refer to. Please add a citation that uses the term GINE explicitly.
7. Pip throws an error when using the author’s requirements.txt to setup the experimental environment.
8. Some typos and grammar mistakes:
* 158, 622: denotes -> denoted
* 190: in this work (unnecessary)
* 278: In meanwhile -> Meanwhile
Questions
Overall, I like the paper. I am open to increasing the score if the authors address the weaknesses listed above and address the questions below.
* Since we are trying to explain the black box and not the data, what if the black box is prioritizing the distant edges? In that case, will TempME force the explanation to be cohesive?
* The authors state the use of a “generative model”. However, it is not found anywhere. The term “generative” brings VAEs and probabilistic graph models to mind, a model that can generate new graph instances that follow a specific distribution or capture the underlying patterns and characteristics of the observed graph data. There is no graph generation involved. Sampling is different from generating. What exactly do the authors mean by a “generative model”? Is it just the ability to generalize to unseen nodes? In that case, the authors should change the terminology as it is misleading.
* In line 194, are C motifs sampled from both u and v, totalling 2C?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
Yes, the authors have addressed the limitations in the appendix.