Taming Local Effects in Graph-based Spatiotemporal Forecasting

Summary

This paper proposes a method to leverage local effects in graph-based spatio-temporal forecasting. The authors claim that existing spatio-temporal graph neural networks are global models, i.e. all nodes share the same set of parameters, and thus may fail to capture some node-specific patterns. On the other hand, local models, in which some layers within the models are node-specifically parameterized, have better performances compared to global ones, but at the cost of many additional parameters. The authors find a method, random node embeddings, to strike a balance between local and global methods. The authors also propose regularizations to improve transferability of the node embeddings and the resulting models. Experiments over real-world data are given where the proposed method achieves consistent improvements over a variety of models and datasets.

Strengths

1. The studied problem is interesting. It challenges the assumption that a shared global STGNN is used for all nodes, which is standard in previous works. 2. The proposed technique, i.e. trainable node embeddings, is simple and sound. The regularization terms designed also make sense. 3. The proposed technique with trainable node embeddings are effective over various models (DCRNN, AGCRN, GWNet) and real-world datasets, which shows the generality of the proposed technique. 4. The experimental results showing that node embeddings and regularizations are effective in terms of knowledge transfer are a plus. Intuitively, people may think that models with node specific parameters will not perform well in unseen nodes, but the results show the opposite.

Weaknesses

1. The proposed method with node specific embeddings is effective, but not new. Specifically, STID [40] proposes exactly the same technique in terms of trainable node embeddings. I am slightly concerned about whether the technical contribution meets the standard of NeurIPS with this existing work. 2. The fine-grained categorization about spatio-temporal graph neural networks do not seem necessary, e.g. T&S, TTS and anistropic VS istropic. I fail to see how introducing these concepts help better understand the paper, and thus I would suggest these parts be removed.

Questions

I really like this paper and since it is in general clearly written, I do not have questions at this time.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This paper presents a methodological framework aimed at rationalizing the inclusion of trainable node embeddings in STGNNs for spatiotemporal forecasting applications. The authors examine the interplay between globality and locality in graph-based spatiotemporal forecasting and provide insights and guidelines for specification design. The paper demonstrates how incorporating trainable node embeddings in STGNNs can effectively combine the advantages of shared message-passing layers with node-specific parameters, while efficiently transferring the learned model to new node sets. The proposed framework is supported by empirical evidence and offers a principled approach for accommodating various node embeddings.

Strengths

1. The authors investigate the interplay between globality and locality in graph-based spatiotemporal forecasting, resulting in five major findings. 2. The paper illustrates how including trainable node embeddings in STGNNs can effectively combine the benefits of shared message-passing layers with node-specific parameters and efficiently transfer the learned model to new node sets.

Weaknesses

1. The paper is not well-organized, making it difficult to understand the main points and arguments presented. 2. The proposed framework adopts the TTS model as an STGNN, but some important TTS methods are not discussed in the related work, such as [1] and [2]. [1] Jianfei Gao and Bruno Ribeiro. On the equivalence between temporal and static equivariant graph representations. In International Conference on Machine Learning, pages 7052–7076. PMLR, 2022. [2] Da Xu, etc. Inductive representation learning on temporal graphs. In ICLR 2020.

Questions

The instructions in line 43 and lines 179-180 may appear contradictory. Could you provide a more detailed explanation?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

see weakness

Summary

This paper examines the interaction between global and local effects in graph-based spatiotemporal forecasting. It addresses the limitations of a single global model by introducing a framework that incorporates trainable node embeddings into graph-based architectures. This framework enables the learning of specialized components and combines the benefits of shared message-passing layers with node-specific parameters. Additionally, the framework facilitates model transfer to new node sets. The paper offers empirical evidence and provides guidelines for adapting graph-based models to the dynamics of each time series to improve prediction accuracy.

Strengths

It is nice to see a paper that investigates the attribution of "local" and "global" learning in modeling spatial-temporal graphs. The evaluation is very comprehensive and the paper is very informative. It may have great impact that can benefit the broad community that researches on spatial-temporal graphs.

Weaknesses

The paper tries to answer a set of very big questions ("local" vs "global"), which I feel could be too hard to find a concrete answer in a 9 pages conference paper. Similar questions can be asked for GNNs as well: Is message-passing more important or the node feature encoding more important? Should I go fully inductive like GCN, GraphSAGE? Or I just stick to non-inductive GNNs? Is the isotropic message-passing enough like vanilla GCNs, or do I need anisotropic message-passing like graph attention networks (GAT)? Do I interleave the MP layers with node encoding layers like most GCNs do? Or I should stack multiple node encoding layers before doing message-passing? I feel it is a little too overwhelming to answer all these questions at once. I really appreciate the authors' efforts to investigate these questions, but it feel less convincing when it fits into a 9-pages conference paper, that each claim will be supported by less empirical evidences. Sometimes I will question that, how does this claim hold for other applications when the nature of a problem changes, would the conclusions change? What's more challenging is that, these questions seem to not having a general answer that hold for all the applications, making it especially hard to draw conclusions by merely relying on empirical studies (or you will need a lot of experiments across much more domains). In general, it is overall a technical solid paper and an ambitious one as well.

Questions

I have to admit that I do not fully understand the paper, though I tried to read through the paper multiple times, so it may be helpful for me to understand the key contributions of the paper if the authors can provide some information of the following: 1. **What is the guideline and main takeaway messages for the audience to design models for spatial-temporal graph learning?** I feel a little lost in a vast amount of information and empirical observations. 2. **How do we know which design is the best for an application? Isn't it application specific?** If we want to know whether the "local" or "global" components in a STGNN are more important for an application, we may want to try it out or are there any ways to know it beforehand? To the best of my understanding, it could be very different from applications to applications, since global information is more important for some of them while for others local information is more important. 3. **Could T&S-AMP design be a generic to go choice?** If we do not know the importance of global or local effect in an application beforehand, can T&S and AMP be an to-go option? Many well-established GNNs for dynamic graphs fall in this category, for example, use attention in MP (AMP) and use alternating message-passing layers and recurrent layers (T&S), e.g., Graph Recurrent Attention Networks (GRAN).

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

No limitations and negative societal impacts are left unaddressed.

Summary

In this paper, the authors explore the influence of locality and globality in graph-based spatiotemporal forecasting architectures. Existing spatiotemporal models are global trained on multiple multivariate timeseires, which can capture the strong dependency among individual nodes in a network. Standard local models such as RNNs learn each timeseries independently which lost the interaction information with other nodes, but are fitted solely on each individual trajectory resulting in good short-term prediction performance. Directly combine the predictions from global and local models would result in a large number of model parameters (introduced by individual models). The authors instead propose to use a learnable embedding vector to represent the locality for each node and incorporate it in the GNN message passing procedure. The guide the learning process for such node embeddings, the authors further proposed two regularization terms to make the model more generalizable, with the assumption that the underlying dynamics of nodes within the same network topology would not differ too much. Experiment result over several benchmark datasets show the proposed method is able to make better prediction results than compared baselines.

Strengths

1. The writing for this paper is very easy to follow 2. The idea to inject local information into existing global spatiotemporal models is interesting. 3. The experiments are comprehensive, though some baselines are missing.

Weaknesses

1. My major concern is the contribution/novelty of this paper. The authors propose to learn a node embedding to mimic the role of local models such as RNN trained on each individual timeseries. First of all, the node embeddings are static, whereas the output of RNN models are dynamic. Those local features for each individual node can changes over time which can be well-captured by any local models. Secondly, the learnable node embeddings seem to me are similar to those exogenous factors specific to each node, how to guarantee the learned embeddings would not serve as the same role as those exogenous factors? Finally, learning these embeddings would make the whole model not able to perform inductive tasks. When a new node/timeseries comes in, one needs to retrain the model instead of directly use the model to do the inference, opposed to existing spatiotemporal GNNs. 2. Also there are some missing baselines in terms of spatiotemporal GNNs, such as continuous graphODE approaches [1][2] and other discrete methods [3]. [1] Huang, Zijie, Yizhou Sun, and Wei Wang. "Learning continuous system dynamics from irregularly-sampled partial observations." Advances in Neural Information Processing Systems 33 (2020): 16177-16187. [2] Song Wen, Hao Wang, and Dimitris Metaxas. 2022. Social ODE: Multi-agent Trajectory Forecasting with Neural Ordinary Differential Equations. In Computer Vision–ECCV 2022: 17th European Conference. [3]Sanchez-Gonzalez, Alvaro, et al. "Learning to simulate complex physics with graph networks." International conference on machine learning. PMLR, 2020.

Questions

1. Can the authors provide running time comparison during the testing stage, as the proposed method would need to retrain the model on unseen(new) nodes. 2. Can the authors visualize some of the learned local node embeddings and show some case study to interpret their semantic meanings?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

2 fair

Contribution

1 poor

Limitations

The authors have not discuss the limitations of their model.