In recent years, there has been a rapid development of spatio-temporal prediction techniques in response to the increasing demands of traffic management and travel planning. While advanced end-to-end models have achieved notable success in improving predictive performance, their integration and expansion pose significant challenges. This work aims to address these challenges by introducing a spatio-temporal pre-training framework that seamlessly integrates with downstream baselines and enhances their performance. The framework is built upon two key designs: (i) We propose a spatio-temporal mask autoencoder as a pre-training model for learning spatio-temporal dependencies. The model incorporates customized parameter learners and hierarchical spatial pattern encoding networks. These modules are specifically designed to capture spatio-temporal customized representations and intra- and inter-cluster region semantic relationships, which have often been neglected in existing approaches. (ii) We introduce an adaptive mask strategy as part of the pre-training mechanism. This strategy guides the mask autoencoder in learning robust spatio-temporal representations and facilitates the modeling of different relationships, ranging from intra-cluster to inter-cluster, in an easy-to-hard training manner. Extensive experiments conducted on representative benchmarks demonstrate the effectiveness of our proposed method. We have made our model implementation publicly available at https://github.com/HKUDS/GPT-ST.
Paper
Similar papers
Peer review
Summary
This paper proposes a novel pre-training framework (GPST) for ST prediction, it mainly integrates the ST parameter personalization scheme and the region-wise semantic association mechanism into the pre-training model, and conduct the training in an unsupervised manner. Besides, the proposed model GPST adopts the hierarchical hypergraph structure to capture the semantic-level association of multiple regions from a global perspective. Extensive experiments are conducted.
Strengths
1. This paper is well-presented and well-organized. 2. This paper introduces a spatio-temporal pre-training framework that can be easily integrated into downstream baselines and improve their performance. 3. Detailed experimental results are provided to validate the model.
Weaknesses
1. The model's performance seems to heavily rely on the pre-training stage. If the pre-training is not done correctly or if the pre-training data is not representative of the test data, the model's performance could suffer, how can this case be handled? 2. Why not include POIs, which should benefit the prediction performance.
Questions
See above.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Noone
Summary
This paper introduce pre-training task for spatio-temporal. The algorithm design includes a spatial-temporal knowledge extractor and a pre-training mechanism. The design of the pre-training methods is to resolve the two limitations of existing spatial-temporal learning: (1) lack of comprehensive personalization (2) insufficient consideration of modelling semantic.
Strengths
This paper study an important problem: GNN-pretraining.
Weaknesses
1. The presentation wasn't very clear, please refer to questions. 2. Missing many recent baselines, e.g., TGAT, JODIE, TGN. The authors can compare with those works using https://github.com/amazon-science/tgl. Besides, there are most more recent methods [1-3] need to be included in discussion. [1] Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks [2] Provably expressive temporal graph networks [3] Do We Really Need Complicated Model Architectures For Temporal Networks? 3. Experiment dataset size are all relatively small. I am not sure this method can scale to very large graph from experiments.
Questions
1. Could you please explicitly summarize what is the SSL pre-training method different from the method on static graph? My impression is the main contribution is due to spatial-temporal vs static graph setting. The mask and predict 2. The hyper-edge definition is not very clear to me. At line 116 "each of which connects multiple vertices", could you please explain how an edges can connect to more than 2 nodes? 3. How intra-class pattern and inter-class relation are defined (line 138). Is this coming from dataset, or the authors define them based on human knowledge per dataset. 4. Is the pre-training task a link prediction task? Do you explicitly consider the potential information leakage issue during pre-training. For example, the positive edges for pre-training might be sampled as a neighbor when computing the node representation. 5. Will GNN size affect the pre-training efficiency?
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
1 poor
Contribution
2 fair
Limitations
No
Summary
The authors introduce a spatio-temporal pre-training framework that can be easily integrated into downstream baselines. The framework comprises personalized parameter learners and hierarchical hypergraph networks. The former enables the acquisition of spatio-temporal personalized representation and the latter capture the intra-class and inter-class correlations among regions. Besides, the authors design an adaptive mask strategy to guide the model in learning diverse relationships across regions. Experiments are conducted on representative benchmarks on four real-world datasets to evaluate the proposed framework.
Strengths
1. The authors propose a novel pre-training framework for Spatio-Temporal prediction which can be easily applied to existing advanced ST neural networks. 2. The authors devise an adaptive mask strategy that guides the model in a progressive manner, facilitating the acquisition of useful knowledge from other categories to recover unknown categories. The comparisons with other mask strategies demonstrate the superiority of the designed mechanism. 3. The authors evaluate both the original performance and enhanced performance with the proposed framework of different baselines on four datasets. The results indicate that the proposed model significantly improves the downstream baseline prediction performance and confirm the effectiveness of the framework.
Weaknesses
1. The presentation of this paper should be improved: (1) The explanations of some concepts are not clear. For instance, the meaning of the symbol $M_{r,t}$ in Section 4.2 is unclear. Does it represent the mask operation? Furthermore, the first-level hyperedges mentioned in Section 4.3.2 lack clear explanation, as it is the first occurrence of the term "first level" in the paper. (2) Equation (4) is referenced multiple times in the paper to illustrate the process of generating embeddings. However, it is necessary to provide a more detailed explanation of how the equation is actually implemented. For instance, in Line 173 of Section 4.2, it would be helpful to provide specific equations for the computations involved. The inclusion of concrete examples would greatly enhance the understanding of Equation (4) and its functionality. (3) In Section 4.2, there are multiple subscripts used, including (r,t), r, and t. It would be better if provide a more detailed explanation to clarify the differences between these descriptions. (4) In Section 5.1.1, the paper mentions the use of the absolute error loss function to optimize the parameters. However, there is a lack of detailed description regarding how this loss function is precisely employed. It would be beneficial to provide a more thorough explanation of how the optimization process utilizes the absolute error loss function, including any specific computations. 2. The authors adopt the hierarchical hypergraph structure to capture the semantic-level association of multiple regions from a global perspective. However, the construction of the hypergraph requires a more detailed description. Specifically, the definition of hyperedges and capsules in Section 4.3 needs to be clarified. The paper mentions that hyperedges are treated as high-level capsules, but what exactly does a capsule consist of? Furthermore, it would be helpful to explain the type of hypergraph connection that a hyperedge represents. 3. Some related work with similar ideas is not discussed. For example, the paper [1] also builds a pre-training model on urban data and builds a personalized model in a specific area. Though the idea is not the same, the author should discuss the difference in their related work at least. 4. It would be beneficial to include a brief discussion on future work in the paper. [1] A Contextual Master-Slave Framework on Urban Region Graph for Urban Village Detection, ICDE 2023.
Questions
1. What’s the “another initial” in Line 206 of Section 4.3.1 mean? 2. Is the aforementioned c in Section 4.4 from Section 4.3.1?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
The authors have not adequately addressed the limitations.
Summary
This paper introduces a spatio-temporal pre-training framework called GPST (Generative Pre-training framework for Spatial Temporal prediction) to enhance the performance of existing models in traffic management and travel planning. The goal is to address the challenges of integrating and expanding refined models, while achieving better predictive performance. The framework consists of two main components. (1) a spatio-temporal knowledge extractor that utilizes personalized parameter learners and hierarchical hypergraph networks. These modules are designed to model personalized representations and semantic relationships between regions, which have been overlooked in previous work. (2) an adaptive mask strategy is proposed to guide the knowledge extractor in learning robust spatio-temporal representations. The paper conducts extensive experiments on representative benchmarks and demonstrates the effectiveness of the proposed GPST method.
Strengths
1. The two research questions are interesting. Question (1) in personalized information extraction is well studied in STGNN. Question (2) is relatively novel since region functions could make a difference but not considered too much in the literature. 2. The experiments results look promising.
Weaknesses
1. Related works should include SSL works from both Graph and Multivariate Time Series. I think current version has some missing references in esp. Graph SSL, which could be related to the strategy developed in this paper (needs further justification) 2. Figure 3 is really small to understand clearly. Visual illustration is better decoupled into text description. 3. Section 4 has too many details that could be moved to appendix since these detailed designs are not quite novel, though important. I'd suggest this section to be more compact and highlighted. 4. I have some concerns about the improvement by using GPST. It is clear in Table 1 that some methods benefit a lot and some not that much. I expect a more detailed analysis on this. Is some design of GPST redundant in GWN, MTGNN, etc? What is the core part of GPST that makes performance better? Or, is GPST dependent of certain architecture designs? How are these two factors (architecture and ssl strategy) decoupled?
Questions
Thanks for clarifying the above mentioned weaknesses.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness
2 fair
Presentation
2 fair
Contribution
2 fair
Limitations
NA
Q1: I think I didn’t make myself clear in the previous feedback. Let me rephrase in another way. Let’s say the time-encoding in Eq. 3 is a contribution as mentioned by author. But if we look at Eq. 3, its just applying an MLP on the input node features, and here the input node features is named as day feature $z^{(d)}$ and week feature $z^{(w)}$. By replacing the $z^{(d)}, z^{(w)}$ with any node features $x$, it looks exactly a static graph algorithm. The same for adaptive mask strategy, I am use it exactly for static graph. Q2: I am still confused. You mean hyper-edge is actually like another node. For example if hyper-edge connect node 1 with node 2+3, then its like (1, hyper-edge), (hyper-edge, 2), (hyper-edge, 3)? Q3: Sorry, I know what is intra-class pattern and inter-class relation. My original question is “Is the intra-class pattern and inter-class relation coming from dataset, or the authors define them based on human knowledge per dataset.” How do you know this exists? Any evidence? Q4: I was asking this because most graph learning model use link prediction to explicitly learn the graph structure. This is often the case for most temporal graph network methods: pre-train on link-prediction then use it for node-classification. Q5: Most pre-training method works well on large model, e.g., the ResNet in vision task and Bert Models in language task, and it doesn’t often works well on GNN. For example if we take a look at open graph benchmark, none of those methods are using pre-training. One of the hypothesis is GNN is small, not suitable for pre-training. Therefore, many graph transformers are proposed and they show pre-training on those large GNN models indeed help. My question originally want to ask whether you have same observation. I should use “effectiveness” instead of “efficiency”. Q6: Temporal graph networks can also handle node classification task if I recall correctly. Could you please elaborate how GPST are integrated with JODIE and TGN?
Thank you very much for your response. We will address the issues you have raised. **Response to Q1:** Thank you for your feedback. The graph structure varies across different time steps in our method, distinguishing it from static graphs that remain unchanged over time. In specific, Eq (3) demonstrates the generation of temporal features based on actual time information. These features can be integrated with the hypergraph structure to produce the dynamic hypergraph structure. For example, the hypergraph connectivity $\textbf{H}'_t$ is customized to the $t$-th time slot using another initial temporal embeddings $\textbf{D}_t^{'}$ by $\textbf{H}'_t=\text{softmax}(\textbf{D}_t^{'\top} \cdot \bar{\textbf{H}}')$, where $\bar{\textbf{H}}'\in\mathbb{R}^{d'\times H_S\times R}$ contains embeddings for each hypergraph connection (Section 4.3.1 start from line 205). Here, $\textbf{H}'_t\in\mathbb{R}^{H_S\times R}$ (also $\textbf{H}'\in\mathbb{R}^{T \times H_S\times R}$) represents the generated dynamic hypergraph structure. This structure is employed for personalized message aggregation on node features at different time steps. However, for static graphs, the aforementioned process does not exist. The proposed adaptive mask strategy exhibits spatio-temporal awareness as it is based on the aforementioned dynamic hypergraph structure. **Response to Q2:** Thank you for your response. The concept of hyperedges and the mentioned another type of node share some similarities. It is akin to a virtual node at another hierarchical level that associates multiple actual nodes. The dinsinction lies in that hyperedges do not represent concrete entities (e.g. regions, roads, intersections) in the physical world, but nodes are usually used to represent entities in the observed data. Furthermore, related works [1-2] have indicated the effectiveness of hypergraphs in clustering or latent factor mining. [1] Inhomogeneous Hypergraph Clustering with Applications, NIPS 2017 [2] Hypergraph Clustering Based on PageRank, KDD 2020 **Response to Q3:** Thank you for your comment. In brief, these patterns are not labeled in the dataset. Instead, we expect the model to learn these latent relations. Our case study serves as evidence of GPST's capability to discover such patterns. Specifically, the employed dataset records urban ST data (e.g. traffic flows). Similar regions within the city exhibit analogous traffic patterns (e.g., high traffic during holidays in commercial districts), while dissimilar regions show interdependencies (e.g., people moving from residential to office areas during peak hours), representing normal human activities. Such patterns are unlabeled in the dataset. Thus, we empower GPST with the capacity to capture these patterns by constructing a learnable hierarchical hypergraph network. We confirm this through the case study (Section 5.3). In Figures 5(a) and 5(b), GPST groups three regions with akin traffic patterns into a single category. The latitude and longitude data further affirm their common classification, like being commercial districts. And Figure 5(c) illustrates potential traffic migration between different-category regions captured by GPST. **Response to Q4:** Thank you for your response. If you have any questions about pre-training tasks, please feel free to discuss them with us. **Response to Q5:** Thank you for your valuable feedback. We fully recognize the significance of having sufficient and high-quality training data for achieving successful pre-training. In the domain of ST prediction tasks, although the dataset may not contain a large number of nodes, each individual node encompasses a substantial amount of temporal features. For instance, the METR-LA dataset records traffic data over a four-month period with a sampling frequency of 5 minutes. This rich historical data serves as a solid basis for the pre-training process of GPST, and its ability to effectively enhance downstream baselines further substantiates this observation. **Response to Q6:** Thank you for your reply. Similar to other baselines, we utilize the representations generated by GPST as inputs to enhance the performance of JODIE and TGN (appendix A.1.3). The implementation details for JODIE and TGN are as follows. (1) JODIE: We assigned static embeddings to each node (or region) and used RNNs to handle the dynamic embedding updates of node features, where GNNs replaced the interactions between users and items. A time encoding layer combined with embedding mapping operations was used to update the dynamic embeddings of nodes to accommodate predictions at different time steps. (2) TGN: MLPs and RNNs were employed as the message and memory functions for updating node features. A multi-head attention mechanism was utilized to aggregate messages from neighbors (constructed based on distance). Thank you again. We welcome further discussion and are available to address any questions you may have.
Thank you for the responses.
The authors tried their best to answer all my questions, and most of my concerns are addressed.But I have to say that not all answers given by authors make sense to me (e.g., major contribution mentioned in Q1, evidence intra- and inter-class pattern is important instead of only intuitions in Q3, why only consider mask-and-prediction but not other graph pre-trianing in Q4, impact on model size in Q5), but this could potentially because I am not familiar with the spatial-temporal setting used in this paper, and may not correctly evaluated this paper as other reviewers. This paper is more like a data-mining paper that apply existing ML algorithms on a spefic real world problem to solved it: This paper take mask-and-predict to solve a traffic prediction problem and using some dataset spefic observation to get better performance. I have to say it is not very easy to digest this paper, and I believe it belongs to data mining conferences (e.g., ICDM, WWW, KDD) more than ML conferences. I was expecting to papers working on **theory** or **empirical study** to reveal some more **general ML problems** (e.g., consider spatial-temporal as a more general setting instead on solving a specific downstream task, e.g., traffic prediction). Therefore, I increased my score slightly given the effort authors made during rebuttal, and I will also reduced my confidence score because I am not familiar with the specific setting in this paper. Suggestions: In the revised version, the authors might want to try other popular pre-training methods, espetially those designed for MPNN with an ablation study to justify why mask-then-prediction is a good choice.
Response to Reviewer 8n4n
We sincerely appreciate your positive feedback. The improvement of the score is a profound affirmation and encouragement for us, and your suggestions are exceptionally insightful. We are fully committed to addressing any questions or concerns you may have regarding this work. ***Comment 1: Discussion of the mentioned questions.*** **Response:** We appreciate your response. Regarding Q3, we consider the intra- and inter-class patterns to be manifestations of the regularities exhibited by urban activities. (e.g., regions with similar functionalities exhibit similar spatio-temporal patterns). Such regularity can be derived from empirical observations of human activities. We will address Q4 in Comment 3. If there are any specific questions or concerns regarding Q1 and Q5, we will make every effort to address them. ***Comment 2: Generalizability of GPST.*** **Response:** We sincerely appreciate your feedback. In this paper, we propose a spatio-temporal pre-training framework to enhance the performance of downstream spatio-temporal models. Following the settings of previous spatio-temporal graph studies [1-2], we focus on the specific task of spatio-temporal prediction in real urban scenarios. Real-world data distributed on spatio-temporal dimensions (e.g. traffic, crime, house pricing) has their unique but common patterns. For example, spatio-temporal features exhibit diverse evolutionary patterns across different time periods, and regions with similar functionalities tend to demonstrate similar spatio-temporal patterns. This paper aims at excavating such patterns with a unified pre-training framework. And our empirical study suggests its generalization ability across a variety of different tasks. [1] Graph wavenet for deep spatial-temporal graph modeling, IJCAI 2019. [2] MSDR: Multi-step dependency relation networks for spatial temporal forecasting, KDD 2022. ***Comment 3 Additional Ablation Experiments.*** **Response:** Thank you for your suggestion. In the Model Ablation Study (Section 5.3), we compared our approach with several generative pre-training strategies, such as GraphMAE. To further investigate why mask-reconstruction is a good choice, we extended our exploration to other types of pre-training approaches, specifically contrastive pre-training. Contrastive learning is a widely adopted pretraining strategy for representation learning with the techniques of data augmentation and information maximization. We selected DGI[3] and GraphCL[4] as baselines, which are two widely recognized and extensively used pretraining strategies in GNNs. DGI: This method learns node representations in graph-structured data by maximizing the mutual information between local representations and global summaries. GraphCL: It incorporates four data augmentation techniques to generate two contrasting views, optimizing the model by maximizing the mutual information between the views. We conducted ablation study on the METR-LA and NYC TAXI datasets, and the results are presented in the table below. Here, "r/" denotes the replacement of the GPST pretraining strategy with the DGI or GraphCL pretraining strategy. The results demonstrate the effectiveness of contrastive pre-training methods in enhancing downstream models, highlighting the adaptability of contrastive learning strategies in our model for representation learning. Notably, our approach achieves the most significant performance enhancement, possibly attributed to the higher correlation between the mask-reconstruction task and the downstream task (regression task). In addition, the proposed adaptive mask strategy also contributes to the model to learn robust spatio-temporal representations. Results on METR-LA dataset | Model/Metrics | MAE | RMSE | MAPE | |:--:|:--:|:--:|:--:| | GWN | 3.15 | 6.44 | 8.67% | | r/ DGI | 3.12 | 6.36 | 8.64% | | r/ GraphCL | 3.10 | 6.30 | 8.50% | | w/ GPST | **3.02** | **6.18** | **8.29%** | Results on NYC TAXI dataset | Model/Metrics | MAE | RMSE | MAPE | |:--:|:--:|:--:|:--:| | GWN | 5.64 | 9.92 | 42.19% | | r/ DGI | 5.37 | 9.30 | 40.20% | | r/ GraphCL | 5.38 | 9.55 | 38.49% | | w/ GPST | **5.12** | **8.89** | **37.49%** | [3] Deep Graph Infomax, ICLR 2019. [4] Graph Contrastive Learning with Augmentations, NIPS 2020.
Thanks for rebuttal
The responses from the authors have clearly addressed the problems pointed out previously. Thanks for all responses.
General Response
We extend our heartfelt appreciation to all the reviewers for their dedicated time and effort in reviewing this work. Their constructive feedback and valuable suggestions have greatly contributed to improving the paper. We are delighted to hear that the reviewers find the problem addressed in our work to be interesting (Reviewer H5wT) and important (Reviewer 8n4n), and consider that the paper is well-presented and well-organized (Reviewer 83m7). We also appreciate the suggestions mentioned by the reviewers, such as enriching the related work (Reviewer H5wT, fFpq, 8n4n), highlighting key aspects of the methodology (Reviewer H5wT), including additional baselines and pre-training methods from similar works (Reviewer 8n4n), providing detailed descriptions of equations and notations (Reviewer fFpq), and improving the figure captions (Reviewer H5wT). We have addressed each reviewer's comments with specific responses, and we will incorporate the necessary revisions in the paper to address their concerns. The key revisions are summarized below, and also mentioned in the corresponding responses to each reviewer. (1) Related Work (Section 2): We have expanded this section based on the insightful suggestions provided by Reviewers H5wT, fFpq, and 8n4n. (2) Methodology (Section 4): To enhance the conciseness and focus of the this section, we have condensed it and moved specific details of routine operations to the supplementary material, as recommended by Reviewer H5wT. (3) Main Results (Section 5.2): Taking into account the valuable feedback from Reviewer 8n4n, we have included additional baseline results into the comparative experiments. (4) Model Ablation Study (Section 5.3): In light of the insightful suggestion from Reviewer 8n4n, we have included some popular pre-training techniques of GNNs in the ablation experiments. (5) Detailed Descriptions: Following the suggestions of Reviewer H5wT, we have supplemented the textual descriptions of Figures 1, 2, and 3. Additionally, in accordance with the suggestions from Reviewer fFpq, we have enriched some detailed descriptions of notations and relevant formulas. We would like to express our gratitude once again to the reviewers for their valuable suggestions.
Thanks for your response
Dear authors; Thanks for your effort to response my questions. I have no future points arising for the discussion. Regards
Thank the authors for reply. I don't have further questions.
Decision
Accept (poster)