A Hierarchical Spatial Transformer for Massive Point Samples in Continuous Space

Transformers are widely used deep learning architectures. Existing transformers are mostly designed for sequences (texts or time series), images or videos, and graphs. This paper proposes a novel transformer model for massive (up to a million) point samples in continuous space. Such data are ubiquitous in environment sciences (e.g., sensor observations), numerical simulations (e.g., particle-laden flow, astrophysics), and location-based services (e.g., POIs and trajectories). However, designing a transformer for massive spatial points is non-trivial due to several challenges, including implicit long-range and multi-scale dependency on irregular points in continuous space, a non-uniform point distribution, the potential high computational costs of calculating all-pair attention across massive points, and the risks of over-confident predictions due to varying point density. To address these challenges, we propose a new hierarchical spatial transformer model, which includes multi-resolution representation learning within a quad-tree hierarchy and efficient spatial attention via coarse approximation. We also design an uncertainty quantification branch to estimate prediction confidence related to input feature noise and point sparsity. We provide a theoretical analysis of computational time complexity and memory costs. Extensive experiments on both real-world and synthetic datasets show that our method outperforms multiple baselines in prediction accuracy and our model can scale up to one million points on one NVIDIA A100 GPU. The code is available at https://github.com/spatialdatasciencegroup/HST.

Paper

Similar papers

Peer review

Reviewer Y8FJ8/10 · confidence 4/52023-07-06

Summary

The authors proposed a hierarchical spatial transformer model for many irregular point samples in continuous spatial domain. Compared with existing methods, the proposed method can model implicit spatial dependency across irregular samples and in multiple scales in continuous space. The proposed model uses a quad-tree hierarchy to conduct efficient attention operations by approximating distant points in a coarse resolution. The model also includes an uncertainty quantification module to capture the varying prediction confidence in different input sample density. Evaluations on several datasets confirm that our method outperforms multiple baselines in accuracy and effectiveness of uncertainty quantification.

Strengths

Overall, this is a solid paper with novel technical contributions and extensive experimental evaluations. • The paper solves a significant problem of spatial representation learning for irregular samples in continuous space. The problem has many significant applications in environment sustainability. It is also important for learning surrogate models to speed up numerical simulations. • The technical novelty of the proposed hierarchical attention architecture is strong. The model uses a quad-tree to learn latent representation of point samples in different subareas in a multi-scale hierarchy. The attention layers use quad-tree to make spatial approximation to overcome the computational bottleneck. The model also has an uncertainty quantification module. • The proposed method has shown promising results on both real-world and synthetic datasets. The evaluation is solid with sensitivity analysis and computational experiments.

Weaknesses

Although the experimental evaluation is extensive, it will be helpful to add some more on how the thresholds of uncertain/certain predictions are determined in UQ metric.

Questions

• How are the thresholds of uncertain/certain predictions determined in UQ metric?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

No limitations on societal impacts.

Reviewer J8MW4/10 · confidence 4/52023-07-06

Summary

This paper proposes a hierarchical transformer to model a large number of irregular point samples in continuous space. This is achieved by a quad-tree hierarchy which could learn the multi-scale spatial representation. So the long-range interactions are recorded. The experiment is performed in three real-world dataset (two water quality datasets and one sea-surface temperature prediction) and one simulation dataset.

Strengths

This submission tries to solve the large number of issues by using quadtree. In my understanding, the quadtree is a classic technique for downsampling points so that the point number will be reduced to an affordable level. And the quadtree could model the long-range interactions without the distribution assumption. The proposed method reduced the computational cost to $O(N\log N)$ or $O(NM)$. The approach presentation is detailed and clear.

Weaknesses

The adopted quadtree technique seems could only work well in the sparse case. As a result, the proposed method is not a general solution for all kinds of large numbers of point samples. The experiment setting is not convincing in the reviewer's current understanding. In particular, Red Tide and Turbidity datasets have a large number of data samples, but it seems only to have one 'epoch'. In other words, The point number is large but the total data size seems very small. And for Darcy flow and Sea Surface, it seems each image only have 400 point samples, it is not a large number set in my understanding, the all-pair transformer could cover it.

Questions

Could the author clarify how the adopted quadtree technique in the paper could solve uniformly distributed cases? Link to the weakness above, as Darcy flow and Sea Surface datasets are set, could the author claim why do we have to feed all point samples in a single feed? In other words, why don't we solve this task in all three datasets by following a classic 'classification setting'? Compared with the all-pair transformer, pros and cons? Could the reviewer know the number of parameters for all-pair transformer and other baselines in Table 2? Any guidance in choosing hyper-parameters for designing quadtree if needed? As in Weakness, my main concern is first two questions.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Reviewer Ma6V5/10 · confidence 5/52023-07-07

Summary

This paper proposes a quad-tree partition of irregularly distributed sample locations. This leads to an algorithm with O(NlogN) complexity.

Strengths

Quad-tree idea is nice, even though not original for irregular grids or pixels.

Weaknesses

The targeted problem in the paper is regression. But the model is framed as an encoder-decoder. The paper does not explain why this seemingly unreasonable choice. The O(NlogN) complexity has a large big-O constant because of sparse matrix operations which are known to be inefficient on GPUs. There are not enough details on models and experiments. For example, the model parameters such as number of layers, hidden size and number of attention heads are not clearly specified. The training hyper parameters such as learning rate and its decay schedule are not explicitly stated. This makes it hard to determine the quality of the experiments, let alone replicating them. Datasets are all 2D with up to 1K sample points. These are well within the capacity of vanilla Transformer on A100, the GPU used in the paper. And the optimal leaf node size is 100. This results in a very shallow quad tree. So the evidence about the efficiency gain is not convincing.

Questions

What is the vocab for the decoder? The solution space for regression is not discrete.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

NA

Reviewer Ma6V2023-08-20

Thanks for the clarification. I have increased my ratings.

Reviewer 7rvT8/10 · confidence 4/52023-07-07

Summary

This paper proposes a hierarchical spatial transformer model for a large number of point samples in continuous space. The model is important for geoscience applications, such as water quality monitoring and air quality monitoring, and operator learning for numerical models. The novel idea includes continuous position-encoding and hierarchical attention layers, which make a trade-off between efficiency and spatial resolution and captures the interactions in multiple spatial scales. The proposed model also has uncertainty quantification that reflect the effect of sample spatial sparsity. The proposed method is compared against multiple baselines on different datasets with significant improvements. There are also computational experiments to show the efficiency of the method.

Strengths

1. The idea of hierarchical attention with a quad-tree structure for massive samples in the continuous space is quite novel. The idea makes a trade-off between computational efficiency and spatial approximation of latent representation. The spatial approximation in a quad-tree structure is well-motivated by the spatial autocorrelation effect. 2. The proposed method has strong technical contributions. There is theoretical analysis of the time complexity of the proposed model and the gain in efficiency. 3. There are extensive experimental evaluations against multiple state of the art methods on both real-world and synthetic datasets. The results show the better accuracy and uncertainty quantification of the proposed method.

Weaknesses

1. There are some minor presentation issues that can be fixed. For example, Table 2 has a typo in the title. It should be “two real-world datasets and a synthetic dataset”. 2. The application paragraph in the introduction can be more detailed to better highlight the impact of the model in operator learning for numerical simulation.

Questions

1. Can you explain in more details how the proposed model can be used in numerical simulations (e.g., ocean current, multiphase flow) in the introduction?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

4 excellent

Limitations

No limitations discussed in the paper.

Reviewer 7rvT2023-08-15

Rebuttal read

Thank you for providing the rebuttal. I believe they addressed my previous questions. The paper makes valid and practical contributions to the machine learning community. The proposed idea can also beneifit a broad range of scientific domains. In my opinion the paper should be accepted.

Area Chair Jy8N2023-08-18

Dear authors, thank you for your rebuttal. I will take into consideration your unanswered responses about the computational complexity, applicability of the method, and comparison with other transformers.

Reviewer Y8FJ2023-08-18

Thank you for the response

Thanks for the responses. The authors have properly addressed my questions. I am fine with an acceptance.

Program Chairsdecision2023-09-21

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC