Advection Augmented Convolutional Neural Networks

Summary

The paper introduces Advection Augmented Convolutional Neural Networks (ADRNet), integrating a semi-Lagrangian push operator into CNNs to improve spatio-temporal sequence prediction. Mimicking the Reaction-Advection-Diffusion equation, ADRNet addresses challenges in long-range information propagation and explainability. Evaluations on scientific datasets (CloudCast, SWE) and video prediction datasets (Moving MNIST, KITTI) show ADRNet's superior performance over baseline models.

Strengths

**Originality**: - Innovative use of a semi-Lagrangian push operator for long-range information in CNNs. **Significance**: - Addresses a key limitation in CNNs for spatio-temporal prediction tasks. - Potential impact in fields like weather prediction and traffic flow analysis.

Weaknesses

**readability**: - The introduction of the method in this paper primarily relies on formulas, lacking the necessary explanations and descriptions, making it less readable. - There are writing errors in the article, making it difficult to understand, such as in line 246. **Scalability**: - Scalability to larger, real-world datasets and more complex models is not thoroughly explored. **Replication of results**: - Although the experimental parameter descriptions are provided, due to the lack of running code, going to reproduce the results of the experiment, however, requires quite a bit of work

Questions

What criteria or methods did you use to determine hyperparameters for different datasets and models? Have you tested ADRNet on larger and more diverse datasets? What were the results? Are there ways to enhance the generative capabilities of ADRNet for more complex video prediction tasks?

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

**Addressing Limitations**: - The paper acknowledges limitations in generative capabilities for video prediction. **Suggestions**: - Explore methods to enhance generative aspects. - Test on larger, more complex datasets and discuss results and challenges. - Discuss potential societal impacts, such as ethical use of predictive models in sensitive areas like healthcare.

Summary

This work is based on an analogy of CNNs to Advection-Diffusion-Reaction PDEs: Classical ResNets can be understood as pure Diffusion-Reaction PDEs, without an Advection Term. Thus, this work proposes to introduce a new layer resembling the Advection Term, to allow for long-range transportation of information. More specifically, they propose to learn the velocity fields, with which to advect the coordinates with a CNN. Advected quantities can then be computed by simply moving the quantities on the old grid to the new coordinates, and then interpolating back to the old grid (which they propose to do in a mass-conserving way, i.e. resembling conservative semi-lagrangian advection). Leveraging this new advection layer, the authors find performance on par with state-of-the-art approaches on a few benchmark datasets from PDE emulation, weather radar forecasting and video prediction. In addition they find indication, that the added advection layer can improve direct longer horizon forecasts for the Shallow Water Equation.

Strengths

1. Introduces a new neural network layer that mimics semi-lagrangian advection 2. Shows on multiple common (and not too easy) benchmarks against strong baselines that the derived ADRNet architecture is competitive 3. Paper is reasonably well written, with some less-important details moved into the appendix.

Weaknesses

Major points: 1. If the CFL conditions are fulfilled, then Advection should be local. In other words, if the time step is small enough (compared to velocity), advection only moves quantities to neighboring pixels. Then, a regular CNN with local filters should be able to capture advection. Hence, i'd expect your layer to be most useful for simulations with large time steps compared to velocity. I am unsure if the datasets presented in this paper fall into this category! You already partially discuss this, in Line 204-207. I suggest to extend the discussion and maybe also study how important long-range advection is in the datasets presented here, e.g. by performing image co-registration between the time steps (you could use 2D cross correlation for this), and then plotting the average displacement distance for each dataset. Potentially compare this with the average displacement your trained models learn inside the advection layers. You should also compare this to the receptive field, that the CNNs have without the advection layer. You could also tailor the benchmark datasets to be more suitable for your model by deliberately increasing their time step and thus the need for your advection layer. Of course if you do this, communicate it clearly, and best report scores for multiple step sizes, which should ideally show how the advection layer becomes more and more important, the longer the time step. 2. For video prediction, optical flow is still often integrated in SOTA approaches. It is similar to advection, and since you propose your model may be useful for video prediction, I suggest you discuss this in the related works. 3. Your experiment section is very short. As is, it is not clear what exactly you evaluate. Please elaborate in the text. 4. The description of the Advection layer should be central, as it is the main contribution. I found it a bit short, in particular the Pseudocode Algorithm 1 does not detail how the Push-Operator works. Also section 3.2 could be extended to describe more in-depth how the conservative regridding works in practice. Maybe even add a figure that describes your approach. 5. Section 2 seems a bit long. I mean it is very nice to draw this analogue of CNNs to the ADR equation to introduce the advection layer, but it is neither essential for understanding the layer itself, nor is it very novel. I suggest to shorten this part. Same holds for Section 3.1 Minor Points: 1. Please place your floating elements on the top of the page, [t], as is pages 7 & 8 are somewhat visually confusing. 2. Typos and missing words, e.g. L248 Figure _ 3. Tab4 what do 10->5, 10->10 and so on mean? Describe it in the caption!

Questions

- How about dilated Convolutions? The recent GraphCast paper introduces long-range connections in their multimesh to solve the issue of propagating information over longer distances. The equivalent for CNNs would be dilated convolutions, that is, the filters contain holes/zeros, and thus allow for propagating information from further away without heavy oversmoothing. It would be cool to see an ablation study against this. - A common application of models, like the PDE emulators in the study, is to use them in long-term autoregressive rollouts. Standard CNNs are often unstable over such longer rollouts. It would be super interesting to see how the ADRNet does in such a setting. - Would have been great if this submission would have had a code repo, i would have liked to try the network myself.

Rating

7

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

The study of limitation is very short and only existing in the survey. Please discuss in the main text, do you observe any artifacts from introducing the advection layer? (e.g. how about at the image boundaries?) How well does training go, is it easier or harder to train?

Summary

This paper proposes a method to enhance the performance of networks in solving PDE problems using CNNs. To achieve this, the authors present a computational approach for efficiently retrieving information from distant points in spatio-temporal datasets. Their main contributions can be summarized as follows: 1) They propose an interpretation method for information using the advection-diffusion-reaction step. To implement this accurately, 2) they introduce a semi-Lagrangian linear layer. 3) They present a training approach for these components. Despite minor issues such as experimental validation, computational efficiency, or the readability of figures, this paper is well-structured mathematically and demonstrates high performance in the field.

Strengths

1. The interpretative range of information in a CNN, considered a function, is limited to its receptive field. This limitation makes it challenging to interpret information outside of this domain. The paper addresses this issue by proposing the Advection-Diffusion-Reaction approach, significantly improving performance. 2. The equations proposed in the paper are clear regarding variable declarations and usage, and they are written in a way that is easy to follow. While aspects like rigor may require a more detailed review, there appear to be no major issues based on my examination. The paper effectively formalizes the challenges faced by existing networks and provides solutions to address these issues.

Weaknesses

1. The most notable drawback is the absence of visualizations, such as attention maps, to confirm that the concept shown in Figures 1 and 2 operates as expected at each feature level, even in toy examples. While the network demonstrates significant performance improvements, experiments are needed to verify whether the concept functions correctly compared to the computational complexity involved. In particular, it would be impressive if a pre-trained model could observe the proposed problem of long-range information being advected and diffused. 2. Overall, there is a need to enhance readability, including improving the concept figures. Specifically, Figures 2 and 5 require better clarity. Additionally, the graph in Figure 1-(c) could benefit from readability improvements. The absence of source code makes it difficult to understand how each step was implemented, which adds to the difficulty of reading and evaluating the paper.

Questions

Check Weakness

Rating

5

Confidence

2

Soundness

3

Presentation

1

Contribution

3

Limitations

Yes

Summary

This study addresses spatio-temporal prediction problems in the physical sciences. Since existing methods for temporal prediction mechanisms based on convolutional neural networks (CNNs) often underperform when propagating long-range information, the authors proposed a physics-inspired architecture that mimics the reaction-advection-diffusion equation in high dimensions.

Strengths

- The motivation of the proposed algorithm is straightforward. - The idea of operator splits to construct a neural network from the advection-diffusion-reaction differential equation is interesting.

Weaknesses

- The authors need to include many essential steps so that readers can fully understand the proposed idea. For example, the approximation described in eq. (8) needs to be explained in detail; the authors merely refer the readers to [1]. This approximation is a crucial process that enables operator splitting, and therefore, a detailed analysis should at least be included in the appendix. - I wonder why the diffusion and the reaction can be combined in a 3$\times$3 convolution layer. I agree that the reaction step can be implemented with a nonlinear 1$\times$1 convolution layer. However, for a diffusion step, as shown in Fig. 2, the dilated convolution is better suited to describe the diffusion step. - There is no quantitative comparison between the proposed ADRNet and other algorithms. Even though Fig. 5 illustrates the results of ADRNet, these frames are not suitable for demonstrating spatio-temporal prediction as there are no significant transitions between temporal frames. - As can be seen in Table. 8 and Table. 9, ADRNet performs poorly compared to the other algorithms and even shows quite a gap to SOTA performance, which raises doubts that the proposed ADRNet can consistently address the spatio-temporal prediction problems.

Questions

- The order of reaction, diffusion, and advection seems to be crucial for the design of the neural network. In Fig. 2, the process is shown in the order advection-diffusion-reaction, while the proposed operator splitting processes the data in the order reaction-diffusion-advection. Is there a particular reason for adopting this sequence in operator splitting? In addition, it would be necessary to empirically investigate whether this method can be generalized to various scientific datasets that do not follow this sequence of processes. - In Table.6, DMVFN [21] is duplicated. Please check it.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

See the weaknesses and questions.

Thank the author for the detailed reply. The reviewer maintains the current score.

Thank the author for the detailed reply, which has solved most of the doubts. I think the current rating is appropriate. The author's work is very interesting to the reviewer, and the author is encouraged to release the source code to support the reviewer to better verify the presented work.