PreDiff: Precipitation Nowcasting with Latent Diffusion Models

Summary

This paper presents the PreDiff for precipitation nowcasting. PreDiff adopts the latent diffusion for prediction. Besides, a special knowledge control design is used in a classifier guidance way to incorporate auxiliary prior knowledge, which can guide the model to satisfy conservation constrain. In both synthetic and real-world benchmarks, PreDiff achieves a good performance.

Strengths

1. This paper presents a practical method for precipitation nowcasting. Using diffusion model can benefit the prediction skill. 2. The proposed knowledge control is reasonable and effective. 3. This paper is well-organized.

Weaknesses

1. The major concern is about novelty. This paper introduces the famous diffusion model techniques into precipitation nowcasting task, including the condition diffusion, latent diffusion model and classifier guidance. The key design is the knowledge control. But accoding to the paper, the knowledge control requires maual designed for every specific task. And there does not exist evidence that anticipated precipitation intensity is a right prior knowledge for precipitation nowcasting. More explanation are expected. 2. More comparing baselines are insufficient. There are some advanced deep models for precipitation nowcasting that the authors should take into consideration, especially the second baseline that is based on the GAN. [1] MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions, CVPR 2021 [2] Skilful precipitation nowcasting using deep generative models of radar, Nature 2021 3. Model efficiency comparisons are also required, including the GPU memory and running time.

Questions

(1) In precipitation nowcasting, how to decide the hyper-parameter n? (2) It is interesting to see the results in Figure 3 (b). I want to know the CSI, Bias and FVD metrics for the $4\sigma$ and $-4\sigma$ models.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

They have discussed limitations.

Summary

This paper applies latent diffusion models to the problem of precipitation nowcasting, developing a modular and flexible method for enforcing external constraints during the generating process. Constraints are general in that they can be specified in terms of any function F from the data to the reals, F:(x,y)->R. In the proposed "Knowledge Control" (KC) method, a separate NN is trained to predict F from the noised latent data. Conveniently this can be trained before training the diffusion model; it is then used while training the diffusion model as a regularizer, punishing the diffusion model for generating samples that violate the constraints. Experiments are performed on a simulation dataset (MovingMNIST) to demonstrate the advantage of the modeling choices and the KC method, and compare diffusion models with other deep learning models. The KC method is used to enforce conservation of energy, and the experiments demonstrate that KC is effective at constraining the generated samples without interfering with the other performance metrics. The model is then applied to the SEVIR precipitation dataset and compared against a number of other deep learning models. Experiments demonstrate the use of the KC method in controlling the overall amount of rainfall.

Strengths

- The knowledge control method will be of high interest to those applying diffusion models to scientific applications, where there are often physical constraints. - Experiments are well done and will be valuable to researchers in this area. Experiments are done on both simulated and real data, with multiple performance metrics, and comparisons to strong baselines. - The paper is well-written.

Weaknesses

None.

Questions

- In experiments, deterministic models are compared against probabilistic models. For the probabilistic models, a single sample was generated from the probabilistic distribution in each forecast. The probabilistic models perform worse than the deterministic models on some metrics. Isn't it likely that the performance of the probabilistic models would be improved if multiple samples were drawn from them?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors concisely and fairly address the limitations.

Summary

This paper introduces a new method for doing precipitation nowcasting. The method is based on building a conditional diffusion model in a global latent space, which is found separately in an autoencoding fashion. Additionally, a “knowledge control” mechanism is introduced to help ensure that the generated samples satisfy known physical constraints. Numerical validations are conducted on a synthetic and a real precipitation dataset.

Strengths

* The problem of precipitation nowcasting is highly significant and has a lot of potential for real-life impact. * The paper is generally well written. I find it easy to understand. Tables and figures are informative. * Lots of details for reproducibility.

Weaknesses

* The model itself is not the most novel. Both latent diffusion and knowledge control (similar to classifier guidance) are not new. This is somewhat compensated by the fact that the model is being applied to a specific and impactful domain. * Despite the seemingly rich set of baselines being compared, some of the most recent SOTA models developed specifically for the precipitation nowcasting problem are missing (despite being mentioned in the paper): Metnet2 (ref 5), DGMR (ref 28) - these models were applied to real weather data, but one should be able to adapt them to the 2nd benchmark dataset considered here. It is important to know where the model stands compared to these, especially considering that the methodology alone does not qualify for accept due to lack of novelty. * The use of the FVD metric is questionable (see questions below). * Value of knowledge control on the precipitation dataset is unclear. * A few other places that need further clarifications (see questions section). I feel in its current state these outweigh the strengths so I vote for borderline reject.

Questions

* Ln 47 - “AR models” refer to autoregressive models? I didn’t see this acronym defined * Equation (1) - what is $p_\mathcal{E}$ exactly? I presume $y$ gets encoded through $\mathcal{E}$ and how do you draw $z_\text{cond}$? * Ln 196: “.. by comparing the statistics of feature vectors extracted from real data and generations” - what is this real data? The references cited on ln 201 (in audio and chemical applications) both use embeddings trained on their respective domain-specific datasets. It is meaningless to compare against references not related to the specific tasks since the feature distributions will be completely different. * Ln 209-210: “.. generate predictions with sharp and clear digits” - this is too handwavy to me. Although the statement looks qualitatively true, a quantitative measure would make it much more rigorous. A potential option is to look at Fourier spectrum -‘blurry’ images tend to show underestimated energy in high frequency modes. * Ln 299-300: “.. highlighting possible extreme cases like rainstorms and droughts” - this is not easy to verify by looking at Figure 3 for someone without domain knowledge. Additional labels or captions would help. * I do not fully understand the purpose of knowledge control discussed in section 3.2.2. Is it only meant to encourage the generation of extreme scenarios? Do you have ways to evaluate them besides FVD?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors mention limitations related to potential misalignment between visual quality and scores, as well as dependence on data quality and quantity.

Summary

This work proposes a latent diffusion model called PreDiff for spatiotemporal forecasting. The main contributions of this paper are two-fold: 1) they use a conditional latent diffusion model to handle the uncertainty of spatiotemporal forecasting. 2) They introduce an explicit knowledge control mechanism to align the model with domain-specific prior knowledge.

Strengths

1. This paper is well-written and easy to follow. 2. The design of the knowledge control mechanism is interesting. It is decoupled with the latent diffusion model, thus providing more flexibility to the whole framework.

Weaknesses

1. My main concern is the experiments. This paper does not conduct enough ablation studies, and the effectiveness of the diffusion model cannot be supported by experiments shown in Tables 1 & 2. 2. Table 2 only reports the mean of CSI at different thresholds, which is insufficient to prove the superiority of the proposed method. CSI with a high threshold correlates with strong weather, thus, it is important in this work to prove that PreDiff can generate better forecasts, which is also one of the main claims of this paper. I suggest authors list CSI at different thresholds in Table 2 to make a comprehensive comparison. 3. The parameter number of PreDiff is significantly higher than that of EarthFormer (about 15 times). Thus the comparison in Tables 1 & 2 is unfair. So is it possible to reduce the number of parameters to make a fair comparison with other methods?

Questions

Please see the weakness.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

Please see the weakness.